PURPOSE

The equations of motion that govern nuclear magnetic resonance lead to an incredible variety of MRI contrast mechanisms and spatial encoding schemes to be accessed via the application of cleverly constructed sequences of applied magnetic fields. However, the full potential of the Bloch equations has been difficult to exploit due to their non-intuitive, nonlinear dynamics which can devolve into chaotic behaviors and otherwise have intractable, non-analytical solutions [1]. Our previous work [4] introduced a model-free reinforcement learning approach to pulse sequence generation, with an AI agent that explores an unknown MR imaging environment with pulse sequence “actions,” and constructs a model through corresponding RF receive-signal “rewards.” In this work, we demonstrate the same AI agent learning to generate optimal RF waveforms to perform slice selection in unknown inhomogeneous B0 settings.

METHODS AND EXPERIMENTS

Our reinforcement learning framework is composed of an artificial intelligence agent which generates pulse sequence actions that interact with its environment, the time-evolution of the imaging samples’ nuclear magnetization governed by the Bloch equations, which produces rewards based on reconstructed image error, that guide the update and refinement of the agent’s actions (Figure 1). Our Bayesian approach to model this system is composed of pulse sequence actions that are generated from a probabilistic distribution p(X). In order to encourage agent to output pulse sequence actions that respects reasonable mathematical constraint and adapts its complexity with respect to the knowledge about the environment, we model p(X) as a dependent Gaussian process with kernelized Fourier basis prior up to degree 4, and put sparse-inducing prior on the wavelet coefficients. The value function which maps the generated action to a predicted image score is modeled using a Bayesian neural network. In order to balance exploitation and exploration in a principled manner, the updated model posteriorproposes the next set of pulse sequence actions by maximizing an acquisition function, the Expected Improvement (EI)5. The MRI physics simulator developed here was inspired by an open-source GPU-based MRI simulation tool, MRILAB4, 6. A MRI simulation wrapper was developed for Python and the simulation computational kernel was implemented entirely within the Python wrapper for Nvidia CUDA – Pycuda. The kernel was executed in parallel within the GPU environment since the Bloch equations can be solved independently for every voxel of the virtual objects. The inputs of the simulation kernel were 1) a set of 3D objects with 10 spins per voxel and variable spin densities; 2) hardware specifications with imaging magnetic field gradients and 3) the transmitting RF pulse, B1 amplitude and phase were generated by the agent. The output of the simulation kernel was the 1D/2D signal with the real and imaginary components. We performed both 1-D and 2-D experiments with different constraints upon the pulse sequence generator. All experiments was carried out with an RF pulse width of 500 µs, a flip angle = 90º, a frequency offset = 0 and a time-step, Δt of 10 µs. For the 1-D experiment, only the Gx encoding gradient was applied with a slice selection gradient in the slice direction. A linear local B0 variation was applied in the x-y direction (as seen in figure). For the 2-D experiments, the encoding gradients (Gx, Gy and Gz) were set to simulate a typical 2D GRE sequence with TE = 5 ms, TR = 10 s, FOV = 12 cm and matrix size = 44 × 40 and receiver BW = 100 kHz. An inhomogeneous B0 field was also applied in the slice direction. No frequency offset was applied to the excitation pulse. The 1-D experiment constrained a zero frequency offset and a zero RF phase for the excitation pulse. As seen in Figure 2, in this experiment the agent converged at low MSE with the corresponding RF amplitude. As compared to the spin density-weighted ground truth, the two agent-generated RF pulses output a correct slice profile of the different objects even in a inhomogeneous B0 field. For the 2-D experiments, RF frequency offset was still constrained to zero, however the RF phase was relaxed so that it varies from -pi to pi. Even though our system was trained on a ground truth with a completely different contrast (M0 proton density), the generated RF pulse was sufficient to accurately perform the slice selection, such that the resulting image accurately reflects that of a typical GRE sequence without dB0 variation.

Acknowledgements

B.Z. was supported by National Institutes of Health / National Institute of Biomedical Imaging and Bioengineering F32 Fellowship (EB022390).

References

[1] Lin, Yung-Ya, Natalia Lisitza, Sangdoo Ahn, and Warren S. Warren. "Resurrection of crushed magnetization and chaotic dynamics in solution NMR spectroscopy." Science 290, no. 5489 (2000): 118-121. [2] Rund, Armin, Christoph Stefan Aigner, Karl Kunisch, and Rudolf Stollberger. Magnetic Resonance RF pulse design by optimal control with physical constraints. IEEE, 2018. [3] Ma, Dan, Vikas Gulani, Nicole Seiberlich, Kecheng Liu, Jeffrey L. Sunshine, Jeffrey L. Duerk, and Mark A. Griswold. "Magnetic resonance fingerprinting." Nature 495, no. 7440 (2013): 187. [4] Zhu, Bo, Liu, Jeremiah, Koonjoo Neha, Rosen, R Bruce and Rosen S. Matthew. “AUTOmated pulse SEQuence generation (AUTOSEQ) using Bayesian reinforcement learning in an MRI physics simulation environment.” ISMRM workshop Machine Learning, March 2018 and In Proceedings of the 26th ISMRM Meeting & Exhibition in Paris, France 2018. [5] Shahriari, Bobak, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando De Freitas. "Taking the human out of the loop: A review of bayesian optimization." Proceedings of the IEEE 104, no. 1 (2016): 148-175. [6] Liu, Fang, Julia V. Velikina, Walter F. Block, Richard Kijowski, and Alexey A. Samsonov. "Fast realistic MRI simulations based on generalized multi-pool exchange tissue model." IEEE transactions on medical imaging 36, no. 2 (2017): 527-537.