Purpose

Recent studies have demonstrated the value of deep learning in RF pulse design^1-4. Particularly, deep reinforcement learning (DRL)⁵ was shown able to predict an RF pulse with specified design criteria³. Here, we expand its application to five pulse design scenarios: small-tip-angle excitation, 90-degree excitation, refocusing, inversion and multi-band 90-degree excitation.

Methods

Our DRL pulse design framework entailed an artificial agent learning from interactions with its environment (Fig. 1). The agent was a deep neural network and the environment involved a Bloch simulator.
The training in each episode (i.e. iteration) was as follows: 1) the agent made observation by taking the target slice profile as input and took action by predicting RF pulse; 2) the predicted RF pulse was fed (along with a constant gradient) to the Bloch simulator to produce the corresponding slice profile; 3) the produced slice profile was used to evaluate a reward with which to update the neural network. The neural network was trained by minimizing the loss (instead of maximizing the reward) using Adam algorithm⁶. The loss was calculated by
$$loss=\lambda_{1}\cdot{MSE_{-}M_{xy}}+\left(1-\lambda_{1}\right)\cdot{MSE_{-}M_{z}}+\lambda_{2}\cdot{R_{-}M_{xy}}+\lambda_{3}\cdot{R_{-}M_{z}}$$
where MSE_M_xy and MSE_M_z are mean squared error (MSE) for transverse and longitudinal components, respectively, used to quantify the deviation of the produced slice profile from the target; R_M_xy and R_M_z are the corresponding ripple levels, defined as sum of the min-max magnitude difference in passband and stopband; λ₁, λ₂ and λ₃ are three hyperparameters tunable to adjust the weightings of individual terms.
For ease of training, the target slice profile was relaxed to a trapezoid shape, with the transition band calculated by the SLR algorithm for 0.01 passband and stopband ripple levels⁷.
For each design scenario, an independent neural network was trained using a custom two-stage optimization. λ₁, λ₂ and λ₃ in the loss definition were adapted (Fig. 2) from stage to stage such that the first stage was devised to reduce both MSE and ripples while the second stage to purely minimize MSE. Further, λ₁ was set to a value close to 1 for excitation pulse designs (to consider mostly M_xy) and a value close to 0 for refocusing and inversion pulse designs (to consider mostly M_z).
Although the neural networks in all design scenarios shared the same architecture of 6 fully-connected layers (including 4 hidden layers each with ReLU activation and 256 neurons), they differed in the output layer. The output layer was designed to output the entire RF waveform for inversion pulse design, whereas it was devised to predict only the first half of the RF waveform (with the second half mirroring the first half) for all the other design scenarios requiring a linear-phase slice profile.
For the small-tip-angle scenario, the single neural network was trained by randomly sampling a pool of target slice profiles prescribed for flip angles from 5 to 80 degrees (in steps of 5 degree) to promote its generalizability. In all scenarios, real-valued RF pulses were predicted, the slice thickness was set to 3 mm, and a total of 400 episodes carried out (300 for stage 1 and 100 for stage 2).
Our DRL framework was implemented using the Flux package in Julia⁸ and all networks trained on a Linux workstation using CPU without parallel computation. For comparison, pulses were also designed with the SLR algorithm⁷.

Results

Our DRL method substantially reduced the slice profile MSE for all pulse design scenarios (Fig. 3) when comparing the resultant slice profile against the target trapezoidal slice profile, and MSE was decreased the most for multiband excitation pulse design (by as high as 86%). When comparing the resultant slice profile against the ideal rectangular slice profile, our DRL method outperformed SLR for flip angles up to 90 degrees, with MSE decreased most for small tip angle excitation (by 15%), though slice profile was moderately degraded (by <10%) for refocusing, and slightly degraded (<1%) for inversion.
The improvement in slice profile for small tip angle pulse design was further confirmed by inspecting the resultant slice profiles of representative flip angles (Fig. 4). For a wide range of flip angles, our DRL method gave rise to a slice profile closer to both target and ideal slice profiles, with more flattened passband observed for relatively small tip angles and with less stopband ripples for relatively large tip angles.
Likewise, our DRL method improved slice profiles for single band and multiband excitation pulses with noticeable suppression of stopband ripples, while leading to visually comparable slice profiles for refocusing and inversion (Fig. 5).
The predicted RF pulses, however, are not as smooth as those from SLR and have increased peak B1.

Discussion and Conclusion

We have demonstrated a deep reinforcement learning framework to design various RF pulses. Our results showed that our DRL framework can effectively predict excitation, refocusing, inversion and multiband excitation pulses for a given target slice profile, with improved performance in ripple suppression as compared to the conventional SLR pulse design algorithm.
Part of our future work is to investigate how best to reduce peak B1 and to examine the utility of other neural network architectures (e.g., RNN⁹ and ResNet¹⁰).

Acknowledgements

This work was supported by NIH grants U01 EB025144, and P41 EB015894.