Introduction

Deep reinforcement learning (DRL) has demonstrated that an artificial intelligence (AI) agent can master complex games (e.g., Go and StarCraft) by repetitively playing self-matches^1,2. Using a similar approach, Shin et al. recently demonstrated that an AI agent can design several types of radiofrequency (RF) pulses^3,4. In this work, we investigated the following questions: Does the design result change a lot for repetitions? Is each module, RF generation and RF refinement, critical for the final results? Can DeepRF provide a reasonable improvement over a range of design parameters?

Methods

[DeepRF] In DeepRF, RF pulses of a desired profile are initially generated from a recurrent neural network (RNN) agent using DRL (RF generation module, Fig 1a) and then subsequently refined using a neural network-like computational graph⁵, referred to as Bloch graph (RF refinement module, Fig 1b). In our setting, 76,800 episodes (1 episode = 1 RF) were performed for one execution of DRL. Since the results of DRL are well-known to be sensitive to initialization⁶, 50 DRL executions with random initializations were performed to reduce the variance of the DRL results. Therefore, a total of 3,840,000 (= 76,8000 × 50) RF pulses are generated using the RF generation module. Then, the top 256 RF pulses were selected and further optimized in the RF refinement module. For each RF, 10,000 iterations of the refinement were performed and the best pulse (e.g., RF with minimum specific absorption rate (SAR)) was chosen as the final design.
[Consistency study] To check the consistency of the design results of DeepRF, we performed three repetitions of DeepRF designs and compared these results. The design objective of DeepRF was to match the simulated profile of an SLR inversion pulse⁷ (time-bandwidth product (TBW) = 4.3, duration = 5.12 ms, passband/stopband ripples = 1%) while penalizing SAR.
[Ablation study] To test the importance of each module in DeepRF (Fig. 1), we ablated each module and performed three repetitions of the above inversion pulse design using the RF generation module or RF refinement module. The initial input pulses for the RF refinement module were RF pulses of random shapes.
[Scalability study] To verify the scalability of DeepRF for a range of a design parameter, four SLR inversion pulses with different TBWs (4.3, 5.6, 6.8, and 8.1) were designed (duration = 12.8 ms, passband/stopband ripples = 1%), and the corresponding DeepRF pulses of the matched profiles were designed with the SAR regularization.

Results

The results of three repetitions of the DeepRF designs are shown in Fig. 2. The pulse shapes of the DeepRF pulses (1^st and 2^nd columns in Fig. 2) show similarity with each other and the SARs of the DeepRF pulses are the same for all the repetitions (17.5 (mG)²sec) and smaller than that of SLR pulse (19.6 (mG)²sec). The slice profiles (3^rd and 4^th columns in Fig. 2) from the DeepRF and SLR pulses perfectly match. The ripple constraints (= 1%) are satisfied in all cases (5^th column in Fig. 2).
The DeepRF design results with the ablation of each module are shown in Fig. 3 and Fig. 4. When the RF refinement module was ablated (Fig. 3), the pulse shapes from the ablation (1^st and 2^nd columns in Fig. 3) are inconsistent, and the slice profiles from the ablation and SLR were different (3^rd and 4^th columns in Fig. 3). The SARs of the ablation results are higher than the original DeepRF results (22.8 or 23.8 or 22.7 vs. 17.5 (mG)²sec). When the RF generation module is ablated (Fig. 4), although the slice profiles from the ablation and SLR were almost the same (3^rd and 4^th columns in Fig. 4), the pulse shapes from the ablation (1^st and 2^nd columns in Fig. 4) are inconsistent. The SARs of the ablation results are higher than the original DeepRF results (18.1 or 18.0 vs. 17.5 (mG)²sec), demonstrating the necessity of combining the RF generation and RF refinement modules.
Finally, the DeepRF design results with the different TBWs are shown in Fig. 5. For all the designs, the pulse shapes of the DeepRF pulses are clearly different from those of the SLR pulses (1^st and 2^nd columns in Fig. 5). The slice profiles from the DeepRF and SLR pulses are almost identical (3^rd and 4^th columns in Fig. 5). The SARs of the DeepRF pulses are smaller than those of the SLR pulses in all TBWs (1^st column in Fig. 5).

Discussion and Conclusions

In summary, our study demonstrates that the DeepRF designs are fairly consistent for repetitions, and the DeepRF designs with the ablation of the RF generation module or RF refinement module were inferior to the original design. Additionally, the DeepRF designs were successfully performed over a range of the design parameter, TBW. Those results suggest that DeepRF is a reliable and scalable RF pulse design method.

Acknowledgements

This work was supported in part by the National Research Foundation of Korea under Grant NRF-2018R1A2B3008445 and in part by the Samsung Research Funding and Incubation Center of Samsung Electronics under Project SRFC-IT1801-09.