Machine‐Learning Based Identification of the Critical Driving Factors Controlling Storm‐Time Outer Radiation Belt Electron Flux Dropouts

Understanding and forecasting outer radiation belt electron flux dropouts is one of the top concerns in space physics. By constructing Support Vector Machine (SVM) models to predict storm‐time dropouts for both relativistic and ultra‐relativistic electrons over L = 4.0–6.0, we investigate the nonlinear correlations between various driving factors (model inputs) and dropouts (model output) and rank their relative importance. Only time series of geomagnetic indices and solar wind parameters are adopted as model inputs. A comparison of the performance of the SVM models that uses only one driving factor at a time enables us to identify the most informative parameter and its optimal length of time history. Its accuracy and the ability to correctly predict dropouts identifies the SYM‐H index as the governing factor at L = 4.0–4.5, while solar wind parameters dominate the dropouts at higher L‐shells (L = 6.0). Our SVM model also gives good prediction of dropouts during completely out‐of‐sample storms.


Introduction
The Earth's radiation belt electrons exhibit various acceleration and loss processes, especially during geomagnetic active time (Li & Hudson, 2019;Ripoll et al., 2020;Tu, Li, et al., 2019).Among these processes, the relativistic electrons in the outer belt, also known as "killer" electrons due to their deleterious effects (Baker, 1998), can exhibit fast and significant losses, known as dropouts (Onsager et al., 2002;Turner et al., 2012).Understanding and predicting the physical processes that control these dropouts has been a fundamental question in space physics due to its scientific and practical importance, which has always been a challenge since various mechanisms can operate simultaneously and interact in a nonlinear way.
There are two major loss processes driving outer belt electron flux dropouts: electrons are either lost through the magnetopause into the interplanetary space due to the magnetopause shadowing with enhanced outward radial diffusion (Bortnik et al., 2006;Ma et al., 2020), or lost to the upper atmosphere via interacting with various plasma waves (Blum & Breneman, 2020;Drozdov et al., 2022;Lyu et al., 2022;Usanova et al., 2014).Recent statistical studies reported the important impact of high solar wind dynamic pressure (P SW ) and southward interplanetary magnetic field (IMF) B z on producing significant relativistic electron dropouts (Gao et al., 2015;Gokani et al., 2022;Hua et al., 2023;Onsager et al., 2007;Yuan & Zong, 2013).Moreover, Hua et al. (2023) suggested that dropouts strongly depend on storm (SYM-H index) and substorm (AE index) conditions.The SYM-H index, equivalent to Dst index, measures the ring current intensity near Earth (Wanliss & • We investigate the critical driving factors controlling dropouts by constructing dropout prediction models using Support Vector Machines (SVMs) • The most informative (critical) inputs controlling dropouts are SYM-H at L ≤ 4.5 and solar wind drivers at L = 6.0 with mixed impact in between • Our ultimate best SVM models can capture the observed relativistic and ultra-relativistic dropouts during completely unseen storm events

Supporting Information:
Supporting Information may be found in the online version of this article.Showalter, 2006).While Boynton et al. (2017) demonstrated the main governing factor of IMF B z for dropouts at energies >1 MeV at L ∼ 4.2, Boynton et al. (2016) suggested the dominant role of both IMF B z and P SW at L = 6.6.The statistical study of Xiang et al. (2018) suggested that electromagnetic ion cyclotron (EMIC) wave scattering dominantly causes dropouts at low L* region during the most active time, while dropouts at high L* region are due to both EMIC wave scattering and magnetopause shadowing accompanied by enhanced outward radial diffusion.Although these studies, based on traditional statistical analysis of satellite data, have comprehensively analyzed the dependence of dropouts on various driving factors, the traditional way has important limitations in that it is difficult to isolate the impact of different driving factors.These different driving factors are intrinsically correlated with each other in a nonlinear way and may nonlinearly interact with each other to produce different effects at different locations and phases of the storm (Ma et al., 2024).The dominant driving factors and physical mechanisms that cause dropouts are still not fully understood.
Since these nonlinearly correlated driving factors and their nonlinear interactions produce dropouts, to unravel these nonlinear correlations between various driving factors and dropouts, we take advantage of machine-learning techniques that can associate inputs with outputs in a nonlinear way, which have been widely used in space weather modeling and forecasting (Bortnik et al., 2016(Bortnik et al., , 2018;;Chu et al., 2017;Ma et al., 2022;Wing et al., 2022), and in discovering the important underlying/missing physical processes (Camporeale et al., 2022;Ma et al., 2023Ma et al., , 2024)).In this letter, we employ Support Vector Machines (SVMs) to construct storm-time electron dropout prediction models over L = 4.0-6.0 using dropout data set based on 5-year Van Allen Probes observations from our previous study (Hua et al., 2023).The SVM models take a time history of different driving factors as inputs.Therefore, we can relate the inputs (i.e., driving factors) to the predicted outputs in the most general and nonlinear way.By comparing the model performance of SVM models that use different driving factors with various lengths of time history (dt) as inputs, we identify the most informative input parameters along with their optimal dt, which are regarded as the critical driving factors of dropouts.The good performance on the test set and the out-of-sample storm events suggests that our model performs reliably and can provide accurate prediction of dropouts.Our machine-learning based analysis is novel from the dual points of view of scientific insight and novel analysis technology, and represents a unique way to get physical insights of critical driving factors from data.

Data
In this study, we adopt the same dropout data set from our previous statistical study (Hua et al., 2023) based on Van Allen Probes (Mauk et al., 2013) observations during all the geomagnetic storms with (SYM-H) min < 50 nT during 2013-2017.There were 110 storm events selected in this study.The detailed dropout selection criteria are provided in Hua et al. (2023).Specifically, we performed superposed epoch analysis of electron fluxes in each storm using the Energetic Particle Composition (ECT; Spence et al., 2013) combined spin-averaged, crosscalibrated fitted data (Boyd et al., 2019).The time of minimum SYM-H ((SYM-H) min ) was selected as t epoch = 0.After binning fluxes into a grid of 0.1L × 6 hr UT in four MLT regions: 00-06, 06-12, 12-18, and 18-24, the dropouts were selected by requiring the flux to either decrease by a factor of >4 within 6 hr, or a decrease by a factor of >4 in 12 hr while the flux decay by a factor of >1.5 in previous two successive time steps.This helps to capture both rapid and more extended dropout events.Hereafter, one dropout sample represents dropout at a given L-shell and MLT region in one 6-hr time bin.We adopt SYM-H, AE, P SW , solar wind density (N P ) and speed (V SW ) at 1-min resolution from OMNI data.

Model Description
The SVM model, first presented in 1995 (Cortes & Vapnik, 1995), has gained great popularity in various applications including remote sensing (Mountrakis et al., 2011), predictions of renewable energy sources (Zendehboudi et al., 2018), and disease diagnosis (Wang et al., 2021) due to its speed and accuracy in solving nonlinear problems with high-dimensional spaces and its robustness when only limited reference data is provided, which is often the case in the real-world problems.SVM is a supervised non-parametric statistical machine learning algorithm.The SVM approach aims to search for the optimal hyperplane (also known as separation margin) that separates the data set into a discrete predefined number of classes.This optimal hyperplane refers to the decision boundary that maximizes the distance between the closest training sample and the separating hyperplane.Numerous SVM tutorials are available in the literature (e.g., Burges, 1998;Chang & Geophysical Research Letters 10.1029/2024GL108268 Lin, 2011;Cortes & Vapnik, 1995;Melgani & Bruzzone, 2004).Considering that the dropout data set is a relatively small data set with tens to hundreds of dropout samples at a given L-shell and energy (Hua et al., 2023), and the time series of various driving parameters will be used as inputs (i.e., a high-dimensional problem), SVM is particularly appealing for dropout predictions.
We employ the SVM model in sklearn.svm(Chang & Lin, 2011) to construct dropout prediction models for each L-shell and energy over L = 4.0-6.0 with a step of ∆L = 0.5 for 891 keV and 2.2 MeV electrons, representing the relativistic and ultra-relativistic electron populations, respectively.We treat the dropout prediction as a classification problem, where we use 0 and 1 to represent "no dropout" and "dropout" samples as the model output (target), respectively.The sigmoid kernel function is used in our SVM model.Note that we tried several different SVM kernels including the polynomial, the radial basis function (RBF), and the sigmoid kernels.While the sigmoid kernel produces overall better performance comparing to others, the polynomial and RBF kernels can also lead to reasonable performance.We sample the whole data set during the 2 days of the storm within [t 0 1, t 0 + 1] (in units of days, and t 0 corresponds to (SYM-H) min ), when the most pronounced dropouts were observed (see Figure S1).In this way, the number of no-dropout data points can be significantly reduced so that the SVM model will focus less on predicting the no-dropout class.Nevertheless, the sampled data set is still highly imbalanced since the no-dropout points outnumber the dropout samples.Therefore, we introduce the key word of "balanced" into the class weight in the training data set, which automatically adjust weights inversely proportional to the class frequency in the model input.
The numbers of the samples of the whole data set at different energies and L-shells vary from ∼1,300 to ∼1,900 due to the satellites' spatiotemporal coverage.For each data sample, the model inputs (features) consist of time series of SYM-H, AE, P SW , N P , and V SW individually or jointly with various lengths of time history window (dt) ranging from 2 to 24 hr with a step of 2 hr.Here, we average these driving parameters into a 5-min time resolution and interpolate in the time domain to fill any data gaps.Additionally, the flag of MLT is also included as input, with 0 and 1 representing night (18-06 MLT) and day (06-18 MLT) side, respectively.Our previous statistics demonstrated that the statistical properties of dropouts strongly depend on MLT, with a much stronger flux decay ratio during active time and a slightly higher occurrence during quiet time on the dayside comparing to nightside.Therefore, it is necessary to include MLT as one of the model inputs.Considering the similar dependence of dropouts on different driving parameters at 18-24 and 24-06 MLT, and 06-12 and 12-18 MLT respectively, the dropouts in our SVM models are split into night (18-06 MLT) and day (06-18 MLT) side.The whole data set is randomly split into training set (70%) and test set (30%) regarding the individual samples.The test set is only used to evaluate the model's out-of-sample performance.By examining the performance of SVM models that use different driving factors with various dt as inputs, we identify the most informative inputs along with their optimal dt, which are regarded as the critical driving factors of dropouts.

Results
We calculate two model performance measures, namely the score and true positive rate (TPR) using the test set.The score represents the model mean accuracy (Tiwari, 2022), defined as the fraction of the correct predictions over the total number of predictions in the test set (n s ): where ŷi is the predicted value of the ith sample and y i is the ground truth value (observations).It is important to note that a larger score does not always equal to a better performance since our data set is highly imbalanced.Due to the smaller number of dropouts comparing to no-dropout samples, a large score can result from a large portion of correctly predicted no-dropout class (i.e., true negative, TN), while the number of correctly predicted dropout class (i.e., true positive, TP) can be small.Therefore, we also consider the TPR (Fawcett, 2006), also known as recall, defined as: TPR = Positives correctly classified Total positives = Correctly predicted dropouts Total observed dropouts . (2) A larger TPR means that the model can correctly predict more dropout samples, which is also our primary concern.
To determine the most important driving factors, we first train SVM models by only using one parameter at a time with various lengths dt. Figure 1 presents the model performance of score (blue) and TPR (red) as a function of dt using various input parameters at indicated L-shells.The models showing good performance (here defined as having score ≥0.80 and TPR ≥ 0.65) are highlighted as the filled symbols.The dt, score, and TPR combinations for the best performance for energies at 891 keV and 2.2 MeV are listed in each panel.The best models represent the models with largest sum of score and TPR.Clearly, V SW is the least informative input, resulting in the worst performance compared to the other parameters.The crucial driving parameters strongly depend on L-shell.From Here, the results below 0.60 are not shown.The dt, score, and TPR for the best performance from good models in black and from poor models in gray are listed in each panel.

Geophysical Research Letters
10.1029/2024GL108268 the first two rows, SYM-H is the most informative input at L ≤ 4.5 that produces the largest TPR (∼0.90) with larger scores (∼0.90) compared to AE, P SW and N p , suggesting the governing factor of storm activity at L ≤ 4.5.Nevertheless, AE, P SW and N p can also lead to good performance, suggesting that different physical processes may operate together in causing dropouts.While strong storm and substorm activities along with enhanced P SW are favorable for the generation of EMIC waves that can cause efficient electron losses (Jun et al., 2019), outward radial diffusion is also enhanced during more active time (Ozeke et al., 2020;Tu, Xiang, & Morley, 2019).From the last row, solar wind parameters (P SW and N P ) are the dominant drivers at L = 6.0, indicating the primary role of magnetopause shadowing accompanied with outward radial diffusion in producing dropouts (Hudson et al., 2014).In the middle region at L = 5.0-5.5,SYM-H and N p lead to similar performance, indicating that both storm and solar wind dynamics are important in causing dropouts.Furthermore, the maximum score and TPR for the best models decreases with increasing L-shell for all driving factors, indicating that different physical processes can be dominant in different dropout events at higher L-shells, whose nonlinear pattern is more difficult for the SVM model to capture.
Among the best models that use different inputs at a given L-shell and energy from Figure 1, we regard the models with largest sum of score and TPR as the models with the best performance using a single driving parameter as input.The visualization of the confusion matrices (Fawcett, 2006) on the test set for these best models is shown in Figure 2, showing ground true labels (from observations) versus predicted labels at the indicated L-shells and energies color-coded by the sample numbers.The corresponding inputs along with the optimal dt are listed on the top of each panel.For 891 keV electrons (Figures 2a-2e), although the SVM model is a relatively simple machine-learning model and the data set is highly imbalanced as shown by the no-dropout samples which are vastly outnumbered the dropout samples, our SMV models using only one driving parameter demonstrate good accuracy in predicting the dropouts with highest score reaching 0.9 at L = 4.5.Moreover, most of the observed dropouts can be correctly predicted as suggested by the number on the top left corner in each panel (number of TP) compared to the number on the bottom left corner (number of false negative, FN).Therefore, our models have relatively good ability to distinguish between the dropout and no-dropout classes.Nevertheless, we have to note the large number of falsely predicted dropout (false positive, FP) shown on the top right corner, suggesting the over prediction of dropouts by our SVM models, possibly due to the manually introduced class weight and the data sampling during the pronounced dropout period.The results for 2.2 MeV electrons show similar trends (Figure 2f-2j).Therefore, our SVM models can give good predictions of whether relativistic and ultra-relativistic electron dropouts will occur by only using the time history of a single driving parameter.
Based on the best models shown in Figure 2, we further add a second driving parameter as input and find its optimal dt to improve the model performance.We iterate this process for the third and the fourth driving parameters.Figures 3a-3d serve as an example to illustrate this process to progressively improve the SVM model at L = 4.0 for 891 keV electrons.Figure 3a is the same as Figure 2a, where we use the time history of SYM-H with dt = 22 hr as input.After trying different second inputs including AE, P SW and N p with dt varying from 2 to 24 hr each time, adding AE with dt = 4 hr as the second input gives the largest sum of score and TPR.This model is regarded as the best model using two input parameters (Figure 3b), where both score and number of correctly predicted dropouts have been improved.Figures 3c and 3d display the results by adding the third and fourth inputs.The performance stops improving when adding the fourth input, meaning that the fourth input (N p ) is less informative compared to the first three inputs, or rather that all its information has already been captured by the previous three inputs.The ultimate best model for dropout predictions at L = 4.0 for 891 keV electrons is the model that uses three inputs (Figure 3c), which gives the largest score and the largest number of correctly predicted dropouts.Similar improvements are performed for SVM models at L = 4.0-6.0 for the 891 keV and 2.2 MeV electrons, with the results of the ultimate best models shown in Figures 3e-3n.Overall, adding more input parameters mainly reduces the number of falsely predicted dropouts (FP) compared to Figure 2, which is important in potential applications in space weather prediction, whilst having slight improvements in the correctly predicted dropouts (TP).This confirms the overall governing effect of the first input on producing the dropouts.Still, we note that the number of FP is relatively large compared to that of TP.
Figure 4 shows the application of our best, multi-parameter SVM models in dropout predictions.It is important to note that the storms selected here during 2018 are completely out-of-sample storm events.Because the AE index from OMNI is not available during these storms, we adopt the SME index (Newell & Gjerloev, 2011) from SuperMAG (Gjerloev, 2012), comparable to the traditional AE index but not identical, which may introduce some errors.Figures 4a-4d show the evolution of P SW , N p , SYM-H, and SME indices during the 20 April 2018 storm.Figures 4e and 4f present the spin-averaged electron fluxes at the color-coded L-shells at 891 keV and 2.2 MeV, with the time corresponding to the predicted dropouts marked as triangles.Although there are some falsely

Geophysical Research Letters
10.1029/2024GL108268 predicted dropouts after t epoch = 0, most of the fast and significant flux decay corresponding to the fast decrease in SYM-H and spikes in P SW and N p are successfully captured by our SVM models.Our SVM models also successfully capture the dropouts during the 6 May 2018 storm (Figures 4g-4l).Therefore, our SVM models can predict dropouts for the completely unseen storm events.

Conclusions and Discussions
In this study, we employed a machine-learning approach to investigate the critical factors driving storm-time outer radiation belt electron flux dropouts over L = 4.0-6.0 using 5-year Van Allen Probes observations.By comparing the performance of SVM models that use a single driving factor with various time history lengths dt as inputs, we attempt to identify the most informative inputs along with their optimal dt, which are regarded as the critical driving factors of dropouts.We further improve the model performance by adding one more input parameter each time and determine their optimal dt.The ultimate best SVM models give good predictions for completely unseen storms.Apart from using machine-learning approach as a black box for dropout predictions, more importantly, this machine-learning technique enables us to extract physical insights by isolating different drivers and ranking their importance via comparing the model performance, which is a unique way to relate the inputs to dropouts nonlinearly comparing to the more traditional statistical methods that predominantly rely on linear correlations.The major conclusions are summarized as follows: 1.The critical driving factors of dropouts strongly depend on L-shell, showing the governing factor as SYM-H at L ≤ 4.5 and solar wind drivers (P SW and N p ) at L = 6.0 with mixed impact at intermediate L-shells.This suggests that storm activities have dominant impact on dropouts L ≤ 4.5, possibly through wave-driven scattering to the atmosphere, while solar drivers predominantly affect the dropouts at higher L-shells, possibly via magnetopause shadowing.2. The SVM models that use single inputs demonstrate good mean accuracy (score = ∼0.8-0.9) and ability to correctly predict dropouts (TPR = ∼0.7-0.9).Adding more inputs mainly reduces the falsely predicted dropouts (FP), which is important in accurate space weather prediction, whilst showing slight improvement in correctly predicting dropouts (TP), which confirms the governing effects of the first input parameters.3. Most of the observed dropouts of both relativistic and ultra-relativistic electrons during the completely out-ofsample storms during 2018 are captured by the best multi-parameter SVM models, suggesting the feasibility of prediction of dropouts only based on geomagnetic indices and solar wind parameters, which is important for future radiation belt forecasting.
It is worth noting that our study is subject to limitations.Since we sample the data set during the 2 days of each storm and manually introduce a "balanced" class weight, our SVM model can focus more on predicting the dropout samples.However, this contributes to the relatively large number of falsely predicted dropouts (FP).
Another common way to handle data imbalance is to resample the training set: either oversample the minority class or undersample the majority class.Dealing with the data imbalance is a ubiquitous problem in space physics since most of the time, the quantity of interest is typically in the minority.This remains as a challenge and future studies are needed to improve the way of tackling data imbalance.

Figure 1 .
Figure1.Model performance indicated by score (blue) and true positive rate (TPR, red) on the test data set using various input parameters, one at a time.From left to right: AE, SYM-H, P SW , N P , and V SW , with different lengths of time history (dt) at L-shells of: (a-e) 4.0, (f-j) 4.5, (k-o) 5.0, (p-t) 5.5, and (u-y) 6.0 for 891 keV (circles, labeled as E1) and 2.2 MeV (triangles, labeled as E2) electrons.The models with good performance (score ≥ 0.80 and TPR ≥ 0.65) are highlighted as the filled symbols.Here, the results below 0.60 are not shown.The dt, score, and TPR for the best performance from good models in black and from poor models in gray are listed in each panel.

Figure 2 .
Figure 2. Visualization of the confusion matrices for the models with best performance using a single driving parameter as input on the test data set at various L-shells corresponding to columns.(a-e) Confusion matrices at indicated L-shells showing true labels (from satellite observations) versus predicted labels for 891 keV electrons color-coded by the sample numbers, with the corresponding input parameter, the optimal length of time history (dt), and score marked on the top of each panel.(f-j) Similar to (a-e) but for 2.2 MeV electrons.

Figure 3 .
Figure 3. (a) Same as panel (a) in Figure 2, with the corresponding driving parameter followed by the length of time history (dt) selected as model inputs given on the top.(b-d) Similar to (a) but for models with best performance by adding one more driving parameter as input each time.Confusion matrices for the models with best performance regardless of numbers of driving parameters used as inputs for (e-i) 891 keV and (j-n) 2.2 MeV electrons at indicated L-shells.

Figure 4 .
Figure 4. SVM predictions of electron flux dropouts using out-of-sample storm events.(a, b) P SW and N p indices during the 20 April 2018 storm.(c, d) SYM-H and SME indices.(e, f) The spin-averaged electron fluxes at the color-coded L-shells at 891 keV and 2.2 MeV from both Van Allen Probes.Here, the time of (SYM-H) min is taken as the epoch 0. The predicted dropouts are marked as triangles.(g-l) Similar to (a-f) but for the 6 May 2018 storm.