On the Considerations of Using Near Real Time Data for Space Weather Hazard Forecasting

Space weather represents a severe threat to ground‐based infrastructure, satellites and communications. Accurately forecasting when such threats are likely (e.g., when we may see large induced currents) will help to mitigate the societal and financial costs. In recent years computational models have been created that can forecast hazardous intervals, however they generally use post‐processed “science” solar wind data from upstream of the Earth. In this work we investigate the quality and continuity of the data that are available in Near‐Real‐Time (NRT) from the Advanced Composition Explorer and Deep Space Climate Observatory (DSCOVR) spacecraft. In general, the data available in NRT corresponds well with post‐processed data, however there are three main areas of concern: greater short‐term variability in the NRT data, occasional anomalous values and frequent data gaps. Some space weather models are able to compensate for these issues if they are also present in the data used to fit (or train) the model, while others will require extra checks to be implemented in order to produce high quality forecasts. We find that the DSCOVR NRT data are generally more continuous, though they have been available for small fraction of a solar cycle and therefore DSCOVR has experienced a limited range of solar wind conditions. We find that short gaps are the most common, and are most frequently found in the plasma data. To maximize forecast availability we suggest the implementation of limited interpolation if possible, for example, for gaps of 5 min or less, which could increase the fraction of valid input data considerably.

Space weather forecasts are typically driven by data obtained upstream of the Earth at the L1 point (e.g., Baker et al., 1990;Chu et al., 2021;Forsyth et al., 2020;Keesee et al., 2020;Smith, Forsyth, Rae, Garton, et al., 2021;Wing et al., 2005), approximately 1.5 million km ahead of the Earth. Given the speed of the solar wind, this provides between 20 and 90 min of warning before solar wind plasma encounters the Earth (Baumann & McCloskey, 2021). To account for the variable time delay between the plasma measured at L1 and the arrival of that plasma at the Earth, many forecast models propagate the measurement to a fixed point relative to the Earth, the bow shock for example, (e.g., Baumann & McCloskey, 2021;Cash et al., 2016). Such methods are present in commonly used scientific datasets, such as the OMNI database (https://omniweb.gsfc.nasa.gov/) (Weimer & King, 2008), and will have implications for data continuity.
Further, most current space weather forecasting models have been developed using post-processed, scientific quality data provided by the spacecraft at L1, or combined data products such as OMNI (e.g., Keesee et al., 2020;Smith, Forsyth, Rae, Garton, et al., 2021;Wintoft et al., 2017). Such data are generally only available after an extended period, beyond the interval in which a timely forecast must be made. Data are also available in near-real time (NRT), usually within 5 min of acquisition by the spacecraft (e.g., Zwickl et al., 1998). However, given limited telemetry and processing the NRT data are an approximation of the full scientific data that will later be available. This may lead to differences in the value that the data take, and also whether data are available for some intervals (e.g., Machol et al., 2013). Nonetheless, the NRT data must be used in order to provide actionable forecasts of upcoming hazardous space weather.
In this work we explore how we may best use the data available in NRT to produce space weather forecasts. We will use the example of the space weather threat to ground based infrastructure through GICs. GICs pose a severe threat to continuous and reliable power network operation in many countries around the world (e.g., Gaunt & Coetzee, 2007;Marshall et al., 2012), particularly at higher latitude locations (e.g., Bolduc, 2002). Even at mid-latitude locations, such as the UK, the estimated cost of a major geomagnetic storm-one that leads to a widespread and long-lasting interruption of electrical supply-has been estimated at billions of dollars a day (Oughton et al., 2017). However, with sufficient warning mitigating actions may be taken that would reduce this cost considerably (Eastwood et al., 2018;Oughton et al., 2019). Therefore, the forecasting and mitigation of large, damaging GICs is a critical endeavor.
In this study we explore the data that are available from the L1 point in NRT and investigate how they may be used by space weather forecasting models. In Section 2 we describe the data, while in Section 3 we evaluate the NRT data, including comparisons to the more commonly used scientific quality data. In Section 4 we show the results of training an example forecasting model on the NRT data, using a model based on Smith, Forsyth, Rae, Garton, et al. (2021). Section 5 then discusses the results in the context of producing space weather forecasts in NRT.

Data
In this study we use and explore the data made available by SWPC (the Space Weather Prediction Center) in NRT from the ACE (Advanced Composition Explorer) and DSCOVR satellites located at L1. ACE launched in 1997 as a part of the NASA Explorer program and has provided solar wind observations since 1998 (Stone et al., 1998). Meanwhile, DSCOVR was a NASA/NOAA mission that launched more recently in 2015. The NRT data is provided at 1 min resolution (Zwickl et al., 1998). For the magnetic field we consider the three GSM (Geocentric Solar Magnetospheric) components of the magnetic field ( , , ) and the total field strength (|B|) (Smith et al., 1998). Meanwhile, for the plasma data we evaluate the derived proton density (n p ), solar wind bulk speed (V) and ion temperature (T i ) (Aellig et al., 2001;McComas et al., 1998).
The ACE spacecraft was the operational real time solar wind monitor at L1 until mid-2017, at which time it was replaced by DSCOVR. In this study we therefore use data from ACE between 1999 and 2015 (inclusive) and from DSCOVR between 2018 and 2020 (inclusive). This selection ensures we are using intervals in which each spacecraft was the primary real time operational solar wind monitor, and therefore prioritized for telemetry downlink. We note that even though DSCOVR is currently the operational real time solar wind monitor, data from ACE may occasionally be used to fill large missing intervals in the DSCOVR data. Due to issues with the spacecraft, DSCOVR data are largely absent between 26 July 2019 and 26 February 2020, however we do not perform any substitution with ACE during this period, and simply use the data that are available. Switching the data between spacecraft that are not co-located or cross-calibrated results in its own challenges that are beyond the scope of this work.
In Sections 3 and Section 4 we use scientific quality data for comparative purposes. For ACE we use H0 data (level 2), that have been re-sampled to the same 1 min cadence as the NRT data. For the magnetic field data this means they are down sampled from a 16s cadence, while the plasma moments are up sampled from a 64s cadence. We note that if we instead down sampled the NRT measurements to 64s prior to running the analysis it would result in very small changes (≤0.4%) to the quantitative comparisons performed. Meanwhile, for DSCOVR we use H0 magnetic field data (down-sampled from 1s resolution), and H1 plasma moments at a 1 min resolution. The DSCOVR H1 plasma moments are currently only available until 27 July 2019, limiting the time interval for which the comparison with the NRT data can be evaluated.

Data Evaluation
The NRT data from the DSCOVR and ACE spacecraft at L1 are available within minutes of recording by the spacecraft and are therefore automatically processed. On the ground further processing steps, such as manual data review, data cleansing, more complex fitting methods and recalibration are used to produce science quality data. It is important to consider any differences between these two types of data when moving space weather models from the science quality data on which they were likely developed/evaluated to the operational NRT data. Though DSCOVR is the current operational solar wind monitor some models require large quantities of training data and thus the historical ACE NRT data are also of interest. Below, we evaluate the NRT data values for ACE and DSCOVR, as well as their continuity.

Data Validity
First, we compare the values returned by the NRT data and the equivalent science quality data that is released later after post-processing. Figure 1 shows a series of two-dimensional histograms of the occurrence of plasma moment values in these data sets for ACE (left) and DSCOVR (right). We note that if either data set (NRT or science) is missing then that interval is not represented in Figure 1. Additionally, in this analysis our base assumption is that the science data are the "correct" values, to which the NRT data represent an initial estimate. Therefore, in an ideal situation we would observe the case where both the science and NRT values are close, and lie along the red dashed lines (or the NRT data are within ±10% of the scientific values). However, we see deviations from this for both the ACE and DSCOVR data.
Inspecting the results for the ACE data (left of Figure 1) we see that the plasma velocity is best represented in the NRT data (Figure 1c), with the histogram most closely following the diagonal line of equality and the vast majority of the data (96%) being bounded by the orange ±10% lines. In contrast, the solar wind density ( Figure 1a) and ion temperature (Figure 1e) are less well captured by the ACE NRT data, showing much larger spreads: only ∼20-30% of both NRT data sets are within 10% of the science values. We also see evidence for a hard-coded lower limit to the NRT ion temperatures (Figure 1e).
Considering the DSCOVR data (right of Figure 1), we once again find that the plasma velocity is best captured by the NRT data ( Figure 1d). The vast majority of the DSCOVR data lie along the red dashed line, with 94% of the NRT data within 10% of the science values. We find similar spread in the DSCOVR density comparison ( Figure 1b) as we did for the ACE data (∼20% within ±10%). Near-real time data from both spacecraft appear to commonly underestimate the plasma density. Finally, considering the NRT temperatures reported by DSCOVR we find 20% of the data within the ±10% lines, slightly less than the 30% found for the ACE data. We note that the DSCOVR spacecraft has been operational for a shorter interval, and so has experienced fewer extreme events (e.g., at larger n p or V). Figure 2 shows a similar comparison of the magnetic field data, comparing the NRT and science data for ACE (left) and DSCOVR (right). Statistically, the ACE data are peaked around the diagonal line of gradient unity, though there is wide spread in values. This suggests that the magnetic field values are often corrected later on, sometimes by 10s of nT. There is noticeably less spread in the comparison of the values (Figure 2a), though interestingly there are a series of values that seem to form a parallel distribution to the red line. This would correspond to a small offset of 5 nT or less, where the NRT data under-reported the value of . There is also evidence of a distribution perpendicular to the line of equality in the results for and , perhaps representing a rotation of the magnetic field in the X-Y plane. We note that despite the spread in the field components reported by ACE, 68% of the NRT measurements of the total field magnitude (|B|) are within ±10% of the science values.
When we compare the DSCOVR results (right of Figure 2), we again find that the NRT data are much more representative of the science data than was observed for ACE, with a clear diagonal distribution dominating the results. As with ACE, the NRT total field strength (|B|) is also very close to that later found within the science data, with 94% of the values being within 10%.

Data Continuity
The magnetosphere is a highly dynamic system that couples to the solar wind on a range of time scales (e.g., Borovsky, 2020;Coxon et al., 2019;Shore et al., 2019). Space weather models therefore often require information regarding the preceding interval of solar wind, rather than a single measurement of the current conditions. Incomplete data-those with communication gaps or missing measurements-may pose a problem for this approach. Data gaps have also been inferred to cause errors in the derivation of coupling functions (Lockwood et al., 2019). There are many reasons why data may be missing from the real time data stream, or should be ignored. Missing data are generally flagged with a code according to the reason, be it an operational consideration (e.g., downlink issues) or problems/failure of the measurement or recording. It may be possible to develop bespoke solutions to account for different missing data flags, however in the following we treat all missing and/or flagged data identically. Figure 3 demonstrates the average yearly occurrence of data gaps with different lengths in the NRT data from ACE (top) and DSCOVR (bottom). The equivalent distributions for the science data sets are shown in black. Starting with ACE, we can see that the most common data gap length in both data sets is 5 min or less, that is, five or fewer data points. We observe that there are nearly 200,000 gaps per year of less than 5 min in the plasma data ( Figure 3b). The ACE NRT magnetic field data is more complete by comparison, and whilst the sub-five-minute data gaps are the most common type, there are only around 4000 of these per year. Comparing the scientific and NRT data, we see that the ACE scientific magnetic field data is much more continuous, with several orders of magnitude fewer gaps per year (of any duration). Meanwhile, for the ACE plasma data we see that there are generally fewer gaps shorter than ∼60 min in the science data, but there are on average more gaps of greater duration, perhaps as longer windows of data are later flagged for removal.

Data Gap Occurrence
Deep Space Climate Observatory NRT data has noticeably fewer data gaps per year than its ACE counterpart, particularly considering the plasma data. However, we do see the same patterns, with sub-five-minute data gaps being the most frequently observed, and the plasma data exhibiting more numerous gaps per year than the magnetic field. For instance, there were around 10,000 sub-five-minute data gaps per year in the plasma data, The results for the science data are provided in black. Note the logarithmic y-axes.
while there are approximately 4000 per year in the NRT magnetic field data. If we consider the equivalent distributions for both the DSCOVR scientific magnetic field and plasma data, we see fewer gaps of less than 5 min, but additional gaps of an hour or longer, suggesting that-as with ACE-longer periods of data are later discarded.

Windowed Data Validity
Space weather models often require continuous data as an input, and many numerical or computational techniques will not return a result should even a single entry be missing. For example, a model may require a window of 30 min of continuous solar wind data as an input (cf. Smith, Forsyth, Rae, Garton, et al., 2021). It is therefore important to examine the fraction of input data windows that gaps would invalidate. This is examined in Figure 4, for the ACE (top) and DSCOVR (bottom) NRT data, plotted in green. In Figure 4 the windows are evaluated as they would present in an operational setting, such that the stride between adjacent windows of data is equal to the cadence of the data (1 min), and therefore the intervals of data tested overlap.
When using the ACE NRT data (Figures 4a and 4b), we can see that if 5 min or fewer of continuous data are required (i.e., the model requires an interval of 5 min of solar wind data as input) then around 90% of the magnetic field data will be available. However, should plasma data also be required then only 55% of input data are valid.
As the required length of input window increases, the percentage of valid data intervals decreases. If two hours (120 min) of continuous input are required then only 75% of magnetic field data suffice, and approximately 1% of plasma data are available.
Comparatively, DSCOVR NRT data are more complete (Figures 4c and 4d), noting the different vertical axis scale. For DSCOVR, should a 5 min window of data be required then 97% of magnetic field data and 96% of plasma data will be valid. Again, this decreases with longer input window lengths, reaching 86% of magnetic field and 72% of plasma data as the input window length reaches 120 min. While this represents a major improvement on ACE, it still has the potential to be a serious issue for forecasting models.
Given how common we have shown short data gaps to be in the NRT data (Figure 3), it is useful to consider how the use of interpolation schemes can increase the quantity of valid, continuous windows of data. Figure 4 shows the application of two maximum lengths of interpolation, in addition to the use of the "raw" data. We can see that if gaps of 5 min or fewer are interpolated (orange stars in Figure 4) then the fraction of valid data intervals increases significantly for both spacecraft and types of data. If the interpolation is permitted over larger gaps of 15 min or less (blue crosses in Figure 4) then the fractions increase again, though this is a smaller improvement than was found for the change from the raw data to interpolation over 5 min or less.

Example NRT Forecasting of Ground Magnetic Activity
We now qualitatively compare space weather forecast models using science quality data and NRT data. The example model we use is the CNN-based (Convolutional Neural Network) model developed by Smith, Forsyth, Rae, Garton, et al. (2021), designed to provide a probabilistic forecast as to whether the rate of change of the ground magnetic field (R) would exceed certain fixed thresholds at an observatory on the ground (i.e., may be linked to an enhanced GIC risk). In these examples we use the magnetic field at the Lerwick (LER) magnetometer station in Scotland as our target. For full model details the interested reader is directed to Smith, Forsyth, Rae, Garton, et al. (2021).
The models were trained on data between 2003 and 2014, with 1998-2002 being used as a validation set during training. This means that the data from 2015 to 2016 can be used as an unseen test of model performance, for example, the selected storms discussed below. The models require copious amounts of data to train, covering as complete a record of possible solar wind conditions as can be sourced, and thus we are limited to the ACE datasets. Training a model on the combined ACE and DSCOVR datasets is beyond the scope of this work. Following on from Section 3.2, we interpolate linearly over gaps smaller than 15 min in order to maximize the data availability, and perform this consistently for both the ACE NRT and science data. We note that in an NRT data situation we will not necessarily know the next available value as the more recent time intervals may have no data, but we can repeat the last recorded value for a short interval of time. A forecast horizon of 180 min has been used for both models.
First, we show the 17/18th March 2015 St Patrick's Day storm in Figure 5, which has been previously noted for its ionospheric (e.g., Astafyeva et al., 2015) and ground impacts (e.g., Carter et al., 2016;Kozyreva et al., 2018). We can compare the NRT and science data in the top three panels of Figure 5, from which we can observe some of the results previously discussed in Section 3. It is clear from the velocity and density that there is more rapid temporal variability in the NRT data than is seen in the post-processed ACE data (i.e., Figures 5a and 5c compared to Figures 5b and 5d). This short term-variability would explain some of the scatter in Figure 1. There are also significant data gaps in the science density data that are not present in the NRT data (e.g., a few hours around 1200 UT on the 17th March), suggesting that this data was later removed. On the other hand, there is a data gap in the NRT magnetic field data that appears to have been filled with further post-processing (e.g., around 1800 UT on the 17th March). Overall, there is an extended period during this storm when both the NRT and science-based models are unable to produce a forecast, corresponding to a gap in the NRT magnetic field data and the scientific density data, emphasizing the operational challenges in providing space weather forecasts. This appears to be precisely the kind of interval when predictions of large ground magnetic field variability would be desirable. Nonetheless, both models show increased probabilities during the storm, qualitatively reflecting the ground truth.
Next, we show a second example geomagnetic storm (Figure 6), this time on the 22/23rd June 2015, which was notably observed to have adverse impacts on a mid-latitude railway system (Liu et al., 2016). This storm offers opportunities to further explore several similarities and differences between the scientific and NRT driven models. The increased short-term variability present in the NRT data is once again apparent, and significant intervals of the density data on the 21 June are not present. Inspecting the NRT solar wind velocity data (Figure 6a) we see two anomalous spikes; at 0400 UT on the 21st June and at 1000 UT on the 23rd June. Naively in the NRT data these may initially resemble very large solar wind shocks, to which space weather models may respond. However, we see that the NRT model does not report increased probabilities around these intervals, presumably as the model has seen this anomalous behavior in the training data and only responds to more sustained changes in the solar wind that are corroborated by changes in other solar wind parameters. Regarding the predicted model probabilities, from single case studies we cannot quantitatively assess the performance of the models, however we can see that during the period of disturbed ground magnetic field activity during the storm that the models are reporting elevated probabilities, as would be desirable, and the NRT and science-based model results qualitatively appear similar. We may compare the metrics returned by the models when applied to the "unseen" NRT and science data between 2015 and 2016. Figure 7 shows three metrics as a function of threshold of the ground magnetic field: the receiver-operator characteristic (ROC), precision-recall (PR) score and Brier skill score (BSS). In terms of ROC score, the scientific and NRT models perform very similarly, while in terms of PR and BSS the model based upon the scientific data performs slightly better. We note that due to the presence of different data gaps the metrics are not precisely comparable, and this will be most evident at the highest thresholds which are less frequently exceeded. Nonetheless, the NRT-based model is achieving at least comparable performance to the model employing the science quality data.

Discussion
The magnetosphere reacts to the impinging solar wind on a range of timescales, and the results are heavily dependent upon the history of the coupled system. For example, even the response to fast changes in solar wind dynamic pressure (e.g., Oliveira & Raeder, 2015;Shinbori et al., 2012;Smith, Forsyth, Rae, Rodger, & Freeman, 2021;Yue et al., 2010) may be modulated by the magnetospheric state that was induced in the preceding intervals (Zong et al., 2021). Further, characteristic timescales for the response of the currents linking the magnetosphere and ionosphere, and consequent ground magnetic field variability, range from 10s of min to several hours Shore et al., 2019). Magnetospheric physics therefore dictates that space weather models need to interpret-in some fashion-the history of the coupled solar wind magnetosphere system. There are two main methods in which models can infer the historical behavior: first, they can use statistical properties of a period of data, such as the variability or range of a solar wind property (e.g., Camporeale et al., 2020;Wintoft et al., 2017;Shprits et al., 2019;Smith et al., 2020;Zhelavskaya et al., 2019). Second, models can process entire time intervals of data and extract the necessary information themselves (e.g., Keesee et al., 2020;Kunduri et al., 2020;Smith, Forsyth, Rae, Garton, et al., 2021). Discontinuities and unreliable values in the NRT solar wind data thus pose potential risks and limitations to the validity and utility of these models. Figures 1 and 2 clearly showed that there are often differences between the reported NRT data and the values that are later found in the post-processed science data. We also showed that the velocity is the most reliable plasma moment returned by the NRT data, compared to the density and temperature. Regarding the magnetic field, while there is considerable spread in the reported magnetic field components, the total magnetic field magnitude is mostly within 10% of the reported NRT value. Meanwhile, we note the limited interval for which DSCOVR data were available has constrained the solar wind conditions to which DSCOVR has been exposed. On inspection, we showed examples of two types of "error" that are present in the NRT data: first, the NRT data tend to exhibit  Figures 5 and 6). The error bars represent the 95% confidence intervals calculated from 100 iterations of a bootstrap method (with replacement).

Data Validity
short time scale variability that can be resolved with post-processing; second, there are occasionally very transient anomalous values present in the data (e.g., spikes, Figure 6).
These factors have strong implications for space weather model design and development. The differences between the science and NRT data suggest that-as is best practice-models should be trained upon the data type that will be used in the future. For example, the larger short term variability of the NRT data will need to be found in both the training and evaluation data. We showed that in Figures 5 and 6, the selected model was able to compensate for the additional short-timescale variability in the NRT data. Additionally, artifacts in the data (such as the anomalous spikes in Figure 6) may also be tolerated by some models. However, this is a particular problem if a statistical parameter such as the range of the solar wind velocity is evaluated (e.g., Smith et al., 2020;Zhelavskaya et al., 2019). This may require simple additional processing of the NRT data, for example, through smoothing or averaging, prior to its use by a forecasting model. Space weather models may employ solar wind-magnetosphere coupling functions (e.g., Milan et al., 2012;Newell et al., 2007) in order to assess the geoeffectiveness of the solar wind (e.g., Tan et al., 2018). We note that the selection of coupling function requires careful consideration of the magnetospheric prediction to be made (Lockwood, 2022;Lockwood & McWilliams, 2021). However, one commonality between the various coupling functions is their reliance on the orientation of the interplanetary magnetic field, and therefore the relative sign and value of and , in particular. As we have shown in Figure 2, particularly with the ACE NRT data, the reliability of these components must be considered with care. Of greatest concern are instances showing evidence of and changing sign upon further processing/calibration (e.g., data in the upper left or lower right quadrants of Figure 2). Table 1 shows the percentage of magnetic field data that has an inconsistent sign between the scientific and NRT data sets. We can see that the fractions are quite considerable, and that-as in the previous evaluations-the DSCOVR data are more consistent than that from ACE. The reliability (and short term variability) in other NRT solar wind parameters, such as the density, should also be considered. These would result in large fluctuations of the coupling functions that would not be present in the post-processed science data.

Data Continuity
To provide a useful and practical NRT forecast of geomagnetic conditions it is important to maximize the length of time for which it is available. Many magnetospheric models require several hours of historical measurements order to provide a result (e.g., Bortnik et al., 2016;Keesee et al., 2020;McGranaghan et al., 2020;Smith, Forsyth, Rae, Garton, et al., 2021;Wintoft et al., 2015). The implementation of such models requires careful design, given the nature of the input data available. Additionally, many space weather models will also wish to maximize the data available to allow the largest possible training data sets.
In this work we have shown that data gaps are more numerous in the NRT data sourced from ACE than they are in the data from DSCOVR. Therefore, if only a limited quantity of training data is required then the DSCOVR data would appear to be a good choice. However, the ACE NRT data covers a much longer period of time. This is important as heliospheric conditions and space weather vary over the solar cycle (Chapman et al., 2018(Chapman et al., , 2020Luhmann et al., 2002) and between cycles Lockwood et al., 2014;Reyes et al., 2021), while rare, extreme events are the most concerning (Kilpua et al., 2015;Owens et al., 2021;Thomson et al., 2011;Vennerstrom et al., 2016;Wintoft et al., 2016). For this reason, the longer time-span ACE NRT data is an appealing training data set despite its limitations.
We also showed that short data gaps are the most common, for both ACE and DSCOVR NRT data. These frequent but short data gaps significantly impair the continuity of the data. For example, if an hour of continuous data is required as a model input then only 3% of ACE NRT plasma data is valid (Figure 4b). A simple solution is to interpolate over short data gaps. Interpolating over gaps of 5 min or less significantly increases the portion of valid ACE NRT plasma data to 80% (if an hour of continuous data is required). Previous forecasting models, using the somewhat analogous one-minute resolution OMNI data, have applied simple linear interpolations over gaps of 10 (Keesee et al., 2020) or 15 min (Smith, Forsyth, Rae, Garton, et al., 2021;Wintoft et al., 2015) to good effect. Figure 4 also shows that the data gaps are far more prevalent in the plasma data, and so models have been developed that only use the magnetic field data (e.g., Wintoft et al., 2015). Such models may give slightly impaired performance, but they would allow forecasts to be produced during times when the more complete models would otherwise be unavailable. Additionally, techniques such as sub-sampling of the available data (e.g., McGranaghan et al., 2020) may be implemented in such a way as to reduce the effects of short data gaps.
While space weather forecasts have often employed simple (e.g., linear) interpolation methods, reconstructions of the solar wind data using statistical methods have also shown great promise (Kondrashov et al., 2014). A "pattern matching" approach could also be employed to identify historical analogous intervals to provide a surrogate input (cf. Haines et al., 2021). Meanwhile, longer time-scale reconstruction of solar wind data has also been performed using ground based indices (Kataoka & Nakano, 2021;Machol et al., 2013) and magnetosheath data (Nabert et al., 2015). However, while ground magnetometer data may be available in NRT, filling NRT data gaps with such reconstructions may be challenging in an operational forecast.
If interpolation is employed then it is useful to consider the solar wind autocorrelation timescales, in order to maximize the data available while minimizing the impact of the interpolation on the quality of the input data. The autocorrelation describes how similar a time series is with a lagged version of itself, and is evaluated for a series of different time offsets, or lags (Appendix A). Figures 8a and 8b show the autocorrelation and partial autocorrelation of four selected NRT solar wind parameters. We will assess these qualitatively. The precise autocorrelations depend on solar wind conditions, but such an analysis is beyond the scope of this work. The autocorrelations have been calculated for the longest continuous intervals of DSCOVR NRT data, though we note that the results do not change significantly based upon the time period selected (so long as it is of sufficient duration and includes both dynamic and quiescent solar wind).
The solar wind properties tested all show significant autocorrelations beyond the 60 min shown in Figure 8i, though we do see that the autocorrelation of has nearly reduced to the 95% confidence bound by 60 min. This suggests that-in a broad statistical sense-gaps shorter than 60 min could be reasonably interpolated. However, this completely neglects the importance of short-timescale variability in the coupling of the solar wind and magnetosphere, for example, rapid changes in parameters found during phenomena such as interplanetary shocks. Therefore, while quiescent solar wind data may undoubtedly be interpolated over relatively large intervals, such large interpolation is unadvised during the active intervals of most importance to space weather models.
An additional complication is created if the data gap is present in the most recently acquired NRT data. Consequently there will be no opportunity to interpolate (e.g., linearly) as one side of the data gap is unknown. In this case techniques such as repeating the last recorded value can be performed, and we should consider the partial autocorrelation: it ignores the data within the lag period (i.e., it treats the lag period as a gap). For the solar wind velocity and density (Figures 8aII and 8bII) the partial autocorrelations diminish rapidly, and are within the 95% confidence intervals by around 15 and 5 min, respectively. Meanwhile, for the solar wind magnetic field magnitude and (Figures 8cII and 8dII) the significant partial autocorrelation timescales are short at around 8 and 5 min, respectively. From this we can conclude that-again a statistical sense-repeating the last recorded value for approximately 5 min will not hugely impact the data quality, while performing this for 15 min or more will begin to become problematic, particularly in active geomagnetic conditions. It would be possible to use this kind of analysis to set variable interpolation limits separately for the magnetic field and plasma data, if models only required one type of data (cf. Wintoft et al., 2015). We emphasize that these are statistical results on a long time interval of continuous data, and therefore likely not representative of more extreme or highly transient solar wind conditions under which data gaps may be more likely (e.g., due to instrument saturation effects). Additionally, these autocorrelations have been calculated from the NRT data which, given the short-time scale variability we have shown above, may cause this to underestimate the "true" solar wind autocorrelation timescales. Nonetheless, these provide a useful estimate of approximate interpolation timescales for the NRT data.
Generally, linear interpolations have been employed in the past (e.g., Smith, Forsyth, Rae, Garton, et al., 2021;Wintoft et al., 2015), however in the future more complex interpolation methods may give more confidence to interpolating over larger data gaps, for example, using similar historical analogs , or the use of auto-regressive models, given the high level of autocorrelation observed. Nonetheless, future space weather missions should look to minimize NRT data gaps, those due to instrument saturation effects for example, (e.g., Nicolaou et al., 2020).

Summary and Conclusions
Space weather hazards pose a severe threat to infrastructure such as electricity networks. Recognizing this threat, models have been developed to forecast magnetospheric parameters, such as indices (Chakraborty & Morley, 2020;Tan et al., 2018;Tasistro-Hart et al., 2021) or direct space weather consequences such as auroral-related particle precipitation (McGranaghan et al., 2020), radiation belt enhancements , or ground magnetic perturbations (Keesee et al., 2020;Smith, Forsyth, Rae, Garton, et al., 2021;Wintoft et al., 2015). Such space weather models are most often trained and evaluated on post-processed scientific quality data from spacecraft upstream of the Earth at the L1 point. In this work we have assessed the validity and continuity of the Near-Real-Time (NRT) data that is available much more rapidly, and showed an example of a forecasting model using such data.
When the NRT data are compared to the post-processed, science quality data we found that the NRT data are subject to greater short-term variability and occasional anomalous values. Nonetheless, the solar wind velocity is mostly accurate to within ±10%. While the NRT solar wind density and temperature are generally close to the more processed values, they are also subject to greater uncertainty. Regarding the magnetic field, the total field magnitude is generally reported to within 10% of its processed value. Meanwhile, there is a greater spread in the individual components of the field, including some occasions when the sign of the and is later corrected. Some models may be able to compensate for the differences between the science quality and NRT data (if they are trained on the appropriate data source), while others may require additional data pre-processing.
When considering an operational space weather model it is important to maximize its availability. To this end we investigated the occurrence and lengths of data gaps that could impede an operational model. We found that short data gaps are frequently found in the NRT data, and are more numerous in the plasma data (compared to the magnetic field data). Additionally, DSCOVR NRT data shows gaps less frequently than the NRT data from ACE. Nonetheless, short interpolations (e.g., over gaps of 5 min or less) can dramatically increase data and hence model availability, even when long continuous windows are required.

Appendix A: Autocorrelation Analysis
In Figure 8 we evaluate the autocorrelation functions of several solar wind parameters. Practically, we employ the python statsmodels package. Theoretically, if we let x 1 , x 2 , …x n be the observations of a sample of the time series, we take the mean of the time series as:̄= The sample auto-covariance function for lag h (h < n) is then defined as: We then take the sample autocorrelation function for lag h as: Meanwhile, the sample partial autocorrelation is the sample autocorrelation between x t and x t+h but with the linear dependence of x t on x t+1 through x t+h−1 removed. For more details the interested reader is directed to Brockwell and Davis (2002).

Data Availability Statement
Information on the NRT solar wind data may be found at https://www.swpc.noaa.gov/products/real-time-solarwind, while we use data provided by SWPC but archived by the BGS (British Geological Survey) which may be found at the UK NGDC https://webapps.bgs.ac.uk/services/ngdc/accessions/index.html#item172549. Scientific DSCOVR data may be accessed at https://cdaweb.gsfc.nasa.gov/pub/data/dscovr/, while the scientific ACE data may be accessed at https://cdaweb.gsfc.nasa.gov/pub/data/ace/.