To date, no formal validation of the new ionosphere nowcast system, United States–Total Electron Content (US-TEC), at the Space Environment Center in Boulder, Colorado, has been published. This paper therefore lays part of the validation groundwork by comparing solutions from Magic, the analysis version of US-TEC, with total electron content (TEC) data from the Fast Onboard Recording of Transient Events (FORTE) satellite. The Magic system uses ground-based GPS observations to reproduce a four-dimensional model of the electron density in the ionosphere. From this model, the TEC between any two points at any time can be obtained. The FORTE satellite, on the other hand, detects the arrival time versus frequency of a broadband signal from a transmitter at Los Alamos. The FORTE-observed group delay provides the TEC along the line of sight between the transmitter and the satellite. These FORTE line-of-sight observations can be compared with TEC values over the same lines of sight in the Magic model. A root-mean-square error (RMSE) calculation statistically compares 178 lines of sight. The RMSE indicates a statistical error of 2.87 total electron content units (1 TECU = 1016 el/m2) between FORTE and Magic, using the current operational GPS station list in US-TEC. How much FORTE and Magic individually contribute to this error remains indeterminable, although the errors are expected to be unique to either system and uncorrelated. Individual contributions of each method to the RMSE are estimated by eliminating observations most affected by raypath bending in FORTE and by varying the number of stations in Magic.
If you can't find a tool you're looking for, please click the link at the top of the page to "Go to old article view". Alternatively, view our Knowledge Base articles for additional help. Your feedback is important to us, so please let us know if you have comments or ideas for improvement.
 Estimating the free electron density is of great interest since unmodeled variability in the ionosphere has placed significant limitations on the accuracy of navigation applications. For this reason, efforts by radio scientists to image the electron density distribution in the Earth's ionosphere have been ongoing for decades [Austen et al., 1988]. Some of the early work used tomographic techniques on data obtained from the United States' Navy Navigation Satellite System (NAVSAT) satellites. These initial results produced the first calculation of a two-dimensional ionosphere reconstruction from a network of receivers on the ground. The introduction of the Global Positioning System (GPS) satellites provided a geographically extensive data source that further improved this solution [Komjathy, 1997; Mannucci et al., 1998]. These methods use data assimilation techniques to combine data from various sources and prior model estimates to obtain a statistically optimal estimate of a state vector that models the ionosphere. With these techniques, researchers have expanded their capability to obtain inversions in four dimensions [Yeh and Raymund, 1991; Fremouw et al., 1992; Bernhardt et al., 1998; Bust et al., 2001], while recent research has moved toward improving the real-time specification of the ionosphere [Hajj et al., 2004; Scherliess et al., 2004; Schunk et al., 2004; Wang et al., 2004]. These recent advances in nowcast data assimilation systems have made it possible to determine ionosphere model values in real time, but the accuracy of these new models requires further assessment.
 One such newly implemented, real-time model, United States–Total Electron Content (US-TEC), which estimates ionosphere electron density, has been developed by the National Oceanic and Atmospheric Administration's (NOAA) National Geodetic Survey (NGS), NOAA's Space Environment Center (SEC), and the Cooperative Institute for Research in Environmental Sciences (CIRES) at the University of Colorado in Boulder. US-TEC is currently under installation for operational use at the SEC in Boulder, Colorado. However, to date, no formal validation has been published concerning the accuracy of US-TEC, although a concurrent study is underway [Araujo-Pradere et al., 2007]. The research presented here provides validation groundwork by comparing Magic, the analysis version of US-TEC, with an independent observation system and solution method for the electron density.
 Magic uses GPS observations from the NGS' Continuously Operating Reference Stations (CORS) network or other GPS data sources to construct a four-dimensional electron density model of the ionosphere [Spencer et al., 2004]. The total electron content (TEC) along a line of sight between any two points at any time may be obtained by integrating along the path through the three-dimensional time-dependent Magic model. Any such path in the Magic model may be compared to an independent data source for validation.
 The Fast Onboard Recording of Transient Events (FORTE) satellite [Jacobson et al., 1999] operated by the Los Alamos National Laboratory (LANL), in Los Alamos, New Mexico, provides this independent data source. The FORTE satellite data estimates the path TEC by analyzing the dispersion or difference in group arrival time versus frequency of a radio signal from a pulsed VHF transmitter located at Los Alamos.
 A root-mean-square error (RMSE) statistically compares a set of 178 FORTE line-of-sight TEC estimates with the corresponding Magic model estimate for each equivalent line-of-sight location and time. The RMSE provides statistical error of 2.87 total electron content units (TECU) between each FORTE observation and its Magic counterpart, where 1 TECU = 1016 el m−2. Insufficient information exists in this study to accurately determine the individual contribution by FORTE or Magic to the total RMSE. However, the possible contribution by FORTE and Magic to the total error is examined and discussed on the basis of the results herein.
2. Magic Overview
 The Magic software package was developed to specify the Earth's ionosphere in three spatial dimensions and time. Following the discussion of Spencer et al. , this software determines the ionosphere electron density field by using tomographic techniques to assimilate a collection of slant path measurements from the CORS GPS system. Magic uses a Kalman filter [Kalman, 1960; Kalman and Bucy, 1961] as the data assimilation algorithm and the 1995 International Reference Ionosphere (IRI95) model [Rawer and Minnis, 1984; Bilitza, 1990, 1997; Bilitza et al., 1993] to provide the propagation model for the state in the Kalman filter. The Kalman filter method provides a statistically “best” estimate of the state whose error has been optimally reduced on the basis of the minimum variance form of the least squares solution given time-dependent observations and a prior model estimate of the solution [Tapley et al., 2004].
 The Magic model uses a set of empirical orthonormal functions (EOFs) to characterize the vertical variation in electron density through the ionosphere. A typical set of three EOFs is illustrated in Figure 1. These orthonormal functions are calculated at the start of each day using a singular-value decomposition algorithm based on the vertical density profiles from the IRI95 model. Like the operational US-TEC, exactly three EOFs are used in the Magic model. Increasing the number of EOFs beyond three achieves only negligible improvement in the accuracy. The state vector in the Kalman filter consists of a set of amplitude coefficients for these EOFs, which describes the vertical structure. The amplitude coefficients for each grid point, with a geographic spacing of 1.5° latitude and 4.0° longitude, are calculated every 15 min. The resultant model provides a succinct representation of the electron density field in four dimensions.
 In Magic and US-TEC, the Kalman filter uses the IRI95 model to propagate the state and its associated covariance matrix to the next observation set, every 15 min. The forward propagation of the state is constructed from a weighted linear combination of the relative spatial/temporal gradients in IRI95 and an absolute estimate from IRI95. The Magic results in this paper follow the operational setting for US-TEC, where 90% of the state propagation is based on the change predicted by IRI95 and 10% of this propagated state consists of the climatology prediction directly from IRI95. These proportions follow the settings for US-TEC, although no formal analysis has been performed to determine if these settings are optimal. Correlations are assumed between neighboring grids in the latitude and longitude directions where the amount of correlation decreases with distance, on the basis of a Gaussian function. No covariance, and therefore no correlation, is assumed in the radial direction. Model errors are also included in the propagation of the state covariance. Details can be found in work by Spencer et al. .
 The primary differences between US-TEC and Magic are: US-TEC is a real-time operational system and Magic can be applied to past data sets. Further, being the analysis version, the number of stations available to Magic is adjustable. Otherwise, the solution method for both US-TEC and Magic rely on the same subroutines. An illustration of moderate storm conditions from 22 February 2007, over North America from the Magic/US-TEC system is shown in Figure 2, where 53 CORS stations as well as additional real-time International Global Navigation Satellite Systems (RTIGS) and NOAA-GPS meteorological stations, indicated on the map, were used in the solution.
3. CORS Network
 The GPS data used in this study are a smaller subset of the National Geodetic Survey's (NGS's) Continuously Operating Reference System (CORS) network of GPS stations. The full CORS network includes several hundred GPS stations that operate continuously throughout the world. The currently operating US-TEC ingests data from 66 stations from the CORS network, although this number may be increased in the near future. All 66 stations used in US-TEC and the various station lists in the Magic analyses are located within the United States and Central America. The NGS provides data in receiver independent exchange (RINEX) format files that can be downloaded from the NGS website: http://www.ngs.noaa.gov/. Figure 3 shows a map from the CORS website that gives the locations of the CORS stations that were operational in February 2007. The number of CORS stations continues to steadily increase as new stations join the network.
 Since the CORS stations used by US-TEC and Magic are located only within the United States and Central America, the US-TEC and Magic models are most accurate over these regions. US-TEC and Magic can extrapolate its solution over nearby outlying areas, in part because the low-elevation raypaths from this region extend for several thousand kilometers beyond its edges, but the errors in the US-TEC/Magic solution will increase as one moves away from the station network. Plans are also underway to include more stations in the operational system, US-TEC, particularly in currently data-sparse regions, which should improve the accuracy of the operational system. Results later will show how the variation in the number of stations in Magic affects the RMSE.
Figure 4 illustrates the data used in the US-TEC and Magic solutions. The points are color coded, indicating the line-of-sight TEC value along each raypath, between the CORS station and GPS satellite, as determined by the calibrated phase differences [Spencer et al., 2004]. The key to the color values is shown in the bar at the right in TECU. The plot displays the intersection points where a set of CORS raypaths intersect a hypothetical shell located at an altitude of 375 km. All of the signal path intersection points within a 15-min interval are shown. Signal paths for stations in coastal Alaska and Central America as well as the continental United States can be seen. This plot covers a time interval during the peak of the geomagnetic storm of 29 October 2003. The anomalously high TEC values due to the storm are visible over Central America, the southern regions of the United States, and adjacent areas.
4. FORTE Overview
 The FORTE satellite [Jacobson et al., 1999] provided satellite-based triggering and recording of pulsed VHF signals. FORTE radio recordings were performed continually from 1997 through 2003. The FORTE satellite, at 800 km altitude and 70 deg. inclination, carried two radio receiver/digitizer systems. The first was a pair of 25 MHz bandwidth receivers, with 50-megasample/s recording, that was tunable to anywhere in the VHF (30–300 MHz) range. The second was a single 300 megasample/s system, which sampled an 85-MHz analog bandwidth channel that could also be tuned throughout the VHF. The work described here was performed with this passband located at 0–85 MHz.
 Typically the ionosphere-refracted signal from a radio pulse has a group delay that varies as f−2 for higher frequencies in this passband, approximately f > 50 MHz, but whose variation versus frequency is stronger than f−2 for lower frequencies [Roussel-Dupré et al., 2001]. Ultimately, at the lowest frequency supporting a transmission path from the ground to the satellite, the group delay diverges and cannot be fully described by a power series in 1/f. At this point, an exact treatment of the fully anisotropic dispersion relation is required [Moses and Jacobson, 2004]. In fact, the higher-order divergence of the group delay at this propagation cutoff provides a wealth of diagnostic information about the ionospheric structure.
 For this study, the use of FORTE is confined to those pulse recordings of signals emitted by the Los Alamos Portable Pulser (LAPP), a research facility for transmitting cooperative VHF pulsed signals to satellites, for use in characterizing those satellites' radio receivers [Holden et al., 1995; Massey et al., 1998]. An example of a FORTE recording of a LAPP pulse is shown in Figure 5. Note the splitting of the signal at lower frequencies into two propagation modes, due to the magnetic anisotropy of the ionospheric dielectric [Massey, 1990; Roussel-Dupré et al., 2001]. Prior to analyzing these data for the present study, the problem could be reduced to a series of precise (to within ±0.05 μs) determinations of the group delay for both magnetic modes. These discrete data on group delay were then fitted by the ray trace model. The model outputs a slant TEC that is integrated over the slant line of sight between the LAPP and the satellite. These slant TECs are the FORTE inputs to the current study.
 The Magic solutions are compared to 178 FORTE observations taken on 30 separate days between 30 June 2000 and 27 November 2001. The time distribution of the observations is provided in Table 1. Each pass of the FORTE satellite over Los Alamos lasts less than 10 min, and the FORTE observations are made at 1-min intervals during each pass. An example of data from a single pass and the Magic TEC value corresponding to each FORTE observation is shown in Figure 6. Both the FORTE and Magic values in Figure 6 show the expected increase in TEC at lower raypath elevation angles at the beginning and end of each pass.
 The effect of varying the number of CORS observing stations in the Magic solutions is examined by varying the number of stations used in the assimilation from approximately 20 stations to approximately 130, with these numbers being averages as some of the stations go online and offline during the analysis period. The differences between the FORTE and Magic solutions were examined as a function of raypath elevation angle as shown in Figure 7. Points from a single FORTE satellite pass are connected by the same line color. The variance in the FORTE-Magic difference decreases with increasing elevation angle for all station quantities.
 In the case of the GPS data used in Magic, errors due to the raypath bending are probably negligible because the frequency is much higher (1.5 GHz instead of tens of MHz). However, the uncertainties in the line-of-sight TEC from the GPS observations alone can come from several sources. In obtaining the TEC from the GPS observation, errors are associated with combining the accurate but ambiguous phase and the noisy but unambiguous pseudorange. Other possible error sources include receiver biases, satellite biases, cycle slip correction, and multipath. This total error from all of these processes could well amount to 1 to 3 TECU alone [Mannucci et al., 1998].
 Errors also arise from the inversion method in Magic, which has limited observations of the vertical structure in the ionosphere. The vertical profile is one of the most difficult features to sense with ground-based data since all of the observation raypaths pass through the entire ionosphere. None are restricted to any particular height, which would provide better sensitivity to height variations. Since the altitude of the FORTE satellite is below the maximum altitude of the Magic solution, only a section of the vertical profile in Magic is compared with the FORTE line of sight. In this case, the altitude-dependent structure in Magic becomes important. Inaccuracies in the vertical structure of Magic will appear when compared to the specified altitude range of FORTE. There may also be some error in the Magic vertical structure since the plasmaspheric content is not explicitly estimated in the model state. Figure 6 suggests that this error may increase on slant paths with greater plasmaspheric content.
 Besides errors in the Magic solution, there are uncertainties in the FORTE estimate of TEC that arise mainly from the bending of the ray. The bending accounts for several TECU so that an accurate estimate of the bending is crucial. The ray trace model in FORTE is estimated to probably reduce the error in FORTE to less that 0.5 TECU [Roussel-Dupré et al., 2001; Moses and Jacobson, 2004].
 In Figure 7, the increase in variance with decreased elevation angle closely matches the obliquity function, accounting for most of the variation with respect to elevation. The correlation of the variance to the obliquity function indicates that raypath bending is not a major contributor to the total error. The primary sources of error probably come from overall modeling errors in Magic, the subsampling of Magic to account for the FORTE altitude, and GPS observation errors.
 In Figure 8, the differences between the FORTE and Magic solutions are plotted against the total electron content from the FORTE solution. The effect of increasing TEC values should be similar to increasing elevation angle as errors in the Magic vertical profile are magnified with increasing TEC. Also as before, the variance of the distributions decreases as the number of CORS stations increases, indicating again that the Magic vertical profile description probably improves as more observations are included in its solution. The variance decreases slightly for lower FORTE TEC values, as shown by the reduced RMS for FORTE values below 60 TECU in Figure 8 as well as Table 2.
Table 2. Root-Mean-Square Error Calculations According to Elevation Angle Cutoff, TEC Cutoff, and Number of Stations
Average Number of CORS Stations in the Magic Solution
Average Number of CORS Stations Within 500 km of Los Alamos
Average Number of CORS Stations Within 1000 km of Los Alamos
 To analyze Figures 7 and 8 in a more quantitative manner, a simple root mean square error (RMSE) is calculated as
where Fi is the ith FORTE observation, Mi is the ith corresponding Magic solution, and n is the number of observations. A table of the RMS errors as a function of elevation angle, total TEC, and the total number of CORS stations in the Magic solution is provided in the first column of Table 2. Alongside the total number of stations are the number of CORS stations located within 500 and 1000 km of LAPP transmitter in Los Alamos in the second and third columns, respectively, to provide an indication of how stations in vicinity of the point of interest affect accuracy.
 The fourth column in Table 2 indicates that the total error between the FORTE and Magic solutions can be reduced to less than about 2.8 TECU if a large enough number of stations is used in the Magic solution. The RMS error appears to approach a “floor” around 2.7 TECU as the number of stations increase. This number more correctly provides a “worst-case” scenario value since the systematic errors due to the FORTE signal path bending and Magic sensitivity to vertical profile variations have not been considered. Both sources are probably contributing to the differences, although the Magic contribution is probably dominant.
 To examine the effects of the accuracy in the vertical description, one may consider only data at higher elevation angles between the FORTE satellite and LAPP transmitter. When data with elevation angles below 55° are removed, the RMS error decreases by a little more than 1 TECU on average, as shown in the fifth column of Table 2. Since an increase in the electron density also contributes to the signal path bending, possible errors in the FORTE solution can be minimized by only considering observations with FORTE observations below 60 TECU. The cutoff, 60 TECU, is chosen somewhat arbitrarily but provides a good balance between minimizing bending effects while maintaining a sufficient number of observations for the statistical analysis. These results are shown in the sixth column where the RMS error is reduced to 2.32 TECU when the 60 US-TEC station list is used, and further to 1.71 TECU if the station list is increased to 133. These values correspond to other concurrent studies [Araujo-Pradere et. al., 2007] of Magic and US-TEC, which indicate an average error of about 2.4 TECU when approximately 60 CORS stations are used, including quiet and geomagnetically disturbed conditions.
 To further reduce the errors in the raypath calculation, one may consider only FORTE observations above 55° elevation and below 60 TECU as shown in the seventh column of Table 2. Using the current operational 60-station list, the error is reduced to 1.66 TECU, and with 133 stations the error is reduced to 1.15 TECU. It is important to note that the seventh column still provides the combined error of both solutions. However, how much each solution is contributing to the error still cannot be clearly determined. Additionally, 28 observations are nearing the minimum number of data points for which some may consider a reliable statistic.
 Unfortunately, a sufficient number of FORTE observations are not available to allow an adequate analysis of Magic and FORTE during geomagnetic storm conditions. The FORTE data do, however, include seven days where the ap index exceeds 30 and two days where the ap exceeds 150. Much of the data is taken within days of several large storms during 2000 and 2001, years near the height of the solar maximum. Although this study uses data from disturbed periods, it is not possible to draw conclusions about Magic's accuracy during geomagnetic storms because of the lack of data on storm days.
 The RMSE values in this study show similarities to a concurrent US-TEC/Magic validation study [Araujo-Pradere et al., 2007]. The study by Araujo-Pradere et al. uses a method, termed differential validation, to compute the increase in phase difference error with time for a particular satellite and receiver pair. If one begins with a 0 phase difference at time = 0, the phase differences will increase with time and the inversion error along each raypath will begin to match the actual error in the USTEC/Magic inversion. This differential validation method indicates an average error of about 2.4 TECU when approximately 60 CORS stations are used, which includes quiet and geomagnetically disturbed conditions. This value from differential validation matches, within a TECU, the RMSE values determined in this study as indicated in Table 2 for 60 stations.
 This research provides an estimate of the combined uncertainty in the Magic and FORTE systems. Using the currently operational 60-station list for US-TEC, comparisons of Magic with FORTE indicate an error of 2.87 TECU when all FORTE observations are used indiscriminately. The overall bias for the 60-station case is approximately 0.29 TECU. The vertical error of 1.66 TECU is approximated by considering only FORTE observations at high elevation angles and low TEC. Increasing the station list in Magic to 133 stations decreases the errors further to 2.70 TECU when all FORTE observations are used and 1.15 TECU when FORTE observations are selected on the basis of high elevation angle and low TEC.
 One should point out that several limitations may exist in this study, and ongoing validation of the FORTE and Magic systems will be necessary. One such limitation, for example, comes from analyzing data at a single location only. The LAPP transmitter, located at Los Alamos, New Mexico, lies near the middle of the CORS station network, insuring optimal coverage. It is not known how accuracies will decrease as one moves away from the primary coverage area. Additionally, it is unknown how this system will perform during storm conditions or how much FORTE or Magic contribute individually to the total error, as other examples. Although more validation will be necessary, the results presented here provide some initial insights into the accuracies of the FORTE and Magic systems.
 The work was supported by the Interagency GPS Executive Board (IGEB). The authors wish also to thank Kenneth Davies and Adela Angel of the University of Colorado and the Space Environment Center for their expertise and extensive help on this project.