SEARCH

SEARCH BY CITATION

Keywords:

  • ionosonde networks;
  • intelligent archives;
  • data assimilation

Abstract

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Routine Unattended Operations
  5. 3. Data Management at Central Repository Sites
  6. 4. Data Management: Lessons Learned
  7. 5. Quantifying Electron Density Profile Uncertainty
  8. 6. Discussion
  9. 7. Summary
  10. Acknowledgments
  11. References

[1] A comprehensive set of technical solutions has been developed for the distributed network of digisonde ionospheric sounders to make their data available in a timely and user-friendly fashion. In a significant engineering effort, 34 digisondes operating worldwide stand as a modern sensor network with on-site intelligent data interpretation, data delivery to multiple locations for the automated storage and dissemination to end users, and a software suite for interactive data visualization/editing and remote network maintenance and sounding schedule control. The paper discusses lessons learned from the system operations and outlines future efforts.

1. Introduction

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Routine Unattended Operations
  5. 3. Data Management at Central Repository Sites
  6. 4. Data Management: Lessons Learned
  7. 5. Quantifying Electron Density Profile Uncertainty
  8. 6. Discussion
  9. 7. Summary
  10. Acknowledgments
  11. References

[2] Maturing information technologies have brought a quiet revolution to the sensor networks that now routinely deliver real-time data, unattended and autonomously, to decision-making systems of high complexity. In the 1980s, the space weather prediction systems started to rely on real-time information on ionospheric conditions derived from ionospheric high-frequency (HF) sounding data. Whereas originally, the real-time data capability came at a substantial operating and maintenance expense, now the time has come to place a wide spectrum of ionosonde data products on computer screens of anyone who asks. The University of Massachusetts-Lowell digisonde group has expended a considerable development effort in recent years toward seamless source-to-destination performance of an ionosonde network, covering all key issues from unattended field operations to automated data quality control and database management. This paper describes the status of the digisonde networking as an example for any modern ionosonde network.

2. Routine Unattended Operations

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Routine Unattended Operations
  5. 3. Data Management at Central Repository Sites
  6. 4. Data Management: Lessons Learned
  7. 5. Quantifying Electron Density Profile Uncertainty
  8. 6. Discussion
  9. 7. Summary
  10. Acknowledgments
  11. References

[3] The Digisonde 256 [Reinisch, 1996a] and its follow-up model, the Digisonde Portable Sounder (DPS) [Reinisch, 1996b] were designed to operate in an unattended, remotely controlled “glass room” environment of the major sensor networks of the U.S. Air Force [Buchau et al., 1995; Reinisch, 1996a], the Australian projects Jindalee Operational Radar Network (JORN) and Jindalee Facility Alice Springs (JFAS) [Barnes et al., 2000; Reinisch et al., 1997], and the European project Digital Upper Atmosphere Server (DIAS) [Belehaki et al., 2005]. Protocols for remote communication to the digisondes allow the user to monitor their operations, command sounding schedules and programs, perform maintenance and software upgrades, and download measurement data. The DPS sounders periodically determine the health status of their hardware subsystems and report their housekeeping data to remote locations for analysis (Figure 1). With a battery backup, four receivers, and two transmitters, DPS is built to be fault tolerant and to sustain reduced-quality operations in the presence of hardware failures. Health and status data are available for remote troubleshooting via the digisonde home pages. A complete digisonde list with links to the station home pages can be found at http://ulcar.uml.edu/stationlist.html.

image

Figure 1. Digisonde housekeeping data indicating health and status of all its subsystems. This example is for Průhonice, Czech Republic (http://digisonda.ufa.cas.cz/) on 24 March 2005, 2014:48 UT.

Download figure to PowerPoint

3. Data Management at Central Repository Sites

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Routine Unattended Operations
  5. 3. Data Management at Central Repository Sites
  6. 4. Data Management: Lessons Learned
  7. 5. Quantifying Electron Density Profile Uncertainty
  8. 6. Discussion
  9. 7. Summary
  10. Acknowledgments
  11. References

[4] With currently available communication bandwidths, making routine ionosonde data available at a central location on a near-real-time basis is more a question of the operator's policy than a technical difficulty. At one such central repository at the University of Massachusetts-Lowell (UML), arriving digisonde data are immediately put under the management of two databases, the Digisonde Ionogram Database (DIDBase) and the Drift Database (Drift-DB), that separately handle two principal modes of digisonde soundings and ionogram and sky map/drift. It was a major effort to immerse the two databases in a software environment that provides easy manipulation and robust operations.

[5] The current data roster at the DIDBase enlists 50 digisondes (http://ulcar.uml.edu/DIDBase/), and new ionogram data are arriving in near real time from 34 locations and offline from archival CD-ROMs and tapes. The recently established Drift-DB is gaining strength and data volume; the currently available data set includes 22 digisondes, of which 10 report data in near real time.

[6] Figure 2 shows the current arrangement of the computer systems managing the UML digisonde data repository. The DIDBase and Drift-DB databases are maintained by two separate Linux servers running the open-source relational database management system “FireBird,” selected for its superior ratio of strength to management effort. Data ingestion is operated by Java applications DIDBFill and DriftFill that provide a comprehensive quality control of incoming data for format errors and content integrity. The FTP incoming server receives data submissions from online digisondes for immediate ingestion. Both DIDBase and Drift-DB databases are provided with interactive data analysis tools, current standard archival output (SAO) Explorer (http://ulcar.uml.edu/SAO-X/) and Drift Explorer (http://ulcar.uml.edu/Drift-X.html), that connect to the databases for read-write access over the Internet using the thin–client layer technology. A Tomcat server provides read access to the ionogram data via an Internet browser.

image

Figure 2. Digisonde data repository. Ionogram and drift data arrive from the network for quality control and ingestion to DIDBase and Drift-DB databases. Databases are available for queries and read-write operations over the Internet using SAO Explorer and Drift Explorer software.

Download figure to PowerPoint

[7] In addition to the interactive data analysis tools mated to the DIDBase and Drift-DB over the Internet and a suite of ingestion and data management utilities, the data repository runs an automated data request execution subsystem (ADRES) [Reinisch et al., 2004]. The ADRES subsystem serves those ionospheric applications that require ionogram-derived data to be 100% manually verified in order to exclude possible errors of the automated ionogram interpretation by the Automatic Real Time Ionogram Scaler With True Height (ARTIST) software [Reinisch and Huang, 1983; Reinisch et al., 2005]. From the end-user point of view, operating ADRES is as simple as placing a text file containing digisonde location(s) and time interval(s) of interest into the appropriate incoming FTP folder of the ADRES server in Lowell and later collecting SAO files with manually verified scaled data in the corresponding outgoing FTP folder. ADRES accepts requests and arranges all involved steps from acquisition of raw ionogram files to release of the verified data to requesting parties. It can also remotely command selected digisondes to switch to a high-cadence campaign measurement mode for the requested period of interest (e.g., during the overhead pass of a spacecraft). Each ADRES request is registered in the DIDBase, and its completion status can be monitored both locally at UML and remotely over the Internet.

4. Data Management: Lessons Learned

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Routine Unattended Operations
  5. 3. Data Management at Central Repository Sites
  6. 4. Data Management: Lessons Learned
  7. 5. Quantifying Electron Density Profile Uncertainty
  8. 6. Discussion
  9. 7. Summary
  10. Acknowledgments
  11. References

[8] DIDBase and ADRES projects have been thoroughly tested during the challenging task of providing ground truth data for calibration and validation (CalVal) of spaceborne UV sensors [Paxton et al., 2002] that measure ionospheric electron density profiles. More than 20 digisondes participated in this CalVal study to provide high-cadence ionospheric characteristics during the overhead passes of the UV sensors on a daily basis. Since its dry run test in 2003, the ADRES subsystem received over 12,000 requests for data from the ionosonde network, has arranged the data acquisition, and has reported over 55,000 manually scaled ionograms for comparison with the UV data. It would be unrealistic trying to accomplish such tasks in the conventional data storage scenario. The high degree of automation provided by ADRES allowed the ionogram scalers to connect to the DIDBase, query it for the subset of ionograms acquired by ADRES per CalVal requests, verify or edit the autoscaling results, commit edited records back to the DIDBase, and rely on the ADRES to collect and submit the verified data to the requesting party.

[9] Our experience with the DIDBase and ADRES operations during 2001–2005 clearly showed the following. (1) Human intervention and interaction must be minimized and automated to ensure reliable operations and quick turnaround. (2) Data survivability and security at remote observatories cannot be guaranteed. Expert supervision of the software/hardware at the ionosonde locations is too expensive. Real-time data must be delivered to a central data repository where commercial-strength data management can be provided cost effectively. (3) The best arrangement for the ADRES operations in terms of timely request completion is that participating ionosondes routinely report their entire data stream for ingestion to the DIDBase. When ADRES searches other data repositories on the Internet for requested digisonde data, performance deteriorates, especially for the retro data requests that are more than 14 days old. The worst scenario is when data have to be requested from the ionosonde managers for manual upload and subsequent ingestion into DIDBase. (4) User interface of scaling software requires a thoughtful effort to be made ergonomic. Committing scaling results to the database should be done by default as the scaler moves to the next ionogram. Keyboard strokes should complement most commands, especially for navigating through the data set. (5) Provision of the WWW interface to the database is the best way to expand its user base. No matter how easy it is to install SAO Explorer software on any platform, there is always a psychological barrier to overcome, and there is sometimes a firewall issue to deal with. (6) Although it seems easier to deliver ionogram-derived data as a small text file holding only requested information, all reported data should retain their full available content in the standard extensible format in order to optimally manage changing requirements. (7) Extensibility of the standard data format is important for the data providers that are unavoidably impacted by new requirements created by end-user applications. Upward compatibility of the standard data format is crucial for data consumers to avoid costly and lengthy software upgrades when new format revisions become available.

5. Quantifying Electron Density Profile Uncertainty

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Routine Unattended Operations
  5. 3. Data Management at Central Repository Sites
  6. 4. Data Management: Lessons Learned
  7. 5. Quantifying Electron Density Profile Uncertainty
  8. 6. Discussion
  9. 7. Summary
  10. Acknowledgments
  11. References

[10] New ionospheric assimilation models such as the Global Assimilation of Ionospheric Measurements (GAIM) [Schunk et al., 2004] differ from prior generation adaptive ionospheric models in that they analyze the uncertainty of the observational inputs before using them as constraints on the physical model drivers. These uncertainties are important for the Kalman filter, currently viewed as the most promising prediction technique for space weather applications, to dynamically evolve into an optimal state where model and observations are matched with the least error.

[11] The ionogram scaling algorithm ARTIST [Reinisch and Huang, 1983] recently underwent a series of modifications that substantially improved the quality of its analysis [Reinisch et al., 2005], and a network-wide upgrade of the ARTIST software is underway. ARTIST is currently being augmented to automatically determine the uncertainty of each profile point in addition to providing a confidence level for the entire profile. Inner and outer boundary profiles are determined that define the uncertainty range for any profile point using a set of anchor points whose uncertainty is known from statistical analysis of prior data. Figure 3 shows the profile anchor points hsE (starting height of E layer), fmE (plasma frequency at the E layer peak), fsF1 (plasma frequency at the start of the F1 layer), fmF1 (plasma frequency at the F1 layer peak), and fmF2 (plasma frequency at the F2 layer peak).

image

Figure 3. Point-by-point uncertainty of the ARTIST electron density profile obtained by calculating inner and outer boundaries enclosing the profile. Both boundaries rest on five anchor points whose uncertainties are known from statistical analysis of differences between automatically and manually scaled values.

Download figure to PowerPoint

[12] The anchor point uncertainties are calculated separately for different times of day, season, location, and geophysical conditions by comparing automatically and manually scaled values on a representative data set. Then, for each profile calculated by ARTIST [Reinisch and Huang, 1983], the algorithm first locates an appropriate set of anchor point uncertainties using the time of measurement, ionosonde location, and the automatically determined spread F characteristics in the ionogram. The appropriate set of anchor point uncertainties is used to obtain the target points for the inner and outer boundaries. The boundaries are then calculated by modifying the ARTIST profile's Chebyshev coefficients so that the boundary fits the target points. The confidence level, which can be used for top-level decisions to accept or reject a profile for assimilation, reflects the complexity of the ionogram (spread F, ionospheric tilt, blanketing Es, absorption, etc.) and the ionogram quality on the basis of signal-to-noise ratios and station performance history.

6. Discussion

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Routine Unattended Operations
  5. 3. Data Management at Central Repository Sites
  6. 4. Data Management: Lessons Learned
  7. 5. Quantifying Electron Density Profile Uncertainty
  8. 6. Discussion
  9. 7. Summary
  10. Acknowledgments
  11. References

[13] The changing world of ionospheric nowcasting and prediction imposes new requirements on the instrument networks providing real-time sensor information. An important consequence of the ongoing changes is the inability of aging file formats to accommodate growing contents of ionogram-derived data. Calls for additional information have forced several revisions of the standard archival format for ionospheric characteristics in the past, each time causing problems for data users and providers because of the substantial software redesign effort associated with such changes. The current standard archival output (SAO) format for scaled characteristics [Gamache et al., 1996] has practically reached the limit of its flexibility for accommodating new needs.

[14] A new format for ionospheric data must satisfy a minimum number of requirements. (1) It must be easily expandable to accommodate specific information needs of particular instruments or researchers. (2) It has to be upward and downward compatible; new revisions or additions should not prevent older software from reading old contents in the new files. (3) It has to be intuitive and self-explanatory for a new user to easily find major data elements. (4) It should be able to store derived ionospheric information not only from digisondes but also from other ionospheric sounders, as well as from ionospheric models such as the International Reference Ionosphere [Bilitza, 2003] and average representative profiles calculated over various periods of time. It should be sufficiently flexible to accommodate relevant data from other instruments like incoherent scatter radars.

[15] Many data exchange problems have recently been solved using a new Extensible Markup Language (XML). The XML standard is a subset of Standard Generalized Markup Language (SGML), ISO 8879:1986(E)). A new SAO format based on the XML model is under active development.

7. Summary

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Routine Unattended Operations
  5. 3. Data Management at Central Repository Sites
  6. 4. Data Management: Lessons Learned
  7. 5. Quantifying Electron Density Profile Uncertainty
  8. 6. Discussion
  9. 7. Summary
  10. Acknowledgments
  11. References

[16] New technical developments are making their way into ionospheric sounding. Taking the digisonde as an example, we presented recent advances in the sensor technology of ionospheric sounding, including self-tested unmanned operations of the ionosonde, the new data management systems DIDBase and Drift-DB, intelligent adjustment of measurement schedules, automated service ADRES for requesting manually verified ionogram-derived data, and new efforts to statistically describe the uncertainties of reported ionospheric characteristics and electron density profiles. The modern ionosonde network provides global geophysical information in real time and can switch to special campaign mode operation by remote command and then report data to a central database. Manual verification of autoscaled data can be performed by experienced scalers located anywhere in the world, as long as Internet access is available. Reports about missing sounding data can arrive from unsolicited sources like radio amateurs checking the latest ionogram image on their laptops. Behind the ease of digisonde operations there stands a suite of robust communication and data management tools that were developed to provide seamless 24/7 solutions to the global network of modern ionospheric sounders.

Acknowledgments

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Routine Unattended Operations
  5. 3. Data Management at Central Repository Sites
  6. 4. Data Management: Lessons Learned
  7. 5. Quantifying Electron Density Profile Uncertainty
  8. 6. Discussion
  9. 7. Summary
  10. Acknowledgments
  11. References

[17] This research was supported in part by AF grant 3F19628-C-0092. The authors thank Terry Bullett for his suggestions and encouragement during the development work.

References

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Routine Unattended Operations
  5. 3. Data Management at Central Repository Sites
  6. 4. Data Management: Lessons Learned
  7. 5. Quantifying Electron Density Profile Uncertainty
  8. 6. Discussion
  9. 7. Summary
  10. Acknowledgments
  11. References
  • Barnes, R. I., R. S. Gardiner-Garden, and T. J. Harris (2000), Real time ionospheric models for the Australian Defence Force, paper presented at 2000 Workshop on Applications of Radio Science (WARS'00), La Trobe Univ., Melbourne, Victoria, Australia, 27–29 April.
  • Belehaki, A., L. Cander, B. Zolesi, J. Bremer, C. Juren, I. Stanislawska, D. Dialetis, and M. Hatzopoulos (2005), DIAS project: The establishment of a European digital upper atmosphere server, J. Atmos. Sol. Terr. Phys., 67, 10921099.
  • Bilitza, D. (2003), International Reference Ionosphere 2000: Examples of improvements and new features, Adv. Space Res., 31, 757767.
  • Buchau, J., T. W. Bullett, A. E. Ronn, K. D. Scro, and J. L. Carson (1995), The digital ionospheric sounding system network of the US Air Force Air Weather Service, in Ionosonde Networks and Stations, Rep. UAG-104, pp. 1620, Boulder, Colo.
  • Gamache, R. R., I. A. Galkin, and B. W. Reinisch (1996), A database record structure for ionogram data, Rep. UMLCAR 96-01, Cent. for Atmos. Res., Univ. of Mass.-Lowell, Lowell, Mass.
  • Paxton, L. J., D. Morrison, Y. Zhang, H. Kil, B. Wolven, B. S. Ogorzalek, D. C. Humm, and C.-I. Meng (2002), Validation of remote sensing products produced by the Special Sensor Ultraviolet Scanning Imager (SSUSI): A far UV-imaging spectrograph on DMSP F-16, Proc. SPIE, 4485, 338348.
  • Reinisch, B. W. (1996a), Ionosonde, in Upper Atmosphere, edited by W. Dieminger, G. K. Hartmann, and R. Leitinger, pp. 370381, Springer, New York.
  • Reinisch, B. W. (1996b), Modern ionosondes, in Modern Ionospheric Science, edited by H. Kohl, R. Rüster, and K. Schlegel, pp. 440458, Eur. Geophys. Soc., Katlenburg-Lindau, Germany.
  • Reinisch, B. W., and X. Huang (1983), Automatic calculation of electron density profiles from digital ionograms: 3. Processing of bottomside ionograms, Radio Sci., 18, 477492.
  • Reinisch, B. W., D. M. Haines, K. Bibl, I. Galkin, X. Huang, D. F. Kitrosser, G. S. Sales, and J. L. Scali (1997), Ionospheric sounding support of OTH radar, Radio Sci., 32, 16811694.
  • Reinisch, B. W., I. A. Galkin, G. Khmyrov, A. Kozlov, and D. F. Kitrosser (2004), Automated collection and dissemination of ionospheric data from the digisonde network, Adv. Space Res., 2, 241247.
  • Reinisch, B. W., X. Huang, I. A. Galkin, V. Paznukhov, and A. Kozlov (2005), Recent advances in real-time analysis of ionograms and ionospheric drift measurements with digisondes, J. Atmos. Sol. Terr. Phys., 67, 10541062.
  • Schunk, R. W., et al. (2004), Global Assimilation of Ionospheric Measurements (GAIM), Radio Sci., 39, RS1S02, doi:10.1029/2002RS002794.