Radio Science

Determining riometer quiet day curves 1. The matrix method

Authors


Abstract

[1] A number of methods have been proposed for the determination of quiet day curves of riometer recordings of cosmic radio noise in polar regions. Most of these methods are based on the distribution of the recorded intensities, in a given siderial time period, over a number of consecutive days. The method proposed in this paper is based on the filtering, using a low-pass filter in the Fourier domain, of a discrete two-dimensional function or matrix. The two dimensions of the matrix are siderial time and day number, with each row of the matrix representing one siderial day. With a low-pass filter the low frequency Fourier coefficients are retained while the high frequency Fourier coefficients are discarded. The highest Fourier coefficient that is retained is the cutoff frequency of the filter. The proposed method was compared to two existing methods using data obtained from a single broad beam antenna. It was found that the proposed method is less sensitive to noise than the existing methods.

 

1. Introduction

[2] Even before the introduction of the riometer there had been a need for the determination or estimation of accurate quiet day curves (QDCs) from cosmic radio noise, for the observation of ionospheric attenuation. Some of the methods devised to determine QDCs needed data collected over a long period of time.

[3] Both wide and narrow beam antennae have been used in riometer work. A new development is the imaging riometer described by Detrick and Rosenberg [1990], Stoker et al. [1992] and Wilson et al. [2001]. Here a phased-array antenna is used to observe the cosmic radio noise in a number of directions centered on the zenith. From these observations it is possible to derive a spatial map of the attenuation in the ionosphere as a function of time. Typical resolutions used are 7 by 7 [Detrick and Rosenberg, 1990] and 8 by 8 [Wilson et al., 2001], giving 49 and 64 areas of observation respectively.

[4] With multifrequency riometry it should in principle be possible to infer the altitude at which attenuation is taking place from the power indices, n, defined by equation image [Hargreaves, 1969]. At two different frequencies

equation image

where A1 and A2 are the attenuations at frequencies f1 and f2 respectively. It has, however, subsequently been shown by Stoker [1987] that n is an indication of three dimensional, that is both horizontal and vertical, structure of ionization in the upper atmosphere. Recently some work has been done investigating the use of computerized tomography to obtain an altitude profile of attenuation in the ionosphere, among others by Afraimovich et al. [1992], Austen et al. [1988], Fremouw et al. [1992], Na and Lee [1994], Pryse and Kersley [1992] and Raymund et al. [1990, 1993, 1994a, 1994b].

[5] With the current increased use of imaging riometers as well as the possible future use of tomography it is becoming increasingly important that quiet day curves are determined accurately.

[6] A QDC is defined as the signal intensity that is observed with a quiet and undisturbed ionosphere. The QDC thus defined is subjected to seasonal and even shorter-term variations. A QDC determined for a given period is therefore only valid for that particular period of time.

[7] Originally the determination of QDCs was based on the assumption that the highest value recorded over a number of days for a given sidereal time interval corresponds to a condition of no attenuation [Heisler and Howler, 1967]. It was also assumed that no drift in equipment parameters took place. The procedure was to translate the received data to sidereal time and compare all the values in sidereal time intervals. The highest “reliable” value for a given sidereal time interval would then be a point on the QDC. Some researchers, for example Heisler and Howler [1967], Lusignan [1960] and Mitra and Shain [1953], however questioned the accuracy of the QDC derived in this way as the minimum attenuation on undisturbed days does not necessarily return to zero and is subjected to a seasonal variation.

[8] Alternatively the QDC value for a given sidereal time interval can be defined as that value which separates the highest 10% (say) of the data values from the remaining 90%. Armstrong et al. [1977] compared the QDCs using 10%, 5% and 3% as criteria and found that the actual percentile value was not very critical. They however proposed that the inflection point on the high side of the peak of the distribution of signal intensities be used instead of an arbitrary percentile value. The basic assumption behind the inflection point method is that the quiet day value of an interval is that value (on the high-signal side of the peak of the distribution) at which the distribution of signal values is decreasing most rapidly with increasing signal level [Krishnaswamy et al., 1985].

[9] In their implementation of the inflection point method Krishnaswamy and Detrick [1983] and Krishnaswamy et al. [1985] viewed the distribution as being made up of a finite number of line segments. The inflection point is then defined as the midpoint of that line segment, on the high-signal side of the peak of the distribution, which has the maximum negative or steepest descending slope. The inflection point of each interval is determined in this way resulting in a QDC which may be irregular with numerous spikes. This raw QDC can then be filtered to produce a smooth QDC.

[10] Krishnaswamy et al. [1985] found that the inflection point method gave better results in the presence of noise if sidereal time intervals of 30 minutes instead of 10 minutes were used. They also did a comparison with the percentile method and found that the two methods produced similar results with clean or smooth data, while the inflection point method produced much better results with noisy data.

[11] The Physics Department of the Potchefstroom University has been running a riometer program at Sanae, Antarctica (70°S, 2°3W, equation image since 1964. Since 1979 La Jolla fast response solid state riometers have been in use with the data being recorded by chart recorders as well as digitally. The traditional method of processing riometer data consisted of tracing a “quiet day” from the chart recordings and using it to calculate the attenuation on the days that were close to the “quiet day”. The problem with this method was that in the absence of a quiet day it would be necessary to use a quiet day that was temporally far removed from the day(s) that had to be processed. This problem was addressed in 1980 when the processing of the riometer data was computerized by using the maximum density method.

[12] In this method two weeks' data are first divided into sidereal time intervals. The QDC value of an interval is now defined as the peak of the distribution of the signal intensities of that interval. The Fourier transform of the QDC values is calculated, the low order coefficients kept unchanged while the high order coefficients are set to zero and the inverse Fourier transform calculated.

[13] Normally coefficients 0 to 5 were used resulting in smooth QDCs determined from as few as seven consecutive days [Drevin and Stoker, 1990]. It is however shown by Drevin and Stoker [2003] that coefficients 0 to 3 would be more than adequate as they contain 95.6% of the AC energy with coefficients 4 and 5 contributing only 2.7% to the AC energy.

[14] An aspect that the percentile, inflection point and maximum density methods have in common is the fact the QDC value for a given sidereal time interval is based on the distribution of the signal strengths observed in that sidereal time interval over a number of days. These methods can therefore be classified as distribution-based methods.

[15] In determining the QDC for a given day the distribution-based methods described above make use of the cosmic radio noise intensities observed over a number of adjacent days. Normally a number of days preceding and following the day in question are used. The QDC value for each sidereal time interval is then based on the distribution of signal intensities observed in that sidereal time interval on each of the days being used. The remainder of the signal intensities, observed on the same day in intervals immediately preceding and succeeding that interval, however, have no influence on the QDC value for that sidereal time interval. It is only when the QDC values are smoothed that the values outside a sidereal time interval have an influence on its QDC value.

[16] In this paper a new method is proposed which makes direct use of the signal intensities observed over the entire day, as well as the preceding and successive days in determining a QDC. The proposed method is to incorporate both the reception pattern of the riometer antenna as well as the seasonal (and shorter term) variations in the quiet, undisturbed ionosphere.

[17] To be able to compare this new method with existing methods it was decided to implement the maximum density method [Drevin and Stoker, 1990] as well as the 90 percentile method [Armstrong et al., 1977]. QDCs were calculated using these two methods with 15-day intervals consisting of the day for which the QDC is being calculated as well as the seven days preceding and following it.

[18] The Fourier transform of each day's QDC values was calculated, the fourth and higher coefficients discarded and the inverse Fourier transform calculated resulting in a smooth QDC. The choice of keeping the three lower order coefficients is based on the results obtained by Drevin and Stoker [2003]. QDCs for day numbers 1 to 7 and 359 to 365 were not calculated. The QDCs obtained with the maximum density method are hereafter referred to as MDM-data while those obtained with the 90-percentile method are referred to as 90PM-data.

2. Initial Data Processing

[19] The data used in this study were recorded during 1983 at Sanae, Antarctica (70°S, 2°3W, equation image using La Jolla fast response solid state riometers at frequencies of 30 and 51.4 MHz. The riometer output voltages were averaged over 2 second intervals, passed through an ADC (analogue to digital converter) and stored on magnetic tape. The integer digital values are bins numbered 0 to 236 with a resolution of 25 mV per bin. (These time and intensity resolutions were in use with the wide beam riometers (1979 to 1994). The resolutions have however been increased with the introduction of the imaging riometer in 1991.)

[20] Noise in the form of external interference as well as calibration signals are present in the data and therefore have to be removed. The external interference includes radio broadcasts as well as interference from the ionosonde in operation at Sanae and is present in the form of spikes of very short duration. The preprocessing procedure as described by Drevin and Stoker [1990] is used to remove the noise and calibration signals from the data. The 1-minute data resulting from this preprocessing procedure are resampled into 512 intervals per day where each interval is either the mean of 2 one-minute values or the median of 3 one-minute values. The choice of 512 intervals is dictated by the use of the fast Fourier transform (FFT) to calculate the Fourier transform of the data. As the FFT is a base 2 algorithm it is preferable that the length of the data be a power of 2. It is of course also possible to resample the data into 1024 intervals instead of 512 intervals. The advantage of using 512 intervals is that less memory is required and processing time is shorter. A further advantage is the smoothing which takes place during the resampling process.

[21] The data are also translated to sidereal time during this resampling process. Since the length of a sidereal day is 1436 UT minutes the last 4 one-minute data values of each universal day are discarded and the remaining 1436 values are translated to sidereal time before being resampled into 512 intervals. The result is that the data of each universal day are in sidereal time.

[22] The result of the preprocessing and resampling procedure described above is a data set which can be used to calculate QDCs. Once the QDCs have been obtained they can be scaled to 1-minute, 2-second or any other desired time resolution and used to calculate the attenuation of the data at that resolution. This data set is stored in the form of a matrix with each day forming a row of the matrix. Each column of the matrix represents a fixed sidereal time interval. This gives a matrix of 365 rows (days) and 512 columns (sidereal time intervals).

[23] The data set resulting from this preprocessing step (hereafter referred to as raw-data) contains gaps as well as some remaining noise. These gaps and the noise are removed from the data set using the following two-pass procedure:

  1. A sliding median of 15 days is calculated, over the whole year, for each of the 512 sidereal intervals. That is, a sliding median of 15 values is calculated for each of the columns in the raw-data matrix.
  2. Gaps that are still present in sidereal intervals (columns) are removed by linear interpolation over such gaps. Linear interpolation is used as it is unlikely that gaps are large, large gaps would imply missing data for the same sidereal interval over a large number of successive days. It is furthermore assumed that the quiet day value for a given sidereal interval does not change much from day to day.

[24] The data set resulting from this smoothing procedure (hereafter referred to as smooth-data) is cleared of most noise, gaps and attenuation events, and is only used in the calculation of quiet days. The raw and smooth-data sets are shown as images in Figure 1. Each row in the images represents the data of one day. Maximum values are represented by white. (The black areas in Figure 1a represent missing data.)

Figure 1.

The 30 MHz riometer data (1983) shown as images: (a) raw-data and (b) smooth-data. Each day is represented by a horizontal line that is divided into 512 sidereal time intervals.

3. Matrix Method

[25] The method consists of the following steps: (1) The data of a number of days are placed in a matrix with each individual day forming a row of the matrix while each of the columns represent a sidereal time interval. (2) The data matrix is transformed using a two-dimensional transform such as the Fourier or cosine transform. (3) The transformed data are multiplied by the transfer function of a filter such as the Butterworth low pass filter. (4) The inverse transform of the filtered data is calculated resulting in a matrix where each row contains the QDC for the day which initially occupied that row.

[26] With an ideal low pass filter all the coefficients up to the cutoff frequency are kept unchanged while the higher frequency coefficients are eliminated by setting them equal to zero. The cutoff frequency therefore is the highest Fourier coefficient that is retained. The transfer function of an ideal filter is defined as:

equation image

where equation image is the distance from the point (u,υ) to the origin of the frequency plane while D0 is the distance from the cutoff frequency to the origin of the frequency plane. This transfer function will consist of a circular area, where all the values are equal to 1, centered on the origin of the frequency plane, with a radius of D0. All the values outside this area will be equal to 0.

[27] In order to be able to retain a different number of coefficients in the horizontal (time) and vertical (day) directions, D0 is rather defined as:

equation image

where θ is the direction of the vector D0, Dx is the cutoff frequency in the horizontal direction and Dy in the vertical direction (Figure 2). The value of Dx is dependent on the antenna reception pattern while the value of Dy depends on the day to day variation. Using this definition of D0 in Equation (2) will result in a transfer function consisting of an elliptical area, where all the values are equal to 1, centered on the origin of the frequency plane. All the values outside this area will once more be equal to 0.

Figure 2.

The ellipse parameters used in Equation (2).

[28] An ideal low pass filter unfortunately suffers from ringing. The alternative is to use a nonideal filter such as the Butterworth filter of which the transfer function is defined as:

equation image

where n is the order of the filter [e.g., Gonzalez and Woods, 1993, p. 208; Pratt, 1991, p. 250]. The definition of D0 in Equation (3) is used to produce a transfer function that is elliptically shaped rather than circular.

[29] The FFT is used to calculate the Fourier transform of the data, but as the FFT is a base 2 algorithm, the number of days for which QDCs are calculated should preferably be a power of 2. Furthermore, as QDCs were calculated only for day numbers 8 to 358 using the maximum density and 90 percentile methods, it was decided to calculate QDCs for that same period using the matrix method. The implication is that two sets of QDCs consisting of 256 days each are to be calculated, the first set for day numbers 8 to 263 and the second set for day numbers 103 to 358, with day numbers 103 to 263 forming the suffix of the first set and the prefix of the second set.

[30] The Fourier transform of day numbers 8 to 263 of the smooth-data is therefore calculated and multiplied with the transfer function of the Butterworth filter (Equation (3)) with equation image and equation image in Equation (2) after which the inverse Fourier transform is calculated. These values of Dx and Dy are based on the results of Drevin and Stoker [2003]. The result, which is the QDCs of day numbers 8 to 263, is shown as an image in Figure 3a while the daily means are shown in Figure 3b. This procedure is then repeated for day numbers 103 to 358.

Figure 3.

(a) The 30 MHz QDCs calculated with the matrix method shown as an image. Each day is represented by a horizontal line that is divided into 512 sidereal time intervals. (b) Daily means of the QDCs shown in (a). (c) Difference between the daily means of the QDCs of day numbers 103 to 263.

[31] Ideally the QDCs for day numbers 103 to 263 should be identical in the two sets of QDCs. The differences between the daily means of day numbers 103 to 263 in these two sets of QDCs are shown in Figure 3c. The nonzero differences at the beginning and end of this period are due to Gibbs's phenomenon.

[32] The Gibbs phenomenon error is caused by discontinuities in a data set. The Fourier transform treats a data set as a single period of a periodic data set resulting in discontinuities at the leading and trailing edges of this specific data set. It can be shown that the Gibbs phenomenon is not only caused by discontinuities in the values of the data set but also by discontinuities in the slope (first derivative) as well as higher order derivatives of the data set [e.g., Thompson, 1992].

[33] The pinned sine transform suggested by Meiri and Yudilevich [1981] for use in image coding and compression removes this discontinuity. It was however shown by Clarke [1983] that the nonconstant zero order basis vector of the discrete sine transform leads to an undesirable distribution of energy among the transform coefficients. A number of alternative pinned transforms were investigated by Drevin [1999] and the use of the pinned discrete Fourier transform was proposed to reduce the error due to the Gibbs phenomenon.

[34] An one dimensional data set, equation image can be pinned by using:

equation image

where y(i) is the pinned data set and p(i) is the pinning polynomial:

equation image

with

equation image

The pinned data set equation image is then transformed using the standard FFT.

[35] As no implied value or slope discontinuity exists in the rows of the data matrix it is not necessary to use a pinned transform to transform the rows of the data matrix. However, a pinned transform is needed to transform the columns of the data matrix as they can contain both value and slope discontinuities (Figure 3b). In this method, therefore, the rows of the data matrix are first transformed using the standard FFT. The zero order column of the transformed rows is nothing else than the daily means of the data while the rest of the columns contain only AC energy. It is therefore only necessary to use a pinned transform for the zero order column while the standard FFT can be used for the rest of the columns.

4. Discussion

[36] The distributions of the 30 MHz attenuations for 1983 as well as the distributions of the 30-51.4 MHz power indices n-values are given in Figure 4. The attenuations and power indices were calculated relative to QDCs determined by the 90 percentile method, maximum density method and matrix method.

Figure 4.

The distributions of 30 MHz attenuations as calculated with (a) 90PM, (c) MDM and (e) matrix method QDCs. The distributions of 30-51.4 MHz power indices (n-values) obtained from attenuations as calculated with (b) 90PM-, (d) MDM- and (f) matrix method QDCs.

[37] The n-values were determined only when the 30 MHz attenuations were greater than 0.4 dB and the 51.4 MHz attenuations were greater than 0.2 dB. The n-values that appear between 3.5 and 4.5 in Figure 4 are mainly due to a level shift in the 30 MHz signal strengths of day numbers 105 to 107.

[38] The 90-percentile method QDCs (90PM-QDCs) resulted in a large number of 0.1 dB attenuations while the maximum density method QDCs (MDM-QDCs) and matrix method QDCs have their peak number of attenuations at 0 dB (Figure 4). This is caused by the fact that the 90PM-QDCs are slightly higher than the QDCs determined with the other two methods as can be seen from Figures 5 and 6.

Figure 5.

Daily means of the 30 MHz QDCs: 90PM (upper solid line), MDM (lower solid line) and matrix method (dotted line).

Figure 6.

Difference (in dB) between daily means of the 30 MHz QDC values (signal strengths). The difference between the MDM and 90PM QDCs is shown in (a), (b) is the difference between the MDM and matrix method QDCs and (c) the difference between the matrix method and 90PM QDCs.

[39] The differences (in dB) between the daily means of the MDM and 90PM QDCs

equation image

are determined where QMDM(d) and Q90PM(d) are the means of the MDM and 90PM QDCs for day d respectively and Δ(d) is the difference between the two method's QDCs for day d. The differences between the daily means of the three methods, applied to the 30 MHz data, are shown in Figure 6.

[40] The shape of the QDCs should stay constant throughout a year. However, variations in amplitude can be expected. The principal sinusoidal shape of a QDC is represented by the first AC coefficient of the QDC. The sidereal times of the maximum and minimum intensities of the QDC can be inferred from the phase angle of the first AC coefficient. These times are only dependent on the spatial variations in cosmic radio noise temperature and should therefore be constant throughout the year. The sidereal times of the 30 MHz and 51.4 MHz daily QDC maxima, QDCs determined by the matrix method, are shown in Figure 7a. The two frequencies exhibit the same variation in time of daily QDC maxima even though the amplitude of the variation in the case of the 51.4 MHz is much smaller. The correlation between the variation observed for the two frequencies is 0.94.

Figure 7.

The sidereal time of the 1983 daily signal strength maxima as derived from the phase angle of the first AC coefficient of each day's Fourier transform of (a) the 30 MHz (solid line) and 51.4 MHz (dotted line) matrix method QDCs and (b) of the 30 MHz MDM (solid line) and 90PM (dotted line) data.

[41] The sidereal times of the 30 MHz MDM and 90PM daily QDC maxima are shown in Figure 7b. When compared with Figure 7a it is significant to note that the variations obtained with the matrix method are much smoother than those obtained with the maximum density and 90 percentile methods. The variations in the time of QDC maximum can be attributed to the method's sensitivity to noise.

[42] The mean times and standard deviations (σ) of the maxima obtained with the different methods are given in Table 1. These results correlate well with the sidereal times of 16:15 and 16:29 respectively for the maxima of the 30 MHz and 51.4 MHz mean day curves obtained by Drevin and Stoker [2003].

Table 1. Times of QDC Maxima
 Matrix MethodMDM 90 PM
 30 MHz51 MHz30 MHz
Mean16:1016:2716:1116:10
σ, minutes29123229

[43] The application of this method to the data obtained with an imaging riometer still has to be investigated. The big difference between wide beam and imaging riometers, as far as the matrix method is concerned, lies in the spatial frequency responses of their antennae reception patterns. The reception pattern of the antenna used by an imaging riometer is very narrow and therefore has a spatial frequency response that is much higher than the spatial frequency response of the reception pattern of the antenna used by a wide beam riometer. The data gathered for each individual area (49 areas with a 7 by 7 imaging riometer) form an individual data set to which the matrix method can be applied. It is therefore not necessary to adapt the method from two to four dimensions. The relationship between the QDCs of neighboring areas should, however, also be investigated.

5. Conclusions

[44] A new method of determining quiet day curves, the matrix method, has been proposed in this paper. Data obtained from a single broad beam antenna was used to test the method and compare it to two existing methods. It was found that this method is less susceptible to noise than is the case with the maximum density and 90 percentile methods. This could indicate that the proposed method is less susceptible to absorptions that occur on a daily basis, as well as to absorptions that last for a number of days. However, the sensitivity of the method to such absorption events still has to be investigated.

[45] An undesirable side effect of filtering in the frequency domain is that errors are introduced at the leading and trailing edges of the data set which is being filtered. This phenomenon is known as the Gibbs phenomenon. Different methods of overcoming this problem were investigated with the pinned Fourier transform being the most effective.

Acknowledgments

[46] The data used in this paper were obtained and processed through the financial and logistical support to the South African National Antarctic Program by the Department of Environmental Affairs and Tourism.

Ancillary