Determining riometer quiet day curves 2. Derivation of filter cutoff frequencies

Abstract

[1] A method for the determination of riometer quiet day curves which is based on the filtering of a discrete two-dimensional function or matrix has been proposed. The two dimensions of the matrix are siderial time and day number, with each row of the matrix representing one siderial day. The filtering is done in the Fourier domain using a low-pass filter. With a low-pass filter the low frequency Fourier coefficients are retained, while the high frequency Fourier coefficients are discarded. The highest Fourier coefficient that is retained is the cutoff frequency of the filter. In this paper the filter cutoff frequencies are derived directly from the antenna reception pattern as well as from the data itself. A number of methods can be used to infer a cutoff frequency from the data; however, only a number of entropy based methods are investigated in this paper.

1. Introduction

[2] The matrix method for the determination of riometer quiet day curves, proposed by Drevin and Stoker [2003], is based on the filtering of a discrete two-dimensional function or matrix representation of the riometer data. This matrix's dimensions are siderial time and day number, with each row of the matrix representing one siderial day. The filtering of this matrix is done in the frequency or Fourier domain. Although the maximum density method [Drevin and Stoker, 1990] also made use of filtering in the frequency domain, the choice of cutoff frequency was subjective, the choice being the cutoff frequency which gave visually acceptable results. The cutoff frequency is the highest Fourier coefficient that is retained.

[3] In this paper, however, we attempt to determine cutoff frequencies objectively. The cutoff frequencies are determined directly from the antenna reception pattern as well as from the data itself. A number of methods can be used to infer a cutoff frequency from the data. However, only a number of entropy based methods are investigated in this paper.

2. Spatial Frequency Response of the Antenna Reception Pattern

[4] The data recorded by a riometer over a 24 sidereal hour period can be viewed as the convolution of the intensity of the cosmic radio noise in a strip centred on the path of the zenith of the receiving station with the reception pattern of the riometer antenna [Detrick and Rosenberg, 1990]. That is:

where h(α, δ) * f(α, δ) is the convolution of the two functions f(α, δ) and h(α, δ) which represent the intensity of the cosmic radio noise and the antenna reception pattern, respectively [e.g., Gonzalez and Woods, 1993, p. 107]. α and δ are the equatorial coordinates, right ascension and declination, respectively. g(α, δ) is the observed intensity with the antenna pointed in the direction (α, δ). As the antenna is fixed on Earth the declination (δ) in Equation (1) will not change, and g(α, δ) will be a function of time only. The intensity observed over a 24 sidereal hour period is therefore a function of right ascension (α), with α increasing from 0 to 24.

[5] In the frequency domain Equation (1) becomes:

where G(u, v), H(u, v) and F(u, v) are the Fourier transforms of g(α, δ), h(α, δ) and f(α, δ), respectively. In the terminology of linear system theory the transform H(u, v) is called the transfer function of the convolution process. It is also the frequency response function of the antenna reception pattern.

[6] The frequency response function of the antenna reception pattern determines the shortest period that can be expected in the variation of the received signal strength. The low frequency components of the Fourier domain characterize variations that take place slowly or over long intervals, while edges and variations that take place over short intervals are characterized by the high frequency components of the Fourier domain [e.g., Gonzalez and Woods, 1993, p. 189]. A narrow reception pattern is characterized by a large change over a short interval and will therefore have large high frequency coefficients. It can therefore be expected that variations with short periods will be observed. A wide reception pattern, on the other hand, has a frequency response consisting mainly out of low frequencies. Consequently the observation of long period variations is all that can be expected when an antenna with a wide reception pattern is being used.

[7] Except for the frequency response of the antenna reception pattern, the shape of the QDC is also influenced by the spatial variations in cosmic radio noise temperature. A QDC with little or no variations can be expected in the case where the temperature of the strip of sky that is swept by the antenna is quite uniform, even if an antenna with a narrow reception pattern is being used. However, the presence of a very strong point source in an otherwise uniform strip of sky cannot result in a variation being observed which is shorter than that which is predicted by the frequency response of the antenna. That is, the spatial frequencies of such a variation cannot be higher than the frequency response of the antenna.

[8] Once the frequency response of the antenna reception pattern is known a low pass filter can be designed to cut out noise and attenuations (variations with periods which are shorter than the minimum indicated by the frequency response of the antenna reception pattern) and leave the quiet day variations intact.

[9] It is therefore necessary to first calculate the reception pattern of the antenna. This was done for the antenna in use during 1983 using MININEC antenna analysis software (see, for example, http://www.emsci.com/ and http://www.qsl.net/wb6tpu/swindex.html). The north-south and east-west sections through the reception pattern are shown in Figure 1a and are representative of the entire reception pattern. It is, however, also clear from Figure 1a that there is a slight deviation from cylindrical symmetry in the antenna reception pattern. As can be seen from Figure 1a, the antenna in use at the time had a −3 dB beam width of approximately 70°. The frequency response of the reception pattern is given by its Fourier transform. Part of the east-west section through the Fourier transform of the antenna reception pattern is shown in Figure 1b. The percentages that each of the first five AC coefficients contribute to the total AC energy of the transfer function of the antenna are shown in Table 1. Also shown are the cumulative percentages of the first five AC coefficients. It is clear that the antenna reception pattern has a low frequency response. The fourth and higher order Fourier coefficients do not contribute significantly to the variation in the recorded data. This means that the fourth and higher coefficients of the Fourier transform of a day's riometer data can be discarded (made equal to zero) without a significant loss of quiet day information.

Table 1. Frequency Response of the Antenna Reception Patterna
Coefficient
12345
• a

The percentages and cumulative percentages of the total AC energy in the low order Fourier coefficients of the frequency response of the antenna reception pattern are given. The DC energy or zeroth Fourier coefficient is excluded.

Percentage51.231.812.61.61.1
Cumulative %51.283.095.697.298.3

[10] The maximum density method [Drevin and Stoker, 1990] made use of Fourier coefficients 0 to 5. The results in Table 1 however indicate that coefficients 0 to 3 would be more than adequate as they contain 95.6% of the AC energy with coefficients 4 and 5 contributing only 2.7% to the AC energy.

[11] It is expected that the AC energy content of the QDCs determined for Sanae, Antarctica (70°S, 2°3W, L=4.0), would approach that which is predicted by the frequency response of the antenna reception pattern as the regions round both the south galactic pole and the galactic centre pass through the antenna reception pattern during each sidereal day.

[12] The data used in this study were recorded during 1983 at Sanae using La Jolla fast response solid state riometers at frequencies of 30 and 51.4 MHz. The noise and gaps were removed from this data, after which the data were resampled into 512 intervals per day and translated to siderial time [Drevin and Stoker, 2003]. The resulting data set is hereafter referred to as the smooth-data set.

[13] Furthermore, QDCs were calculated from the smooth-data set using the maximum density method [Drevin and Stoker, 1990]. To determine the QDCs 15 day intervals consisting of the day for which each QDC is being calculated as well as the seven days preceding and following it were used. The Fourier transform of each day's QDC values was calculated, the fourth and higher coefficients discarded and the inverse Fourier transform calculated resulting in a smooth QDC. The choice of keeping the three lower order coefficients is based on the results of Table 1. QDCs for day numbers 1 to 7 and 359 to 365 were not calculated. The QDCs obtained with the maximum density method are hereafter referred to as MDM-data.

[14] A mean day curve for 1983 was determined from the smooth-data set. This was done by first subtracting the daily means from the smooth-data. Thereafter the mean of each sidereal interval of the entire year's data was calculated resulting in a mean day curve. This was done for both the 30 MHz and 51.4 MHz smooth-data. The mean day curves of the two frequencies are shown in Figure 2a while the Fourier power spectra of the two mean day curves are shown in Figure 2b. Only the first 10 AC coefficients of the power spectra are shown. The power spectra of the mean day curves correlate well with the frequency responses of the reception patterns of the antennae that were in use at the time thereby confirming the theoretical prediction given above. (The theoretical frequency response functions of the reception patterns of the 30 MHz and 51.4 MHz antennae were the same.)

[15] There is an indication in Figure 2a of shorter period variation in the 51.4 MHz mean day curve than in the 30 MHz mean day curve. As can be seen in Figure 2b, coefficients 2 and higher of Fourier transform of the 51.4 MHz mean day curve contain a slightly larger fraction of the transform energy than is the case with the 30 MHz mean day curve. This could imply a narrower reception pattern for the 51.4 MHz antenna than for the 30 MHz antenna. There is also a small phase difference between the first AC coefficients of the two mean day curves with the 30 MHz mean day lagging about 4° behind the 51.4 MHz mean day curve. (The first AC coefficient represents the principal sinusoidal shape of the mean day curve.) An explanation for this phase difference could be that the 51.4 MHz antenna pointed slightly to the west of the 30 MHz antenna. The 51.4 MHz antenna pointing slightly to the north of the 30 MHz antenna would also result in a maximum being observed slightly later at 51.4 MHz than at 30 MHz. The antennae in use at Sanae were double dipoles. It is possible that the individual dipole elements were not quite horizontal and that the two dipoles of each antenna were not in the same horizontal plane. Therefore it is quite plausible that the imperfections in the antennae, hinted at in the above discussion, did in fact exist. The difference between the mean day curves of the two frequencies (Figure 2a) may be attributed to these imperfections in the antennae. A further factor which could have had an influence on the reception patterns of the antennae is the formation of ice on the antenna dipole elements.

3. Spatial Frequency of Day-to-Day Variation

[16] A variation in the daily mean of the received power of the cosmic radio noise has been observed [Drevin and Stoker, 1990]. The daily means of the 30 MHz data are shown in Figure 3. It is clear from Figures 3b and 3c that both a long-term variation spanning the entire year as well as shorter period variations are present in the QDCs. The short period variations are evident throughout the year. These variations are to a large degree present in both the 30 MHz and the 51.4 MHz data. The main exception is the dip between day number 90 and 110 which is only present in the 30 MHz data. When viewing the data matrix in the vertical dimension these real QDC variations are represented by low frequency coefficients in the Fourier transform of the data while noise and attenuations are represented by high frequency coefficients. In order to filter the data it is therefore necessary to determine a cutoff frequency with which the real QDC variations in the data will be retained while noise and attenuations are discarded. However, it is not possible to determine such a cutoff frequency analytically and it therefore has to be inferred from the data itself.

[17] A number of methods to infer a cutoff frequency directly from data have been investigated by Drevin [1999]. In this paper, however, we only make use of a number of entropy based methods.

3.1. Entropy Thresholding

[18] Given a data set, a series of (discrete) functions which converge to the original data set can be constructed by taking the Fourier transform of the data set and calculating the inverse Fourier transforms of the first N, N = 1, 2, 3, …, coefficients. It is hypothesized that in the case of noiseless data the entropy or information content [Shannon and Weaver, 1949; Soofi, 1994] of the individual functions (inverse Fourier transforms) will increase, as N increases, to a maximum after which it will stay constant. Assuming this behaviour of the entropy, it is further hypothesized that the cutoff frequency (coefficient) is that value of N after which the entropy stays constant.

[19] To test this hypothesis three series of inverse Fourier transforms, converging to the daily means of the raw, smooth and MDM-data of days 8 to 263, were constructed for N = 1, 2, 3, …, 128. The values of the inverse Fourier transforms were rounded to give integer values, i, in the same range (0 to 236) as the original data. The raw-data contain noise in the form of absorptions as well as external interference. Most of this noise has been removed from the smooth-data while the MDM-data, being quiet day curves, can be assumed to be noise free.

[20] The entropy values of each of the inverse Fourier transforms in the three series were calculated using the Shannon entropy measure, defined as [Shannon and Weaver, 1949]:

with

where p = (p1, p2, p3, …, pN) with pi the proportion of the (inverse Fourier transform) data points with the intensity value i. The entropies of the inverse Fourier transforms of the first N coefficients are shown in Figure 4. The entropy values are scaled to the interval (0, 1), with the maximum entropy set equal to 1. With the smooth and MDM data there is a fast increase in the value of the entropy as N is increased to between 5 and 15, after which the entropy does not increase significantly as more coefficients are added. The entropy curve of the raw-data, on the other hand, displays a much slower increase as N increases. The first hypothesis made at the beginning of this section is therefore verified by the actual behaviour of the entropy curves. The value of N after which the entropy stays constant could be defined as the coefficient at which the entropy is smaller or equal to the minimum of the entropies of all the higher frequency coefficients, while the entropy of the next frequency is greater than the minimum of the entropies following it. This choice can be stated as:

with N subject to both the conditions:

where HN is the entropy of the inverse Fourier transform of the first N coefficients. Equation (2) can, however, result in a number of values for fc. If this is the case then the fc which best fits the definition of “leveling of” has to be chosen visually. This intuitive method results in fc = 18 in the case of the raw-data, fc = 11 in the case of the smooth-data and fc = 12 in the case of the MDM-data (Figure 4 and Table 2). The vertical lines in Figure 4 are at N = 18 in the case of the raw-data, N = 11 in the case of the smooth-data and at N = 12 in the case of the MDM-data. The horizontal lines indicate the minimum value of the entropies to the right of the vertical lines. In the case of the smooth and MDM-data an alternative “intuitive” cutoff coefficient can be chosen to the left of the vertical line, viz. the coefficient where the entropy is maximum:

with N subject to both the conditions in Equation (3). For both the smooth and MDM-data this method results in N = 7.

Table 2. Cutoff Frequenciesa
RawSmoothMDM
• a

The cutoff frequencies were determined using the following methods: the intuitive entropy methods as specified by Equations (2) and (4); the distance between the entropy curve and the diagonal, identical results were obtained with the Euclidean and K-L distance measures; the K-L distance between the entropy curve and the model specified by Equation (7).

Equation (2)181112
Equation (4) 77
Distance2866
Equation (7)1998

3.2. Entropy and Kullback-Leibler Distance

[21] The Euclidean distance of each entropy curve from the diagonal (1, min(entropy)) to (128, 1) is shown in Figure 5a. Only the AC coefficients are used and a diagonal is defined for each data set. The coefficient that results in the entropy which is furthest from the diagonal is used as cutoff frequency. These values are N = 28 for the inverse Fourier transforms of the raw-data and N = 6 for both the smooth-data and the MDM-data, which corresponds closely with the intuitive values obtained in Section 3.1 (Table 2).

[22] The Kullback-Leibler distance (K-L distance) between the distribution, fx, of observed data and a model or predicted distribution, gx is given by [Kullback and Leibler, 1951]:

subject to the condition:

Equation (5) is not a metric distance as it is not symmetric, that is By adding I(f, g) and I(g, f) together we have:

which is symmetric and is therefore a metric distance [Brink and Pendock, 1996]. Using the K-L distance in the form:

to obtain the K-L distance between the entropy and diagonal at each N instead of the Euclidean distance, results in exactly the same cutoff coefficients (Figure 5b).

[23] The diagonal can be replaced by a model representing the desired increase of the entropy as N is increased. This model can then be adapted until the K-L distance (Equation (6)) between it and the entropy is a minimum. A simple model would be for the entropy to increase linearly up to a maximum after which it stays constant. The cutoff frequency could then be defined as the coefficient where the maximum is reached. That is:

where fc is the cutoff frequency and Hmin and Hmax are the minimum and maximum entropies, respectively. The K-L distance for all values of fc are then calculated (Figure 5c) and the fc which results in the minimum distance is used as cutoff frequency. The results obtained with this method are fc equal to 19 for the raw-data, 9 for the smooth-data and 8 for the MDM-data (Table 2).

4. Discussion

[24] Silverman [1983, 1986] found that if one smoothes a little more than one would in order to optimise mean square error the “bumps” which result are all true bumps rather than noise induced bumps. Donoho [1992] views the thresholding of wavelets as an abstraction of this problem. To avoid the presence of “false bumps” or noise induced structures in smoothed data it is therefore necessary to smooth a little more than one would to achieve optimal mean-squared error. Genovesa and Stark [1996], Pulliam and Stark [1993], Stark [1992,1993], and Stark and Hengartner [1993] give two examples of geophysical studies where structures in reconstructed data caused interpretation difficulties. As one might easily be tempted to interpret such noise induced structures as having actual physical significance, it is better to err on the side of a cutoff frequency which is too low than one which is too high when choosing a cutoff frequency.

[25] With the exception of the results obtained for the raw-data, all the cutoff frequencies are in the range of 6 to 12 coefficients. It was therefore decided to use a cutoff frequency equal to 10. The period of this cutoff is between 25 and 26 days which is close to the ±27 day period of the oscillations in the last 100 to 120 days of the MDM-data.

5. Conclusions

[26] The matrix method is based on the filtering of the riometer data in the frequency (Fourier) domain. The maximum density method also made use of filtering in the frequency domain. In the maximum density method the choice of cutoff frequency was subjective, the cutoff frequency being the frequency that gave visually acceptable results. In this paper, however, an attempt was made to determine cutoff frequencies objectively. The result is that the cutoff frequency in one direction is determined directly from the antenna reception pattern while it is inferred from the data itself in the other. A number of entropy based methods that can be used to infer a cutoff frequency from the data were investigated.

[27] It is shown that the cutoff frequency determined from the antenna reception pattern correlate well with the Fourier power spectra of mean day curves. The cutoff frequency inferred directly from the data also correlate well with variations in the data found by Drevin and Stoker [1990].

[28] This reseach was done using data of a single year (1983), obtained at a single station using a wide beam antenna. Further research should be done to compare the annual variation of daily means as shown in Figure 3 using data from different years and obtained at different latitudes.

[29] The validity of using the frequency response of the antenna reception pattern could also be further tested by using data obtained with antennae with different beam widths. Data from different latitudes should also be included in this testing to determine the sensitivety of the method to different amounts of diurnal galactic noise variation.

Acknowledgments

[30] The data used in this paper were obtained and processed through the financial and logistical support to the South African National Antarctic Program by the Department of Environmental Affairs and Tourism.