SEARCH

SEARCH BY CITATION

Keywords:

  • rainfall;
  • rain rate;
  • breakpoints;
  • hidden semi-Markov model;
  • spatial variability

Abstract

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Hidden Semi-Markov Model
  5. 3. Fitting the HSMM
  6. 4. Spatial Variation
  7. 5. Summary and Conclusion
  8. Acknowledgments
  9. References

[1] Spatially, rainfall is very variable, and a common problem is that for a particular point of interest no observations are available. So a spatial model is required to provide suitable estimates at such points. The problem is further compounded by the many questions that can be asked regarding rainfall, with each question generally needing a particular spatial model for a particular data type. The approach here is through a temporal model of high temporal resolution rain rate data such that the spatial variability of rainfall is then, in part, that of the parameters of the model. Other aspects entailing contemporary differences in rainfall from point to point are not dealt with; only those based on long-term point statistics are considered. The breakpoint format of rainfall data, which records the rain rate and the times when it changes, implicitly provides high temporal resolution rainfall information in a highly compressed form. A Markov model was chosen so that its states could be aligned with the different physical processes that occur in the atmosphere and are associated with rainfall. However, the data consist only of rain rates and durations with no labels indicating the prevailing process for each datum; thus the states in the model are “hidden.” The model was fitted to data from 20 locations in central New Zealand, and its parameters were then mapped using a thin-plate smoothing spline and verified through simulations of artificial breakpoint data sets and the subsequent extraction of a few statistics for which independent spatial variations were available. Thus, through similar simulations at any arbitrary point a set of model parameters is available, and many rainfall questions can be answered. The model, being physically based, also provides some insights into rainfall processes.

1. Introduction

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Hidden Semi-Markov Model
  5. 3. Fitting the HSMM
  6. 4. Spatial Variation
  7. 5. Summary and Conclusion
  8. Acknowledgments
  9. References

[2] Determining the spatial variability of environmental measurements is of fundamental importance and provides a means of answering the common problem that at a particular point of interest no measurements may have been made. Thus a spatial model for the particular measurement is required to provide suitable estimates. In this paper, rainfall is the particular concern, and its spatial variability based on long-term point statistics is dealt with rather than that entailing contemporary differences in rainfall from point to point. Generally, rainfall statistics are calculated from data derived by monitoring the accumulation of rain as time progresses, and, although these statistics show less spatial variability than short-term measurements, it is still high.

[3] There are many questions that can be asked regarding rainfall ranging, for example, from what is the mean annual accumulation to what is the mean annual 10 minute maximum fall and in a direct approach each question would need its own spatial model. Rather than that, however, the approach here is through a model of high temporal resolution rain rate data such that the spatial variability of rainfall is captured through that of the parameters of the model. Essentially, the model is fitted at a number of sites where appropriate data is available then each parameter of the model is mapped using a thin-plate smoothing spline [Hutchinson, 1995]. From the mappings, many rainfall questions can be answered for an arbitrary point through the simulation of an artificial data set from the model parameters estimated for that point; the statistics concerned can then be extracted from that data set. With the additional use of the thin-plate smoothing spline, the methodology is similar to that of Sansom [1999], who considered only three points, each with 15 years of data, that spanned New Zealand with a spacing around 500–1000 km. The results presented here are from 20 points, each with 5 years of data, from the southern part of New Zealand's North Island, an area of ∼200 km by 200 km.

[4] To answer arbitrary questions about rainfall, the high temporal resolution rain rate data referred to above must measure rainfall as completely as possible. Consider, rainfall is generally observed as its accumulation over a given interval of time at a point in space. However, this totally ignores its intermittency (arguably the most fundamental property of rain), which, compared to measuring accumulations, is much more difficult to assess. Measurements at high frequency are required to provide not only the mean rain rate during periods of continuous rain between dry times but the lengths of both the wet and the dry times as well. Two words should be emphasized: point and continuous, i.e., instant by instant. However, a measurement can neither be made at a point nor at an instant and usually is made by collecting rain over an area of a few hundred square centimeters (i.e., the catch area of a gauge) and during some collecting interval. Thus, simplistically, better assessment of the instantaneous point rainfall rate could be achieved by reducing both the collecting area and interval.

[5] Consider also that rain gauge measurements are generally stored as the accumulation (or mean rate) over a fixed period. However, if the period is too long and collection area too large compared to the scale of the variation of rainfall then this fixed period representation is a poor approximation to reality. On the other hand, attempts to improve the situation by using shorter periods and smaller areas not only lead to large data sets but are limited because rainfall is a discrete process. It occurs as raindrops and, in the limit, during what could sensibly be recognized as a time of rainfall, there would be large oscillations from no rain (observations between drops) to intense rain (observations during a drop). However, Marshall and Palmer [1948], Joss and Waldvogel [1968] and Torres et al. [1994] relate the distribution of raindrop size to a rain rate which might be termed the ambient rate since it is a steady rate applying for an arbitrary period during which, and to an arbitrary volume over which, the distribution of raindrop size is stationary. Thus the fundamental point measurement for rain, rather than raindrops, is the temporal variation of the ambient rate which Sansom and Thomson [1992] and Sansom and Gray [2002] noted can be well approximated as a series of periods of arbitrary length each with the rate constant throughout and with its ends defined by abrupt changes of rate. This type of temporal variation could be stored as fixed period amounts but large data sets would result as a short fixed period would have to be used to accurately locate in time when a change in ambient rate occurred. Also, the actual time of change would not be accorded any particular importance whereas it is fundamental. Both these objections are overcome by using breakpoints which are the times when the ambient rain rate changes and the breakpoint data set consists of those times and the associated rates or, equivalently, the durations of steady rates (with dry times having zero rate). Further details regarding breakpoints and their use are given by Sansom [1992] and Barring [1992].

[6] A recent source of breakpoint data is the drip style gauge [Stow et al., 1998; Sansom and Gray, 2002], which forms the caught rain into approximately equisized drips that are then simply counted, and so it is similar to a tipping bucket in that the amount is discretized, but each drip is much smaller than the typical bucket size. This gauge itself does not produce breakpoints, but they can be automatically extracted [Sansom, 1997]. A further source developed by Sansom et al. [2001] uses an interpolation procedure on radar imagery to produce estimates of rain rate every 15 s from 15 min images; a breakpoint series can then be extracted for any pixel using the same process as developed for the drip gauges. However, the major source of breakpoint data, and that of the data from the 20 stations used in this paper, is still the manual digitisation of pluviographs from Dines tilting siphon automatic rain gauges [Sansom, 1987]. The locations of the stations used and the orography of central New Zealand is shown in Figure 1, where the main area of interest is indicated as that part of the North Island south of 40°S. Two stations were just to the north of this area, and another two farther north were also included to provide data from high elevations, and a station on the South Island was also included. These additional stations were needed to facilitate the mapping phase of the study.

image

Figure 1. Central New Zealand's orography and the locations of the stations. The numbers indicate places where the estimated model parameters are examined in detail in section 4.

Download figure to PowerPoint

[7] The modeling of breakpoint data has progressed from its decomposition into lognormal components by Sansom and Thomson [1992] to the hidden Markov model (HMM) by Sansom [1998] and finally the hidden semi-Markov model (HSMM) by Sansom [1999] and Sansom and Thomson [2001]. The model and its physical interpretation is described in section 2, while the fitting of the model to the data is given in section 3, and the determination and interpretation of the spatial variation is given in section 4. The final section provides a summary and conclusions.

2. Hidden Semi-Markov Model

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Hidden Semi-Markov Model
  5. 3. Fitting the HSMM
  6. 4. Spatial Variation
  7. 5. Summary and Conclusion
  8. Acknowledgments
  9. References

[8] The data that are to be modeled do not contain details of the collection of individual raindrops. Instead the controlling parameters of the process that produces a stationary drop distribution size (DSD) of raindrops are summarized within the data through the ambient rain rate associated with each such DSD [Marshall and Palmer, 1948; Joss and Waldvogel, 1968; Torres et al., 1994]. Essentially, certain conditions prevail for some interval within the precipitation generating mechanism (PGM); a stationary DSD is maintained through that interval; the overall effect is an ambient rain rate; breakpoint data capture the ambient rates and their durations. A particular type of PGM may well last for longer than the period of a single stationary DSD since rapid changes within the PGM will lead to another DSD being established for some further time. Furthermore, since rainfall is generated by at least two processes, one convection and the other frontal systems, there are at least two types of PGM. Other PGMs may well exist, but, initially, a given number each with its own characteristics should not be imposed; rather the ability, within the modeling, to cope with any number of PGMs is needed. Then the modeling will suggest how many PGMs exist and a null PGM will complete a set such that each moment of time can be attributed to a particular PGM. The null PGM covers conditions when precipitation is not possible but, not all dry time can be attributed to it since dry intervals may occur even within the duration of a PGM. For example, convective activity can be long lasting but it is intermittent in nature with dry periods interspersed between individual showers.

[9] These durations of, and rapid changes between, either DSDs or PGMs relate as much to space as to time since the point measures reflect not only its temporal development but also the spatial differences in the PGM as it passes overhead. Thus, in spatial terms, areas of precipitation are separated by null PGM areas; each area of precipitation consists of contiguous subareas over each of which a particular PGM is operating; each subarea is further divisible into contiguous areas each with a stationary DSD. As such spatial patterns move over a gauge, the DSD or PGM is steady or changes rapidly, provided the boundaries between different areas are narrow, and its structure appears as a hierarchical division of time. There are three levels to this hierarchy: (1) durations of stationary DSDs or the breakpoint data itself which can be termed the wet or dry durations, or simply the “wets” or “drys”; (2) durations of PGMs consisting of a series of DSDs and together composing an “episode”; and (3) a sequential series of PGMs which together compose an “event” with the null PGMs being the interevent drys.

[10] The hierarchy is illustrated in Figure 2 where one event from the top timeline has been expanded on the second timeline into a series of four PGMs, which have been labeled as either “Rain Episode” or “Shower Episode.” The choice of names for the episodes is just illustrative, and it may be that different or additional names might become apparent through the modeling. The breakdown of the PGMs into DSDs is shown on the third timeline where “w” and “d” indicate wets and drys, respectively, and each “w” or “d” represents one breakpoint. The duration of the breakpoints is shown in Figure 2 as a section of the timeline, and a rain rate is associated with each duration, the rate being zero for those labeled with a “d”. From the timelines a sequence of labels can be generated which describe the state of the system at any time. For example, at the bottom of Figure 2 and starting from the left, at the end of an interevent dry periods (“I”) a period of rain starts (“Rw”), which has a change of rate (another “Rw” is shown) before a short dry break occurs (“Rd”); eventually, rain turns to showers with a dry break (“Sd”) until a shower actually occurs (“Sw”) and so on. Note that, since only the “w” or “d” can be definitely assigned, these states are not directly or completely observed and must be inferred from the breakpoint data itself.

image

Figure 2. Hierarchical division of time into (top) large-scale precipitation events and dry interevents, (middle) rain or shower episodes, and (bottom) individual episodes of wet and dry breakpoint durations. The “d” and “w” indicate dry and wet, respectively, which is known from the breakpoint data, whereas the “R”, “S,” and “I” indicate rain or shower episodes and dry interevents, respectively, which are not known from the breakpoint data.

Download figure to PowerPoint

[11] Thus the HSMM was chosen to model the breakpoint data since its unobserved states form a suitable hierarchy for rainfall activity evolving over time. Within the unobserved states, the breakpoint observations are modeled as bivariate (for the wet data) and univariate (for the dry data) mixtures of lognormal distributions while the states follow a semi-Markov model in which with each observation a transition between states takes place with a probability that depends only on the initial and final states of the transition. However, self-transitions are forbidden by setting zeros in the diagonal of the transition matrix whose other elements represent the probabilities of other transitions. Instead, persistence within a state is handled through a dwell time distribution that specifies the probabilities that a state will persist for n (n = 1, 2, 3,…) breakpoints: A modified geometric proved a suitable choice. HSMM and HMM modeling have been used in meteorological contexts [e.g., Zucchini and Guttorp, 1991; Sansom, 1998, 1999; Hughes et al., 1999], and more general references include Rabiner [1989], Elliot et al. [1995], MacDonald and Zucchini [1997], and Sansom and Thomson [2000, 2001].

[12] As noted above, the number of PGMs and hence states was not imposed rather guidance for the selection of model order was made through the Bayesian information criterion (BIC) [Schwarz, 1978] and from physical considerations. As is the case with other HSMM and HMM applications, the recursions involved needed to be appropriately scaled in order to preserve numerical precision. Details of this are given by Sansom and Thomson [2000] as are methods needed for the treatment of censored data. Monte Carlo methods were used to obtain variance estimates for the parameters.

3. Fitting the HSMM

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Hidden Semi-Markov Model
  5. 3. Fitting the HSMM
  6. 4. Spatial Variation
  7. 5. Summary and Conclusion
  8. Acknowledgments
  9. References

3.1. Initial Model Order and Dwell Time Parameterization

[13] Much of the fitting procedure used was as detailed by Sansom [1999], but a short summary is given here and some differences are noted. The fitting procedure is a maximum likelihood procedure, which is iterative, so initial values are required. Following Rabiner [1989], uniform probabilities are sufficient to initialize the transition probability matrix, and, from Sansom [1999], if the dwell time probabilities are nonparametric (as is assumed initially) it is also sufficient to initialize these as uniform. However, according to Rabiner [1989], the parameters of the state distributions need to be initialized with values near to those that give the global maximum in the likelihood otherwise convergence may only reach a local maximum. To minimise this risk, initial values were randomly selected many times and the fitting procedure followed to convergence. Then the fit with the greatest likelihood was taken as the global maximum. Each random selection was restricted to feasible values with all the location parameters confined to the range of the data and the scale/correlation parameters kept to the same order as those of the data. While this method did not guarantee a global maximum, it was a practical strategy, which, since many such fits were made, provided some degree of confidence that the global maximum had been reached.

[14] In this way, Sansom [1999] fitted to the same data set a series of models each with a different number of states. The states were labeled as R1, R2, S1, and S2, where R refers to rain and S refers to showers and the following integers refer to subdivisions: I, interevent dry, and E, error (so named since Sansom and Thomson [1992] found a component that could be attributed to imperfections in the manual digitizing process). In general, the greater the number of states the smaller the BIC and hence the better the statistical fit but, the physical cause for each state was also assessed. This sometimes suggested that the same label was suitable for two states because they were collocated with similar links to other states. In such a case, even if the model was statistically justifiable according to the BIC criterion, it was considered that the decomposition had progressed too far and it was unlikely that two physically distinct states had been found. It might have been that seasonality or some other low-frequency variation in the data was being seen. A nine state model with four drys (R, S1, S2, and I) and five wets (R1, R2, S1, S2, and E) was adopted by Sansom [1999] since all the more complicated models had at least two states with the same physical interpretation. Sansom [1999] showed that that model was suitable throughout New Zealand, and it was chosen as the starting point for the analysis of the data from the 20 stations considered in this paper.

[15] Data for the 5 year period from January 1988 to December 1992 were used for most of the 20 stations but a few had slightly earlier or later 5 year periods. The data from Wellington (41°17′S, 174°46′E) are shown in Figure 3 by a histogram of the drys at the top and a scatterplot of the wets below. Wellington can be used as an effective example for all the 20 stations since, to the first order, the data from all stations are similar, and so the same model could be fitted at all stations. If this were not the case, then the spatial modeling presented in this paper would not be possible. In Figure 3 all rates and durations have been logarithmically transformed in an effort to make the data more Gaussian and in the light of suggestions by many authors [e.g., Biondini, 1976; Kedem et al., 1994] that many types of rainfall measurement are lognormally distributed. It can be seen that the transformation did not completely normalise the data; the drys have a heavy upper tail and the contours in the wet scatterplot are distorted from being elliptical. It was assumed that the logarithms of the data could be well represented by a mixture of normals or, equivalently, that the original data could be well represented as a mixture of lognormals with each component in the mixture corresponding to a state in the HSMM.

image

Figure 3. The breakpoint rainfall data and its fitted hidden semi-Markov model (HSMM). (top) Histogram of the 3713 dry durations. (bottom) Scatterplot of the 9390 bivariate wet observations. Both the time and the rain rate axes are logarithmic, and the central portion of the bivariate wet data has been contoured with isolines of observed frequency. The goodness of fit of the model to the dry data is shown by the curve superimposed on the histogram, while that of the wet data is shown by the dashed lines within the contoured portion of the scatterplot. These show the difference between the scatterplot's frequency contours and the equivalent ones that could have been drawn from the values of the fitted parameters, with the darker being the ±5% difference contour; the lighter shows where there is no difference. The solid line across the lower panel is explained in the text.

Download figure to PowerPoint

[16] In the nine state model of Sansom [1999] state E was attributed to poor digitizing and involved data over a wide range of rates but all with short durations, i.e., those breakpoints which would be digitized with the most difficulty. However, the HSMM could now be fitted with the shortest durations censored, so that only knowledge of their occurrence was used in the fitting procedure but their values ignored. Thus, as given by Sansom and Thomson [1998], the wet data to the left of the solid line in Figure 3 were censored, and with the number of wet states reduced to four, an eight state model fitted. The BIC for the nine state model at Wellington was 69888.6, and that of the eight state model of the censored data was 69095.4, which is much smaller, indicating a significantly better model.

[17] One of the reasons for fitting an HSMM to the data, rather than an HMM, is that the dwell time distributions are implicitly geometric for the HMM, and it is far from certain that rainfall behaves that way. Indeed, because it is uncertain how the dwell times are distributed, a nonparametric dwell time distribution over durations of 1, 2,…, D was initially fitted, where D needed to be long enough to capture the longest likely dwell periods but short compared to the amount of data. For all the data sets concerned, dwell periods of at most 30 breakpoints were expected and about 13,000 data were available, and so a D of 50 was chosen. The nonparametric dwell time distribution for Wellington is shown in Figure 4 with a geometric, fitted to the nonparametric probabilities, superimposed. Clearly, the geometric is a poor fit for three of the four states, and the necessity for an HSMM rather than an HMM is established. Also shown is the fit of a more parsimonious parametric model, fitted using the methods of Sansom and Thomson [2000, 2001]. This new parametric form, which is a modified geometric with a free parameter for the probability of a dwell of 1 and a geometric tail for dwells greater than 1, fits the nonparametric probabilities adequately and lowered the BIC to 67402.9 which, again, indicates a significantly better fit.

image

Figure 4. The fitted nonparametric dwell distributions (vertical solid lines) together with the fitted geometric distributions (dashed lines) and modified geometric distributions (points) superimposed.

Download figure to PowerPoint

3.2. Alignment of the States Between the Stations and Final Selection of Model Order

[18] The data from all other stations were also fitted to HSMMs with four dry states and four wet states using both nonparametric dwell times and ones distributed as the modified geometric specified above. The latter gave similar improvements in the BIC for the other 19 stations to that found for Wellington. Figure 5 shows the model structure in its final form using Wellington again as an example, but the structure for the fit at this stage differed from the final one in only a few details. Furthermore, much similarity was found in the structures of all 20 fits with the relative locations and scales of all states and the interconnections between the states being qualitatively similar for the fits from station to station. The main differences from the structure of Figure 5 were in the dry states where more separation occurred between the two with the shortest durations, while one of the others often tended to stretch rather unrealistically from the shortest durations to the longest. However, the fitting procedure does not label the states except as state 1, state 2, etc., and, in general, it was not the case that state N at a particular station was equivalent to state N at any other station.

image

Figure 5. The fitted state distributions: (top) univariate lognormal for the dry durations and (middle) bivariate lognormal for the wet breakpoint observations below. The labels are convenient choices with some support from the text and from state to state the same valued contours are used for the bivariate distributions. (bottom) Transitions between and dwells within the states of the HSMM. The states are placed schematically with the dry ones as large solid dots along a time line indicated by the upper axis and the wet ones as large open dots in the time-rain rate plane indicated by the lower axes. Transitions are shown by lines with attached arrows indicating the direction of transition; most, but not all, such lines have arrows pointing each way. The heaviness of the lines indicates the importance of the transitions, with the heavy solid lines representing those transitions each covering at least 4% of the total number of transitions and lighter lines representing those covering between 4 and 2% each. At each wet state's location is shown the mean number of breakpoints that occur before a shift to another state takes place; for the drys this is always just one before a transition to a wet state takes place.

Download figure to PowerPoint

[19] Using Wellington as a convenient standard, for each other station, in turn, all combinations of a subset of its model parameters were compared to the same subset from Wellington using k mean clustering [Hartigan and Wong, 1979]. Thus, for example, if only three states were concerned and S1, S2, and S3 represented the parameters of the states at a station and W1, W2, and W3, those at Wellington then clustering would be sought in the following seven-observation multivariate data set i.e., {W1,W2,W3; S1,S2,S3; S1,S3,S2; S2,S1,S3; S2,S3,S1; S3,S1,S2; S3,S2,S3}. The clustering was performed in stages with two clusters being found at each stage and the combinations in the cluster not containing the Wellington values being discarded. This was repeated until only one combination clustered with the Wellington values, and it was taken to have its states aligned with Wellington's. Occasionally, more than one combination was in the final cluster, and a selection was made manually. A final visual check was made using plots similar to Figure 5, and some further changes were made. The alignment of the states was not strictly necessary for the next stage of the fitting, but it did facilitate the identification of poorly fitted states, which was made through the estimation errors of the model parameters.

[20] To assess estimation errors, fits were made to simulated data based on the parameters estimated by the fit to the actual data. At each station, 50 simulated data sets and 50 sets of parameter estimates were created in this way using much the same number of breakpoints as that in the original data set for the station concerned. (Technically, the number kept the same was that of the state visits, which was estimated from the original data using equation (6) from Sansom and Thomson [2001]). The standard error of each parameter estimate from the actual data was then estimated by its standard deviation over the 50 simulations with any bias in the estimation procedure being checked by comparing the parameter estimates from the actual data set to the mean of that parameter's estimates from the 50 simulations. The statistic of interest (Z) is the ratio of the difference between the mean, over the 50 simulations, of the parameter (pm) and the estimate of the parameter (pe) to the standard deviation, over the 50 simulations, of the parameter (sm) divided by the square root of 50:

  • equation image

[21] Rather than computing such statistics for all 72 parameters of the model, only those for the 12 location parameters (i.e., the four mean rates and four mean durations for the wets and the four mean durations for the drys) were found. For unbiased estimates this statistic should be normally distributed; however, 38% of them had values less than −3, and 8% had values greater than +3, leaving only 54% between −3 and +3 rather than the 99.7% that might be expected for normal variates. The parameters most affected were those for the wet states that had been most heavily censored, by up to 10% each, although at most the overall censoring was 2% of the data.

[22] The initial censoring line had been based on that successfully used by Sansom and Thomson [1998]; now simulations were used to assess the optimum positioning of a censoring line. From the parameters fitted to the actual data sets at each of the 20 stations, data sets were simulated which were then censored to varying degrees, and the model was refitted. Both the slope and the position of the censoring line had to be changed from those initially chosen, with the solid line in Figure 3 being a suitable compromise between removing enough poor data to prevent it adversely affecting the fit and not so much that the biases found above reoccurred. The overall censoring was reduced to 0.5% or less. The BICs increased (up to 67,566.3 for Wellington), but, on repeating simulations to assess the estimation errors of the fitted model parameters, the statistic of equation (1) was more normally distributed with 88% between −3 and +3, 2% greater than +3, and 10% less than −3. However, half of those below −3 were for the mean duration of the longest dry periods. This general bias to a shorter mean duration for state I could result from data defects and/or by its representation as a lognormal component being an unsuitable choice.

[23] Defects in the dry data, which had not been censored, were unlikely to occur at the high end where the length of the durations was much greater than any error that manual digitizing could introduce, but the shortest periods were prone to poor digitizing with a high probability of unreasonably short periods being spuriously introduced. Such an excess of short dry periods might give rise to the tendency, as noted earlier, for state I to stretch from the shortest durations to the longest. Thus the censoring of short dry durations was introduced and optimized through simulations and refitting after various censoring thresholds had been applied. In general, the removal of the few extremely short durations, less than 0.1% of the whole data set, resulted in significant differences in the parameter estimates, but any further censoring rapidly led to much larger differences; the minimal censoring was introduced.

[24] With regard to the suitability of state I being represented by a lognormal component, Sansom [1995] and Sansom and Thomson [1998] refer to a companion to state I which might be termed state M, where M refers to multiple since its originates from the concatenation of two or more other dry states. Primarily, it would be expected to arise if between two occurrences of state I a weak event took place, which did not give any precipitation at the observation site. It is known that such events do occur from synoptic weather observations that commonly report precipitation near to but not at the point of observation. Such cases cannot be recognized from just the data, and, even if they were, those dry durations could not be separated into their parts, but the existence and handling of such behavior should be incorporated within the model.

[25] Three ways were tried. A fifth dry state was introduced, but this state M produced as many biased estimates as state I had previously; also the BICs increased markedly which was mainly due to the 10 extra parameters in the model. Second, rather than introduce an extra state, state I was modeled as a mixture of two lognormals, which ensured that the two components of state I would behave in the same way regarding transitions to other states, and this only increased the number of model parameters by three. This model structure still produced biases but only in 9% of the parameters, which were not all associated with state I, and the BICs all decreased; however, for all stations one of the other dry states had significant presence across the whole range of dry durations. The third method was to retain state I as a mixture but reduce the total number of dry states to three, since in the previous scheme the state stretching across all durations clearly had no particular function and might well be superfluous. As with the previous scheme, 9% of the parameter estimates had some bias, again not all associated with state I, but most the BICs improved even further, and none of the dry states stretched across the whole duration range.

[26] Thus a model with three drys states with the state I requiring a two-component mixture was best, but having reduced the number of states, it seemed prudent to check that the second component of state I was strictly necessary. When it was dropped, still 6% of the parameters had some bias, all of the BICs increased, and once again state I stretched across the whole duration range; the second component of state I was needed. The components of this HSMM fit to the Wellington data are shown in Figure 5, where the univariate and bivariate normal component distributions are shown scaled to a size appropriate to the relative frequencies of the states. The scaling factors for the wet states are determined from the estimated transition probability matrix and the dwell time distributions. A comparison between the fit and the data is given in Figure 3. For the drys the overall fitted distribution is superimposed on the histogram and shows a good fit between the data and the model. For the wets the percentage difference between the empirical distribution of the wets and the overall fitted distribution is shown by dashed lines. It can be seen in Figure 3 that the bulk of the wets is modeled well, and even around the extremes the absolute difference is not often more than 5%.

[27] The bottom panel of Figure 5 shows those transitions, which collectively account for 94% of all the transitions that occurred in the data. The states are placed in the same relative positions as in the top panels except that the I and M substates are shown together (i.e., the solid dot on the right) at a position that is the mean of their combined distribution. It can be seen that eight transitions, in four pairs, are indicated as each contributing over 4% to the total number, and they collectively account for 70% of the total number. These four connections partitioned six of the seven states into two groups. The first group linked S1 with R and I/M (to which R1 and R2 are also weakly linked), and the second group joined S2 with S and I/M. The first group appears to represent the sequence of showers and rain associated with a frontal system; the second group represents periods of convective activity. The two groups are weakly linked from S2 to S1 but are largely independent since otherwise they are separated by interevent drys. It is also interesting to note that heavy and persistent rain (R2) always occurs after an interevent dry and decays to lighter and less persistent rain (R1).

3.3. Restricting Transitions and the Final Model

[28] The necessary conditions for the proposed spatial variability scheme to be feasible and effective are the following: The same model structure applies over the whole area where the spatial variability is being found, the model's parameters vary slowly and smoothly with dependence only on position and orography, and the model is physically meaningful and capable of similar interpretation at all stations. The first condition has been imposed, but the others needed to be verified. It has been noted that all the data sets were similar to Figure 3 and that, at least superficially, all the model fits were similar to Figure 5; thus the second condition could be assumed.

[29] To further ensure that fits from station to station were as compatible as possible, a refitting procedure was followed in which each station was refitted using the fits from itself and all the other stations as initializations and then taking the fit with the highest likelihood as the best fit. In general, it was not the case that the best fit at a station was that initialized by its current fit, and refitting was repeated until most stations were best fitted from themselves. It is this fit for Wellington that is shown in Figure 5. Fits to the other stations were broadly similar, with the main differences being in the position of state I and in the less important connections between states (i.e., the minor structure depicted within plots of the kind shown in the bottom panel of Figure 5).

[30] Twelve of the stations had the I and M states similar to Wellington, where their relative locations and scales can be justified as follows: State I represents the lognormal distribution of interevent durations (variate X), whose mean and variance relate to that of the normal distribution, N(μ,σ2), of the logarithmically transformed data, i.e.,

  • equation image

If two interevent durations became concatenated because the event between resulted in no precipitation at the observation site, then the total duration would be an observation from the distribution of 2X or when logarithmically transformed from the normal distribution N(μ′, σ′2). It can be shown that

  • equation image

Since the durations spanning a missed event (or events) might also include one or more S or R type drys, an exact fit to this argument was not found. However, for the values encountered, the relations imply that σ′2 is somewhat less than σ2 and that μ′ exceeds μ by ∼1 or 2; these features can be seen in the Wellington fit and the 11 similar to it. By refitting using initializations from these 12 stations, similar features were imposed at the other eight stations, and some progress was made toward ensuring that the third condition was satisfied; that is, the model has the same physical interpretation at all stations.

[31] Apart from forbidding self-transitions, no restrictions had been imposed on which transitions could occur. Within all the data sets a dry period never follows a dry period; thus all the probabilities for transitions between the dry states were zero. In addition, some of the remaining possible transitions had probabilities which were close to zero, and by setting them to zero in all initializations it was possible to impose further common structure onto the fits to all stations. As noted above, variation between stations fits existed in the less important connections between states, and if the mean across the stations for a particular transition probability was under 0.05, then it was set to zero for all stations. This reduced the number of parameters associated with the transition matrix from 35 to 17, and the BICs for all but one of the stations decreased. Also, it became clear that state R2 could have two functions: Either after a mean persistence of six breakpoints before a transition it connected to state R1 and occasionally state I and so was associated with frontal activity, or after a mean persistence of three breakpoints it connected to state S2 and state S and so was associated with convection. The relatively high persistence, which would not generally be associated with convection, supports frontal activity as the more likely function for state R2. However, all stations can have state R2 forced into taking on one function or the other by setting the appropriate transition probabilities to zero. In each case the number of parameters was reduced from 17 to 12, and the BICs for 16 of the 20 stations indicated that the model having state R2 involved in frontal activity was better.

[32] To again ensure that fits from station to station were as compatible as possible, the refitting procedure was followed. For state I/M, 10 stations retained the physically acceptable pattern described above, and, as before, by refitting using initializations from only those 10 stations, similar features were imposed at the others. The BICs for those stations increased, but all BICs were still smaller, except at two stations, than those for the unrestricted transitions model. Also, refitting to data sets simulated using these fits resulted in nearly 99% of values of the statistic of equation (1) being between −3 and +3 with a worse case of −5.9 for a parameter from one of the stations whose BIC was larger than its value for the unrestricted transitions model. For this station, state I/M was located at much lower values than for all other stations, so it was not used in the determination of the spatial variation, especially as it was located close to another station; it was one of the overlapping pair just south of 41oS and east of 175°E in Figure 1.

4. Spatial Variation

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Hidden Semi-Markov Model
  5. 3. Fitting the HSMM
  6. 4. Spatial Variation
  7. 5. Summary and Conclusion
  8. Acknowledgments
  9. References

[33] The HSMM is not a spatial model since the model parameters can only be estimated at a point without regard to any data away from that point. An initial assumption that the same model structure would apply over the whole area had been made and was subsequently endorsed by the similarity found between the fits at the 19 stations, which, in turn, supported the view that the model was physically meaningful. In particular, states were located in physically understandable ways as were the transitions between them; moreover, not all possible transitions seemed to occur. Consequently, some transitions were explicitly excluded which not only reduced the number of parameters in the model but also ensured that the model had the same physical interpretation at all stations. The further assumption that the spatial variations of the model's parameters should be slow and smooth was also supported by the similarity of the 19 station fits.

[34] A desired consequence of that assumption is that given the values of the parameters at relatively few points, they can be adequately estimated elsewhere without needing any independent information apart from the position and height at the points where the parameters are known. The questions at this stage were as follows: Would the 19 stations be enough? Would the accuracy of parameter estimation at those points be sufficient despite a spatial model not being used? Overall, how could the adequacy of any estimations be assessed? Since all available data had been used, no independent direct verification would be possible, but the estimated parameters could be used to simulate data sets at a spatial density comparable to that of stations that provide daily rainfall observations. Then the spatial variations of statistics extracted from the simulations could be compared to those of statistics of the same type already available from the daily rainfall network.

4.1. Application of the Thin-Plate Smoothing Spline

[35] The spatial variation of the model parameters for the 19 remaining stations was determined using Hutchinson's [1995] partial thin-plate smoothing spline interpolation technique, which deals with dependencies on position and orography. The spline is a surface that is fitted to spatially distributed data with some error assumed at each data point, so it can be smoother than if the data were fitted exactly. A single parameter controls the smoothing and is often chosen by the method of generalized cross validation (GCV), in which each station is omitted in turn from the estimation of the fitted surface and the mean error is found. This is repeated for a range of values of the smoothing parameter, and then the value that minimizes the mean error is taken to give the optimum smoothing. However, in many climatological data sets, which often have little data and are noisy, using the GCV can result in unrealistically smooth maps with unacceptably large differences between the data and the spline fit. To address this, Zheng and Basher [1995] manipulated the signal and error characteristics of the data and spline fit and found that enforcing a global value for the ratio of signal to error (a quality measure available from the spline fitting procedure) provided a useful and intuitive procedure for understanding and controlling the fitting. The larger the signal to error ratio is, the closer the fitted surface passes through the data and the smaller the error of the fitted values.

[36] The final structure of the HSMM model had 49 parameters each of which needed its spatial variation determined, but if a spline surface had simply been found for each, then it would have been unlikely that the spline parameter estimates at an arbitrary point constituted a valid and consistent set. In particular, the sum of the transition probabilities from each row of the matrix must sum to one. An attempt to build in such dependency was made by dividing the HSMM parameters into two groups and applying the following methodology: (1) Find spline surfaces for the first group using the GCV criterion and retrieve their spline estimates at the station locations; (2) refit the HSMM at each station using the spline estimates of the first group as fixed constants rather than parameters (i.e., only the parameters of the second group are reestimated); (3) find spline surfaces for the refitted second group using the largest possible signal to noise ratio so that the spline fit at the station points would be nearly exact. Thus the final spline fit at each station yielded parameters close to those that maximized the probability of the station's data given the model and the transition matrix's row sums were close to unity.

[37] This scheme allowed the members of at least one group of parameters the freedom to have the smoothing of their spatial variations optimized using the full potential of the spline's ability to differentiate actual spatial variation from noise. Such freedom for the other group had to be sacrificed to ensure consistency through the parameters and to maintain their values close to the most likely values. The 49 parameters can be divided naturally as the parameters of the state distributions as one group and all the probabilities of the dwell distributions and transition matrix as another. The question of which of these groups was the “first” was settled by letting each in turn be the “first” and then refitting the HSMM and using the refitted parameters to simulate long breakpoint data sets from which some basic statistics were extracted for comparison with similar statistics extracted from the actual data sets.

[38] The statistics chosen for the comparison were the mean total annual accumulation (annual total), the mean annual maximum fall in 10 min (10 min fall), and the mean annual number of days with at least 1 mm of rain (wet days). Because the model has no seasonality, just simulating single years would probably have given too wide a range of annual totals since in any particular simulated year a season might well be overrepresented; for example, it could be like one long winter. So simulations were made for 5 year periods, the same period length as the data, to ensure good estimates of the annual total and of the standard error of 5 year totals, which needed to be multiplied by √5 to annualize the standard error. This also applied to wet days but not to 10 min falls where the mean of the 10 min fall as well as its standard error needed adjustment since the annual 10 min fall is not linearly related to the 5 year 10 min fall. The adjustment was as follows: Apply a double log transform to the set of five yearly 10 min falls, extract the mean and divide it by 1.290, or extract the standard error and divide it by 1.515, then back transform the results. These divisors were determined empirically.

[39] With one statistic per panel, Figure 6 shows a comparison of the three statistics at the 19 stations with a group of five whisker plots for each station. Each whisker plot shows the mean value and a 90% confidence interval (CI). From left to right the whisker plots in each group are for the whole long-term record of the station (LT); the actual 5 year period used in this study (ACT); 100 simulations using the HSMM parameters (HSMM); 100 simulations using GCV spline estimates of the state distribution parameters and refitted HSMM estimates of the probabilities of the dwell distributions and transition matrix (GCV1); as GCV1 but GCV spline estimates of the probabilities and refitted HSMM state parameters (GCV2). Generally, the 5 year period used was an acceptable sample with respect to the longer-term behavior with only a few occasions when the LT and ACT 90% CIs did not overlap. Also, the HSMM parameters before any adjustment by the spline generally gave simulated data sets from which the extracted statistics were close to those from the actual data. This further endorses the model and its particular structure as appropriate for a wide ranges of scales since it reproduces statistics with durations from 10 min up to a year and with amounts ranging from the zeros of the dry days to annual totals of a 1000 mm or more.

image

Figure 6. A comparison of the three statistics (one per panel) from five different sources at the 19 stations with a group of five whisker plots for each station, which have been ordered from left to right by the mean annual total over the data period. Each whisker plot shows the mean value and a 90% confidence interval (CI), and the annual total values have been plotted at half their real values for the station on the extreme right. From left to right the whisker plots in each group are as follows: for the long-term record (LT); for the actual 5 year period used (ACT); for 100 simulations using the HSMM parameters (HSMM); for 100 simulations using GCV spline estimates of the state distribution parameters and refitted HSMM estimates of the probabilities (GCV1); and as GCV1 but GCV spline estimates of the probabilities and refitted HSMM state parameters (GCV2).

Download figure to PowerPoint

[40] Figure 6 shows, in particular, that the GCV2 simulations are closer to ACT than are the GCV1. A simple scoring scheme was used to provide an objective assessment: (1) if the GCV mean lay within the ACT CI or if the ACT mean lay within the GCV CI, then score 2 or (2) if the GCV and ACT CIs overlap then score 1, giving for 19 stations and 3 statistics an overall maximum score of 114. GCV1 scored 93, and GCV2 scored 103. This implies that either the HSMM estimations of the probability parameters are less certain than those of the state distributions and/or the simulation of data sets is more sensitive to the values of the state distribution parameters. In the first case, optimal spline fitting using the GCV criterion should enhance the estimation of the probability parameters since it provides some spatial considerations such that neighboring stations gain mutual support for their parameter estimates. The second case was examined by repeating the spline fitting at various signal to error ratios rather than using the GCV. For statistics from simulations to be as good as those of GCV2, a ratio of no less than 4 to 1 was required, but Hutchinson and Gessler [1994] caution that the ratio should not exceed unity, otherwise the spatial patterns lose robustness and become sensitive to extra data.

4.2. Verification of the Thin-Plate Smoothing Spline Fit

[41] Thus the GCV2 scheme was adopted, but, before the final step of using the largest possible signal to error ratio for fitting the state distribution parameters, the possibility that the dependence of the parameters on height might be lost with the imposition of a high signal to error ratio was checked. All model parameters were fitted at a series of ratios. In general, the parameters tended to limiting values as the signal to error ratio increased; moreover, the absolute values at the limits were larger at the higher ratios, and the standard errors of the parameter's estimations were smaller.

[42] Spline estimates of the HSMM parameters were produced for a 6 km grid over the North Island of New Zealand to the south of 40°S. Although the model has 49 parameters, a total of 55 spline surfaces were fitted, with the extra six arising from the transition matrix's row sums being unity, so each row has one less parameter than it has nonzero probabilities. Also, one of its rows only has one nonzero probability; thus there are six extra rather than seven. Using the largest possible signal to error ratio in deriving the spline surfaces should have resulted in consistent parameters at each of the 563 grid points; in particular, the transition matrix's row sums should have been unity. However, for four of the states this was not always the case, with two of the wet states having a few row sums of up to 1.74 and 1.28 but means (over the 563 grid points) of 1.04 and 1.02, respectively. To achieve consistency, the transition matrix rows were simply scaled by their row sums.

[43] At each of the 563 grid points, 5 years of breakpoint data were simulated 100 times and the annual totals, 10 min falls, and wet days were extracted as described above. The spatial variations of these statistics derived from the simulations are shown in the top panels of Figure 7. The patterns of variation do accord with expectations; that is, all the statistics increase with elevation (Figure 1 shows the orography), and with most rain-bearing airstreams being from the west, a weak rain shadow exists in eastern areas. Comparisons with independently derived spatial variations of the same statistics are shown in the bottom panels of Figure 7. In each case the difference between the HSMM and independent values as a percentage of the HSMM value at each grid point has been mapped. For the annual totals the independent derivation [Thompson et al., 1997] was not available for areas north of ∼40.3°S, and although the mean difference over the grid points was 2.5%, areas with HSMM values over 20% larger in the ranges and the southwest and over 20% smaller in some parts of the east coast can be seen. Despite a general appearance of better agreement than existed for the annual totals, for 10 min falls the mean difference over the grid from the independent derivation [Thompson, 2002] was −10.2%, and departures of over 20% existed. The independent derivation for wet days was based on a GCV spline fit to 211 daily reporting stations; again, the agreement appears better than that of the annual totals, but departures of over 20% existed, and the overall mean was 3.2%.

image

Figure 7. (top) Spatial variations of the three statistics from the HSMM method. (bottom) Percentage difference between the HSMM method and an independent estimation. Solid lines show the 0% contours, dashed lines show the +20% contour, and dotted lines show the −20% contour.

Download figure to PowerPoint

[44] For each statistic both the HSMM and the other derivation provided estimations with their own errors from the actual, but unknown, value, so the degree of agreement was adequate and shows both methods to be at least partially successful. However, the HSMM method used only 5 years data from each of 19 stations, whereas longer records from many more stations were used by the independent derivations, so they were more demanding with respect to network density and record length. Also, the independent methods were each developed specifically for the statistic concerned, while the HSMM method is not particular to the three statistics of Figure 7, and any annual statistic can be estimated through simulations. The spatial variations of the comparisons in the bottom panels of Figure 7 show systematic patterns with large contiguous areas of positive and negative percentage difference. In particular, the HSMM fields were comparatively more extreme with the wet areas wetter and the dry ones drier, that is, higher annual totals and 10 min falls and more wet days at higher elevations and, to a lesser extent, in the west. However, the main promoter for this pattern was an extremely wet, high-elevation station out of the area of interest, that is, the most northerly and westerly station shown in Figure 1. It was included as a unique source of breakpoint data from high elevation without which larger errors would have resulted. A high-elevation station in the area of interest and some more data from the east coast would have been beneficial. Also, the Thompson et al. [1997] estimation of annual totals had a bias to slight underestimation in the area just east of the main ranges where the largest area of over 20% difference occurs in the bottom left panel of Figure 7, so the HSMM could well be more correct. It is also likely that, being based mainly on low-elevation stations, the spline estimation of wet days would have underestimated the increase of their number with elevation, and, again, the HSMM could well be more correct.

4.3. Physical Interpretation of the Model and its Spatial Variation

[45] From the 563 grid points eight have been chosen, and their parameters are displayed in Figure 8 in the manner of the bottom panel of Figure 5. The eight, whose positions are indicated by the numbers in Figure 1, are referred to below as GP1 to GP8, and they were chosen in the light of the orography and the spatial patterns of Figure 9 and 10

image

Figure 8. The transition probabilities and dwell times, shown in the style of the bottom panel of Figure 5, for the positions indicated in Figure 1 by numerals are denoted in Figure 8 by GP1, etc.

Download figure to PowerPoint

image

Figure 9. Spatial variations of some selected model parameters and derived statistics for the wet states. The maximum and minimum values over the whole field for the particular parameter or statistic are shown.

Download figure to PowerPoint

image

Figure 10. Spatial variations of some selected model parameters and derived statistics for the dry states. The I and M components of state I/M are shown separately, and the images in the last column are the probabilities from the I/M row of the transition matrix. The maximum and minimum values over the whole field for the particular parameter or statistic are shown.

Download figure to PowerPoint

[46] A rainfall event starts with a transition from the interevent dry state designated as I/M in the text and shown by solid octagons in Figure 8. The basic pattern of connections in the panels of Figure 8 is most clearly illustrated by GP3 where two types of event can be seen. The simplest is just a period of showers with occasional dry breaks, and it consists of the states indicated by triangles in Figure 8; this type will be termed a convective event with the wet and dry states designated as CS and S. The other type of event is more complex with three wet states (the square designated as LR for light rain, the diamond as HR for heavy persistent rain, and the octagon as FS for frontal showers) and a dry state (the solid square, R). It may well represent the passing of a frontal system, so it will be termed a frontal event. For GP3 such an event can start with any of its wet states but cannot end from HR, and once the system has reached the FS stage, it cannot revert directly to LR, although there can be interchanges between LR and HR. No dry breaks occur from HR, but both LR and FS can have dry breaks, i.e., the transitions to R.

[47] There were 19 nonzero probabilities in the model's transition matrix, but only 14 transitions are shown for GP3 in Figure 8. The ones shown account for 93% of all transitions, so some other less frequent ones take place. For other GPs those may be frequent enough to feature, whereas some from GP3 may not appear. In particular, GP3 is unusual in not showing the transition back from a convective event to I/M, which must occur unless a transition to a frontal event takes place. The latter occurs, but not vice versa, at all GPs apart from GP3 and GP4 with generally less than 1% of the transitions being from CS to FS. Also, at half the GPs (2, 4, 5, and 6), FS can develop into LR, not often at GP2 and GP5 but for over 4% of the transitions at GP4 and GP6. The model also has provision for the return of LR to FS, but no example of that appears in Figure 8. Finally, frontal events can end from HR, but in Figure 8 the only example is GP4. For each GP, there were generally seven or eight transitions, which covered at least 80% of the transitions, and another six or seven, each representing ∼2% and together bringing the total to 99%, so there were usually four or five transitions which together only contributed 1%, but these differed from GP to GP.

[48] Figure 8 gives insight into the temporal structure of events and, as given above, allowed the physical interpretation of the model and the labeling of its states. Figures 9 and 10 display the spatial structure, but, rather than giving the patterns for all the HSMM parameters, only some selected parameters and derived fields are shown. The derived fields are the fractional representation that a state has of the total number of breakpoints derived from the transition matrix and the dwell time distributions and, in Figure 9 alone, the mean dwells derived from the two parameters of the dwell time distributions. Thus, in some comprehensible way, Figures 9 and 10 show most of the parameters apart from the standard deviations and correlations of the state distributions, which, in general, showed little spatial variation. The standard deviations tended to increase with height. The correlations between the duration and rate variates in the wet breakpoints were always negative and became more negative with height but were generally in the range −0.3 to −0.5 except for FS, where values ranged from −0.6 to −0.8.

[49] Although there are no discontinuities and they smoothly change from one to the other, three rainfall regimes can be distinguished: (1) the low-lying areas to the west of the main ranges that run southwest to northeast and the low-lying areas immediately to the east of those ranges; (2) those ranges themselves together with other areas of high elevation near GP3 and GP6; and (3) the remaining eastern areas especially near the coast. The behaviors of rainfall in these areas are as follows.

  1. Mean wet durations and dwell times are generally small, but rates can be at their highest (CS near GP2) or lowest (LR and HR near GP5). Dry breaks within events are generally short but ones within convective events are at their longest in the northwest; I/M are generally long. Most breakpoints are CS, which is also the state from which most events end, whereas events most frequently start with FS. GPs 1, 2, and 5 show that convective events can connect to FS, which is only weakly, or indirectly through R, connected to LR. Most LR occurs at the start of a frontal event and may have associated periods of HR.
  2. Persistence is highest in the ranges and especially near GP4. Also, mean wet durations are longest except for LR, which does have its longest values near GP6 but only medium values in other ranges. Both LR and HR mean rates are at their highest, but FS rates are low and at their lowest near GP3. S dry breaks are at their shortest, but R breaks are at their longest and about as long as I/M drys, which mainly consist of I and are much less frequent than elsewhere. Most breakpoints are LR with a large contribution from HR especially near GP3, which is also the state from which most events end, whereas events most frequently start with FS, although near GP3 as many start with LR or HR. GPs 3, 4, and 6 show that convective events rarely connect to FS which is generally strongly connected to LR.
  3. The main difference from other low lying areas is that, as illustrated by GPs 7 and 8, the connection between FS and LR is even weaker, and with few dry breaks from LR, frontal events are more distinct than elsewhere. However, with the main access to FS from LR being through I/M, the FS associated with a particular event may be sufficiently temporally separated from the LR to seem to be part of a separate event. Also, the mean duration and dwell for HR is higher.

5. Summary and Conclusion

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Hidden Semi-Markov Model
  5. 3. Fitting the HSMM
  6. 4. Spatial Variation
  7. 5. Summary and Conclusion
  8. Acknowledgments
  9. References

[50] The model of Sansom [1999] was refined using the methods of Sansom and Thomson [2000, 2001] and applied to the 20 stations shown in Figure 1, but one station was discarded. The new methods allowed the data to be censored by a minimal, but sufficient, amount to eliminate the worst outliers in the data. They also handled the estimation of the parameters of the modified geometric distribution that was used for the dwell times in a particular state. Several ways of treating the dry data were examined with the best being to deal with the interevent drys distribution as a mixture of two normal components, i.e., the actual interevent drys and ones that had doubled up through the nondetection of a weak event. The final refinement was to explicitly exclude the rarest transitions in such a way that the final model structure was physically believable. The BIC was used throughout the model fitting but never at the expense of physical credibility, which occasionally meant the adoption of models with a lower likelihood than the maximum that could be found. In particular, to ensure a common model structure over all the stations, refitting was done at all stations using for initializations the fits of just those (usually a majority) that were physically credible. This resulted in the most likely models that had the desired structure but not always in the overall most likely model.

[51] Hutchinson's [1995] partial thin-plate smoothing spline was used to determine the spatial variation of the model's parameters. Initially, the model's probability parameters were fitted spatially, and spline-estimated values were recovered at the station points using the GCV criteria. The model's other parameters were then reestimated from the data using a modified form of the HSMM fitting procedure in which the probability parameters were treated as fixed constants; that is, the spline estimated values were used and not changed. Basic statistics for the stations were recovered well from data sets simulated using probabilities from the spline fit and the modified HSMM estimated state distribution parameters (see Figure 6). This collection of parameters was then used in high signal to noise ratio spline fits, and sets of parameters were estimated for a 6 km grid over the area of interest shown in Figure 1. These fits were verified by extracting statistics from simulations based on the parameters at each grid point and comparing them to independent estimates of the same statistics. On average, they compared well, but areas of systematic discrepancy existed. However, even if the HSMM method is not as accurate as the independent methods (i.e., those of Thompson et al. [1997] and Thompson [2002] and the spline fit to daily data), it is less demanding with respect to network density and record length. Also, the HSMM method is not particular to the three statistics chosen for verification, so any annual statistic can be estimated through simulations, whereas the independent methods were each developed specifically for the statistic concerned; the HSMM method is more general.

[52] The physical interpretation of the model is that the interevent periods are a mixture of single interevent periods and ones that have doubled up. Events are either convective and consist of some showers with dry breaks, or they relate to frontal passages when three wet states, again with dry breaks, are possible. The frontal states represent periods of heavy persistent rain or lighter rain or light, probably postfrontal, showers and dry breaks during the light rain or showers. These event types are not everywhere as distinct as has just been suggested since heavier convective showers can change to lighter frontal showers without an intervening interevent dry period. Indeed, about the east coast such changes are more probable than a change from light rain to light showers, and frontal events are mainly periods of light and heavy rain with any postfrontal showers separated from the frontal event by an interevent dry. The distinction is also blurred in the ranges where the convective states are much less frequent than elsewhere and frontal states are much more persistent.

[53] This interpretation largely maintains consistency with the anecdotal ideas and traditional views, which were used as guidance in the development of the model, but some aspects of Figures 8, 9, and 10 do not go easily with that intuition. In particular, although they cover the long dry periods that can easily be associated with interevent dry times, I/M mean values were found to be shorter than might be expected. Also, a clearer distinction between event types would be desirable. The model structure was chosen with care but still may not be the one that best reflects all the physical processes that take place; each element of the model should have a corresponding physical element, and all physical elements should be represented in the model. Furthermore, the transitions that were retained in the model were those that appeared most prevalent, but some of the retained rarer ones could still have no physical relevance, or some that were dropped maybe should have been retained. However, the model does enable in a general way the estimation of the spatial variation of annual rainfall statistics and provides useful and important insights into rainfall processes.

Acknowledgments

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Hidden Semi-Markov Model
  5. 3. Fitting the HSMM
  6. 4. Spatial Variation
  7. 5. Summary and Conclusion
  8. Acknowledgments
  9. References

[54] This work was carried out under Foundation for Research, Science and Technology contract CO1X0030.

References

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Hidden Semi-Markov Model
  5. 3. Fitting the HSMM
  6. 4. Spatial Variation
  7. 5. Summary and Conclusion
  8. Acknowledgments
  9. References