Spatial and temporal variation of rainfall over New Zealand



[1] A hidden semi-Markov model (HSMM) was applied to high temporal resolution rain rate data, and, after determining a transition structure for all of New Zealand, the model's parameters from fits at about 90 sites were mapped. The addition of bogus sites was required before a verification gave an acceptable result for some annual statistics over a 0.1° by 0.1° (i.e., about 10 km) spaced grid. Synthetic rainfall series were generated at each of the grid points, and various aspects of rainfall climatology were mapped. The rainfall process can be viewed as a succession of precipitation events which individually break down into a succession of episodes of either rain type precipitation (i.e., large spatial scale and persistent) or shower type precipitation (i.e., small-scale and short lived). The synthetic data sets showed that most rainfall events consist of a single shower and the rest rarely consist of more than two interchanges between rain and showers. These interchanges take place between periods of rain and showers centered at the higher end of the range of rain rates, and the transition from rain to showers is about three times more likely than that from showers to rain. The area with the highest probabilities for these transitions is the Southern Alps in the west of the South Island of New Zealand.

1. Introduction

[2] For short timescales, weather radar images and time series of high-resolution rain gauge measurements indicate that rainfall has great variability through both space and time. The variability reflects the range of action of the precipitation generation mechanisms (PGMs) and the understanding of these can be deepened by taking a long-term view through the extraction of longer timescale statistics. These highlight the signal within the noise and it is their variability that is described in this paper. Thus the long-term spatial variability of rainfall is described rather than that of contemporary differences from point to point and, temporally, it is that of the overall tendencies of the evolution of rainfall events averaged over many occurrences that is described.

[3] In a similar past study for New Zealand, Sansom [1999] used only three sites which were widely separated and so revealed little spatial detail but were sufficient to highlight some common features especially regarding the temporal variability. At all locations, wet and dry times could be put into two groups such that one (i.e., rain) was characterized by longer periods of lighter precipitation with few dry breaks, while the other (i.e., showers) had shorter but generally heavier periods of precipitation with often long dry breaks in between. The large-scale spatial variability of the rainfall climatology was assessed through model statistics with the frequency of events and the amounts from and durations of rain, as distinct from showers, being found to be the most variable. Also, usually only one episode of rain and one of showers constituted a precipitation event.

[4] The methodology of Sansom [1999] was extended by Sansom and Thompson [2003] to include the continuous, and detailed, determination of spatial variability. Several ways of treating the dry data were examined with the best found being to deal with the interevent drys distribution as a mixture of two normal components, i.e., the actual interevent drys and the ones arising from the concatenation of two consecutive interevent drys when a weak event gave no rain at the observation site. As in the earlier study, events were either convective and consisted of some showers with dry breaks or related to frontal passages with three wet states, again with dry breaks, possible. The frontal states represented periods of heavy persistent rain or lighter rain or light, probably postfrontal, showers and dry breaks during the light rain or showers. These event types were not everywhere well defined since heavier convective showers could change to lighter frontal showers without an intervening interevent dry period. Indeed, in parts of the study area such changes were more probable than a change from light rain to light showers and frontal events were mainly periods of light and heavy rain with any postfrontal showers separated from the frontal event by an interevent dry. The distinction was also blurred in the ranges where the convective states were much less frequent than elsewhere and frontal states much more persistent.

[5] However, the area considered by [Sansom and Thompson, 2003] was only that part of New Zealand's North Island which is south of 40°S (see Figure 1) and little was determined regarding the evolution of rain. This lack is addressed in this paper and the coverage extended to be country-wide by using high temporal resolution rainfall data sets from a number of sites throughout New Zealand. Section 2 describes the data sets, section 3 describes a model to which they were fitted, and in section 4 the fitting procedures are described. The model captures the short-term temporal variability in its parameters while section 5 describes the application of a thin-plate smoothing spline to determine their spatial variability. Using parameters estimated from the spatial interpolation, the synthetic data generated at all points over New Zealand on a 0.1° by 0.1° spaced grid were used, as described in section 6, in a verification against independent data. In section 7, the synthetic data are used to illustrate the spatial and temporal variation of rainfall and a summary is given in section 8.

Figure 1.

Schematic orography of New Zealand with the higher elevations shown with the darker shading except the highest elevations (>3000 m) in the Southern Alps which run near the west coast of the South Island are shown in white. The sites with breakpoint data are shown as black dots, and the bogus stations referred to in section 6 are shown as open circles. Also shown are the seven regions and their names.

2. Breakpoint Data

[6] The type of rainfall data that was analyzed is termed “breakpoints” and Sansom [1999] and Sansom and Thompson [2003] both provide in depth descriptions with J. Sansom et al. (Enhancing the physical significance of rainfall breakpoints through two-dimensional video distrometer data, submitted to Journal of Geophysical Research, 2007) giving additional information regarding the physical significance of breakpoints. Briefly, the more common form of high temporal resolution rainfall data might, for example, be 5 min totals with the periods starting at the hour and extending to 5 min past, then from 5 min past to 10 min past etc. Clearly such data, with an arbitrary time base, have no physical basis whereas breakpoints are based on the observation that, in near continuous time, an accumulation v. time plot of rainfall consists of a series of straight line segments of variable length and slope. Furthermore, these segments will start at arbitrary times and not, as in the example, at a particular 5 min division of the hour. Thus rainfall progresses as a series of periods of random duration each with a steady rain rate throughout and these durations and rates are recorded as the breakpoint data. So no arbitrary choice of time base is made. It is variable but is physically based as a change of rain rate occurs when a change takes place in the drop size distribution (DSD) of the raindrops [Marshall and Palmer, 1948; Torres et al., 1994; Sansom et al., submitted manuscript, 2007]. Breakpoint data sets, therefore, consist of the rates (or amounts) and durations of constant rain rate with the durations of dry periods having a zero rain rate.

[7] A recent source of breakpoint data is the drip style gauge [Sansom and Gray, 2002] which forms the caught rain into approximately equisized drips that are then simply counted, and so it is similar to a tipping bucket in that the amount is discretized, but each drip is much smaller than the typical bucket size. A further source developed by Sansom et al. [2001] uses an interpolation procedure on radar imagery to produce estimates of rain rate every 15 s from 15 min images; a breakpoint series can then be extracted for any pixel using the same process as developed for the drip gauges [see Sansom and Gray, 2002, section 5a]. However, the breakpoint rainfall data used in this paper were extracted from the daily pluviographs of Dines tilting siphon automatic rain gauges by manual digitization. The locations of the 88 gauges used are given in Figure 1. Data from a further four sites were also used but were discarded because of probable imperfections in the manual digitization of the pluviographs or problems in the subsequent fitting to the model. For most of the locations, the 5 year period from January 1986 to December 1990 was used but a few had some earlier or later years and just one or two had no overlap with the 1986–1990 period although all fell within 1982–1996.

3. Brief Review of the HSMM

[8] A hidden semi-Markov model (HSMM) was used to model the breakpoint data and Sansom [1999] and Sansom and Thomson [2001] provide in-depth descriptions of the model, its particular suitability for breakpoints and its physical interpretation. Briefly, the states of the HSMM align with the different PGMs that occur in the atmosphere such as those for convective precipitation or for synoptic-scale frontal precipitation. The HSMM is embedded in a hierarchy of timescales which is shown in Figure 2. Precipitation events and interevent dry periods comprise the longest scale at the top then, within each precipitation event, rain and shower PGM episodes comprise the medium scale, while at a fine scale are the periods of steady rain, or brief dry periods within the episodes, that are captured by the breakpoint data.

Figure 2.

Hierarchical division of time into large-scale precipitation events and dry interevents at the top, then events into rain or shower episodes, and finally the episodes into individual wet and dry breakpoint durations. The “d” and “w” indicate dry and wet, which are known from the breakpoint data, whereas the “R,” “S” and “I” indicate rain or shower episodes and dry interevents which are not known from the breakpoint data.

[9] In Figure 2 one event from the top timeline has been expanded on the second timeline into a series of four PGMs which have been labeled as either “Rain Episode” or “Shower Episode.” The names for the episodes are just illustrative as it was found that four types of wets and three types of drys were required [Sansom and Thompson, 2003]. The breakdown of the PGMs into DSDs is shown on the third timeline where “w” and “d” indicate wets and drys respectively and each “w” or “d” represents one breakpoint. The duration of the breakpoints is shown in Figure 2 as a section of the timeline and a rain rate is associated with each duration, the rate being zero for those labeled with a “d.” From the timelines a sequence of labels can be generated which describes the state of the system at any time. For example, at the bottom of Figure 2 and starting from the left, at the end of an interevent dry period (“I”) a period of rain starts (“Rw”) which has a change of rate (another “Rw” is shown) before a short dry break occurs (“Rd”); eventually rain turns to showers with a dry break (“Sd”) until a shower actually occurs (“Sw”) and so on.

[10] Note that, since only the “w” or “d” can be definitely assigned, the states are not directly or completely observed (i.e., they are hidden) and must be inferred from the breakpoints. Within the hidden states, the breakpoints are modeled as bivariate (for the wet data) and univariate (for the dry data) mixtures of lognormal distributions. The states follow a semi-Markov process in which the probability of a transition from one state to another depends only on the initial and final states of the transition. These probabilities form the transition matrix in which semi-Markov, in contrast to Markov, implies self-transitions are not allowed and the length of time within a state, or its persistence, is handled by a dwell time distribution. Dwell time is simply the number of breakpoints that occur before there is a change to another state, such as a change from showers to rain, or from rainfall to an interevent dry. Further details of the HSMM are given by Sansom and Thomson [2000, 2001, 2007]. The parameters of the dwell distributions and the probabilities in the transition matrix are estimated in the model-fitting process together with the parameters of the state distributions.

[11] As mentioned above to adequately represent the breakpoint data by the HSMM, four types of wets and three types of drys were required [Sansom and Thompson, 2003]. Sansom and Thompson [2003, Figures 3 and 5] show a typical example of a breakpoint data set and its fit to a HSMM; both data set and model resemble all those of the 88 sites used in this paper. The seven states of the HSMM will be referred to by RD, SD, I/M for the drys and RL, RH, SL, SH for the wets. The dry ones are, respectively, the dry breaks during rain and shower episodes and the longer dry intervals between events (i.e., I for interevent) but sometimes a site might not detect an event and two I states might combine (i.e., an M, for multiple-interevent, occurs). These I and M states cannot be distinguished in the data and are modeled together as a mixture. The wet states are either rain (R, the longer-duration breakpoints) or showers (S, the shorter-duration ones) and the subscripts L and H distinguish between ones with light rain rates and those with heavy rain rates.

4. Fitting the HSMM to Breakpoint Data

[12] Fitting breakpoint data to HSMMs was done regionally with New Zealand subdivided into 7 overlapping regions (see Figure 1), and each of the 88 sites was assigned to up to three regions. Over the North Island four regions were defined (northern, western, eastern and southern North Island, i.e., NNI, WNI, ENI and SNI respectively) while over the South Island three regions were defined (northern, central and southern South Island, i.e., NSI, CSI and SSI respectively). All the South Island sites were assigned to either NSI or SSI so CSI was redundant but it was found to be useful in identifying and aligning the rainfall states.

4.1. Model Fitting

[13] Sansom [1999] examined models with different numbers of states ranging from 4 to 11 and selected a 9 state model which differed from the Sansom and Thompson [2003] 7 state model by having subdivisions of the SD state and an extra wet state labeled E. The latter was directly attributable to the very short duration breakpoints arising from imperfections in the manual digitization process. However, Sansom and Thompson [2003] found that by censoring out these imperfections, which accounted from less than 0.01% of the breakpoints at most locations, a much better (according to the Bayesian Information Criteria (BIC) [Schwarz, 1978]) and simplified HSMM resulted. Similarly, the model was improved by not subdividing SD but allowing the I state to become the mixture I/M instead. Furthermore, the need for the dwells to be dealt with explicitly within an HSMM rather than an HMM (hidden Markov model) was established by Sansom [1999] and further improved by Sansom and Thompson [2003] in which the dwell distributions were found to be best parameterized as modified geometrics where the probability of a dwell of 1 is a free parameter and there is a geometric tail for longer dwells.

[14] This paper adopts the Sansom and Thompson [2003] model and only a brief summary of the fitting procedures for the HSMM model is given here but more details are given in Appendix A. The fitting is an iterative maximum likelihood procedure which requires sets of initial values for the three parameter sets (i.e., the state distributions, the transition matrix and the dwell distribution). Following the strategy of Rabiner [1989], uniform probabilities were used for the transitions and dwell times but the state distribution parameters were initialized many times by random selections, and the fit with the greatest likelihood was taken as the global maximum. The random selections were restricted to the range of feasible values for the location parameters, while the dispersion and correlation parameters were restricted to be the same order as that of the data. This pragmatic approach did not guarantee a global maximum but, the large number of fits to the data provided confidence that the global maximum was achieved (step 1 of Appendix A).

4.2. Regional Models

[15] Fitting the sites individually within each of the regions, showed there was much similarity over the sites in each region. The similarity was in the relative locations and scales of all seven states and in the model structures, i.e., the interconnections between the states and their strengths, or frequency of occurrence. As in the work by Sansom and Thompson [2003], this encouraged the search for common regional (and, later, national) structures that were to be drawn from the HSMM fits and then imposed upon them.

[16] To ensure fits from station to station were as compatible and similar in structure as possible, first all stations within each region were aligned such that the state labels (RD, SD, I/M, RL, RH, SL, SH) were consistent from site to site (step 2). This is an essential and necessary condition if the spatial variability of the HSMM parameters is to be assessed and interpreted in a physically meaningful way. Next, and still only within one region, each site was refitted using for initializations the current fit (i.e., the estimated values of the model parameter values) from itself and the current fits from all other stations of the region. The fit with the highest likelihood at each site was taken to be the best fit and the site whose prior fit had given rise to this best fit was also noted then refitting was repeated until most of the stations were best fitted from themselves (step 3). Then the models of the region were formed into clusters to identify the most common HSMM structures using k-mean clustering [Hartigan and Wong, 1979]. Two significant clusters could always be identified and a significance test was performed to find whether a third cluster needed to be considered but two were always found to be sufficient. An HSMM was built from the cluster with the most members and each site was refitted using that HSMM for the initialization (step 4). Generally, one cluster dominated with the differences between the clusters not being large. In the SNI region, for example, 20 of the 29 sites formed the largest cluster and the difference between the clusters arose mainly in the mean durations of the dry states.

[17] Apart from the inherent property of HSMMs that self-transitions are forbidden, no restrictions had been placed on which transitions could occur, so step 4 of the procedure for imposing and enhancing commonality within a region was to set to zero those transition probabilities that for all the sites within a region were close to zero (i.e., less than 0.05). With this new structure for the transition matrix and two other candidate structures (see step 5 of Appendix A), each site was refitted and the process repeated until at the final step 5 there were no further small probabilities that could be set to zero. Using representative sites, the final structures for the seven regions are shown in Figure 3. The states are placed at their mean duration and rate with the dry ones shown as large solid dots along a time line indicated by the upper axis and the wet ones as large open dots in the time–rain rate plane indicated by the lower axes. Transitions are shown by lines with arrows indicating the direction of transition and most, but not all, such lines have arrows pointing each way. The heaviness of the lines indicates the strength of the transitions with the heavy solid lines representing those transitions each covering at least 4% of the total number of transitions, lighter lines for those covering 2–4%, dashed for 1–2%, and dotted for under 1%. At each wet state's location is shown the mean number of breakpoints that occurs before a shift to another state takes place; for the drys this is always just one before a transition to a wet state takes place.

Figure 3.

The HSMM structures for the regions of Figure 1 in terms of which transitions between states are allowed and the relative frequencies of the transitions. The states are positioned either in the rate/duration plane for the wets or along a duration axis for the drys at their distribution's location parameter. The mean dwell (i.e., number of breakpoints during which no change of state takes place) in the wets states is shown by the number in the circles at the states' locations: a state change always follows a dry breakpoint. Also, in the bottom right diagram, the structure of the national HSMM is shown schematically with the states labeled as they are referred to in the text.

4.3. A National HSMM for New Zealand

[18] For each region, an HSMM with the same structure for each site had been imposed through the refitting and clustering process and this was endorsed by the intraregional similarity of the fitted models prior to disallowing any transitions. The next step was to form a common structure for New Zealand as a whole using those of Figure 3 as a guide. Figure 3 shows that the number of retained interconnections varies from 14 to 19 and of these 11 are common to all areas, another two are each common to all but a single area and, only one was unique to a single area. Also, there are a few features that are common to either the North or the South Island; that is, only in the North Island is there a return path from SH to I/M while the SL to SD and return, and that from SL to SH only appear in the South Island.

[19] Thus, for each island an HSMM structure was formed that comprised the most prevalent interconnections between the seven states of the constituent regions with the less common ones omitted. The bottom right diagram of Figure 3 shows a schematic of the resultant structures where the 13 lighter arrowheads were common to both island, the rightmost darker one was particular to the North Island and the other darker ones only occurred in the South Island HSMM structures. It can be seen from Figure 3 that there were potentially two candidate structures for a New Zealand HSMM; one containing a superset of 18 state transitions formed by combining both the island structures, and the other being the minimal set of the 13 state transitions that were common to both.

[20] As well as the supersets and minimal sets of both islands, the North Island pattern of interconnections was applied to South Island data sets and the South Island pattern to North Island data sets. Thus at any particular site three state transition structures were imposed on the HSMM. After refitting each station to each of the three structures, 84 out of the 88 locations had their best fit (i.e., lowest BICs) with the superset of transitions. The four sites where the superset was not favored were in the northern half of the North Island where the best model was found to be that with a South Island transition structure. This is closer to the superset than to the North Island structure so the superset structure shown in the bottom right diagram of Figure 3 was adopted as the national HSMM. Also, a further exercise in determining a national HSMM made through combining the seven regions into three (i.e., northern, NNI/WNI/ENI; central, SNI/NSI; and southern, CSI/SSI) rather than two (i.e., North Island and South Island) had the same result.

[21] The depiction of the national structure is schematic but retains the essentials of the regional fits and, as noted by Sansom and Thompson [2003], the states were located in physically understandable ways. An exception is that, as shown in Figure 3, although somewhat exaggerated since they were often nearly coincident, the mean duration of SD was occasionally smaller than that of RD but the larger variability of SD still ensured that, as might be expected, the longest intraevent dry breaks arose from SD rather than RD; that is, they occur within showery episodes rather than periods of rain. Regarding the transitions between the states, not all of the ones that were possible occurred but those that did were also physically understandable. For seven states where self-transitions for the four wet states and all dry-dry transitions between the three dry states are excluded, there are 36 possible transitions but only 18 of these needed to be retained. Furthermore, the retained transitions show two distinct groupings: one grouping appears to represent the convective shower process (the S states), and the other representing the precipitation processes associated with frontal systems (the R states). Apart from the expected connections through I/M, which as the null-PGM is in neither group but does provide a connection between them, the two groups were largely independent except for the interchange between RH and SH but this was generally weak and only found to be both way over the South Island. This linkage did not fit with anecdotal expectations and raised the question: is it necessary to impose physical structure on the HSMM rather than treating it as purely a statistical model based on the lowest BIC? On the other hand, if, as was done, adopting the structure highlighted by the BIC led to valid consequences then the “anecdotal expectations” were poor and some new understanding is reached.

5. Applying the Thin-Plate Spline

[22] Sansom and Thompson [2003] showed that that the spatial variability of rainfall can be captured by mapping the HSMM parameters from a network of sites within a region. For this paper the national structure was applied to all of the sites of Figure 1 so that the spatial variability of the model parameters over the whole of New Zealand could be assessed, and interpreted in a physically meaningful way.

[23] The spatial variation of the HSMM parameters can be estimated at all locations from a few independent variables that include geographic location and height, using a thin-plate smoothing spline technique developed by Hutchinson [1995]. The spline is a surface that fits spatially distributed data with some small error assumed at each data point, so that the surface can be smoother than if the data were fitted exactly. A single parameter controls the smoothing and is often chosen by the method of generalized cross validation (GCV) in which the effect of omitting each site in turn is examined. However, in many climatological data sets, which often have few data and are noisy, using the GCV can result in unrealistically smooth maps with unacceptably large differences between the data and the spline fit. To address this, Zheng and Basher [1995] manipulated the signal and error characteristics of data and spline fit and found that enforcing a global value for the signal to error ratio (a quality measure available from the spline fitting routine) provided a useful and intuitive procedure for understanding and controlling the fitting. The larger the signal, in relation to the error, the closer the fitted surface passes through the data and the smaller the error of the fitted values.

[24] The method developed and implemented by Sansom and Thompson [2003] was based on such considerations and was used to determine the spatial variation of the parameters of the HSMM model across New Zealand. The independent variables in the spline fitting were latitude and longitude with elevation used as a covariate; that is, the parameters being fitted were given global rather than local dependencies on elevation. The elevation was smoothed by being replaced at each site by its mean value within a 15 km radius of the site but the degree of smoothing was found to have little overall influence as a range of elevation smoothing was applied with little change in the result. However, the inclusion of elevation as a covariate was necessary as will be described in the next section. The fitting of the HSMM parameters to thin-plate splines was a multistep process as follows: (1) split the parameters into two logical groups: the static parameters, which define the distributions of durations and rates, and the dynamic parameters, which comprise those of the dwell distributions and the probabilities of the transition matrix; (2) fit spline surfaces to the members of the dynamic group using the GCV criterion, obtain spline estimates at the station locations, and scale the transition probabilities from a state to enforce the constraint that they must always sum to one; (3) refit the HSMM model at each station using the spline estimates of the dynamic parameters as fixed constants, rather than as parameters to be estimated, and so reestimate the static group of parameters; and (4) fit spline surfaces to the refitted model using the largest possible signal to error ratio, such that the fit at the station locations is nearly exact.

[25] Hutchinson and Gessler [1994] cautions using signal to error ratios larger than unity in fitting spline surfaces, as was implemented in step 4, since the spatial patterns could lose robustness and become sensitive to extra data. However, the scheme above allowed the dynamic parameters the freedom to have the smoothing of their spatial variations optimized using the full potential of the spline's ability to differentiate actual spatial variation from noise. Unfortunately, to achieve such freedom, that for the static group had to be sacrificed in step 4 to ensure consistency through the parameters and maintain their values close to the most likely values.

[26] The final structure of the national HSMM had 48 parameters but 54 different surfaces were produced through the application of the thin-plate spline. The difference arises from: the RD state only having the exit to RL so the probability of that transition is unity and has no spatial variability; and, as mentioned above in step 2, transition probabilities from a state must always sum to one so the number of parameters reduces by 1 for each state of the model. Some of these 54 surfaces will be shown below but before that a verification of the model is presented in the next section.

6. Verification of HSMM and Spline Fits

[27] An initial verification was made by comparing some annual statistics extracted from 100 years of synthetic data generated from the HSMM fits at the 88 breakpoint data sites with actual observations at those sites. The historical series covers the period 1 January 1945 to 31 December 2002 and were extracted from a 151 station daily rainfall data set used by Thompson [2006]; 88 of these were also breakpoint sites. The statistics generated were: the mean annual total rainfall; the mean annual maximum rainfall in 1 d; the mean annual number of wet days (i.e., days with at least 1 mm of rain); and, the mean annual maximum dry spell (i.e., run of days with less than 1 mm of rain). Only annual statistics were extracted as the HSMM is not a seasonal model but, as it models the average structure over the period of record, it should yield annual statistics. The same statistics were also generated from the daily rainfalls and the two sets of statistics were compared at each site through a standardized score defined by:

equation image

where equation imageH is the median of the historical series, equation imageS is the median of the synthesized series, and MADp is the pooled mean absolute deviation, i.e., the deviations of the synthetic data from their median at the site together with those for the historic data from their median.

[28] Figure 4 shows histograms of Ds which, ideally, should be zero for all sites and rainfall statistics but their distributions are away from zero such that the synthetic data gave too little annual rainfall with too many dry days, too few wet days and the maximum 1-d rainfalls were too small. Thus the HSMM has a dry bias which arises because it is not constrained to follow a seasonal pattern so simulations do not contain the persistence that may exist at seasonal timescales. For example, the concentration of heavy rain may be curtailed at times when it should be more persistent and so bias 1-d maxima in the simulations to smaller values than actually occur. Similarly, without the propensity for a “wet season” to occur, a dry period may become extended beyond what could naturally occur giving overlong dry spells and diminished annual totals.

Figure 4.

Histograms of the site-by-site differences between the median of the synthetic annual statistic and the median of the historic ones divided by their pooled mean absolute deviation. At the top of each panel the statistic concerned is indicated with its bias (i.e., the mean of the scaled differences).

[29] Including seasonality in the HSMM is a nontrivial task requiring, as a first step, a physical investigation to determine where seasonality is expressed within the rainfall process so that the model can be enhanced in a physically meaningful way. Then, as a second step, the development, implementation and verification of the algorithms for fitting the seasonal HSMM is required. The first step is described by Sansom and Thomson [2007] and work is underway on the second. The analysis below demonstrates that a fully seasonalized model would not have the model biases of the current HSMM by estimating from the historic record the annual rainfall statistics expected if there were no seasonality in rainfall. Thus, rather than introducing seasonality into the HSMM, the historic record was deseasonalized.

[30] Deseasonalized statistics were estimated from the historic records by finding at each station and for each statistic the month that is most similar to the year as a whole because that is what the HSMM synthesizes. Thus, for each station the months that most closely average a twelfth of the mean annual total and number of wet days were identified, and those months that most frequently yield the maximum 1-d fall and dry spell were also identified. The Ds were then recalculated by using those typical months rather than the full year with the total and wet days actual values multiplied by 12, and each synthetic year for the maximum 1-d fall and dry spell was reduced to 31 d to be equivalent to that of the deseasonalized maximum 1-d fall and dry spell. Figure 5 shows histograms for these deseasonalized Ds which still had nonzero values at some sites but the distributions were centered closer to zero and the overall biases were much smaller and not significant.

Figure 5.

Similar to Figure 4 but based on the deseasonalized historic data in which only the month that most resembles the year overall is retained. For the annual total and number of wet days, the values for the historic months are multiplied by 12, and for the other statistics the synthetic year is reduced to 31 d.

[31] The mean values of these deseasonalized Ds can also be used to assess the need for orography in the spatial model, i.e., should elevation be included as a covariate in the spline fitting. Table 1 shows the variation of the mean Ds over a number of scenarios: the first and second rows of data show the change that occurs when orography is dropped completely; the third and fourth rows show the effect of applying orography to either the static or the dynamic group of parameters; and the fifth row shows the result of only using orography with the 22 parameters that showed a strong relationship with elevation. These were: the static parameters for states SD, RD and I/M, the mean rain rates for the wet states and the dynamic parameters for SH and I/M. However, not all the smallest mean Ds occurred for this selection of parameters with orography and no obvious pattern exists in this selection. The necessity for including orography in some way is underlined by all the mean Ds being of largest magnitude when orography is excluded and, although none of the smallest mean Ds occur when orography is applied throughout, only one is least in either of the divisions between static and dynamic parameters. Thus, the simplest and most effective strategy is to apply orography to all parameters.

Table 1. Mean Values of Ds Over the 88 Breakpoint Sites for the Specified Annual Statistics and Scenariosa
StaticDynamicTotal RainWet DaysDry SpellsMax 1-d
  • a

    That is, parameter groups to which orography is applied. The grouping for the fifth row of data is given in the text.

No orographyno orography−0.425−0.5000.908−0.314
Orographyno orography−0.294−0.3840.756−0.181
No orographyorography−0.275−0.3930.872−0.247
Orography (22)no orography (32)−0.342−0.2560.753−0.264

[32] The verification above concerns only the observed and synthesized values at the breakpoint sites but the spline fitting enables the estimation of the HSMM parameters and, hence, the synthesizing of data at arbitrary points. Such points could be chosen at the many (currently about 600) daily rainfall stations that are spread across New Zealand and the actual and HSMM rainfall climates could be compared at those points. However, the data from the daily stations have been used [Tait et al., 2006] to produce nationwide estimates of daily rainfall on a day-by-day basis since 1960. The same thin-plate smoothing spline was used to make estimates at over 11000 points on a 0.05° latitude/longitude grid and a manual analysis of the mean annual rainfall rather than the orography was used as an independent covariate. The mean annual rainfall derived from their 46 years of daily estimates is shown in Figure 6a while the mean annual rainfall derived by synthesizing 50 years of breakpoints from the HSMM parameters estimated at 2868 points on a 0.1° latitude/longitude grid is shown in Figure 6b.

Figure 6.

Comparison of (a) the historical mean annual rainfall total with (b) that synthesized using the HSMM. (c) The absolute difference between the historical and synthesized is shown as a percentage, and (d) the difference has been standardized at each point by dividing by the pooled mean absolute deviation at that point. The legend labels are for the midpoint of the particular range, so for linear scales the boundaries are the arithmetic mean but the harmonic mean for log scales.

[33] The spatial variations shown in Figures 6a and 6b are similar apart from the synthetic one clearly being (as might be expected from the site-by-site comparisons above) somewhat drier than the historical but this degree of similarity was only achieved by introducing “bogus” stations. Before their introduction, for the South Island, although the pattern was similar to that shown in Figure 6b the values in the Southern Alps, which run down most of the western side of the island, were far too low. More disturbingly almost the entire North Island had annual rainfalls of no more than 1000 mm with, in particular, the Tararua Ranges of the North Island, near 41°S, being underestimated by up to 6000 mm whereas the regional study of Sansom and Thompson [2003] using a similar technique had successfully modeled that area. That regional study had used a slightly different model structure and restricting the spline analysis of this study's model to that area still produced reasonable results. However, extending the area to include those parts of the South Island north of 42°S immediately degraded the results; that is, the new national model was valid for a small enough region but not when extended into (as can be seen from Figure 1) a relatively data sparse area. On such extensions the spline needs more guidance and, since no more data were available, the approach adopted was that of Zheng and Basher [1995] in which values were inserted at places where experience suggested values could be well estimated. Thus, the data from the station high up on Mount Taranaki at about 39°S 174°E was repeated at a point with the same elevation in the northern Tararua Ranges, where the annual rainfall is known to be similar and its general exposure to rain bearing airstreams is similar. The result was that the mean annual rainfall in central New Zealand between 40°S and 42°S became well estimated by introducing this one bogus station.

[34] The process of reaching the result shown in Figure 6b was to extend from the area already successfully dealt with until some significant deficiencies were observed in the mean annual rainfall field. Then to correct the most significant deficiency, a single bogus station was added at the position of the deficiency with the selection being guided by prior knowledge regarding the actual mean annual rainfall at that position, i.e., the Tait et al. [2006] estimate, which is taken to be the best one available. Also the exposure and elevation of that position was considered such that an existing station with a rainfall climate similar to that expected at the point of deficiency was selected for that point. After ensuring the addition of the bogus station was successful, the area of coverage was further extended such that after the addition of 14 bogus stations Figure 6b resulted. The six bogus stations in the South Island near the Southern Alps were added together as a single step rather than one by one as the common problem in that area was to persuade the model to replicate the enormous gradients in the mean rainfall field. The spline had difficulty in maintaining both the observed orographic enhancement in the west and degree of rain shadow in the east because, not only is there a lack of data in the area, but also there was little choice in what bogus data might be used: suitable observations were just not available. Thus, although it can be seen from Figure 6c that the absolute percentage difference between the historical and synthetic mean annual rainfall fields in the North Island and eastern seaboard of the South Island was often below 10% and mainly below 20%, in the Southern Alps and the area immediately to the east, differences were often about 50% and occasionally over 100%. Another problem area was the western seaboard of the North Island between about 38°S and 40°S where the spline may well have attributed the high rainfalls on Mount Taranaki to its westerly position rather than to its height while just to the north the spline had, perhaps, reversed the situation and not been able to replicate the levels of coastal rainfall. Thus, some spatially systematic differences were found and have been expressed as the standard score, Ds, in Figure 6d where the dryness in and west of the Southern Alps and along parts of the western coast of the North Island can be seen. The systematic excess rain to the east of the Southern Alps and near Mount Taranaki can also be seen but in other regions the spatial distribution of Ds is not significantly systematic.

[35] The station-by-station verification was also made through consideration of three other annual statistics and, while the selection of bogus stations was only made through the mean annual rainfall, the others were also viewed in a manner similar to that of Figure 6. The worst performance of the HSMM was for the replication of the mean annual maximum dry spell where, in general, the estimate was twice the actual (i.e., the Tait et al. [2006] estimates). The other two gave better results: for the number of wet days the difference between the HSMM and observations (i.e., the Tait et al. [2006] estimations) was about 10–20%; and, for the maximum 1-d rainfall the differences were about 20–30% with the same spatially systematic differences seen for the mean annual rainfall. Histograms for the Ds scores for all four annual statistics are given in Figure 7 which shows that the bias for each statistic was similar to the corresponding one in Figure 4. Thus the estimations at the grid points were much the same as those at the stations and this was despite the introduction of bogus stations which suggests that their introduction did not force the result too far toward a reality that the HSMM cannot replicate. Furthermore, the results presented below in Figures 8a, 8b, 9, 10, 1111 , and 12 do not show any anomalous behavior at the positions where the bogus stations were introduced.

Figure 7.

Similar to both Figures 4 and 5 but for the 2868 points of the 0.1° latitude/longitude grid at which rainfall was synthesized.

Figure 8a.

Spatial variations of the frequency of occurrence, mean duration and mean rain rate of the rain wet states of the HSMM.

Figure 8b.

Spatial variations of the frequency of occurrence, mean duration and mean rain rate of the shower wet states of the HSMM.

Figure 9.

Spatial variations of the frequency of occurrence and mean duration of the dry states of the HSMM.

Figure 10.

Spatial variations of (a) the frequency of occurrence of rain episodes (shower episode frequency is just 100 – rain episode frequency) and the mean durations of (b) rain and (c) shower episodes in terms of the number of breakpoints.

Figure 11.

Episode statistics: their mean durations, accumulations and fraction of wet time for (left) rain and (right) showers.

Figure 12.

(a and b) The transition probabilities that generate multiple episode events. Event statistics: (c) fraction of events that involve more than one episode, (d) the mean number of R/S episode pairs, (e) fraction of events that are a single shower episode, and (f) the fraction that start with an R episode.

7. Results and Discussion

[36] Sansom and Thomson [2007] showed that the probabilities of transitions between states and the dwell times within states (i.e., the dynamics of the model) vary from season to season and that the HSMM's bias arises from not incorporating this nonhomogeneity into the model. In contrast, the statics of the HSMM (i.e., the state distributions of the durations and the rates of the breakpoints) were not seen to be seasonal. Thus, in terms of the hierarchy of Figure 2, the lack of seasonality will have least effect at the lowest level where the dynamics only control the frequency with which each state occurs and, in Figures 8 and 9, an unbiased view can be presented for each of the HSMM states of the spatial variation of the mean breakpoint durations and rates. Also as the HSMM was shown to be unbiased for deseasonalized data it provides valid insights into the large-scale spatial and long-term temporal variations of rainfall at the episode and event levels of the hierarchy (Figures 10, 11 and 12, respectively).

[37] Figure 8 shows the spatial variations for each of the wet states of, in the top row, the fraction of the overall number of breakpoints with which the state occurs, its mean breakpoint duration in the middle row and its mean breakpoint rate in the bottom row. The most frequent states, although not the ones with the greatest overall durations, are RL and SH which contribute over 20% of the breakpoints in most places and over 30% in the east of the North Island for RL and in western areas for SH. Such frequency is also achieved by SL in southeastern areas of the South Island and, to a lesser extent, by RH in western areas of the South Island. Over all states the patterns of variation of the mean durations are quite similar but, in Figure 8a, the scale for the R states is twice that of the S states. The pattern shows a slight tendency, except for SL, for longer mean durations in eastern areas whereas the tendency for mean rates is for them to be heavier in western areas. This is particularly so for the H states but much less so for the L states which show little spatial variation.

[38] Figure 9 shows the spatial variations for each of the dry states of, in the left column, the fraction of the overall number of breakpoints with which the state occurs and its mean breakpoint duration in the right column. The frequency of the I/M state has much the same range as the frequencies of the wet states but those for RD and SD are much lower with the former at only 1% in the southwest of the South Island and never more than 5% elsewhere. The spatial variation of SD is stronger and, although around 5% in many areas, rises to over 10% in the south of the South Island and the northwest of the North Island. In those same areas the mean durations of SD are longest whereas, apart from the probably spurious maximum in the southwest of the North Island, the pattern for RD shows little spatial variation. Also for I/M little variation occurs over the North Island whereas the strong rain shadow affect of the Southern Alps can be seen in the rapid change from the shortest I/M durations in the west of the South Island to the longest ones in the east.

[39] The middle level of Figure 2 concerns the progression of PGMs and illustrates that a particular episode continues while the succession of breakpoints is of one type, either R or S. In Figure 2 the only distinction within each type is that some are wet and others dry but at the bottom right of Figure 3 it is shown that while the states of the national model also form two groups, the R and S groups, within each group the types L, H and D exist; that is, the wet ones are subdivided. Figures 10 and 11 show the spatial variations of some statistics associated with the episode level of the rainfall hierarchy.

[40] Data sets can be synthesized from the HSMM using the spatial variations of its parameters (some of these are indicated in Figures 8 and 9) as determined by the thin-plate smoothing spline. Such data are complete as not only do the generated breakpoints have a duration and rate but they also have a state label. Figure 10 was derived from the state labels of synthetic data sets by discarding the I/M states and the subscripts of the R and S states. Thus in Figure 10a the number of R groups is shown as a percentage of the total number of R and S groups while in Figures 10b and 10c the mean number per episode in each type of group is shown. Figure 10a shows that most episodes are of the shower type but that, for most of the North Island except the far North, about 40% are of the rain type. The least frequency of rain episodes is in the Southern Alps but, in terms of breakpoint numbers, the episodes are longest in that area. Shower episodes are generally much shorter than rain ones but they, too, are longest in the Southern Alps. Figure 11 shows that showers episodes are also generally shorter than rain with accumulations about 25% that of rain (the rain scale is four times that for showers in Figure 11 (middle)). Except for the far southeast, the longest rain episodes occur in the South Island and the highest accumulations from rain occur in the west and also in the northeast of the North Island. Dry breaks occur during episodes and Figure 11 (bottom) shows that these are longest for showers especially in the south of the South Island and the northwest of the North Island whereas, in most areas, rainfall persists for at least 70% of the time during rain episodes.

[41] The example event shown in Figure 2 consists of two Rain/Shower episode pairs and the transition structure shown at the bottom right of Figure 3 confirms that such a situation is possible because the R and S groups are connected by the two-way link between RH and SH. If this link did not exist then within an event there could be no change of episode type; each event would either by a single rain episode or a single shower episode. At the end of section 4, the physical validity of this link was questioned not because a link between the R and S groups was thought unlikely (indeed, some link would be expected anecdotally) but because a link between the L states seemed more likely; that is, frontal rain eases off to light showers preceding heavier ones. However, of the four possible links between the wet R and S states only that between the H states was chosen by the HSMM fitting procedure and, although generally weak at the stations, Figures 12a and 12b show that areas exist where frequent transitions through this link would occur. The spatial variation of the link's transition probabilities are shown in Figures 12a and 12b to be low in the southeast of both islands, especially that for SH to RH, and small over most of the North Island but quite significant about the Southern Alps. The degree of significance can be seen in Figures 12c and 12d where the first shows the fraction of events that are at least one R/S pair of episodes and the second shows the mean number per event of such pairs. Clearly, the higher probabilities ensure that 10–15% of all events in the west of the South Island are at least one R/S pair and that over all such events the mean number of R/S pairs is 1.5–2 in that area. Elsewhere, the fraction of multiple episode events is much less and Figure 12e shows that most single episode events are a single shower episode. A slightly different view of this is given in Figure 12f where the spatial variation of the fraction of events that start with a rain episode is given; this is highest in the North Island and lowest over the Southern Alps. The mean event durations, amounts and fraction of wet time could be mapped as it was for both rain and shower episodes in Figure 11 but the patterns are very similar to that for rain for durations and amounts although the values are less because the mixture of rain and showers dilutes them. The influence of the showers in the events also makes the event wet time fraction similar to that for showers.

8. Summary and Conclusions

[42] The verification against long-term annual statistics showed that the HSMM had a bias toward a drier than actual climate but the comparison with deseasonalized statistics showed that there were times of the year when the HSMM would be unbiased. Thus the results summarized and conclusion drawn below are principally for those times but they are indicative for the remainder of the year where slightly more persistence would be likely. So the statistics mapped in Figures 10b, 10c, 11, 12c and 12d are underestimates in some areas while other parts of Figures 10 and 12 and Figures 8 and 9 are probably unbiased. These considerations have been allowed for below.

[43] The primary state labels R and S were chosen on the basis of the duration of the breakpoints so the states covering the wet breakpoints with longer durations were taken to be representing rain while the shorter ones represent showers. No direct effort was made to discriminate data between periods during which the PGM was independently identified as either rain (frontal) or showers (convective). Thus the R and S labels are not firmly tied to the type of PGM but they can always be interpreted as representing, respectively, either the more or the less persistent precipitation. The secondary labels were straightforward extensions with H and L simply referring to the wet states covering the breakpoints with higher or lower rain rates respectively while D refers to the dry states that clearly connected with either R or S. These simple choices gave consequences that were consistent with anecdotal expectations; for example, dry breaks are less frequent during rain than during showers. However, beyond these consistencies that underpin the alignment of the R/S labels with the R/S PGMs, these are conclusions particular to this study.

[44] Regarding temporal variation, in the long-term 80% of all events consist of a single shower, or occasionally two or three showers spread over 1 or 2 h, yielding about 1 mm of rain. Remaining events are either a single period of rain or a few interchanges between rainy and showery periods. The latter have much the same character as those that occur singly while, in the long term, the rainy periods have 5–10 changes in intensity, often last 4–5 h with rain falling for at least 80% of the time, and yield 5–15 mm of rain. The interchange from rain to showers is more frequent than the reverse and both occur between the H states; that is, heavy rain can be followed by an intense shower but not a light shower, or, the longer breakpoints with high rates can be followed by shorter ones with high rates but not by ones with low rates.

[45] This temporal behavior has spatial variation. The Southern Alps (the high range that can be seen in Figure 1 in the west of the South Island) experience the highest frequency of interchanges between rain and showers, and back to rain. It is also the area with the highest frequency of events starting with a shower and of changes of intensity during rain or showers. The west of the South Island in general has the highest mean accumulation during and the highest mean duration of events while the lowest occurs in the southeast of the South Island where, together with the southeast of the North Island, the lowest frequency of R/S interchanges occur. However, the driest parts of New Zealand are in central parts of the south of the South Island (see Figure 6a) to the north of where the lowest mean accumulations occur but coincident with where the longest gaps between events occurs (see Figure 9, bottom right).

[46] Only a general overview of the temporal and spatial patterns in New Zealand has been given here. However, the synthetic rainfall data sets could be analyzed to provide many more details particularly with regard to variability so, while Figures 10 and 11 show mean conditions, maps of various percentiles could also be produced. Indeed, although it is biased toward a drier climate, the HSMM does seem to reproduce the correct variability [Sansom and Renwick, 2007]. The problem of the bias should be dealt with by extensions of the work described by Sansom and Thomson [2007] which are now underway. The problem of replacing the bogus stations and generally increasing the density of the network is also being addressed but is likely to be longer term than developing a seasonal HSMM.

Appendix A

[47] The steps taken to establish a national HSMM are the following:

[48] 1. For each site fit an HSMM in which all transitions are allowed. For initial values use uniform probabilities in the transition matrix and the dwell distributions and random values for the static parameters. From many such initializations, iterate to convergence and adopt the fit with the highest likelihood as what can be termed the “best” fit for this stage.

[49] 2. Over each region compare the fits from all sites with each other to find the set of stations with the greatest number of common features, i.e., in diagrams such as Figure 3, those with similar locations for the states, similar frequencies of transition, and similar dwell times. For sites other than those in this set, permute the states until a combination is found for each site which allows it to be included in the set. The outcome of this stage is the realigned set in which, for example, index-1 for all the sites could be given the label RL.

[50] 3. Again within each region, use each site to initialize fits at all the sites. Assuming a region has N sites, after the fitting there are N fits for each site and of these one is the best and will be used in the next round of this refitting. Also for each site note is taken of which of the N gave the fit with the highest likelihood. Several rounds of such refitting are made until most sites give rise to their own best fit. This refitting process reenforces the realignments made in step 2 above.

[51] 4. Again within each region, use k-mean clustering on the HSMM parameters to group the sites into two sets and also into three sets. Apply significance test of Hartigan and Wong [1979] to ensure the split into two sets is sufficient to highlight whatever difference might exist between the sites. Build an HSMM from the biggest cluster and use as the initialization for fits to all the sites within the region. This step ensures that the fits for those sites within the smallest cluster are not due to a local maximum in the likelihood surface having “captured” the fit and prevented a fit at the global maximum.

[52] 5. Again within each region, generate three different transition matrices from those for the individual sites by finding for each particular transition its median, over the sites, first when all sites are considered, secondly for those sites in the biggest cluster of a k-mean division into two groups, and Last for those in the smallest cluster. In each case, transition probabilities of less than 0.05 are set to zero and the matrix scaled so that all rows sum to unity. If the three have the same structure (i.e., transitions which can take place and have nonzero probabilities are common to all three and so forbidden transitions with zero probabilities are also common) then the final structure for the region is determined. If the three do not have the same structure then for each site fits are made from each of the candidate structures and a comparison made through the BIC such that the structure which gives rise to the most best fits is adopted for another round of this step. This step attempts to prevent the model fitting procedure taking what might be termed “statistical advantage” of the presence of transitions which have no physical foundation. It also reduces the number of model parameters.

[53] 6. To establish a national model, common features of the region structures were determined and, as fully explained in section 4.3, a national structure was selected.


[54] This work was carried out under Foundation for Research, Science and Technology contract C01X0701.