Identification of nonlinearity in rainfall-flow response using data-based mechanistic modeling



[1] Data-based mechanistic (DBM) modeling is an established approach to time series model identification and estimation, which seeks model structures and parameters that are both statistically optimal and consistent with plausible mechanistic interpretations of the system. This paper describes the application of the DBM method to 10 min, relatively high precision, rainfall-flow data, including observations of both surface flow and subsurface flow. For a generally wet winter period, the preferred surface flow model is nonlinear in flow generation and linear in routing, while the preferred subsurface flow model is linear in flow generation and nonlinear in routing. These models have mechanistic interpretations in terms of mass balance, hydrodynamics, and conceptual flow pathways. The four-parameter surface and subsurface flow models explain 91% and 96% of the variance of the corresponding observations. Other plausible models were identified but were less parsimonious or were more reliant on prior perceptions. For a wet summer validation period, the models performed as well as in the calibration period; however, when a long dry spell was included, the performance deteriorated. It is speculated that this is because of complex wetting-drying dynamics and potential nonstationarity of the soil properties that are not sufficiently revealed in the available data. Conceptual models informed by the DBM results matched the DBM model performance for subsurface flow but gave poorer performance for the more complex surface flow responses. It is concluded that the DBM method can identify nonlinearity in both flow generation and routing and provide conceptual insights that can go beyond prior expectations.

1. Introduction

[2] A general challenge in the field of hydrological modeling is model structure identification [Dunn et al., 2008]. There are several practical criteria which may be applied for selecting a model structure, for example, a model which needs no more than the available data and produces the required output; a model with which the modeler is familiar and which has a history of working well; a model which, after estimation of parameter values, seems to perform satisfactorily on the catchment under investigation or on an analog catchment; or a model of suitable complexity so that uncertainty in parameter values is minimized. In practice, for any application, normally only one or perhaps a few readily available model structures are considered without using a rigorous selection process, and consequently, many potentially better models are overlooked. It is commonly noted that ideally, the search for the optimal model structure should be approached more systematically, and the development of identification procedures has received significant attention from hydrological modelers [e.g., Wagener et al., 2003; Young, 2003; Lee et al., 2005; Kuczera et al., 2006; Lin and Beck, 2007; Liu and Gupta, 2007; Son and Sivapalan, 2007; Clark et al., 2008; Fenicia et al., 2008; Bulygina and Gupta, 2009; Reichert and Mieleitner, 2009].

[3] A common model identification procedure is based on developing prior hypotheses about hydrological processes in the catchment under study, conceptualizing these into a numerical model, and testing the performance (usually against measurements of flow from the catchment outlet) and, if performance is unacceptable, then rejecting and renewing the hypothesis, proposing a new model, and so on. This may be called the “hypothetico-deductive” approach [Young, 2003]. A feature of this approach is that the prior range of potentially relevant processes may be wide, leading to comprehensive, highly parameterized, highly uncertain, “physics-based” models [Beven and Freer, 2001]. While this type of model has a role in speculative scenario analysis [e.g., Jackson et al., 2008], it does not easily allow for the identification of the dominant modes of response that are most valuable for developing conceptual understanding and for uncovering behavior that was unexpected a priori.

[4] An alternative procedure for model structure identification is the inductive, data-based mechanistic (DBM) approach, which has developed over many years from its original conception [Young, 1978] to the first use of the name DBM by Young and Lees [1993]. Its initial application to rainfall-flow modeling [e.g., Young, 1974, 1993; Whitehead et al., 1976; Young and Beven, 1994] has led to continued use for this purpose over a number of years [e.g., Young, 2001, 2003; Young et al., 2007; Taylor et al., 2007].

[5] The DBM approach aims to impose minimal prior constraints on the model and to evolve the model through the assimilation and analysis of measured data. By relying as much as possible on information in the measurements, such an ideology minimizes the subjectivity in hypothesis making, and by seeking the simplest possible model structures supported by the data (i.e., parsimony), the approach aims to minimize the risk of accepting wrong, overparameterized models and seeks to isolate the few dominant modes of response that dictate the dynamic behavior of the system [Young and Ratto, 2009, 2011]. Recognizing that any set of hydrological measurements gives an incomplete picture of the physical system, a range of possible response modes may well be omitted with such an approach, which is dependent on the information content in the data. It follows that a data-based model should not be used to extrapolate far beyond the realm of the measurements on which it is based. However, increased confidence in predictions and increased applicability of the model for exploring scenarios may be justified if the model has physical characteristics that relate well to physical processes being investigated, despite its largely empirical origin.

[6] The DBM approach aims to identify statistically optimal yet physically plausible models. It is normally based on statistical identification of linear or nonlinear transfer functions and the subsequent decomposition of these transfer functions into models that may have a physical interpretation, using algorithms available in the CAPTAIN Toolbox for Matlab (the fully functional CAPTAIN Toolbox for Matlab can be downloaded from and is available for use, without a license, for 3 months). Only models that are considered to perform well, are statistically well defined and parsimonious, and have an acceptable physical interpretation are accepted. The identified structure may then be used in a number of ways: for example, the assessment of conceptual models [Ratto et al., 2007], real-time forecasting [Young, 2002; Romanowicz et al., 2006], identification of spatial signals in response [McIntyre and Marshall, 2010], and providing links with reduced-order models that are used in the “emulation” of associated physically based models [Young and Ratto, 2009, 2011].

[7] This paper describes the application of the DBM modeling approach to identify nonlinearity in the rainfall-flow relationship and the subsequent attempt to infer a conceptual model on the basis of the DBM modeling results. As opposed to previous DBM applications to rainfall-flow modeling, which have used hourly to daily data, this study uses 10 min resolution measurements of rainfall and of flow in two parallel flow pathways. It is proposed that because of the small spatial scale in this case, the rainfall measurements represent the catchment average rainfall more precisely than typical DBM applications where there is significant noise in the rainfall inputs associated with spatial sampling error. With this “low-noise” rainfall, along with dual-pathway flow data, we are able to extract more information about nonlinear processes than has previously been achieved using DBM modeling, including internal routing nonlinearity, rather than just nonlinearity in the effective rainfall input.

2. DBM Modeling

2.1. Linear Transfer Function

[8] The general form of a single input discrete time linear transfer function is as follows:

equation image

where x is the measured model input, equation image is the model output, a = [a1a2 ··· an]T and b = [b0b1 ··· bm]T are parameter vectors, equation image is the uniform sampling interval used in the model (= tktk−1), k is the time step number (sampling index), zr is the backward shift operator (i.e., zrxk = xkr), equation image is the time delay between the onsets of x and equation image (restricted to being an integer multiple of the sampling interval equation image when using a discrete time model), and m + 1 and n are the number of numerator and denominator terms, respectively. Within the DBM context, the parameters of equation (1) are usually estimated using the refined instrumental variable methods [Young, 1984 and references therein] (enlarged and revised version in text, in press, 2001) and available in the CAPTAIN Toolbox. In the context of DBM rainfall-flow modeling, where equation image is the estimated streamflow, it is common for the measured input x to be defined as the estimated effective rainfall equation image, in which case equation (1) acts as a flow-routing function. Here and for the rest of the paper, we use the term “flow routing” to describe the movement of effective rainfall through the subsurface and/or surface of a catchment to a specified outlet location. While the generation of effective rainfall and its movement as flow through a catchment are not distinct processes, they are commonly treated this way in rainfall-flow models [Beven, 2001], and this is a fundamental assumption used in the DBM modeling described below.

[9] Equation (1) defines a family of model structures depending on how many terms are included and the time delay (i.e., the values of m, n, and equation image). Naturally, we seek a model structure that gives a good fit between modeled and observed response (flow in our context), as measured, for example, using the coefficient of determination (RT2) applied to the simulated model residuals [Pedregal et al., 2007, equation 6.54]. RT2 is equivalent here to the Nash-Sutcliffe efficiency, which is commonly used in hydrological modeling. To ensure reasonable parsimony, it is common to also evaluate a criterion that moderates the measure of fit with a measure of parameter uncertainty, for example, the well-known Akaike information criterion [Akaike, 1974] or Young's information criterion (YIC) [Young, 1990]. The final model selection criterion is that the identified transfer function should be decomposable into a conceptually plausible system of hydrological storages and potentially also pathways that bypass the storage [e.g., Young and Lees, 1993; Young, 2005].

[10] The simplest form of equation (1) that is commonly used in hydrological modeling is

equation image

which is a discrete time solution to a single linear store. A time constant (or residence time) representing the storage effects can be derived by conversion to continuous time using the well-known expression [e.g., Young, 1984]

equation image

[11] A steady state gain parameter G can also be derived from equation (2) in the form

equation image

where in the case where equation image and equation image are measured in the same units, G > 1 would imply that the system is gaining water from somewhere other than effective rainfall and G < 1 would imply a loss of water. Another transfer function commonly used in hydrological applications is the second-order process

equation image

which if the roots of the denominator polynomial 1 + a1z−1 + a2z2 are real, can be decomposed, for example, into two first-order systems in parallel or series or as a feedback connection, where the parallel connection is normally the most physically appropriate in the present context. These types of model structures are commonly identified as suitable when using daily resolution data [e.g., Young, 1993; Young and Beven, 1994; Lees, 2000; Young, 2003]. More complex transfer functions have been identified using hourly resolution data: for example, Young et al. [1997] and Young [1998, 2008a] identify models equivalent to two stores plus a instantaneous bypass flow, while Young [2010, 2011] identifies a third-order model with three stores and a bypass flow.

2.2. Nonlinearity and State-Dependent Parameter Estimation

[12] Effective rainfall is usually estimated from measured rainfall using a nonlinear model. Various conceptual soil moisture models are available and may be applied prior to the linear transfer function identification [e.g., Jakeman et al., 1990; Young, 2003; Chappell et al., 2006]. However, in keeping with the data-based ethos of the DBM approach, it is more common to employ an empirical model described by

equation image

where f(y) is an empirical nonlinear function of a chosen catchment wetness index y and r is measured rainfall. Originally, for example, Young [1993] and Young and Beven [1994] identified a power law function of observed flow q as the wetness index [see also Young et al., 1997; Lees, 2000, Young, 2002; Romanowicz et al., 2006; McIntyre and Marshall, 2010]

equation image

[13] Here equation image is a parameter that needs to be identified as part of the model fitting procedure, with high values signifying a highly nonlinear response and a zero value being equivalent to a linear effective rainfall (or loss) model. The coefficient c is not identified but can be selected by the modeler, for example, to impose the constraint equation image or to ensure that the total effective rainfall is equal in volume to the resulting flow. While the use of observed flow as a wetness index restricts the predictive use of the model to forecasting applications [e.g., Romanowicz et al., 2006] and precludes the direct application of the model to long-term predictive simulation, the objective here is identification of the form of nonlinearity, which can later inform the development of such a predictive simulation model.

[14] Alternative nonlinear models for the estimation of effective rainfall may be prespecified, their parameters optimized, and their performances intercompared [e.g., Ochieng and Otieno, 2009]; however, prespecification is somewhat unsatisfactory given the goal of minimizing prior assumptions about model structure. Instead, the nature of the nonlinearity can be examined using nonparametric state-dependent parameter estimation [Young, 2000, 2003]. Combining equations (1) and (6), the full rainfall-flow model then takes the form

equation image

or, equivalently, each parameter in the b vector can be considered to be a function of y, i.e.,

equation image

[15] The nature of the functions bi(y) (i = 1, m) can be explored by estimating the vector b as time variable using state-dependent parameter estimation techniques [see Young, 2000, 2003]. Plotting bi against y then shows the nature of any significant relationship that can then be parameterized. For example, in the hydrological context, using y = qk, various authors have justified the parameterized power law of equation (7) in this way [e.g., Lees, 2000; Ratto et al., 2007; Young, 2008a]. Equation (9) may be generalized further to allow parameters ai, i = 1, 2, …, n, to also be dependent on qk, which may signify internal nonlinearity in the routing model and time variable recession characteristics. Although this kind of nonlinearity exists in theory and nonlinear flow-routing models have been used in DBM and other applications [e.g., Moore and Bell, 2002; Romanowicz et al., 2006; Young et al., 2006; Segond et al., 2007], the dominant nonlinearity has traditionally been linked to flow generation via an “effective rainfall” input nonlinearity, rather than any form of flow-routing nonlinearity [Wheater et al., 1993; Beven, 2001]. Indeed, in the majority of flow-routing applications, combinations of linear stores or other simple unit hydrograph functions are employed. Also, prior to the case study in section 3, it seems that the DBM approach had not been used to help distinguish between the two sources of nonlinearity in the rainfall-flow response.

3. Case Study

[16] The case study aims to investigate whether with minimal prior hypotheses but good quality, high-resolution, supporting data the DBM method can be used to identify the nonlinear structure of a rainfall-flow model.

3.1. Case Study Data

[17] The plot under study (Figure 1) is within the Pontbren experimental catchment in Powys, mid-Wales, United Kingdom (see Marshall et al. [2009] for a location plan). The plot is part of a field used for sheep grazing and is agriculturally improved, meaning it has been underdrained by a network of tile drains and at various unknown dates has been plowed, fertilized, and reseeded with more productive grasses and clover. Soils are silty clay loams of the Cegin and Sannan series, with a distinct upper layer of approximately 30 cm depth overlying a relatively impermeable lower layer [Marshall et al., 2009]. For example, median sampled saturated hydraulic conductivities for the upper and lower layers were 1.67 and 0.018 m/d, respectively [Wheater et al., 2008]. Tile drains are located at around 750 mm beneath the surface. There is some evidence of mole activity that together with other bioturbation effects and soil structural change, could potentially create a macropore network. The climate is wet, with average annual rainfall in the field equal to 1449 mm (as measured between 1 April 2007 and 31 March 2009). Within the field is an instrumented plot (Figure 1) with a quite well defined but not isolated catchment area. The surface water catchment of the plot has an area of 0.0044 km2, with average surface slope of 12.5%, while the tile drain network implies an approximate subsurface catchment area of 0.0036 km2.

Figure 1.

Features of the case study plot.

[18] The plot was monitored between July 2005 and July 2009, although there are significant gaps in the data series [McIntyre and Marshall, 2010]. Measurements included rainfall, overland flow, subsurface drain flow, soil water pressure at a network of sites at multiple depths, and groundwater levels [Marshall et al., 2009]. The rainfall is monitored at 10 min resolution by a tipping bucket gauge with a nearby storage gauge (there was no significant difference in volumes between the two). The surface flow is intercepted by a gutter at the bottom edge of the plot, which transmits flow to a 45° V notch weir box in series with a tipping bucket to verify measurements. The flow from the network of tile drains collects in one main tile drain, the outlet of which is connected to a 45° V notch weir box, which is also in series with a tipping bucket. The surface and drain flows are measured at 1 min intervals, and the average flows over 5 min intervals are logged. Review of the time series flow data (e.g., Figure 2) illustrates that the flow from the plot is dominated by subsurface drain flow, with surface flow relatively intermittent and short-lived. For example, from November 2006 to October 2007, 94% of the recorded flow volume was drain flow. Flow peaks are also usually dominated by drain flow, although in wet conditions the peak surface flow can exceed the peak drain flow. Both pathways exhibit relatively flashy responses, with sudden bursts of rainfall in wet periods causing the drain flow to peak typically within 40 min of peak rainfall and the surface flow within 20 min. The response of the groundwater is, in general, a slow seasonal signal, without notable response to individual wet periods, which is assumed to mean that little water infiltrates though the relatively impermeable subsoil.

Figure 2.

Model results in the winter calibration period using the three parallel linear stores (equation (10)). The corresponding results using the preferred model is in Figure S3.

[19] While we consider the flow data to be a relatively accurate record of what is collected in the weir boxes (relative, that is, to flow data typically used in catchment-scale rainfall-flow studies), it cannot easily be argued that the measurements accurately represent the total plot outflow. First, it may be assumed that the surface flow collection gutter did not always intercept all the surface flow because of intermittent interference from animals, and during the largest recorded flow event (17–18 January 2007) the surface flow downpipe (connecting the gutter to the weir box) was observed to be partly blocked by storm debris. Second, the bounds of this plot were defined by topography and the identifiable subsurface drainage network, and there may have been significant unmeasured outflow particularly from throughflow. And third, although the water table variability is generally small and slow [Marshall et al., 2009], there is likely to be some unmeasured groundwater recharge. A water balance calculation for a winter period, assuming no net storage (as supported by tensiometer measurements) and that evaporation losses are equal to potential evaporation, suggests that 18% of the outflow is unmeasured. In summer, in periods of drain flow less than 0.02 L/s, some diurnal variations of up to approximately 0.001 L/s can be observed (e.g., Figure 6), with peaks at around 4:00 A.M. every day. The same signal appears in the soil water pressure measurements; hence, it is presumed to be related to the diurnal cycle of transpiration.

[20] The data period used for the DBM model identification is 10 November 2006 to 25 January 2007. This period is chosen because the relevant variables have been continuously monitored during this period and there are many events of varied magnitude and duration but without snowfall occurrence. Marshall et al. [2009] noted that large cracks appeared in the plot's soil during the summer of 2006 and that this affected the flow response for several months afterward. This will be considered when interpreting results. Although this winter period is described here as “wet,” it contained relatively dry spells, including a 15 day period with almost zero rainfall, and soil water pressure measurements show that the plot was not continually saturated [Marshall et al., 2009]. A second, drier period of 9 May 2007 to 30 July 2007 is used for validation. Rainfall-flow data are shown in Figure 2 for the winter calibration period (see Figure 6 in section 3.7 for the summer calibration period).

3.2. DBM Model Identification With Time-Invariant Parameters

[21] The DBM methods were implemented using the discrete time transfer function identification modules within the CAPTAIN toolbox [Pedregal et al., 2007; Taylor et al., 2007]. Continuous time transfer function modeling is an alternative approach, also available within CAPTAIN, which can provide better defined and less biased parameter estimates in cases where fairly rapidly sampled data are available. This is possible in this case, but the discrete time results yielded similar inferences and are reported here. The sampling interval used is the smallest possible with the available rain gauge data, equation image min. The simplified refined instrumental variable [see Young, 2008b] option of the RIVBJ routine within the CAPTAIN Toolbox is used for transfer function parameter estimation.

[22] The model identification strategy begins with a relatively straightforward and conventional DBM model identification exercise and then progressively explores avenues aimed at extracting more information from the data. Hence, to begin with, the aggregated surface and subsurface flow is used as the observed flow, and a linear flow-routing model is serially connected with a conventional nonlinear flow generation model, equation (7). Subsequently, models are fitted to the data from the two individual flow pathways to look for new information. Information about nonlinearity is then explored using state-dependent parameter estimation. It is assumed that any losses from the plot, including possible unmeasured components of inflow or outflow discussed previously, and any error in the estimated catchment area, are accounted for implicitly in the parameters of the model. This may affect only the volume scaling, for example, parameter c in equation (7), although if inflow or outflow volumes are related to the wetness of the plot, it is expected to also affect the nonlinearity parameter equation image.

[23] Note that the unit of flow used in all the modeling is mm/equation image, where equation image s. Therefore, the aggregated flow is not a single measurable variable; rather, it corresponds to Qs/As × 600 + Qd/Ad × 600, where Q is flow in L/s, A is catchment area in m2, and the subscripts s and d refer to the surface and subsurface catchments, respectively. For the first stage of analysis, the observed surface and drain flow are aggregated in this way for the purpose of fitting the model. The nonlinearity parameter equation image is optimized by maximizing RT2 for each considered transfer function structure. This included all combinations of models within the ranges m = 1–3 and n = 1–3, giving nine model structures in total. The performance and optimized parameters of two of the tested structures are given in Table 1 (models 1 and 2). Model 2, with two parallel stores, gives the best RT2 value (0.92), and this model is about the same as model 1, a single store, in terms of YIC (−11.6 compared to −11.9). YIC is a logarithmic measure, so that a more negative value, coupled with a comparatively high RT2, indicates a better identified model: for a more detailed explanation; see Young [1990] and Appendix 3 of Young [2001]. The identification of two parallel stores as a suitable model for this plot is neither new nor surprising: the same result was found by McIntyre and Marshall [2010] in their broader analysis of the Pontbren streamflow data, and it is known a priori from the measurements that there are dual flow pathways. Nevertheless, this result provides a benchmark to assess whether or not the model complexity can be developed systematically by applying the DBM analysis to the separated surface and drain flow components of flow.

Table 1. Optimized Performance and Parameter Values for Model Defined by Equation (7) Combined With Different Transfer Functions for the Winter Period
ModelModeled VariableaStructure n, m, δFlow Generation ParametersLinear Routing ParametersNonlinear Routing ParametersTime Lag δ (min)Gain GNumber of Parameters NPerformance
cλT1 (h)T2 (h)split to T1βαYICbRT2
  • a

    Here s is the measured surface flow, and d is the measured drain flow.

  • b

    Young's information criterion (YIC) is not calculated for the nonlinear models because the parameter covariance is unknown.

  • c

    Nonlinear one-store routing models.

1q = s + d1, 1, 01.650.372.3     0.714−11.90.87
2q = s + d2, 2, 01.920.440.9712.50.47   0.816−11.60.92
3s1, 1, 01.020.820.35     0.764−12.40.91
4s2, 2, 00.990.810.273.650.51   0.846−8.00.92
5s1, 1, 0c0.730.70   0.650.25 0.955 0.93
6s1, 1, 0c0.680.68   0.690.28  4 0.93
7d1, 1,     0.804−12.20.88
8d2, 2, 02.640.571.3721.80.45   0.806−10.40.91
9d1, 1, 1c0.52−0.06   0.541.04101.076 0.96
10d1, 1, 1c0.62    0.440.9310 4 0.96

[24] For each of the two flow components the same array of nine different transfer function structures was tested. At this stage, we maintained the nonlinear model of equation (7) for each flow component and, again, optimized equation image for each tested linear transfer function. Selected results are included in Table 1. For the surface flow component, a single linear store with equation image and equation image (model 3 in Table 1) is selected as the best compromise between performance and parsimony, and for the drain flow, a parallel store model equation image and equation image (model 8) is preferred. Superimposing these two models gives the transfer function

equation image
equation image
equation image

where s is the measured surface flow, d is the measured drain flow, equation image is the estimated effective rainfall going to surface flow, and equation image is the estimated effective rainfall going to drain flow. Equation (10a) is equivalent to three parallel flow pathways [e.g., Young et al., 1997; Young, 2005, 2008a], but unlike in previous models, the split between the three pathways is not constant because of the presence of parallel nonlinear effective rainfall models (equations (10b) and (10c)). The aggregated surface and drain flow time series obtained from equation (10) is shown in Figure 2, along with aggregated observed flow, rainfall, and estimated effective rainfall. In Figure 2, the flow is log transformed to exaggerate the low flow errors. The RT2 value for the aggregated flow is 0.93. Although this is only 0.01 more than the original two parallel store models and therefore, arguably, a small if not insignificant performance improvement, the more complex transfer function facilitates a simulation which can distinguish between surface and subsurface pathways. This distinction is potentially important, for example, if the DBM results are used to develop models for assessing impacts of soil and drainage management.

[25] A feature of the result in Figure 2 is that the two highest flow peaks (on the 13 December and the 18 January) are modeled well only because the estimated effective rainfall is significantly higher than (almost twice the volume of) the observed rainfall. This is mathematically possible because the constraint equation image is not imposed upon the nonlinear model, equation (7). In general, the occurrence of equation image> rk may be considered physically reasonable because if rainfall measured from one or more rain gauges is treated as an estimate of catchment average rainfall, it is expected to contain significant spatial sampling errors. And, of course, the DBM model is inherently stochastic, so that the estimated uncertainty has to be taken into account in any evaluation of the model characteristics. In the case study, however, errors arising from sampling the catchment area are presumed to be small because the plot covers an area of only around 65 m × 65 m, with the rain gauge located almost centrally within it, and comparisons with storage gauge readings show no volume bias (although both may suffer from undercatch). Furthermore, closer examination of the hydrographs in Figure 2 (e.g., Figure 3) shows that the accumulated volume of flow over each of the two largest events (12–13 December and 17–18 January) is overestimated by the model despite the good fit to peaks, supporting the view that equation image has been overestimated. Adding to concerns about the model of equation (10), Figure 2 illustrates persistent underestimation of low flows. Therefore, despite its impressive performance in terms of RT2, we do not accept the model of equation (10) on the grounds that the generated effective rainfall is not physically realistic and there is a low flow bias. Instead, we go on to more critically assess the nonlinear component of the model.

Figure 3.

Model performances for four example events (a–d) using equation (10) and (e–h) using equation (16).

3.3. State-Dependent Parameter Analysis for Drain Flow

[26] We proceed now to examine whether the nonlinear component of the drain flow model can be improved using state-dependent parameter estimation. To facilitate clear signals of state dependence, the simplest of the applicable transfer function structures, equivalent to one single store, is preferred as a starting point [Young, 2003]. Adapting equation (2) to this form, we obtain

equation image

[27] Both the a1 and b0 parameters were simultaneously identified as state-dependent parameters, using the estimation procedure described by Pedregal et al. [2007, chapter 5] and Young et al. [2001], assuming for now that equation image. The default settings in the CAPTAIN Toolbox SDP function were used, except that the code for the number of iterations used to estimate the noise variance ratio smoothing parameter was set to −2 on the recommendation of Pedregal et al. [2007, p. 98] and the integrated random walk [see Pedregal et al. 2007, p. 19] was used to ensure a smooth estimate of the state-dependent parameters. As well as using the observed drain flow (d) as the state on which the parameters are dependent, as specified in equation (11), the observed surface flow (s), the aggregated flow (s + d), and the plot average soil water tension (p) were also considered, with the aim of identifying the most representative catchment wetness index (y) for use in equation (6). Although state dependence was present in all cases (the results are shown in the auxiliary material in Figure S1), by far, the least noisy dependence was achieved using d as the state, and these results are presented in detail in Figure 4 and subsequent analysis.

[28] Figures 4a and 4b show the relationships b0(d) and a1(d) for drain flow in the winter period. Figure 4a supports the power law function, equation (7), previously assumed, although the result may also arise from dependency of a1 on the value of b0, as explained below. The state dependence of a1 for drain flow is illustrated in Figure 4b, showing that the time constant T decreases as flow increases. This result may be interpreted in two ways: the state dependency of T illustrated by Figure 4b may be because a single drain flow store has been used to represent a dual-pathway drain flow system (this is easily confirmed by applying the single-store state-dependent parameter analysis to the outputs of a parallel store model); alternatively, or additionally, the result may be interpreted as a kinematic routing effect.

Figure 4.

Model parameters as functions of flow in the winter period: (a) parameter b0 in the drain flow model, (b) parameter a1 in the drain flow model, (c) parameter b0 in the surface flow model, and (d) parameter a1 in the surface flow model.

3.4. Proposed Mechanistic Interpretations of the State Dependence

[29] The kinematic wave model [e.g., Beven, 2001, p. 177] requires that equation image where v is celerity, q is flow, and equation image and equation image are parameters related to the physical properties of the flow path. Celerity may also be treated as equal to a representative length equation image divided by a representative time of travel, which in this case, is the time constant T since there is no time delay equation image; hence, equation image. Using the approximation to equation (3), equation image parameter a1 can then itself be expressed in the form

equation image

where equation image and equation image have been combined into one parameter equation image with units (mm/equation image. The same form of a1(dk) is observed when fixing equation image min; hence, (12) is valid for either equation image or equation image min. This theoretically derived expression is consistent with the form of the empirical relationship in Figure 4b, where equation image < 1. Therefore, the nonlinear kinematic routing is accepted as a plausible model, and equation (11) becomes

equation image

[30] If the gain parameter G is forced to be constant, from equation (4), equation image and the drain flow model would become

equation image

[31] An alternative explanation for the result in Figure 4a can now be seen: a1 may vary because of kinematic routing effects, and hence, b0 must vary in order to maintain a constant steady state gain between rainfall and flow. Therefore, the state-dependent analysis results for the drain flow lead to three alternative propositions: (1) there are two parallel linear routing stores and nonlinear flow generation, (2) there is one nonlinear routing store as in equation (14) with linear flow generation, or (3) there is a mixture of the nonlinear flow generation and nonlinear routing which, assuming a single store, may be expressed as

equation image
equation image

3.5. Assessment of the Proposed Mechanistic Interpretations

[32] The first proposal, that there are two parallel linear routing stores and nonlinear flow generation, has already been assessed (the drain flow model in equation (10)) and was discounted. The second and third propositions are now explored by optimizing the parameters of the mixed model, equation (15). If the optimum value of equation image is not significantly different from zero, then the second proposition is supported, while if it is significantly greater than zero, then the third proposition is supported. Prior to optimizing the parameters, the observed drain flow dk in equation (15a) is replaced by simulated drain flow at the preceding time step equation image. This prevents feedback effects that cause instability in the validation period results (described in section 3.8). Parameters G, equation image, equation image and equation image are optimized to the drain flow observations using Matlab's nonlinear solver “fminsearch” with RT2 as the criterion. The time delay parameter equation image is also optimized by trial and error, and the c parameter is fixed, as in the previously optimized models, so that the volume of effective rainfall equals the volume of observed flow. The optimized parameter values are shown in Table 1 (model 9). The optimized value of equation image (−0.06) implies that overall, there is less flow generated during wetter periods than in drier periods; however, this value is not considered to be significantly different from zero, and so our second proposition is supported: a linear drain flow generation model.

[33] Another feature of the optimized equation (15) is that G is close to unity. This is perhaps unsurprising, as the value of c has been fixed so that the volume of effective rainfall is equal to the value of observed flow, so that, in principle, there should be no need for a gain parameter. However, in practice, G ≈ 1 is a nontrivial result because it signifies that consistency between observed and simulated drain flow volumes coincides with the RT2 optimal model (this consistency was not present when using linear routing, as indicated by the values of G in Table 1). Fixing equation image and G = 1 and reoptimizing equation image and equation image yield the values in Table 1 (model 10) with associated RT2 = 0.96. This is a remarkably good performance for a four-parameter (c, equation image and equation image) model. Furthermore, unlike the majority of DBM rainfall-flow models, this model avoids the use of observed flow as a surrogate measure of wetness and therefore may be used directly as a simulation model for prediction.

[34] However, model 10 is questionable for at least three reasons. First, recalling that the plot soil was not continually saturated, a linear flow generation model is contrary to the general experience that flow generation under unsaturated soil conditions is a nonlinear process. A second related issue is that the value equation image is higher than expected a priori: calibrated values for catchment-scale models typically range from 0.2 to 0.7 [e.g., Wittenberg, 1993; Segond et al., 2007], and theoretical analysis for unconfined saturated subsurface flow gives a value of 0.5 [Moore and Bell, 2002]. These two issues point to the possibility of interaction between equation image and equation image within the optimization, so that equation image is lower than expected and equation image is higher than expected. However, plotting the response of RT2 against the parameters of equation (15) shows unambiguously that value of equation image is close to zero and that equation image is close to 0.93. This plot is included in the auxiliary material in Figure S2.

[35] The remaining question about model 10 is whether a dual-pathway drain flow model, where one or both pathways are nonlinear in the form of equation (15), might perform better. Indeed, when a state-dependent analysis was applied to the disaggregated fast flow component of drain flow df (i.e., the observed drain flow minus the modeled slow drain flow from equation (10)), it did indicate nonlinearity of the form evident in Figure 4b. To answer the question, therefore, a nonlinear fast drain flow model was optimized conditional on the linear slow drain flow model specified within equation (10).

[36] With the RT2 value remaining at 0.96, the addition of the slow flow store was not justified. Furthermore, a response surface analysis of the type shown in Figure S2 indicated poor identifiability, especially for the slow flow b0 parameter. On the other hand, an attraction of this dual-pathway drain flow model is the more reasonable values (in terms of prior expectations) obtained for equation image and equation image (0.18 and 0.65, respectively). In other words, there seems to be a more reasonable split of nonlinearity between the flow generation component and the fast flow-routing component. While at this stage in the analysis a dual-pathway model cannot be ruled out as an alternative description of the drain flow system, precedence is given to the more parsimonious single-store model.

3.6. Surface Flow Model

[37] For the surface flow model, using surface flow as the state (rather than drain flow, aggregated flow, or soil water pressure) produced the clearest signals of state dependence in the parameters. Figure 4c shows a surprising reduction in b0 at the highest values of surface flow, and Figure 4d shows a1 increasing with increasing flow, particularly at the upper range of flows. This unexpected state dependence is largely because the surface runoff coefficient for the largest storm event (17–18 January 2007) was considerably less than those for the medium size events, and its time constant was larger. This is consistent with the field observation that the surface flow downpipe (see section 3.1) failed to collect and efficiently transmit the full surface flow on that occasion. If a nonlinear model of the form of equation (15) is pursued for the surface flow, despite the lack of clear evidence in Figure 4, then the optimized parameter values are as shown in Table 1 (model 5 when G is optimized and model 6 when G is fixed to 1.0).

[38] Model 6 has the same number of parameters as the linear single-store model, while the RT2 value has improved from 0.91 to 0.93. Also, the value equation image is comparable with established values for overland flow; for example, the classic Chezy friction equation for surface flow would give equation image. This supports an argument that the DBM results should be overruled in this case, given prior knowledge of process nonlinearity and field measurement problems. However, in keeping with the DBM ethos that adding complexity to the model on the basis of prior expectations should be avoided wherever possible, the linear surface flow routing in equation (10) is favored, although the nonlinear counterpart cannot be ruled out as an alternative plausible description.

3.7. Preferred Aggregated Flow Model

[39] Combining the preferred surface and subsurface models (models 3 and 10 in Table 1) leads to

equation image
equation image
equation image

[40] Equation (16) gives an RT2 value for the aggregated flow of 0.97. To illustrate the performance in detail, four example events are included in Figure 3, comparing results to those obtained from the benchmark linear routing model, equation (10). The performance over the full winter time series is shown in the auxiliary material in Figure S3. These illustrations show that the overestimation of effective rainfall during the largest events, 12–13 December and 17–18 January, arising from the assumption of equation (10) that all the nonlinearity was in the flow generation has been solved. Figure S3 shows that the low flow bias has also been substantially solved, and the absence of a drain flow steady state gain (i.e., G = 1.0) signifies a more satisfactory mass balance than is implicit in equation (10). Adoption of the alternative more complex models (dual-pathway nonlinear drain flow routing and/or nonlinear surface flow routing) produced the same or only marginally better performances.

[41] Some properties of the model residuals obtained from equation (16) are shown in Figure 5. Figures 5b and 5d show some bias toward negative residuals at the medium range of rainfall and flows. This indicates some systematic error in the drain flow recessions where, potentially, underestimation of medium flows is needed to achieve good peak flow and base flow performance. The bias persists if we use the dual-pathway nonlinear drain flow routing, and so the bias cannot easily be interpreted as lack of flow pathways. Figures 5b, 5c, and 5d also show positive residuals during the large event of 17–18 January 2007, which was dominated by surface flow. These residuals are associated largely with timing errors (see Figure 3h); however, interpretation is complicated by the scope for observation error in surface flow during this event, as already noted. There is no evidence in Figures 3, 4, 5, or S3 that the swelling of the soil observed by Marshall et al. [2009] following the dry summer of 2006 has altered the flow response within this winter period.

Figure 5.

Error analysis (equation (16), winter period). (a) The x axis and y axis are curtailed so that the distribution tails are more visible: the actual peak value of relative frequency is 0.32 in the bin centered around an error of −0.0006 mm equation image. (b–d) The actual maximum errors are also shown.

3.8. A Summer Validation Period

[42] All the results presented so far are for the wet winter period 10 November 2006 to 25 January 2007. Insight into processes may be gained by closer analysis of the summer “validation” period of 9 May to 30 July. This period followed a dry April where observed flows fell to zero by the beginning of May (for this reason 1 May was not used as the start of the period; a zero initial condition is problematic because of the flow dependence of parameters). The dry spell continued until 15 June with only two significant flow events, and flows were sufficiently low that diurnal transpiration signals of around 0.001 L/s became visible, as previously noted. The subsequent wet period lasted until the end of July (Figure 6). Given the differences between the winter and summer conditions, this is a difficult validation period because some mechanisms operative over the summer period may not have been active over the winter, so the results reported in this section should be considered with this in mind.

Figure 6.

Model results in the summer validation period using equation (16).

[43] The model of equation (16), estimated using the winter period, performed well on the wet summer period (Figure 6, from 15 June), with an RT2 value for the aggregated flow of 0.96, and if the parameters are reoptimized (as before, for drain and surface flow models separately), their values are close to those estimated for the winter period. However, equation (16) performs less well if the whole 2 month summer period is included (the entire period in Figure 6), with an RT2 value for the aggregated flow of 0.83. The loss in performance lies in both surface and drain flow components. The error structure over the summer period is dominated by overestimation of flow events within and immediately after the dry weather and underestimation of similarly sized events in the subsequent wetter period. If the summer period is extended to include the dry weather in August, this pattern repeats.

[44] However, reoptimizing the parameters within the model structure of equation (16) (including equation image in the drain flow model), giving an optimal RT2 value of 0.91 for the aggregated flow for the period shown in Figure 6, does not recover the level of performance achieved in the wet periods. Plausible reasons for the apparent structural change in the model during and after the dry periods are the increased significance of the diurnal transpiration cycle, the presence of the strong nonlinearity and hysteresis usually found in drying-wetting cycles of clay rich soils [e.g., Topp, 1971], the presence of infiltration excess flow (although there is no observed evidence of this), changes in the properties of the grass, and/or changes in the soil structure, for example, a stronger role of macropores after dry periods. As noted, Marshall et al. [2009] reported large cracks appeared in the plot's soil during the dry summer of 2006, and these appeared to affect the groundwater dynamics. However, there was no visible change to the soil structure or unusual groundwater dynamics during the 2 month validation period. Furthermore, there was no evidence that the change in response was related to the available potential evaporation estimates. A state-dependent parameter estimation exercise on the 2 month summer period produced results similar to those for the winter period, except with a very noisy relationship between b0 and flow for the drain flow model. This implies that influences other than rainfall and integrated antecedent wetness affected the variability of drain flow generation in the summer, potentially relating to the factors suggested above. Examination of runoff and soil moisture data over multiple experimental plots is currently underway to help resolve this.

3.9. Wetness Index Simulation

[45] One motivation for the DBM analysis is to assist in identification of a model that is capable of predicting flow response to scenarios of future rainfall inputs. The routing models identified here may be applied directly to prediction and so can the wet period drain flow generation model. The optimal surface flow and dry period drain flow generation models, however, used observed flow as a surrogate wetness index. A possible solution is to use the flow estimated at the previous time step; however, this led to unsatisfactory results for both drain and surface flows. Therefore, the value of the DBM analysis for making predictions partly rests in its potential for use in multistep-ahead forecasting when the data exhibits time delays and to inform hypotheses about models that can simulate catchment wetness.

[46] A range of simple models were tested, and the discussion here focuses on the one which gave the most satisfactory results. The DBM results in Figure 4 are closely associated with models that replace the observed flow in equation (7) with a simulated catchment wetness index [Young, 2003]. This includes the models described by Jakeman et al. [1990] and the subsequent versions described by Ye et al. [1995] and McIntyre and Al Qurashi [2009]. For the winter period, attempts to optimize these models led to the same conclusion as the DBM analysis: the generated flow is not significantly related to variability of wetness, and the linear effective rainfall model of equation (16b) is preferred. For the summer period, a three-parameter version of the catchment wetness index model (which assumes a first-order loss from the soil store and that effective rainfall is proportional to wetness raised to a power equation image) combined with the nonlinear routing store in equation (16a) achieved a satisfactory performance over the full range of summer flows, with RT2 = 0.95. Arguably, such simple drain flow generation models would not have been presumed to work well prior to a DBM analysis, and parameter values would have been significantly biased if the routing nonlinearity had not been separated out. However, given the difficulty of addressing the hysteretic and nonstationary behavior proposed in section 3.8, identifying a single predictive model to simulate flows over both the winter and summer periods is more challenging and has not been addressed in this work.

[47] Identifying a predictive simulation model of surface flow also remains a challenge The best performance for surface flow in the winter period was RT2 = 0.79 using a five-parameter catchment wetness index model, which accounted for losses using daily estimates of potential evaporation and included an infiltration excess parameter, along with a linear routing store structure (as in model 3 in Table 1). Using a nonlinear surface-routing store (as in model 5) improved the result only slightly, with RT2 = 0.81. In the summer period, the best model (the five-parameter catchment wetness index model with a single linear routing store) achieved only RT2 = 0.55 after optimization. It seems that the surface flow generation processes are not controlled solely by a lumped soil wetness index: the observed flow was by far the best wetness index, as suggested in the previously cited DBM rainfall-flow modeling studies.

[48] It may be conceptually attractive to use the same wetness index for the generation of flow for all pathways, as is normal practice in conceptual modeling at both plot and catchment scales [e.g., Lee et al., 2005; Krueger et al., 2010]. The previous DBM results (Figure S1) indicate that this may not be effective for the case study plot, and this is confirmed by testing wetness index models and explicit soil moisture accounting models, including those proposed for this same plot prior to DBM analysis [Wheater et al., 2008]. While it would not be reasonable to suggest that the surface and subsurface flow processes are independent, the DBM results illustrate that the separation is stronger than presumed a priori.

4. Conclusions

[49] This paper has demonstrated the identification and estimation of a plot-scale rainfall-flow model using the data-based mechanistic (DBM) modeling approach, including the allocation of nonlinearity between the flow generation and routing components of the model. The data were considered to be more accurate than typically used for rainfall-flow modeling because of the double measurement systems used for flow and low errors in spatial rainfall estimates. Parsimonious models of two parallel pathways, drain flow and surface flow, were identified independently, and the models were superimposed into a model of total flow. In winter and summer wet periods, a linear model was identified for the drain flow generation (i.e., a proportional loss model), and there was stronger than expected nonlinearity in drain flow routing. The wet period performances of the surface flow and drain flow models (RT2 = 0.91 and RT2 = 0.96) are remarkable for models with only four parameters, and superimposing the models achieved an even better performance for aggregated flow (RT2 = 0.97).

[50] Despite the good performance in relatively wet periods, the model was less successful when forced to alternate between very dry and very wet periods, and it is speculated that this is due to nonlinearity in soil moisture dynamics and/or nonstationarity in the vegetation or soil structure, for example, due to opening and closing of macropores, as observed by Marshall et al. [2009]. The DBM method could be used to explore the cause of this variability if suitable additional data were considered, for example, air temperature, which may be a primary cause of the structural change in the soil. A conceptual model that was based on the DBM modeling results and which matched the DBM performance was found for drain flow but was elusive for surface flow.

[51] The search for methods to assist in identification of model structures has received considerable interest from hydrologists, as illustrated by the papers cited in section 1. Two features stand out from these and other papers: the difficulty of distinguishing between candidate model structures given data and parameter uncertainty and, in the context of rainfall-flow modeling, the common result that all the nonlinearity is included in the effective rainfall generation model. The latter feature is partly a consequence of the former: the typical accuracy of rainfall-flow data may not permit unambiguous identification of routing nonlinearity. In addition, arguably, the linear routing is commonly used simply for convenience because of the numerical efficiency of the analytical solutions.

[52] In contrast to these previous studies, a significant aspect of the DBM modeling reported in this paper is the identification and estimation of nonlinear state dependency within the model (i.e., in the feedback within the model, rather than just in the effective rainfall nonlinearity at the model input). Here the effect of state dependency is to change the recession characteristics of the drain flow pathway as a nonlinear function of the soil wetness, revealing that the associated time constant decreases nonlinearly as the drain flow increases. This is the first DBM rainfall-flow modeling study that has clearly identified such characteristics. Furthermore, the paper has shown that apportioning nonlinearity between flow generation and flow-routing components in this way has considerable rewards in terms of the model performance and the physical plausibility of its internal functioning. For example, the identification of a nonlinear loss model conditional on a linear routing model may corrupt the internal realism of the model despite good flow performance. Having reached this conclusion using high-quality experimental data, future investigation is required using more typical, catchment-scale, operational data sets.


[53] This research was partly funded by the UK Flood Risk Management Research Consortium Phase 2, EPSRC grant EP/F020511/1. The CAPTAIN toolbox ( was used under license from Lancaster University. The Pontbren experiment is possible due to the support of the Pontbren Group, Llyn Hir landowners, and Coed Cymru.