Reduced-order models (ROMs) approximate the high-dimensional state of a dynamic system with a low-dimensional approximation in a subspace of the state space. Properly constructed, they are used to significantly reduce the computational cost associated with the simulation of complex dynamic systems such as flow and transport in the subsurface. A key component in model reduction is to construct the subspace where we look for approximate solutions. In this work, we apply model reduction in inverse modeling and use the solution parameter space of underdetermined geostatistical inverse problems to construct the subspace in which we seek approximate solutions for any given parameters needed in the inversion process. The subspace is constructed by collecting state variable (e.g., pressure) distributions in the flow domain. Each of the distributions, called snapshots, contains the result of full forward model simulation for a given test with a basis vector in the solution parameter space as input parameters. We then use linear combinations of the snapshots to approximate the forward model solution for any parameters needed in inverse modeling. In geostatistical inverse modeling, the solution parameter space is spanned by the cross-covariance of measurements and parameters; hence, we name the ROM as the geostatistical reduced-order model (GROM). We also show that with minor loss of accuracy in the forward model, the accuracy in parameter estimation is still high, and the saving in computational cost is significant, especially for large-scale inverse problems where the number of unknowns is enormous.
 The use of reduced-order models (ROMs) is becoming popular in approximating complex and computationally expensive numerical models, such as the simulation of subsurface flow and transport processes [Vermeulen et al., 2004; Markovinović and Jansen, 2006; Razavi et al., 2012]. These processes often occur in a large three-dimensional heterogeneous domain, may involve multiple fluid phases, and are governed by nonlinear flow and transport equations that are computationally expensive to solve. The issue of computational cost is even more serious in resources management optimization, inverse modeling, and uncertainty quantification in which multiple forward simulations of these processes are needed.
 Model reduction is first introduced in the analysis of turbulent flow [Lumley, 1967; Berkooz et al., 1993]. An earlier detailed review of model reduction with the Karhunen-Loève (KL) expansion can be found in Newman . The idea of model reduction is to constrain the solution of the forward model to a subspace of the actual solution space (e.g., Galerkin projection). The subspace usually has a much lower dimension so that at the cost of minor accuracy loss, the computational saving in finding the reduced-order solution instead of the exact solution is substantial. Further, the approximation error is manageable and often trivial for field applications that involve a fair amount of noise in measurement data. To construct the subspace, one first computes with the full model a number of state variable (e.g., pressure) distributions, or snapshots, in the space domain for a variable set of times or parameters. The snapshots are then orthogonalized to remove redundant information and form a basis set to which the reduced-order solution will be projected.
 ROMs have been developed for single-phase and multiphase flow problems. For example, Cazemier et al.  use proper orthogonal decomposition (POD) and Galerkin projections in driven cavity flow simulations. Vermeulen et al.  use similar strategies for groundwater flow simulations. Markovinović and Jansen  use model reduction to construct solution predictors for iterative solution of multiphase flow problems in porous media. Cardoso and Durlofsky  construct linearized ROMs for multiphase subsurface flow problems. Siade et al.  propose methods for snapshot selection on the time axis for groundwater flow model reduction, and each snapshot is the hydraulic head configuration in the whole space domain at a certain time. More recently, Razavi et al.  categorize model reduction as a type of surrogate modeling strategy and provide a review of its use in water resources applications.
 ROMs have also been used in parameter estimation and inverse modeling, and most of their use can be summarized in two ways. The first way is to select in an efficient manner candidate samples for Monte Carlo (MC) sampling, especially Markov chain Monte Carlo (MCMC) sampling. The more accurate a ROM is, the higher the probability that a poor candidate will be filtered out by the ROM before checking it with the full model. For instance, Lieberman et al.  propose an MCMC algorithm with model reduction for statistical inverse problems of groundwater flow. In their work, the authors construct a reduced steady-state model with the objective of covering the parameter space by choosing parameter sets via a greedy algorithm that uses the prior information of parameters. Watzenig et al.  use ROMs in electrical capacitance tomography to accelerate MCMC sampling. The other way is to find the optimal parameters that meet a certain set of criteria such as data fitting and spatial smoothness requirements in a calibration-by-optimization mindset. In field applications, the approximation error from ROMs is often smaller than the noise in measurement data so that the parameter estimation is almost not affected. For example, Park et al.  use model reduction to solve inverse heat transfer problems that involve unknown time-varying heat source functions. Vermeulen et al.  use model reduction in inverse modeling of groundwater flow problems with six to nine unknown parameters. Siade et al.  use a ROM with quadratic programming to estimate up to 15 parameters in groundwater flow problems. Pasetto et al.  use the POD to reduce the computational cost associated with Monte Carlo uncertainty analysis in a problem of groundwater flow driven by randomly distributed recharge. In all of these applications, the number of parameters estimated is very small, and the way in which snapshots are selected in the parameter domain makes the construction of ROMs computationally prohibitive and hence the approximation inaccurate when the number of parameters is large.
 In this work, we apply model reduction to underdetermined geostatistical inverse problems in which a large number of unknown parameters are estimated using a limited amount of measurements. We propose a new way of snapshot selection for such underdetermined inverse problems based on the low-dimensional solution space of linear underdetermined geostatistical inverse problems. With this method, the number of snapshots is proportional to the dimension of the solution space, which is often m + p, where m is the number of measurements and p is the number of unique mean values in the parameter domain (1 if we assume a homogeneous mean field). Since the solution space is spanned by the cross-covariance of measurements and parameters, we refer to this ROM as the geostatistical reduced-order model (GROM).
 This paper is arranged as follows. In section 2, we first review concepts and discuss methods in model reduction and geostatistical inverse modeling with an example in groundwater flow. Then we develop efficient snapshot selection strategies for underdetermined geostatistical inverse problems. In section 3, we use numerical examples in groundwater flow to (1) examine the accuracy of the new snapshot selection strategy and (2) test efficacy and efficiency of the new GROM in estimating synthetic Gaussian parameter fields with noise-free numerical data. In section 4, we apply the GROM to steady-state hydraulic tomography to estimate non-Gaussian parameter fields with noisy laboratory-collected hydraulic data. We then discuss the results in section 5.
 In this section, we first use a groundwater flow problem as an example to review the general concept and process of model reduction with Galerkin projection. We then discuss about inclusion of the reduced-order model (ROM) in geostatistical inverse modeling and derive the adjoint equation for sensitivity computing. Then we propose a new snapshot selection strategy in the parameter domain for underdetermined geostatistical inverse modeling and name the so-constructed ROM as the geostatistical reduced-order model (GROM).
 In this part and the rest of this paper, we use subscript “r” to represent systems or solutions with respect to the reduced model, and “ ” over a symbol to represent the corresponding variable approximated with the reduced model.
2.1. Model Reduction With Galerkin Projection
 Let us start from a steady-state groundwater flow model in a space domain Ω, i.e.,
subject to conditions at Dirichlet boundary and Neumann boundary :
where K represents hydraulic conductivity; Q is the pumping(−)/injection(+) rate at location xQ, δ is the Dirac delta function, h1 is the constant boundary head value, and n is a unit vector pointing outward the domain Ω.
 After discretizing Ω into n elements and solving equations (1)-(3) numerically, we obtain a linear system of n equations such as
where represents the vector of all elementwise unknown parameters and h is a vector of hydraulic head at all elements in the domain. We refer to equation (4) as the full model and for simplicity, we will write for A(s) and for b(s) in the rest of this paper.
 After projecting the state vector to an nr-dimensional subspace ( ) spanned by orthonormal columns in a basis matrix , i.e.,
where approximates h by a linear combination of basis vectors in P and hr contains coefficients that we need to solve for in the ROM, we have
 However, this system has more equations (n) than unknowns (nr), and hence, it does not have a solution. As a result, instead of requiring the residual be zero in , we make it zero in the range space of P and reach a reduced-order model (ROM) given as
where and . By solving equation (7), we make the residual perpendicular to the range space of P.
 To construct the basis matrix P, we first create the snapshot matrix . It has ns columns and each column is a snapshot, which by definition, is the head configuration (h) in the domain Ω for a certain test with a specific set of parameters s [Newman, 1996]. Then we compute the empirical covariance matrix whose incomplete eigenvalue decomposition leads to after retaining only nr eigen-pairs corresponding to relatively large eigenvalues [Siade et al., 2010]. The eigenvalue decomposition of is closely related to that of a much smaller matrix . Indeed, if , we have , which means that the eigenvector of is S times that of , and they share the same corresponding eigenvalue.
 We need to mention here that traditionally the model reduction is only in the time domain for transient problems, which means snapshots of the system state h are taken at a set of times and h at any other time is approximated by a linear combination of the snapshots [Newman, 1996]. An optimal snapshot selection method along the time axis is also proposed in Siade et al. . Another well-known model reduction technique in the time domain is the use of temporal moments [Cirpka and Kitanidis, 2000a; Nowak and Cirpka, 2006; Yin and Illman, 2009]. Particularly, Leube et al.  generalize temporal moments with temporal characteristics, and it is evident that snapshots collected along the time axis are temporal characteristics with base functions , where δ is the Dirac delta function and tk is the time when the snapshot is taken. Although it is feasible to reduce the model in both parameter and time domains, for demonstration purposes, in this paper we only consider model reduction in the parameter domain.
2.2. Geostatistical Inverse Modeling With ROM
 Following Kitanidis  for the full model, we can also write a similar objective function for parameter estimation with the ROM, i.e., the posterior density function of parameters conditioned on measurement data :
where is a zonation matrix related to the prior mean values of s, and it is often an all-one vector (p = 1) if s has a homogeneous mean field; contains the prior mean values of s; where is the measurement operator; is the prior covariance matrix of s, and is the observation error covariance matrix.
 Assuming a noninformative prior for , i.e., a uniform distribution on , and integrating it out from equation (8) leads to
where . The optimal estimate of parameters s is the one that maximizes equation (9). This is a nonlinear inverse problem because the forward model , or the flow model in equation (1), is a nonlinear function with respect to unknown parameters s. As shown in Kitanidis  and briefly explained in the following subsection, the maximization is often done with the Newton's method that requires the linearization of with respect to s. Indeed, similar to the original head equation in (1), we can also derive the adjoint equation for the sensitivity of with respect to s.
 Let us write the measurement operator as , then for the ROM, we can write the sensitivity matrix [ ] as
where can be easily found by solving adjoint-state equations with respect to equation (7). Indeed, we have
where is a zero vector with the ith row replaced by one, and and can both be easily computed without solving any linear systems. Notice that here we have used the derivative rule for the inverse of a matrix, i.e., .
 The question left is how to select a set of snapshots for the construction of P that provides relatively accurate approximation of h for any parameter sets needed in the inversion process while at the same time is computationally inexpensive to find.
2.3. Snapshot Selection and the GROM
 Previous studies use random or deterministic sampling of the parameter domain to take snapshots. For example, to select snapshots, Siade et al.  use preassumed upper and lower bounds to enumerate all possible combinations of the limiting values of the parameters. However, this method is not applicable to inverse problems with a large number of unknown parameters. Another option is Monte Carlo (MC) sampling of the parameter space [Pasetto et al., 2011]. However, due to the slow convergence rate of MC methods, accuracy of the ROM is often poor when the number of snapshots is much smaller than that of unknown parameters. Further, increasing the number of snapshots increases computational cost and hence decreases the benefit of using a ROM.
 We are interested here in a special type of inverse problems. The underdetermined inverse problems have more unknowns (n) than measurements (m), and they are common in areas such as groundwater hydrology where measurements are sparse and unknowns are plenty due to multiscale heterogeneity of the subsurface porous medium. One can easily have more than a million unknowns to estimate for a 3-D groundwater flow and transport problem. However, as Kitanidis  points out, the estimate of unknown parameters can actually be constrained in an m + p dimensional subspace spanned by and X, where is the sensitivity of the measurements to unknowns for the full model. In his analysis for a nonlinear forward model, the parameter estimation at iteration i + 1 is written as
where the coefficients are found by solving the following equations system:
in which ; ; and H is the sensitivity matrix evaluated at si.
 As a result, we propose the use of for snapshot selection to construct the ROM, i.e., we use each column in as input parameters to run the full model and get a snapshot, totaling m + p snapshots. Then we follow the process introduced in section 2.1 to find the basis matrix P and hence construct the ROM. Ideally, if the forward model h(s) is linear, its output for any possible solution of the inverse model can be constrained in a subspace spanned by outputs using each of the columns in . Nonetheless, even for nonlinear forward models where H changes from one iteration to the next, the snapshots, and hence the matrix P, do not need to be updated for each iteration because (1) the observation error will most likely override the approximation error and (2) the change in the sensitivity matrix H and hence QHT between iterations may be small. Thus, P only needs to be constructed once and if necessary, updated every several iterations and whenever discrepancy between measurements and simulation remains large and cannot be reduced. This results in significant savings in computational cost because computing the sensitivity matrix H is extremely expensive and probably marks the bottleneck step of the entire inversion with the Newton's method. After all, for each step in Newton's method, it requires as many full model runs as the number of unknown parameters or the number of measurements, whichever is smaller. With the ROM, is much easier to compute and the solution of the inverse problem is now constrained in the subspace spanned by X and , which has a rank of .
 Similarly, in other geostatistical methods such as cokriging and the successive linear estimator (SLE) [Yeh et al., 1996], estimates of the unknowns are constrained in an m-dimensional subspace spanned by QHT (X. Liu et al., Fast iterative implementation of large-scale nonlinear geostatistical inverse modeling, submitted to Water Resources Research, 2013), and we can just use full model outputs with columns in QHT as parameter inputs to construct the basis matrix P. If a very short correlation length is used in the covariance function, the covariance matrix Q turns diagonal and the solution space turns HT, which is also the solution space for deterministic inverse methods such as the least squares method and the minimum length method [Menke, 2012]. It means that for these methods, we can also construct ROMs from the columns of HT and use them for the inversion.
 Realizing that QHT is simply the cross-covariance between measurements and unknowns, we name the ROM as the geostatistical reduced-order model (GROM). Further, we can also approximate it using analytical results such as those from Dagan  for randomly heterogeneous porous media. At least, such cheaply acquired splines can be used as a good starting point for model reduction.
 To summarize the process, a flowchart of inverse modeling with the GROM is presented in Figure 1 and explained below:
 (1) With an initial guess of the parameters s0, compute the sensitivity matrix H and the solution parameter space .
 (2) Scale each column of so that it represents a reasonable mean (e.g., the mean of s0) and variance of the parameters. There are at least two reasons for this step: (1) the values in each column of may be physically meaningless and unacceptable to the forward model; and (2) due to nonlinearity in the forward model h(s), the GROM's high accuracy is limited to a certain range of the parameters.
 (3) For each scaled column of , run the full forward model h(s) and put the output solution in the columns of the snapshot matrix S.
 (4) Find the nr leading eigen-pairs of SST; construct the basis matrix P and hence the GROM.
 (5) Solve the inverse problem with the GROM and get the estimated parameters sest.
 (6) If observation data are not fitted adequately with sest, go to step 1 with sest as the initial guess. Otherwise, end the inversion procedure. This step is optional because in most field applications, the observation error will be greater than the GROM's approximation error.
3. Numerical Examples With the GROM
 In this section we use a synthetic two-dimensional aquifer as shown in Figure 2. This aquifer has dimensions of 161 cm in length and 75.6 cm in height to resemble the structure in two laboratory-constructed sandboxes that will be introduced in the next section. Starting from a homogeneous ln(K) field (p = 1) for the construction of the GROM, we first test the accuracy of the GROM in approximating steady-state hydraulic heads for different heterogeneous ln(K) fields. For this aquifer, we first generate a synthetic Gaussian hydraulic conductivity field with the fast Fourier transform (FFT) [Dietrich and Newsam, 1997] and another one with a truncated Karhunen-Loève (KL) expansion, with which we produce numerical pumping test data to examine the performance of the GROM in geostatistical inverse modeling. Eight numerical pumping tests (2-D pumping rates range from 0.29 to 0.31 cm2/s) have been conducted in this aquifer at eight locations and pressure responses collected at all 48 ports (Figure 2) excluding the pumping port, which leads to steady-state hydraulic head measurements. We assume constant-head boundary conditions at the left, right, and top boundaries, and no-flow boundary conditions at the bottom. The flow domain is discretized into a grid, and the forward problem is solved with the finite-volume method.
3.1. Approximation Accuracy
 To see how well the GROM approximates the full model, we start from a homogeneous ln(K) field ( ) for the synthetic aquifer in Figure 2 and compute the sensitivity matrix H to get , for which we use an exponential covariance function for Q (variance = 3.0, correlation length = 50 and 10 cm in the horizontal and vertical directions, respectively). Each column of is then linearly scaled to have a mean value of −3.0 and a variance of 3.0. We then run the forward model for the eight pumping tests and get steady-state hydraulic head snapshots with ln(K) fields from each column. Then we construct the basis matrix P by retaining eigen-pairs of with the largest nr eigenvalues. We use in our study because it retains over 99.99% of the energy/expected variance of the empirical covariance matrix [Newman, 1996; Vermeulen et al., 2004] (Figure 3) and it provides accuracy high enough for inverse modeling. This represents a dimension reduction from 20,000 for the full model to 200 for the reduced model. We also refer the readers to Vermeulen et al.  for a more detailed discussion on the choice of nr. Then we sample 200 random K vectors in the range space of (random linear combination of columns in ) and plot a histogram of the normalized approximation error in Figure 4a. Surprisingly, given nonlinearity in model , we see that the GROM still well approximates the full model, and the mean value of the relative approximation error is about . Given that the pressure cm for all pumping tests, this corresponds to an absolute approximation error (i.e., ) of about .
 Similarly, in Figure 4b we show approximation error of h for random ln(K) vectors in the range space of with the GROM constructed from the analytical covariance function following results in Dagan [1985, equation ((4)d)]. To use this analytical result, we assume a constant pressure gradient in the entire sandbox, which is different from the flow field induced by pumping and used to compute the numerical covariance matrix QHT. However, the approximation accuracy is still good and the relative approximation error is about , which corresponds to an absolute approximation error of about .
 For comparison purposes, we plot in Figure 4c a histogram of the approximation error in y for vectors in with the ROM constructed with random Gaussian ln(K) fields from . Here and Q are the initial guess (−3.0, homogeneous) and prior exponential covariance matrix (variance = 3.0, homoscedastic) for parameters s, respectively. In Figure 4d, we also display a histogram of the approximation error for random Gaussian ln(K) fields in with the GROM constructed with vectors in . For both cases, the relative approximation error is about (or 0.38 for the absolute approximation error), which is also the level of accuracy we would expect for any ROMs constructed by Monte Carlo sampling in the domain .
 These figures show that statistically the accuracy of the GROM is better for ln(K) fields in the range space of than in , which is reasonable since the basis matrix P is constructed from K fields in the columns of . Furthermore, use of Monte Carlo sampling to construct the ROM leads to a large degradation of approximation accuracy for vectors in the solution space . The accuracy of the GROM for Gaussian ln(K) fields in is also low; however, as shown in the following numerical examples and applications, its impact on the efficacy of inverse modeling is insignificant because the ln(K) estimation is constrained in the subspace spanned by and accuracy of the GROM outside this subspace is not of a major concern. Furthermore, as demonstrated in the next section, the cross-covariance matrix, QHT changes slightly from the beginning of inversion with a homogeneous parameter field, to the end of inversion with a heterogeneous parameter field.
3.2. Estimation of Synthetic Parameter Fields
 For test purposes, we first generate a synthetic Gaussian ln(K) (K in cm/s) field with the Fast Fourier Transform (FFT) [Dietrich and Newsam, 1993] for the aquifer shown in Figure 2. We use an exponential covariance function and a constant mean value for the generated field. The generated ln(K) field has a mean value of −0.96 (or 0.38 cm/s for K) and a variance of 0.69. It is plotted in Figure 5a and it represents ln(K) values that we will estimate using steady-state pressure measurements. We use pressure responses at all ports except the pumping port during eight pumping tests for the inverse model. In total, measurements are used. Thus, this is a highly underdetermined inverse problem.
 In this application, we start from a homogeneous initial field ( ) and follow the same process as described above to construct the basis matrix P—each column of is linearly scaled with mean value −3.0 and variance 3.0—and the GROM. We then solve an inverse problem with the GROM as shown in equation (9). The result from inversion is presented in Figure 5c. It shows that with the GROM, the inverse model returns an estimated ln(K) field that is very similar to the true field. Although due to the limit in the number of measurements and the pressure smoothing effect caused by large hydraulic diffusivity, the algorithm is not able to resolve the small-scale variability of the randomly heterogeneous field shown in Figure 5a, we clearly see that all major high and low hydraulic conductivity zones are well captured in their locations, shapes, and values. Further, we notice that the accuracy of the GROM is still high even though the initial ln(K) value (−3.0) used to construct the GROM is different from the mean value of the true ln(K) field (−0.96). This point is also verified with all other GROM cases in the rest of this manuscript, in which we use the same initial ln(K) value to construct the GROM for true ln(K) fields with various mean values different from −3.0.
 Next we generate a relatively smoother synthetic Gaussian ln(K) field with truncated Karhunen-Loève (KL) expansion by keeping 50 eigen-pairs corresponding to the 50 largest eigenvalues of the prior covariance matrix. The realization has a mean ln(K) value of −1.43 (or 0.24 cm/s for K) and a variance of 0.83. It is plotted in Figure 5b and the same pumping tests with the same boundary conditions as in the previous numerical example are simulated to produce synthetic hydraulic head measurements. We also use as the initial guess, and the result of inverse modeling with the GROM is shown in Figure 5d. Again, we notice that the estimated ln(K) field well resembles the true one. Further, this estimation is better than the previous one (Figure 5c) due to the fact that both the ROM and the geostatistical inversion algorithm use smoothness assumptions [Newman, 1996; Liu and Kitanidis, 2011].
 In Figure 6, we display the fitting between the true and estimated ln(K) values, and that between the simulated and measured hydraulic head for the two cases. It confirms that even with approximation errors in the GROM, the inverse model is still able to estimate parameters and fit measurement data with high accuracy.
 For comparison purposes, we also use the full model to estimate the ln(K) fields with the same set of data, and the results are presented in Figures 5e, 5f, and 7. We see that although the estimation is slightly better than that with the GROM, the difference is almost unidentifiable in capturing major features of the true ln(K) fields as well as in fitting the data.
 The high accuracy of GROM for nonlinear inverse problems can be partly explained by the small change in the cross-covariance matrix QHT from the beginning of inversion with a homogeneous ln(K) field, to the end of inversion with the estimated heterogeneous ln(K) field. In Figure 8, we show cross-covariance maps between ln(K) and a few selected measurements. The estimated heterogeneous ln(K) field is for the true ln(K) field generated with FFT (Figures 5a and 5c). It is clear that although heterogeneity affects H and hence QHT, the major spatial patterns in QHT remain the same. We also notice the difference in the actual cross-covariance values shown in the maps, which is due to the difference between the homogeneous ln(K) value (−3.0) and the mean value of the estimated ln(K) field (−0.94).
 In this section, we use hydraulic data from real laboratory sandbox experiments to test the performance of the GROM under more realistic situations. We apply the model reduction methodology developed above to steady-state hydraulic tomography for hydraulic experiments in two laboratory-constructed sandboxes. As shown in Figures 9a and 9b, the sandboxes are of the same dimension (193 cm (L) × 82.6 cm (H) × 10.2 cm (D)) but different sand structures. Dimensions of the sand body in the sandbox are the same as the synthetic aquifer shown in Figure 2. Locations of observation ports and pumping ports at the back of the sandbox are also shown in Figure 2. The first sandbox—Sandbox 1 (Figure 9a)—has rectangular blocks manually packed with sand finer than the background sand. The laboratory-tested hydraulic conductivity values with permeameter tests range from 0.016 to 0.26 cm/s. Similar to numerical examples in the previous section, eight pumping tests have been conducted in this sandbox at eight locations, and pressure responses have been collected at all 48 ports (Figure 2) excluding the pumping port. Water tanks at both sides of the sandbox are used to keep constant-head boundary conditions at the left, right, and top boundaries during pumping tests. At the bottom, we have no-flow boundary conditions. Both steady-state [Illman et al., 2007, 2008; Liu and Kitanidis, 2011] and transient [Liu et al., 2007; Yin and Illman, 2009] pressure records have been used for hydraulic tomography to estimate the spatially varying hydraulic conductivity field. The other sandbox—Sandbox 2 (Figure 9b)—has been constructed by sediment transport in the sandbox, and similar pumping tests have been conducted in it for steady-state [Illman et al., 2010, 2012] and transient [Berg and Illman, 2011] hydraulic tomography. For more details on these two sandboxes, please refer to Illman et al. [2007, 2010]. In this work, we use steady-state hydraulic head measurements collected during the eight pumping tests to estimate 20,000 ln(K) values on a grid.
4.1. Characterization of Laboratory Sandboxes
 With the same process as in the numerical examples, we apply the model reduction methodology to steady-state hydraulic tomography in the two laboratory-constructed sandboxes. Results are shown in Figures 9c and 9d, and an overlay view of the tomogram and the sandbox photograph is presented in Figures 9e and 9f. As we can see, in both cases, the tomogram well replicates the sand block distribution in the sandbox, and major patterns of the heterogeneous aquifers are well captured.
 In Figure 10, we display the fitting between simulated and measured hydraulic head for the two sandboxes. It confirms that even with measurement noise, geostatistical error due to the non-Gaussian true parameter field, and approximation error in the GROM, the inverse model is still able to fit measurement data with high accuracy.
4.2. Use GROM-Based Estimation as Initial Guess
 Due to nonlinearity in the model h(s), the accuracy of the GROM cannot be infinitely high, and it affects the estimation accuracy of the inverse model. Thus, if better estimation results are still expected and required, the cheaply acquired result from inverse modeling with the GROM can be used as an initial guess for inverse modeling with the full model (maximizing equation (9) with replaced by h). We use this strategy for the problem in Sandboxes 1 and 2, and the estimated ln(K) fields are presented in Figures 9g and 9h.
 The fitting between measured and simulated hydraulic head with the full model is also shown in Figure 10. We can see that the level of fitting is very similar with the full model and with the GROM. In fact, a slightly better fitting is acquired with the GROM for Sandbox 1. This is due to the fact that the approximation error of the GROM is very small and smaller than the noise in measured hydraulic head data, so both the full model and the GROM fit the data equally well.
4.3. Computational Savings
 We run all simulations in Matlab R2012b on a Windows 7 PC without any parallelization schemes. The full forward problem is discretized with the finite-volume method and solved with the preconditioned conjugate gradient method (Matlab built-in function pcg). However, the reduced system in equation (7) is solved with a direct method (LDL decomposition) as it is dense but small (Matlab built-in left-division operator). We use rectangular discretization of the simulation domain (even spacing in each direction) and hence the FFT-based method for multiplication with the covariance matrix [Nowak et al., 2003]. Line search is inserted between two linearizations to secure convergence [Liu and Kitanidis, 2011]. As reported in Table 1, the total simulation time for the two synthetic inverse problems is 221 s (FFT) and 250 s (KL), respectively, with the GROM; and 390 s (FFT) and 544 s (KL), respectively, with the full model. For Sandboxes 1 and 2, the total simulation time is 259 and 256 s, respectively, with the GROM; and 553 and 1856 s, respectively, with the full model. In general, the saving in CPU time ranges from 43% to 86%.
Table 1. CPU Time and Numbers of Iterations
CPU Time (s)
Full model after GROM
Full model after GROM
Full model after GROM
Full model after GROM
 To break down the CPU time to parts, we take the inverse problem with GROM for Sandbox 1 as an example. Constructing the GROM takes about 63% of the CPU time (21% on computing and 42% on computing snapshots and the basis matrix P); evaluating the sensitivity matrix of the GROM takes 11% of the time; forming and solving the linearized geostatistical inverse system takes 9% of the time; and line search takes 16% of the time. Similar breakdown of CPU time is also reported for Sandbox 2 and the two synthetic cases.
 As pointed out in Cirpka and Kitanidis [2000b], the computational cost for inverse problems is dominated by repetitive simulation of the full forward model, which is particularly true for large-scale problems. For inversion with the full model, the total number of full model simulations depends on the number of iterations or Newton steps to reach convergence. The CPU cost of constructing the GROM is equivalent to that of two iterations with the full model. For Sandboxes 1 and 2, respectively, 6 and 16 iterations are needed to converge for the inverse problem with the full model, so the saving in CPU time is between 67% and 88%.
 When the result from GROM-based inverse modeling is used as an initial guess of parameter values to inverse modeling with the full model, the number of iterations needed for convergence is reduced to 4 and 12 for Sandboxes 1 and 2, respectively; thus, the total CPU cost is still no more than that of the original inverse model starting from a homogeneous initial guess. Furthermore, as demonstrated in the numerical examples and applications, this step is usually unnecessary because the GROM is accurate enough to produce desirable results of inversion.
5. Discussion and Conclusions
 In this paper, we develop geostatistical reduced-order models (GROMs) in the parameter domain to solve underdetermined inverse problems. We use the solution parameter space of underdetermined geostatistical inverse problems to take snapshots for constructing the GROM. As the solution parameter space is mapped by the cross-covariance matrix between the parameters and measurements, we avoid the use of statistical sampling of the high-dimensional parameter space. Further, the solution parameter space of underdetermined inverse problems has a low dimension so that only a few snapshots are needed for the GROM to reach high approximation accuracy for any parameters in the solution space.
 We then show with synthetic and laboratory-collected hydraulic head data that the accuracy of the GROM is very high in the solution space of geostatistical inverse problems, and inverse modeling with the GROM presents results that are very close to those using the full forward model. However, the computational cost is significantly lower with the GROM in both CPU time and the total number of full forward model simulations.
 The fact that the GROM only needs to be constructed once from the initial (usually homogeneous) parameter field may lead to even greater computational cost savings in uncertainty quantification when a number of realizations are generated by running the inverse model for multiple times, as it means that for all realizations except the first one, we only need the GROM and do not need the full model at all. Nonetheless, we shall make no claim that this is true for all cases. In fact, the GROM suffers from the forward problem's nonlinearity the same way as all other ROMs do. This is because that the Galerkin projection is a linear projection while the forward problem is nonlinear. The quantification of GROM's approximation error caused by nonlinearity and its inclusion in an inverse model is still a valuable topic for future research. However, at this point, we recommend an empirical way of updating the GROM when the fitting error between the observed and predicted data remains large and cannot be improved with iterations. Practically, the error of the GROM should be kept at the same order of magnitude as the observation error [Tarantola, 2004, p. 21].
 The GROM also has great potential use in MCMC sampling. We can use the GROM to construct a surrogate model to filter out those realization candidates that are not likely to be accepted by the distribution function with the full model. Such application can increase the acceptance rate of MCMC sampling with the full model to a significantly high level and hence largely reduce the computational cost.
 The simultaneous model reduction in parameter, time, and frequency domains for inverse transient problems is also of great interests. Ideally, we will be able to get an even larger amount of savings in computational cost when these model reduction strategies are combined.
 This work was funded by the Assistant Secretary for Fossil Energy, National Energy Technology Laboratory (NETL) of the U.S. Department of Energy under contract DE-AC02-05CH11231. Additional funding was provided by the Earth Sciences Division of Lawrence Berkeley National Laboratory through Early Career Development grants. We also thank Wolfgang Nowak, Arvind Saibaba, and the other anonymous reviewer for their valuable comments and suggestions.