#### 3.1. Pattern Error

[19] To address the questions raised in the last section and generalize the decomposition of pattern errors to vectors, we start from the matrix identity

where cov(**F**, **O**) = 〈(**F** − 〈**F**〉)(**O** − 〈**O**〉)^{⊺}〉 is the covariance matrix [*Feller*, 1968] and var(**A**) ≡ cov(**A**, **A**), where **A** = **D**, **O**, **F**. Scalar variables are one-dimensional vectors, which means the matrices reduce to the familiar scalar quantities: var(*D*), var(*O*), var(*F*) and cov(*F*, *O*). For a vector variable, the trace of the covariance matrix is the sum of its diagonal elements and so,

By taking the trace of equation (13) and normalizing by (*σ*_{O}^{2} + *σ*_{F}^{2}) instead of just *σ*_{O}^{2}, we have

where

[20] *α* is the normalized error variance, first defined in *Koh and Ng* [2009] for vector and scalar variables. There, the diagnostic was shown not to depend on observation variability, thus yielding insights into the forecast errors in a mesoscale model.

[21] *ρ* is a natural generalization of the correlation coefficient for vector variables, already defined by *Dietzius* [1916]. Other definitions of vector correlation also exist [e.g., *Court*, 1958; *Crosby et al.*, 1993], but *Aparna et al.* [2005] in evaluating the modeling of sea breezes found the correlation diagnostic based on Dietzius' definition to be least noisy.

[22] *η* is a new measure called “variance similarity” or “similarity” for short. It is defined here as the ratio of the geometric mean to the arithmetic mean of *σ*_{O}^{2} and *σ*_{F}^{2}. Like _{F}employed implicitly in the non-dimensional Taylor diagram, the variance similarity*η* compares the standard deviations of the forecast and observation. But unlike _{F}, the variance similarity *η* has the following advantages:

[23] 1. It is invariant under the exchange of observation and forecast. Since this means *η* is equivalent for reciprocal values of *σ*_{F}/*σ*_{O} (i.e. *x* and 1/*x*, where *x* ∈ ^{+}), the under- and over-prediction of atmospheric variability are penalized equally.

[24] 2. It increases monotonically as the variance of observation and forecast approach each other. *η* ≪ 1 indicates that the observation and forecast fluctuate with very different amplitudes; *η* = 1 denotes matching standard deviation of variations, *σ*_{F} = *σ*_{O}.

[25] In fact, all three metrics, *α*, *ρ* and *η*, are non-dimensional, symmetric with respect to the observation and forecast, and valid for both scalar and vector variables.

[26] Motivated by the analogy to the cosine rule in *Taylor* [2001], the relation in equation (16) can be visualized geometrically, by taking either *ρ* or *η* as the cosine of an angular coordinate leaving the other as the radial coordinate. Here, we define the angular coordinate *ϕ* as

*ϕ* is 0 when forecast and observation vary to the same extent; *ϕ* is *π*/2 when the observation or forecast is constant. For *σ*_{F} ≈ *σ*_{O}, *ϕ* is approximately equal to the fractional difference between *σ*_{F} and *σ*_{O} (Appendix A). Equation (16) becomes

Figure 2, described in detail by its caption, shows how *α*, *ρ* and *ϕ*are related geometrically, and shall henceforth be called the “correlation-similarity” diagram.

[28] Equation (16)resolves non-zero*α* into contributions from the disagreement in phase (*ρ* < 1) and in amplitude (*η*< 1) between observation and forecast variations, which is visualized geometrically on the correlation-similarity diagram: the (*ρ*, arccos *η*)-plane is divided up by isolines of*ρ*/*η.* As *ρ*/*η* = *y* *ρ* = *y* cos *ϕ*, isolines of *ρ*/*η* = *y* are circles of diameter *y* as shown in Figure 2. Models that lie within (outside) the *ρ*/*η* = 1 circle are dominated by phase (amplitude) errors and the smaller (larger) the value of *ρ*/*η*, the greater is the relative contribution from phase (amplitude) errors.

[29] In the correlation-similarity diagram,*α* is denoted by the vertical distance. At the bottom, *α* = 0 denotes no pattern error when forecast and observation variability match in both amplitude and phase. In the middle, *α* = 1 denotes that the model is only as well as a random forecast made from knowing the climatological mean and having equal chance of predicting an anomaly in one direction or the opposite, i.e. cov(**F**, **O**) = 0 (cf. equations (13) and (17)). At the top, *α*= 2 denotes maximum pattern error when forecast and observation vary with the same amplitude but are exactly out-of-phase. Therefore,*α* calibrates the pattern error made by a model against that made by the mentioned random forecast.

[30] With reference to Figure 2, for fixed non-zero*η*, *α* is minimized by increasing correlation *ρ.* For fixed positive *ρ*, minimizing *α* implies maximizing *η* which means the forecast variability approaches the observed variability (equation (16)). For fixed negative *ρ*, minimizing *α* requires minimizing *η*, i.e. increasing disagreement in the amplitude of the variations, which may sound perplexing at first. But actually, when the forecast variations tend to oppose the observed variations (and hence exhibiting negative correlation), it makes sense for the least disagreement to be achieved by either forecast or observation being nearly constant. Nonetheless, *η* 0 also corresponds to either *σ*_{F} ∞ or *σ*_{O} ∞. This would mean *α* is minimized simply because the normalization reference tends to infinity. But this is a common problem for normalized measures. For example, _{D} is also minimized in regions where the observed variability *σ*_{O} is very large for negative *ρ* (cf. Figure 1). Luckily, negative correlations usually indicate some severe problem in the underlying physics or dynamics and so models in practical use seldom fall into this category.

[31] As for *η* = 0, *α* is independent of *ρ*, which is desirable because when either forecast or observation is constant, *ρ* is undefined. In fact, *α* can be proven from first principles to be unity in this case. On the other hand, for *ρ* = 0, *η* is well-defined, but *α* is independent of *η* and equals to one. The model performs only as good as a random forecast, but even so, one may reasonably expect to distinguish among those forecasts that rightly reproduce the observed amplitude of variations, those that do not, and those that are constant. So, although *α* is a good measure of the pattern error against the random forecast, it is not a good skill score to use when the model performs similarly to a random forecast, i.e. when *α* ≈ 1. But in such cases, *η* itself clearly provides the necessary supplementary information and so one might be motivated to design a simple skill score:

[32] Now, *α** is in no way unique: it is similar to the skill scores already proposed in section 5 of *Taylor* [2001]. In contrast, the definition of *α* is special in the sense that it arises naturally from the symmetric normalization of the centered MSE and results in a simple relation for pattern, phase and amplitude errors (equation (16)). Thus, we focus on the separate use of *α* and *η* in the present work and refer the interested reader to *Taylor* [2001] for a discussion on designed skill scores similar to *α**.

#### 3.2. Bias

[33] The correlation-similarity diagram only represents the pattern error and its decomposition into phase and amplitude errors. Thus, there is still a need to incorporate the bias into the diagnostic framework. The generalization ofequation (5) for vectors is the matrix equation

By taking the trace, we have

[34] We divide equation (24) by (*σ*_{O}^{2} + *σ*_{F}^{2}) to remove the dependence on observation variability while preserving the symmetry under the exchange of forecast and observation. Using equation (17) and with some rearrangement and substitutions,

where

[35] *δ* is the normalized RMSE (NRMSE). *σ* is the normalized pattern error (NPE), as it arises from the normalization of the standard deviation of the error fluctuations. *μ* is the normalized bias (NBias). The use of *σ*_{D} to standardize the bias 〈**D**〉 is recommended when the two populations are not independent [*Rosenthal*, 1991]. The magnitude of NBias is also the paired t-statistic multiplied by , where *N* is the number of degrees of freedom in the data sample. Despite *N* being unknown because of spatiotemporal correlation among data values, NBias is a direct measure of statistical significance: larger NBias implies more significant bias. So a comparison of NBias between two regions is also a comparison of the statistical significance of model bias in the two regions.

[36] To present the diagnostics on a polar plot, we can define an angle *γ* based on the NBias by

Note that for small bias, tan *γ* ≈ *γ* and so *γ* approximates NBias. Equation (25) can be rearranged as

For the magnitude of NBias less than 0.5, the bias makes negligible contribution to the total error (i.e. RMSE) since for |tan*γ*| =≲ 0.5, cos *γ* ≳ 0.9 and *δ* ≈ *σ.* Figure 3 illustrates the geometric relation between *σ*, *δ* and *γ* embodied in equation (30). It shows the decomposition of the total error into bias and pattern error contributions in a non-dimensionalized manner and will henceforth be called the “error decomposition” diagram. For a scalar variable, positive and negative bias are distinguished by plotting in the right and left quadrant respectively. For a vector variable, only the right quadrant is used (which could be interpreted as the 2D meridional projection of a 3D hemispherical plot in which the azimuthal angle denotes the direction of the vector bias).

#### 3.3. Vector Variables

[37] For a vector variable **A**, *σ*_{A}^{2} ≡ tr[var(**A**)] does not capture all independent pieces of information in var(**A**). For a two-dimensional vector like horizontal wind (*u*, *v*), there are three independent pieces of information pertaining to variance, exemplified by var(*u*), var(*v*) and cov(*u*, *v*). *Koh and Ng* [2009] proposed the use of an ellipse to capture all this information. Figure 4 (reproduced from *Koh and Ng* [2009]) shows such an ellipse, where *a* and *b* are respectively the square roots of the larger and smaller eigenvalues of var(**A**). (Such representation can be generalized to an *n*-ellipsoid for*n*-dimensional vectors.) Taking the vector **A** as the error **D** of some two-dimensional vector variable, the centered MSE,*σ*_{D}^{2} = *a*^{2} + *b*^{2}, would be a measure of the size of the “error ellipse”.

[38] Suppose the eigenvector with eigenvalue *a*^{2} makes a clockwise angle of *θ* with the vertical axis (Figure 4), where 0 ≤ *θ* < *π.* Using the horizontal and vertical unit vectors as the basis, Appendix B shows that

where *ε*_{s} is a symmetrized measure of eccentricity of the ellipse, defined as:

For *ε*_{s} ≪ 1, *ε*_{s} is the fractional difference between *a* and *b* (proven in Appendix B).

[39] Evidently from equation (31), besides *σ*_{D}, two other diagnostics are necessary to complete the description of vector pattern errors: *ε*_{s} and *θ* which measure respectively the extent of anisotropy and the preferred direction of the vector errors. Both diagnostics can be displayed in another polar plot, the “error anisotropy” diagram (Figure 5) in which the radial distance is *ε*_{s} and the polar angle is 2*θ*, reflecting the order-2 rotational symmetry of the orientation of the error ellipse.*ε*_{s} = 0 corresponds to isotropy while *ε*_{s} = 1 corresponds to maximal anisotropy (i.e. the vector errors align in a straight line). Note that *ε*_{s} and *θ* characterize the vector pattern error but are not errors per se.