SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Model and inequality constrained hypotheses
  5. 3. Bayes factors and prior probabilities
  6. 4. Issues with prior specification
  7. 5 Alternative prior specification methods
  8. 6. Simulation study
  9. 7. Concluding remarks
  10. Acknowledgements
  11. References
  12. Appendix A: Derivation of the Bayes factor
  13. Appendix B: Bayes factor based on inline image
  14. Appendix C: Generalized inverse
  15. Appendix D: Information paradox
  16. Appendix E: Intrinsic Bayes factors

Several issues are discussed when testing inequality constrained hypotheses using a Bayesian approach. First, the complexity (or size) of the inequality constrained parameter spaces can be ignored. This is the case when using the posterior probability that the inequality constraints of a hypothesis hold, Bayes factors based on non-informative improper priors, and partial Bayes factors based on posterior priors. Second, the Bayes factor may not be invariant for linear one-to-one transformations of the data. This can be observed when using balanced priors which are centred on the boundary of the constrained parameter space with a diagonal covariance structure. Third, the information paradox can be observed. When testing inequality constrained hypotheses, the information paradox occurs when the Bayes factor of an inequality constrained hypothesis against its complement converges to a constant as the evidence for the first hypothesis accumulates while keeping the sample size fixed. This paradox occurs when using Zellner's g prior as a result of too much prior shrinkage. Therefore, two new methods are proposed that avoid these issues. First, partial Bayes factors are proposed based on transformed minimal training samples. These training samples result in posterior priors that are centred on the boundary of the constrained parameter space with the same covariance structure as in the sample. Second, a g prior approach is proposed by letting g go to infinity. This is possible because the Jeffreys–Lindley paradox is not an issue when testing inequality constrained hypotheses. A simulation study indicated that the Bayes factor based on this g prior approach converges fastest to the true inequality constrained hypothesis.

1. Introduction

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Model and inequality constrained hypotheses
  5. 3. Bayes factors and prior probabilities
  6. 4. Issues with prior specification
  7. 5 Alternative prior specification methods
  8. 6. Simulation study
  9. 7. Concluding remarks
  10. Acknowledgements
  11. References
  12. Appendix A: Derivation of the Bayes factor
  13. Appendix B: Bayes factor based on inline image
  14. Appendix C: Generalized inverse
  15. Appendix D: Information paradox
  16. Appendix E: Intrinsic Bayes factors

In past decades there has been considerable development of the evaluation of hypotheses containing inequality constraints. A practical advantage for this development is that applied researchers often formulate their expectations using inequality constraints. In a repeated measures model, for instance, one may expect that the first measurement mean is smaller than the second measurement mean, the second measurement mean is smaller than the third measurement mean, and the third measurement mean is smaller than the fourth measurement mean: inline image. Subsequently, when evaluating an inequality constrained hypothesis, researchers obtain a direct answer to the question to what degree their expectations are supported by the data. Testing inequality constrained hypotheses also has a technical advantage because they result in more statistical power. The simplest example is that the classical one-sided p-value of the standard null hypothesis against a one-sided alternative is half the size of the two-sided p-value when the one-sided alternative is supported by the data. See Barlow, Bartholomew, Bremner, and Brunk (1972), Robertson, Wright, and Dykstra (1988), and Silvapulle and Sen (2004) for an overview of classical methods for testing inequality constrained hypotheses. For recent developments on this topic using Akaike-type information criteria and Bayesian selection criteria, see Hoijtink, Klugkist, and Boelen (2008) and Hoijtink (2011), and the references therein.

In this paper, a Bayesian approach will be considered. A straightforward method for evaluating inequality constrained hypotheses in a Bayesian framework is by computing the posterior probabilities that the inequality constraints hold. This approach, however, may not result in desirable testing behaviour. For example, let us consider the repeated measures model with four measurements where the interest is in testing inline image (i.e., the means monotonically increase) versus inline image (i.e., the means do not monotonically increase). Now let us consider a hypothetical data set with 10 observations with sample means of inline image, and a maximum likelihood (ML) estimate of the covariance matrix of inline image; this corresponds to an effect size of inline image, which implies a ‘small’ to ‘medium’ effect (Cohen, 1988). The posterior probabilities that the inequality constraints of inline image and inline image hold are given by inline image and inline image, respectively. These probabilities can simply be estimated as the proportion of posterior draws satisfying the inequality constraints. Hence, inline image receives more posterior evidence than inline image based on these posterior probabilities. Because inline image is supported by the data with a small to medium effect, testing hypotheses based on the posterior probabilities may not be preferable.

This result can be explained by the fact that the difference in complexity (or parsimony) of inline image and inline image is ignored when computing posterior probabilities that the inequality constraints hold. For this reason, we shall focus on the Bayes factor (Jeffreys, 1961), which is known to balance between fit and complexity as an “Ockham's razor” (e.g., Kass & Raftery, 1995). In the Bayes factor, the ‘relative complexity’ of an inequality constrained space under inline image is quantified through the prior probability that the inequality constraints hold, and the ‘relative fit’ is quantified through the posterior probability that the inequality constraints hold (both relative to the unconstrained space).

In the Bayes factor, prior specification plays a crucial role when testing a null hypothesis with equality constraints on the parameters due to the Jeffreys–Lindley paradox (Lindley, 1957; Bartlett, 1957; Jeffreys, 1961). The paradox implies that the evidence for the null hypothesis can become arbitrarily large when specifying the prior variance large enough. When testing inequality constrained hypotheses however, the Jeffreys–Lindley paradox does not play a role because the prior probability that the inequality constraints hold is not affected by the prior variance when the unconstrained prior is centred on the boundary of the constrained space (Klugkist & Hoijtink, 2007). Therefore, Bayes factors are well suited for testing inequality constrained hypotheses. However, three issues must be taken into account when using the Bayes factor for this purpose.

First, the complexity of the inequality constrained parameter spaces can be ignored. As will be shown in this paper, Bayes factors based on non-informative improper priors and partial Bayes factors based on posterior priors (O'Hagan, 1995; Berger & Pericchi, 1996, 2004) ignore the complexity when testing inequality constrained hypotheses.

Second, Bayes factors may not be invariant for linear one-to-one transformations of the data. This is the case, for instance, when using the constrained posterior priors of Mulder, Hoijtink, and Klugkist (2010), which can be seen as a multivariate generalization of Jeffreys' (1961) notion of balanced priors. Mulder et al. proposed an automatic prior specification method that resulted in priors that were located on the boundary of the constrained parameter space with diagonal covariance structure. One of the important implications of the proposed methodology was that every ordering of the parameters was equally likely a priori.

Although this idea of balancedness seems intuitively appealing, a problem is that the prior probability that the inequality constraints hold using a balanced prior (e.g., inline image and inline image under inline image and inline image, respectively, in the example above) depends on the parameterization of the model. As will be shown, when formulating the inequality constraints on the mean differences between subsequent measurements instead of the actual measurement means, the resulting Bayes factor may result in violations of linear invariance.

Third, the information paradox can be observed. This paradox has been investigated when testing the standard null hypothesis (an equality constrained hypothesis); see Zellner (1986), Berger and Pericchi (2001), and Liang, Paulo, Molina, Clyde, and Berger (2008). The paradox states that the Bayes factor for an unconstrained hypothesis against the null hypothesis does not converge to infinity as the evidence against the null hypothesis accumulates, for fixed sample size. In this paper, we present an equivalent formulation of the paradox when testing inequality constrained hypotheses. Similarly to when testing equality constrained hypotheses, the information paradox can also be observed when testing inequality constrained hypotheses when using Zellner's (1986) g prior.

For this reason, two methods are developed that avoid the three issues mentioned above. In the first method, minimal training samples are transformed such that the resulting posterior priors are located on the boundary of the constrained space with identical covariance structure to that of the original posterior prior. The second approach is based on the g prior, where g goes to infinity. This is allowed because the Jeffreys–Lindley paradox is not an issue when testing inequality constrained hypotheses. The resulting Bayes factor is relatively simple to compute and avoids the information paradox.

The paper is organized as follows. First, the multivariate normal linear model and the inequality constrained hypotheses are formulated in Section 2. Section 3 describes the Bayes factor when testing inequality constrained hypotheses, including a short discussion of the choice of prior probabilities for the hypotheses. In Section 4 the three issues are discussed in detail in relation to posterior priors, balanced priors, and the g prior. Section 5 describes two new Bayesian methods for testing inequality constrained hypotheses. A simulation study is discussed in Section 6. Some concluding remarks are made in Section 7.

2. Model and inequality constrained hypotheses

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Model and inequality constrained hypotheses
  5. 3. Bayes factors and prior probabilities
  6. 4. Issues with prior specification
  7. 5 Alternative prior specification methods
  8. 6. Simulation study
  9. 7. Concluding remarks
  10. Acknowledgements
  11. References
  12. Appendix A: Derivation of the Bayes factor
  13. Appendix B: Bayes factor based on inline image
  14. Appendix C: Generalized inverse
  15. Appendix D: Information paradox
  16. Appendix E: Intrinsic Bayes factors

Models will be considered that belong to the multivariate normal linear model, where a p-dimensional dependent variable inline image depends on a k × p mean parameter matrix Θ via k explanatory variables inline image and error covariance matrix Σ according to

  • display math(1)

for i = 1,…,n, so that Y =  + E, where inline image, inline image, and inline image, with inline image. Under this model, the non-informative improper Jeffrey's prior is given by inline image, resulting in the posterior

  • display math(2)

where N(θ;m,K) denotes the multivariate normal density with mean m and covariance matrix K, inline image denotes the inverse Wishart density with scale matrix C and degrees of freedom ν, θ is the vectorization of Θ, that is, θ = vec(Θ), the ML estimate is given by inline image, and the sum of squares equals inline image. Furthermore, the marginal posterior of Θ has a matrix t distribution given by inline image.

Throughout this paper, testing problems are considered between a finite number of T inequality constrained hypotheses inline image on the mean parameters, where inline image or inline image with inline image. Furthermore, it is assumed that a solution exists to the system of equations inline image, where inline image and inline image. This implies that the inequality constrained parameter spaces share a common boundary. Furthermore, we shall refer to the hypothesis with no constraints on θ as the unconstrained hypotheses, denoted by inline image. Note here that the likelihood function under inline image, denoted by inline image, is a truncation of the likelihood function under inline image, that is, inline image, where inline image is the identity function. Therefore, the hypothesis index will be omitted in the likelihood h and parameters θ and Σ when possible in order to simplify the notation.

3. Bayes factors and prior probabilities

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Model and inequality constrained hypotheses
  5. 3. Bayes factors and prior probabilities
  6. 4. Issues with prior specification
  7. 5 Alternative prior specification methods
  8. 6. Simulation study
  9. 7. Concluding remarks
  10. Acknowledgements
  11. References
  12. Appendix A: Derivation of the Bayes factor
  13. Appendix B: Bayes factor based on inline image
  14. Appendix C: Generalized inverse
  15. Appendix D: Information paradox
  16. Appendix E: Intrinsic Bayes factors

A natural choice for the prior under an inequality constrained hypothesis inline image is to take the truncation of the unconstrained prior inline image under inline image (Berger & Mortera, 1999; Klugkist, Laudy, & Hoijtink, 2005), that is, inline image where the prior probability inline image is a normalization constant. Consequently, the Bayes factor of inline image against inline image can be expressed as

  • display math(3)

(see Appendix A). Hence, the Bayes factor is equal to the ratio of the posterior probability that the inequality constraints hold under inline image, which serves as a measure of ‘relative fit’ of inline image, and the prior probability that the inequality constraints hold under inline image, which serves as a measure of ‘relative complexity’ or size of inline image (both relative to inline image).

It is important to note that the prior probabilities that the inequality constraints hold under inline image, inline image, are not equal (or proportional) to the prior hypothesis probabilities inline image. The inline image must be chosen separately. We recommend using equal prior probabilities for inline image, for t = 1,…,T, that is, inline image, as a default choice (unless there is specific prior knowledge). If we use the prior probabilities that the inequality constraints hold as prior probabilities for the hypotheses, that is, inline image, the posterior probabilities of the hypotheses become equal to (or proportional to, in the case of more than two hypotheses) the posterior probabilities that the inequality constraints hold. For example, when testing inline image versus inline image using the unconstrained prior inline image, so that inline image and inline image, the posterior hypothesis probabilities become

  • display math

Hence, the relative complexity measures of the inequality constrained hypotheses, inline image, are cancelled out so that the resulting posterior hypothesis probabilities become equal to the posterior probabilities that the inequality constraints hold, inline image. This would result in a bias towards the ‘larger’ hypothesis inline image, for p > 2. In the remainder of this paper, posterior hypothesis probabilities and prior hypothesis probabilities will not be further discussed. Therefore, when discussing the posterior (or prior) probability that the inequality constraints of inline image hold under inline image, we refer to inline image (or inline image).

4. Issues with prior specification

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Model and inequality constrained hypotheses
  5. 3. Bayes factors and prior probabilities
  6. 4. Issues with prior specification
  7. 5 Alternative prior specification methods
  8. 6. Simulation study
  9. 7. Concluding remarks
  10. Acknowledgements
  11. References
  12. Appendix A: Derivation of the Bayes factor
  13. Appendix B: Bayes factor based on inline image
  14. Appendix C: Generalized inverse
  15. Appendix D: Information paradox
  16. Appendix E: Intrinsic Bayes factors

4.1 Posterior priors: Ignoring relative complexity

A commonly used method to avoid ad hoc or subjective prior specification is by using a training sample, which is a subset of the data that results in a posterior prior (e.g., Lempers, 1972; O'Hagan, 1995; Berger & Pericchi, 1996). The remainder of the data is then used for hypothesis testing. Generally it is recommended to use minimal training samples which result in proper posterior priors that contain minimal information, so that maximal information is used for hypothesis testing.

In the multivariate normal linear model, a minimal training sample inline image consists of inline image random observations from [Y|X], where inline image has rank k. The training sample is used to update the non-informative improper Jeffreys prior inline image to obtain a posterior prior inline image given by (2) where the ML estimate inline image, the sum of squares inline image, inline image, and inline image are substituted for inline image, S, X, and n, respectively. The posterior prior under inline image is then a truncation of the unconstrained posterior prior, that is, inline image, where the posterior prior that the inequality constraints hold, inline image, is a normalizing constant. The partial Bayes factor then yields (Berger & Pericchi, 1996)

  • display math(4)

where inline image denotes the remainder of the data after removing inline image from Y and inline image refers to the Bayes factor based on the non-informative improper Jeffreys prior inline image.

Lemma 1. The Bayes factor of an inequality constrained hypothesis inline image against the unconstrained hypothesis inline image using non-informative improper priors is equal to the posterior probability that the inequality constraints hold under inline image, that is, inline image.

Proof. See Appendix B.

Consequently, expression (4) becomes inline image. As the evidence for an inequality constrained hypothesis inline image accumulates in the sense that each element of inline image and inline image goes to infinity, then inline image, so that the partial Bayes factor converges to 1. This illustrates that the posterior prior probability that the inequality constraints hold, inline image, does not reflect the complexity of an inequality constrained space. Consequently, the resulting partial Bayes factor ignores the difference in complexity between inline image and inline image when the evidence for inline image accumulates.

4.2 Balanced priors: Violation of linear invariance

Mulder, Hoijtink, and Klugkist (2010) proposed the following quantification of the complexity of an inequality constrained hypothesis.

Definition 1. The uniform complexity of an inequality constrained hypothesis inline image, with inline image, is defined as

  • display math(5)

where the multivariate uniform density is defined as

  • display math(6)

For example, the uniform complexity of an order-constrained hypothesis inline image yields inline image, and the uniform complexity of a p-dimensional one-sided hypothesis inline image equals inline image. Consequently, an unconstrained prior under inline image was referred to as ‘balanced’ (as a multivariate generalization of Jeffreys' (1961) principle), if the prior probability that the inequality constraints hold under inline image is equal to the uniform complexity of inline image, that is, if

  • display math(7)

holds for all t = 1,…,T. Based on this property, a typical property of a balanced prior for θ is that it is centred on the boundary of the inequality constrained space with a diagonal covariance structure. Note that posterior priors do not satisfy (7).

Automatic prior specification methods have been proposed that satisfy (7) where the prior variances were based on minimal training samples, which were referred to as constrained posterior priors (Mulder, Klugkist, van de Schoot, Meeus, Selfhout & Hoijtink, 2009; Mulder et al., 2010; Mulder, Hoijtink, & de Leeuw, 2012). It has been shown that the resulting Bayes factors are consistent and have good frequency properties for a broad class of hypotheses with equality constraints and inequality constraints (e.g., Hoijtink, 2011). However, balanced priors do not result in Bayes factors that are invariant for linear transformations of the data because uniform complexity inline image depends on the parameterization of the model.

Example 1. Let inline image and consider an order constrained hypothesis inline image, where inline image is a (p − 1) × p matrix where the rows are permutations of (1,−1,0,…,0), and the relative complexity is given by inline image. Therefore, a balanced prior would be inline image, and the resulting Bayes factor of inline image against inline image would be inline image. Now we consider a one-to-one transformation of the data according to inline image, where inline image, inline image, inline image, and inline image. For the transformed data, the inequality constrained hypothesis becomes inline image, with inline image. Under this parameterization, a balanced prior would be inline image, and the Bayes factor would be equal to inline image. This may differ substantially from inline image when p > 2.

4.3 Zellner's g prior: Information paradox

Zellner's (1986) g prior is a popular prior for Bayesian variable selection which results in Bayes factors that are invariant for linear transformations of the data. To our knowledge, the g prior has not been investigated in detail when testing inequality constrained hypotheses.

For the multivariate normal linear model, the g prior under inline image can be written as

  • display math(8)

where inline image should be chosen as an anticipated parameter vector and g can be chosen according to the preferred prior weight where a large (small) value for g indicates a diffuse (informative) prior. When testing inequality constrained hypotheses, we suggest using the restricted maximum likelihood (RML) estimate (Judge, Griffiths, Carter Hill, & Lee, 1980; Mulder et al., 2010) based on the complete data as a default choice, that is,

  • display math(9)

where inline image. The RML estimate inline image maximizes the likelihood under the linear restrictions inline image. Note here that inline image is a ‘weak’ generalized inverse of inline image (Appendix C). Because the distance between the prior mean inline image and the ML estimate will be minimal, prior shrinkage will be minimal when choosing inline image.

When using the encompassing prior approach, the Bayes factor can be expressed as

  • display math(10)

where inline image is the conditional prior probability that the inequality constraints hold conditional on Σ, which is independent of g (see Appendix A), and inline image is the conditional posterior probability that the inequality constraints hold. Note that g = 1 is plugged into the conditional prior probability because inline image is independent of g. A derivation is given in Appendix A.

4.3.1 Information paradox

When testing inequality constrained hypotheses, we shall refer to the information paradox when the Bayes factor inline image of an inequality constrained hypothesis inline image against its complement inline image converges to a constant inline image when each element of inline image goes to ∞, while keeping the sample size n fixed. The paradox occurs when using the Bayes factor based on the g prior in (10), as illustrated in the following example.

Example 2. Consider a one-sided hypothesis inline image against inline image, when independent and identically distributed (i.i.d.) data come from inline image. The g prior is given by inline image and inline image, so that the conditional prior probability equals inline image, for all inline image, so that inline image, where

  • display math

with inline image. When the evidence for inline image accumulates in the sense that inline image, the Bayes factor of inline image against inline image in (19) becomes

  • display math

for fixed g and n. Consequently,

  • display math

Thus, the information paradox occurs.

5 Alternative prior specification methods

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Model and inequality constrained hypotheses
  5. 3. Bayes factors and prior probabilities
  6. 4. Issues with prior specification
  7. 5 Alternative prior specification methods
  8. 6. Simulation study
  9. 7. Concluding remarks
  10. Acknowledgements
  11. References
  12. Appendix A: Derivation of the Bayes factor
  13. Appendix B: Bayes factor based on inline image
  14. Appendix C: Generalized inverse
  15. Appendix D: Information paradox
  16. Appendix E: Intrinsic Bayes factors

5.1 Posterior priors based on transformed training samples

In order to obtain partial Bayes factors that incorporate the relative complexity of an inequality constrained hypothesis, a linear transformation of a minimal training sample is applied according to

  • display math

where inline image is the RML estimate of Θ as in (9) based on a training sample inline image. Consequently, the sufficient statistics yield

  • display math
  • display math

Hence, the ML estimate of θ based on inline image becomes equal to the RML estimate of the minimal training sample inline image under the linear restrictions inline image. Furthermore, the sums of squares remain unchanged.

The posterior prior based on the transformed minimal training sample then yields

  • display math(11)

Unlike the constrained posterior prior of Mulder et al. (2010) with diagonal covariance structure to ensure balancedness according to (7), the covariance structure of the posterior prior in (11) is based on the sums of squares of the training sample inline image. This is an important property to ensure invariance of linear transformations of the data. Below we discuss two different partial Bayes factors based on this posterior prior.

5.1.1 The partial Bayes factor inline image

According to the same steps as in Appendix A, the partial Bayes factor inline image is obtained by using the remainder of the data inline image for hypothesis testing, that is,

  • display math(12)

where

  • display math(13)

with inline image, inline image, inline image, inline image, and inline image. This partial Bayes factor works as an Ockham's razor when testing inequality constrained hypotheses where the posterior prior probability inline image serves as a measure of relative complexity, which was not the case for the posterior prior probability based on inline image, and the posterior probability based on inline image and inline image serves as a measure of relative fit of inline image. However, the information paradox can be observed when using this partial Bayes factor as a result of shrinkage to the prior mean.

Example. (continued) A minimal training sample inline image consists of inline image randomly selected observations, say inline image, from y. Furthermore, inline image, for all l, and inline image, inline image, and inline image inline image, where inline image, inline image inline image, inline image, inline image, and inline image. Therefore, inline image, for fixed n, as inline image. Consequently,

  • display math

and

  • display math

as inline image.

5.1.2 The partial Bayes factor inline image

To avoid the paradox, the posterior of the complete data set Y is used instead of a posterior based on inline image and inline image, that is,

  • display math(14)

Interestingly, the partial Bayes factor in (14) differs from the standard partial Bayes factor inline image in (4) in terms of the correction factor, that is, inline image instead of inline image, respectively. Furthermore, the partial Bayes factor in (14) avoids the information paradox.

Theorem 1. As each element of inline image goes to ∞, the partial Bayes factor inline image goes to ∞ for inline image against inline image.

Proof. See Appendix D.

Example. (continued)  The marginal posterior of θ is inline image, where inline image is independent of inline image. We compute the limit of inline image as inline image:

  • display math
5.1.3 Intrinsic Bayes factors based on inline image

Because of the dependence of the partial Bayes factor inline image on the randomly selected training sample, a method needs to be specified to ‘average’ over the partial Bayes factors over all possible training samples inline image. The resulting Bayes factor is referred to as an intrinsic Bayes factor (IBF: Berger & Pericchi, 1996, 2004). Several intrinsic IBFs have been proposed which average over the partial Bayes factors in different ways (e.g., arithmetic mean, geometric mean, or median). Here we shall use the median IBF, which can be expressed as

  • display math(15)

where Med[·] denotes the median over all L training samples. Berger and Pericchi (1998) also considered another definition of the median IBF where the correction factor is equal to the median of the ratio, that is, inline image, instead of the ratio of the medians. Here, (15) is used so that it automatically satisfies the coherence property inline image. Note that the training sample that is the median of inline image may not be the training sample that is the median of inline image. A short discussion of other IBFs can be found in Appendix E.

5.2 A g prior approach which avoids the information paradox

To avoid the information paradox when testing an equality constrained hypothesis against inline image, Liang et al. (2008) proposed to use a mixture of g-priors. When testing inequality constrained hypotheses, we propose a slightly simpler solution by letting g go to infinity. A similar solution was suggested by (Hoijtink, 2011, Ch. 10) with the exception that the g prior approach proposed here is invariant for linear transformations of the data.

First note that the respective posteriors in (19) converge in distribution to

  • display math
  • display math

as g [RIGHTWARDS ARROW] ∞. Therefore, we propose the Bayes factor

  • display math(16)

A method for computing (16) is through

  • display math

where inline image and inline image are the kth draws from inline image and inline image, respectively, and inline image is the sth posterior draw from inline image.

Theorem 2. As each element of inline image goes to ∞, the Bayes factor inline image goes to ∞ for inline image against inline image.

Proof. See Appendix D.

Example. (continued) Expression (16) yields

  • display math

which is equal to the partial Bayes factor inline image, for all l, so that inline image, and the information paradox is avoided.

6. Simulation study

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Model and inequality constrained hypotheses
  5. 3. Bayes factors and prior probabilities
  6. 4. Issues with prior specification
  7. 5 Alternative prior specification methods
  8. 6. Simulation study
  9. 7. Concluding remarks
  10. Acknowledgements
  11. References
  12. Appendix A: Derivation of the Bayes factor
  13. Appendix B: Bayes factor based on inline image
  14. Appendix C: Generalized inverse
  15. Appendix D: Information paradox
  16. Appendix E: Intrinsic Bayes factors

A simulation study was carried out to investigate the differences between the Bayes factors that were discussed in this paper. We consider the testing problem of inline image versus inline image for i.i.d. data that come from inline image, for i = 1,…,n, for four different populations: inline image and inline image; inline image and inline image; inline image and inline image; and inline image and inline image. A balanced Bayes factor inline image was obtained from a relatively non-informative conjugate balanced prior, where inline image, in which inline image, for v = 1 and 2, and inline image. The results are displayed in Figure 1. Each line represents the expected weight of evidence for inline image against inline image (i.e., the natural logarithm of the Bayes factor) of a different Bayes factor, based on 100 simulated data sets from each of the four populations when increasing n from 4 to 25. Based on these results we conclude the following. First, the Bayes factor inline image, based on the g-prior with g [RIGHTWARDS ARROW] ∞, converges fastest to the true hypothesis, followed by the median IBF based on the transformed minimal training samples, inline image, and the balanced Bayes factor inline image. Second, when the population is located on the boundary, such as inline image, the Bayes factors inline image, inline image, and inline image prefer inline image over inline image because the relative complexity of inline image is smaller than inline image, while the relative fit is equal. Third, the Bayes factor based on the non-informative improper prior, inline image, that is, the ratio of posterior probabilities that the inequality constraints hold, shows a preference for the more complex inline image.

7. Concluding remarks

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Model and inequality constrained hypotheses
  5. 3. Bayes factors and prior probabilities
  6. 4. Issues with prior specification
  7. 5 Alternative prior specification methods
  8. 6. Simulation study
  9. 7. Concluding remarks
  10. Acknowledgements
  11. References
  12. Appendix A: Derivation of the Bayes factor
  13. Appendix B: Bayes factor based on inline image
  14. Appendix C: Generalized inverse
  15. Appendix D: Information paradox
  16. Appendix E: Intrinsic Bayes factors

When testing inequality constrained hypotheses using the Bayes factors, several issues must be taken into account. First, the Bayes factor may ignore the complexity of the inequality constrained parameter spaces. This is the case with the partial Bayes factors as a result of posterior priors that are centred around the likelihood. This issue can be avoided by centring the unconstrained prior on the boundary of the constrained parameter space. Second, the Bayes factor may not be invariant for one-to-one transformations of the data. This can be observed when using balanced priors which are centred on the boundary of the constrained parameter space with zero correlations. This issue can be avoided by incorporating the covariance structure of the data in the prior. Third, the information paradox can be observed. This is a result of too much prior shrinkage which can be observed when using Zellner's g prior.

Two alternative approaches were proposed that avoid these issues. First, partial Bayes factors were constructed based on transformed minimal training samples which result in posterior priors that are centred on the boundary of the constrained parameter space with a covariance structure equal to that for the training sample. Second, a Bayes factor based on the g prior was considered where g went to infinity, denoted by inline image.

The numerical results showed that the Bayes factor inline image converged fastest to the true inequality constrained hypothesis, followed by the intrinsic Bayes factors based on partial Bayes factors of transformed minimal training samples, followed by intrinsic Bayes factors based on standard partial Bayes factors. For this reason, the Bayes factor inline image is recommended when testing inequality constrained hypotheses. As a next step, it would be interesting to also include hypotheses containing equality constraints on the parameters. In this case, inline image cannot be used due to the Lindey–Jeffreys paradox. The proposed IBFs based on transformed minimal training samples would then be a good choice.

References

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Model and inequality constrained hypotheses
  5. 3. Bayes factors and prior probabilities
  6. 4. Issues with prior specification
  7. 5 Alternative prior specification methods
  8. 6. Simulation study
  9. 7. Concluding remarks
  10. Acknowledgements
  11. References
  12. Appendix A: Derivation of the Bayes factor
  13. Appendix B: Bayes factor based on inline image
  14. Appendix C: Generalized inverse
  15. Appendix D: Information paradox
  16. Appendix E: Intrinsic Bayes factors
  • Barlow, R., Bartholomew, D., Bremner, J., & Brunk, H. (1972). Statistical inference under order restrictions. New York, NY: John Wiley.
  • Bartlett, M. (1957). A comment on D. V. Lindley's statistical paradox. Biometrika, 44, 533534.
  • Berger, J. O., & Mortera, J. (1999). Default Bayes factors for nonnested hypothesis testing. Journal of the American Statistical Association, 94, 542554. doi:10.1080/01621459.1999.10474149
  • Berger, J. O., & Pericchi, L. R. (1996). The intrinsic Bayes factor for model selection and prediction. Journal of the American Statistical Association, 91, 109122. doi:10.1080/01621459.1996.10476668
  • Berger, J. O., & Pericchi, L. R. (1998). Accurate and stable Bayesian model selection: The median intrinsic Bayes factor. Sankhyā, Series B, 60, 118.
  • Berger, J. O., & Pericchi, L. R. (2001). Objective Bayesian methods for model selection: Introduction and comparison (with discussion). In P. Lahiri (Ed.), Model Selection, Institute of Mathematical Statistics Lecture Notes 38 (pp. 135207). Beachwood, OH: Institute of Mathematical Statistics.
  • Berger, J. O., & Pericchi, L. R. (2004). Training samples in objective Bayesian model selection. Annals of Statistics, 32(3), 841869. doi:10.1214/009053604000000229
  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
  • Harville, D. A. (2008). Matrix algebra from a statistician's perspective. New York, NY: Springer.
  • Hoijtink, H. (2011). Informative hypotheses: Theory and practice for behavioral and social scientists. New York, NY: Chapman & Hall/CRC.
  • Hoijtink, H., Klugkist, I., & Boelen, P. A. (2008). Bayesian evaluation of informative hypotheses. New York, NY: Springer.
  • Jeffreys, H. (1961). Theory of probability (3rd ed.). New York, NY: Oxford University Press.
  • Judge, G. G., Griffiths, W. E., Carter Hill, R., & Lee T.-C. (1980). The theory and practice of econometrics. New York, NY: John Wiley.
  • Kass R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773795. doi:10.2307/2291091
  • Klugkist, I., & Hoijtink, H. (2007). The Bayes factor for inequality and about equality constrained models. Computational Statistics and Data Analysis, 51, 63676379. doi:10.1016/j.csda.2007.01.024
  • Klugkist, I., Laudy, O., & Hoijtink, H. (2005). Inequality constrained analysis of variance: A Bayesian approach. Psychological Methods, 10, 477493. doi:10.1037/1082-989X.10.4.477
  • Lempers, F. B. (1972). Posterior probabilities of alternative linear models. Rotterdam, The Netherlands: Rotterdam University Press.
  • Liang, F., Paulo, R., Molina, G., Clyde, M. A., & Berger, J. O. (2008). Mixtures of g priors for Bayesian variable selection. Journal of the American Statistical Association, 103, 410423. doi:10.1198/016214507000001337
  • Lindley, D. V. (1957). A statistical paradox. Biometrika, 44, 187192. doi:10.2307/2333251
  • Moreno, E., Bertolino, F., & Racugno W. (1998). An intrinsic limiting procedure for model selection and hypotheses testing. Journal of the American Statistical Association, 93, 14511460. doi:10.2307/2670059
  • Mulder, J., Hoijtink, H., & de Leeuw, C. (2012). BIEMS: A Fortran 90 program for calculating Bayes factors for inequality and equality constrained model. Journal of Statistical Software, 46.
  • Mulder, J., Hoijtink, H., & Klugkist, I. (2010). Equality and inequality constrained multivariate linear models: Objective model selection using constrained posterior priors. Journal of Statistical Planning and Inference, 140, 887906. doi:10.1016/j.jspi.2009.09.022
  • Mulder, J., Klugkist, I., de Schoot, A., Meeus, W., Selfhout, M., & Hoijtink, H. (2009). Bayesian model selection of informative hypotheses for repeated measurements. Journal of Mathematical Psychology, 53, 530546. doi:10.1016/j.jmp.2009.09.003
  • O'Hagan, A. (1995). Fractional Bayes factors for model comparison (with discussion). Journal of the Royal Statistical Society, Series B, 57, 99138.
  • Robertson, T., Wright, F. T. & Dykstra, R. (1988). Order restricted statistical inference. New York, NY: Wiley.
  • Silvapulle, M. J., & Sen, P. K. (2004). Constrained statistical inference: Inequality order and shape restrictions (2nd ed.). Hoboken, NJ: John Wiley.
  • Zellner, A. (1986). On assessing prior distributions and Bayesian regression analysis with g-prior distributions. In P. K. Goel & A. Zellner (Eds.). Bayesian inference and decision techniques: Essays in honor of Bruno de Finetti (pp. 233243). Amsterdam, The Netherlands: North-Holland/Elsevier.

Appendix A: Derivation of the Bayes factor

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Model and inequality constrained hypotheses
  5. 3. Bayes factors and prior probabilities
  6. 4. Issues with prior specification
  7. 5 Alternative prior specification methods
  8. 6. Simulation study
  9. 7. Concluding remarks
  10. Acknowledgements
  11. References
  12. Appendix A: Derivation of the Bayes factor
  13. Appendix B: Bayes factor based on inline image
  14. Appendix C: Generalized inverse
  15. Appendix D: Information paradox
  16. Appendix E: Intrinsic Bayes factors

General formulation

The Bayes factor is derived when using the encompassing prior approach of Berger and Mortera (1999) and Klugkist et al. (2005). Let the unconstrained (or encompassing) prior under the unconstrained hypothesis inline image be denoted by inline image. Then, inline image, where the normalizing constant is equal to the prior probability inline image and inline image is the indicator function. Furthermore, the likelihood function inline image under inline image is also a truncation of the likelihood function inline image under inline image, that is, inline image. The Bayes factor of inline image against inline image is given by

  • display math(17)

where inline image is the posterior probability that the inequality constraints hold. This expression was derived by Klugkist et al. (2005) in a different way.

The g prior approach

When using the unconstrained g prior in (8), a natural choice for the conditional prior under inline image is inline image, where the normalizing constant is given by

  • display math(18)

where inline image, which is independent of g. Therefore, we shall plug g = 1 into (18). Similarly, we denote inline image. Then the Bayes factor becomes

  • display math(19)

where inline image and inline image, with inline image and inline image. Interestingly, (19) shows that the Bayes factor cannot be expressed as a ratio of the marginal posterior and the marginal prior probability that the inequality constraints of inline image hold, as in (17), but as the ratio of the conditional probabilities which is computed via the marginal posterior of Σ.

Appendix C: Generalized inverse

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Model and inequality constrained hypotheses
  5. 3. Bayes factors and prior probabilities
  6. 4. Issues with prior specification
  7. 5 Alternative prior specification methods
  8. 6. Simulation study
  9. 7. Concluding remarks
  10. Acknowledgements
  11. References
  12. Appendix A: Derivation of the Bayes factor
  13. Appendix B: Bayes factor based on inline image
  14. Appendix C: Generalized inverse
  15. Appendix D: Information paradox
  16. Appendix E: Intrinsic Bayes factors

The restricted maximum likelihood estimate of θ under Rθ = r in (9) was defined as

  • display math(21)

where inline image is the unrestricted ML estimate of θ, R is a q × p restriction matrix, r is a vector of length q, and X is n × p matrix. For the sake of simplicity we omitted the indices. Furthermore,

  • display math

so that inline image holds, and therefore inline image is a generalized inverse of R. There are four conditions for the generalized inverse matrix G of R (Harville, 2008), namely,

  1. the general condition, RGR = R;
  2. the reflexive condition, GRG = G;
  3. the normalized condition, inline image; and
  4. thereverse normalized condition, inline image.

It can be easily checked that inline image only satisfies conditions 1 and 2. Therefore, inline image is also referred to as a ‘weak’ or ‘reflexive’ generalized inverse matrix.

Note that we cannot simply plug in a different generalized inverse matrix for inline image. For example, if a generalized inverse matrix, say, inline image, were substituted for inline image in (21), for which inline image held, the resulting expression would be independent of the data because inline image.

Appendix D: Information paradox

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Model and inequality constrained hypotheses
  5. 3. Bayes factors and prior probabilities
  6. 4. Issues with prior specification
  7. 5 Alternative prior specification methods
  8. 6. Simulation study
  9. 7. Concluding remarks
  10. Acknowledgements
  11. References
  12. Appendix A: Derivation of the Bayes factor
  13. Appendix B: Bayes factor based on inline image
  14. Appendix C: Generalized inverse
  15. Appendix D: Information paradox
  16. Appendix E: Intrinsic Bayes factors

The limit of the partial Bayes factor inline image and the Bayes factor inline image (based on the g prior) are computed for inline image against inline image, as each element of inline image goes to ∞.

First, note that the limit of the posterior probability that the inequality constraints of inline image hold converges to 1, i.e.,

  • display math

where inline image. Therefore,

  • display math

and

  • display math

so that

  • display math

Hence, the information paradox is avoided for this partial Bayes factor.

Second, we derive the limit of the conditional posterior probability

  • display math

for all inline image. Therefore,

  • display math

Following the same steps, it can be shown that the conditional posterior probability that the constraints of inline image hold goes to 0, as inline image, and therefore

  • display math

Consequently,

  • display math

Thus, the information paradox is avoided.

Appendix E: Intrinsic Bayes factors

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Model and inequality constrained hypotheses
  5. 3. Bayes factors and prior probabilities
  6. 4. Issues with prior specification
  7. 5 Alternative prior specification methods
  8. 6. Simulation study
  9. 7. Concluding remarks
  10. Acknowledgements
  11. References
  12. Appendix A: Derivation of the Bayes factor
  13. Appendix B: Bayes factor based on inline image
  14. Appendix C: Generalized inverse
  15. Appendix D: Information paradox
  16. Appendix E: Intrinsic Bayes factors

Different intrinsic Bayes factors can be used for averaging over the partial Bayes factors inline image which are coherent in the sense that inline image and inline image, such as the encompassing arithmetic IBF,

  • display math(22)

the geometric IBF,

  • display math(23)

and the median IBF,

  • display math(24)

where inline image is the Bayes factor for inline image against inline image based on a non-informative improper prior which is equal to the ratio of posterior probabilities that the inequality constraints hold, that is, inline image. The results for the encompassing arithmetic IBF and the geometric IBF were very similar to those for the median IBF in the simulation study in Figure 1.

image

Figure 1. Expected weight of evidence (inline image) based on five different Bayes factors for testing inline image vs inline image: the Bayes factor based on non-informative improper prior inline image, the median IBF based on standard training samples inline image, inline image, and transformed training samples inline image, inline image, a balanced Bayes factor inline image, and the Bayes factor inline image based on the g-prior where g [RIGHTWARDS ARROW] ∞. The results are based on four different populations with mean θ and covariance matrix [1ρ; ρ1].

Download figure to PowerPoint

When an IBF behaves like an actual Bayes factor as the sample size goes to infinity, the corresponding prior of this limiting Bayes factor is referred to as the intrinsic prior (e.g., Berger & Pericchi, 1996; Moreno, Bertolino, & Racugno, 1998; Berger & Mortera, 1999). The IBFs in (22)–(24) have an intrinsic prior that is improper because the limit of the correction factor inline image as n [RIGHTWARDS ARROW] ∞ (e.g., inline image in the encompassing arithmetic IBF) only depends on inline image, which are independent of the sufficient statistic of θ. For this reason, the IBFs in (22)–(24) would be recommended instead of the Bayes factors based on these improper intrinsic priors.

To illustrate the sensitivity of the geometric IBF to extreme training samples, that is, when inline image is very close to zero, we consider the data set

  • display math(25)

and two one-sided hypotheses inline image and inline image on the means of a bivariate model inline image. The sample mean equals inline image, for all inline image, which implies evidence for inline image against inline image, and the sums of squares S are approximately equal to [84;47.33] when inline image is close to zero. Note here that inline image only depends on the training sample correlation, inline image, because the posterior prior based on inline image is centred on 0. Furthermore, inline image is monotone in inline image, and therefore, inline image is obtained from the same training sample as Medinline image. When a minimal training sample (with inline image) consists of any combination of the first four observations and ε is close to zero, the posterior prior probability inline image will be approximately equal to .5, and inline image will be approximately equal to 0. This greatly affects the geometric IBF, as can be seen in Table E1. When inline image, there is even a slight preference for inline image against inline image which conflicts with the posterior probability of inline image. The median IBF and encompassing arithmetic IBF seem relatively unaffected by inline image.

Table E1. The outcomes of the encompassing arithmetic IBF, the geometric IBF, and the median IBF for the data set in (25) for different values of inline image
  inline image inline image inline image
inline image 5.23.35.5
inline image 5.41.35.8
inline image 5.40.55.8