How to practise Bayesian statistics outside the Bayesian church: What philosophy for Bayesian statistical modelling?

• Then I saw Tom Bayes – Now I'm a believer,

• Without a trace – of doubt in my mind!

• I'm a Bayesian (oooh) – Oh I'm a believer –

• I couldn't p now if I tried!

• - Brad Carlin, Bayesian Believer

1. Introduction

Unlike most other statistical frameworks, Bayesian statistical inference is wedded to a particular approach in the philosophy of science (see Howson & Urbach, 2006); this approach is called Bayesianism. Rather than being concerned with model fitting, this position in the philosophy of science primarily addresses theory choice. Naturally, in some cases there exists a relation between scientific theories and statistical models, and this relation can be so tight that choosing the model is tantamount to accepting the theory. However, in many cases of data analysis, the statistical model bears only an indirect relation to scientific theory, and in such cases the act of statistical modelling is distinct from the act of theory choice.

If one takes seriously the distinction between statistical modelling and theory evaluation, it becomes clear that one who takes a Bayesian approach in one of these areas need not take it in the other. Thus, one who utilizes statistical techniques that are known as ‘Bayesian’ (e.g., computes a Bayes factor) may not adhere to the philosophy of science that goes by the same name; and one committed to the philosophy of Bayesianism may, for a host of reasons, employ techniques that arise from non-Bayesian statistics (e.g., a classical t-test). The former stance, where one utilizes Bayesian machinery without adhering to the Bayesian account of science, has become quite commonplace in statistical modelling. This is because Bayesian model-fitting routines occasionally offer considerable resources where frequentist approaches struggle, and thus there is a pragmatic reason to use them.

At conference dinners, modellers who use Bayesian statistics without adhering to Bayesianist philosophy are usually called ‘practical Bayesians’. The paper by Gelman and Shalizi (2013) is perhaps the first attempt to systematically justify and underpin the position of the practical Bayesian, and thereby offers a welcome addition to the literature. Gelman and Shalizi's alternative philosophy is primarily shaped by Popper's view that scientific propositions are to be submitted to repeated criticism in the form of strong empirical tests. For them, best Bayesian statistical practice involves formulating models using Bayesian statistical methods, and then checking them through attempts to falsify and modify them. At the same time, Gelman and Shalizi reject the orthodox Bayesian view that statistical inference is inductive inference, which involves updating subjective probability estimates that hypotheses are true. Their hope is that the philosophical foundations they provide for Bayesian statistical practice will benefit both the practice of statistics and the philosophy of science.

We welcome Gelman and Shalizi's paper because it serves to explicate the idea that, to practise Bayesian statistics, one need not first be converted and baptized in the Bayesian church. Practical Bayesianism is a viable position that can be taken by one who subscribes to non-Bayesian philosophy of science without internal contradiction. As Gelman and Shalizi show, many of the procedures commonly followed in Bayesian statistics can be accommodated by alternative perspectives. However, the particular account that Gelman and Shalizi have is not the only alternative, and it is questionable whether it is adequate in all situations where Bayesian statistical modelling arises. In this commentary, we will draw attention to alternatives that may be more fruitful in accommodating Bayesian practice.

2. The philosophy of science and the philosophy of data analysis

Bayesianism, in the philosophy of science, holds that rational agents should update their belief in various theories according to the famous formula P(T|E) = P(E|T)P(T)/P(E), where T denotes a theory and E the evidence. Bayesian statistics works on the idea that models should be evaluated via the analogous formula P(M|D) = P(D|M)P(M)/P(D), in which M is a statistical model and D the data. In discussions on Bayesian statistics, it often tacitly assumed that T = M and E = D, so that statistical modelling is a special case of Bayesian philosophy. However, in many cases, these identifications are not viable. In particular, it is hard to uphold that statistical model and substantive theory are typically one and the same.

For instance, we may entertain the theory that males and females do not structurally differ in intelligence. In statistical work, we may then subsequently analyse IQ scores and test the statistical model F(IQ) = F(IQ|Sex), where F(·) denotes the population distribution function. The statistical model is not identical to the theory, unless one is willing to assume that IQ and intelligence are interchangeable terms, which is a gross simplification on any of the psychometric theories of intelligence currently on offer. One can readily see the importance of this distinction by considering the appropriate theoretical move upon finding, in empirical work, that the statistical model is not appropriate for the data, so that the distributions of IQ are not invariant across sex. Clearly, one can accept this conclusion without necessarily accepting that intelligence differs across the sexes.

Because the theory is not identical to the statistical model, one has to fix a relation between them to get the scientific process going. Given that identity of T and M is clearly not an option, one may alternatively construct the relation to be one of logical implication (i.e., it is assumed that T entails M), but even this is, we think, is too much to honour when faced with the exceedingly messy process of statistical testing in science. Even in order to get the entailment of M from T, one has to fix many auxiliary hypotheses (e.g., the validity of the IQ test as measure of intelligence, the distribution of error scores, the dichotomous treatment of sex, etc.), and if one ponders this issue a little it is evident that there are not just many such hypotheses, but an infinity of them. As a result, our working hypothesis ought to be that statistical inference and theory choice are not games that are played in the same ball park.

Given this conclusion, one has to wonder whether it really makes any difference for our evaluation of the theoretical hypothesis in question whether we evaluate the statistical hypothesis F(IQ) = F(IQ|Sex) through Bayesian statistics or otherwise. Suppose John evaluates the hypothesis by computing a posterior distribution for the statistical hypothesis. Is John then automatically mandated to generalize his Bayesian behaviour to the evaluation of the theory? Suppose Jane simply evaluates the hypothesis using a frequentist t-test. Does this preclude Jane from entering the conclusion into a Bayesian chain of reasoning to find out what she should believe about the theory in question? We think it is clear that no such generalizations follow. John could evaluate his results according to a hypothetico-deductive scheme, while Jane could plug hers into the Bayesian philosophical framework. There is no reason suppose that statistical data analysis and theory evaluation should follow the same theoretical precept.

This, we think, is one of the most important conclusions to draw from the paper by Gelman and Shalizi, who work out the connection between a Bayesian data-analytic framework and a hypothetico-deductive account of theory evaluation in detail. However, their vigorous defence of this analysis, which claims that Bayesian practice sits ‘much better’ with hypothetico-deductive accounts across the board, suggests that Bayesian practitioner should now trade their Bayesian account for Gelman and Shalizi's hypothetico-deductive theory. This, we think, is a non sequitur, for two reasons. First, that Gelman and Shalizi are able to accommodate Bayesian practice in a hypothetico-deductive framework does not entail that such practice could not be accommodated by one of the many other philosophies of science that have been developed in the past centuries. Second, the philosophy of science that Gelman and Shalizi have in store is unnecessarily impoverished – so impoverished, in fact, that it is doubtful whether their version of hypothetico-deductivism is able to sustain their data-analytic work in the first place.

3. Popper and the hypothetico-deductive method

Gelman and Shalizi maintain that formulating, checking and revising models accords well with sophisticated forms of hypothetico-deductive inference. However, for them, sophisticated hypothetico-deductivism essentially amounts to a meagre version of Popper's (1969) falsificationist view of hypothetico-deductive method. By stripping away many of the elements of Popper's philosophy of critical rationalism that make his falsificationist theory of science a rich account of scientific inquiry, the authors are left with an impoverished and rather unsophisticated account of the hypothetico-deductive method. Notably, Gelman and Shalizi reject Popper's confirmation-theoretic notion of corroboration, but they offer nothing in its place. Further, perhaps influenced by Popper's view that there is no logic to discovery, they offer no methodological account of model formulation. For them, hypothetico-deductive inquiry is little more than the injunction to engage in repeated strong testing of hypotheses about models, along with a commitment to the view that this should be done by exploiting deductive inference only. We acknowledge that Gelman and Shalizi speak of adding a neo-Popperian Lakatosian flavour to their thinking in related publications, but the ideas of Lakatos' methodology of scientific research programmes do no real work in their philosophy of Bayesian modelling.

In choosing a Popperian view of scientific confirmation, Gelman and Shalizi explicitly reject the confirmationist account of hypothetico-deductive method promoted by Carl Hempel. Actually, their reference to Hempel (1965) is to his early instance confirmation view of scientific confirmation, not to his later account of the hypothetico-deductive method. Hempel proposed the idea that, in scientific confirmation, hypotheses are confirmed by discovering their positive instances. In his formalization of this idea, Hempel required that the evidence entailed the development of the relevant hypothesis. This is quite different from hypothetico-deductive inference in which the evidence is deductively entailed by the hypothesis, and confirmation occurs through successful predictive testing.

More importantly, there are now available a number of sophisticated variants of the hypothetico-deductive method that Gelman and Shalizi might have made use of in formulating their own account of this method. Sprenger (2011a) provides a useful overview and defence of modern thinking about the hypothetico-deductive method. Interestingly, Sprenger (2011b) also recently proposed an account of confirmation that unifies Hempel's insight that hypotheses are confirmed by their instances and the core hypothetico-deductive idea that hypotheses are confirmed by their successful predictions. This modern hybrid account of confirmation has an important advantage over that outlined by Gelman and Shalizi: it allows for an objective notion of inductive support – something that we think Gelman & Shalizi's model testing strategy, in fact, requires. At the same time, it features strong hypothetico-deductive testing of a falsificationist kind, and it allows for the piecemeal testing of entire theories. Both of these are desirable features of scientific modelling for Gelman and Shalizi.

In addition to drawing from Popper, Gelman and Shalizi make brief heuristic use of some of Thomas Kuhn's (1970) ideas about science. They suggest that Kuhn's distinction between normal and revolutionary science is somewhat analogous to their distinction between learning within a Bayesian model and checking the model either to discard or expand it. However, they caution about pushing the analogy too far, correctly pointing out that most model checking and reformulation is puzzle-solving work, not revolutionary change, that takes place within a single paradigm. We think this disanalogy renders a serious appeal to Kuhn's theory of science as basically inappropriate for their particular philosophy, as indeed it is for the social sciences generally.

4. Modes of scientific inference

Gelman and Shalizi follow Popper in declaring that deductive inference is all there is to scientific inference. For them, this allows for the strong testing of Bayesian models by constantly checking them via their deductively derived predictions. However, unlike Popper, they acknowledge that science does trade in inductive inference of a material kind in which the premises and conclusion of inductive arguments contain reference to context-specific facts. Furthermore, they acknowledge that inductive statistical inferences to unobserved cases can be drawn on a background of deductive models. So it would seem that, for them, science admits both deductive and inductive modes of reasoning.

We would go further and claim that in addition to deductive and inductive inference, science makes heavy use of abductive inference – a form of inference, moreover, that we think has a place in Gelman and Shalizi's view of model generation and model revision. In a word, abductive inference is explanatory inference, and it involves reasoning to, or from, hypotheses that explain relevant facts (e.g., Magnani, 2001). For example, the statistical method of exploratory factor analysis involves the abductive generation of latent factors to explain patterns in multivariate data. Further, inference to the best explanation (briefly mentioned by Gelman and Shalizi) is an abductive approach to theory appraisal in which explanatory reasoning forms the basis for evaluating rival theories.

Although Gelman and Shalizi describe their modelling philosophy as falsificationist in nature, it is more than this. For, when a model (or a component of a model) is confronted with negative evidence, the model (or its relevant parts) can be revised by means other than straight rejection or elimination. Often, this modification of a hypothesis will be seen to plausibly explain the anomalous data. We think that when scientists engage in such model revision, they engage in abductive reasoning, whether they know it or not. Thus, it would seem that Gelman and Shalizi's account requires an extension to abductive inference if it is to account for standard practices of model revision.

Two further comments on the inference forms involved in modelling are in order. Gelman and Shalizi regard the process of checking and ruling out possible misspecifications of a model as consistent with the strategy of eliminative induction. However, in this context, they think the word induction is a misnomer, and they enlist the support of Kitcher (1993) in maintaining that the strategy really embodies a deductive argument. However, Kitcher is concerned with the successive elimination of actual theories that rival the theory of interest, not with successive checks for possible inconsistencies in a single model. We think that both inductive and deductive eliminative strategies are used in science, and that because of the uncertainties in social science research, the aspect of model checking referred to here by Gelman and Shalizi is more realistically construed as an inductive strategy.

Our final comment on scientific inference is to point out that Gelman and Shalizi's bald characterization of classical Fisherian and Neyman–Pearsonian inference as deductive in nature is wrong, or at least simplistic. If anything, Fisher was an inductivist, and Neyman seemed to endorse both inductive and deductive forms of inference. However, a proper characterization of the forms of inference involved in these two statistical traditions is demanding and complex (Rivadulla, 1991).

5. Conclusion

Gelman and Shalizi have done the statistical community a great service by decoupling Bayesian data analysis and Bayesian philosophy of science. We fully agree with the conclusion that Bayesian data analysis can be justified in non-Bayesian ways. However, we think it is important to emphasize that the actual approach chosen by Gelman and Shalizi is but one of many. It should be recognized that one can consistently be a practical Bayesian without being a philosophical Bayesian, but it should also be recognized that one can maintain that consistency in a variety of ways, not just through hypothetico-deductivism. We ourselves have reservations about the suitability of Popper's theory-centred falsificationist account of science for data modelling which, despite their comments to the contrary, we take to be Gelman and Shalizi's real focus.

In a sense, Gelman and Shalizi may trade one problem for another, not because they are substituting hypothetico-deductive philosophy for Bayesianism, but because they are attempting to characterize data analysis in such a grand theory in the first place. Perhaps the philosophy of data analysis cannot be tied uniquely to one of the major theories of scientific inference. The process of data analysis could be argued to feature elements that are reminiscent of many several theories in the philosophy of science – including inductive, deductive, and abductive accounts. However, it is also guided and constrained by strongly pragmatic concerns, ranging from the available money and time to the computational resources of computers, that are alien to Bayesianism, but also to the type of hypothetico-deductive theories that Gelman and Shalizi enlist. In fact it is improbable, in our view, that a one-to-one mapping between a philosophy of inference and a philosophy of data analysis could ever be achieved.