Extending the Box–Cox transformation to the linear mixed model

Summary.  For a univariate linear model, the Box–Cox method helps to choose a response transformation to ensure the validity of a Gaussian distribution and related assumptions. The desire to extend the method to a linear mixed model raises many vexing questions. Most importantly, how do the distributions of the two sources of randomness (pure error and random effects) interact in determining the validity of assumptions? For an otherwise valid model, we prove that the success of a transformation may be judged solely in terms of how closely the total error follows a Gaussian distribution. Hence the approach avoids the complexity of separately evaluating pure errors and random effects. The extension of the transformation to the mixed model requires an exploration of its potential effect on estimation and inference of the model parameters. Analysis of longitudinal pulmonary function data and Monte Carlo simulations illustrate the methodology discussed.