Several issues are discussed when testing inequality constrained hypotheses using a Bayesian approach. First, the complexity (or size) of the inequality constrained parameter spaces can be ignored. This is the case when using the posterior probability that the inequality constraints of a hypothesis hold, Bayes factors based on non-informative improper priors, and partial Bayes factors based on posterior priors. Second, the Bayes factor may not be invariant for linear one-to-one transformations of the data. This can be observed when using balanced priors which are centred on the boundary of the constrained parameter space with a diagonal covariance structure. Third, the information paradox can be observed. When testing inequality constrained hypotheses, the information paradox occurs when the Bayes factor of an inequality constrained hypothesis against its complement converges to a constant as the evidence for the first hypothesis accumulates while keeping the sample size fixed. This paradox occurs when using Zellner's g prior as a result of too much prior shrinkage. Therefore, two new methods are proposed that avoid these issues. First, partial Bayes factors are proposed based on transformed minimal training samples. These training samples result in posterior priors that are centred on the boundary of the constrained parameter space with the same covariance structure as in the sample. Second, a g prior approach is proposed by letting g go to infinity. This is possible because the Jeffreys–Lindley paradox is not an issue when testing inequality constrained hypotheses. A simulation study indicated that the Bayes factor based on this g prior approach converges fastest to the true inequality constrained hypothesis.

This paper presents a procedure to test factorial invariance in multilevel confirmatory factor analysis. When the group membership is at level 2, multilevel factorial invariance can be tested by a simple extension of the standard procedure. However level-1 group membership raises problems which cannot be appropriately handled by the standard procedure, because the dependency between members of different level-1 groups is not appropriately taken into account. The procedure presented in this article provides a solution to this problem. This paper also shows Muthén's maximum likelihood (MUML) estimation for testing multilevel factorial invariance across level-1 groups as a viable alternative to maximum likelihood estimation. Testing multilevel factorial invariance across level-2 groups and testing multilevel factorial invariance across level-1 groups are illustrated using empirical examples. SAS macro and Mplus syntax are provided.

We discuss the statistical testing of three relevant hypotheses involving Cronbach's alpha: one where alpha equals a particular criterion; a second testing the equality of two alpha coefficients for independent samples; and a third testing the equality of two alpha coefficients for dependent samples. For each of these hypotheses, various statistical tests have been proposed. Over the years, these tests have depended on progressively fewer assumptions. We propose a new approach to testing the three hypotheses that relies on even fewer assumptions, is especially suited for discrete item scores, and can be applied easily to tests containing large numbers of items. The new approach uses marginal modelling. We compared the Type I error rate and the power of the marginal modelling approach to several of the available tests in a simulation study using realistic conditions. We found that the marginal modelling approach had the most accurate Type I error rates, whereas the power was similar across the statistical tests.

Of the several tests for comparing population means, the best known are the ANOVA, Welch, Brown–Forsythe, and James tests. Each performs appropriately only in certain conditions, and none performs well in every setting. Researchers, therefore, have to select the appropriate procedure and run the risk of making a bad selection and, consequently, of erroneous conclusions. It would be desirable to have a test that performs well in any situation and so obviate preliminary analysis of data. We assess and compare several tests for equality of means in a simulation study, including non-parametric bootstrap techniques, finding that the bootstrap ANOVA and bootstrap Brown–Forsythe tests exhibit a similar and exceptionally good behaviour.

Multi-group latent growth modelling in the structural equation modelling framework has been widely utilized for examining differences in growth trajectories across multiple manifest groups. Despite its usefulness, the traditional maximum likelihood estimation for multi-group latent growth modelling is not feasible when one of the groups has no response at any given data collection point, or when all participants within a group have the same response at one of the time points. In other words, multi-group latent growth modelling requires a complete covariance structure for each observed group. The primary purpose of the present study is to show how to circumvent these data problems by developing a simple but creative approach using an existing estimation procedure for growth mixture modelling. A Monte Carlo simulation study was carried out to see whether the modified estimation approach provided tangible results and to see how these results were comparable to the standard multi-group results. The proposed approach produced results that were valid and reliable under the mentioned problematic data conditions. We also present a real data example and demonstrate that the proposed estimation approach can be used for the chi-square difference test to check various types of measurement invariance as conducted in a standard multi-group analysis.

A new oblique factor rotation method is proposed, the aim of which is to identify a simple and well-clustered structure in a factor loading matrix. A criterion consisting of the complexity of a factor loading matrix and a between-cluster dissimilarity is optimized using the gradient projection algorithm and the k-means algorithm. It is shown that if there is an oblique rotation of an initial loading matrix that has a perfect simple structure, then the proposed method with Kaiser's normalization will produce the perfect simple structure. Although many rotation methods can also recover a perfect simple structure, they perform poorly when a perfect simple structure is not possible. In this case, the new method tends to perform better because it clusters the loadings without requiring the clusters to be perfect. Artificial and real data analyses demonstrate that the proposed method can give a simple structure, which the other methods cannot produce, and provides a more interpretable result than those of widely known rotation techniques.

The two-sample Student t test of location was performed on random samples of scores and on rank-transformed scores from normal and non-normal population distributions with unequal variances. The same test also was performed on scores that had been explicitly selected to have nearly equal sample variances. The desired homogeneity of variance was brought about by repeatedly rejecting pairs of samples having a ratio of standard deviations that exceeded a predetermined cut-off value of 1.1, 1.2, or 1.3, while retaining pairs with ratios less than the cut-off value. Despite this forced conformity with the assumption of equal variances, the tests on the selected samples were no more robust than tests on unselected samples, and in most cases substantially less robust. Under conditions where sample sizes were unequal, so that Type I error rates were inflated and power curves were atypical, the selection procedure produced still greater inflation and distortion of the power curves.

For one-way fixed effects ANOVA, it is well known that the conventional F test of the equality of means is not robust to unequal variances, and numerous methods have been proposed for dealing with heteroscedasticity. On the basis of extensive empirical evidence of Type I error control and power performance, Welch's procedure is frequently recommended as the major alternative to the ANOVA F test under variance heterogeneity. To enhance its practical usefulness, this paper considers an important aspect of Welch's method in determining the sample size necessary to achieve a given power. Simulation studies are conducted to compare two approximate power functions of Welch's test for their accuracy in sample size calculations over a wide variety of model configurations with heteroscedastic structures. The numerical investigations show that Levy's (1978a) approach is clearly more accurate than the formula of Luh and Guo (2011) for the range of model specifications considered here. Accordingly, computer programs are provided to implement the technique recommended by Levy for power calculation and sample size determination within the context of the one-way heteroscedastic ANOVA model.

The ‘deterministic-input noisy-AND’ (DINA) model is one of the more frequently applied diagnostic classification models for binary observed responses and binary latent variables. The purpose of this paper is to show that the model is equivalent to a special case of a more general compensatory family of diagnostic models. Two equivalencies are presented. Both project the original DINA skill space and design Q-matrix using mappings into a transformed skill space as well as a transformed Q-matrix space. Both variants of the equivalency produce a compensatory model that is mathematically equivalent to the (conjunctive) DINA model. This equivalency holds for all DINA models with any type of Q-matrix, not only for trivial (simple-structure) cases. The two versions of the equivalency presented in this paper are not implied by the recently suggested log-linear cognitive diagnosis model or the generalized DINA approach. The equivalencies presented here exist independent of these recently derived models since they solely require a linear – compensatory – general diagnostic model without any skill interaction terms. Whenever it can be shown that one model can be viewed as a special case of another more general one, conclusions derived from any particular model-based estimates are drawn into question. It is widely known that multidimensional models can often be specified in multiple ways while the model-based probabilities of observed variables stay the same. This paper goes beyond this type of equivalency by showing that a conjunctive diagnostic classification model can be expressed as a constrained special case of a general compensatory diagnostic modelling framework.

Several methods are available to estimate the total and residual amount of heterogeneity in meta-analysis, leading to different alternatives when estimating the predictive power in mixed-effects meta-regression models using the formula proposed by Raudenbush (1994, 2009). In this paper, a simulation study was conducted to compare the performance of seven estimators of these parameters under various realistic scenarios in psychology and related fields. Our results suggest that the number of studies (k) exerts the most important influence on the accuracy of the results, and that precise estimates of the heterogeneity variances and the model predictive power can only be expected with at least 20 and 40 studies, respectively. Increases in the average within-study sample size () also improved the results for all estimators. Some differences among the accuracy of the estimators were observed, especially under adverse (small k and ) conditions, while the results for the different methods tended to convergence for more optimal scenarios.

Haberman (2008) suggested a method to determine if subtest scores have added value over the total score. The method is based on classical test theory and considers the estimation of the true subscores. Performance of subgroups, for example, those based on gender or ethnicity, on subtests is often of interest. Researchers such as Stricker (1993) and Livingston and Rupp (2004) found that the difference in performance between the subgroups often varies over the different subtests. We suggest a method to examine whether the knowledge of the subgroup membership of the examinees leads to a better estimation of the true subscores. We apply our suggested method to data from two operational testing programmes. The knowledge of the subgroup membership of the examinees does not lead to a better estimation of the true subscore for the data sets.

Multivariate ordinal and quantitative longitudinal data measuring the same latent construct are frequently collected in psychology. We propose an approach to describe change over time of the latent process underlying multiple longitudinal outcomes of different types (binary, ordinal, quantitative). By relying on random-effect models, this approach handles individually varying and outcome-specific measurement times. A linear mixed model describes the latent process trajectory while equations of observation combine outcome-specific threshold models for binary or ordinal outcomes and models based on flexible parameterized non-linear families of transformations for Gaussian and non-Gaussian quantitative outcomes. As models assuming continuous distributions may be also used with discrete outcomes, we propose likelihood and information criteria for discrete data to compare the goodness of fit of models assuming either a continuous or a discrete distribution for discrete data. Two analyses of the repeated measures of the Mini-Mental State Examination, a 20-item psychometric test, illustrate the method. First, we highlight the usefulness of parameterized non-linear transformations by comparing different flexible families of transformation for modelling the test as a sum score. Then, change over time of the latent construct underlying directly the 20 items is described using two-parameter longitudinal item response models that are specific cases of the approach.

In 2004, Hunter and Schmidt proposed a correction (called Case IV) that seeks to estimate disattenuated correlations when selection is made on an unmeasured variable. Although Case IV is an important theoretical development in the range restriction literature, it makes an untestable assumption, namely that the partial correlation between the unobserved selection variable and the performance measure is zero. We show in this paper why this assumption may be difficult to meet and why previous simulations have failed to detect the full extent of bias. We use meta-analytic literature to investigate the plausible range of bias. We also show how Case IV performs in terms of standard errors. Finally, we give practical recommendations about how the contributions of Hunter and Schmidt (2004) can be extended without making such stringent assumptions.

Precursors of the reliability generalization (RG) meta-analytic approach have not established a single preferred analytic method. By means of five real RG examples, we examine how using different statistical methods to integrate coefficients alpha can influence results in RG studies. Specifically, we compare thirteen different statistical models for averaging reliability coefficients and searching for moderator variables that differ in terms of: (a) whether to transform or not the coefficients alpha, and (b) the statistical model assumed, distinguishing between ordinary least squares methods, the fixed-effect (FE) model, the varying coefficient (VC) model, and several versions of the random-effects (RE) model. The results obtained with the different methods exhibited important discrepancies, especially regarding moderator analyses. The main criterion for the model choice should be the extent to which the meta-analyst intends to generalize the results. RE models are the most appropriate when the meta-analyst aims to generalize to a hypothetical population of past or future studies, while FE and VC models are the most appropriate when the interest focuses on generalizing the results to a population of studies identical to those included in the meta-analysis. Finally, some guidelines are proposed for selecting the statistical model when conducting an RG study.

A common question of interest to researchers in psychology is the equivalence of two or more groups. Failure to reject the null hypothesis of traditional hypothesis tests such as the ANOVA F-test (i.e., H₀: μ₁ = … = μ_k) does not imply the equivalence of the population means. Researchers interested in determining the equivalence of k independent groups should apply a one-way test of equivalence (e.g., Wellek, 2003). The goals of this study were to investigate the robustness of the one-way Wellek test of equivalence to violations of homogeneity of variance assumption, and compare the Type I error rates and power of the Wellek test with a heteroscedastic version which was based on the logic of the one-way Welch (1951) F-test. The results indicate that the proposed Wellek–Welch test was insensitive to violations of the homogeneity of variance assumption, whereas the original Wellek test was not appropriate when the population variances were not equal.

Random item effects models provide a natural framework for the exploration of violations of measurement invariance without the need for anchor items. Within the random item effects modelling framework, Bayesian tests (Bayes factor, deviance information criterion) are proposed which enable multiple marginal invariance hypotheses to be tested simultaneously. The performance of the tests is evaluated with a simulation study which shows that the tests have high power and low Type I error rate. Data from the European Social Survey are used to test for measurement invariance of attitude towards immigrant items and to show that background information can be used to explain cross-national variation in item functioning.

This study aims to reparameterize ordinary factors into between- and within-person factor effects and utilize an array of the within-person factor loadings as a latent profile which encapsulates all score responses of individuals in a population. To illustrate, the Woodcock–Johnson III (WJ-III) tests of cognitive abilities were analysed and one between- and two within-person factors were identified. The scoring patterns of individuals in the WJ-III sample were interpreted according to the within-person factor patterns. Regression analyses were performed to examine how much the within-person factors accounted for the person scoring patterns and criterion variables. Finally, the importance and applications of the between- and within-person factors are discussed.

This paper reports on a simulation study that evaluated the performance of five structural equation model test statistics appropriate for categorical data. Both Type I error rate and power were investigated. Different model sizes, sample sizes, numbers of categories, and threshold distributions were considered. Statistics associated with both the diagonally weighted least squares (cat-DWLS) estimator and with the unweighted least squares (cat-ULS) estimator were studied. Recent research suggests that cat-ULS parameter estimates and robust standard errors slightly outperform cat-DWLS estimates and robust standard errors (Forero, Maydeu-Olivares, & Gallardo-Pujol, 2009). The findings of the present research suggest that the mean- and variance-adjusted test statistic associated with the cat-ULS estimator performs best overall. A new version of this statistic now exists that does not require a degrees-of-freedom adjustment (Asparouhov & Muthén, 2010), and this statistic is recommended. Overall, the cat-ULS estimator is recommended over cat-DWLS, particularly in small to medium sample sizes.

In this journal, Zimmerman (2004, 2011) has discussed preliminary tests that researchers often use to choose an appropriate method for comparing locations when the assumption of normality is doubtful. The conceptual problem with this approach is that such a two-stage process makes both the power and the significance of the entire procedure uncertain, as type I and type II errors are possible at both stages. A type I error at the first stage, for example, will obviously increase the probability of a type II error at the second stage. Based on the idea of Schmider et al. (2010), which proposes that simulated sets of sample data be ranked with respect to their degree of normality, this paper investigates the relationship between population non-normality and sample non-normality with respect to the performance of the ANOVA, Brown–Forsythe test, Welch test, and Kruskal–Wallis test when used with different distributions, sample sizes, and effect sizes. The overall conclusion is that the Kruskal–Wallis test is considerably less sensitive to the degree of sample normality when populations are distinctly non-normal and should therefore be the primary tool used to compare locations when it is known that populations are not at least approximately normal.

In applications of item response theory, assessment of model fit is a critical issue. Recently, limited-information goodness-of-fit testing has received increased attention in the psychometrics literature. In contrast to full-information test statistics such as Pearson’s X² or the likelihood ratio G², these limited-information tests utilize lower-order marginal tables rather than the full contingency table. A notable example is Maydeu-Olivares and colleagues’M₂ family of statistics based on univariate and bivariate margins. When the contingency table is sparse, tests based on M₂ retain better Type I error rate control than the full-information tests and can be more powerful. While in principle the M₂ statistic can be extended to test hierarchical multidimensional item factor models (e.g., bifactor and testlet models), the computation is non-trivial. To obtain M₂, a researcher often has to obtain (many thousands of) marginal probabilities, derivatives, and weights. Each of these must be approximated with high-dimensional numerical integration. We propose a dimension reduction method that can take advantage of the hierarchical factor structure so that the integrals can be approximated far more efficiently. We also propose a new test statistic that can be substantially better calibrated and more powerful than the original M₂ statistic when the test is long and the items are polytomous. We use simulations to demonstrate the performance of our new methods and illustrate their effectiveness with applications to real data.

When two test forms measure the same construct but are independently modelled using item response theory, the two forms’ respective metrics cannot be assumed to be equivalent. Thus, before comparing parameter estimates across forms, a linear transformation must be applied to at least one form's scale. The mean-sigma method is a well-known procedure for estimating this adjustment when a common set of items appears on both forms. In this paper, I show both analytically and empirically (through a small simulation study) that the mean-sigma estimators of the transformation constants are biased. While this systematic error was modest relative to random error under the conditions studied here, it is nevertheless intrinsic and its magnitude is conditional on extrinsic design features that include the number of anchor items and the quality of their difficulty estimates.

Multilevel mediation analysis examines the indirect effect of an independent variable on an outcome achieved by targeting and changing an intervening variable in clustered data. We study analytically and through simulation the effects of an omitted variable at level 2 on a 1–1–1 mediation model for a randomized experiment conducted within clusters in which the treatment, mediator, and outcome are all measured at level 1. When the residuals in the equations for the mediator and the outcome variables are fully orthogonal, the two methods of calculating the indirect effect (ab, c – c′) are equivalent at the between- and within-cluster levels. Omitting a variable at level 2 changes the interpretation of the indirect effect and will induce correlations between the random intercepts or random slopes. The equality of within-cluster ab and c – c′ no longer holds. Correlation between random slopes implies that the within-cluster indirect effect is conditional, interpretable at the grand mean level of the omitted variable.

Multiple-set canonical correlation analysis and principal components analysis are popular data reduction techniques in various fields, including psychology. Both techniques aim to extract a series of weighted composites or components of observed variables for the purpose of data reduction. However, their objectives of performing data reduction are different. Multiple-set canonical correlation analysis focuses on describing the association among several sets of variables through data reduction, whereas principal components analysis concentrates on explaining the maximum variance of a single set of variables. In this paper, we provide a unified framework that combines these seemingly incompatible techniques. The proposed approach embraces the two techniques as special cases. More importantly, it permits a compromise between the techniques in yielding solutions. For instance, we may obtain components in such a way that they maximize the association among multiple data sets, while also accounting for the variance of each data set. We develop a single optimization function for parameter estimation, which is a weighted sum of two criteria for multiple-set canonical correlation analysis and principal components analysis. We minimize this function analytically. We conduct simulation studies to investigate the performance of the proposed approach based on synthetic data. We also apply the approach for the analysis of functional neuroimaging data to illustrate its empirical usefulness.

A new test is proposed for the problem of comparing two independent groups in terms of some measure of location. The proposed test () uses a one-step M-estimator and a bootstrap-t method with the procedure proposed by Özdemir and Kurt (2006). Eight methods were compared in terms of actual Type I error and power when the underlying distributions differ in skewness and kurtosis under heterogeneity of variances. For the 21 theoretical distributions, the Yuen test with the bootstrap-t method was the most favourable, followed by test. For the five real data sets, the proposed test and percentile bootstrap method with the one-step M-estimator performed best.

Laenen, Alonso, and Molenberghs (2007) and Laenen, Alonso, Molenberghs, and Vangeneugden (2009) proposed a method to assess the reliability of rating scales in a longitudinal context. The methodology is based on hierarchical linear models, and reliability coefficients are derived from the corresponding covariance matrices. However, finding a good parsimonious model to describe complex longitudinal data is a challenging task. Frequently, several models fit the data equally well, raising the problem of model selection uncertainty. When model uncertainty is high one may resort to model averaging, where inferences are based not on one but on an entire set of models. We explored the use of different model building strategies, including model averaging, in reliability estimation. We found that the approach introduced by Laenen et al. (2007, 2009) combined with some of these strategies may yield meaningful results in the presence of high model selection uncertainty and when all models are misspecified, in so far as some of them manage to capture the most salient features of the data. Nonetheless, when all models omit prominent regularities in the data, misleading results may be obtained. The main ideas are further illustrated on a case study in which the reliability of the Hamilton Anxiety Rating Scale is estimated. Importantly, the ambit of model selection uncertainty and model averaging transcends the specific setting studied in the paper and may be of interest in other areas of psychometrics.

Even though many educational and psychological tests are known to be multidimensional, little research has been done to address how to measure individual differences in change within an item response theory framework. In this paper, we suggest a generalized explanatory longitudinal item response model to measure individual differences in change. New longitudinal models for multidimensional tests and existing models for unidimensional tests are presented within this framework and implemented with software developed for generalized linear models. In addition to the measurement of change, the longitudinal models we present can also be used to explain individual differences in change scores for person groups (e.g., learning disabled students versus non-learning disabled students) and to model differences in item difficulties across item groups (e.g., number operation, measurement, and representation item groups in a mathematics test). An empirical example illustrates the use of the various models for measuring individual differences in change when there are person groups and multiple skill domains which lead to multidimensionality at a time point.