A Monte Carlo study comparing PIV, ULS and DWLS in the estimation of dichotomous confirmatory factor analysis

Abstract 

We conducted a Monte Carlo study to investigate the performance of the polychoric instrumental variable estimator (PIV) in comparison to unweighted least squares (ULS) and diagonally weighted least squares (DWLS) in the estimation of a confirmatory factor analysis model with dichotomous indicators. The simulation involved 144 conditions (1,000 replications per condition) that were defined by a combination of (a) two types of latent factor models, (b) four sample sizes (100, 250, 500, 1,000), (c) three factor loadings (low, moderate, strong), (d) three levels of non-normality (normal, moderately, and extremely non-normal), and (e) whether the factor model was correctly specified or misspecified. The results showed that when the model was correctly specified, PIV produced estimates that were as accurate as ULS and DWLS. Furthermore, the simulation showed that PIV was more robust to structural misspecifications than ULS and DWLS.

1. Introduction

Confirmatory factor analysis (CFA) is a widely used statistical tool in test development which allows researchers to test hypotheses about the structure of a scale. Typically, model parameters within CFA are estimated using maximum likelihood (ML) estimation (Browne, 1984; Bollen, 1989). ML assumes that the observed variables are continuous and that they follow a multivariate normal distribution, or equivalently that their covariance matrix has a Wishart distribution. However, test developers often work with items that have a binary response format (e.g. Nestler, Back, & Egloff, 2011). In this case, the assumptions behind ML are typically not met, leading to significant estimation problems, including, for instance, that the model parameters as well as their standard errors are inaccurately estimated (e.g. Babakus, Ferguson, & Jöreskog, 1987; Muthén & Kaplan, 1992).

Given these findings, a number of alternative estimation methods have been developed (Christofferson, 1975; Muthén, 1984; Jöreskog & Sörbom, 1996). All of these methods assume that a continuous latent variable underlies the observed responses to a binary item, and that the specified CFA model holds for these latent continuous variables and not the observed binary variables (see Bollen, 1989, pp. 439–445, for a general introduction). Model parameters are estimated by employing refined versions of a weighted least squares (WLS) approach (Browne, 1984); these versions draw on the tetrachoric correlations among the dichotomous items and on their asymptotic covariance matrix. Overall, robustness studies have shown that these estimation methods perform better than ML (e.g. Muthén & Kaplan, 1992; Beauducel & Herzberg, 2006). Simulations, however, have also found that they are only partially robust to misspecified factor models as they are system-wide estimators: they estimate all model parameters in one step (Bollen & Maydeu-Olivares, 2007).

The aim of the present paper is to compare the performance of a recently suggested alternative equation-by-equation estimator to the established approaches in the estimation of binary CFA models. Specifically, we compared the polychoric instrumental variable (PIV) estimator suggested by Bollen and Maydeu-Olivares (2007) to the unweighted least squares (ULS) estimator and the diagonally weighted least squares (DWLS) estimator in a Monte Carlo study. In the next section we introduce the dichotomous factor analysis model. We will then give a brief description of the standard estimation methods (e.g. ULS, DWLS) and introduce the PIV estimator.

2. The dichotomous factor analysis model

Consider a questionnaire consisting of n items y_j ( j = 1,…, n) that individuals rated by using two response alternatives. The dichotomous factor analysis model assumes (a) that a latent variable underlies the observed binary variable y_j, and (b) that both are related via a threshold relationship: y_j = 0 if _, and y_j = 1 if , respectively, where τ_j0 = −∞ and τ_j2 = ∞.

Furthermore, the latent responses y* are linear functions of m latent factors η:

• (1)

where y* denotes the n× 1 vector of latent response variables, Λ is an n×m matrix of factor loadings, η is an m× 1 vector of factors, and ɛ denotes an n× 1 vector of errors terms. By assumption, the factors and the errors are normally distributed, the expectation of both is zero, and the factors and the error terms are uncorrelated.

It follows, then, that the latent distributions are normally distributed, that they have an expectation of zero, and that their covariance matrix Σ is given by:

• (2)

where Ψ is the covariance matrix of the latent factors η. The covariance structure hypothesis thus holds for the latent distributions and not for the observed binary responses. To estimate the model parameters, Σ is substituted by the matrix of tetrachoric correlations between the binary responses, P, and Θ is set to Θ = I− ΛΨΛ^T. The latter is done to identify the model but has the consequence that the error variances are no longer model parameters, and, more generally, that a correlation structure is analysed (see Bentler & Salavei, 2010, for an overview on the analysis of correlation structures; see also Muthén, & Asparouhov, 2002, for an approach for analysing covariance structures when there are binary indicators). Furthermore, it can be shown that the dichotomous factor analysis model is mathematically equivalent to the normal ogive version of the two-parameter item response model and to the graded response model (see McDonald, 1999; Bollen, Bauer, Christ, & Edwards, 2010).

2.1. Standard estimation and testing approach

Imagine that we have a drawn a random sample of size N from the dichotomous factor model. Computing model parameters starts with the estimation of the thresholds, τ_j1, and the tetrachoric correlations among the items (Olsson, 1979). Then the asymptotic covariance matrix, , of the tetrachoric correlations is estimated. Finally, a least squares function F is minimized to estimate the model parameters (Muthén, 1978):

• (3)

where θ is the vector of (independent) model parameters, reflects the vector of estimated tetrachoric correlations, ρ(θ) denotes the restrictions imposed on the population tetrachoric correlations by the parameter vector θ, and , is a positive definite weight matrix.

The standard approaches to estimating model parameters differ in their choice of the weight matrix W. In ULS (Muthén, 1978, 1984), to begin with, W is set to an identity matrix (i.e. W = I). In WLS (Browne, 1984), by contrast, W is the inverse of the estimated asymptotic covariance matrix of the tetrachoric correlations (i.e. ). Finally, in DWLS (Jöreskog & Sörbom, 1996), W is set to . Note that the name that Flora and Curran (2004) and Muthén, du Toit, and Spisic (1997) use for DWLS is robust WLS. Also, they use a diagonal matrix V instead of W to estimate the model parameters. In contrast to W, V includes not only the asymptotic variances of the tetrachoric correlations, but also the asymptotic variances of the thresholds. However, all approaches – including ULS, WLS, DWLS, and robust WLS – use the full asymptotic covariance matrix, , to obtain standard errors and the chi-square test statistic.

All three estimators can be employed to obtain model parameters in popular software packages such as LISREL (Jöreskog & Sörbom, 1996) or Mplus (Muthén & Muthén, 1998). In both programs, ULS and WLS are termed accordingly. DWLS, however, is called robust DWLS in LISREL and WLSMV in Mplus. Additionally, although both programs use the same method to estimate the tetrachoric correlation, they differ in their estimation of the asymptotic covariance matrix. Specifically, whereas LISREL employs the procedure described in Jöreskog (1994), Mplus uses the method described in Muthén (1984). The two methods differ in their treatment of the threshold parameters, but are asymptotically equivalent (Muthén & Satorra, 1995). Also, Dolan (1994) showed that they produced similar results in the case of CFA even with a sample size of N = 200.

ULS, WLS and DWLS yield consistent estimates of the model parameters that are asymptotically normal, and asymptotically correct standard errors can be computed (see Bollen & Maydeu-Olivares, 2007). Furthermore, simulation studies have found that whereas WLS performs adequately only for large sample sizes, DWLS and ULS perform well for small ones too (e.g. Dolan, 1994; Flora & Curran, 2004; Beauducel & Herzberg, 2006; Forero, Maydeu-Olivares, & Gallardo-Pujol, 2009). Therefore, both are typically used to estimate the parameters of a CFA model. However, recent simulations have found that they are only partially robust to structural misspecifications (Bollen & Maydeu-Olivares, 2007), and both lead to biased parameter estimates and standard errors when the factor loadings are small, when the latent continuous variables are skewed, and when there are only a few factor indicators (Forero et al., 2009).

2.2. Polychoric instrumental variable estimator

Given these results, and given that these conditions are typically found in applied settings, the performance of alternative estimation methods should be investigated. One of the alternatives is the polychoric instrumental variable estimator recently proposed by Bollen and Maydeu-Olivares (2007). The basic idea behind the PIV estimator is to compute the factor loadings in a first step and to estimate the variance and covariance model parameters – based on the values of the factor loadings – in a second step. Specifically, whereas the factor loadings are obtained using a non-iterative procedure employing instrumental variables (IVs), the variance–covariance parameters are estimated using an iterative procedure.

IV estimation is a special case of generalized method of moments estimation (Baum, Schaffer, & Stillman, 2003; Hall, 2005) and is typically employed in econometrics (e.g. Angrist, Imbens, & Rubins, 1996) to investigate the effect of a predictor on a criterion in a linear regression when the predictor is measured with error or is systematically related to other determinants of the criterion. In this case, the assumptions of ordinary least squares regression are not met, and the model parameters cannot be consistently estimated. A solution to this problem is to find one or more exogenous variables that affect the predictor but not the criterion (cf. Morgan & Winship, 2007), and to use these IVs to compute consistent versions of the regression parameters.

Bollen (1996) extended this approach to structural equation models (SEMs) for continuous variables.¹ Applied to a CFA model, he showed that each indicator is a linear function of the factor loadings of this indicator, the indicator used to scale the latent factor, and a composite disturbance term containing the error of the scaling indicator and the error of the actual indicator. Given that the scaling indicator is correlated with the composite disturbance term – the disturbance contains this error term – IVs are used to compute the factor loadings. Specifically, in a two-stage regression (2SLS), the scaling indicator is first regressed on the IVs. The regression coefficients obtained are then used to compute predicted values of the scaling indicator, and the actual indicator is then regressed on these predicted values. The regression coefficient of this second regression is the desired factor loading.

The 2SLS/IV estimator yields consistent estimates of the factor loadings when the IVs meet certain requirements. The IVs must be (a) correlated with the scaling indicator, (b) unrelated to the errors in the composite disturbance term, and (c) sufficient in number so that there are at least as many IVs as there are scaling indicators (cf. Bollen, 1996; Bollen & Maydeu-Olivares, 2007).² If the CFA model is identified, the potential IVs are a subset of the other indicators, and the structure of the CFA model can be used to determine which of these indicators satisfy conditions (a) and (b). The IVs are thus model implied, and it is this feature that differentiates Bollen's 2SLS/IV approach from the typical usage of IVs in econometric contexts where the IVs are taken from outside of the model. To indicate this conceptual difference, the 2SLS/IV approach is called model-implied instrumental variables (MIIV) in more recent publications (e.g. Bollen & Bauer, 2004).

Recently, Bollen and Maydeu-Olivares (2007) generalized the MIIV estimator to categorical variables. As in the case of continuous indicators, factor loadings are first computed using IVs in a two-stage regression, whereby these computations draw on the tetrachoric correlations among the indicators. Once the factor loadings have been obtained, variances of the factors and the covariance among them are computed. Therefore, the factor loadings are entered into equation (3), and the ULS variant of F is used to estimate the remaining model parameters. Finally, Bollen and Maydeu-Olivares (2007) also provided formulae to compute standard errors for the factor loadings and for the variance and covariance model parameters.

2.3. The current simulation

To date, only Bollen and Maydeu-Olivares (2007) have done a simulation study in which the performance of PIV relative to ULS has been investigated. In their study, they used a two-factor model with high factor loadings (λ = .80), and the model was either correctly or incorrectly specified. They found that PIV provided parameter estimates that were as accurate as estimates obtained using ULS when the model was correctly specified. When the CFA model was incorrectly specified (the cross-loading of an item was set to zero), PIV produced more accurate parameter estimates compared to ULS. The aim of the present research was to replicate and extend these findings. Specifically, we examined the performance of PIV under different settings of factor model size, factor loading magnitude (i.e. indicator reliability), sample size, and non-normality of the latent distributions (see Table 1 for an overview of design factors).³ Furthermore, we investigated the effect of a structural misspecification. Finally, we compared the performance of PIV not only to ULS but also to DWLS, as the latter is also widely used in estimating dichotomous CFA models.

Table 1. Factors investigated in the current simulation

Factor	Levels
Model size	2 levels: two-factor model or three-factor model
Factor loading	3 levels: λ = .40, .55, .70
Sample size	4 levels: N = 100, 250, 500, 1,000
Latent distributions	3 levels: normal, moderately or extremely non-normal
Misspecification	2 levels: no or yes

3. Method

3.1. Outcome variables and data generation

3.1.1. Continuous latent responses

In each of the factor loading conditions, three continuous latent response population distributions (N = 20,000) were generated using the R package mvtnorm (Genz et al., 2011). The first distribution was multivariate normal with skewness and kurtosis of zero. The second distribution differed moderately from normality by having a skewness of 2 and a kurtosis of 7, and the final distribution was extremely non-normal with a univariate skewness of 3 and kurtosis of 20. These parameters were chosen as they are representative of levels of non-normality typically encountered in applied research (see Flora & Curran, 2004; Yu, 2002). To simulate the latent distributions with non-zero skewness and kurtosis, we used the procedures suggested by Fleishman (1978) and Vale and Maurelli (1983). For each of the three distributions, 1,000 random samples of four different sample sizes were drawn: 100, 200, 500, and 1,000.

3.1.2. Binary observed responses

After drawing (continuous) samples from the latent distributions, we transformed each latent response variable into a dichotomous variable by dichotomizing it at a threshold of τ = 0.

3.2. Model specification

The population correlation matrix on which samples were based conformed either to a model consisting of two correlated factors each measured by four indicators (see Figure 1(a)) or to a model consisting of three correlated factors each measured by four indicators (see Figure 1(b)). Depending on the factor loading condition, factor loadings on the true latent factor were set to either λ = .70, λ = .55, or λ = .40; otherwise they were set to λ = 0. The uniquenesses were all set to either θ = .51 in the high factor-loading condition, θ = .69 in the moderate, or θ = .84 in low. Finally, the population covariance between the latent factors was set to 0.3, and the variances of the factors were set to 1.

Figure 1. (a) Two-factor model and (b) three-factor model simulated in the present study. Variables in circles refer to latent variables (i.e. latent factors, η_i, or latent continuous response variables, y_i*, respectively), and variables in rectangles to dichotomous response variables. In the case of the misspecified two-factor model, the fourth and eighth indicators were set to load on the second and the first latent factors, respectively (see dotted lines). For the misspecified three-factor model, the fourth (eighth, twelfth) indicator was set to load on the third (first, second) latent factor (again see dotted lines).

In all replications, two model specifications were tested (again see Figure 1). The first model was correctly specified such that all indicators were allowed to load on their respective true factor and the two or three factors, respectively, were allowed to correlate. In the case of the incorrectly specified factor model, one observed indicator of each factor in the true model was specified to load on the other, wrong, factor, even though the factor loadings on the respective incorrect factors were zero in the population. With respect to the two-factor model, for example, item 4 was set to load on factor 2, and item 8 was set to load on factor 1.

3.3. Data analysis

3.3.1. Parameter estimation

Parameter estimates and standard errors were computed using R (R Development Core Team, 2011; the R code can be found in the supplementary materials, available with the online version of this paper). Specifically, thresholds and tetrachoric correlations were obtained by employing the two-stage procedure described in Olsson (1979). Then, an estimate of the asymptotic covariance matrix, , was computed by implementing the procedures of Jöreskog (1994) and Christofferson and Gunsjö (1996). In the case of ULS, DWLS, and PIV (when the variances of the factors and the covariance between them had to be estimated), the function F was minimized using the R function nlminb(). Factor loadings were estimated with a starting value of 1.00, variance parameters with a starting value of 0.05, and for covariance parameters, the starting value was set to 0.⁴

Finally, to obtain the factor loadings according to the PIV estimator, we selected IVs that satisfied all three requirements mentioned earlier. Consider, for example, the estimation of the factor loading of the second indicator in the case of the two-factor model (see Figure 1(a)). To estimate it, IVs are needed to predict the scaling indicator of the respective factor (i.e. ). These IVs must be (a) correlated with and (b) uncorrelated with the disturbance term that contains the error terms of and . All other indicators, , meet these two requirements as either they load on the same latent factor or the two latent factors are correlated, and all errors are uncorrelated. Of this set of potential IVs, finally, we used the IVs that should be most strongly – according to the CFA model – related to the scaling indicator. In the case of the second indicator, and were chosen to compute the factor loading (see Table 2 for the IVs for the other indicators).

Table 2. Instrumental variables (IV) used to estimate the factor loading of the indicator y* in the two-factor model

Correctly specified			Incorrectly specified
y*	z	IV	y*	z	IV
2	1	3, 4	2	1	3, 8
3	1	2, 4	3	1	2, 8
4	1	2, 3	4	5	6, 7
6	5	7, 8	6	5	4, 7
7	5	6, 8	7	5	4, 6
8	5	6, 7	8	1	2, 3

3.3.2. Goodness-of-fit statistic

In addition to examining the accuracy of the three estimators, we also investigated how well an estimator-specific goodness-of-fit statistic performed in the different conditions. These goodness-of-fit tests were based on the test statistic , where N denotes the size of the sample, and is the value of the fit function with the estimated parameter values inserted (see Equation 3). When the weight matrix W is correctly specified, that is, , then T is asymptotically χ²-distributed with d degrees of freedom. If this assumption is not met, as in the case of ULS and DWLS, T is a mixture of d independent one-degree-of-freedom chi-square variables. A goodness-of-fit test can then be obtained by rescaling T so that, for example, its mean and variance equal those of a chi-square variate with d degrees of freedom. In the latter case, T is generally not χ²-distributed, but can well be approximated by a χ² distribution with d degrees of freedom (Bentler & Savalei, 2010, p. 14). We obtained these mean- and-variance-adjusted statistics for the ULS and the DWLS estimators by implementing the formula as suggested, for instance, in Bentler and Savalei (2010, equation 1.25); for the PIV estimator, we used the equations given in Bollen and Maydeu-Oilvares (2007). As before, we used R to compute the test statistics.

4. Results

The results for the two- and three-factor models were very similar. Therefore, we present the results of the two-factor model only (see Figure 1(a)). Tables that report the results for the three-factor model can be found in the supplementary materials. Also, the results in each simulated condition are included in the supplementary materials. In Section 4.1 present the results concerning convergence failures. Then we go on to report the results concerning the performance of the three estimation methods in parameter estimation (Section 4.2) and standard error estimation (Section 4.3). Finally, in Section 4.4, we report the results concerning the test statistic.

4.1. Convergence failures

Table 3 displays the percentage of convergence failures depending upon estimator, magnitude of true factor loading, and sample size. When the factor model was correctly specified, most convergence failures emerged in conditions of a small sample size and a small true factor loading. This effect was more pronounced for ULS and DWLS than for PIV. Overall, PIV yielded a very small rate of convergence failures. A similar result pattern emerged for the misspecified model, although failures to converge arose here also in conditions of moderate and high true factor loadings. Again, PIV yielded the smallest rate of convergence failures. Finally, non-normality had no systematic effect in either model specification condition.

Table 3. Rates (%) of convergence failures for correctly and incorrectly specified models depending on estimator, magnitude of true factor loading (λ) and sample size (N)

λ	N	ULS		DWLS		PIV
		Correct	Incorrect	Correct	Incorrect	Correct	Incorrect
.40	100	21.9	26.7	21.4	25.9	0.1	0.1
	250	3.9	11.5	3.7	10.7	0.1	0.1
	500	0.2	2.8	0.2	2.4	0.0	0.1
	1,000	0.0	0.3	0.0	0.3	0.0	0.1
.55	100	1.8	8.5	2.1	8.0	0.1	0.1
	250	0.0	0.8	0.0	1.2	0.0	0.0
	500	0.0	0.1	0.0	0.1	0.0	0.0
	1,000	0.0	0.0	0.0	0.0	0.0	0.0
.70	100	0.0	2.7	0.4	1.9	0.0	0.0
	250	0.0	0.2	0.0	0.2	0.0	0.0
	500	0.0	0.0	0.0	0.0	0.0	0.0
	1,000	0.0	0.0	0.0	0.0	0.0	0.0

In accordance with earlier research, cases in which the estimator did not converge were removed from the analysis. In contrast to earlier research, however, we kept replications in which Heywood cases occurred in the analysis. This was done as Heywood cases reflect natural variability in parameter estimates, and excluding them thus leads to biased parameter means and parameter variances (see Savalei & Kolenikov, 2008).

4.2. Relative bias of parameter estimates

To compare the performance of the three estimators, we computed the relative bias of the parameter estimates. To this end, a mean factor loading per replication was obtained by averaging the estimated factor loadings across the indicators. For the misspecified factor model, two mean factor loadings per condition were computed: one for the factor loadings of the correctly specified indicators and one for the incorrectly specified indicators. The relative bias of these mean estimates, and of the covariance, was then computed using , where denotes the estimated parameter and θ the true parameter value. Relative bias below 10% was considered acceptable. Relative bias of 10–20% was considered substantial, and above 20% unacceptable (see Forero et al., 2009).

Examining the relative bias of factor loadings in the correctly specified models, first, we found (see Table 4) that PIV tended to underestimate the true factor loading, whereas the other two estimators tended to overestimate it. Overall, PIV produced less bias (−4.4%) than ULS (10.6%) and DWLS (10.4%). As can be seen in Table 4, PIV yielded more accurate factor loading estimates in almost all conditions of low or moderate true factor loadings. When the factor loading was high, ULS and DWLS outperformed PIV. Besides these effects, the magnitude of the true factor loading had a negative influence on relative bias (low, 13.6%; moderate, 3.3%; high, −0.2%) and sample size had a positive influence (N = 100, 11.9%; N = 250, 6.5%; N = 500, 2.6%; N = 1,000, 1.1%). Importantly, the negative effect of a small sample size was stronger the lower the true factor loading. This result pattern held for all estimators alike, although it was more pronounced for ULS and DWLS. Finally, non-normality had negligible effects on the accuracy of factor loading estimates.

Table 4. Average relative bias (%) of factor loading estimates for correctly and incorrectly specified models depending on estimator, magnitude of true factor loading (λ) and sample size (N)

λ	N	ULS		DWLS		PIV
		Correct	Incorrect	Correct	Incorrect	Correct	Incorrect
Note. In the case of incorrect model specification, average relative bias refers to correctly specified factor loadings.
.40	100	66	45	63	44	−31	−49
	250	29	26	31	28	−6.1	−13
	500	11	12	11	11	0.3	0.4
	1,000	4.9	5.2	4.9	5.1	1.6	3.9
.55	100	20	15.9	17	16	−6.0	−10.2
	250	4.3	6.0	4.1	5.5	−1.9	−3.3
	500	1.8	1.9	1.8	1.7	−0.8	−2.5
	1,000	0.4	−0.1	0.4	−0.1	−0.7	−2.8
.70	100	2.9	6.2	2.4	4.5	−3.9	−3.2
	250	0.8	2.1	0.6	1.4	−1.9	−1.3
	500	0.1	1.0	−0.1	0.4	−1.5	−1.4
	1,000	−0.1	0.3	−0.3	−0.1	−1.1	−1.2

For the covariance estimate, we found that all three estimators yielded substantial biases, albeit PIV was more accurate (12.1%) than ULS (17.1%) and DWLS (19.5%). Also, the magnitude of the true factor loading (low, 19.5%; moderate, 14.7%; high, 14.6%), and the non-normality of the latent distributions had an influence on the quality of parameter estimates (normal, −0.1%; moderately non-normal, 15.2%; extremely non-normal, 34.2%). As can be seen in Table 5, PIV produced high biases in all conditions of a low true factor loading. In the other two factor loadings conditons, biases were greater the more non-normal the latent distributions were. For ULS and DWLS the negative effect of the non-normality of the latent distributions holds in all true factor loading conditions alike. Finally, biases were smaller the larger the size of the samples (N = 100, 19.9%; N = 250, 16.2%; N = 500, 14.8%; N = 1,000, 14.1%).

Table 5. Average relative bias (%) of covariance estimates for correctly and incorrectly specified models depending on estimator, magnitude of true factor loading (λ) and non-normality (NN)

λ	NN	ULS		DWLS		PIV
		Correct	Incorrect	Correct	Incorrect	Correct	Incorrect
Note. Levels of factor non-normality are 1 = multivariate normal, 2 = moderately non-normal, and 3 = extremely non-normal.
.40	1	−0.4	76	1.1	78	−17	10.1
	2	31	114	33	120	16	63
	3	41	144	45	153	26	74
.55	1	0.5	80	2.1	90	−1.3	57
	2	9.7	96	12	110	8.4	62
	3	32	129	35	149	33	106
.70	1	2.0	87	3.3	113	1.9	58
	2	7.8	94	10.3	127	10.1	70
	3	31	121	34	156	32	99

Concerning the incorrectly specified model (Tables 4 and 5), ULS as well as DWLS overestimated the loadings of the correctly specified items (9.1% and 8.8%, respectively), whereas PIV underestimated them (–6.9%). Furthermore, as in the case of the correctly specified model, relative biases were larger the smaller the true factor loading (low, 7.9%; moderate, 2.2%; high, 0.7%), and this effect was larger the smaller the size of the sample. When the true factor loading was moderate or high, all three estimators yielded acceptable biases at N = 250. Finally, non-normality of the latent distributions again had a negligible effect. For the falsely specified items (see Table 6), PIV produced the most accurate estimates as compared to ULS and DWLS. As can be seen, sample size had a positive effect for ULS and DWLS but not for PIV, and PIV estimates got larger – and hence more biased – the greater the magnitude of the true factor loading. Finally, for the covariance estimate, all three estimators produced unacceptable average biases in almost all conditions. However, it is noteworthy that this bias was lower – but still unacceptable – when PIV was used (see Table 5).

Table 6. Means of the factor loading estimates of the incorrectly specified indicators depending on estimator, magnitude of true factor loading (λ), and sample size (N)

λ	N	ULS	DWLS	PIV
.40	100	.84	.83	.08
	250	.67	.66	.12
	500	.50	.49	.14
	1,000	.40	.40	.13
.55	100	.85	.79	.18
	250	.65	.61	.19
	500	.52	.53	.18
	1,000	.48	.49	.18
.70	100	.81	.76	.23
	250	.63	.65	.23
	500	.59	.63	.23
	1,000	.58	.62	.22

4.3. Relative bias of standard errors

We computed the relative bias of standard errors for the factor loadings and the covariance using . Here, SE is the average standard error of the parameter estimates across valid replications, and is the standard deviation of the respective parameter estimate.

Concerning the standard errors of the factor loadings in the correctly specified factor model, PIV yielded worse estimates (10.7%) than ULS (−4.8%) and DWLS (−5.7%). As a closer inspection of Table 7 reveals, this result is mainly due to unacceptable and substantial standard errors in conditions of a low true factor loading and a small sample size (i.e. N = 100 and N = 250); in all other conditions, PIV produced acceptable average standard errors. For ULS and DWLS, acceptable average standard errors emerged at N = 1,000 when the true factor loading was low and at N = 250 when the true factor loading was moderate. For the covariance, similar biases in standard errors occurred for all three estimators. Specifically, these were substantial and unacceptable when the true factor loading was low or moderate and sample size was small (N = 100 or N = 250). Finally, for both parameters, non-normality had no systematic effects on standard error estimates.

Table 7. Average relative bias (%) of standard errors depending on estimator, correctly or incorrectly specified models (MSP), magnitude of true factor loading (λ), and sample size (N)

MSP	λ	N	ULS			DWLS			PIV
			λT	λF	ψ	λT	λF	ψ	λT	λF	ψ
Note. λT and λF, respectively, refer to the average standard error bias concerning the correctly specified or falsely specified indicators. Levels of the factor model misspecification are C = correct, and IC = incorrect.
C	.40	100	25		−23	22		−25	75		34
		250	−19		−15	−19		−16	12		14
		500	−15		−7.9	−19		−8.6	3.6		5.8
		1,000	−3.6		−3.5	−3.8		−3.8	4.7		4.3
	.55	100	−25		−13	−27		−17	9.0		6.5
		250	−5.6		−6.2	−6.3		−7.6	3.8		−0.3
		500	−1.8		−0.6	−2.1		−1.3	2.7		2.5
		1,000	1.6		0.4	1.3		0.1	3.4		2.1
	.70	100	−8.1		−5.5	−9.1		−9.9	5.6		−2.3
		250	−1.7		−2.8	−2.1		−4.4	2.8		−0.3
		500	−0.1		1.5	−0.1		0.1	2.9		1.9
		1,000	1.5		3.3	1.6		2.8	3.1		3.7
IC	.40	100	34	36	−8.7	31	29	−10.1	79	70	32
		250	7.0	−2.6	−6.4	6.5	0.1	−8.4	43	32	12
		500	−11	−35	−5.1	−13	−38	−6.0	6.2	13	5.1
		1,000	−3.8	−49	−1.8	−3.1	−49	−1.8	5.6	3.6	−4.2
	.55	100	5.9	−5.2	−8.0	−6.0	−18	−10.9	22	11	−2.1
		250	−9.2	−44	−6.7	−7.0	−49	−6.3	4.1	3.5	−13
		500	−3.0	−61	1.1	−1.1	−58	1.7	3.3	3.7	−13
		1,000	11.4	−48	6.1	8.6	−45	6.3	4.8	3.1	−14
	.70	100	−6.2	−41	−8.7	−22	−49	−10	4.5	3.7	−19
		250	2.4	−57	−5.5	−4.0	−45	−4.0	2.0	2.2	−23
		500	8.6	−47	6.3	−0.1	−41	2.7	1.3	2.3	−22
		1,000	10.1	−42	9.1	1.5	−39	3.4	2.1	4.1	−20

Regarding the misspecified factor model (Table 7), PIV produced acceptable standard errors for both factor loading estimates at N = 1,000 when the true factor loading was low and at N = 250 when the true factor loading was moderate. For ULS and DWLS standard error bias was unacceptable at N = 100 for the correctly specified indicators. In the case of the falsely specified items both estimators produced unacceptable standard errors in almost all conditions. Concerning the covariance, whereas standard errors were substantial or unacceptable in almost all conditions for PIV, they were acceptable in almost all conditions for the other two estimators.

4.4. Chi-square test statistic

Table 8 presents means and rejection rates (at α = .05) for the chi-square test statistics depending upon estimator, magnitude of true factor loading, and sample size (non-normality had no systematic effects). For correctly specified models, overall rejection rates were near the cut-off value for ULS (4.8%) and DWLS (5.7%), and slightly lower for PIV (3.1%). As can be seen in Table 8, the test statistic based on the PIV estimates tended to under-reject models in almost all conditions. For ULS and DWLS, by contrast, under-rejections only occurred in conditions with low factor loadings and small sample sizes. Regarding the misspecified factor models, the test statistic yielded reasonable rejection rates for all three estimators except when the true factor loading was small and the sample size was small (N = 100 or N = 250) or when it was moderate and sample size was small (N = 100).

Table 8. Mean of the chi-square test statistic and type 1 error rate (at α= 5%) depending on estimator, correctly or incorrectly specified models, magnitude of true factor loading (λ), and sample size (N)

λ	N	ULS				DWLS				PIV
		Correct		Incorrect		Correct		Incorrect		Correct		Incorrect
		χ	5%	χ	5%	χ	5%	χ	5%	χ	5%	χ	5%
.40	100	13.9	2.2	15.9	5.4	14.7	4.4	16.8	9.4	6.4	0.7	6.9	1.6
	250	16.6	3.9	25.0	35	16.9	4.7	25.7	38	9.8	1.2	12.9	18
	500	17.6	4.9	38.1	78	17.8	5.3	38.8	79	13.3	1.7	25.3	58
	1,000	17.9	5.0	63.2	99	18.0	5.2	64.1	99	15.9	2.8	53.5	96
.55	100	14.1	4.3	25.3	53	14.9	7.0	27.0	60	9.5	1.8	16.2	35
	250	15.8	4.0	53.8	98	16.2	5.0	55.6	98	13.4	2.1	44.4	92
	500	16.6	5.1	100	100	16.9	5.6	102	100	15.3	3.7	93.2	99
	1,000	16.8	5.4	192	100	17.0	5.6	194	100	15.8	4.4	188	100
.70	100	12.7	5.2	45.5	98	13.5	7.8	47.2	98	10.9	3.2	40.9	93
	250	13.9	5.0	112	100	14.5	5.5	112	100	12.9	4.1	112	100
	500	14.6	5.2	223	100	15.1	5.6	221	100	13.6	4.9	230	100
	1,000	14.9	6.2	441	100	15.4	6.3	434	100	13.9	6.1	463	100

5. Discussion

The present simulation study was implemented to investigate the performance of the polychoric instrumental variable estimator in the estimation of a dichotomous confirmatory factor analysis. Specifically, we examined how (a) the factor model size, (b) the magnitude of the true factor loading, (c) the non-normality of the latent distributions, and (d) the size of the sample affected the performance of PIV. Also, its robustness to a model misspecification was tested. Finally, we compared the performance of PIV to two other well-established system-wide estimators, namely, ULS and DWLS.

The results showed, first, that PIV (like ULS and DWLS) provided accurate parameter estimates in most conditions when the model was correctly specified. In some of these conditions, for instance, when the true factor loading was small or moderate, PIV even provided more accurate estimates than ULS and DWLS. Furthermore, we found that the magnitude of the true factor loading had an impact on PIV factor loading estimates and PIV covariance estimates. Non-normality of the latent distributions, by contrast, only affected PIV covariance estimates. Importantly, both of these effects also arose for ULS and DWLS. Thus, overall, PIV produced estimates that were as accurate as ULS and DWLS, and it even outperformed ULS and DWLS in estimation accuracy in some conditions.

Second, when the model was misspecified, the quality of the parameter estimates for the correctly specified factor loading was the same for all three estimators. PIV, however, was more robust to the structural misspecification than ULS and DWLS for the falsely specified factor loadings. This result, however, is to be expected given that ULS and DWLS use all the available information to estimate the model parameters, including, for instance, not only the assignment of an indicator to a latent factor but also whether the latent factors are correlated. PIV, by contrast, employs only instrumental variables (IVs) – the other indicators – to compute the factor loadings. Also, given that the magnitude of an IV estimate depends on the strength of the relation between the IV and the predictor, the bias of the falsely specified factor loading is expected to increase the higher the correlations between the falsely assigned indicator and the other indicators used to compute the factor loading estimate. Note that this not only explains why the bias in parameter estimates got larger the higher the magnitude of the true factor loading, but also suggests that PIV will perform even better the more the correlation between the latent factors approaches zero. Finally, although PIV provided more accurate covariance estimates than ULS and DWLS, these were nevertheless unacceptable in almost all conditions. Again, this result is to be expected as the PIV estimator is most probably robust for the factor loadings but need not be robust for the covariance parameter (see Bollen & Maydeu-Olivares, 2007, for more information concerning the conditions on the robustness of variance–covariance parameters).

The present findings are thus consistent with earlier results showing that PIV as well as MIIV are more robust against structural misspecifications than other system-wide estimators. Although these results are important, they may be of limited value in applied contexts, as it may be more crucial to locate the structural misspecification in applied work (Saris, Satorra, & van der Veld, 2009). We believe, however, that even for this question an application of the MIIV approach is suitable. Specifically, recent evidence suggests that the over-identification tests (see footnote 2) used to test the appropriateness of IVs are a good means for diagnosing the source of a misspecification in the case of structural equation models with continuous indicators (Kirby & Bollen, 2009). The idea behind this approach is that for MIIV, the IVs are unambiguously determined by the hypothesized model structure. When an over-identification test concerning a set of IVs thus fails, this implies that the model structure that suggested these IVs must be misspecified. We think that it is an interesting task for future research to investigate whether this approach is also suitable for SEMs with ordinal indicators.

Third, concerning standard error bias, the present simulation found that ULS and DWLS provided more accurate estimates than PIV when the model was correctly specified. This was, however, mainly due to biased standard errors in cases of a small true factor loading. A potential explanation of this might be that the correlations between the two indicators that served as IVs and the indicator for which the factor loading had to be estimated were too low. This weakness of the IV has been found to influence the asymptotic behaviour of IV estimators before (see Baum et al., 2003; Bound, Jaeger, & Baker, 1995).

Finally, the test statistics for all three estimators performed well in the case of the misspecified factor model, although all three tended to under-reject the wrong factor model when the true factor loading was small and the size of the sample was smaller than N = 1,000.

PIV also tended to under-reject models when the model was correctly specified. Here, better results emerged for the other two estimation methods.

In short, then, the present work shows that the PIV estimator is an interesting alternative method for the estimation of the parameters of a dichotomous confirmatory factor analysis. When the factor model was correctly specified, PIV yielded results similar to ULS and DWLS in most cases, and it was more robust to structural misspecifications than ULS and DWLS. However, the present simulation is just a first step in investigating the performance of the PIV estimator. It would be a worthwhile task for future research to examine PIV in other contexts, such as full structural equation models or models with exogenous observed covariates, to conclusively answer whether it is a true alternative to the more common system-wide estimators.

Acknowledgements

We thank Boris Egloff, Stanislav Kolenikov, Stefan Schmukle and three anonymous reviewers for very helpful comments on an earlier version of this article.

Footnotes

• 1

  A number of earlier accounts used a two-stage least squares 2SLS/IV approach to obtain model parameters. Madansky (1964) used IVs to estimate the factor loadings in an exploratory factor analysis. Hägglund (1982) and Jöreskog (1983) extended this approach to factor analysis models with uncorrelated errors. Finally, Jöreskog and Sörbom (1996) suggested a 2SLS estimator for the parameters of the structural part of an SEM. First, a 2SLS/IV approach is first used to estimate the factor loadings of the measurement models together with the covariance matrix of the latent factors. Second, this matrix is then used by a 2SLS estimator to estimate the parameters of the structural part of the SEM. Bollen's (1996) 2SLS/IV approach differs from all three approaches in that it allows for correlated errors and provides the asymptotic covariance matrix of the model parameters. Finally, it differs from Jöreskog and Sörbom's (1996) approach in that Bollen's method estimates the parameters of the structural part of the SEM without estimating the measurement model first.

• 2

  If there are as many IVs as there are scaling indicators, the equation is said to be exactly identified; if there are more IVs than scaling indicators, the equation is called over-identified. In the latter case, a validity test can be performed to determine whether a set of IVs is uncorrelated with the composite disturbance term (see Sargan, 1958, for the test statistic; or Baum et al., 2003, for variants of this test).

• 3

  We ignored different conditions of model size (i.e. the number of indicators) in our simulation as other simulation research showed that the effect of model size on parameter estimates and standard errors is notably small unless the number of indicators is fewer than five (Flora & Curran, 2004; Forero et al., 2009).

• 4

  To obtain starting values for the factor loading estimates, other software packages such as Mplus (Version 5 and higher), LISREL, or the Stata confa module (Kolenikov, 2009) use IV-based approaches. We did not use such an approach here as we wanted to compare the performance of the three estimators given the same initial (estimation) situation.