Critical Values for Yen's Q3: Identification of Local Dependence in the Rasch Model Using Residual Correlations

Abstract

The assumption of local independence is central to all item response theory (IRT) models. Violations can lead to inflated estimates of reliability and problems with construct validity. For the most widely used fit statistic Q3, there are currently no well-documented suggestions of the critical values which should be used to indicate local dependence (LD), and for this reason, a variety of arbitrary rules of thumb are used. In this study, an empirical data example and Monte Carlo simulation were used to investigate the different factors that can influence the null distribution of residual correlations, with the objective of proposing guidelines that researchers and practitioners can follow when making decisions about LD during scale development and validation. A parametric bootstrapping procedure should be implemented in each separate situation to obtain the critical value of LD applicable to the data set, and provide example critical values for a number of data structure situations. The results show that for the Q3 fit statistic, no single critical value is appropriate for all situations, as the percentiles in the empirical null distribution are influenced by the number of items, the sample size, and the number of response categories. Furthermore, the results show that LD should be considered relative to the average observed residual correlation, rather than to a uniform value, as this results in more stable percentiles for the null distribution of an adjusted fit statistic.

Keywords: Monte Carlo simulation; Rasch model; Yen’s Q3; local dependence; residual correlations.

Conflict of interest statement

Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Figures

Figure 1.

The empirical 95th and 99th percentiles in the empirical distribution of Q3,max for dichotomous items. Note. Gray horizontal dashed lines indicate 0.2 and 0.3, respectively.

Figure 2.

The empirical 95th and 99th percentiles in the empirical distribution of Q3,max for polytomous items. Note. Gray horizontal dashed lines indicate 0.2 and 0.3, respectively.

Figure 3.

The empirical 95th and 99th percentiles in the empirical distribution of Q3,* for dichotomous items. Note. Gray horizontal dashed lines indicate 0.2 and 0.3, respectively.

Figure 4.

The empirical 95th and 99th percentiles in the empirical distribution of Q3,* for polytomous items. Note. Gray horizontal dashed lines indicate 0.2 and 0.3, respectively.

Figure 5.

The empirical distribution of Q3,max based on 10,000 data sets simulated using item and person parameters from the Makransky and Bilenberg (2014) data.

Figure 6.

The empirical distribution of Q3,* based on 10,000 data sets simulated using item and person parameters from the Makransky and Bilenberg (2014) data.

Figure 7.

The test information in four models with item deletion and in the model with item combination.