Impact of Not Fully Addressing Cross-Classified Multilevel Structure in Testing Measurement Invariance and Conducting Multilevel Mixture Modeling within Structural Equation Modeling Framework

In educational settings, researchers are likely to encounter multilevel data without strictly nested or hierarchical but cross-classified multilevel structure. However, due to the lack of familiarity and limitations of statistical software with cross-classified model, most substantive researchers adopt then the less optimal approaches to analyze cross-classified multilevel data. Two separate Monte Carlo studies were conducted to evaluate the impacts of misspecifying cross-classified structure data as hierarchical structure data in two different analytical settings under the structural equation modeling (SEM) framework.

Study 1 evaluated the performance of conventional multilevel confirmatory factor analysis (MCFA) which assumes hierarchical multilevel data in testing measurement invariance, especially when the noninvariance exists at the between-level groups. We considered two design factors, intra-class correlation (ICC) and magnitude of factor loading differences. This simulation study showed low empirical power in detecting noninvariance under low ICC conditions. Furthermore, the low power was plausibly related to the underestimated ICC and the underestimated factor loading differences due to the redistribution of the variance component from the crossed factor ignored in the analysis.

Study 2 examined the performance of conventional multilevel mixture models (MMMs), which assume hierarchical multilevel data, on the classification accuracy of class enumeration and individuals? class assignment when the latent class variable is at the between (cluster)-level. We considered a set of study conditions, including cluster size, degree of partial cross-classification, and mixing proportion of subgroups. From the results of the study, ignoring a crossed factor caused overestimation of the variance component of the remaining crossed factor at the between-level which was redistributed from the ignored crossed factor in the analysis. Moreover, no SEM statistical program can conduct MMM and take into account of the cross-classified data structure simultaneously. Hence, a researcher should acknowledge this limitation and be cautioned when conventional MMM is utilized with cross-classified multilevel data given the inflated variance component associated with the remaining crossed factor. Implications of the findings and limitations for each study are discussed.