Exploring the complexity of second language writers' strategy use and performance on an integrated writing test through structural equation modeling and qualitative approaches

Integrated writing tasks that combine reading, listening, and writing have become increasingly popular in assessing academic writing. These tasks are seen to offer more authenticity, improve fairness, and provide positive washback effects of the test on learning and teaching of English around the globe. However, the integrated nature of these tasks can pose some issues, such as construct-related validity and verbatim source use. Given that the inferences made from test scores depend upon the construct of the measure, it is important to have a working knowledge of how strategies are used on integrated writing tests as part of the process of construct validation. This study investigates the relationship between second language writers’ strategy use and performance on an integrated reading-listening-writing test using structural equation modeling and qualitative approaches. Data were collected from 161 non-native English-speaking students. The students first took an integrated reading-listening-writing test and followed by a strategy inventory on how they thought while completing the test. Twenty students, ten in the high-performance group and ten in the low-performance group, participated in a retrospective interview. Exploratory factor analysis (EFA) was used to identify the clusters of items based on three hypothetical factors: Rhetorical, Self-Regulatory, and Test-Wiseness Strategy Use. Confirmatory factor analysis (CFA) was then utilized to test the hypothetical relations between observed and latent variables. Subsequently, structural equation modeling (SEM) was used to model the relationship between students’ self-reported strategy use and their test performance. The data collected from retrospective interviews, an open-ended questionnaire, and planning sheets were analyzed to triangulate quantitative results and provide supplementary information in interpreting the quantitative data. The study illuminates the nature of integrated writing strategy use, the nature of integrated writing performance, and the relationship between strategy use and performance on an integrated reading-listening-writing test. The results of the study have implications for second language writing assessment and instruction as well as theory in second language academic writing.