Mathematical literacy assessment design : a dimensionality analysis of Programme for International Student Assessment (PISA) mathematics framework

The National Research Council (NRC) outlines an assessment design framework in Knowing What Students Know. This framework proposes the integration of three components in assessment design that can be represented by a triangle, with each corner representing: cognition, or model of student learning in the domain; observation, or evidence of competencies; and interpretation, or making sense of this evidence. This triangle representation signifies the idea of a need for interconnectedness, consistency, and integrated development of the three elements, as opposed to having them as isolated from each other. Based on the recommendations for research outlined in the NRC's assessment report, this dissertation aims to conduct a dimensionality analysis of Programme for International Student Assessment (PISA) mathematics items. PISA assesses 15-year olds' skills and competencies in reading, math, and science literacy, implementing an assessment every three years since 2000. PISA's mathematics assessment framework, as proposed by the Organisation for Economic Co-operation and Development (OECD), has a multidimensional structure: content, processes, and context, each having three to four sub-dimensions. The goal of this dissertation is to show how and to what extent this complex multidimensional nature of assessment framework is reflected on the actual tests by investigating the dimensional structure of the PISA 2003, 2006, and 2009 mathematics items through the student responses from all participating OECD countries, and analyzing the correspondence between the mathematics framework and the actual items change over time through these three implementation cycles. Focusing on the cognition and interpretation components of the assessment triangle and the relationship between the two, the results provide evidence addressing construct validity of PISA mathematics assessment. Confirmatory factor analysis (CFA) and structural equation modeling (SEM) were used for a dimensionality analysis of the PISA mathematics items in three different cycles: 2003, 2006, and 2009. Seven CFA models including a unidimensional model, three correlated factor (1-level) models, and three higher order factor (2-level) models were applied to the PISA mathematics items for each cycle. Although the results did not contradict the multidimensionality, stronger evidence was found to support the unidimensionality of the PISA mathematics items. The findings also showed that the dimensional structure of the PISA mathematics items were very stable across different cycles.