An automated test assembly for unidimensional IRT tests containing cognitive diagnostic elements

Large-scale assessments are typically administered numerous times per year using parallel test forms. The traditional methods of constructing parallel test forms are based on manually selecting items for given test specifications such as content balancing. This methods are cumbersome, time consuming, and inefficient. To overcome these problems, an automated test assembly has been used successfully in test construction to assemble conventional IRT tests (van der Linden, 1994). However, these conventional large-scale assessments only provide a single summary score that indicates the overall performance level or achievement level of a student in a single learning area. For assessments to be more effective, tests should provide useful diagnostic information in addition to single overall scores. One approach is using a Cognitive Diagnosis modeling. The purpose of this research is to develop an algorithm for generating information-rich tests by combining Cognitive Diagnosis with the traditional IRT approach that not only produce a single score to measure an examinee’s ability level but also provide diagnostic information. This study describes a new method of automated test assembly, which incorporates diagnostic techniques with existing IRT-based testing assembly methods. The purpose of Cognitive Diagnosis modeling is to provide useful information by estimating individual knowledge states by assessing whether an examinee has mastered specific attributes measured by the test (Embretson, 1990; DiBello, Stout, & Rousses, 1995; Tatsuoka, 1995). Attributes are skills or cognitive processes that are required to perform correctly on a particular item. If standardized testing could incorporate assessments of the various attributes constituting the item, then students, parents, and teachers would be able to see where a student stands with respect to mastering the item. Such information could be used to guide the learner toward areas requiring more study. Helping students to identify their intellectual strengths and weaknesses is more informative and instructive than simply giving them a single score that represents their overall ability. By being able to assess where they stand in regard to the attributes that compose an item, students can plan a more effective learning path to be desired proficiency levels. Even though Cognitive Diagnosis has attracted considerable attention from researchers, few studies have described how to assemble a test that conforms to given cognitive criteria. If such a test could be assembled, it would provide more specific identification of the areas where students needs to improve their skills. Also, it would provide diagnostic feedback to teachers, who could then address the specific needs of individual students. In this way, the test becomes an active tool in the educational process rather than just a passive score report. The proposed automated test assembly method and its corresponding computer algorithm will be developed to construct tests automatically from a given item bank while assuring the tests conform to specifications from both conventional IRT scaling and the Cognitive Diagnostic aspects. The method employs the commonly used Zero-One (0/1) Linear Programming Method. This study describes a new method of automated test assembly, which incorporates diagnostic techniques with existing IRT-based testing assembly methods using Maxmin, Minimax, and Maximum Information Methods. A major goal of this research is to identify a set of the most reasonable constraints in Cognitive Diagnosis and to integrate those new constraints into traditional IRT scaling. Most traditional test assembly methods tend to select best test items to form a test under given test specifications, such as content balancing, item difficulties, item formats, reliabilities, test length, and many more (van der Linden, 1998). For this research, a component to deal with Cognitive Diagnosis is added to the current existing automated test assembly method based on IRT. The research described in this dissertation sought to apply and improve available technologies to automate this task and thereby contribute to a new area of educational research. By implementing the Cognitive Diagnostic approach within the traditional standardized test assembly methods, testing specialists will find that using the algorithm introduced in this dissertation might prove useful to test development.