An examination of comprehensibility in a high stakes oral proficiency assessment for prospective international teaching assistants

This study investigated the construct of comprehensible English in the context of oral proficiency assessment for international teaching assistants. I carried out a three-part mixed method design to explore instructor rater judgments, results of a speech analysis, and how specific speech variables might have influenced judgments on the assessment criteria. Each step focused on a failed/passed assessment comparison made possible through archived data from which 10 individuals initially failed the oral proficiency test but within the same year retook the task and received a passing score. Part A evaluated the perspective of the instructor raters through the rating scale judgments provided on the assessment evaluation forms. In the second part of the study, I coded and scored grammatical, temporal, and phonological variables that occurred on two-minute excerpts of a field-specific summary task from the set of 10 failed and then subsequently passed assessments performed by the same individuals. I inspected the speech analysis results to evaluate differences in the values of specific speech variables on the set of failed performances in comparison to the set of passed performances. In Part C, I conducted 10 case studies to compare each individual's rating scale judgments and rater comments on grammar, fluency, and pronunciation from their failed and their passed assessment with the results from the speech analysis of grammatical, temporal, and phonological variables. The case study approach facilitated a broad inspection of the interrelation among the rater perspectives on the assessment criteria and the speech analysis results. The study findings showed evidence of an interrelation between temporal and phonological variables on rater judgments of comprehensibility, as well as the role of pronunciation as a criterion for oral proficiency assessments. I concluded with implications for future research on the interrelation among speech variables that influence listener perceptions of comprehensibility and the use of pronunciation as a speaking assessment criterion.