A Comparative Evaluation of Three Situational Judgment Test Response Formats: Additional Assessment of Construct-related Validity, Subgroup Differences, Susceptibility to Response Distortion, Test-taker Reactions, and Internal Psychometric Properties

The primary objective of the present study was to investigate the construct-related validity of three situational judgment test (SJT) response formats. The present study addressed a potential common method bias threat arising from a shared-common-response-method effect that serves as a plausible alternative explanation for the posited differences-in-g-loading (i.e., more demanding in terms of information processing) explanation, which suggests that the variance in scores is due to the differences in the cognitive and information-processing demands of each response format. Thus, the present study used a design in which the three integrity-based SJT response formats were crossed with a GMA test and a personality measure using the same response formats, and 492 undergraduate students were randomly assigned to one of nine study conditions associated with the possible combinations of response formats. White?Hispanic and sex-based subgroup differences, susceptibility to response distortion, and test-taker reactions concerning the three response formats were also assessed, along with a comparative assessment of the internal consistency, test-retest, and alternate-form reliabilities of the three response formats. The results of this study generally supported the differences-in-g-loading explanation for the observed effects. In addition, mixed results were obtained for the differences in SJT scores between Whites and Hispanics, although the only significant difference was found for the rate response format, which favored White respondents. Consonant with the construct assessed by the SJT, women outperformed men on all three response formats, particularly the rank-SJT. The results indicated that the relationship between response format and response distortion was strongest for the rate-SJT, followed by the rank- and then the most/least-SJTs. Participants displayed the most favorable reactions to the most/least-SJT. Lastly, the internal consistency and test-retest reliability estimates were highest for the rate-SJT, while the alternate-form reliability estimates were highest for the rank- and most/least-SJTs. In summation, in the context of noncognitive constructs (e.g., integrity), the rate-SJT appears to be the superior, preferred response format, with its main drawback being its susceptibility to response distortion.