Browsing by Subject "construct validity"
Now showing 1 - 2 of 2
Results Per Page
Sort Options
Item Evidence of Construct-Related Validity for Assessment Centers: More Pieces of the Inferential Pie(2010-07-14) Archuleta, KathrynMuch research has been conducted on the topic of the construct-related validity of assessment centers, however a definitive conclusion has yet to be drawn. The central question of this debate is whether assessment centers are measuring the dimensions they are designed to measure. The present study attempted to provide more evidence toward the improvement of construct-related validity. The first hypothesis involved determining whether opportunity to observe and opportunity to behave influenced discriminant and convergent validity. The second hypothesis addressed the debate over evaluation method and examined which method, within-exercise or within-dimension, yielded more favorable internal construct-related validity evidence. The third hypothesis explored the call for exercise scoring in assessment centers and compared the criterion-related validity of exercise versus dimension scores within the same assessment center. Finally, the fourth objective looked at the relationship of the stability of the dimensions with internal construct-related validity, specifically convergent validity evidence. A developmental assessment center used in two applied settings supplied the data. Two administrations of the assessment center were conducted for low to mid-level managers in a state agency (N = 31). Five administrations were conducted in a professional graduate school of public administration that prepares students for leadership and managerial positions in government and public service (N = 108). The seven administrations yielded a total sample size of 139 participants. Analysis of multi-trait-multi-method (MTMM) matrices revealed that, as hypothesized, a lack of opportunity to behave within exercises, operationalized using behavior counts, yielded poor discriminant validity. Assessor ratings of opportunity to observe and behave did not produce hypothesized results. Consistent with the second hypothesis, secondary assessors, who represented the within-dimension evaluation method, provided ratings that demonstrated better construct-related validity evidence than the ratings provided by primary assessors, who represented the within-exercise method. Correlation and regression analyses of the dimension/performance relationships and the exercise/performance relationships revealed neither dimensions nor exercises to be the better predictor of supervisor ratings of performance. Using MTMM, partial support was found for the fourth objective: those dimensions that were more stable across exercises yielded better convergent validity evidence versus those dimensions that were more situationally specific. However the differences were not statistically significant or large. Overall results of this study suggest that there are some areas of design and implementation that can affect the construct-related validity of assessment centers, and researchers should continue to search for ways to improve assessment center construct-related validity, but should also look for ways other than MTMM to assess validity.Item The Substantive Validity of Work Performance Measures: Implications for Relationships Among Work Behavior Dimensions and Construct-Related Validity(2012-10-19) Carpenter, NichellePerformance measurement and criterion theory are critical topics in the fields of I/O psychology, yet scholars continue to note several issues with the criterion, including empirically redundant behaviors, construct and measure proliferation, and definitions that conflict. These interconnected problems hinder the advancement of criterion measurement and theory. The goal of this study was to empirically examine the issues of theory/construct clarity and measurement as they exist regarding work performance behaviors. This study's first objective was to clarify definitions of core performance behaviors, particularly to resolve issues of construct proliferation and conceptual conflict. Universal definitions of four core criterion constructs (i.e., task performance, citizenship performance, counterproductive work behavior, and withdrawal) were developed that integrated existing definitions of similar behaviors. Each definition reflects a parsimonious conceptualization of existing performance behaviors, which serves to clarify existing, and at times divergent, criterion conceptualizations. Importantly, these integrated definitions represent commonly-held definitions of the constructs and replace the largely discrepant accumulation of definitions. The second objective was to determine whether existing items assumed to measure the four core work performance behaviors were judged by raters to represent their respective constructs. The results showed that of the 851 items examined, over half were judged to not represent their respective constructs which, importantly, replicated previous research. Additionally, the results highlight items that match their respective construct definition and contain minimal overlap with non-posited constructs. Finally, the third objective was to determine the implications of using the problematic items for both the empirical relationships among work performance behaviors and evidence of construct-related validity. The results provided preliminary evidence that while nomological networks are minimally affected, relationships among some work performance dimensions are significantly affected when problematic items are removed from measures of performance constructs. This dissertation demonstrated the need for more attention to the construct labels placed on the behaviors described in work performance items, as there are potentially adverse consequences for theory and measurement. Ultimately, the results of this study showed that work performance behaviors/items have often been assigned incorrect construct labels which, subsequently, may cast considerable doubt on the theoretical and empirical understanding of the criterion domain.