View
57
Download
1
Category
Tags:
Preview:
DESCRIPTION
To Use the Teamwork Test -- Or Not? A Psychometric Evaluation. Janet L. Kottke California State University, San Bernardino Kimberly A. French University of South Florida Rhiannon J. Kirchner California State University, San Bernardino Presented to PTC of Southern California, July 23, 2013. - PowerPoint PPT Presentation
Citation preview
To Use the Teamwork Test -- Or Not? A Psychometric Evaluation
Janet L. KottkeCalifornia State University, San Bernardino
Kimberly A. FrenchUniversity of South Florida
Rhiannon J. KirchnerCalifornia State University, San Bernardino
Presented to PTC of Southern California, July 23, 2013
2
TKSA Development
• Stevens and Campion (1994; 1999) develop the Teamwork KSA Test – 35 items, multiple choice format (4 options,
dichotomous scoring)• Reviewed teamwork literature – Identified 14 KSAs identified with effective
teamwork
Kottke, French, & Kirchner PTC July 23, 2013
3
14 KSAs → 5 facets
• These 14 KSAs represent– Five facets• Conflict resolution• Collaborative problem solving• Communication• Goal setting and performance management• Planning and task coordination
Kottke, French, & Kirchner PTC July 23, 2013
4
5 facets → 2 dimensions
• Two higher-order dimensions: – Interpersonal KSAs • Conflict resolution• Collaborative problem solving• Communication
– Self-management KSAs • Goal setting and performance management• Planning and task coordination
Kottke, French, & KirchnerPTC July 23, 2013
5
Teamwork KSA Structure
Kottke, French, & KirchnerPTC July 23, 2013
O’Neill, Goffin, & Gellatly, 2012, p. 37
6
Knowledge, Skill, and Ability (KSA) Requirements for Teamwork
I. INTERPERSONAL KSAs A. Conflict Resolution KSAs
1. The KSA to recognize and encourage desirable, but discourage undesirable, team conflict. 2. The KSA to recognize the type and source of conflict confronting the team and to implement an
appropriate conflict resolution strategy. 3. The KSA to employ an integrative (win-win) negotiation strategy rather than the traditional
distributive (win-lose) strategy. B. Collaborative Problem Solving KSAs
4. The KSA to identify situations requiring participative group problem solving and to utilize the proper degree and type of participation.
5. The KSA to recognize the obstacles to collaborative group problem solving and implement appropriate corrective actions.
C. Communication KSAs 6. The KSA to understand communication networks, and to utilize decentralized networks to
enhance communication where possible 7. The KSA to communicate openly and supportively, that is, to send messages which are: (1)
behavior- or event-oriented; (2) congruent; (3) validating; (4) conjunctive; and (5) owned. 8. The KSA to listen nonevaluatively and to appropriately use active listening techniques. 9. The KSA to maximize consonance between nonverbal and verbal messages, and to recognize
and interpret the nonverbal messages of others. 10. The KSA to engage in ritual greetings and small talk, and a recognition of their importance.
II. SELF-MANAGEMENT KSAs D. Goal Setting and Performance Management KSAs
11. The KSA to help establish specific, challenging, and accepted team goals. 12. The KSA to monitor, evaluate, and provide feedback on both overall team performance and
individual team member performance. E. Planning and Task Coordination KSAs
13. The KSA to coordinate and synchronize activities, information, and task interdependencies between team members.
14. The KSA to help establish task and role expectations of individual team members, and to ensure proper balancing of workload in the team.
(Adapted from Steven & Campion, 1994, Table One, p. 505)
7
Sample Question
• When you set work goals for yourself or your work team, what are the best goals to set?
A. Set goals to "do your best." B. Set general and broad goals. C. Set specific and detailed goals. D. Set easy and simple goals.
Kottke, French, & KirchnerPTC July 23, 2013
8
Criterion validity
• Several studies have found predictability of supervisor and observer ratings of team effectiveness (rs range .20 to .56)– Ellis, Bell, Ployhart, Hollenbeck, & Ilgen, 2005– Leach, Wall, Rogelberg, & Jackson, 2005– McClough & Rogelberg, 2003– Morgeson, Reider, & Campion, 2005
• Mixed results in predicting team member peer ratings
Kottke, French, & KirchnerPTC July 23, 2013
9
Questions about predictive validity
• Miller, 2001– Issues raised about Stevens & Campion validation studies
• Team performance was predicted less well (r = .44) than task work (r = .56)
• TKSA predicted team performance only slightly better than did an aptitude test– But: TKSA added incremental variance beyond GMA
• Aptitude data suggest individual aptitude, not teamwork aptitude measured
– Miller’s own study finds no significant relationship between team performance and TKSA scores
Kottke, French, & KirchnerPTC July 23, 2013
10
Questions about psychometrics
• O’Neill, Goffin, & Gellatly (2012) examine the TKSA – Comprehensive analysis• Reviewed all known studies using TKSA• Classic test theory analysis• Confirmatory factor analysis• Exploratory factor analysis• Convergent, discriminant validation• Criterion validity analysis
– Disappointing results all aroundKottke, French, & Kirchner
PTC July 23, 2013
11
Our plan
• Replicate O’Neill CTT & CFA analyses with fresh samples
• Data sources– Had used the TKSA Test for a group project in several
classes (sample 1)– Collected new data to address unexpected finding
from sample 1 that aptitude was negatively related to teamwork interest (sample 2)
– Collected data for a thesis project on teamwork and task interdependence (sample 3)
Kottke, French, & Kirchner PTC July 23, 2013
12
The 3 samples
• Students from a mid-size southwestern University. – Sample 1 = 251 college students
• Cohort groups, upper level undergraduate-level psychology course, term-length group project.
– Sample 2 = 279 college students • Recruited from psychology and business courses
– Sample 3 = 404 college students • From upper-level undergraduate courses in both psychology
and business departments, either a long-term (i.e., several weeks) or term-length group project
Kottke, French, & Kirchner PTC July 23, 2013
13
Work experience of samples
• Although students, vast majority were employed – 96% in sample 1– 82.4% in sample 2– 88.4% in sample 3
• Work experience ranged from means of 5.5 to 7.3 years – SDs ranged from 5.6 to 6.3 years
Kottke, French, & KirchnerPTC July 23, 2013
14
Table 1. Demographic Descriptive Statistics
Sample 1 Sample 2 Sample 3Demographic Characteristic M SD M SD M SDAge 25.1 6.6 23.1 7.0 23.9 6.1Work experience (years) 7.3 5.9 5.5 6.3 5.9 5.6Number of term-length group projects - - 1.8 2.0 - -Number of work group projects - - 4.4 11.9 - -
Gender (% Female) 82.1% 74.9% 65.1%Ethnicity African American 9.2% 11.5% 8.2% Asian American 4.4% 4.7 % 4.5% Hispanic 45.4% 47.3% 45.0% White 35.1% 26.9% 29.7% Other 6.0% 8.2% 11.9%Education Freshman 0.4% 16.5% 0.0% Sophomore 2.8% 14.3% 4.7% Junior 43.4% 33.3% 50.2% Senior 51.0% 34.1% 42.6% Graduate 2.0% 0.4% 0.2% Other 0.4% 0.4% 0.5%%Respondents with work experience 96.0% 82.4% 88.4%Respondents with supervisor experience 40.2% 24.7% -
Kottke, French, & KirchnerPTC July 23, 2013
15
Analyses conducted
• Analytic Approach– In each sample we evaluated
• Classical test theory– Item, subscale, and total measure properties,
» Means, standard deviations, and item-total correlations – Subscale and total measure means, standard deviations, and internal
reliability • Structure using categorical confirmatory factor analysis
(Mplus) – One factor model, two factor model, and a five factor model
» For multi-factor models, both a covaried model (all factors were allowed to correlate), as well as a higher-order model (includes higher-order general teamwork KSA factor)
Kottke, French, & KirchnerPTC July 23, 2013
16
Supplemental
• Supplemental analysis– To ensure psychometric issues are not a result of student
samples, ran classical test analyses on the subset of working respondents from each sample.
– Results from working samples were not substantially different than full samples• For example, average absolute difference in factor and total
means was .11 for sample 1 [SD = .07], .21 for sample 2 [SD = .19], and .14 for sample 3 [SD = .21].
• The average difference in inter-item correlations was .02 [SD = .01])
• Thus, only the results from the full samples are presented here.
Kottke, French, & KirchnerPTC July 23, 2013
17
Table 2.Teamwork KSA Test item-level means, standard deviations, and item-total correlations for all samples.
Sample 1 Sample 2 Sample 3 Subscale Item Statistics M(p) SD ri-f ri-h ri-t M(p) SD ri-f ri-h ri-t M(p) SD ri-f ri-h ri-t
Conflict resolution mean 0.69 0.43 0.14 0.22 0.26 0.64 0.45 0.08 0.22 0.17 0.67 0.44 0.05 0.16 0.17Conflict resolution SD 0.16 0.07 0.09 0.14 0.15 0.19 0.06 0.05 0.11 0.14 0.20 0.07 0.05 0.11 0.12
Communication mean 0.54 0.48 0.13 0.17 0.17 0.54 0.48 0.08 0.13 0.15 0.53 0.48 0.11 0.14 0.16Communication SD 0.13 0.02 0.06 0.06 0.07 0.13 0.02 0.09 0.11 0.11 0.15 0.02 0.07 0.08 0.07Collaborative problem solving mean 0.42 0.47 0.13 0.18 0.20 0.42 0.48 0.14 0.16 0.18 0.40 0.47 0.11 0.14 0.16Collaborative problem solving SD 0.14 0.02 0.09 0.16 0.17 0.13 0.02 0.09 0.10 0.10 0.13 0.02 0.09 0.08 0.09Goal setting & performance management mean 0.60 0.47 0.20 0.22 0.23 0.58 0.48 0.07 0.13 0.19 0.59 0.48 0.19 0.25 0.28Goal setting & performance management SD 0.12 0.04 0.07 0.04 0.05 0.12 0.01 0.07 0.07 0.05 0.11 0.01 0.06 0.02 0.02Planning and task coordination mean 0.64 0.47 0.15 0.18 0.24 0.63 0.47 0.13 0.17 0.23 0.64 0.47 0.15 0.22 0.24Planning and task coordination SD 0.10 0.03 0.02 0.02 0.07 0.12 0.03 0.05 0.09 0.09 0.13 0.04 0.03 0.02 0.05Grand mean 0.55 0.47 0.14 0.19 0.21 0.54 0.48 0.10 0.15 0.17 0.54 0.47 0.12 0.17 0.19Grand SD 0.16 0.04 0.08 0.10 0.12 0.15 0.03 0.08 0.10 0.10 0.16 0.03 0.07 0.08 0.08Notes. M(p) = item means, SD = item standard deviations, ri-f = corrected item-factor correlation,, ri-h = corrected item-higher order factor correlation, ri-t = corrected item-total correlation.
RESULTS: Item means, SDs, item total correlations
Kottke, French, & KirchnerPTC July 23, 2013
18
Table 3. Teamwork KSA Test facet and total means, standard deviations, reliability, and item correlation summary for all samples
Sample 1 Sample 2 Sample 3
No. of Items M SD K-R 20
Inter-item correlation
M SD K-R 20
Inter-item correlation
M SD K-R 20
Inter-item correlation
Teamwork KSA Test Score
M Range M Range M RangeInterpersonal KSAs 25 13.05 3.55 .58 .06 -.12, .32 12.84 3.36 .52 .05 -.17, .23 12.72 3.29 .51 .04 -.15, .19 Conflict Resolution 4 2.75 .97 .25 .09 -.06, .26 2.55 .95 .15 .05 -.01, .14 2.67 .91 .10 .03 -.11, .16 Communication 12 6.49 2.04 .36 .05 -.11, .26 6.53 1.89 .24 .03 -.10, .22 6.41 1.99 .33 .04 -.11, .19 Collaborative Problem Solving
9 3.82 1.71 .34 .05 -.12, .23 3.76 1.72 .35 .06 -.08, .22 3.63 1.64 .28 .04 -.07, .18Self-Management KSAs 10 6.23 1.99 .49 .09 .00, .27 6.06 1.87 .39 .06 -.08, .22 6.11 2.10 .54 .11 -.01, .26 Goal Setting & Performance Management
5 3.01 1.29 .40 .12 .04, .24 2.90 1.16 .17 .03 -.06, .14 2.93 1.29 .38 .11 .03, .23 Planning & Task Coordination
5 3.22 1.20 .30 .08 .00, .18 3.17 1.19 .27 .07 -.05, .22 3.19 1.21 .32 .09 -.01, .26Teamwork KSA Overall 35 19.29 4.84 .69 .06 -.14, .32 18.81 4.69 .65 .05 -.17, .25 18.82 4.70 .67 .05 -.15, .26
More Results: Scale means, SDs, inter item correlation means, KR 20s
Kottke, French, & KirchnerPTC July 23, 2013
19
Confirmatory Factor Analysis Results on following slide
Kottke, French, & KirchnerPTC July 23, 2013
20
Table 4. Summary of confirmatory factor analysis results
Sample Model 2 df CFI RMSEA
Absolute Loading Mean Loading Range
Sample 1 Five factor, covariedb 198.12* 158 .85 .032 .34 -.10 to .83 Revised five factor, covarieda 309.23** 152 .41 .064 .41 -.05 to .92 Five factor, higher order Did not converge Revised five factor, higher ordera 359.39** 154 .24 .072 .44 -.03 to 1.00 Two factor, covaried 202.701* 159 .84 .033 .34 -.51 to .82 Two factor, higher order Did not converge Revised two factor, higher ordera 202.95* 156 .36 .066 .36 -.09 to 1.00 Single factor 202.701* 159 .84 .033 .34 -.10 to .82Sample 2 Five factor, covariedb 211.12 183 .88 .022 .29 -.08 to .58 Revised five factor, covarieda 308.61** 182 .47 .048 .38 -.09 to .71 Five factor, higher order Did not converge Revised five factor, higher ordera 368.40** 182 .22 .058 .38 -.09 to 1.00 Two factor, covaried 212.79 184 .88 .023 .30 -.49 to.62 Two factor, higher order Did not converge Revised two factor, higher ordera 363.73** 182 .24 .057 .32 -.08 to 1.00 Single factor 212.54 184 .88 .022 .30 -.08 to .62Sample 3 Five factor, covariedb 327.42** 213 .78 .036 .33 -.01 to .61 Revised five factor, covarieda 471.13** 211 .51 .055 .40 .01 to .71 Five factor, higher order Did not converge Revised five factor, higher ordera 565.28** 209 .32 .065 .40 .01 to 1.00 Two factor, covaried 345.70** 214 .75 .039 .32 -.55 to .52 Two factor, higher order Did not converge Revised two factor, higher ordera 527.38** 211 .40 .061 .32 .03 to 1.00 Single factor 353.40** 214 .74 .040 .30 -.01 to .52
*p < .01, **p < .001Kottke, French, & Kirchner
PTC July 23, 2013
21
Discussion • To be fair to the test authors
– They recommend the subscales be used for training and developmental purposes only
– In all of their publications, they use only total scores– So, whether the test authors intended a hierarchical structure is not
a certainty• Yet, with their careful development that categorized the 14 KSAs
into specific facets, one would expect meaningful structure• Criterion validity has been found, but issues here as well
– TKSA test correlates highly with general mental ability– Some incremental predictability
Kottke, French, & KirchnerPTC July 23, 2013
22
Conclusion
• Very limited evidence for adequate psychometric properties
• Use with caution– Recognize the test’s limitations
Kottke, French, & KirchnerPTC July 23, 2013
23
Thank you
Kottke, French, & KirchnerPTC July 23, 2013
24
Questions?
Kottke, French, & KirchnerPTC July 23, 2013
25
ReferencesEllis, A. J., Bell, B. S., Ployhart, R. E., Hollenbeck, J. R., & Ilgen, D. R. (2005). An evaluation
of generic teamwork skills training with action teams: Effects on cognitive and skill-based outcomes. Personnel Psychology, 58, 641-672.
Leach, D. J., Wall, T. D., Rogelberg, S. G., & Jackson, P. R. (2005). Team autonomy, performance, and member job strain: Uncovering the teamwork KSA link. Applied Psychology: An International Review, 54, 1-24.
McClough, A. C., & Rogelberg, S. G. (2003). Selection in teams: An exploration of the Teamwork Knowledge, Skills, and Ability test. International Journal of Selection and Assessment, 11, 56-66.
Miller, D. L. (2001). Reexamining teamwork KSAs and team performance. Small Group Research, 32(6), 745.
Morgeson, F. P., Reider, M. H. & Campion, M. A. (2005). Selecting individuals in team settings: The importance of social skills, personality characteristics, and teamwork knowledge. Personnel Psychology, 58, 583-611.
O'Neill, T. A., Goffin, R. D., & Gellatly, I. R. (2012). The knowledge, skill, and ability requirements for teamwork: Revisiting the teamwork‐KSA test's validity. International Journal of Selection And Assessment, 20, 36-52.
Stevens M. J. & Campion M. A. (1999). Staffing work teams: Development and validation of a selection test for teamwork settings. Journal of Management, 25, 207-228.
Stevens, M. J. & Campion, M. A. (1994). The knowledge, skill, and ability requirements for teamwork: Implications for human resource management. Journal of Management, 20, 503-530.
Kottke, French, & KirchnerPTC July 23, 2013
Recommended