1_Pub-Miller-Tucker_IC-Mil_IJIR_201509 (2)

International Journal of Intercultural Relations 48 (2015) 120–136

Contents lists available at ScienceDirect

International Journal of Intercultural Relations

journa l homepage: www.e lsev ier .com/ locate / i j in t re l

Addressing and assessing critical thinking in interculturalcontexts: Investigating the distance learning outcomes ofmilitary leaders

John W. Millera,∗, Jennifer S. Tuckerb

a Air Command & Staff College, Gunnison Court Pike Road, AL 36064, United Statesb US Army Research Institute, United States

a r t i c l e i n f o

Article history:Available online 31 July 2015

Keywords:Critical thinkingIntercultural competenceSelf-RegulationSituational judgment testCultural dimensionsDMISMilitary leaders

a b s t r a c t

Intercultural competence is a goal of many educational and training programs for militaryleaders who, when deployed overseas, are required to think critically and make strategicdecisions in culturally complex environments. Critical thinking skills are not only essentialleadership tools, but are also the keys to development of intercultural competence. The pri-mary objective of this study is to gain a better understanding of the relationship betweencritical thinking and intercultural competence. Another objective is to determine whichintercultural competencies learners find most challenging. Research methodology includeda thorough examination of the curriculum, content, testing data, and end-of-course surveyresults of a non-credit, self-paced, instructorless online course for 2241 mid-career AirForce officers. Assessing critical thinking and intercultural competence development is achallenge in any educational context, but especially in online, self-paced courses withoutthe benefit of teacher or student interaction. Situational judgment tests (SJTs) are a prac-tical technique for assessing progress. Results revealed a significant relationship betweencritical thinking and intercultural competence SJT scores. Item analysis indicated that somescenarios were more challenging than others from an intercultural perspective. Accordingto the findings, certain cultural differences challenge the decision-making ability of militaryofficers in high-stress, high-visibility situations. These include culture-general knowledge(Mindset), empathic communication skills (Skillset), and a curious, open, and nonjudgmen-tal attitude (Heartset). Participants scored significantly higher, however, when analyzingSJTs situated in more mundane intercultural workplace situations. The study’s findings sup-port the efficacy of utilizing SJTs to develop intercultural competence, especially in onlinelearning environments.

Published by Elsevier Ltd.

1. Introduction

Critical thinking – the ability to reflect and withhold judgment before making a decision – is essential for success in highereducation and beyond (Facione, 2010). Intercultural competence is another set of cognitive skills also widely recognized asintegral to academic and professional success. This skillset also promotes effective and appropriate decision-making, butwithin contexts that are culturally complex (Deardorff, 2009, p. 479). Indeed, the U.S. military views both critical thinking

∗ Corresponding author.E-mail addresses: [email protected], [email protected] (J.W. Miller).

http://dx.doi.org/10.1016/j.ijintrel.2015.07.0020147-1767/Published by Elsevier Ltd.

dx.doi.org/10.1016/j.ijintrel.2015.07.002

http://www.sciencedirect.com/science/journal/01471767

http://www.elsevier.com/locate/ijintrel

http://crossmark.crossref.org/dialog/?doi=10.1016/j.ijintrel.2015.07.002&domain=pdf

mailto:[email protected]

mailto:[email protected]

dx.doi.org/10.1016/j.ijintrel.2015.07.002

J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations 48 (2015) 120–136 121

Table 1The OODA loop and Facione’s six skills.

Observe Orient Decide ActOODA loop The leader detects:

- Unfoldingcircumstances

- Outside information

The leader examines:

- Cultural traditions- Genetic heritage- New information- Previous experience- Analyses & synthesis

The leaderhypothesizes the bestcourse of action

The leader:

- Tests &evaluateshypothesis

Six CT skills Interpretation andself-regulation

Analysis andself-regulation

Inference andself-regulation

Explanation andself-regulation

Evaluation andself-regulation

and intercultural competence as essential to effective leadership (Abbe & Halpin, 2009-2010). Because military leaders areroutinely deployed to foreign lands, sometimes at a moment’s notice, the ability to think critically and make effective andappropriate decisions in intercultural environments has attained priority status in education and training programs in theU.S. Air Force (Selmeski, 2009).

There is general acknowledgement that the requisite cognitive skills for critical thinking and intercultural competenceoverlap (Deardorff, 2004, 2006, 2009; J. M. Bennett, 2009; M. J. Bennett, 2013). In fact, M. J. Bennett (2013) has noted thatthe process for developing intercultural competence “parallels the development of critical thinking” (p. 109). Clearly, bothskillsets are related and both are integral to effective communication and leadership. Yet surprisingly little research hasfocused on the relationship between critical thinking and intercultural competence.

The present study seeks to shed light on the linkages between critical thinking and intercultural competence by examiningthe content of a self-paced, online course for mid-career officers in the U.S. Air Force and the testing data of the more than3200 students who completed the course. Another goal of this research is to determine which intercultural competencieslearners find most challenging.

2. Theoretical background

In this section, we review theories related to critical thinking, intercultural competence, online education, constructivistpedagogy, and situational judgment tests.

2.1. Critical thinking models

Dewey (1910) described critical thinking as an active process of purposeful reflection during which judgment is sus-pended. Building on his work, many others have followed in his footsteps (e.g., Ennis, 1979; Glaser, 1941; Paul, 1993),including Facione (1990, 2010) and Boyd (1995).

2.1.1. Facione’s modelFacione (2010) asked an interdisciplinary panel of 46 scholars to conceptualize the cognitive and affective aspects of

critical thinking, Facione’s expert panel reached consensus on six essential cognitive skills: analysis, interpretation, inference,evaluation, explanation, and self-regulation (p. 6). The panel characterized critical thinking as “the process of purposeful, self-regulatory judgment,” a definition that highlights the importance of self-regulation (p. 22). In his explanation of the results,Facione described self-regulation as a “remarkable” skill that embodies metacognition. Self-regulation is metacognitivebecause it “double checks” the thinking process by working in concert with the other five cognitive skills as well as itself(p. 7). It is an inner dialogue where personal opinions, beliefs, and assumptions are questioned and initial judgments arereconsidered.

The study’s panel of experts did not agree solely on the essential components of critical thinking’s cognitive skills. Thepanel also reached consensus on what constitutes a “critical spirit,” a set of “dispositions” or attitudes that provide an indi-vidual with the proper mental platform for thinking critically (p. 8). These attitudes include qualities such as inquisitiveness,open-mindedness, flexibility, and prudence in suspending judgment. However, before enacting these cognitive and atti-tudinal skills, critical thinkers must understand how to apply each skill and why each is important. While Facione (2010)does not directly designate this technical knowledge as a specific set of skills, he implies it by providing his readers with acomprehensive description of the cognitive skills. The three skillsets – cognitive, attitudinal, and knowledge skills – form aholistic suite of abilities that, when consciously applied, will result in competent critical thinking.

2.1.2. Boyd’s OODA loop modelWhile Facione’s (2010) model defines critical thinking, Boyd’s (1995) OODA loop describes a recursive process for applying

cognitive skills when making decisions or solving problems. Known to virtually all uniformed Air Force personnel, OODA is anacronym that stands for observe, orient, decide, and act (Boyd, 1995, slide 4). The loop is a “time competitive decision cycle”originally created to help fighter pilots respond intelligently to tactical situations. Despite the narrowness of its original

122 J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations 48 (2015) 120–136

purpose, OODA can be applied to any situation that requires critical thinking (Osinga, 2007, p. 6). A graphic alignment ofboth models illustrates how Facione’s six skills can be applied in real world situations within the OODA framework (seeTable 1).

2.2. Intercultural competence models

Intercultural competence, like critical thinking, is an active process of reflection during which judgment is suspended.Deardorff (2006), J. M. Bennett (2009), M. J. Bennett (2013), and Ting-Toomey (1999) sought to tease out the requisite skillsand processes necessary for competent intercultural relations. Other scholars have taken an etic approach (e.g., Hall, 1981;Hofstede, 1991; Triandis, 1995), arguing that cultures can be better understood by comparing values and beliefs that emergein divergent styles or behaviors (e.g., high/low context dimensions; individualist/collectivist dimensions).

2.2.1. Deardorff’s study of intercultural competenceUsing research methodology similar to Facione’s (2010), Deardorff (2006) consulted a panel of intercultural scholars to

define intercultural competence and identify those cognitive and attitudinal skills essential to the development of com-petent intercultural communication and decision-making. Deardorff’s panel did not reach consensus on specific cognitiveor attitudinal skills. Nonetheless, 80% of the panel did agree on 22 integral components. Among these were two of the sixcritical thinking skills to emerge from Facione’s research: analysis and interpretation. Other critical thinking skills identi-fied by Deardorff were comparative thinking, cognitive flexibility, and the ability to withhold judgment and relate. Thesethree metacognitive skills require purposeful reflection on one’s own inferences and judgments and are thus examples ofself-regulation in practice.

In addition, the majority of Deardorff’s (2006) experts also agreed on a suite of attitudes required for competent inter-cultural communication: respect, openness, and curiosity (p. 254). Two of these align closely with components of the criticalspirit articulated by Facione’s (2010) panel: open-mindedness and curiosity. Deardorff’s panel also identified a third groupof skills: cultural knowledge. According to Deardorff’s (2006) experts, one must have acquired “[d]eep knowledge andunderstanding of culture (one’s own and others)” to engage in competent intercultural communication (p. 250).

2.2.2. J.M. Bennett’s model of intercultural competenceIn an effort to organize the requisite skills for effective intercultural relations, J. M. Bennett (2009) grouped these com-

petencies into three dimensions that roughly align with the three skill groupings to emerge from both Facione (2010) andDeardorff’s (2006) work: knowledge, skills, and attitude. Bennett calls these three dimensions “mindset,” “skillset,” and“heartset” (p. 97). The mindset is the knowledge dimension and encompasses culture-general models and knowledge aboutcultural awareness along with generalized data about specific cultures. The skillset or behavioral dimension, on the otherhand, puts into practice such critical thinking skills as analysis, interpretation, inference, empathy, active listening, andinformation gathering. Bennett also includes within the skillset the ability to problem solve while building and maintain-ing relationships, tasks that require the engagement of all the critical thinking skills. The third dimension, the heartsetgrouping, aligns with Deardorff’s (2006) attitudinal set and Facione’s critical spirit by espousing qualities that lead to effec-tive self-regulation. Bennett includes in this area interpersonal qualities such as “curiosity, initiative, nonjudgmentalness,open-mindedness, [and] tolerance of ambiguity” (p. 97).

2.2.3. M.J. Bennett’s developmental model for intercultural sensitivityJ. M. Bennett’s (2013) developmental model for intercultural sensitivity (DMIS) provides a framework for describing

how people experience cultural difference. The DMIS describes six types of intercultural sensitivity ranging from “shallowexperience. . .to more complex perception,” making it a useful curriculum development tool for determining the level ofculture-general content a specific group of learners might need (p. 86). The six stages of development are: denial, defense,minimization, acceptance, adaptation, and integration. The first three (denial, defense, and minimization) are classified asethnocentric whereas the second group of three (acceptance, adaptation, and integration) are ethnorelative (pp. 84–99).

A key DMIS stage for educators tasked with developing curriculum for military personnel is minimization. The DMISstages progress from the ethnocentric stages of denial, defense, and minimization to the ethnorelative stages of acceptance,adaptation, and integration. Bennett (2013) characterized minimization as a transitional position between the ethnocentricand ethnorelative stages. While behavior at this stage remains essentially ethnocentric, individuals operating within thisposition are able to recognize differences between their own and other cultures. According to Bennett, they remain proneto judge these differences in behavior or style as a deficiency rather than as indicative of a divergent worldview. Given thetypical kinds of work and living arrangements most uniformed military personnel are assigned to when deployed overseas,the majority of midcareer Air Force officers are likely to fit comfortably within the DMIS’s minimization stage.

2.2.4. Ting-Toomey’s observe, describe, interpret, suspend judgment modelTing-Toomey (1999) singled out analysis as a salient intercultural competency which she operationalized into a four-step

model: observe, describe, interpret, and suspend judgment or “O-D-I-S” (p. 269). In addition to analysis, three other criticalthinking skills are apparent in Ting-Toomey’s model: interpretation, explanation (analogous with “describe”), and self-


Table 2The OODA loop and O-D-I-S.

Observe Orient Decide ActOODA loop The leader detects:

- UnfoldingCircumstances

- Outside Information

The leader examines:

- Cultural Traditions- Genetic Heritage- New Information- Previous Experience- Analyses & Synthesis

The leaderhypothesizes the bestcourse of action

The leader:

- Tests & evaluateshypothesis

ODIS Observe Describe Interpret Suspendjudgment

Decide how toengage(unstated step)

Engageappropriately(unstated step)

regulation (analogous with “suspend judgment”). One interesting aspect of O-D-I-S, is its striking similarity to the observeand orient stages of Boyd’s OODA loop (see Table 2).

2.2.5. Etic models of cultureEtic models of culture are grounded on the assumption that commonalities can be found in the values and behaviors

of divergent cultures. Typically characterized by the use of microcultural dimensions or categories, etic models make itpossible to generalize values related to specific behaviors. Examples of microcultural dimensions linked to social practices in aspecific cultural setting are Hall’s (1981) low and high context communication styles as well as monochronic and polychronictime orientations (Hall, 1976). Another example is Hofstede’s (1991) individualist and collectivist values. Livermore (2010)categories of rational versus mystical views of the world are another.

There are dangers associated with the use of etic models. Unless properly articulated and presented, they can lead tostereotyping and reductionist thinking. Despite these drawbacks, however, they are valuable teaching and assessment toolsbecause the continua representing the contrasting categories of the various models help students to better understand howtheir own cultural values and the behaviors associated with them differ from those of other cultures.

2.3. Online learning pedagogy and assessment

The emergence of the Internet, the World-Wide Web, and mobile communication technologies during the 1980s and90s have transformed higher education. These innovations, wrote Harasim (2012), challenge the educational commu-nity to “rethink and reassess [its] teaching practices and pedagogical approaches” (p. 3). The greatest challenge lies inthe development of relevant self-paced online learning environments and assessment instruments.

2.3.1. Constructivist online instructionA constructivist approach to online learning, instruction, and assessment, according to Harasim (2012), must focus on “the

role of the learner or group of learners” (p. 68). Constructivist pedagogical assumptions are akin to some of the basic principlesunderpinning both critical thinking and intercultural competence; learning within the constructivist paradigm emergesthrough a process of purposeful reflection and dialogue. Indeed, Gold (2001) pointed out another similarity between thesetwo competencies and the constructivist paradigm when he observed that online courses built on constructivist principlestypically assign authentic, real-world tasks that help students “integrate other understandings of multiple perspectivesthrough reflection” (p. 36).

Building on general assumptions of purposeful reflection, multiple perspective taking, and dialogue, Gold (2001) identifiedthree principles common to all constructivist education programs. These principles are: (1) a curriculum that progresses from“the specific to the general, from the concrete to the abstract”; (2) content that incorporates “prior assimilated structures”;and (3) assessment instruments that test “conceptual understanding,” not rote memorization (p. 38). These three principlespresent a unique challenge for curriculum developers tasked with designing self-paced online courses.

2.3.2. From the specific to the general, from the concrete to the abstractThe introduction of real-world scenarios can lead learners to a better understanding of abstract concepts. In an online

instructorless learning environment, an appropriate story or scenario not only draws student interest, but also providespractice in the use of conceptual frameworks to analyze an ambiguous situation. Before course designers can competentlycreate such meaningful content, however, they must first have a clear picture of their students’ educational background,work history, and professional interests.

2.3.3. Use of assimilated models to facilitate learningUsing familiar conceptual models to help students acquire new ones is another way to teach unfamiliar abstract con-

cepts. According to Lantolf and Thorne (2006), this learner-centered technique facilitates learning by moving students fromthe known to the unknown and is best exemplified by the use of “scaffolds” (p. 274). Scaffolds are collaborative learning


Fig. 1. Vygotsky’s zone of proximal development (ZPD) (Miller, 2011).

strategies that activate the knowledge and skills learners have already mastered to perform tasks or grasp concepts thatwould otherwise be beyond their abilities. These peer-to-peer, tutor-to-student, and teacher-to-student strategies take placewithin Vygotsky’s (1978) Zone of Proximal Development (ZPD) (pp. 79–91). The ZPD is a conceptual frontier lying betweenwhat the learner is capable of performing unaided and what he or she cannot do even with assistance (see Fig. 1).

In a self-paced online environment where peers and an instructor are absent, course designers must build scaffoldinginto the content through the use of tables, graphics, and online learning tools (e.g., simulations, videos, online quizzes withfeedback for incorrect answers). As Harasim (2012) noted, these tools cannot, in and of themselves, “engage the learnerin constructivist interactions such as discussion, debates or other knowledge-building interactions” (p. 76). Nonetheless, ifcourse designers and curriculum developers have a clear understanding of the learners’ abilities, it is possible to constructscaffolds that will engage them in an inner dialogue of purposeful reflection. A simple example of scaffolding built intocourse content appears in Tables 1 and 2. In Table 1, Boyd’s (1995) OODA loop is presented in tandem with Facione’s (2010)critical thinking model while Table 2 scaffolds OODA alongside Ting-Toomey’s (1999) O-D-I-S model.

2.3.4. Assessment of progress through situational judgment testingA student’s motivation is inextricably linked to a conscious awareness that he or she is making progress. As mentioned

previously, students in self-paced online learning environments are not able to question instructors or interact with class-mates. The absence of dialogue makes it difficult for learners to check their understanding of new content. One way to addressthis challenge is through situational judgment tests (SJT). SJTs are assessment instruments that provide both a form of dia-logue between the student and the content as well as the “authentic tasks” recommended by Gold (2001, p. 36). Hauenstein,Findlay, and McDonald (2010) describe SJTs as “scenario-based assessments designed to simulate judgment processes in thework context” (p. 263). Known originally as “critical incidents” (Ng & Rayner, 2010), SJTs were first introduced by the AirForce during World War II as a way to sort out top performers for aircrews (p. 90).

In addition to providing practice in real-world problem solving, SJTs also lend face validity to the assessment process.In her description of this concept, Anastasi (1988) asserted that “the test [must] ‘look valid”’ to the students in order for itto have face validity (p. 144). By composing SJTs that require students to consider how new concepts and skills are appliedeither in familiar situations or within the context of scenarios to which they can easily relate, course designers ensure thatstudents will not only be engaged in the process, but also learn from the experience and accept that the assessment is a validone.

2.4. Summary and research questions

The primary objective of this study is to gain an understanding of the relationship between critical thinking and inter-cultural competence. Another objective is to determine which intercultural competencies learners find most challenging.


Research methodology included a thorough examination of the curriculum, content, testing data, and end-of-course surveyresults of a non-credit, self-paced, instructorless online course for mid-career Air Force officers.

Research questions that guided us in this study are: What are the relationships between critical thinking skills anddifferent types of intercultural competencies? What aspects of intercultural competence (mindset, skillset, and heartset)are more challenging for participants when responding to scenario-based SJT items?

3. Method

3.1. Participants

As stated previously, this study drew its data from a self-paced, non-credit orientation course for an Air Force distancelearning program for mid-career officers. The program is comprised of 9 self-paced and 3 instructor-led online coursesdesigned to improve the professional knowledge and leadership ability of Air Force majors and major-selects to preparethem for promotion to Lieutenant Colonel. Known by its acronym “ORIN,” this course has been in operation since September2012. More than 10,400 students have either taken ORIN or are currently enrolled in it. Enrollment for the entire program asof this writing is 8900 students. All students in the program hold at least a bachelor’s degree. Many have earned a master’sdegree. Because the vast majority of students are either Air Force Majors or Major-selects, most have been deployed overseasat least once. All have had to reside in diverse regions within the U.S. Students enrolled during the March 2013–June2013 timeframe completed a voluntary end of course survey (ns = 3205–3336). The data for the analyses investigatingthe relationship between critical thinking and intercultural competence, item difficulty, pattern of responding, and otherpsychometric properties of the Lesson 5 SJTs reflected a smaller sample of these students (n = 2241).

3.2. Procedure—overview of the course

This study examined the curriculum, assessment instruments, and testing data of ORIN, a self-paced, asynchronous,online course for mid-career officers. Students must take the orientation course before enrolling in the program’s credit-bearing courses. ORIN has two goals: (1) to introduce students to the program’s policies and procedures; and (2) to enhancestudent potential for academic success by introducing lessons on critical thinking, intercultural competence, and academicwriting. The course has six lessons. The first introduces the students to the 12-course program and orients them to theBlackboard course management system. The last lesson provides an overview of the requirements for academic writing atthe post-baccalaureate level. The middle four lessons, however, are designed to answer four questions students might askabout critical thinking. These are: Lesson 2: “Why is critical thinking important?”; Lesson 3: “What is critical thinking?”;Lesson 4: “How can critical thinking be applied to academic study?” and Lesson 5: “How can I use my critical thinkingskills to communicate more effectively across cultures?” While the second lesson gives students a broad introduction to theimportance of critical thinking from the perspective of military history and the fourth provides detailed instruction on howto read critically, the third and fifth lessons are the focus of this study. We refer to the third lesson as the critical thinkinglesson and the fifth as the intercultural competence lesson.

The critical thinking lesson provides a comprehensive introduction to this cognitive skillset. Content is built aroundFacione’s (2010) six critical thinking skills. Boyd’s OODA loop is also included in the lesson. The intercultural competencelesson’s content introduces general concepts related to intercultural communication, including Bennett’s (2009) taxonomy ofintercultural competencies, Ting-Toomey’s (1999) ODIS model, and key micro-cultural dimensions. Desired student learningoutcomes for this lesson align with the Air Force’s definition of intercultural competence:

The ability to quickly and accurately comprehend, then appropriately and effectively act in a culturally complex environ-ment to achieve the desired effect—without necessarily having had prior exposure to a particular group, region or language.(Selmeski, 2009, p. 1)

a Students are able to check their progress by taking scenario-based assessments (SJTs) that simulate dialogue and promotereflection while also delivering robust feedback for each multiple choice answer, whether right or wrong.

Lesson 5 begins with an introduction to the basic concepts of intercultural communication followed by content thatillustrates how principles of critical thinking can be applied to the task of making decisions in culturally complex situations.The course’s intercultural competence lesson addresses a 5-year Air Force-wide educational initiative to develop “cross-culturally competent” (sic) officers and enlisted men and women (Selmeski, 2009, p. 1). Known as the Quality EnhancementPlan or QEP, this initiative was required by the Commission on Colleges of the Southern Association of Colleges and Schoolsfor re-affirmation of accreditation. Because of this initiative, uniformed personnel at all levels have been the beneficiaries ofa broad variety of education and training programs focusing on the development of intercultural competence. As a result, theAir Force officers who participated in this research project, have received multiple opportunities to develop their interculturalcompetence to include the following: (1) pre-deployment training consisting of both online instruction with interculturalcompetency-based simulations for Iraq and Afghanistan and face-to-face training with regional subject matter experts; and(2) exposure to intercultural competence curricula during their most recent professional military education as Captainswhether they took it online or in residence.


Table 3Critical thinking SJTs in critical thinking lesson.

Critical thinking SJT summary Critical thinking skills/model

A well-digging project in a developing country Self-regulation:

• Question assumptions

Anti-American sentiment in the Kyrgyz Republic OODA loopSelf-regulation

• Move from Observe to Orient stage• Analyze to determine root causes

An ambiguous situation with a shop clerk in an English-speakingcountry with high context communication style

ObservationSelf-regulation

• Take time to observe• Question assumptions• Ask questions

An ambiguous situation while standing in line in a developing countrywith a high-context communication style

ObservationSelf-regulation

• Take time to observe• Question assumptions• Ask questions

Building relationships with coalition partners and analyzing a problem EthnocentrismSelf-regulation

• Employ purposeful reflection• Take time to observe• Question assumptions

The content in Lessons 2–5 supports learning objectives across the curricula of the program’s 12 courses while provid-ing a review and reinforcement of concepts related to intercultural competencies previously developed in other Air Forceeducation and training programs. Lesson 5 also articulates directly with three other courses in the program: leadershipand command, national security studies, and joint forces planning. Content in these three courses includes case studiesand online simulations where students must negotiate and make operational and strategic-level decisions in interculturalcontexts.

3.3. Measures

3.3.1. Critical thinking SJTsThe assessment for the critical thinking lesson consists of 10 multiple choice items. Five of the 10 questions are SJTs

and contain scenarios that are all situated in developing countries. Each SJT presents students with an ambiguous conflictsituation triggered by differences in cultural values and behaviors and test students’ critical thinking ability to analyze real-world situations and make appropriate decisions (Table 3). Further, the five SJTs require students to apply their knowledgeof critical thinking’s five cognitive skills and the metacognitive skill of self-regulation.

The total critical thinking score was calculated by taking an average of the correct responses made on the 10 items (10points each for each correct response on 5 knowledge-based questions and 5 SJTs; M = .79). Although participants are ableto take as many iterations of the test as needed to reach the passing score of 80%, only their scores from the final attempt onthe critical thinking assessment were used in the subsequent analyses. When conducting the analyses to obtain an internalconsistency reliability estimate, the results produced a negative Cronbach’s Alpha and indicated many negative inter-itemcorrelations. The results of an exploratory factor analyses indicated a six-factor solution with three of the items not loadingon any of these six factors. These results, suggesting a multidimensional scale, taken together with the negative reliabilityestimate (alpha) supports the idea from prior research that the use of internal consistency reliability estimates for SJTs isproblematic. Waugh (2004) explained that:

Even at the item level, situational judgment tests are multidimensional and heterogeneous in nature. That is, a typical itemmeasures more than one construct and the items measure the various constructs to different degrees. Internal consistencyreliability estimates, on the other hand, assume that a single construct or the same set of constructs (to the same degree)underlies the items. Thus, coefficient alpha usually underestimates the reliability of situational judgment tests (p. 6–7).

Therefore, in addition to the correlation between the average critical thinking and intercultural competence scores, weprovide the correlations between the individual critical thinking items and the average intercultural competence score. The


items for both the critical thinking and intercultural competence assessments were automatically randomized such thateach student received one of 10 different random orders of items each time he or she took the assessments.

3.3.2. Intercultural competence SJTsThe intercultural competence assessment consisted of 12 SJTs that were modeled after Cushner and Brislin’s (1996)

cultural assimilator methodology and test the students’ ability to practice mindset, skillset, and heartset concepts and skills(see Appendix A). Three scenarios were adapted from Cushner and Brislin’s work, two were based on research conductedwith Army Soldiers (Rosenthal, Wadsworth, Paulin, Hooper, & Bhawuk, 2009), four were drawn directly from the coursecontent, one was developed by the first author of the present study from his experiences as a Peace Corps Volunteer, andtwo were based on tasks performed by Air Force members in overseas locations as specified in Air Force doctrine. The finalversion of each scenario was reviewed by at least one Air Force officer whose expertise matched the particular tasks beingperformed in the scenario. Some scenarios were further modified in response to feedback from subject matter experts.

Below is an SJT Progress Check from the intercultural competence assessment. To answer correctly, students must employmindset, skillset, and heartset skills and be willing to disobey a direct order from their commander. Making such a decisionwith a minimal amount of information requires self-regulation, patience, respect, interest, openness, the ability to adapt,and a certain degree of courage. An officer will not do so lightly. Willfully disobeying an order from a superior without justcause can be considered dereliction of duty and subject to prosecution under the Uniform code of military justice (Article 92;see Fig. 2).

As participants are able to take as many iterations of the test as needed to reach the passing score of 80%, their scores wereaveraged across all of the individual testing iterations to obtain the average intercultural competence score. We decidedto average the intercultural competence scores across iterations rather than select only the score from the final iterationbecause we were interested in examining the variability in responses across multiple iterations of the assessment. Although12 intercultural competence SJTs were developed for the assessment, the online course randomizes 10 items for eachparticipant and provides 10 points per each correct score. Thus, the total intercultural competence score consists of anaverage of the correct responses on 10 of the 12 items (M = 81%).

To obtain a reliability estimate, all 12 items were coded as 1s (correct) or 0s (incorrect; alpha = .59). Although this reflects alower reliability estimate, it is a typical estimate for SJTs, which tap multiple constructs across scenarios (Weekley, Ployhart,& Holtz, 2006). This estimate also may be lower because, in order to conduct item analyses with these data, participantsmay have received a score of 0 for an SJT they did not receive. To address this issue and to better understand the pattern ofresponding to the intercultural competence SJTs, we provide the correlations between the average critical thinking score andthe individual intercultural competence SJT items in addition to the correlation between the average critical thinking andintercultural competence scores. Finally, to analyze the pattern of responding on the distractor items, incorrect responseswere subsequently recoded as 1s.

3.3.3. Student end-of-course surveyAt the completion of the course, students were asked to respond to an end-of-course survey consisting of 10 items on

a 6-point Likert scale (1 = strongly disagree, 6 = strongly agree). Students rated the course’s ability to teach critical thinkingand intercultural competence skills (see Table 8 for the list of items and results.) The data were collected anonymously.Therefore, it was not possible to investigate relationships between the self-report items and the SJTs.

4. Results

In this study, we sought to uncover the relationship between critical thinking and intercultural competence. We were alsointerested in determining which intercultural competencies students found most challenging to enact as they responded tothe intercultural situations presented in the SJT questions. The following results emerged from this investigation.

4.1. The relationship between critical thinking and intercultural competence

The results demonstrated that critical thinking was significantly related to the overall measure of intercultural com-petence (r = .28, p < .001) and significantly related to many of the individual intercultural competence SJT items (Table 4).Further examination of the pattern of relationships indicated that critical thinking was more strongly related to scenariosreflecting mindset and skillset intercultural competencies. This makes sense since these two sets of competencies encom-pass a working knowledge of culture general concepts such as microcultural dimensions as well as the ability to applythat knowledge to real-world situations. Each of the intercultural competence SJT scenarios either implied or specificallydescribed their cultural contexts using one or more microcultural dimensions. It is not surprising then, that the resultsindicate use of critical thinking skills to a larger degree when the students applied their understanding of high/low context,individualism/collectivism, time orientations, and rational/mystical religious beliefs to make informed decisions.

As the results for the critical thinking measure indicated a multi-dimensional construct, Table 5 provides the correlationsbetween the individual critical thinking items and the overall intercultural competence score. The results demonstrated thatthe strongest, positive relationships were found among items reflecting self-regulation and inference as well as interpretationand observation skills. The results also revealed an unexpected significant negative correlation between one of the critical


Fig. 2. SJT #9 in the intercultural competence lesson ACSC DL 6.0. (2012).


Table 4Nonparametric correlations (Spearman’s Rho) for the critical thinking average percent correct score and the individual intercultural competence SJTs.

SJT domains Critical thinking(n = 1732) Average percent correct

Mindset/Skillset: Cognitive skills; tendencies of collectivist cultures .06*

Mindset/Skillset: Cognitive skills; tendencies of collectivist cultures .16**

Mindset/Skillset: Cognitive skills; tendencies of high-context cultures .09**

Mindset/Skillset: Cognitive skills; Importance of respect .05Heartset: Self-regulation: Show patience, respect, interest, and openness .10**

Heartset: Self-regulation: Suspend judgment; show respect, interest, and openness .13**

Heartset: Self-regulation: Show respect, interest, and openness .04Mindset/Skillset: Cognitive skills; tendencies of high-context cultures .14**

Heartset: Self-regulation: Show patience, respect, interest, and openness .02Mindset/Skillset: Cognitive skills; tendencies of high-context cultures .01Mindset/Skillset: Cognitive skills; time orientations; tendencies of high-context cultures .16**

Mindset/Skillset: Cognitive skills; tendencies of collectivist cultures .20**

*p < .05.**p < .01.

Table 5Nonparametric correlations (Spearman’s Rho) for the intercultural competence average percent correct score and the individual critical thinking items.

Critical thinking domains Intercultural competence(n = 2092) Average percent correct

Comprehension question of interpretation −.02Comprehension question of observation .03Knowledge question of inference .07**

Knowledge question of questioning .04Knowledge question of judgment .02Self-regulation SJT .13**

OODA loop SJT .03Self-regulation, OODA loop, intercultural context SJT −.05*

Self-regulation, OODA loop, intercultural context SJT .06**

OODA loop, ethnocentrism, intercultural context SJT .04

*p < .05.**p < .01.

Table 6SJT item difficulty and distractor response results.

Intercultural competence skills and microcultural dimensions Correct Response Collapsed Distractor 1 Distractor 2 Distractor 3(n = 1919) % Correct (ns)a % Responded (ns)b % Responded (ns)b % Responded (ns)b

Mindset/Skillset: Collectivist culture 59% (1636) 16% (438) 22% (607) 12% (326)Mindset/Skillset: Collectivist culture 73% (2024) 18% (504) 1% (38) 5% (124)Mindset/Skillset: Low/High Context 64% (1782) 38% (1052) 1% (20) 1% (27)Mindset/Skillset: Respect 60% (1656) 4% (123) 4% (100) 8% (220)Heartset: Self-regulation 66% (1822) 1% (23) 4% (105) 1% (17)Heartset: Self-regulation 66% (1821) 2% (41) 2% (56) 1% (38)Heartset: Self-regulation 59% (1644) 1% (33) 8% (234) 6% (177)Mindset/Skillset: Low/High Context 77% (2138) 9% (258) 2% (68) 5% (125)Heartset: Self-regulation 25% (704) 6% (164) 22% (600) 6% (178)Mindset/Skillset: Low/High Context 47% (1316) 3% (86) 1% (22) 1% (34)Mindset/Skillset: Time Orientations; Low/High Context 78% (2155) 4% (102) 5% (150) 3% (95)Mindset/Skillset: Collectivist culture 80% (2213) 2% (51) 1% (28) 3% (91)

a Students are allowed to keep taking the SJTs until they provide the correct response, thus, correct responses were collapsed across iterations such thata student received a score of #6’7# for any correct response across the iterations attempted. The first attempts at the SJT would reflect much lower percentcorrect results.

b Columns may sum to over 100% because the same students are reflected in more than one column depending on their pattern of erroneous responsesacross iterations. These columns reflect the percentage of students who provided incorrect responses prior to possibly providing a correct response. Onthe other hand, columns may not sum to 100% due to missing data (students were only given 10 of the 12 items per attempt).

thinking SJTs and intercultural competence (see item number 8 in Table 5). Some possible explanations for this result areprovided in the Section 5.

4.2. Intercultural competence SJT results

The results of the item difficulty analysis and patterns of responding for each intercultural competence SJT are presentedin this section. Table 6 shows the intercultural competencies and the microcultural dimensions for each SJT as well as the


Table 7Point biserial correlations with the overall intercultural competence SJT score.

Intercultural competence SJTs Point biserial correlations*(n = 2241)

SJT1 .44SJT 2 .24SJT 3 .42SJT 4 .21SJT 5 .21SJT 6 .23SJT 7 .26SJT 8 .18SJT 9 .25SJT 10 .22SJT 11 .15SJT 12 .09

* All correlations were significant at the p < .01 level.

percent correct. For these analyses, as students were allowed to retake the SJTs until they could obtain a satisfactory score,correct responses were coded as 1s and were collapsed across testing iterations. Table 6 also shows students’ pattern ofincorrect responding which illustrates initial interpretations of the intercultural situations.

The range in the percent correct results indicates that some intercultural situations (SJTs) were more challenging forstudents than others (25–80%; Table 6). Even though the students could retake the assessment, these results indicated thatthe situations described in some of the SJTs were particularly challenging for them. Only 25% of the students respondedcorrectly to SJT 9 which taps heartset intercultural competencies while requiring students to avoid a typical low context,individualist response declining the invitation to meet with the host nation Minister, and select a more appropriate highcontext, collectivist response (see Fig. 2 or the Appendix A for a more detailed description of the scenario). Students struggledto espouse the heartset intercultural competencies necessary for effective performance in this intercultural situation such asbeing open-minded and tolerant of an ambiguous situation. Students needed to incorporate these self-regulatory attitudesin their decision making in order to are necessary to avoid selecting an ethnocentric response.

An examination of the pattern of incorrect responding was helpful in determining students’ initial interpretations of theintercultural situations. For example, the results indicated a clear pattern of incorrect responding for SJT 3 (38%, Table 6) inwhich students provided the reason why the local national indicated a much shorter distance than the real distance (seeAppendix A). The results revealed an ethnocentric bias such that the largest percentage of incorrect responses reflected abelief that the local national did not know how to tell time. Further, the results indicated that for this intercultural situation,students did not possess the intercultural mindset and skillset competencies to interpret this situation accurately and selectthe culturally appropriate answer (that the villager had a high context communication style that prevented him from givingthe aid worker bad news). Similarly, the incorrect response selected most often by students in response to SJT 1 (22%)reflects an ethnocentric perspective and a low ability to elicit the necessary mindset and skillset intercultural competenciesto respond correctly. The focus of the scenario is on General Zinni and Klotz’s (2007) experience dealing with a refugee crisisalong the Turkish–Iraqi border in the aftermath of the first Gulf War (see Appendix A). Students who responded incorrectly tothis intercultural situation believed that the most appropriate response would be to enlist the help of Christian missionariesand Peace Corps Volunteers rather than the tribal leaders among the refugees.

On the other hand, the high percentage of correct answers to those situations that described real-world examples of inter-personal communication in intercultural contexts suggests that it was easier for students to tease out pertinent microculturaldimensions and use them to analyze an interpersonal situation and recommend an appropriate response. For example, theresults for SJT 11 suggested that most of the students responded correctly to this intercultural situation on their first attemptbecause of the high percent correct (78%, Table 6) and low occurrences of incorrect responding (see Appendix A). The inter-cultural situation reflected intercultural mindset and skillset competencies and the microcultural dimensions of low andhigh context communication and time orientation to analyze and interpret an intercultural conflict situation (see AppendixA). Although the scenario was complex due to the amount of information included, it also provided clues that the culture’spreferred communication style was high context. Thus, if the students comprehended the differences between low and highcontext, they also could provide the correct response. Similarly, most of the students responded correctly to SJT 12 on theirfirst attempt as witnessed by the high percent correct (80%, Table 6) and low occurrence of incorrect responding. Studentswere likely able to discern the appropriate response by eliminating options that either did not reflect a culturally appropri-ate response or did not seem plausible given the information contained in the scenario (see Appendix A). Another reasonwhy SJTs 11 and 12 received a high number of correct responses might be related to the scenarios themselves, not onlythe intercultural competencies and microcultural dimensions represented therein. Both stories are based on fairly commonintercultural problems in the workplace that students may have experienced or heard about while deployed, making it easierfor them to place themselves in those situations (see Appendix A).

To investigate how each individual SJT related to the overall SJT measure, point biserial correlations were obtainedbetween each SJT and a sum of all of the SJTs (Table 7). Higher point-biserial estimates for particular items suggest thatstudents who respond correctly to these items are also responding correctly to the other items in the assessment. As SJTs 1


Table 8Student feedback: intercultural competence and critical thinking competencies.

This course enhanced my ability to: Strongly agree Agree Slightly agree Slightly disagree Disagree Stronglydisagree

Total agree Totaldisagree

Totalresponses

Understand my own cultural values and personal biasesFrequency 613 1745 665 94 81 28 3023 203 3226Percentage 19% 54% 21% 3% 3% 1% 94% 6%Recognize the cultural information embedded in artifacts, beliefs, values, and assumptions (e.g., physical settings, stories,symbols, rituals, language, and social structure)Frequency 665 1791 617 74 62 27 3073 163 3236Percentage 21% 55% 19% 2% 2% 1% 95% 5%Think of more than one reason for a person’s behaviorFrequency 795 1739 534 77 59 26 3068 162 3230Percentage 25% 54% 17% 2% 2% 1% 95% 5%Understand the points of view of people from different cultural backgroundsFrequency 756 1740 568 76 57 28 3064 161 3225Percentage 23% 54% 18% 2% 2% 1% 95% 5%Systematically identify consistencies, patterns and relationships among pieces of cultural informationFrequency 614 1763 662 86 61 28 3039 175 3214Percentage 19% 55% 21% 3% 2% 1% 95% 5%Incorporate cultural information into my decision-making processesFrequency 727 1744 587 79 56 29 3058 164 3222Percentage 23% 54% 18% 2% 2% 1% 95% 5%Ask questions to elicit in-depth cultural information and fundamental valuesFrequency 659 1715 642 93 65 31 3016 189 3205Percentage 21% 54% 20% 3% 2% 1% 94% 6%Formulate explanations that reflect an understanding of local culturesFrequency 653 1774 604 89 56 29 3031 174 3205Percentage 20% 55% 19% 3% 2% 1% 95% 5%Analyze information and evidence that challenge cultural stereotypesFrequency 680 1775 587 85 61 28 3042 174 3216Percentage 21% 55% 18% 3% 2% 1% 95% 5%This course improved my critical thinking skillsFrequency 657 1514 836 143 109 64 3007 316 3323Percentage 20% 46% 25% 4% 3% 2% 90% 10%

ACSC DL 6.0. (2013).

and 3 related most strongly with the overall score (r = .44, p < .01, r = .42, p < .01, respectively), these items may be particularlygood at evaluating students’ mindset and skillset intercultural competencies and their knowledge of collectivist values andlow and high context communication styles (see Appendix A). That is, the items more reliably discriminate between goodand poor performers on these competencies and dimensions. Interestingly, the two SJTs with the lowest correlations, SJT11 (r = .15, p < .01) and SJT 12 (r = .09, p < .01), had the highest percent correct scores. Although most of the students wereanswering the SJTs correctly on their first attempt, these items may not be optimal indicators of improvement in the students’intercultural competence. This is because it is likely that the students who responded poorly to the other SJTs respondedcorrectly to these items. These SJTs should be further investigated for possible revisions.

Nonparametric correlations (Spearman’s Rho) were obtained for all of the SJT items (the full correlation matrix is notprovided due to space limitations but can be made available by the first author). Although all of the correlations were sig-nificant due to the large sample size of the present study, several relationships are noted because they highlight the need tofurther examine the content of these intercultural situations. Specifically, SJTs 11 and 12 had one of the strongest positivecorrelations (� = .89, p < .01) while SJTs 9 and 10 had the only negative correlations (� = −.33, � < .01). These results could bedue in part to the idea that the intercultural competencies required to successfully perform in both of these interculturalsituations are very different (heartset vs. mindset/skillset). SJT 9 requires students to self-regulate and overcome their eth-nocentric biases within a military context whereas SJT 10 requires the students to elicit the mindset and skillset interculturalcompetencies and determine the most appropriate strategy for working with host country nationals in a complex situation(see Appendix A). Students who possess the skills to perform well in one context may not possess the necessary skills toperform well in the other.

4.3. Student feedback

The results for the end-of-course survey indicated that most students agreed that the course developed interculturalcompetence and critical thinking skills (94–95% agreed across all of the items; Table 8). Further, 90% of the students reportedthat the course improved their critical thinking skills.


5. Discussion

5.1. Findings

The present study sought to uncover the relationship between critical thinking skills and intercultural competencieswhile also seeking to tease out those competencies that were most challenging for learners to enact. Assessment data andstudent feedback metrics from ORIN, a non-credit, self-paced, online course for Air Force Majors were examined to answerthese questions.

Although one might intuitively surmise that a robust relationship exists between critical thinking and interculturalcompetence, little research to date has provided the evidence needed for theory building. The findings of the present researchprovide empirical support for a relationship between critical thinking and intercultural competence and indicate a moderateassociation between these skillsets. The modest strength of the relationship is somewhat surprising given the overlap of themindset/skillset dimensions of intercultural competence with critical thinking skills and indicates that an additional set ofknowledge, skills and attitudes are needed to perform successfully in intercultural contexts. For example, students seemedto have the most difficulty in responding correctly to the intercultural competency items reflecting Bennett’s (2009) heartsetdimension.

The results of the SJTs that described low-stress intercultural conflicts in the workplace suggested that students mighttend to be more adept at making interculturally competent decisions in such real-world situations. However, the findingsalso suggest that there are certain cultural differences that are likely to challenge the decision-making capability of militaryofficers in high-stress situations. These areas include mindset skills related to an understanding of differences in timeperception (polychronic versus monochronic) as well as skillset (relationship building and information gathering), heartsetcompetencies (risk taking and tolerance of ambiguity), and overcoming an ethnocentric perspective (Bennett, 2009). Onesuch scenario (SJT 9) involved a humanitarian operation in a highly collectivist culture with a high-context communicationstyle (Fig. 2 and Appendix A). Students were asked to decide whether or not to report directly to their commander asordered or comply with a high-ranking ministry official’s request for a meeting. In a collectivist culture, such a meetingwould help to forge a relationship that might well bear fruit later on. Most took the safer route and opted to report totheir commander. It seems likely that students perceived the meeting as a risky venture fraught with ambiguity ratherthan an opportunity for relationship building. These results highlight the dilemma military personnel face in such highchallenge situations. As members of an order-driven organization, military officers, when placed in an ambiguous scenariowhere the stakes are high, must make decisions that, for them, may entail ethical considerations that supersede the desireto display cultural sensitivity. Furthermore, as members of a military organization that is quintessentially hierarchical innature, mid-career officers would be extremely cautious in agreeing to commit to a meeting with an official at the levelof Minister without the prior approval of their commander. Given these caveats, such a question would lend itself betterto discussion than a multiple choice SJT with only one correct answer. Yet as an SJT, this scenario can have value whenmultiple iterations are allowed and robust feedback for answers is provided. Students willing to take the time to re-read thescenario and study the answer feedback learn that making such a culturally sensitive choice could well be beneficial to themission.

Another area of interest to emerge from the data was the negative correlation for self-regulation in critical thinkingSJT item number 8 and the overall intercultural competence SJT score. This result was surprising because the scenariotested for the utilization of self-regulation, a critical thinking skill that the students understood and employed success-fully in other SJTs. One possible explanation is that the scenario described a communication problem that the studentsmay have encountered while deployed. In this scenario, a shop clerk in a foreign country where English was the officiallanguage communicated assent to a question by raising her eyebrows and sharply drawing in her breath. This led manystudents to assume that the clerk (a) did not speak English, (b) was communicating surprise, or (c) was displaying anti-American behavior. (The correct answer was to ask her for clarification.) Situations where one has previously performedsuccessfully can sometimes lead to a misplaced assumption of rightness. Indeed, Ambady, LaPlante, and Johnson (2001)cautioned that an individual’s intuition can be very accurate, but only when operating in highly familiar professional orsocial situations. Leaders who rely solely on intuition in intercultural contexts run the risk of misinterpreting – or miss-ing completely – a wealth of socio-cultural information, even when the situation, at least on the surface, seems easilyunderstandable.

Of all the critical thinking skills, self-regulation was found to be the key to responding correctly to the interculturalscenarios. Prior research has described self-regulation as a malleable skill. When developed, it allows individuals to maintaina high level of performance in stressful situations. However, development of this skill takes time. Baumeister (2000) likenedself-regulation to a muscle that becomes depleted over the short term, but strengthens with continuous exercise overtime. Military personnel required to work in divergent cultural contexts that are also dangerous will quickly find their self-regulation skills depleted. Yet those who have been educated and trained to self-regulate should be better able to bounceback, recognize their biases, and control their behavior.

The results of the SJTs and other data were also helpful in ascertaining the competency level of the students. The coursewas specifically designed to present content related to critical thinking and intercultural competence that is appropriatefor students operating at the minimization level of cultural sensitivity on the DMIS scale (Bennett, 2013, 2009). Taking intoconsideration that students had more difficulty answering higher challenge, ambiguous questions, but were able to answer


lower challenge questions with some facility, this was most likely an accurate assumption. However, determining the exactlevel of the DMIS the students had reached by the end of the course was outside the scope of the present study. WhileHammer’s (2012) 50-item questionnaire known as the Intercultural Development Inventory has been found to establish anindividual’s DMIS level, it was not feasible for inclusion in a study involving thousands of military personnel in a government-run educational program.

On the other hand, the results of the end-of-course feedback provide a modicum of insight into the students’ self-awareness with regard to cultural differences as well as their overall assessment of the learning experience. Bennett (2009)designated development of cultural self-awareness as the primary task for students at the minimization stage. According tothe end of course survey results, 94% of the students who took the course agreed that it had enhanced their understanding oftheir own cultural values and personal biases and 95% believed that the course had enhanced their ability to “[u]nderstandthe points of view of people from different cultural backgrounds.” This suggests that for many, some level of cultural self-awareness was achieved.

5.2. Study limitations

The present research has several limitations, due to the procedures involved in conducting both the data collection and thepsychometric analyses. First, as students are able to take the assessments as many times as necessary to reach the 80% markand move forward in the course, the results may have been affected by the responses of some students who tried to achieve80% during their first attempt by guessing. We tried to minimize these potential errors by averaging the responses fromall of the attempts a student made on the Lesson 5 assessments. Averaging across iterations also provided more variabilityin the scores and reduced the inherent restriction of range in the final responses as by this time most have had multipleopportunities to respond correctly.

It is important to note that the nature of the intercultural competence SJTs also pose challenges when analyzing theirpsychometric properties. For example, the SJTs assessed multiple constructs and are likely heterogeneous at the item level(e.g., responses reflect students’ abilities to tap both cognitive skills and personality/motivation characteristics) in answeringthe items. In this case, researchers posit that it is inappropriate to use Cronbach’s Alpha to estimate reliability becauseit underestimates the reliability of the measure (Whetzel & McDaniel, 2009). As test–retest and parallel forms are themost appropriate reliability estimation methods for SJTs (Whetzel & McDaniel, 2009), developers of the course curriculumshould consider creating additional SJT items so that these methods could be employed and accurate reliability estimatesobtained.

5.2.1. Course structureThe most serious limitation to both research and pedagogy, however, is the course’s delivery method. An online, self-

paced course, asynchronous learning environment presents several hurdles for course designers tasked with developingintercultural competence lessons. Heartset learning outcomes such as suspending judgment, ethno-relative perspective-taking, and tolerance for ambiguity are competencies that would be more effectively taught if collaborative learning toolswere compatible with a self-paced online course design. While the SJT instruments examined in the present researchhave some interactive qualities, particularly when their robust feedback is taken into account, they are delivered in thecold medium of the written word. Other educational and training techniques suitable for asynchronous, self-paced onlinelearning environments such as video-based, interactive SJTs and desktop avatar simulations possess game-like quali-ties that enhance motivation (Deci & Ryan, 2002; Malone & Lepper, 1987; Rieber & Noah, 2008; Turner & Paris, 1995).Such instructional tools allow individuals to be more fully immersed in an intercultural context and thus have a betteropportunity to develop and demonstrate the behaviors associated with the heartset dimension. The development cost forthese innovative environments, however, place them outside the reach of many educational programs, as was the casewith ORIN. An added hurdle for distance education programs in military institutions is the stringent cybersecurity meas-ures taken to protect their online networks. These constraints make the incorporation of non-secure interactive websitesuntenable.

One way to ameliorate the limitations of self-paced online courseware is to install a “wiki.” Wikis, according to Mackenzie,Forgarty, and Khachadoorian (2013), are “a collection of loosely structured, collaboratively edited, web-linked content ona particular subject.” Mackenzie and Wallace (2013) proposed that wikis are “conducive to a constructivist perspective”because they present students with an interactive staging area to “apply course concepts to their own experiences,” makingthem ideal for self-paced online courses (p. 247). Installing a wiki would introduce a zone of proximal development allowingstudents to voluntarily describe intercultural experiences where their self-regulation skills (e.g., the ability to take an alter-nate perspective or deal with ambiguity) were tested and then analyze their decision-making based on their understandingof the lesson content (Vygotsky, 1978). The wiki platform also allows for other students taking the course at the same timeto comment and collaborate. The most compelling of the wiki stories would also be available for curriculum developers touse as SJTs in future iterations of the course.


5.3. Future research directions

In looking at the positive results of the end-of-course survey, a fruitful area for research would be to further establish thenomological net of critical thinking and intercultural competence. Research along these lines might entail an investigation ofmotivational constructs such as those proposed by Van Dyne, Ang, and Livermore (2010). In their study of cultural intelligence(CQ), they designated motivation as a separate intercultural competency comprised of three types of motivation: intrinsic,extrinsic, and self-efficacy. Self-efficacy, the belief that “you will be effective in a cross-cultural encounter” (p. 135), may bethe most powerful of the three types of motivation for educators and trainers to consider as they develop curricula and lessonplans. If students and trainees emerge from an education or training experience with the basic skills needed to effectivelyand appropriately plan, communicate, relate, and negotiate in a culturally complex environment, and they are confidentabout implementing those skills, they are more likely to feel ready to navigate and interact within the culture. In turn, themore effective they are in their intercultural experiences, the more they will be able to rely on intuition to make appropriatedecisions. Such confidence, when tempered by the conscious application of self-regulation, also breeds the type of curiositythat Bennett (2009) described, the kind that eventually leads to effective and appropriate decision making through theunconscious use of heartset skills such as patience, respect, interest, openness, multiple perspective-taking, tolerance forambiguity, and the suspension of judgment.

Another area for future research is an investigation of the distal and proximal nature of the intercultural competencies andattributes on intercultural communication behaviors. This approach would provide useful information for both researchersand practitioners. For researchers, the presence of distal predictors often suggests that their effects on performance criteriain intercultural contexts will be indirect and affected by environmental factors and/or other individual characteristics. Forpractitioners, the presence of proximal predictors suggests that these may be more malleable and thus cost effective toinclude in training programs.

A compelling area for study in the near term would be the unique role of self-regulation. Qualitative research could beconducted with students at the same career level in the Air Force, but who are participating in an instructor-led onlinecourse having similar content and objectives. Such a research platform could expand our understanding of the ways thismetacognitive skillset is enacted. (At least one such course is currently offered by the Air Force.) Within the context of theonline course’s discussion board, students would be invited to become participants and receive one or more of this study’smost challenging SJT questions to ponder and discuss. In this approach, however, the scenarios would be rewritten as open-ended questions to better stimulate discussion. By sharing stories in writing on the discussion board and commenting ontheir classmates’ experiences, participants would engage in purposeful reflection and dialogue, essential requirements forlearning from the constructivist paradigm. As a research platform, this project would provide opportunities to ask follow upquestions on the discussion board and elicit more stories of self-regulatory practices.

Qualitative interviews could also be conducted with participants from both instructor-led and self-paced courses later onto tease out more thick descriptions from their experiences. This line of research might well lead to a better understandingof self-regulation. As mentioned previously, another way to generate qualitative data for research within the context of anasynchronous, self-paced course would be to exploit available online technology through the installation of a wiki com-ponent. The literature mentions perspective taking, suspension of judgment and questioning of assumptions as requisiteheartset competencies. However, these are but two manifestations of self-regulation in intercultural contexts. What othersare being successfully employed?

Clearly, more research into self-regulation is needed. Intercultural competence requires an abundance of self-regulatoryinterpersonal skills (e.g., empathy, patience, openness, and tolerance of ambiguity). Even the best critical thinkers in academicor professional contexts may perform at a less than competent level when operating in a culturally complex environment.There is a need for more empirically based research on self-regulation so that education and training programs can maximizethe transfer of its benefits to real-world contexts.

6. Conclusion

The findings of the present research provide empirical evidence that intercultural competence and the ability to thinkcritically are closely related. Our findings also indicate that the metacognitive ability to self-regulate, a “heartset” skillcommon to both critical thinking and intercultural competence, is the most challenging to implement. It is our belief thatinstructional designers for distance education courses can develop online learning with the goal of enhancing interculturalcommunication behaviors if they take a constructivist approach to curriculum design and create a dialogue between thestudent and the instructional content. SJTs provide such a vehicle by giving students a rationale for test answers. This isespecially helpful with incorrect responses. The SJTs perform the duties of an instructor or classmate by providing scaffoldingthat lays the groundwork for greater awareness and understanding to take place.

Despite the many constraints posed by instructor-less online education programs in military institutions, more can bedone to enhance the experience. In the spirit of Vygotsky’s (1978) ZPD, the implementation of wikis that allow for studentcollaboration in a self-paced online environment can motivate students, increase the potential for learning, and enhance theoverall learning experience.


Appendix A.

Intercultural competence lessonSituational judgment tests (lesson 5)A summary1. Knowledge/Skills: Collectivist Culture

Summary: Scenario is based on Gen Zinni’s struggle to work with Kurdish refugees on the Iraq–Turkey border in the aftermath of Desert Storm. Youare asked to decide which group to work with to communicate with the refugees: (a) Tribal leaders, (b) Western-educated professionals, (c)Christian missionaries and Peace Corps Volunteers, or (d) European/Middle Eastern/Indian business entrepreneurs/other expatriates who speak thelocal language

2. Knowledge/Skills: Collectivist CultureSummary: You ask a local mayor to provide workers for a community project, but the workers are inadequate. Does the town tend to have: (a)Collectivist cultural values, (b) Nuclear family system cultural values, (c) Clock time cultural values, or (d) Individualist cultural values?

3. Knowledge/Skills: Low/High ContextSummary: You are a USAID worker whose car breaks down in a rural area of a developing nation. You ask an old man in a nearby village how long itwould take to walk to your destination. He says you can get there in about 4 hours, but in 4 hours you can only get to the next village where youspend the night. The old man: (a) Had a high-context/indirect communication style, (b) Didn’t know how to tell time, or (c) Was anti-American, or(d) Didn’t realize how slowly Americans walk

4. Knowledge/Skills: Importance of RespectSummary: Col Blaine Holt (now Gen Holt) while commander of Manas Transit Center, Kyrgyz Republic, uncovered several reasons for anti-Americansentiment. Which was the most damaging? (a) The U.S. doesn’t respect the Kyrgyz people, (b) Manas’ association with the oppressive government,(c) U.S. money went to Kyrgyz government, not local population, or (d) Little U.S. aid trickled down to the people who needed it

5. 3C Attitude: Self-regulationSummary: You are mentoring the national police force of a developing nation to develop protocols for natural disasters. However, locals don’t seemto be interested in helping out. They tend to believe there’s no point to doing anything since all natural disasters are acts of God, as is survivalafterward. You should: (a) Explain to people they already take steps to combat the elements, (a) Ask national government to help, (c) Rankcommunities by their level of compliance. Those who rank highest get aid first, or (d) Nothing can be done. The project is doomed

6. 3C Attitude: Self-regulationSummary: You are ordered to investigate charges that a U.S. airbase has caused environmental contamination in a developing country. You learnthat it’s not the airbase, but a drought causing the problem. However, local populace is skeptical of Western science. You should: (a) Talk to officialsand their families; listen to their complaints. Resolution will take time. (b) Tell officials their data is inaccurate. (c) Show empirical scientificevidence that proves your case, or (d) Your evidence is irrefutable. No need to meet with lower level officials. Go see the Minister of Health

7. 3C Attitude: Self-regulationSummary: An atypical snowstorm on an island nation is blocking the runway on a U.S. airbase. You have no heavy equipment. Locals have strongcollectivist and spiritual (rather than secular) value system. The snow is a bad omen. They won’t help unless a shaman blesses the airstrip. Youshould: (a) Have a shaman bless the airbase, (b) Go to highest approval authority and request assistance, (c) Assume it’s a face-saving issue becauseequipment is damaged. See if Air Force can repair it. (d) Seek alternatives. Superstition shouldn’t get in the way of the mission.

Knowledge/Skills: Low/High ContextSummary: You are in charge of a construction project in a country with a strong tribal tradition. The land for the project has cultural significance fora nearby village, so they must give you permission. When meeting with the village elders, you notice that some young men are spokespersons. Theelders seem to nod in agreement when you ask for permission, so you assume that all is well. But when the contractor arrives to begin the project,entry is blocked. The best explanation is: (a) The locals have a high-context communication style and were loath to say no directly. (b) Because thelocal culture is hierarchical, you should not have talked with the young spokespersons. (c) People in high-context cultures often change theirminds. (d) The decision was superseded by a powerful elder after the meeting ended.

8. 3C Attitude: Self-regulationSummary: As the newly designated TALCE commander of a humanitarian effort in a developing nation, you have been asked to report to yourcommander upon arrival in-country. However, when you arrive, you learn that the Minister of Health wants to meet with you first. Should you sendword that: (a) You’re honored and at his disposal. (b) Pass on your apologies, but you’ve been ordered to report to your commander. (c) You’rehonored, but you’ve been ordered to report to your commander. Can you meet with him later? (d) You’re happy to meet with him, but it must brief.You’ve been ordered to report to your commander.

Knowledge/Skills: Low/High ContextSummary: You must organize the repair of a U.S. airfield in a developing country and hired a local contractor, but local tribesmen starteddemonstrating—a first for the airbase. The contractor hired workers from a rival tribe. He says the locals have no skilled workers. You meet with atribal elder who talks about tribal culture and the tribe’s loyalty to the airbase, but not about the demonstrations. When asked, he says “Honor islike money.” The elder is: (a) Telling you indirectly that the airbase’s relationship with the tribe is in jeopardy. (b) Telling you indirectly the problemis a minor one since he talked about loyalty and honor. It would be disloyal and dishonorable to break trust. (c) Telling you indirectly there is noproblem since he didn’t respond to your question about the demonstrations. (d) Telling you indirectly that he is willing to accept a bribe.

Knowledge/Skills: Time Orientations; Low/High ContextSummary: You’re an Acquisitions Officer in a developing country who must negotiate a utilities contract within 3 months. A government agencytells you to obtain permits for the work. Your POC assures you it’s an easy process. However, you are continually required to obtain letter afterletter from local agencies. Angry, you do some research and determine the culture tends to be polychronic with a high-context communicationstyle. This helps you understand that: (a) Your POC did not feel a strong urgency to get the permits quickly and also didn’t want to give you badnews directly. (b) In high-context cultures, the more assertive you are, the better. (c) High context cultures are unfazed by an outsider’s anger. (d)Bureaucracies in such cultures tend to add on unnecessary requirements arbitrarily.

Knowledge/Skills: Collectivist cultureSummary: You’re an ACSC student and are friends with an IO in your class who talks about his extended family. You were placed in the same groupto complete a class assignment. You listen to your friend’s briefing, give him some pointers, and ask difficult questions to prepare him. Next day,however, he skipped a planned dinner with you. What’s the problem? (a) His “face” was damaged by your criticism in front of others. (b) He comesfrom an individualistic culture and felt he didn’t get enough time to practice with the group. (c) He’s overly sensitive. It has nothing to do withculture. (d) He comes from a culture that tends to value low-context communication. He wanted more feedback from your group.


References

Abbe, A., & Halpin, S. (2009–2010). The cultural imperative for professional military education and leader development. Parameters, (Winter), 20–31.ACSC DL 6.0., 2012. ACSC DL 6.0 ORIN Lesson 5 progress check: Item 9. [Test item] Maxwell AFB, AL. Air Command and Staff College.ACSC DL 6.0. (March–June 2013). End-of-course survey results: Orientation and introduction course. Maxwell AFB, AL. ESS.Ambady, N., LaPlante, D., & Johnson, E. (2001). Thin slice judgments as a measure of interpersonal sensitivity. In J. A. Hall, & F. J. Bernieri (Eds.),

Interpersonal sensitivity: theory & measurement. Mahwah, NJ: Lawrence Erlbaum Publishers.Anastasi, A. (1988). Psychological testing. New York: Macmillan.Baumeister, R. S. (2000). Self-regulation and depletion of limited resources: does self-control resemble a muscle. Psychological Bulletin, 126(2), 247–259.Bennett, J. M. (2009). Transformative training: designing programs for culture learning. In Michael Moodian (Ed.), Contemporary leadership and

intercultural competence (pp. 95–110). Thousand Oaks, CA: Sage.Bennett, M. J. (2013). Intercultural practices: international/multicultural education. In Milton J. Bennett (Ed.), Basic concepts of intercultural

communication: paradigms, principles, and practices (2nd ed., pp. 105–136). Boston, MA: Intercultural Press.Boyd, J.R., 1995. The essence of winning and losing [PowerPoint slides]. Defense and the National Interest. Retrieved from

<http://dnipogo.org/john-r-boyd/>.Cushner, K., & Brislin, R. (1996). Intercultural interactions: a practical guide (2nd ed.). Thousand Oaks, CA: Sage Publications.Deardorff, D. K. (2004). Internationalization: in search of intercultural competence. International Educator, 13, 13–15.Deardorff, D. K. (2006). Identification and assessment of intercultural competence as a student outcome of internationalization. Journal of Studies in

Intercultural Education, 10, 241–266.Deardorff, D. K. (2009). Implementing intercultural assessment. In D. K. Deardorff (Ed.), Sage handbook of intercultural competence (pp. 477–492).

Thousand Oaks, CA: Sage.Deci, E. L., & Ryan, R. M. (Eds.). (2002). Handbook of self-determination research. In. Rochester NY: University of Rochester Press.Dewey, J. (1910). How we think. Chicago: D. C. Heath and Co.Ennis, R. H. (1979). A conception of rational thinking. In J. R. Coombs (Ed.), Philosophy of education 1979: proceedings of the thirty-fifth annual meeting of the

philosophy of education society (pp. 3–30). Bloomington, IL: Philosophy of Education Society.Facione, P. A. (1990). The Delphi report—critical thinking: a statement of expert consensus for purposes of educational assessment and instruction. Millbrae, CA:

California Academic Press.Facione, P. A. (2010). Critical thinking: what it is and why it counts. Millbrae, CA: Peter A. Facione, Measured Reasons and the California Academic Press.Glaser, E. M. (1941). An experiment in the development of critical thinking. New York: AMS Press.Gold, S. G. (2001). A constructivist approach to online training for online teachers. Journal of Asynchronous Learning Networks, 5(1), 35–57.Hall, E. T. (1976). Beyond culture. New York: Anchor Books.Hall, E. T. (1981). The silent language. New York: Anchor Books, Doubleday.Harasim, L. (2012). Learning theory and online technologies. New York: Routledge.Hammer, M. (2012). The intercultural development inventory: a new frontier in assessment and development of intercultural competence. In M. Vande

Berg, R. M. Paige, & K. H. Lou (Eds.), Student learning abroad (pp. 115–136). Sterling, VA: Stylus Publishing (Chapter 5)Hauenstein, N. M. A., Findlay, R. A., & McDonald, D. P. (2010). Using situational judgment tests to assess training effectiveness: lessons learned evaluating

military equal opportunity advisor trainees. Military Psychology, 22, 262–281.Hofstede, G. (1991). Cultures and Organizations: Software of the Mind. Europe; Maidenhead, Berkshire, UK: McGraw-Hill Book Co.Lantolf, J. P., & Thorne, S. L. (2006). Sociocultural theory and the genesis of second language development. Oxford, UK: Oxford University Press.Livermore, David. (2010). Leading with Cultural Intelligence. New York: AMACOM.Mackenzie, L., Fogarty, P., & Khachadoorian, A., 2013, September 4. A model for online military culture education: Key findings and best practices.

Educause, Retrieved from: <http://www.educause.edu/ero/article/model-online-military-culture-education-key-findings-and-best-practices/>.Mackenzie, L., & Wallace, M. (2013). Cross-cultural communication contributions to professional military education: a distance learning case study. In R.

G. Sands, & A. Greene-Sands (Eds.), Cross-cultural competence for a 21st century military: culture, the flipside of COIN (pp. 239–258). Lanham, MA:Lexington Books.

Malone, T. W., & Lepper, M. R. (1987). Making learning fun: a taxonomy of intrinsic motivations for learning. In R. E. Snow, & M. J. Farr (Eds.), Aptitude,learning, and instruction, III: conative and affective process analysis (pp. 223–253). Hillsdale, NJ: Lawrence Erlbaum Associates.

Miller, J. W. (2011). The face in the machine: the challenge of online teaching and learning. In Proceedings of distance learning instructor colloquium.Maxwell AFB, AL: Air Command & Staff College (June)

Ng, R., & Rayner, S. (2010). Integrating psychometric and cultural theory approaches to formulate alternative measure of risk perception. Innovation: TheEuropean Journal of Social Science Research, 23(2), 85–100.

Osinga, F. P. B. (2007). Science, strategy and war: the strategic theory of John Boyd. New York: Routledge.Paul, R. (1993). Critical thinking: how to prepare students for a rapidly changing world. Berkeley, CA: Foundation for Critical Thinking.Rosenthal, D. B., Wadsworth, L., Paulin, C., Hooper, A. C., & Bhawuk, D. P. S. (2009). Navigating the human terrain: development of cross-cultural perspective

taking skills. Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.Selmeski, B. (2009). Air university quality enhancement plan, 2009–2014: cross-culturally competent airmen. Maxwell AFB, AL: Air Force Cultural & Language

Center.Ting-Toomey, S. (1999). Communication across cultures. New York: Guilford Press.Triandis, H. C. (1995). Individualism and collectivism. Boulder. CO: Westview Press.Turner, J., & Paris, S. G. (1995). How literacy tasks influence children’s motivation for literacy. The Reading Teacher, 48(8), 662–673.Van Dyne, L., Ang, S., & Livermore, D. (2010). Cultural intelligence: a pathway for leading in a rapidly globalizing world. In K. M. Hannum, B. McFeeters, &

L. Booysen (Eds.), Leadership across differences: cases and perspectives. San Francisco, CA: Pfeiffer.Vygotsky, L. (1978). Mind in society: the development of higher psychological processes. In M. Cole, V. John-Steiner, S. Scribner, & E. Souberman (Eds.),.

Cambridge, MA: Harvard University Press.Waugh, G. W. (2004). Situational judgment test. In D. J. Knapp, R. A. McCloy, & T. S. Heffner (Eds.), Validation of measures designed to maximize 21st-century

army NCO performance (ARI technical report 1145) (pp. 6-1–6-25). Arlington, VA: U.S. Army Research Institute for the Behavioral and Social Sciences(DTIC No. ADA 423602).

Weekley, J. A., Ployhart, R. E., & Holtz, B. C. (2006). On the development of situational judgment tests: issues in item development, scaling, and scoring. InJ. A. Weekley, & R. E. Ployhart (Eds.), Situational judgment tests: theory, measurement, and application (pp. 157–181). Mahwah, NJ: Erlbaum.

Whetzel, D. L., & McDaniel, M. A. (2009). Situational judgment tests: an overview of current research. Human Resource Management Review, 19, 188–202.Zinni, T., & Klotz, T. (2007). The battle for peace: a vision of America’s power and purpose. New York: Palgrave Macmillan.

http://dnipogo.org/john-r-boyd

http://www.educause.edu/ero/article/model-online-military-culture-education-key-findings-and-best-practices

Documents

1_Pub-Miller-Tucker_IC-Mil_IJIR_201509 (2)