6
265 TEACH (Training To Enable/Achieve Culturally Sensitive Healthcare) Russell Maulitz, M.D., Ph.D.; Thomas Santarelli; Joanne Barnieu; Larry Rosenzweig; AhN Na Vi; Wayne Zachary, Ph.D. CHI Systems, Inc. (rmaulitz, tsantarelli, jbamieu, /rosenzweig, ahnnayi, wzachary) @chisystems.com Bonnie O'Connor, Ph.D. Consultant BOConnor'at"Lifespan.org Abstract. Personnel from diverse ethnic and demographic backgrounds come together in both civilian and military healthcare systems, facing diagnoses that at one level are equalizers: coronary disease is coronary disease, breast cancer is breast cancer. Yet the expression of disease in individuals from different backgrounds, individual patient experience of disease as a particular illness, and interactions between patients and providers occurring in any given disease scenario, all vary enormously depending on the fortuity of the equation of "which patient happens to arrive in whose exam room." Previously, providers' absorption of lessons-learned depended on learning as an apprentice would when exposed over time to multiple populations. As a result, and because providers are often thrown into situations where communications falter through inadequate direct patient experience, diversity in medicine remains a training challenge. The questions then become: Can simulation and virtual training environments (VTEs) be deployed to short-track and standardize this sort of random-walk problem? Can we overcome the unevenness of training caused by some providers obtaining the valuable exposure to diverse populations, whereas others are left to "sink or swim"? This paper summarizes developing a computer-based VTE called TEACH (Training to Enable/Achieve Culturally Sensitive Healthcare). TEACH was developed to enhance healthcare providers' skills in delivering culturally sensitive care to African-American women with breast cancer. With an authoring system under development to ensure extensibility, TEACH allows users to role-play in clinical oncology settings with virtual characters who interact on the basis of different combinations of African American sub-cultural beliefs regarding breast cancer. The paper reports on the roll-out and evaluation of the degree to which these interactions allow providers to acquire, practice, and refine culturally appropriate communication skills and to achieve cultural and individual personalization of healthcare in their clinical practices. 1. INTRODUCTION There is a mismatch between the needed grasp of cultural expectations growing out of highly diverse patient populations in 21 sl century America, and the sensitivity that medical providers currently bring to bear to interact with those expectations. Through no fault of their own, health-providers often emerge from a training system that gives little attention to communication skills in general, not to mention those particular "soft people skills" requisite to effective clinical management of the many diverse (ethnically and gender) subpopulations that make up American society. Virtually every medical specialty has called for specific, measurable improvements in providers' ability to confront diversity, partly to improve the care of specific patients and partly to reduce larger socio-cultural disparities [1]. Thus, a team of physicians, cognitive scientists, and usability engineers for this project approached the problem of cultural diversity training with the notion that there is minimal evidence to prove existing interventions provide lasting change in cultural understanding of patient needs, or that it modifies trainee attitudes or behaviors toward diverse patient- populations. With funding from the National Institutes of Health's (NIH) National Center on Minority Health and Health Disparities (NCMHHD), the project team created and performed preliminary evaluations of an extensible, malleable, case-authorable, tutoring and assessment system entitled TEACH (Training to Enable/Achieve Culturally Sensitive Healthcare), with an initial panel of interactive clinical cases that explores all aspects of the appropriate care of female African-American breast cancer patients (AABCPs). In the following content and in our conclusion, we characterize each of these desirable feature- sets, e.g., extensibility, as they have been engineered into the TEACH system. To satisfy the "all aspects" requirement, we conducted exhaustive research on several spectrums of AABCPs' needs along several https://ntrs.nasa.gov/search.jsp?R=20100012852 2020-06-19T20:57:16+00:00Z

TEACH (Training To Enable/Achieve Culturally Sensitive ...€¦ · 2.1 TEACH FilterConcept Stephen Krashen wrote extensively about his hypotheses on Second Language Acquisition. He

  • Upload
    others

  • View
    5

  • Download
    0

Embed Size (px)

Citation preview

Page 1: TEACH (Training To Enable/Achieve Culturally Sensitive ...€¦ · 2.1 TEACH FilterConcept Stephen Krashen wrote extensively about his hypotheses on Second Language Acquisition. He

265

TEACH (Training To Enable/Achieve Culturally SensitiveHealthcare)

Russell Maulitz, M.D., Ph.D.; Thomas Santarelli; Joanne Barnieu; Larry Rosenzweig; AhN NaVi; Wayne Zachary, Ph.D.

CHI Systems, Inc.(rmaulitz, tsantarelli, jbamieu, /rosenzweig, ahnnayi, wzachary) @chisystems.com

Bonnie O'Connor, Ph.D.Consultant

BOConnor'at"Lifespan.org

Abstract. Personnel from diverse ethnic and demographic backgrounds come together in both civilian andmilitary healthcare systems, facing diagnoses that at one level are equalizers: coronary disease is coronarydisease, breast cancer is breast cancer. Yet the expression of disease in individuals from differentbackgrounds, individual patient experience of disease as a particular illness, and interactions between patientsand providers occurring in any given disease scenario, all vary enormously depending on the fortuity of theequation of "which patient happens to arrive in whose exam room." Previously, providers' absorption oflessons-learned depended on learning as an apprentice would when exposed over time to multiplepopulations. As a result, and because providers are often thrown into situations where communications falterthrough inadequate direct patient experience, diversity in medicine remains a training challenge. Thequestions then become: Can simulation and virtual training environments (VTEs) be deployed to short-trackand standardize this sort of random-walk problem? Can we overcome the unevenness of training caused bysome providers obtaining the valuable exposure to diverse populations, whereas others are left to "sink orswim"? This paper summarizes developing a computer-based VTE called TEACH (Training to Enable/AchieveCulturally Sensitive Healthcare). TEACH was developed to enhance healthcare providers' skills in deliveringculturally sensitive care to African-American women with breast cancer. With an authoring system underdevelopment to ensure extensibility, TEACH allows users to role-play in clinical oncology settings with virtualcharacters who interact on the basis of different combinations of African American sub-cultural beliefsregarding breast cancer. The paper reports on the roll-out and evaluation of the degree to which theseinteractions allow providers to acquire, practice, and refine culturally appropriate communication skills and toachieve cultural and individual personalization of healthcare in their clinical practices.

1. INTRODUCTION

There is a mismatch between the needed graspof cultural expectations growing out of highlydiverse patient populations in 21 sl centuryAmerica, and the sensitivity that medicalproviders currently bring to bear to interact withthose expectations. Through no fault of theirown, health-providers often emerge from atraining system that gives little attention tocommunication skills in general, not to mentionthose particular "soft people skills" requisite toeffective clinical management of the manydiverse (ethnically and gender) subpopulationsthat make up American society.

Virtually every medical specialty has called forspecific, measurable improvements in providers'ability to confront diversity, partly to improve thecare of specific patients and partly to reducelarger socio-cultural disparities [1].

Thus, a team of physicians, cognitive scientists,and usability engineers for this projectapproached the problem of cultural diversity

training with the notion that there is minimalevidence to prove existing interventions providelasting change in cultural understanding ofpatient needs, or that it modifies traineeattitudes or behaviors toward diverse patient­populations. With funding from the NationalInstitutes of Health's (NIH) National Center onMinority Health and Health Disparities(NCMHHD), the project team created andperformed preliminary evaluations of anextensible, malleable, case-authorable, tutoringand assessment system entitled TEACH(Training to Enable/Achieve Culturally SensitiveHealthcare), with an initial panel of interactiveclinical cases that explores all aspects of theappropriate care of female African-Americanbreast cancer patients (AABCPs). In thefollowing content and in our conclusion, wecharacterize each of these desirable feature­sets, e.g., extensibility, as they have beenengineered into the TEACH system.

To satisfy the "all aspects" requirement, weconducted exhaustive research on severalspectrums of AABCPs' needs along several

https://ntrs.nasa.gov/search.jsp?R=20100012852 2020-06-19T20:57:16+00:00Z

Page 2: TEACH (Training To Enable/Achieve Culturally Sensitive ...€¦ · 2.1 TEACH FilterConcept Stephen Krashen wrote extensively about his hypotheses on Second Language Acquisition. He

266

relevant continua, including (a) the clinicalspectrum from screening to advancedchemotherapy and prognosis, (b) thesociological spectrum from lower socioeconomicto higher socioeconomic personae; and, (c) thecultural spectrum from patient expectationsabout bodily appearance to spiritual belief­systems that may impact patient autonomy.Further below, we present several findings thataddress these continua, in terms of systemtechnical features and the implications of suchsystems for their future expansion.

The appropriate target audience for TEACH isanyone on the training continuum, ranging frommedical students to continuing medicaleducation. In our pilot case-panel of thespectrum of AABCPs, however, we haveconcentrated thus far on house staff trainees­that is, interns and residents-with particularemphasis on surgical subspecialties most oftenresponsible for coaching breast cancer patients.Our focus groups have, therefore, includedpredominantly surgeons, but also haverepresentation from internal medicine and socialwork. Future instances of TEACH, as it isextended to other problems in culturalcompetency-training, will broaden this equationconsiderably.

As TEACH has developed, focus groupmembers' views (those of healthcare providersand patients) have unanimously reflected themedical education literature's prevailing view:that there continues to be a mismatch betweenthe gender and-especially-ethnic diversity ofthe caregivers and that of the populations theytreat.

2. TRAINING APPROACH

Training for culturally competent communicationrequires not only knowledge of relevant culturalbeliefs, barriers, and coping strategies, but alsodeveloping communication skills. To identifycommunication training objectives, we examinedand synthesized literature on models ofculturally competent communication andprovider communication training in cancer care.A number of organizations published reportscontaining models or frameworks for trainingcultural communication (e.g., U.S. Departmentof Health and Human Services Office of MinorityHealth and Agency for Healthcare Research andQuality). We reviewed models presented inthese reports and pared those down to includethose likely to support communication withfemale AABCPs. An example of one model isStuart and Leibermann's BATHE model [2].

Furthermore, we identified communication skillsrelevant to physicians who are working withpatients who have chronic and sometimes

terminal conditions [3]. This literature helped usidentify specific communication skills duringpatient consultation that lead to positiveoutcomes. We were then able to derive specificlearning objectives that are pervasive acrossany relevant cultural beliefs, barriers, and copingstrategies. Additionally, we conducted anextensive literature review on stories of variousfemale AABCPs to try to elicit the patient'sviewpoint on dealing specifically with a breastcancer diagnosis. Finally, through relatedresearch, we identified clinical progressionstages associated with a cancer diagnosisduring which the needs of the patient maychange, thus leading to changes in the relevantdialogue and adjustments to requiredcommunication skills.

2.1 TEACH Filter Concept

Stephen Krashen wrote extensively about hishypotheses on Second Language Acquisition.He identified his fifth hypothesis as the "affectivefilter" or a type of blockage. The three variableshe identified were motivation, self-confidence,and anxiety. He stated that people whoseaffective filter was high, meaning, theypossessed a negative attitude toward languagelearning with low levels of motivation and self­confidence and high levels of anxiety, wouldacquire less language and achieve less thanthose with a low affective filter [4]. His theory,although not completely proven, has gonemostly unchallenged.

For the purpose of developing a "patient profile"for the TEACH initiative, a parallel can be madewith Krashen's "affective filter". We will call thisthe "patient-doctor filter effect". A patient, in thiscase a female AABCP, is associated withseveral filters. The higher these filters, the more"blockage" exists. When a blockage of anyproportion exists, it is more difficult for aphysician to achieve a successful interaction.TEACH is aimed at instructing the physicians(not the patients) the communication skills(which are strategic in nature); therefore, it is inrelation to identifying the possible filter types thathave come into play and to demonstratingverbal and non-verbal actions that appropriatelywork within those filters, and, thereby lead to asuccessful interaction.After reading and summarizing several journalarticles on the topic of female AABCPs andcollaborating with Dr. Bonnie O'Connor, SubjectMatter Expert on African American culture andfolklore, seven filters were identified, whichcould come into play singularly or in combinationwith one or more other filters. These filters drivethe patient's expectations of the conversation,

Page 3: TEACH (Training To Enable/Achieve Culturally Sensitive ...€¦ · 2.1 TEACH FilterConcept Stephen Krashen wrote extensively about his hypotheses on Second Language Acquisition. He

267

her behaviors, and her decoding of messagesfrom the physician. These TEACH filters are:

• Healthcare Coverage equates withQuality of Care

• Breast Cancer Equates with a DeathSentence

• Breast Cancer = Treatment CausingHair Loss or Loss of Sexuality

• God Works Through Doctors• Discrimination• Culturally Indispensable Roles as

Caregivers• In tragedy, preferred coping strategy

method is "Positive Reappraisal"(positivism and spiritualism)

2.2 Training Interpersonal Skills

Oncologists are facing difficult conversationsdue to the patient's perspective of: fears aboutphysical illness, psychological affects, death,treatment, friends and family, and finances,social status, and job [3]. Seemingly, when boththe patient's and the doctor's perspectives arecombined, behaviors are complex. In the caseof TEACH, these perspectives are also joinedwith the cultural beliefs or filters that may bepresent.

Discrepancies and conflicts emerge by generalrule across the entire filter concept with thetypical behaviors, such as asking closedquestions to gather information quickly, usingjudgmental responses, reassuring the patientbefore knowing her needs or concerns, forgingahead with the physician's agenda withoutconsideration to the patient's agenda, and notexhibiting effective listening skills.

These above-mentioned behaviors reveal thatthe physician is overlooking who the patient isand where the patient is coming from. Theinterpersonal and communication skills learningobjectives for TEACH are based on the learnerexhibiting the opposing behaviors after thetraining is completed. In other words, thephysician asks open questions in order hear thepatient's concerns.

TEACH scenarios are designed such that thephysician is given several opportunities to reactin different ways to a patient's question orstatement; one of those ways leads to the mostsuccessful interaction, whereas the other wayslead to either continued worry or even anger onthe part of the patient. An example would bethat the patient with the Positive ReappraisalCoping Strategy states, in a calm voice, that shecan handle whatever the physician has told her.The physician has a choice to either commenton her strength or to ask if she reallyunderstands the gravity of her situation. The

former will lead to a successful interaction,whereas the latter may even anger the patient.This example demonstrates how TEACHcombines both the filter concept and theeffective physician communication skills.Additionally, when the filter concept andcommunication skills are joined by the clinicalstages of progression, a comprehensiveinstructional model emerges as seen in thefigure below.

.... '....

l-- 1=;- 1-'-- i/:==":-=;,~"1'::"""C":.... _..

Figure 1: TEACH Instructional Model

Considering that the approach to learningobjective derivation for TEACH was based onthe filter concept and on identified relevantcommunication skills, TEACH lends itself totransfer the communication skills to a differentset of filters. In other words, other minoritypopulations and clinical contexts could also beanalyzed in the same way, through literaturereview and subject matter expertise, in order todevelop a set of filters related to that particularminority group.

3. TEACH VIRTUAL TRAININGENVIRONMENT

TEACH includes a game-engine-based "player"that can execute interactive avatar-basedscenarios for cultural skills training. The TEACHvirtual training environment incorporates alibrary of cognitive-model-controlled Non PlayerCharacters (NPCs) that facilitate the delivery ofcultural-familiarization training. Through the useof a canonical cognitive model of NPC behaviorsusing a cognitive architecture and a genericscripting language, TEACH scenarios can beencoded and mapped to scenario-specific NPCdialog and behaviors, thereby providing a set ofvirtual NPCs with which the trainee can interact.

An important training requirement for TEACHwas the use of avatars that possess adequatelevels of visual and behavioral fidelity. Thisfidelity includes interactive avatars capable of arange of affect and expressivity. We utilized apre-existing cultural training system, termedVECTOR [5], because it provided a great

Page 4: TEACH (Training To Enable/Achieve Culturally Sensitive ...€¦ · 2.1 TEACH FilterConcept Stephen Krashen wrote extensively about his hypotheses on Second Language Acquisition. He

268

degree of flexibility while requiring onlyincremental modifications.

Of critical importance was the issue of avatarveracity for modeling the target population offemale AABCPs. The requirements for TEACHtraining dictated that the scenarios shouldinclude voice-acted speech coupled with avatarscapable of a range of paralinguistic expressivity.Because these features were not required in thecreation of the VECTOR system, this needpresented a technology gap. To address thisgap, we integrated a high-fidelity character­animation and lip-syncing tool, FaceFX [6], inorder to provide highly interactive avatarscapable of conveying subtle, non-verbal cues.Using FaceFX provides a smooth pipeline forprocessing voice-acted .wav files against avatarspeech (i.e., dialogue) and produces characterasset files which are then used to drive high­realistic game avatars.

The TEACH VTE avatars serve as a form of"virtual" standardized patient and provide forsimulated physician-patient interactions withfemale AABCPs. The trainee is expected tomaintain trust with each avatar bycommunicating in ways that show deference forthe patient's cultural norms (i.e., "filters") andcommunication expectations. The simulatedpatient speaks via a voice-actor while thephysician-trainee selects responses from textpresented on the screen. One of the forms ofperformance feedback provided dynamically tothe trainee is a "trust bar" based on traineeresponses (in the top left), and which is anaggregate measure of patient trust. Additionalmeasures of performance are calculated andstored into the TEACH trainee database for off­line use by an instructor or trainingadministrator.

Figure 2: TEACH Patient-Trainee Interaction

3.1 Avatar Requirements

A range of dimensions were identified whenconsidering the range of characteristics whichwould need to be accounted for in the designand development of the AABCP avatars..Ideally, we wanted to be able to vary the avatarsalong this range in order to develop a group ofrepresentative prototypical avatars. Exampledimensions are: visual appearances, behaviorsand mannerisms, language and vernacular, andsocio-demographic information.

3.2 Patient Interviews and AvatarRefinement

On May 6 and May 8, 2009, ten patients, whowere all female AABCPs, participated in 45­minute to one hour interviews during which timethe patients viewed two videos from the TEACHsystem, reviewed slides showing the avatarsand physical setting, and participated in alengthy discussion with the interviewer using aseries of structured questions.

The length of each interview varied slightly assome patients provided more data than others,and some provided more feedback on thephysical avatars. With patients who providedless information than others, the interviewer didnot pressure the patient but rather let theinterview move more quickly. With thosepatients who spoke more freely about personalfeelings, the interviewer allowed the interview tolast longer and extrapolated data for the study.

Overall, the patients had very positive feedbackabout the TEACH avatars in that their answersto the questions seemed to parallel with what wehad been exploring and even, at times, hadstruggled with. With respect to the latter, thepatients were able to shed some light on theseaspects. No patient remarked that TEACH wastotally unbelievable, and all patients remarkedthat the patients in the videos "reminded them ofsomeone", whether this was physical andattitudinal or whether it was an actualconversation that either they have had with aphysician or someone they know has had.Finally, based on responses, it was evident thatseveral of the seven filters, representing variouscultural beliefs, were a match to the beliefs heldby the interviewees.

The categories of feedback from the patientsrelate to themes, such as visualadditions/changes to the avatars, changes inavatar gestures, expressions, or movement, andenvironmenUsetting changes. Suggestedchanges and enhancements to the avatarsincluded adding jewelry, changing bodyfeatures, changing clothes, changing facial

Page 5: TEACH (Training To Enable/Achieve Culturally Sensitive ...€¦ · 2.1 TEACH FilterConcept Stephen Krashen wrote extensively about his hypotheses on Second Language Acquisition. He

269

expressions, and placing the patient in adoctor's office instead of an exam room.

3.3 TEACH Authoring Facility

Despite the successes in applying serious­games to soft-skills training, scenario contentgeneration is often an obstacle in developinggame-based training systems, particularly interms of cost. Furthermore, a common criticismof serious games has been the lack of asystematic approach to linking learningobjectives to scenario content. This is anoteworthy discrepancy as research has beensuggested that if a scenario is linked withtraining objectives, trainees are more likely tolearn the underlying content [7].

To this end, an important design anddevelopment challenge was including anauthoring capability within TEACH. Such afacility provides two advantages: 1) it allows forsystematic and repeatable manipulation ofexisting scenarios in order to supportexperimentation within this virtual trainingenvironment; and, 2) it provides the ability forend-users of the system to add content in a waythat positions scenario creation in the largercontext of training-objective articulation,performance measurement, andfeedback/assessment.Previously, a VECTOR scenario editorcomponent was developed to allow for theefficient creation of new game-based scenariosand to integrate instructional design principlesinto the authoring process to promote moreeffective training scenarios [8]. To address thisconcern, the VECTOR scenario authoring toolswere augmented to support the uniquerequirements of TEACH, such as voice-actingand Face-FX processing.

- -- .- -

Figure 3: TEACH Scenario Editor

To facilitate consistent scenario creation, aprocess-model for scenario authoring isincluded within the TEACH scenario authoringtool. To make scenario authoring more

accessible to a wider audience (i.e., beyond"game" engineers), a cinematic metaphor wasused to create the design of the authoring toolinterface. The use of cinematic metaphors hasbeen successfully used in similar VTEparadigms [9],[10]. Scenario authoring withinTEACH encompasses a number of trainingaspects, including training objectivespecification, scenario information, environmentspecification, plot organization, vignette creation,and scenario generation.

4. CONCLUSIONS

Next steps for the TEACH environment willinclude comparative-efficacy studies of the VEas essentially a "clinical intervention" for thetraining health of providers. The project teamwill deploy varying "doses" of TEACH, alongsidetraditional didactic or "paper" exposition ofcultural-competency norms, to trainees inrandomized controlled fashion. We will alsoseek to define the extent to which any effect, ifobserved, is sustained over time, probably byexposing our sub-populations of trainees, aftertheir differential exposure to traditional orexperimental training-interventions, to the samemetrics at points in time both immediatelyfollowing then at an appropriate point (sixmonths or more) remote from the interventions.Hypotheses formed as a result of the TEACHexperience are two-fold. From a clinical­effectiveness point of view, we hypothesize thatclinician-trainees experiencing TEACH-like VE'swill demonstrate a degree of sustained culturalawareness that is dependent on intensity andrecency of VE exposure. From the standpoint ofthe expansion of audiences for TEACH andsimilar VE's, we hypothesize that the followingfour metrics will predict success:

1. Scalability allows for an applicationcreated for an initial small group trainingframework to be enlarged to a muchgreater 'N' of trainees withoutconcomitant increase in cost.

2. Extensibility provides for addition of newtypes of virtual cases within previously­designed VE's, allowing new anddifferent forms of norms and filters to beimparted and assessed.

3. Evaluability allows for the directapplication of comparative-effectivenessmetrics to a system. Such a featureprevents the simplistic aspect of sometraining programs' "show-it-and-trust-it"approach to any domain knowledge.

4. Authorability gives tools to non-technicaldomain experts, such as clinicians,permitting them to populate caseswithout having to supply code or

Page 6: TEACH (Training To Enable/Achieve Culturally Sensitive ...€¦ · 2.1 TEACH FilterConcept Stephen Krashen wrote extensively about his hypotheses on Second Language Acquisition. He

270

otherwise contend with excessivelytechnically-constrained requirements:

5. ACKNOWLEDGMENTS

The authors wish to express their gratitude tothe National Center on Minority Health andHealth Disparities (NCMHHD) for support undercontract 1R43MS001642-02. We also thankPriscilla Grant and the extramural programsstaff at the NCMHHD, Roy Gay, M.D., for helprecruiting patient focus group evaluations; WallySmith, M.D., for clinical research methodologyand assistance with minority health issues; andAri Brooks M.D., and Andres Castellanos, M.D.,both for assistance with provider focus grouprecruitment and comparative efficacy evaluation;all of the brave women who participated in thecancer-patient interviews; Katonya Mosley forassistance with cultural appropriatenessmeasures and technical voice recording ofcontent; and Jim Stokes and Inez Nelson fortechnical assistance in content management,database design, and authoring systemdevelopment.

6. REFERENCES

1. Betancourt, J.R.; Carrillo, J. E.; & Green, A.R. (1999). "Hypertension in multicultural andminority populations: linking communicationto compliance," Curr Hypertens Rep, 1(6),482-488.

2. Stuart, M.R. & Leibermann, J.R. (1993). Thefifteen minute hour: applied psychotherapyfor the primary care physician. New York:Praeger.

3. Buckman, Robert, M.D. (1992). How toBreak Bad News. Johns Hopkins UniversityPress: Baltimore, MD.

4. Krashen, S. (1981). Second LanguageAcquisition and Second Language Learning.Oxford: Pergamon.

5. Deaton, J.; Barba, C.; Santarelli, T.;Rosenzweig, L.; Souders, V.; McCollum, C.;Seip, J.; Knerr, B. & Singer, M. (2005)."Virtual Environment Cultural Training forOperational Readiness (VECTOR)," TheJournal of Virtual Reality, vol. 8, pp. 156-167.

6. King, S. (2008). "Animating Speech inGames." Lecture Notes in ComputerScience, pp. 234-245, SpringerLink.

7. Belanich, J.; Sibley, D. & Orvis, K. L. (April,2004). "Instructional characteristics andmotivational features of a PC-based game."ARI Research Report 1822. U.S. ArmyResearch Institute for the Behavioral andSocial Sciences: Alexandria, VA.

8. Barba, C.; Santarelli T.; Glenn FA; Bogert,D. & Belanich, J. (2006b). "An Approach toScenario Authoring for Virtual Environment

Training," Proceedings of the 2006Conference on Behavior Representation inModeling and Simulation (BRIMS), May,2006, Baltimore, MD.

9. Seif EI-Nasr, M. (2005). "Desktop 3-DInteractive Drama - Applying DesignPrinciples from the Performance Arts,"Human Computer Interaction International.NV, July 21-27.

10. Cavazza, M. & Charles, F. (2002)."Character-based interactive storytelling,"ACM Joint Conference on AutonomousAgents and Multi-Agent Systems, Intary.