Strategies for Research Methods Tutoring using

Latent Semantic Analysis

David Allbritton Peter Wiemer-Hastings

Elizabeth Arnott Jesse Efron

DePaul University

http://reed.cs.depaul.edu/peterwh/RMT

Talk presented at the annual meeting of the Society for Computers in Psychology, November 2003, Vancouver BC, Canada.

AbstractThe concepts and skills taught in psychology research methods courses are fundamental to our field, yet are often difficult for students to master. RMT (Research Methods Tutor, a descendent of Autotutor) is an intelligent tutoring system that will supplement classroom instruction by engaging students in conversation about course topics, using LSA (latent semantic analysis) to understand student responses. Pedagogical strategies implemented in RMT are described, along with plans for evaluation and deployment.

The Need for Tutoring in Psychology Research Methods

• The skills taught in research methods are fundamental to psychological science

• Students have difficulty mastering these fundamental concepts in the classroom

• Tutoring can help

Advantages of Expert Human Tutoring

• Flexibility

• Effectiveness – 2 Sigma (Bloom, 1984) – the “Holy Grail” of ITS Research

Advantages of an ITS (Intelligent Tutoring System)

• Practicality

• 24/7 Availability

• Lowering Social Barriers

• Low Cost (after development, that is)

Theoretical Issues in ITS Architecture Selection

• Representation of Domain Knowledge– Deep or Shallow?

• Student Model– Deep or Shallow?

• Interface– Constrained (Menus) or Natural Language?

Theoretical Issues in Choice ofPedagogical Style

• Didactic vs. Socratic

• Conceptual vs. Problem-Oriented

• Conversational vs. Pedagogical

• Collaborative vs. Competitive

• Authentic vs. Artificial Task Setting

Effects Sought on the User

• Comprehension (Clarity)

• Motivation

• Appropriately Challenging (ZPD)

Our Approach with RMT

• Architecture

• Pedagogy

• Implementation

• Evaluation

• Deployment

RMT: Architecture

• LSA for semantic representation – Word meanings are vectors in a semantic space

– Sentences are “bags of words”

– Cosine between vectors defines similarity of meaning

• Dialog Transition Network to guide dialog moves– Example of DTN

• Shallow knowledge of domain• Shallow student model

RMT: Pedagogy

• Conversational• Authentic Task

– “You have just been hired as a research assistant. Your first task is to design a study..”

• Three Levels of Questions / Problems– Conceptual– Analytic– Synthetic

Conceptual Question

• Why are studies conducted outside of the lab more likely to have problems with internal validity?

• Model Answer:– Because the experimenter has more control

over conditions in the lab. – Unanticipated confounds are more likely in a

field setting.

Analytic Question

• Now I’ll give you a description of an experiment, and ask you to evaluate the validity of the experiment. (..description of experiment…)

• The concept of internal validity in research means that changes in the dependent variable are a direct result of changes in the independent variable, and not of some other variable. What are some ways that other things besides the lighting may have influenced the test scores?

Analytic Question (Cont.)

• Model Answer– Students took the exam at different times of the

day.– One group is older than the other. – The two grades have different teachers.

Synthetic Question

• Now, I am going to describe an experiment, and you are going to help me set it up.

• An experimental psychologist believes that plant owners who interact with their plants will be happier than plant owners who do not. She identifies individuals who have never owned a plant and tests their happiness level using the Happiness Inventory. She then divides the non-plant owners into two groups: treatment and control. Describe how she should assign the participants to the two groups, what she should do to each group, and what each group should be told to do.

Synthetic Question (Cont.)

• Model Answer– Participants should be randomly assigned to two

groups. – The treatment group should receive a plant.– The treatment group should be instructed to interact

with the plant. – Control group should receive a plant but not be

instructed to interact with it.– Both groups should be asked to return after a period of

time to be re-tested on the Happiness Inventory.

RMT: Implementation

• Interface– Tutor is animated avatar– Text-only version for comparison– Student types responses in a text box

• Curriculum Script

Curriculum Script

• Topics– Validity, reliability, ethics, types of research,

experimental design

• XML schema defines structure of script– Questions– Model Answers– Prompts, hints, and splices– Dependencies

RMT: Evaluation

• Usability and Pilot Testing

• Experimental Evaluation of Tutor Features– RMT with avatar vs. text-only interface– RMT with authentic task vs. traditional tutoring – RMT vs. web-based tutorials

Pilot Testing: Agents

• One of our avatars’ voice needs work:– “The animation was fine, but the woman’s

voice was annoying.” (“Miyako” avatar)

• The “Merlin” avatar was well-liked:– “I liked his facial expressions.” – “He was funny looking. Relieved stress in a

way.” – “The animation was cool. The speech was

monotonous.”

Pilot Testing: Tutor

• “It was easy because he asked a question and in case you got it wrong, he will help you to get to the right answer.”

• “It didn’t just give you the right answer if you got it wrong.”

• “It was creative and forces you to be clear with your ideas.”

Pilot Testing: Summary (N=26)

• Things they particularly liked:– The feedback

– The hints and prompts

– Multiple chances to get the right answer

– Various ways of asking questions

– Being led to the right answer

– The Merlin animation

• Suggestions for improvement:– Mostly technical problems (esp. speech, agents)

RMT: Deployment

• Usability Testing (now)

• Experimental implementation in selected course sections (soon)

• Available for all research methods courses at DePaul (eventually)

Advantages of an LSA-based System

• Lower development cost than expert system

• “Cheap” way of doing NLP

• Works well for lists, on an all-or-nothing basis

Disadvantages of an LSA-based System

• No domain knowledge – only as good as the curriculum script

• Natural language understanding is only approximate– No syntax: “IV causes DV” = “DV causes IV”

– “Heavy” words can overpower rest of sentence

• Not good at figuring out how much two sets overlap (only good at all-or-nothing matches)

RMT at DePaul: http://reed.cs.depaul.edu/peterwh/RMT