What can CTAT do for you? Overview of the CTAT track Vincent Aleven, Bruce McLaren and the CTAT team 3rd Annual PSLC LearnLab Summer School Pittsburgh,

What can CTAT do for you?Overview of the CTAT track

Vincent Aleven, Bruce McLaren

and the CTAT team

3rd Annual PSLC LearnLab Summer SchoolPittsburgh,

June 18-22, 2007

What can CTAT do for you? - 2 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007

© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007

If you are not in the CTAT track, why would you listen to this talk?• CTAT relevant to most other tracks:

– TuTalk: TuTalk dialogue tutor agent and CTAT intelligent tutor would make an awesome duo!

– In Vivo: could do an in vivo experiment with CTAT-based tutors (happens all the time!)

– Data Shop: many data sets in the Data Shop were generated using CTAT-built tutors

– TagHelper: Hmm…. maybe you are naturally curious about CTAT

• Track hopping is allowed!– E.g., if in the In Vivo track, could attend (some) CTAT

sessions



Overview

• What is “a tutor?”– And would you want to build one?

• Why do you need CTAT to build one?• What more precisely can you do with

CTAT?– Short movie of authoring with CTAT– Examples of projects that have used CTAT– Key choices to make when authoring with

CTAT

• Planned activities in the CTAT track



A simple example: Fraction Addition

• Interact with the fraction addition tutor on the CTAT web site …



Key elements of interactivity in CTAT Tutors1. Problem-solving environment, often with

interactive tools

2. Tutorial guidance, in the form of a. implicit yes-no feedback on correctness of problem

steps b. feedback messages for commonly-occurring errors c. next-step hints (on demand or tutor initiated) ==> can handle multiple student strategies for

solving problems

3. Adaptive problem selection based on student performance solving problems with the tutor

Koedinger, K. R., & Aleven V. (in press). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review.



Kinds of Computer Tutors

Intelligent tutoring systemse.g., Sherlock

Model-tracing tutors

e.g., Andes Cognitive Tutors

e.g., Algebra

Tutoring systems

CAI e.g., Microsoft’s PersonalTutor

Constraint-based tutors

e.g., SQL Tutor

Example-tracing tutors

e.g., Stoichiometry, French Culture Tutor

Can be built with CTAT



Cognitive Tutors: Real-world success of Intelligent Tutoring Systems technology

• Cognitive Tutor Courses– Goal: Improve quantitative literacy & fluency– Computer-based one-to-one tutoring

• Based on computational models of student thinking• Based on the concept of Model Tracing

• Used by many students in many schools– Algebra Cognitive Tutor: In over 2,000 schools in the USA,

300,000 students per year– Geometry Cognitive Tutor: In 350 schools– Most widely used intelligent tutoring system?

• University created a company, Carnegie Learning, to disseminate

Algebra Cognitive Tutor Sample

Use graphs, graphics calculator

Analyze real world problem scenarios

Use table, spreadsheet

Use equations, symbolic calculator

Tutor learns about each student

Tutor follows along, provides context-sensitive Instruction



Replicated Field Studies• Full year classroom experiments• Replicated over 3 years in urban schools• In Pittsburgh

& Milwaukee

• Results:50-100% better on problem solving & representation use.

15-25% better on standardized tests.

Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8.

0

10

20

30

40

50

60

Iowa SAT subset ProblemSolving

Represent-ations

Traditional Algebra Course

Cognitive Tutor Algebra



ACT-R: A Cognitive Theory of Learning and Performance

• Big theory … key tenets:– Learning by doing, not by listening or watching

– Production rules represent performance knowledge:

These units are: Instruction implications:• modular • context specific

isolate skills, concepts, strategiesaddress "when" as well as "how"

Anderson, J.R., & Lebiere, C. (1998). The Atomic Components of Thought. Erlbaum.



• Cognitive Model: A system that can solve problems in the various ways students can

Strategy 1: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + ac = d

Strategy 2: IF the goal is to solve a(bx+c) = d THEN rewrite this as bx + c = d/a

Misconception: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + c = d

Cognitive Tutor Technology:Use ACT-R theory to individualize instruction



3(2x - 5) = 9

6x - 15 = 9 2x - 5 = 3 6x - 5 = 9

Cognitive Tutor Technology:Use ACT-R theory to individualize instruction• Cognitive Model: A system that can solve problems in

the various ways students can

If goal is solve a(bx+c) = dThen rewrite as abx + ac = d

If goal is solve a(bx+c) = dThen rewrite as abx + c = d

If goal is solve a(bx+c) = dThen rewrite as bx+c = d/a

• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction



3(2x - 5) = 9

6x - 15 = 9 2x - 5 = 3 6x - 5 = 9

Cognitive Tutor Technology:Use ACT-R theory to individualize instruction• Cognitive Model: A system that can solve problems in

the various ways students can

If goal is solve a(bx+c) = dThen rewrite as abx + ac = d

If goal is solve a(bx+c) = dThen rewrite as abx + c = d

• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

Hint message: “Distribute a across the parentheses.”

Bug message: “You need tomultiply c by a also.”

• Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing

Known? = 85% chance Known? = 45%



CTAT motivation: Make tutor development easier and faster!• Development costs of instructional technology are

high– E.g., ~300 dev hours per hour of instruction for Computer

Aided Instruction (Murray, 1999)

• Cognitive Tutors: – Large student learning gains as a result of detailed cognitive

modeling– ~200 dev hours per hour of instruction (Koedinger et al,

1997)– Requires PhD level cog scientists and AI programmers

• Solution: Easy to use Cognitive Tutor Authoring Tools (CTAT)Murray, T. (1999). Authoring Intelligent Tutoring Systems: An Analysis of the state of the art. The International Journal of Artificial Intelligence in Education, 10, 98-129.

Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to

school in the big city. The International Journal of Artificial Intelligence in Education, 8, 30-43.



CTAT goal: broaden the group of targeted authors• Instructional technology developers (e.g.,

instructional media dept. at university, or developers of on-line courses)

• Researchers interested in intelligent tutoring systems

• Instructors (e.g., computer-savvy college professors)

• Learning sciences researchers interested in using computer-based tutors in their experiments– Within the PSLC, CTAT-based tutors are often used in in

vivo experiments



How to reduce the authoring cost?• Less programming, more automation

– Drag & drop interface construction– Demonstration-based programming– New: Machine learning and data mining

• Human-Computer Interaction (HCI) methods– User studies, summer schools, informal & formal

comparison studies

• Exploit tools already in use– Component-based architecture with “standard” tool-

tutor protocol– Off-the-shelf tools and languages (e.g., Netbeans,

Eclipse, Flash, Jess)



CTAT supports two types of tutors

• Cognitive Tutors and Example-Tracing Tutors• Behavior is similar (though not identical), both support

the same elements of interactivity:– problem-solving environment with tutorial guidance:– yes/no feedback on steps (can handle multiple student

strategies!)– Specific feedback messages for common errors– Next-step hints

• Technology and development are very different– Example-Tracing Tutors: built through “programming by

demonstration” - authors creates examples of how the students should solve problems

– Cognitive Tutors: built by AI programming (much harder) - author creates rule-based cognitive model



Movie Showing How an Example-Tracing Tutor is built



Authoring with CTAT and Flash

Behavior Recorder

Flash authoringenvironment with

tutor interface shown

Web browser withExample-Tracing tutor

embedded in HTML



Authoring an Example-Tracing TutorStep 1: Create a User Interface

– Create the graphical user interface (GUI) used by the student

Step 2: Demonstrate Behavior– Demonstrate correct, alternative correct, and incorrect solutions

Step 3: Generalize– Specify how demonstrated behavior could vary within given

problem• allowed order of steps• allowed variants for a given step

Step 4: Annotate the Graph – Annotate solutions steps in the resulting “behavior graph” with:

• hint messages, • error messages, • labels for concepts or skills associated with actions

Test and Iterate on Steps 1-4 …

Step 5: Publish Tutor to Web

Behavior Recorder

Data Shop

CTAT’s Modular Architecture

Student Interface(or external problem-solving environment)

Cool Modes

CyclePad

Java Swing

Flash

Learner Management System

LMS

Tutor Shop

Cognitive Model Development Tools

TDK (Lisp-based)

Jess tools (Java-based)

Editor

Eclipse

Tutor Engine

Tertle (Lisp-based) + Model Tracer

Jess (Java-based) + Model Tracer

Example Tracer (Java-based)

Example Tracer (Flash-based)

GUI Builder

IntelliJ

Dreamweaver

Code Warrior

Netbeans

Flash MX 2004



DataShop - Inspection & Evaluation of CTAT Data

Learning CurveReport



DataShop - Inspection & Evaluation of CTAT Data

Error Report



CTAT’s Track Record• CTAT-based tutors have been used in

(approx. 15) experiments in Geometry, Chemistry, Chinese, and French

• Over 300 users (you not included)

• Four releases in past year (1.6, 2.0, 2.1, 2.2)

• CTAT Web site– 35,000 unique visitors in 2006, 580,000 hits

• CTAT Downloads– 2,200 in 2006 (14% CMU & Pitt)



http://ctat.pact.cs.cmu.edu



Development time comparison• Past estimates of ITS development

200:1 for robust beta versions used in real classrooms

• Example-Tracing Tutor Development25:1 for initial alpha versions: (Koedinger et al, 2004)

# Of PseudoTutors

DesignTime

Dev.Time

InstructionalTime

Design/Devto Instr.

Economics 11 3600 2190 180 32.2

Math Assistments 20 810 1170 98 20.2

LSAT 3 240 3000 180 18.0

Language Learning 8 210 575 50 15.7

Totals 4860 6935 508 23.2

• If creating a robust beta doubles dev time, we would have ~50:1 ratio -- a reduction of 4 times!

Koedinger, K., Aleven, V., Heffernan, N., McLaren, B. M., and Hockenberry, M (2004). Opening the Door to Non-Programmers: Authoring Intelligent Tutor Behavior by Demonstration;. In the Proceedings of the Seventh International Conference on Intelligent Tutoring Systems (ITS-2004).



Where does CTAT fit into the Realm of Authoring Tools?

Murray survey (1999) discusses 7 categories of authoring tools for intelligent tutoring systems (ITS), including:

• Device Simulation - RIDES (Munro et al 1997)• Tutoring Strategies - REDEEM (Ainsworth et al 2003;

Major et al 1997)• Domain Expert System - Demonstr8 (Blessing 1997)

Cognitive Tutor Authoring Tools:• “Domain Expert System” category of authoring systems• No commercially successful tools for authoring model-

tracing, Cognitive Tutors• First to provide functionality to simplify & speed cog.

tutor development



Other Example-Tracing Tutors built with CTAT

Chinese - tone study

Chinese - listen and readFrench culture

FrenchChemistry (stoichiometry)

Geometry (self-assessment) Geometry (on-line testing)



Chinese: tone studyWang, Perfetti, Liu, et al.



Stoichiometry studyMcLaren et al.



Geometry self-assessment tutorRoll et al.



French cultureOgan et al.



Help seekingAleven, McLaren, Roll, et al.



FrenchJones et al.



Chinese: listen and readMei, Haney, et al.



Division Tutor (Stefan King)



Mass Production: Create template behavior graph and spreadsheet with data for each problem

Each row containsvalues for one variablein the Behavior Graphtemplate

Each columncontains data for1 problem

A Behavior Graph iscreated automaticallyfor each problem bymerging the spreadsheetdata into the template



What is Mass Production?• Many learning situations require repetition of

similar problems, but authoring many similar Example-Tracing Tutors is hard– It is time-consuming to demonstrate individual

problems– Inconsistency between problems is often introduced

• Examples:– Practice fraction addition problems in a math class– Drill on proper use of articles in a language class

• CTAT mass production allows us to use a single demonstration as the basis for many tutored problems of the same type



The Mass Production process



Self-explanation of incorrect examplesJulie Booth et al.



Another example - Thermodynamics

Simulator hookedup with CTAT

CTAT provides tutoringfor analyzing power plant efficiency



What type of tutor shall I build? Cognitive Tutors v. Example-Tracing Tutors

• Cognitive Tutor (e.g., Algebra Cognitive Tutor, Logic Tutor)– Technology: uses rule-based cognitive model to monitor students

(implemented in Jess)– Development: Difficult to build; requires AI programming – Advantages: General for a class of problems; can deal with subtle

dependencies of steps, many solution paths, etc.– Sometimes an advantage: SimStudents module writes rules for you

(experimental use of machine learning; tried on a small scale)

• Example-Tracing Tutors (the other examples in this talk)– Technology: uses examples of problem solutions to monitor students– Development: programming by demonstration, “Mass Production”

facilitates authoring of many isomorphic problems (in Excel) – Advantages: Much easier to build; for non-programmers



How to make the choice?• Often, example-tracing tutors are a good choice

– Easier to build– At this point in time, their web delivery is better supported

• Cognitive Tutors only if– You have (AI) programming experience (possibly gained during

this summer school), and– You deal with problems that have subtle dependencies among

steps, or many different solutions paths• Best to build an example-tracing tutor first

– Rapid prototype! And supports cognitive task analysis– Then decide whether to move on to Cognitive Tutor– Example-Tracing Tutors re-usable when developing a Cognitive

Tutor: Examples guide planning, serve as semi-automated test cases

http://ctat.pact.cs.cmu.edu/index.php?id=tutortypes



During the summer school

• Learning to create Example-Tracing Tutors is easy, learning to create Cognitive Tutors is hard

• The CTAT track will cover development of Cognitive Tutors and Example-Tracing Tutors– In your project you could decide to focus mainly on

example-tracing tutors– (Then again, this is your chance to get some

mentoring as you build a Cognitive Tutor)

• If you are not in the CTAT track, but interested in learning to build tutors in limited time, it is best to focus on example-tracing tutors



Options for developing a user interface for your tutor (no

programming!)• Java with Netbeans GUI Builder

– Freely available– Web delivery via WebStart (we are still gaining

experience with this)

• Flash (from Adobe)– Easy to deploy on the web: With plug-in, compatible

with many browsers– Not free



Choices, choices, …• $$$ may be a factor …

• Whether you know Flash or Java need not be a factor– Unless you need to program custom behaviors

• Web delivery of Cognitive Tutors only supported with Java, but web delivery of Example-Tracing tutors is easier in Flash

• Sets of widgets/components are differenthttp://ctat.pact.cs.cmu.edu/index.php?id=reference-widgetshttp://ctat.pact.cs.cmu.edu/index.php?id=reference-components

• See also:http://ctat.pact.cs.cmu.edu/index.php?id=java-flash_differenceshttp://ctat.pact.cs.cmu.edu/index.php?id=tutortypes

• Our recommendation: use Flash if you can



During the summer school

• Examples shown in demos and lectures will come mainly from Java / Netbeans

• For your projects, you are free to choose– If the long-term prospects for your project

are uncertain, then the available widgets/components should be decisive

– Otherwise, all factors mentioned on the previously slide should probably be considered



Summary of choices to be madewhen using CTAT for real• Cognitive Tutor or Example-Tracing

Tutors?– If Cognitive Tutors, use SimStudents?– If Example-Tracing Tutors, use Mass

Production?

• Tutor interface in Flash or Java?– Also possible to program your own



Overview of CTAT track activitiesMonday, June 18 11:00 Discuss project focus 16:00 Finalize project focus

Tuesday, June 19 9:15 How to: ACT-R / Building a cognitive model in Jess 10:00 Hands-on: Create Example-Tracing Tutor or on-line test with CTAT 11:45 Do think alouds of other pair 13:30 How to: Model tracing with a Jess cognitive model 14:15 (NSH 3305) Demo: Create cognitive model with CTAT 16:00 Demo: Create tutor GUI with Netbeans; Design & create GUI from think aloud

Wednesday, June 20 9:30 Hands-on: Create Cognitive Tutor for fraction addition 11:45 Hands-on: Create GUI and ETT for your domain 14:15 Hands-on: Write English production rules/Create your own Cognitive Tutor 15:45 Hands-on: Create your own Cognitive Tutor/Try out Simstudents

Thursday, June 21 10:45 Hands-on: Extend your own Cognitive Tutor 13:30 Hands-on: Finalize Cognitive TutorAfternoon Prepare presentations

Friday, June 22Morning/early afternoonProject presentations



Demo of CTAT



Thank you!

Documents

What can CTAT do for you? Overview of the CTAT track Vincent Aleven, Bruce McLaren and the CTAT team 3rd Annual PSLC LearnLab Summer School Pittsburgh,