Parametric Design of Interactive Experiences

A Trace-based Framework for Analyzing and Synthesizing Educational ProgressionsErik Andersen Sumit Gulwani Zoran Popovi

Hello, my name is Erik Andersen and I am a graduate student in the Center for Game Science at the University of Washington. I worked on this project with my advisors Zoran Popovic and Sumit Gulwani at Microsoft Research. Today, I am going to present a new framework that greatly increases our ability to automate task design and progression design for many domains including software tutorials, education, and video games. My hope is that these ideas will enable you to increase the amount of automation in your own research and reduce the need for expert design.1Procedural Learning

One of the most important domains of education is procedural learning. People learn to follow procedures to build an airplane, to follow a cooking recipe, to succeed at some kinds of video games, and to solve math problems. What is common to all of these domains is that there is an algorithm that we want the student to learn. This algorithm involves atomic steps but may also include conditional branches and loops.2Software Learnability

Dong et al. CHI 2012Procedural learning is also important to HCI because it is a part of software learnability. For example, Tao Dong and some colleagues from Adobe presented some work last year in which they created a set of minigames to help people learn how to execute procedures within Adobe Photoshop. HCI researchers often want to evaluate the degree to which a user interface facilitates the learning of procedures, and increase learnability as much as possible.3

The standard way people learn how to execute procedures is by solving progressions of practice problems. If we look at a math textbook, we may notice many different problem types. Some are sort of standard 2-D tables, other problem types include shapes and line drawings. What is common to all of these different kinds of problems is that there is a progression of problems that start easy and get harder. We have learned some principles of how to design an effective progression from psychology and education.4FlowSkillChallengeAnxietyBoredomM. CsikszentmihalyiFor example, Csikszentmihalyis theory of flow suggests that as the learners skill improves, problem difficulty should also increase proportionately.5Zone of Proximal Developmentcan do nowcan do with guidancecant do yetL. VygotskyLev Vygotsky proposed that learners have a zone of proximal development, essentially there is always a set of things the learner can do now, a set of things the learner cannot do yet, and a set of things that they should learn next in order to advance their knowledge. Expert designers often apply principles such as these when creating progressions, but we lack a pragmatic and automatic way to do this in general. Furthermore, progressions are typically authored by hand, and little can be reused across problem domains. If we want to teach something new, or if we want to build an adaptive progression that can actually change depending on how the student is doing, we need a lot of authoring. Just to give one example, it has been estimated that 200-300 hours of content are necessary to produce a single hour of content for cognitive tutors.6

Therefore, we believe that automating progression design can dramatically reduce this effort. In order to automate the creation of progressions for all procedures, we need to understand procedural learning not just as an algorithm and a set of practice problems, but as a collection of smaller conceptual building blocks. We can then systematically introduce these building blocks with a set of knobs to create a progression of problems.7Procedural TracesWhat are traces?

What can we do with traces?We propose that we can create a set of building blocks for procedural learning by examining the specific set of decisions that the learner needs to make when solving a practice problem with a target procedure. We call this a trace. First I will give a concrete example of what I mean, and then I will explain why this representation is useful.8

To illustrate how this works, lets look at an example of an algorithm. This is a worksheet on integer comparison from a textbook called JUMP math which is used in 15% of Canadian schools.9same # of digits?NOgreater # of digits?NOreport lessYESreport greaterYESfirst digit same?first digit greater?NOreport lessYESreport greaterNOsecond digit same?second digit greater?NOreport lessYESreport greaterNOYESYES...Explain algorithm. We argue that one of the most useful characteristics of a practice problem for procedural learning in terms of creating progressions is the pathway that the solution procedure takes when solving that problem. In other words, what is the specific set of decisions the problem makes you think about, and what do you decide at each of these moments? We call this a trace.10same # of digits?NOgreater # of digits?NOreport lessYESreport greaterYESfirst digit same?first digit greater?NOreport lessYESreport greaterNOsecond digit same?second digit greater?NOreport lessYESreport greaterNOYESYES...DDSLHDGGSWe will indicate traces through this procedure using letters. We will represent the full trace as a sequence of letters that are accumulated when the execution flow hits them.11same # of digits?NOgreater # of digits?NOreport lessYESreport greaterYESfirst digit same?first digit greater?NOreport lessYESreport greaterNOsecond digit same?second digit greater?NOreport lessYESreport greaterNOYESYES...DDSLHDGGS

LI will now show some of the traces for this procedure. This is L which corresponds to the situation where the first number has fewer digits than the second number and is therefore smaller.12same # of digits?NOgreater # of digits?NOreport lessYESreport greaterYESfirst digit same?first digit greater?NOreport lessYESreport greaterNOsecond digit same?second digit greater?NOreport lessYESreport greaterNOYESYES...DDSLHDGGSL,

HThis is H.13same # of digits?NOgreater # of digits?NOreport lessYESreport greaterYESfirst digit same?first digit greater?NOreport lessYESreport greaterNOsecond digit same?second digit greater?NOreport lessYESreport greaterNOYESYES...DDSLHDGGSL, H,

DSThis is DS, and so forth.14same # of digits?NOgreater # of digits?NOreport lessYESreport greaterYESfirst digit same?first digit greater?NOreport lessYESreport greaterNOsecond digit same?second digit greater?NOreport lessYESreport greaterNOYESYES...DDSLHDGGSL, H, DS,

DG15same # of digits?NOgreater # of digits?NOreport lessYESreport greaterYESfirst digit same?first digit greater?NOreport lessYESreport greaterNOsecond digit same?second digit greater?NOreport lessYESreport greaterNOYESYES...DDSLHDGGSL, H, DS, DG,

DDS16same # of digits?NOgreater # of digits?NOreport lessYESreport greaterYESfirst digit same?first digit greater?NOreport lessYESreport greaterNOsecond digit same?second digit greater?NOreport lessYESreport greaterNOYESYES...DDSLHDGGSL, H, DS, DG, DDS, DDGSo now we have identified some traces but there are really a lot more.17same # of digits?NOgreater # of digits?NOreport lessYESreport greaterYESfirst digit same?first digit greater?NOreport lessYESreport greaterNOsecond digit same?second digit greater?NOreport lessYESreport greaterNOYESYES...DDSLHDGGSCompare 313 to 324.This provides a useful way to characterize problems. I would like everyone to think about 18same # of digits?NOgreater # of digits?NOreport lessYESreport greaterYESfirst digit same?first digit greater?NOreport lessYESreport greaterNOsecond digit same?second digit greater?NOreport lessYESreport greaterNOYESYES...DDSLHDGGSCompare 313 to 324.

19What can we do with traces?Rank their difficulty

Analyze and compare progressions

Synthesize new progressionsWhat can we do with traces?Rank their difficulty

Analyze and compare progressions

Synthesize new progressionsPartial OrderingLoop executes longer = hardermore digits in number comparisonmore columns in subtraction

Exceptional conditions = harderif boolean expression: statementborrowing in subtractioncarrying in addition

n-gramsWe can expand this partial ordering to a larger family of partial orderings by considering n-grams of statements. 231-gramsthe boy threw the balln-grams are an idea from natural language processing that provide a formal way to consider sequences of words.242-gramsthe boy threw the ball25nn-grams1H, L, D, S, G2DD, DS, DG3DDS, DDGWe can also consider n-grams of statements. In the integer comparison algorithm I showed before, here are the possible n-grams. We can look at a progression to look at which n-grams are missing; for example I mentioned that H and L were missing from the JUMP math progression before.26n-gram based Partial OrderingWe can also define a partial ordering with n-grams

statement coverage => is every statement of T2 covered by T1?path coverage => is the entire trace of T2 contained inside T1?

point out that there are more metrics27What can we do with traces?Rank their difficulty

Analyze and compare progressions

Synthesize new progressionsif length(n1) < length(n2): return less L if length(n1) > length(n2): return greater H for i 0 to length(n1) 1: D if n1[i]< n2[i]: return less Sif n1[i]> n2[i]: return greater Greturn equal

29

JUMP MathThis framework for analyzing practice problems is useful because we can create graphs to visualize progressions. This graph shows the progression of problems found in the JUMP Math curriculum. It starts with the trace DDG, then moves to DG, then DDDS, and back to DDG. This progression does not appear to be very systematic and instead appears to move between the problem types at random.30

JUMP MathSingapore MathWe can also compare progressions from different textbooks. The blue nodes and lines indicate a problem progression from a different textbook. We can see that this progression includes two traces, H and L, that are completely missing from the JUMP Math progression. Furthermore, this other progression moves back and forth between shorter traces that involve fewer decisions, whereas the JUMP Math progression moves quickly into harder traces like DDDS and DDDG. The other progression never reaches these longer traces, which correspond to comparing two three-digit numbers in which the first two digit pairs are the same.31

JUMP MathSingapore MathBoth of these curricula were optimized but they are different. What do we conclude? They are both optimal? What is more likely is that they are both suboptimal.32What can we do with traces?Rank their difficulty

Analyze and compare progressions

Synthesize new progressionsif length(n1) < length(n2): return less L if length(n1) > length(n2): return greater H for i 0 to length(n1) 1: D if n1[i]< n2[i]: return less Sif n1[i]> n2[i]: return greater Greturn equal

DDS11 < 12it turns out there is 34ProblemTimeUnique TracesAddition64 min1433Fraction Computation16 min72Fraction Reduction2 min29Integer Comparison2 min26Subtraction2 min157Prime Factorization2 min70some you can just enumerate, but some you cant.35DragonBox

we wanted to try this on something other than K-12 math, especially something that has a really really huge design space.36DragonBox

So we did the same thing for this game.37

This is huge. This is a graph of the traces for this game up to length three. The black edges represent what was in the original progression, and the red edges represent the traces discovered by our process. The original game had 100 levels and we were able to automatically generate thousands of traces in a few hours, including all of the traces in the original progression. Therefore, we were able to replicate hundreds of hours of design effort in one single automatic design process, and we can use our partial ordering to assemble them into some progression, or just stick with the original one. You wouldnt necessarily want to give ALL of these levels to the user, but this greatly increases our ability to build adaptive progression that can diagnose misconceptions by choosing highly targeted levels.38Validating Our Partial Ordering20 participants20 level pairsUsers ranked pairs: easier, harder, the sameCalculated Root-Mean-Squared-Errorour ordering: 0.35random baseline: 1.00We wanted to confirm that our partial ordering is correct. So we did a study.39Summarynew representation of problem difficulty

allows us to:rank problem difficultyanalyze and compare progressionsgenerate an entire progression

pragmatic way to make content fit theory

To summarize, I have introduced a new representation of the space of problem difficulty for procedural learning. I have shown how it can be used to rank the difficulty of practice problems and I presented a little bit of data supporting these ideas. I have also shown how we can use this framework to analyze, compare, and generate progressions. But what excites me most about this work is that were able to use the natural hierarchies that exist within the model of what we are trying to teach, in this case a procedure, to suggest a pragmatic, automatic, and general way to assemble educational content conforming to theories of how people learn. This is where I think there is the most potential for future work and I wonder if there are other domains of learning in which we can automate content design in a similar way. 40AcknowledgmentsHelp: David Yamanoha and Ryoko Nozawa

Funding: Bill and Melinda Gates Foundation, Office of Naval Research, Hewlett Foundation, Adobe, and Intel

Contact me: eland@cs.washington.edu

I would like to thank some people who helped me run the user study and digitize problems from the worksheets.41From hereautomate other thingstutorialsdiagnosis of student errorsFlowSkillChallengeAnxietyBoredomWe can provide a theoretical argument for secondary objectives by considering basically says that players reach maximal enjoyment when a games challenge is matched to their skill level. If the challenge increases too quickly, the player will become anxious, and if the challenge increases too slowly, the player will become bored. 45ImplicationsGames

Education

Games are a series of interesting decisions. Sid MeierArtifacts

Artifacts Progressions

same # of digits?NOgreater # of digits?NOreport lessYESreport greaterYESfirst digit same?first digit greater?NOreport lessYESreport greaterNOsecond digit same?second digit greater?NOreport lessYESreport greaterNOYESYES...DDSLHDGGSCompare 32 to 314.Quiz. What is the trace of 32 and 314. (L)50same # of digits?NOgreater # of digits?NOreport lessYESreport greaterYESfirst digit same?first digit greater?NOreport lessYESreport greaterNOsecond digit same?second digit greater?NOreport lessYESreport greaterNOYESYES...DDSLHDGGSDDDDG1112 > 111111111 and 1111251

Recently there has been a surge in interest in using technology in education. We have seen the rise of massively open online courses through companies like Coursera. Online video tutorials have become popular through websites like the Khan Academy. There have also been games for learning. I was one of the co-creators of a math learning game, Refraction. For a short time last year, an algebra learning game called DragonBox actually became the most bought app in Norway in the Apple app store.52

In all of these domains, 53ProblemBookProblemsTraces% genAdditionSS:M5841788%SMS5764864%Fraction Addition & SubtractionSS:M6613100%SMS24815100%JUMP1311195%Fraction ReductionSS:M6613100%JUMP5611100%Integer ComparisonSS:M246100%JUMP8812100%if it didnt generate it made one like it. also straightline procedure54HCIeducationsoftware engineeringvideo games

Today I am going to tell you about a project that combines ideas from HCI, education, software engineering, and video games. I know that you, the members of the audience, may belong to one, or all, or none of these areas.55Promisenew representation of problem difficulty

cananalyze problem setsgenerate an entire progressionRegardless of what you are interested in, my objective is to empower you in two ways. First, I want to propose a new representation of the space of difficulty of practice problems for procedural learning. Second, I want to demonstrate the power of this representation, by showing how we can use it to automatically evaluate the completeness of a problem set, and generate complete progressions of practice problems.56Promisenew representation of problem difficulty

cananalyze problem setsgenerate an entire progressionWe can even use it at the scale of generating all of the levels of a popular educational game. My hope is that you will be able to use these ideas to automate certain aspects of your own research.57if length(n1) < length(n2): return less L if length(n1) > length(n2): return greater H for i 0 to length(n1) 1: D if n1[i]< n2[i]: return less Sif n1[i]> n2[i]: return greater Greturn equal

58same # of digits?NOgreater # of digits?NOreport lessYESreport greaterYESfirst digit same?first digit greater?NOreport lessYESreport greaterNOsecond digit same?second digit greater?NOreport lessYESreport greaterNOYESYES...same # of digits?NOgreater # of digits?NOreport lessYESreport greaterYESfirst digit same?first digit greater?NOreport lessYESreport greaterNOsecond digit same?second digit greater?NOreport lessYESreport greaterNOYESYES...DDSLHDGGSDDS11 < 12Quiz. What is the trace of 323 and 314. (DG). It turns out that there are software verification tools that do this automatically.60

DragonBox LevelSolveraction1action2action3...

AABAAAABABCn-gram based Partial Orderingthe boy passed the ballunigrams: the, boy, passed, ballbigrams: the boy, boy passed, passed the, the balltrigrams: the boy passed, boy passed the, passed the ballWe can expand this to a larger family of partial orderings by considering n-grams62n-gram based Partial OrderingInteger Comparisonunigrams: H, L, D, G, Sbigrams: DD, DG, DStrigrams: DDG, DDSWe can expand this to a larger family of partial orderings by considering n-grams63Promisenew representation of problem difficulty

cananalyze problem setsgenerate an entire progressionIn this talk I have introduced have dened a trace-based framework for using execution traces to explore the space of problem progressions for any procedural task that can be specied as a computer program. I have shown how this trace-based framework can categorize problems according to their traces, dene partial orderings that can compare two problems and assemble a setof problems into a progression, and analyze the thoroughness and depth of existing progressions.64Promisenew representation of problem difficulty

cananalyze problem setsgenerate an entire progressionI have also shown how we can use tools from the software engineering community to generate progressions in a single automatic process. I strongly believe that automating educational content generation in this way will greatly reduce the design effort required to create effective learning progressions and reduce the barriers towards creating scaffolding for procedural learning.65

This is huge66

Fraction Addition and SubtractionInteger Additionsame book67