Algorithm Visualization · •Two main learning approaches: –Conceptualize algorithm at a high...

Algorithm VisualizationAlgorithm VisualizationReflections and Future DirectionsReflections and Future Directions

John StaskoJohn Stasko

Information Interfaces Research GroupInformation Interfaces Research Group

School of Interactive Computing & GVU CenterSchool of Interactive Computing & GVU Center

Georgia TechGeorgia Tech

stasko@cc.gatech.edustasko@cc.gatech.edu

http://http://www.cc.gatech.edu/~staskowww.cc.gatech.edu/~stasko

ICER – Sept. ‘07 2

IntroductionIntroduction

•• BS Math 1983, BS Math 1983, BucknellBucknell

•• PhD CS 1989, BrownPhD CS 1989, Brown

–– TANGO algorithm animation systemTANGO algorithm animation system

•• On faculty at Georgia Tech since ‘89On faculty at Georgia Tech since ‘89

ICER – Sept. ‘07 3

TANGO

Polka

Polka-RC

ConcurrentProgramsPVM/ConchPThreadsHPF

PVaniM

EvaluationPairheap

Intro Algos

BH & DFS

HW Scenario

Lens

Choreographer

Algorithm Animation Program Visualization

Trees

LargeDataSets

Dance

Information Mural

XTango

89

90

91

92

93

94

95

96

97

98

99

Polka-3D

Samba

ICER – Sept. ‘07 4

Algorithm Animation’s HistoryAlgorithm Animation’s History

•• Quick look back for those who may not Quick look back for those who may not

be so familiarbe so familiar

ICER – Sept. ‘07 5

Sorting Out SortingSorting Out Sorting

•• Seminal work in areaSeminal work in area

•• 30 minute video produced by Ron 30 minute video produced by Ron

BaeckerBaecker at Toronto in 1981at Toronto in 1981

•• Illustrates and compares nine sorting Illustrates and compares nine sorting

algorithms as they run on different data algorithms as they run on different data

setssets

ICER – Sept. ‘07 6

Sorting Out SortingSorting Out SortingVideo

ICER – Sept. ‘07 7

Early Early AlgoAlgo AnimAnim SystemsSystems

•• SmalltalkSmalltalk--based systems, Animusbased systems, Animus

–– Ralph London & Rob Ralph London & Rob DuisbergDuisberg, ,

Univ. of WashingtonUniv. of Washington

•• BALSABALSA

–– Marc Brown, Brown Univ.Marc Brown, Brown Univ.

ICER – Sept. ‘07 8

BalsaBalsa

•• Used in “electronic classroom” at BrownUsed in “electronic classroom” at Brown

•• Introduced use of Introduced use of

multiple views and multiple views and

interesting event interesting event

modelmodel

M. Brown Computer ‘88

ICER – Sept. ‘07 9

Brown’sBrown’sElectronicElectronicClassroomClassroom

Circa 1983

Apollo workstations

ICER – Sept. ‘07 10

TangoTango

•• Added smooth animation to algorithm Added smooth animation to algorithm

animationanimation

•• Simplification of the design/programming Simplification of the design/programming

process (easier programming model and process (easier programming model and

framework)framework)

•• Formal model of the animation, the PathFormal model of the animation, the Path--

Transition ParadigmTransition Paradigm

ICER – Sept. ‘07 11

TangoTango

StaskoIEEE Computer ‘90JVLC ‘90

Multipleframes frombubblesort

ICER – Sept. ‘07 12

POLKAPOLKA

Video Demo

Stasko & KraemerJPDC ‘93

ICER – Sept. ‘07 13

PolkaPolka--3D3D

•• 33--D & VR version, same animation modelD & VR version, same animation model

VL ‘93

Stasko & Wehrli

Quicksort side view

Blue dots are as in2-d view

Colored planes represent exhanges

ICER – Sept. ‘07 14

StudentStudent--Built AnimationsBuilt Animations

•• SambaSamba

–– Simple animation scripting languageSimple animation scripting language

circle 1 0.8 0.8 0.1 red halfline 2 0.1 0.1 0.2 0.2 green thinrectangle 3 0.1 0.9 0.1 0.1 blue solidtext 4 0.0 0.0 0 black Hellocircle 6 0.3 0.3 0.2 wheat solidtriangle 7 0.5 1.0 0.6 0.8 0.4 0.9 cyan solidbigtext 8 0.2 0.2 0 black Some Big Textmoveto 1 6moverelative 3 0.05 -0.4jumprelative 4 0.4 0.4lower 1color 6 blue

StaskoSIGCSE ‘97

- Embed print statements in any program to generate

ICER – Sept. ‘07 15

TransitionTransition

•• In the mid ’90’s, my work began to In the mid ’90’s, my work began to

transition from building systems to transition from building systems to

thinking about the pedagogical thinking about the pedagogical

implications of the technologyimplications of the technology

ICER – Sept. ‘07 16

EvaluationEvaluation

•• Big question:Big question:

–– Do algorithm animations help students learn Do algorithm animations help students learn

and understand better?and understand better?

•• I think that this issue has dominated I think that this issue has dominated

algorithm animation research of the last algorithm animation research of the last

~8 years~8 years

ICER – Sept. ‘07 17

How to Evaluate?How to Evaluate?

•• What’s a good way to evaluate the What’s a good way to evaluate the

pedagogical value of an algorithm pedagogical value of an algorithm

animation?animation?

Ponder

ICER – Sept. ‘07 18

My WorkMy Work

•• Conducted a number of empirical studies Conducted a number of empirical studies

to answer questionto answer question

–– Compare learning with animation to learning Compare learning with animation to learning

withoutwithout

–– An evolving methodologyAn evolving methodology

•• Concluded with a couple reflective Concluded with a couple reflective

articlesarticles

ICER – Sept. ‘07 19

1. Pairing Heaps1. Pairing Heaps

•• Classical experimental designClassical experimental design

–– Exam scenarioExam scenario

–– AnimAnim vs. no vs. no animanim, followed by test, followed by test

•• Results: Just having animation doesn’t make Results: Just having animation doesn’t make learning happenlearning happen

•• Reflection: Difficult for student to leverage Reflection: Difficult for student to leverage animation of complex algorithm when they animation of complex algorithm when they don’t understand algorithm and visual mapping don’t understand algorithm and visual mapping yetyet

•• Reflection: The better students benefit more…Reflection: The better students benefit more…

Stasko, Badre & LewisInterCHI ‘93

ICER – Sept. ‘07 20

2. Introductory Algorithms2. Introductory AlgorithmsSorting, graphsSorting, graphs

•• Lab style methodologyLab style methodology

•• Results: Benefits to animation, though Results: Benefits to animation, though

unevenuneven

•• Reflection: Interaction is the keyReflection: Interaction is the key

–– Students who enter their own data sets into Students who enter their own data sets into

the algorithm benefited from animationthe algorithm benefited from animation

Lawrence, Badre, & StaskoVL ‘94

ICER – Sept. ‘07 21

3. Binomial Heap & DFS3. Binomial Heap & DFS

•• Classical experimental design with Classical experimental design with

prediction addedprediction added

–– Is animation like prediction?Is animation like prediction?

–– Does it help the student to anticipate what Does it help the student to anticipate what

will happen next and learn from that?will happen next and learn from that?

•• Results: Mixed findings, some supportResults: Mixed findings, some support

•• Reflection: We need a new methodologyReflection: We need a new methodology

Byrne, Catrambone & StaskoComputers & Ed ‘99

ICER – Sept. ‘07 22

4. Binomial Heaps4. Binomial Heaps

•• Change to HW scenarioChange to HW scenario

–– Provide student with learning objectives, materials, Provide student with learning objectives, materials, and questions up frontand questions up front

–– Give unlimited work timeGive unlimited work time

•• Results: Significant benefit to animationResults: Significant benefit to animation

–– Students worked longerStudents worked longer

•• Reflection: Animations appear to help Reflection: Animations appear to help motivationmotivation

–– Make a complex algorithm less intimidatingMake a complex algorithm less intimidating

–– Students with animation were “happier”Students with animation were “happier”

Kehoe, Stasko & TaylorIJHCS ‘00

ICER – Sept. ‘07 23

5. Samba5. Samba

•• Students built their own animations Students built their own animations

during algorithms classduring algorithms class

–– Performance on final exam on questions Performance on final exam on questions

about those algorithms was excellentabout those algorithms was excellent

–– Anecdotal evidence that this helped Anecdotal evidence that this helped

understandingunderstanding

–– Students spent a lot of time tweaking Students spent a lot of time tweaking

graphicsgraphics

StaskoSIGCSE ‘97

ICER – Sept. ‘07 24

Unreported WorkUnreported Work

•• InIn--class assessment of class assessment of quicksortquicksort

•• Worked with focus groups of studentsWorked with focus groups of students

•• Developed task analysis methodology for Developed task analysis methodology for

ensuring that animations were illustrating ensuring that animations were illustrating

the “right” thingsthe “right” things

ICER – Sept. ‘07 25

Little Victories…Little Victories…From *******@felix.cc.gatech.edu Wed Feb 25 19:59:22 1998Date: Wed, 25 Feb 1998 19:52:17 -0500To: stasko@cc.gatech.eduFrom: Susan ****** <*******@cc.gatech.edu>Subject: cs3158

I know you always wanted feedback on using algorith m animations...Well, I have another instance where I think it help ed me...

I was taking a sample Computer Science GRE,a question was asked that had five sorting algorith ms as thepossible answers.

I was pondering the question and thinking about the algorithms- some that we've been exposed to for years are easy to remember,almost inherent. But then I came to "shellsort," a nd I stopped fora second, going "aw geez... that was a complicated one..." butthen suddenly I could see that picture of the anima tion in my headand could remember it as clear as a bell.

Thought you'd enjoy it....Susan ******

ICER – Sept. ‘07 26

Let’s Step Back a MomentLet’s Step Back a Moment

•• Some important questions:Some important questions:

–– What does it mean to understand an What does it mean to understand an

algorithm?algorithm?

–– What makes a good algorithm animation?What makes a good algorithm animation?

–– How should algorithm animations be used?How should algorithm animations be used?

–– ……

ICER – Sept. ‘07 27

UnderstandingUnderstanding

•• What does it mean for a student to What does it mean for a student to

understand an algorithm?understand an algorithm?

•• What are we evaluating in these studies?What are we evaluating in these studies?

Ponder

ICER – Sept. ‘07 28

Understanding an Algorithm isUnderstanding an Algorithm is

•• Knowing which tasks, or types of tasks, for which the algorithm Knowing which tasks, or types of tasks, for which the algorithm is suited or appropriateis suited or appropriate

•• Knowing how to “run” the algorithm, knowing its procedures, Knowing how to “run” the algorithm, knowing its procedures, operations, steps and chronologyoperations, steps and chronology

•• Knowing the algorithm’s purpose and what it can accomplishKnowing the algorithm’s purpose and what it can accomplish

•• Analyzing the algorithm’s complexity and efficiencyAnalyzing the algorithm’s complexity and efficiency

•• Having the ability to code or program the algorithmHaving the ability to code or program the algorithm

•• Being able to articulate about the algorithm in one’s own highBeing able to articulate about the algorithm in one’s own high--level (natural language) wordslevel (natural language) words

•• Having the ability to transfer knowledge about it to another Having the ability to transfer knowledge about it to another algorithm or domainalgorithm or domain

•• Knowing where the algorithm fits into the bigger Knowing where the algorithm fits into the bigger taxonomy/methodology of algorithms in generaltaxonomy/methodology of algorithms in general

ICER – Sept. ‘07 29

Typical Student ApproachTypical Student Approach

•• Focus group findingsFocus group findings

•• Two main learning approaches:Two main learning approaches:

–– Conceptualize algorithm at a high level Conceptualize algorithm at a high level (English discourse). Students referred to (English discourse). Students referred to this as being able to describe it in their own this as being able to describe it in their own terms terms (top down)(top down)

–– Start with pseudo code and get good grasp Start with pseudo code and get good grasp of how it works mechanically, including of how it works mechanically, including tracing on data sets tracing on data sets (bottom up)(bottom up)

Newstetter

ICER – Sept. ‘07 30

Learning OrderLearning Order

•• A. Knowing which tasks, or types of tasks, for A. Knowing which tasks, or types of tasks, for which the algorithm is suited or appropriatewhich the algorithm is suited or appropriate

•• B. Knowing how to “run” the algorithm, B. Knowing how to “run” the algorithm, knowing its procedures, operations, steps and knowing its procedures, operations, steps and chronologychronology

•• C. Knowing the algorithm’s purpose and what C. Knowing the algorithm’s purpose and what it can accomplishit can accomplish

•• D. Analyzing the algorithm’s complexity and D. Analyzing the algorithm’s complexity and efficiencyefficiency

•• E. Having the ability to code or program the E. Having the ability to code or program the algorithmalgorithm

•• F. Being able to articulate about the algorithm F. Being able to articulate about the algorithm in one’s own highin one’s own high--level (natural language) level (natural language) wordswords

•• G. Having the ability to transfer knowledge G. Having the ability to transfer knowledge about it to another algorithm or domainabout it to another algorithm or domain

•• H. Knowing where the algorithm fits into the H. Knowing where the algorithm fits into the bigger taxonomy/methodology of algorithms in bigger taxonomy/methodology of algorithms in generalgeneral

•• A & CA & C

oror

B & FB & F

•• Other pair aboveOther pair above

•• EE

•• DD

•• HH

•• GG Ponder

ICER – Sept. ‘07 31

Algorithm Animation’s Value?Algorithm Animation’s Value?

•• For which of the learning objectives can For which of the learning objectives can

algorithm animations help?algorithm animations help?

•• Perhaps better: For which will algorithm Perhaps better: For which will algorithm

animation have greater learning animation have greater learning

potential?potential?

ICER – Sept. ‘07 32

Assess AA UtilityAssess AA Utility

•• A. Knowing which tasks, or types of tasks, for which the A. Knowing which tasks, or types of tasks, for which the algorithm is suited or appropriatealgorithm is suited or appropriate

•• B. Knowing how to “run” the algorithm, knowing its procedures, B. Knowing how to “run” the algorithm, knowing its procedures, operations, steps and chronologyoperations, steps and chronology

•• C. Knowing the algorithm’s purpose and what it can accomplishC. Knowing the algorithm’s purpose and what it can accomplish

•• D. Analyzing the algorithm’s complexity and efficiencyD. Analyzing the algorithm’s complexity and efficiency

•• E. Having the ability to code or program the algorithmE. Having the ability to code or program the algorithm

•• F. Being able to articulate about the algorithm in one’s own higF. Being able to articulate about the algorithm in one’s own highh--level (natural language) wordslevel (natural language) words

•• G. Having the ability to transfer knowledge about it to another G. Having the ability to transfer knowledge about it to another algorithm or domainalgorithm or domain

•• H. Knowing where the algorithm fits into the bigger H. Knowing where the algorithm fits into the bigger taxonomy/methodology of algorithms in generaltaxonomy/methodology of algorithms in general

Ponder

ICER – Sept. ‘07 33

What Makes a Good Algorithm Animation?What Makes a Good Algorithm Animation?

•• What are the elements of an effective What are the elements of an effective

visualization/animation?visualization/animation?

•• How does one design a visualization that How does one design a visualization that

students can learn from?students can learn from?

Ponder

ICER – Sept. ‘07 34

Good AA DesignGood AA Design

•• Very little work done on this topicVery little work done on this topic

•• Design has largely been a very individual, Design has largely been a very individual,

intuitive matterintuitive matter

•• Involves graphic design, pedagogy, Involves graphic design, pedagogy,

understanding of misunderstandings, understanding of misunderstandings,

story telling, artistic sense, …story telling, artistic sense, …

ICER – Sept. ‘07 35

Multiple ViewsMultiple Views

•• Can be quite valuable but also Can be quite valuable but also

problematicproblematic

–– We can only look one place at a time (people We can only look one place at a time (people

may not notice the text at the bottom of one may not notice the text at the bottom of one

important view)important view)

ICER – Sept. ‘07 36

HeapsortHeapsort Case StudyCase Study

•• Taught psychology faculty colleague the Taught psychology faculty colleague the

heapsortheapsort algorithm using an existing algorithm using an existing

animationanimation

•• Q & A session uncovered many problems Q & A session uncovered many problems

with the animationwith the animation

–– 20+ changes made20+ changes made

ICER – Sept. ‘07 37

HeapsortHeapsort ChangesChanges

•• Show numbers appear in array view Show numbers appear in array view first (reading them in). Then place first (reading them in). Then place them in the tree view and (somehow) them in the tree view and (somehow) highlight the correspondences. highlight the correspondences.

•• Put better text region somewhere for Put better text region somewhere for telling viewer what is happening. telling viewer what is happening.

•• Use circle with small lines out sides Use circle with small lines out sides (indicating (indicating subtreesubtree) to tell where a heap ) to tell where a heap is being checked.is being checked.

•• Change color of node "dropping down" Change color of node "dropping down" in in heapifyheapify operation. Rather than operation. Rather than putting the "H" there.putting the "H" there.

•• Put some text like "Comparing key Put some text like "Comparing key values" and "Swapping with biggest“ in values" and "Swapping with biggest“ in tree view when that occurs. Similarly, tree view when that occurs. Similarly, put some text in array view to draw put some text in array view to draw viewers attention to swaps there.viewers attention to swaps there.

•• Put up text to tell when valid heap has Put up text to tell when valid heap has been built. been built.

•• Put up text to indicate when root and Put up text to indicate when root and

last leaf are being swapped. last leaf are being swapped.

•• Change color of node when it has been Change color of node when it has been

moved out of tree and is in place (do in moved out of tree and is in place (do in

both views). both views).

•• Take circle away at this later stage of Take circle away at this later stage of

algorithm. algorithm.

•• Change color of node moving down in Change color of node moving down in

heapifyheapify..

•• Pause longer after Pause longer after heapifyheapify finishes. finishes.

•• Put text on both sides of line in array Put text on both sides of line in array

view, indicating regions that are sorted view, indicating regions that are sorted

and unsorted respectively. and unsorted respectively.

•• Take out that "Elevate #" text at top.Take out that "Elevate #" text at top.

•• Move line over in array view to very left Move line over in array view to very left

end.end.

Video Demo

ICER – Sept. ‘07 38

The Learner is Your UserThe Learner is Your User

•• Understanding an algorithm well Understanding an algorithm well

handicaps you in designing an animation handicaps you in designing an animation

of itof it

•• We should learn from HCIWe should learn from HCI

–– Have learners as part of your design teamHave learners as part of your design team

–– Iteratively prototype animation Iteratively prototype animation

–– Test it with other learnersTest it with other learners

ICER – Sept. ‘07 39

Algorithm Animation’s PlaceAlgorithm Animation’s Place

•• How should we use algorithm animations How should we use algorithm animations

for learner benefit?for learner benefit?

–– What’s the scenario of use?What’s the scenario of use?

–– When?When?

–– Where?Where?

–– How? How?

Ponder

ICER – Sept. ‘07 40

Potential Use ScenariosPotential Use Scenarios

•• In classIn class

–– When and how?When and how?

•• LabLab

•• OfflineOffline

ICER – Sept. ‘07 41

Typical ClassTypical Class

•• A discussion of the motivation for an algorithm and how it mightA discussion of the motivation for an algorithm and how it mightbe used in the real world. Frequently, analogy is used to give be used in the real world. Frequently, analogy is used to give the students a feeling for the practical value of the algorithmthe students a feeling for the practical value of the algorithm

•• A discussion of the high level methodology and general A discussion of the high level methodology and general operations of an algorithm. Here, a very informal description ioperations of an algorithm. Here, a very informal description is s provided, attempting to give the students a "feel" for the provided, attempting to give the students a "feel" for the algorithmalgorithm

•• A discussion of the detailed methodology and operations of an A discussion of the detailed methodology and operations of an algorithm, with a focus on tracing the algorithm in pseudoalgorithm, with a focus on tracing the algorithm in pseudo--code, code, understanding its control flow and memory usageunderstanding its control flow and memory usage

•• A discussion of the algorithm's computational complexity, that iA discussion of the algorithm's computational complexity, that is, s, an analysis of its space and time requirementsan analysis of its space and time requirements

•• A discussion revolving around the presentation of an algorithm A discussion revolving around the presentation of an algorithm animation illustrating the algorithm's methods and operationsanimation illustrating the algorithm's methods and operations

ICER – Sept. ‘07 42

RepresentationRepresentation

•• Note that each lecture activity involves a Note that each lecture activity involves a

different kind of representationdifferent kind of representation

•• Each uses a set of artifacts and Each uses a set of artifacts and

discussion terms from a different discussion terms from a different

domain/languagedomain/language

ICER – Sept. ‘07 43

ComponentsComponents

•• Motivation/ExampleMotivation/Example

•• HighHigh--level level

methodologymethodology

•• Detailed Detailed

methodologymethodology

•• Complexity analysisComplexity analysis

•• Algorithm animationAlgorithm animation

•• RealReal--world things, world things,

artifacts, problemsartifacts, problems

•• English discourseEnglish discourse

•• Pseudo code or Pseudo code or

programming lang.programming lang.

•• Math notationMath notation

•• Graphics and Graphics and

visualizationvisualization

ICER – Sept. ‘07 44

ReflectionsReflections

•• Running the sessions amazingly instructive to Running the sessions amazingly instructive to

meme

–– Often saw just how an animation wasn’t helping a Often saw just how an animation wasn’t helping a

studentstudent

•• Two papers reflect on the evaluation findings of Two papers reflect on the evaluation findings of

many researchersmany researchers

–– “Meta“Meta--EvalEval of AV,” of AV,” HundhausenHundhausen, Douglas, & , Douglas, & StaskoStasko, ,

JVLCJVLC , June ’02, June ’02

–– ““AlgoAlgo VisVis,” ,” StaskoStasko & & HundhausenHundhausen, appears in , appears in CS CS Education Research, Education Research, ’04’04

ICER – Sept. ‘07 45

ResultsResults

•• Findings definitely mixed, but some studies do Findings definitely mixed, but some studies do

show benefitsshow benefits

–– ------> Potential for pedagogical aid is there, but just > Potential for pedagogical aid is there, but just

can’t throw animation at algorithm and expect it to can’t throw animation at algorithm and expect it to

helphelp

–– Interaction & engagement is keyInteraction & engagement is key

•• Blindly watching algorithm animation not really helpfulBlindly watching algorithm animation not really helpful

•• Student must interact with animation and be engagedStudent must interact with animation and be engaged

–– Animations help motivation, can make algorithm less Animations help motivation, can make algorithm less

intimidatingintimidating

ICER – Sept. ‘07 46

Future WorkFuture Work

•• What’s still missing in algorithm What’s still missing in algorithm

visualization research?visualization research?

–– Further evaluation (somewhat)Further evaluation (somewhat)

–– Simpler animation construction (somewhat) Simpler animation construction (somewhat)

–– Design principlesDesign principles

–– Giant web indexGiant web index

–– Integrated curriculum Integrated curriculum

–– Interactive classroom blackboard toolsInteractive classroom blackboard tools

ICER – Sept. ‘07 47

Controversial ThoughtsControversial Thoughts

•• Age breeds cynicismAge breeds cynicism

•• Let me push a few buttonsLet me push a few buttons

ICER – Sept. ‘07 48

Teach, Don’t RationalizeTeach, Don’t Rationalize

•• The people who seem to benefit most The people who seem to benefit most

from algorithm animations are those who from algorithm animations are those who

already understand the algorithmalready understand the algorithm

•• MaybeMaybe algoalgo animanim helps instructors helps instructors

present better (or think they present present better (or think they present

better) and that’s why they like itbetter) and that’s why they like it

–– But do students learn better?But do students learn better?

Ponder

ICER – Sept. ‘07 49

No Good Use ScenarioNo Good Use Scenario

•• It takes too much time to use an It takes too much time to use an

algorithm visualization well in classalgorithm visualization well in class

•• Difficult to get students to use them Difficult to get students to use them

offlineoffline

Ponder

ICER – Sept. ‘07 50

Software VisualizationSoftware Visualization

•• Good introductionGood introduction

to field to field

•• Many AA chaptersMany AA chapters

–– HistoryHistory

–– DesignDesign

–– SoSSoS

–– EvaluationEvaluation

ICER – Sept. ‘07 51

AcknowledgmentsAcknowledgments

•• Research supported by National Science Research supported by National Science

Foundation and the GVU CenterFoundation and the GVU Center

•• More infoMore info

–– http://www.cc.gatech.edu/gvu/iihttp://www.cc.gatech.edu/gvu/ii

–– http://http://www.cc.gatech.edu/gvu/softvizwww.cc.gatech.edu/gvu/softviz

–– stasko@cc.gatech.edustasko@cc.gatech.edu

ICER – Sept. ‘07 52

Thanks for Your Attention!Thanks for Your Attention!

•• Questions?Questions?