Tangible Math - cs.stonybrook.edu

Tangible MathLori L. Scarlatos

Department of Computer and Information Science, Brooklyn College of the City University of New York,2900 Bedford Avenue, Brooklyn, NY 11210

Email: [email protected]

Educators recognize that group work and physical involvement with learning materials can greatlyenhance the understanding and retention of difficult concepts. As a result, math manipulatives - such aspattern blocks and number lines - have increasingly been making their way into classrooms and children'smuseums. Yet without the constant guidance of a teacher, students can easily become distracted,confused, or frustrated.

This paper describes how math games with tangible user interfaces can address this need. Tangible userinterfaces allow the students to work together in a physical environment, while the underlying computersystem provides the guidance that a teacher would. This paper discusses pedagogical principles and anapproach to designing and developing games that utilize tangible technologies. A library of functions,designed specifically for tracking visual tags in math games, is presented. The paper concludes withexamples of math games that have been prototyped with this library.

Keywords: tangible user interface, physical computing, multi-modal input, collaborative learning, mathmanipulatives

INTRODUCTIONIn an increasingly "flat earth" economy, mathand science education is vital to a country'scompetitiveness in a global market. PresidentBush reiterated this national priority in hisFebruary 2006 State of the Union speech (Bush2006). Yet it is not enough for students to learnmath; they must also understand it.

Math games – using manipulatives, puzzles,and physical activities – provide numerousbenefits in the learning environment. First, theyprovide concrete representations of abstractconcepts, thereby helping more children tounderstand those concepts. Second, they provideand opportunity for children to explore and testtheir understanding of math concepts. Third,they enable groups of children to work together,talk about the problem at hand, and learn fromone another. Fourth, they give active young This paper, which primarily presents new results and un-published work, contains references to some work that hasbeen presented in conference and technical papers, as citedin the references.

children the opportunity to be physicallyengaged in their lessons, as opposed to sittingthrough a "boring" lecture or filling outworkbooks. Finally, these physical activitiesgive children a fun way to practice their mathskills, which helps with retention and transfer ascognitive scientists have shown.

Because of all these benefits, schools andmuseums alike have embraced the use ofphysical learning activities to supplementlearning. Teachers will often divide their classesinto small groups and have them work on aparticular manipulative. Other times, the teacherwill use overhead displays to talk about themanipulatives while the children work at theirdesks with their own pieces. At the GoudreauMuseum for Mathematics in Art and Science,children look forward to being able to play withthe math puzzles after a more formal lesson. Thetrouble comes in trying to direct and help thirty-five children working on twenty-two puzzles allat once. Students often need an attentive teacheror guide, who will ask thought-provokingquestions and make suggestions that will help

them to discover the solution on their own. If theteacher is busy with one group, other groups thatare "stuck" will quickly get bored and often findinappropriate activities to occupy themselveswith. At the New York Hall of Science, student"explainers" are available to provide help asneeded. This museum has discovered that havinga knowledgeable "guide on the side" helps theirvisitors to get more out of the exhibits andactivities. Yet in some settings, this is notalways practical.

We address this issue by developing mathgames that employ tangible user interfaces.Tangible user interfaces are ideal because theyallow students to explore, test, and practicemathematics both physically and collaboratively.By making the math manipulatives part of theuser interface, the computer application is awareof what the children are doing, and can thereforeprovide appropriate feedback as needed. Thesoftware can also record everything the childrendo, so the teacher can refer back to it later.

Our contributions in this area are twofold.First, we have developed a pedagogicalapproach that emphasizes physical activity,collaboration, practice, and teachercustomization. Second, we have created a libraryof functions that enable a developer to rapidlyprototype a game with vision-based tracking in atangible user interface. This library is availablefor free on our website. We have used both theapproach and the library to develop a series ofmath games with tangible user interfaces.

This paper presents the background researchmotivating this work, the pedagogical ideals thatguide this work, and the tools that we use tocreate educational applications. It also describessome of the math games that we developed,including one with a front-end tool that enablesthe teacher to customize an educationalapplication to suit his or her curriculum, class,even individual students. Although the focus ofthis paper is on math games, the approach – andthe tools – described here may be readilyextended to other learning activities in thesciences, social sciences, and language arts.

RELATED WORKThe educational literature is riddled withexamples of how collaborative and exploratory

learning leads to greater understanding. This hasled to calls for teachers to engage students in"active" learning. The National Council ofTeachers of Mathematics (2000) states,"Students must learn mathematics withunderstanding [by] actively building newknowledge from experience ...". Holt (1983)argues that children, when encouraged toexplore on their own, will come to understandmathematical principles that most educatorswould not dare to try to teach them. Gardner(1983) theorizes that different people can be"intelligent" in a variety of different ways, andthat teachers must therefore address thesemultiple intelligences in order to teacheffectively.

As a result, math teachers are beingencouraged to involve students in collaborativeactivities, plan differentiated tasks, and use"workstations" to present the same material indifferent ways (Bransford et al, 2000). Inresponse, teachers increasingly rely on mathmanipulatives – such as Cuissenaire Rods,Pattern Blocks, and Tangrams – to illustratemath concepts and provide students with a wayof further exploring those concepts. Apparentlychildren who normally have trouble with mathfind the concepts more understandable whenthey are represented by tangible objects.

Tangible User InterfacesTangible user interfaces (TUI) are computer-human interfaces in which the boundariesbetween data and representation are blurred(Ishii 1997). With this paradigm, datarepresentations are tangible, or physical, entitiesin the environment. Manipulating these tangibleentities results in corresponding manipulationsof the data in the computer's memory. As aconsequence, working with the data becomes farmore intuitive (and potentially collaborative)than it is with a traditional graphical userinterface using a mouse or keyboard. At a timewhen teachers are being encouraged to makelearning experiences more hands-on, researchersare exploring ways of using TUI for educationalpurposes.

For example, several TUI projects aredesigned to develop children's language artsskills by encouraging them to tell stories (Zhou

et al, 2004 and Moher et al, 2005). In both ofthese cases, the TUI provides props for tellingthe stories as well as a means of recording thestories. In a variation on this theme, Stanton et al(2001) have had children help to design aninterface that enables other children to tell theirstories. The advantage of involving children inthe design process is that the result is morelikely to be usable by children than anapplication written by adults with little insightinto how children think (Druin 1999).

Tangible user interfaces are also used toteach children about how systems work(McNerney 2004, Zuckerman 2005). In both ofthese cases, children physically connectcomputational components to build a system.Resulting connections among these componentsdetermines the behavior of the system; childrenare then encouraged to make changes to thesystem configuration to see the effects on theoutput. Horn and Jacob (2006) have taken thisidea a step further, having developed a tangibleprogramming language called Quetzal. Elementsof the language are represented by physicalcomponents that look a lot like elements in aflow chart. Children connect these (in flow chartfashion) to "write" a simple program for an RCXbrick.

A more recent trend is to look forframeworks for analyzing and creatingapplications with tangible user interfaces.Marshall et al (2003) define different ways ofconceptualizing tangibles to support learning.Specifically, they look at how learners will usethe tangibles ("ready-at-hand" versus "present-to-hand"), the learning activities they willengage in (expressive versus exploratory), andthe model embodied by the tangibles (theoreticalversus practical). Calvillo-Gomez et al (2003)and Ullmer et al (2005) both propose using"token+constraint" paradigms as a way ofunderstanding all tangible user interfaces. Inthese paradigms, tokens are the physicalrepresentations of data and constraints definehow those tokens can be manipulated. Fishkin(2004) uses metaphor (i.e. the way that thetangible represents abstract concepts) andembodiment (i.e. how closely tied the input andoutput representations are) in his taxonomy forcomparing and contrasting work on tangible userinterfaces.

Our project is unique in that it focuses onTUI in math and science educational software(Scarlatos 2002). Building on this initial work,we have developed our approach to a pointwhere our principles and tools may be used by amuch wider audience to create a variety ofTangible Math (and science) games.

PEDAGOGICAL APPROACHOur approach to designing Tangible Mathsoftware is to extend tried-and-true math puzzlesand activities – many recommended by theNational Council of Teachers of Mathematics(2000) – using both TUI design concepts andgameplay design guidelines. Culminating fromthe experience of building and testing our owneducational software with tangible userinterfaces, we have developed a series ofpedagogical guidelines that are described below.While each of these guidelines may appear to bedistinct, all must be blended together to create acohesive whole.

Physical ActivityPhysical involvement in a learning activity –whether a child is moving objects or his/her ownbody – helps to make the activity moremeaningful for the child. This is partly becausethe physical objects help children to see – andtherefore better understand – abstract conceptsin a new way. The other benefit of physicalinvolvement is that the child must use more thanhis or her brain in the activity. It is difficult todoze off when one is jumping around on anumber line.

The trick is discovering the best physicalrepresentation of the abstract concepts beingtaught, so that manipulation and computer-supported feedback are leveraged to enhancelearning. Abstract concepts must be mapped tophysical entities and environments in a way thathelps the child to make the connection betweenthe concrete and the abstract. For example,quantity may be translated from "area of apolygon" to "number of triangles that fit in thisshape". One could argue that the kind ofunderstanding that evolves from the physicalnotion of moving tiles around may be differentfrom the mental image produced by looking at apolygon.

Collaborative LearningGroup work increases the learning potential ofan activity by allowing children to scaffold offof each other's prior knowledge. This has beenshown to be especially effective for girls (Inkpenet al 1995). Talking about the activity also helpschildren to develop meta-cognitive connectionsthat are useful in transference of the knowledge.

Yet if one expects children to collaborate,one must provide a learning environment thatgives all of the children equal access to the dataand equal opportunities to manipulate that data.We, for example, have developed a TICLE(tangible interfaces for collaborative learningenvironments) table that we use in most of ourtangible math games (Scarlatos 2002). This table(shown in figure 1) allows several children towork with a puzzle or math manipulative at thesame time. A camera beneath the table (lookingup) eliminates the problem of visual obscuration,while a touch screen gives all of the childrenequal access to the software's extra help.

Figure 1. Children using a TICLE table have equalaccess to the tangible data (puzzle pieces) and

further information (from the touch screen).

Practice That's FunPractice helps to reinforce concepts that havebeen learned so that they are more deeplyengrained in the learner's mind. This is thepurpose of doing homework, though mostchildren find typical homework assignments tobe boring. Although new concepts are besttaught by teachers, computer software is wellsuited to guiding students as they practiceapplying those concepts.

The best way to make practice fun is tomake a game of it. Increasing difficulty, coupledwith increasing rewards, helps to keep theactivity interesting. In our observations, we havenoticed again and again that when children feelthat they have worked hard and then "won", theyare eager to try the activity again, even if it ispracticing math concepts. And how manychildren finish their homework and then think, "Iwon!"?

Immediate FeedbackOne of the key advantages of having a computer"watch" what the children are doing is that thecomputer can provide immediate feedback. Oursoftware typically provides three types offeedback.

First, when the children make progresstoward a goal, encouragement is given: eitherverbal, visual, and/or a numeric score. Whendesigning the form of the encouragement, anunderstanding of the system of rewards providedby commercial video games, as described byOxland (2004), is essential. Our observations(Scarlatos et al 2002) have revealed that toomuch encouragement is seen as condescending,and can even become annoying.

Alternatively, if the rules of the game arenot followed (e.g. the child does not use thecorrect pieces), the software must providefeedback that gets the child back on track. Theneed for this type of feedback is unique to TUIbecause, unlike a typical computer or videogame, the data representations in a TUI are notunder the control of the computer. We generallyaddress this with a gentle reminder about whatthe goal is, and what constraints go along withthat goal. Again, it is important not to beadamant about this; repeated reminders orcorrections can become distracting anddiscouraging.

Finally, when the students are "stuck", thesoftware must be able to give them hints.Students should have to ask for hints explicitly.Again, we have observed that most children willinitially want to try to solve a problem withoutany hints. It is only when they have failed tomake progress over a period of time (typicallytwo to five minutes) that they will ask for help.The hints must be contextually relevant, i.e.

selected based on the state of the game or whatthe students have already done. At the sametime, the hints must be designed to get childrento think about the problem at hand; tellingchildren to "put this there" doesn't help them tosolve a similar problem later. This ability totransfer knowledge, and solve problems withoutthe help of a computer later on, is an essentialskill that all math games should strive todevelop.

Information GatheringOne of the greatest advantages of using acomputer for learning activities is that thecomputer can record what the student has done.The teacher may then review this informationlater, to determine where the student (or theclass) is having trouble and deduce what theyneed to receive additional instruction on.

All of our tangible math games recordstudent performance data for the teacher to lookat later on. Storing this data in a flat file (with atime/date stamp) is simplest to implement andallows for anonymous activity. For moredetailed analysis, it is preferable to store theinformation in a database. This allows theteacher to track a particular student's progressover time, look for classroom trends, or getdifferent "views" of different student groups.

Teacher CustomizationDespite a developer's best efforts, a computergame can never meet all of a teacher's needs fora particular subject and class combination. Evenwithin a single class, a teacher may feel the needto provide slightly different learning activitiesfor different groups of children.

For this reason, most of our tangible mathapplications rely on an input text file. This filedefines a wide variety of parameters, from whatthe problem or puzzle is, to what tangibleentities are being tracked, to what hints shouldbe given in what circumstances. We then can(and have) develop(ed) a separate teacher'sinterface which is used to define theseparameters.

TECHNICAL APPROACHWhen developing math games, we take aniterative development approach using a

multimedia authoring tool. For most of ourapplications, the tool we have chosen to use isMacromedia Director. In addition to being anexcellent rapid prototyping tool, Director alsoprovides a powerful programming languagecalled Lingo. When Lingo does not satisfy all ofa developer's needs, Director's capabilities areeasily extended by integrating code librariesknown as Xtras.

The remainder of this section describes ourown Anna Xtra, which we use to track visualtags in a collaborative learning game. Wecreated this Xtra because existing libraries didnot adequately support the tracking of visualtags in games that use tangible user interfaces.The functions supported by this library, and theirunderlying algorithms, were first described byMbogho and Scarlatos (2005). Here, we describehow Anna is used to build tangible math games,supplementing the descriptions with codesamples. This Xtra is available as freeware ono u r w e b s i t e :http://www.sci.brooklyn.cuny.edu/~lori/TICLE/.

Vision-Based TaggingIn our early work with tangible user interfaces,we experimented with computer vision using athird-party Xtra. Although we had good resultsfrom the Tangram application that we produced(Scarlatos 2002), calibration was difficult. Thisproblem arose because the functions werelooking for RGB colors with given tolerances orallowable variance. Although small changes inlight are nearly imperceptible to human eyes,they create changes in the RGB values that makethe color look very different to a computer. Infact, some colors that are close to one another inRGB space can look much more different to ahuman (e.g. a color with red added to it) thancolors that are far apart (e.g. a color that islightened by adding equal parts of red, green,and blue).

In response to this problem, we developedour own computer vision Xtra which we callAnna. Anna differs from earlier computer visionXtras in that it looks for matching colors in twodifferent color spaces: YIQ and HSV. YIQcorresponds to the color system used in video,and allows us to specify different tolerances interms of luminance (Y) and chrominance (IQ).

Consider, for example, an environment wherelight conditions can change drastically, such asoutdoors or near a window. When looking for aspecific color, we would want the chrominance(IQ) to be a close match, but could allow muchgreater variance in the luminance (Y) values.

HSV (hue, saturation, value) correspondsmore closely to human vision. In ourexperiments, we have discovered that lookingfor a match in HSV space produces the mostreliable results when tracking visual tags. Thereason for this is that, to make the tags stand out,we have chosen to use highly saturated colors.Using HSV we can narrowly restrict the range ofhues (H) being sought, set a minimum saturation(S) threshold, and allow for greater variance invalue (V or luminance). As a result, our tagtracking is much less sensitive to changes inlight conditions in the environment.

An application using Anna is initialized bycreating a tag reader object, which is used in allsubsequent calls to Anna functions. Figure 2provides the Lingo code for creating anddestroying this object.1

-- Initialize Annaglobal TRobj -- Anna object

on startMovieTRobj = new(xtra "Anna")

end startMovie

-- Clean up at the endon stopMovieTRobj = 0

end stopMovie

Figure 2. Lingo code to start up Anna Xtra.

Tagging StrategiesAs we continued to build math games using

computer vision, we identified the followingvisual tagging strategies as effective for anumber of 2D puzzles and manipulatives.Objects with these tags may be tracked on aTICLE table (with the tags on the underside).They may also be viewed by a camera fromabove (with the tags facing up) or from the side(with the pieces stacked vertically).

1 In Lingo, "--" marks the beginning of a comment. The endof a line marks the end of a statement; statements that needmore than one line use the continuation character "\".

Space-Filling TagsSome math manipulatives and puzzle pieces arecreated from repetitions of a single elementalshape. For example, each Cuisenaire rod is alinear arrangement of N cubes, where Nrepresents the length, area, and numeric value ofthe rod. Pentomino pieces are arrangements offive squares, just as Tetris pieces arearrangements of three squares. Pattern blocks are(mostly) shapes made up of one or moreequilateral triangles: hexagon, trapezoid,rhombus and triangle. All of these manipulativescan be used to teach children about measuringarea and creating shapes. Thus in games usingthese types of puzzle pieces, it is most importantto track how the pieces fill the space.

Figure 3. Space-filling tags mark the center of eachelemental shape, using color to identify the larger

piece using that shape.

Space-Filling Tags are created by placing acolored spot in the exact center of eachelemental shape. For example, a hexagon wouldbe tagged with six spots; each pentomino piecewould have five spots. The colors of the spotsidentify the overall shape of the piece. Theresulting pattern of spots seen by the camera, asshown in Figure 3, can then be used to orientand fill a triangular or square grid.

Figure 4 shows the Lingo code forinterpreting these tags and using thatinformation to fill in a grid space, which isrepresented in the computer's memory. Thiscode assumes that functions (handlers) havebeen specified to check the size of the boundingrectangles (to make sure they are really tags) and

to calculate a grid index based on a tag's positionin the image.

-- Given an array of colors (CList)-- and a list of color labels (CLabel)-- fill in a Grid with those labels

global TRobj -- Anna object

on fillGrid CList, CLabel, Grid

-- initialize the Grid = all emptyrepeat with i = 1 to Grid.countGrid[i] = #empty

end repeat

-- Capture image of the game stategetImage(TRobj)

-- Loop through all of the tag colorsrepeat with i = 1 to CList.count

-- find rectangles bounding-- all tags with this colorcp = CList[i] -- color parametersbounds = hsv_getBoundingRecs(TRobj,\

cp[#color], cp[#hueRange],\cp[#satLimit], cp[#valRange])

-- get label to fill the grid withthisLabel = CLabel[i]

-- loop through all rectangles, &-- label grid elementsrepeat with b in bounds

-- ensure spot is the right sizeif sizeOK(b) then

-- get the Grid index (gi)-- for this locationgi = getIndex(b)

-- label the grid elementGrid[gi] = thisLabel

end ifend repeat

end repeat

end fillGrid

Figure 4. Detecting and interpreting space-filling tagswith Anna.

Orientation TagsFor some puzzle and game pieces, such asTangrams and Scramble Squares, it is importantto know not only what piece is visible andwhere, but also how it is turned (oriented)relative to the other pieces. This information canbe used to determine a relative configuration or

state of the puzzle that is independent of rotationand translation factors. The game can then usethe state information to determine how close theplayers are to reaching a solution. The game canalso use this information to select an appropriatehint when asked.

Orientation Tags are composed of threeclosely spaced collinear spots of color, similar tothe strategy used by Underkoffler and Ishii(1998). One color (such as yellow) is reservedfor marking the center of mass for the piece. Thecolors of the two adjacent spots uniquelyidentify the piece. So, using three colors (inaddition to the color reserved for the centerspot), one can uniquely identify nine pieces. Thethree spots form a vector, with its tail at thecenter, which indicates the orientation of thepiece. Figure 5 shows a collection of Tangrampieces with orientation tags.

Figure 5. Orientation tags identify the puzzle pieceswhile also showing their location and orientation.

Figure 6 lists the Lingo code for finding aset of tags. Here, it is assumed that a Tag class(parent script) has been defined that includesproperties representing an identifier (indicatingthe type of piece it is), an absolute location, andan orientation vector. Methods are defined toadd spots to the tag pattern (addSpot2 andaddSpot3) and then update the properties of thetag (update) based on these. If the spots in a tagobject are not collinear, or the pattern of colorsis not recognized in this game, then the updatemethod returns a value indicating that the result

is not valid. The code in figure 6 also assumesthat a function (getClosestTag) has been definedto determine which Tag is closest to a givenbounding rectangle. Because this function usespartitioning techniques to increase efficiency, avalue of VOID is returned if no tag center iswithin a reasonable distance of the givenrectangle. The function inRange determineswhether a tag is reasonably within the givendistance of a given bounding rectangle.

Using the Tag data generated by thisfunction, a game can then match these tags toknown pieces of the puzzle, to determine wherethose pieces are. Related to an internalrepresentation of the piece's geometry, thisinformation can then be used to determine howtwo pieces are touching. This, in turn, can beused to determine the current state of the puzzleor game (Scarlatos 2002).

-- Given a list of colors (CList, where-- CList[1] = the color of the center)-- and a target distance between spots-- (dist, measured center-to-center),-- return a list of Tags, each one-- having an identity, location-- and orientation vector


on findTags CList, CLabel, dist

-- initialize the Tag listTag = []nTags = 0 -- number of tags found


-- Find all the center spotscp = CList[1] -- color parametersbounds = hsv_getBoundingRecs(TRobj,\


-- Create new tag for each spot found-- using bounds to indicate positionrepeat with b in boundsif sizeOK(b) thennTags = nTags + 1Tag[nTags] = new(script "Tag", b)

end ifend repeat

-- Loop through remaining tag colorsrepeat with i = 2 to CList.count

-- get color label for tag infothisLabel = CLabel[i]

-- find rectangles bounding-- all spots of this colorcp = CList[i] -- color parametersbounds = hsv_getBoundingRecs(TRobj,\


-- loop through all rectanglesrepeat with b in bounds


-- find the closest tagnearest = getClosestTag(b, Tag)

-- if something was returned,-- see if this is part of a tagif (nearest <> VOID) thenif inRange(b, nearest, dist) thennearest.addSpot2(thisLabel, b)

else if inRange(b, nearest, \dist*2) then

nearest.addSpot3(thisLabel, b)end if

end ifend if -- sizeOK

end repeat -- every rectangleend repeat -- every color

-- Update the tags, eliminating any-- that are invalidrepeat with i = nTags down to 1

-- update tag, returning statusOKstatus = Tag[i].update()

-- delete any tags that are not OKif NOT OKstatus thendelete(Tag, i)

end ifend repeat

-- Return resulting list of tagsreturn Tag

end findTags

Figure 6. Detecting orientation tags with Anna.

Attribute TagsBoth mathematicians and scientists useclassification and data analysis techniques tocategorize things. Classification and dataanalysis are important components of the mathand science standards (Van Der Wallen 2000),and are tested on state standardized tests. Togain experience with this activity, attributematerials are typically used. These may bestructured sets – with exactly one piece for every

possible combination of values – or unstructuredsets – made up of sample objects collected bythe teacher or children. The attributesthemselves may be either binary or multi-valued.

Binary attributes either have a givenproperty, or they don't. For example, a rock maybe either shiny or not shiny. Binary attributes areespecially useful for a classification activity inthe sciences known as dichotomous sorting.Binary Attribute Tags employ a solid-coloredspot to represent the presence of an attribute, anda spot with a hole in it to represent the absenceof that attribute. The color of the spot indicateswhich attribute it is. Figure 7 shows the under-sides of an unstructured attribute set with fivedifferent attributes.

Figure 7. Binary attributes are either there (solid) ornot there (hole).

Figure 8 shows Lingo code that determines,for a given collection of objects, what attributesthose objects do and do not have. Although thiscode does not consider the locations of theattributes, the locations may be readily accessed.This could then be used to imply an ordering ofa set or membership in a Venn diagram.

-- Given an array of colors (CList) and-- % of size determining solidity (f)-- return a list of attributes (Alist)-- indicating how many positive (#yes)-- and negative (#no) values are found


on getAttributes CList, factor

-- initialize the AlistAlist = []

-- Capture image of the game state-- & save a copy in cast member 1getImage(TRobj)showImage(TRobj, 1)




-- loop through all rectangles,-- counting attributesyesAtt = 0noAtt = 0repeat with b in bounds


-- count # of pixels with colorcount = hsv_countPixels(TRobj, 1,\

cp[#color], b, cp[#hueRange],\cp[#satLimit], cp[#valRange])

-- update count accordinglyif (count >= (sizeOf(b) * f)) thenyesAtt = yesAtt + 1

elsenoAtt = noAtt + 1

end ifend if

end repeat

-- Add a record to the AlistaRecord = [#yes : yesAtt, #no :

noAtt]Alist[i] = aRecord

end repeat

-- Return the resultsreturn Alist

end getAttributes

Figure 8. Detecting and interpreting binary attributetags with Anna.

Multi-valued attributes appear in activitiesthat focus on sets, Venn diagrams, andpartitionings. Multi-Valued Attribute Tags canbe created from a bi-directional tri-color barcode(Mbogho and Scarlatos 2005). Two of the colorsin the barcode represent binary values (0/1).Bars in these colors represent a 4-bit number(that is then repeated backwards) and mark thestart (and end) of the code. By repeating the

code in this manner, the orientation of thebarcode becomes unimportant. The third(background) color in the barcode represents anattribute category. Thus an image containingmany different barcodes, representing severaldifferent attributes, can be interpreted withoutregard for location, orientation, or scale. Figure9 shows a rock with three different attributeswith values 0 (red), 3 (yellow), and 1 (blue).

Figure 9. Bi-directional barcodes represent differentattribute values, indicated by the background color.

Figure 10 shows the Lingo code for readingthese barcodes. This function returns, for eachattribute, a list of [value, position] pairs toindicate what attribute values are found where.This may then be used to check set orderings orlocation in a Venn diagram.

-- Given an array of attribute-- names (Nlist) and colors (CList),-- return a list of attributes (Alist)-- that specifies the value & location-- of each tag found for that attribute


on getBarCodes CList, NList

-- initialize the AlistAlist = []



-- get attribute namethisName = NList[i]



-- loop through all rectangles,-- reading barcode valuesvalues = []repeat with b in bounds

-- ensure barcode is the right sizeif sizeOK(b) then

-- read the value of the barcodenum = hsv_readTag(TRobj, \

cp[#color], b, cp[#hueRange],\cp[#satLimit], cp[#valRange])

-- add it to the listadd (values, [num, centerOf(b)])

end ifend repeat

-- Add a record to the AlistaddProp(Alist, thisName, values)

end repeat

-- Return the resultsreturn Alist

end getBarCodes

Figure 10. Detecting and interpreting multi-valuedattribute tags (bi-directional bar codes) with Anna.

GAMESNearly all of the games that we have developeduse wooden pieces marked with visual tags.Early on, we discovered that children tend to bevery rough with objects that they are handling,especially in museum settings. The woodenpieces can (and have) been stacked, knockedover, and dropped on the floor; the most that wehave to do is occasionally touch up the tags.

TangramsThe tangram is a classic puzzle used to teachgeometry concepts, such as comparing area andcreating shapes sets (Van de Walle 2000).Tangrams also help to develop problem-solvingskills. Invented centuries ago in China, tangrampuzzles have appeared in books, games, andstate proficiency exams.

Figure 11. Tangram puzzle with tangible userinterface.

We created software that uses orientationtags to track the tangram pieces on a TICLEtable (figure 11). Children select a Tangrampuzzle to solve, which determines the solutionbeing sought and the set of hints that mayappear, depending on the state of the puzzlewhen they ask for help. While the children areworking on the puzzle, the softwarecontinuously polls the computer vision system todetermine whether or not the children aremaking progress. Encouragement is offeredperiodically when they do make progress.Context-sensitive hints, as shown in figure 12,ask the children to solve sub-problems that aredesigned to help them "see" what they need todo next with the larger puzzle. The softwarerecords a history of moves made by the children,along with a history of the hints seen.

Figure 12. Hints are designed to help students to"see" how they might fill in a space in the puzzle.

Pattern BlocksPatten blocks are simple polygonal shapes thatinclude a hexagon, a quadrilateral (half of thehexagon), a rhombus (one third of the hexagon),and an equilateral triangle (one half of therhombus). These blocks can be used in a widevariety of ways to teach several differentconcepts. For example, because the smallerblocks can be combined to create larger blocks,these can be used to teach fractions,multiplication, and division. Pattern blocks canalso be used to teach tessellation concepts.

Figure 13. Problem posed by Pattern Block software.

We developed software that helps childrento understand the concept of area by asking themto cover a pattern using specific pattern blocks.The software only looks at the space-filling tagson these pieces if the question was answeredwrong; if the wrong pieces are being used, thesoftware reminds the children how they canrecognize the pieces they are looking for.

Cuisenaire RodsCuisenaire Rods have been used in classroomsfor decades (Holt 1983) to teach a variety ofnumber concepts. Among these are thedifference between odd and even numbers;addition; multiplication as multiples of a givennumber; and fractions.

We have developed an application thatfocuses on the concept of odd versus evennumbers. Children are given a sequence ofincreasingly difficult exercises that reinforce theconcepts. Children may also choose to solvepuzzles, in which they are asked to fill in a space

using a particular type of Cuisenaire rod. Figure14 shows one of these puzzles. In all of theseactivities, the computer checks to ensure that thechildren are using the right type of rods (i.e.odd- or even-numbered length).

Figure 14. Puzzle using Cuisenaire rods reinforceslessons on odd versus even numbers.

Tower of HanoiThe Tower of Hanoi is a classic puzzle knownand loved by many. For younger children, it is achallenging puzzle that can be madesuccessively harder by adding more and moredisks to the puzzle. For older children, it is anideal tool for demonstrating the concepts ofrecursion and recurrence functions.

Figure 15. Children playing with the Tower of Hanoi.

Our Tower of Hanoi software uses anoverhead camera to track the movement ofbrightly-colored disks on a stand with three pegs(figure 15). In this application, the entire disk isa tag: if it disappears from view for an extendedperiod of time, it is assumed that either the disk

is not in play, or it is underneath a larger disk.Either condition constitutes a violation of therules. While the students work with the tower, adiagram of Serpinski’s triangle shows thesequence of steps as a path through a graph. Inthis diagram, the optimal solution to the problemis represented by the shortest path from onecorner of the triangle to another. The diagramtherefore helps children to "see" when they havemade an extraneous move.

Attribute SortingClassification (the process of making decisionsabout how to categorize things) is an importantskill that is employed both in math and inscience. Children typically learn to classifyobjects by working with attribute sets (Van deWalle 2000).

We have developed a program thatencourages children to analyze and categorize anunstructured attribute set assembled by theteacher. As shown in figure 16, the advantageof using real objects in a set is that children canmake observations that would not be possiblewith virtual objects ... such as testing to seewhich rocks are harder than a nail. In addition tosaving a list of hints seen and the state of thesystem at key stages, the software collects andstores the students' own reflections about whatthe objects on the table have in common.

Figure 16. Children testing the hardness of a rock forthe attribute sorting program.

A separate teachers' program allows teachersto define the attribute sets they are using, and thehints that they wish to offer to the students.Hints can be anything that can be displayed in aweb page, making this accessible to both

technical novices and technically savvy teachers.We have had teachers use this interface to definethe attributes of rocks, leaves, flying insects, andbuttons.

FUTURE TRENDSThis work is continuing in three ways: throughdevelopment, testing, and extension.

Development includes the creation ofadditional software activities with tangible userinterfaces. In the area of attribute sorting, weplan to create math games that use VennDiagrams, "Which One Doesn't Belong", andordered sets. We have also been working on anapplication that uses pattern blocks to learnabout and create tessellations. In addition, weplan to develop teachers' interfaces for more ofour games.

Testing involves bringing our software tothe classroom and having teachers evaluate thepracticality and benefits of using the games.Although we have been able to observe oursoftware being used in museum settings,classroom use will allow us to learn more aboutthe long-term benefits of using math games withtangible user interfaces.

Extension involves moving from two-dimensional puzzles to the third dimension. Weare currently working with motion capture andother multimodal inputs to track student activitywithin a CAVE environment. This extensionwill encourage even more activity on thestudents' part, as well as increase the realm ofpossible learning activities that we can track.

CONCLUSIONWe have presented both a pedagogical approachand software tools for developing educationalsoftware with tangible user interfaces. We havealso described a series of software games thatwe developed using the presented approach andtools. Although the focus of our work has beenon mathematics, the tagging strategies andpedagogy are clearly applicable to science, andprobably other topics as well.

ACKNOWLEDGEMENTSThe author wishes to acknowledge the manytalented students who have worked on thisproject. Audrey Mbogho wrote the Anna Xtra;

Magdalena Jaworska, Shalva Landy, and IreneLam worked on the games. Dave Ferguson,Eleanor Miele, Rosamond Welchman, FrederickGardener, and Tony Scarlatos provided valuableinsights and suggestions throughout this project.The participation and support of Beth Deaner,director of the Goudreau Museum ofMathematics in Art and Science, and PreetiGupta and Frank Signorelli, from the New YorkHall of Science, have been indispensible.

This work has been supported by theNational Science Foundation under grantnumbers 9984385 and 0203333.

REFERENCESBransford, J.D. et al, editors (2000) How People

Learn, National Academy Press.Bush, G.W. (2006) State of the Union: American

Competitiveness Initiative. Summarya v a i l a b l e o n l i n e a thttp://www.whitehouse.gov/news/releases/2006/01/20060131-5.html (February 2006).

Calvill-Gamez, E.H., Leland, N., Shaer, O. andJacob, R.J.K. (2003) The TAC Paradigm:Unified Conceptual Framework to RepresentTangible User Interfaces, Proceedings of theLatin American conference on Human-computer interaction CLIHC '03, pp. 9-15.

Druin, A., Bederson, B., Boltman, A., Miura, A.,Knotts-Callahan, D., Platt, M. (1999).Children As Our Technology DesignPartners. In Druin, A. (ed.), The Design ofChildren's Technology, Morgan KaufmanPublishers, Inc., San Francisco, CA, pp. 52-72.

Fishkin, K.P. (2004) A taxonomy for andanlaysis of tangible interfaces, Personal andUbiquitous Computing, 8(5), pp. 347-358(Sept. 2004).

Gardner, H. (1983) Frames of Mind: The Theoryof Multiple Intelligences, Basic Books.

Holt, J. (1983) How Children Learn , Da CapoPress.

Horn, M.S. and Jacob, R.J.K. (2006) Tangibleprogramming in the classroom: a practicalapproach, CHI '06 extended abstracts onHuman factors in computing systems CHI'06, pp. 869-874 (April 2006).

Inkpen, K., Booth, K. S., Klawe, M. & Upitis, R.(1995). Playing Together Beats Playing

Apart, Especially for Girls. Proceedings ofComputer Support for CollaborativeLearning '95 (CSCL).

Ishii, H. and Ullmer, B. (1997) Tangible Bits:Towards Seamless Interfaces BetweenPeople, Bits and Atoms, Proceedings of theSIGCHI conference on Human factors incomputing systems, Atlanta, GA, pp. 234-241.

Marshall, P., Price, S. and Rogers, Y. (2003)Conceptualising tangibles to supportlearning, Proceeding of the 2003 conferenceon Interaction design and children, Preston,England, pp. 101-109.

Mbogho, A. and Scarlatos, L. (2005). TowardsReliable Computer Vision-Based TangibleUser Interfaces. Proceedings of IASTED-HCI 2005 (Human Computer Interaction),Phoenix, AZ, November 2005.

McNerney, T.S. (2004) From Turtles toTangible Programming Bricks: Explorationsin Physical Language Design, PersonalUbiquitous Computing, vol. 8, pp. 326-337.

Moher, T., Watson, B., Kim, J., Hindo, C.,Gomez, L., Fransen, S. and McEneany, T.(2005) StoryGrid: a tangible interface forstudent expression, CHI '05 ExtendedAbstracts on Human factors in computingsystems, Portland, OR, pp. 1669-1672 (April2005).

National Council of Teachers of Mathematics(2000) Principles and Standards for SchoolMathematics, National Council of Teachersof Mathematics.

Oxland, K. (2004) Gameplay and Design,Addison Wesley.

Scarlatos, L.L. (2002). An Application ofTangible Interfaces in CollaborativeLearning Environments, SIGGRAPH 2002

Conference Abstracts and Applications,125-126.

Scarlatos, L.L., Landy, S.S., Breban, J.,Horowitz, R. and Sandberg, C. (2002). Onthe Effectiveness of Tangible Interfaces inCollaborative Learning Environments,CUNY Graduate Center Technical ReportTR-200204.

Stanton, D., Bayon, V., Neale, H., Ghali, A.,Benford, S., Cobb, S., Ingram, R., O'Mally,C., Wilson, J. and Pridmore, T. (2001)Classroom collaboration in the design oftangible interfaces for storytelling,Proceedings of the SIGCHI conference onHuman factors in computing systems ,Seattle, Washington.

Underkoffler, J, Ishii, H. Illuminating Light: AnOptical Design Tool with a Luminous-Tangible Interface, Proceedings of CHI ’98,542-549.

Van de Walle, J.A. (2001) Elementary andMiddle School Mathematics: TeachingDevelopmentally, 4th Edition, AddisonWesley Longman.

Zhou, Z.Y., Cheok, A.D., Pan, J.H. and Li, Y.(2004) An interactive 3D explorationnarrative interface for storytelling,Proceeding of the 2004 conference onInteraction design and children: building acommunity, Maryland, pp. 155-156 (June2004).

Zuckerman, O., Arida, S. and Resnick, M.(2005) Extending tangible interfaces foreducation: digital montessori-inspiredmanipulatives, Proceedings of the SIGCHIconference on Human factors in computingsystems, Portland, OR, pp. 859 - 868 (April2005).

Documents

Tangible Math - cs.stonybrook.edu