Does Tangible Music Interfaces helps children learn?

Marco Vieira University of Madeira

Centro de Ciências Exatas e da Engenharia Campus da Penteada, 9000-390 Funchal

marcofilipevieira@gmail.com

ABSTRACT The heart of tangible user interface (TUI) is the different ways which human computer interaction can be enhanced by just using everyday physical objects. In other words, each physical object corresponds to a particular piece of digital information or part of a virtual model. Therefore, users can use these objects to manipulate digital information. There are several advantages for using TUI’s, such as: the immediate tactile feedback, this means that users receive feedback from the physical objects as they grasp and manipulate them; also the space-multiplexed input, i.e., each tangible representation serves as dedicated controller occupying its own space, and encourages two-handed and multi-user simultaneous interaction, and thus TUI is perfect for collaboration allowing concurrent manipulation of information by multiple users. This manipulation of physical objects can be a particular educational benefit; in this paper, it will be discussed why this technology is good for learning in a specific domain (music). Lastly, it will be presented a TUI suggestion to enhance music learning through this novel technology.

Keywords Tangible User Interfaces, music, children musical education, haptic, learning.

ACM Classification Keywords H5.5. Information interfaces and presentation (e.g., HCI): Sound and Music Computing. K.3.m Computers and education: Miscellaneous

General Terms Performance, Theory, Verification.

INTRODUCTION The idea of tangible user interfaces (TUI) is relatively novel and it was described as a new paradigm of interaction. Tangible interfaces are defined as “the gap

between the worlds of bits and atoms” [4], permitting humans to use digital information with physical objects [4].

Nowadays tangible user interfaces are being well known by all public in general. Moreover, these tangible interfaces are good for so many reasons such as, interaction environment, which makes people get more involved with it and at same time, they have fun with this kind of technology.

One of the several areas that this technology can be very helpful is music. There are many tangible music interfaces built in these days for so many purposes. The most known purpose is live concerts (mostly used by DJs), because with these technologies is possible to achieve some neat music effects that people like to hear.

Another good thing about these tangible music interfaces is that they can help children learn better some musical concepts [3], as they get to see the results by using their bodies through movements that actually helps them understand those theoretical concepts. Therefore, they can really comprehend musical concepts through some practice about those concepts. It’s easier for an infant to learn something through practical exercises rather through theoretical ones.

It is believed that physical action is important in learning and thinking as we can conclude from some empirical works, namely from Piaget and Bruner work, which shows that children can frequently solve problems when given concrete materials to work with before they can solve it symbolically [11]. Other belief is that sometimes children can’t explain verbally some knowledge, which they have previously acquired without the help of gestures [11]. Furthermore, they found that children are able to solve mathematical problems (e.g. fractions) moving physical materials [11]. These kinds of interfaces are a great approach to the enhancement of learning multiple subjects at school, or in a general way, helping develop some skills since we incline to think that physical interaction through haptic senses is straightforward and natural, consequently it makes easier the comprehension of how things work and what they mean [4].

Through expressive activity, the learner is able to demonstrate his understanding of a certain domain of

study, highlighting misconceptions or gaps in their knowledge [2].

There are several domains for TUI’s, for example: chemistry or molecular biology [13], narrative or rhetoric [14], and music [1,10]. In this paper, it is argued the benefits of using TUI’s for learning purposes in a special domain, which is music. This work tries to prove that TUI’s are capable of improving music learning with children.

Finally, it will be suggested a tangible interface with the intention of improving music learning based on all advantages of other related works. This proposal is only an idea, it has to be developed and done some empirical work to prove that this suggestion reaches its goals.

LEARNING WITH TANGIBLE INTERFACES Tangible interfaces have a big number of diversity of application domains. One of them is learning (which is the objective of this paper), which has received a lot of interest due to the general view within education that manipulation of physical manipulatives can be an educational advantage [1].

In one of Paul Marshall’s papers [1], he claims that there is not enough empirical work on this thematic. Therefore, he calls for a greater focus on empirical work in this area, and he proposes a framework of six perspectives on learning with tangible interfaces. Those perspectives are: learning benefits of this kind of interaction; learning domains, which explains the types of learning tasks usually supported by tangible interfaces; exploratory and expressive activity which is a category of learning activity; integration of representations that already exists; concreteness and sensory directness which says that the physical objects should be treated separately when thinking about tangible interfaces; and lastly effects of physicality which concerns about the effects of learning with physical objects [1].

Learning Advantages with TUI’s Paul Marshall refers that “(…) if perception and cognition are closely interlinked, then using physical materials in a learning task might change the nature of the knowledge gained through interacting with virtual materials” [1]. With this, it’s possible to say, that when you learn something or think about it, you have a mental model in you head of that knowledge. Thus, if we have a physical object that represents that mental model of the knowledge in our heads, then we can conclude that manipulating these objects, the learning process becomes more intuitive, which is the most important benefit of these kinds of interfaces [2]. Furthermore, there is the Piagetian developmental theory about the manipulation of physical objects, which supports the idea that this manipulation helps kids developing their thoughts on something [1]. Therefore, it means that with these interfaces, learning

experience for children turn out to be more natural and effective. It was made some work, which we can observe that tangible interfaces are appropriate for engaging children in playful learning [1], in other words, children can learn through games (with the aim of helping children learn something), historically children have played with physical objects to learn a variety of skills [11]. Tangibles have the potential for providing innovative ways for children to play and learn, through novel forms of interacting and discovering and the capacity to bring the playfulness back into learning [11]. So, we can say that links between physical action and digital effects might increase the commitment and thinking. Additionally, as the TUI’s are more natural (according to what was said before), it is more accessible for novices, younger kids, and people with learning disabilities. This a very good advantage, because anybody can use the system [1].

Additionally, there are several projects, which implies that TUI’s might be appropriate for collaborative learning [1]. This is very good, because when people are able to learn together, this experience of learning have another dimension. In other words, it is different learning alone rather accompanied. When you are learning in a group, you have support from the group whenever you need to, so you really learn. However, when you’re alone, you don’t get that much support and it becomes harder when you struggle with something that you’re having some difficulties. This will make your learning process slower than the usual. As TUI permits you to learn with someone else, your learning experience is different and with higher quality, this is another good reason, why we should use TUI’s.

Expressive and Exploratory Tangible Systems Learners can create an external representation of a certain domain, or an idea, or an understanding about something [1]. This is what expressive activity is. In other words, when people are learning something, always produce a mental representation of that learning which could be about any domain. In addition, tools are often used to enhance learners organize theirs ideas making them more concrete and explicit supporting their learning about the domain [1, 2]. Therefore, when this knowledge is externalized, they can reflect upon the model or representation, and see if it corresponds to reality. Summarizing, by making their individual understanding explicit through external representation, it’s then easier to obverse any possible inconsistencies, conflicting beliefs and incorrect assumptions [2].

In Paul Marshall’s paper [2], he concludes that expressive activity is “(…) where the tangible will embody in some way the learner’s behavior, either physically or digitally (…)”. Concluding, expressive activity is the actions of the user upon the tangible interface, with the intention of manipulating it through physical manipulatives in a way,

that he confirms his mental model or representation about something until all inconsistences all clear.

On the other hand, we have exploratory activity, which consists of an exploring activity upon a model presented by someone else [2]. This means, that the learner discovers the model or a tangible interface made by some designer in order to learn something [2]. There are two types of reflecting learning that we can observe from the exploratory activity: first, the learner can learn how to utilize the tangible itself for a certain task. Therefore, the learner in this phase tries to know what he can do with the system. The second type of reflecting it’s related to the exploration of the model through their interaction with the tangible interface [2]. In other words, the leaner explores an existing model of certain topic, and then he might relate this new information to a personal experience. However, the model can conflict with learner’s existing mental model of comprehension of that topic, leading to a cognitive growth [1].

To resume, in Paul Marshall’s paper [2], he states the exploratory activity as “(…) where the tangible embodies a model provided to the learner by the designer”. This means that the manipulation of physical materials provides learners a new way of gaining extra information about a determined domain. Concluding, tangible systems offer a good environment for learning due to the minimal cognitive effort required understanding how this kind of technology works and more attention could be used onto the underlying domain [1].

RELATED WORK Currently there are a big number of musical TUI’s. Some of them are built for the entertainment purpose, but others are built with learning objectives. Some of those TUI’s constructed for entertainment, can also help learning, because sometimes, children really learn something with toys and those kind of stuff.

One of the musical TUI’s, which enhances learning, is D-Touch, this TUI is comprised by a marker software detector, camera, and of course, the markers [7,8]. The system consists of a sheet of paper with a printed grid layout called the active area. Users place the markers on that active area which is monitored by the camera, and then, there is a software for tracking the markers, and produces a previous action programmed according to the marker type and its position. It was developed three musical applications: Augmented Musical Stave, Tangible Drum Machine, Physical Sequencer [7]. The Augmented Musical Stave is the one with more importance for learning, since it teaches children score notation [7, 8].

Another musical TUI that helps learning, is BodyBeats, which their goal is to help children engage their whole bodies while interacting with computers and learning musical patterns [9].

In addition, there is Vusik, which is a TUI with the aim for children to explore music and sound, while offering the capability for more advanced music composition [6]. Vusik’s main purpose is the ability for their users to better understand music construction, making it more accessible for children or novices and thus, leveraging the creative approaches for all [5].

One more musical TUI is the Tangible Notes [10], which was designed for children or novices to learn and discover musical notes and scales. With this tangible interface, it is possible to overcome the problem of learning how the pitch sounds like from the scores on papers. In this project, there is playing board, and physical manipulatives, which represents the musical notes. This physical manipulatives have different colors that correspond to the duration of the pitch. Therefore, children place the musical notes (“tangible notes”) on the playing table in order to hear the matching sound of that pitch [10]. This system empowers the intuitive learning environment for music discovery, furthermore it inspect the feasibility of tangible interactions in music field.

Lastly, Alissa Antle et. al wanted to prove that embodied metaphors help children learn [3]. So they did an empirical work trying to demonstrate if embodied interaction with children can help them learning abstract concepts related to music. Their project is called “The Sound Maker” and they used a camera in order to track children’s movements in a rectilinear space [3]. According to the children’s movements in the space, it is played a sound and its musical parameters (tempo, volume, pitch) are changed accordingly. Thus, for tempo controlling, children have move around the space, like running, they just have to move form one place to another. Secondly, for volume controlling they have to do quick movements, for instance waving or stomping feet. Finally, for the pitch controlling, it’s calculated by the proximity of each other, then if they are close together the pitch sound will be higher otherwise, it will be lower. After this, they carried out the empirical study with children. They asked children to do it in pairs some tasks, and in the end, they had to explain what they did [3]. With their study, they concluded that children were able to really learn these musical concepts through this tangible interface. In addition, children’s physical demonstration of understanding aided their verbal abilities to explain the concepts.

FUTURE DIRECTION/ SUGGESTIONS Summarizing all work reviewed in this paper, there are already some good musical TUI’s which improves children learning experience. Nevertheless, there is still a lot of work to do, it is necessary to build more and more interfaces of this kind in order to continue enhancing musical learning with children through TUI’s.

Based on this, and also “D-Touch” and “Tangible Notes”, this paper presents another tangible interface with the aim of teaching children or novice how to interpret musical scores with different clefs. By doing this, children have a new way of learning how to read a certain score. Taking advantage of the TUI’s benefits, such as: immediate feedback from physical manipulatives; manipulation of the physical objects actually helps with children thoughts or understanding [1].

The project that this paper presents is entitled by “The Haptic Stave”, and it is comprised by a table, a camera, several markers, and marker tracking software (ReaTIVsion). The table is the working area, where there is a big stave drawn and then we have one area to put the clef marker (see figure 1).

Figure 1 - The Haptic Stave Prototype

The previous figure shows the prototype for the working area of “The Haptic Stave”. In the figure, the number one, show the space where the clef marker should be, secondly the number 2, shows the markers which represents the musical notes (e.g. whole note, half note, quarter note, and half note), and lastly we have the markers for the dynamics (number three), which could be piano, forte, etc. To use the system, children just have to grasp the markers onto the stave. Additionally, it will be played a sound according to the position of the marker on the stave. That sound will be the corresponding pitch of the position on the stave that will be in accordance with the clef (with different clefs, the pitches are also different). And for instance, if the chosen musical note is a quarter note, then the duration of that pitch will be only one beat. In “The Tangible Notes” system, they used different colors for the duration, but here it is actually used the real representation of the note, giving a more realistic environment. Furthermore, children also have the possibility of playing with the dynamics, so they just have to place the dynamics markers bellow the stave. For e.g., if they place a “piano marker” bellow the musical note, the sound of that note will be lower until he picks up that marker.

This is possible, by using a camera on the table for the intention of detecting different markers present on the table and identifies them (through a tracking software like

reacTIVion), so it is possible to perform the action for each marker identified.

Finishing, with this TUI, children are able to learn some musical concepts, like dynamics, musical notes duration and clefs. This approach provides a tangible user interface for children or beginners to explore and enhance their music learning experience

For the future, it would be interesting to develop this TUI, do some empirical work, and see what the results are. This system takes advantage of the tangible interface’s properties, which really help children learn as it was said before in this article.

CONCLUSION The major objective of this paper is to prove that tangible user interfaces are good for learning purposes. Throughout this work, this theory is supported by other studies that had already been carried out. Those studies reveal some interesting aspects for this work. First, physical objects enhance learning, because through expressive and exploratory activities can help children make clearer their particular thought or knowledge about something. In consequence, misunderstandings or gaps in their comprehension disappear by confronting it with their experiential understanding of a model. Second, the physical action is important in learning, and was already confirmed by Piaget and Bruner’s work [11]. Third, all the studies shows that the manipulation of physical objects makes richer children’s learning, as they have a immediate feedback from the physical manipulation, they can know and observe what are the differences if they do a certain action, what is easier with physical manipulation rather than with mental models about a domain. It is simpler for the organization of thoughts or mental models of some knowledge.

Moreover, it was said that historically children learned playing with physical objects. In addition, this technology permits children to play, and explore them meanwhile, they’re learning a lot about something. In this case, children can for instance try placing different markers on table, and it will produce a sound according to that marker, and thus they can make their own music, and learn the musical notes or the dynamics concept as it was suggested in “The Haptic Stave”.

This approach permits collaboration between users, and the act of teaching another infant to do same behavior makes obvious that learner’s explicit comprehension. Therefore, children’s learning experience is highly improved when they can learn at the same time with other children. Finally, this kind of interface is simple to use, permitting anyone without computer background knowledge to use it, making it suitable for children.

REFERENCES 1.Marshall, P. (2007). Do tangible interfaces enhance

learning? Proceedings of the 1st international conference on Tangible and embedded interaction - TEI ’07 (p. 163). New York, New York, USA: ACM Press. doi:10.1145/1226969.1227004.

2.Marshall, P., Price, S., & Rogers, Y. (2003). Conceptualizing tangibles to support learning. IDC ’03.

3.Antle, A. N., Droumeva, M., & Corness, G. (2008). Playing with The Sound Maker  : Do Embodied Metaphors Help Children Learn  ? 7th international conference on Interaction design and children (pp. 178-185).

4.Ishii, H., & Ullmer, B. (1997). Tangible Bits  : Towards Seamless Interfaces between People , Bits and Atoms. CHI ’97 Proceedings of the SIGCHI conference on Human factors in computing systems.

5.Pon, A., Ichino, J., Sharlin, E., & Eagle, D. (2011). Vuzik  : Music Creation and Comprehension through Painting.

6.Composition, G. M., Creation, M., Education, M., & Interaction, S. (2011). Vuzik  : Music Visualization and Creation on an Interactive Surface. Computer.

7.Costanza, E., Shelley, S. B., & Robinson, J. (2003). INTRODUCING AUDIO D-TOUCH  : A TANGIBLE USER INTERFACE FOR MUSIC COMPOSITION AND PERFORMANCE. 6th Int. Conference on Digital Audio Effects (pp. 1-5).

8.Costanza, E., Shelley, S. B., & Robinson, J. (2003). D-TOUCH  : A CONSUMER-GRADE TANGIBLE INTERFACE MODULE AND MUSICAL APPLICATIONS.

9.Zigelbaum, J. (2006). BodyBeats  : Whole-Body , Musical Interfaces for Children. CHI ’06 extended abstracts on Human factors in computing systems (pp. 1595-1600).

10.Wang, T.-hsien. (1997). Tangible Notes  : Exploring Virtual Information via Physical Objects. Interfaces.

11.O’Malley, C., & Fraser, D. S. (2004). Literature Review in Learning with Tangible Technologies. Policy (p. 52).

12.Jensenius, A. R. (2007). ACTION – SOUND Developing Methods and Tools to Study. Musicology.

13.Fjeld, M., Hobi, D., Winterthaler, L., Zurich, E. T. H., Zurich, C.-, Voegtli, B., & Juchli, P. (2004). Teaching Electronegativity and Dipole Moment in a TUI. ICALT ’04 (pp. 4-6).

14.Stanton, D., Bayon, V., Neale, H., Ghali, A., Benford, S., Cobb, S., Ingram, R., et al. (n.d.). Classroom Collaboration in the Design of Tangible Interfaces for Storytelling. CHI ’01.