gud

Abstract—Touch is obviously an important channel along with vision and speech for natural Human Robot Interaction. However, as most service robots are generally specialized for their own service, touch-centered shape design and additional costs/computation less related to their own tasks can represent a limit to the application of a touch system on a service robot. This paper originated from the motivation to apply a touch system with lower costs/computation to robots without design modifications. The proposed touch recognition system features hardware that is simply composed of charge-transfer touch sensor arrays, an accelerometer and a temporal decision tree classifier intended for online recognition and computational time reduction. Experiments performed by 12 people shows the practicability of the system. The results showed an average recognition rate of 83% with respect to the 4 touch patterns of hit, beat, rub and push.

I. INTRODUCTION OUCH is one of the most important channels for information exchanges between a human and a robot [1].

The usefulness of touch in understanding human affective behavior has elicited interest in Human-Robot Interaction [2-6]. Physical interaction between humans occurs when one person wants to express a favor through such actions as shaking hands, hugging, and tapping, or when one person tries to harm another by hitting and pinching that person. Robots need to recognize these human intentions in order to determine if they may be beneficial or harmful to the robot.

Several means of touch recognition have been proposed thus far. Most were motivated by the desire to create a robotic system based on touch-centered communication channels [3-4]. These robot systems tend to have soft-cover skin so that touch recognition plays the important role of providing more an affective relationship with a human. Traditional robot systems generally have hard-cover skin due to the functionality and ease of manufacturing of these types of robots. In many cases of service robots, as they are designed with the goal of their own service any functions not

Manuscript received February 29, 2008. This work was performed for the

Intelligent Robotics Development Program, one of the 21st Century Frontier R&D Programs funded by the Ministry of Commerce, Industry and Energy of Korea.

Seong-yong Koo is with Human-Robot Interaction Research Center, KAIST, Daejoen, Korean (e-mail: koosy@robot.kaist.ac.kr).

Jong-gwan Lim is with Human-Robot Interaction Research Center, KAIST, Daejoen, Korean (e-mail: limjg@robot.kaist.ac.kr).

Dong-soo Kwon is with Human-Robot Interaction Research Center, KAIST, Daejoen, Korean (e-mail: kwonds@kaist.ac.kr).

related to this tend to be removed or diminished. Though service robots that have adopted a touch system have increased according to the demand for more natural HRI, shape design according to a priority toward a type of services remains the space limitations when adopting a touch system. In order to avoid such limits, additional design modification is required, which leads to increased costs. It is the first motivation of this paper to propose a touch system that can be applied to a robotic system without design modification and that allows richer interaction with a human and via a natural user interface.

Another problem is rooted in what is known as multirate time series pattern recognition. The strategy in designing a touch system can be mainly represented by touch receptor-based design (a bottom-up approach) and touch-pattern-based design (a top-down approach). The case of Huggable [3] and WENDY [5] are suitable examples. These robots are extremely reciprocal counterparts to each other in terms of design processing. Regardless of the strategy, the final state of touch recognition is the design of a recognizer such as HMM or ANN among others. Conventionally, pattern recognition was executed directly after end point detection, implying that the recognition result is not shown immediately once touch occurs. As each touch pattern differs in terms of the time duration, it can be considered a time delay to users in the event of a relatively long term touch such as “rub”. Figure 1 illustrates this situation. The figure clearly shows that the recognition results are delayed until end point is detected in the conventional method. Therefore, this study proposes a method in which the recognition results are produced immediately, even while the touch end point remains undetermined.

Fig 1. Pattern recognition time

In this research, a hardware system is designed based on a

simple idea for service robots with space limits that exist due

Online Touch Behavior Recognition of Hard-cover Robot Using Temporal Decision Tree Classifier

Seong-yong Koo, Jong Gwan Lim, Dong-soo Kwon, Human-robot Interaction Research Center, KAIST, Daejeon, Korea

T

Proceedings of the 17th IEEE International Symposium on Robot and Human Interactive Communication, TechnischeUniversität München, Munich, Germany, August 1-3, 2008

978-1-4244-2213-5/08/$25.00 ©2008 IEEE 425

to the use of a hard-cover and a shape design that do not collectively consider touch. In addition, means of recognizing a touch pattern while the touch is being done is proposed using multiple windowing and a temporal decision tree classifier.

II. TOUCH PATTERNS AND FEATURES SELECTION In this section, possible touch patterns are categorized for

a hard-cover robot and the features that can characterize each pattern are investigated.

A. Touch Pattern Categories Situations of physical interference and intended contact

with a robot are categorized in [5]. Table I shows 31 verbs that describe possible contact with a robots.

In order to determine touch patterns that can be classified for a hard-cover robot, the definition of each verb from Cambridge English Dictionary [7] was referenced, and features such as the contact time, the incidence of a repeat, the force, purpose, object used in the touch, direction, and surface form were determined. According to the similarity between features, they were categorized as shown in Fig 2.

Fig 2. Touch classification according to feature similarity

B. Selection of Touch Patterns to be recognized Among the 31 touch actions that were determined, four 4

touch patterns were selected for recognition, hit, beat, push and rub. Certain types of touch such as a pinch or a pull cannot be executed in a hard-cover robot; hence, sub trees from volume in Fig 2 are excluded in this study. Four high classes of other patterns are hit, beat, push and rub.

Features for classifying four touch patterns are defined from the definitions of patterns. As seen in Fig 1, hit and beat are delineated from push and rub according to the amount of time a human maintains this type of touch to the robot. The distinguishable feature of hit and beat involves whether these actions are conducted repeatedly; push and rub involves changes of the contact area because when a human pushes an object in a direction normal to the surface, the contact area does not change. In contrast, a rub implies a change in the contact area.

Table II shows the feature states of each touch pattern.

III. SYSTEM DESCRIPTION This section describes the hardware and system

architecture of the touch recognition interface system that is used in the head of a hard-cover service robot.

A. Hardware A head of the service robot is covered by round-shaped

plastic as shown in Fig 3(a).

(a). Round shape hard-cover (b). Sensors below the cover

Fig 3. Hardware of touch recognition system Although it is difficult to deform the cover and attach

additional devices to the hard cover, it is necessary to detect 4 features of the force, contact time, repetition, and contact area changes. For these reasons, two non-contact sensors were chosen: 3x3 charge-transfer touch sensors (QT113) along with an accelerometer. A charge-transfer touch sensor can detect whether a human hand is located a short distance

TABLE I VERBS RELATED TO CONTACT TO THE ROBOT

Touch Beat Pick Scrub Hit Push Pull Grasp

Collide Thrust Tug Grip Smite Poke Draw Seize

Pat Jab Drag Pinch Tap Jog Tweak Stroke Slap Nudge Scrape Scratch

Punch Prod Rub

TABLE II FEATURE STATES OF EACH TOUCH PATTERN

Force Contact time Repeat Contact area change

Hit N / H Short No No Beat L / N Short Yes No Rub L Long No Yes Push L / N Long No No

L : low force, N : normal force, H : High force

426

from the electrode. The accelerometer is used to calculate the touch force by sensing the vibration of the hard cover. Both can be attached to the inside of the cover, as shown in Fig. 3(b).

B. Real- time and online system Real-time and online recognition implies that all of the

processes should be done in a specific time with the result given at each specific time regardless of whether or not a human is touching the system. This system makes it possible to recognize touch patterns even while a human is touching the system.

Fig. 4 shows the system architecture. The microprocessor in the figure has three processes: a preprocessor to filter raw data and convert it into meaningful data, a feature extractor to create features from the data, and a classifier. All sensor data are acquired once every 10ms, and touch patterns are classified every 40ms.

Fig 4. Real time and online system architecture

IV. TOUCH RECOGNITION This section describes the touch recognition method that

accomplishes these two objectives. 1) Touch pattern recognition of hit, beat, rub and push 2) Online classification

The multi-windowing concept for feature extraction and the temporal decision tree classifier are designed to accomplish the two objectives.

A. Preprocessing Preprocessing occurs at every 10ms to filter touch sensor

data and to store data to be manipulated for features in the next step.

Input data are defined as follows, 1) 9 Touch sensor data: )8...0]([ =ikTi 2) Acceleration data: ][kA

A digital filter is used for the digital values of the touch sensors due to the noise. It is designed according to the rule that states that an item of data is accepted when the value continues over three steps. An example of filtered data is shown below.

:][:][

kTkT

fi

i

00011111100000.....1000010011101000

The last four filtered items of touch data and 20 items of acceleration data are stored for the next step involving multi-windowing feature extraction.

Output data of preprocessing are defined as follows:

1) Last 4 filtered each touch data: ]3[~][ −kTkT fifi

2) Last 20 acceleration data: ]19[~][ −kAkA

B. Multi-windowing feature extraction There are four features that are used to characterize each

touch pattern. These are the force, contact time, repetition, and changes in the contact area. Each feature can be extracted by investigating sensor data for different periods. The force is calculated by acceleration of the vibration of the cover for 200ms. Contact area changes are calculated in this manner for 40ms, and contact time / repetitions are calculated for 10ms using touch sensor data. These three windows move forward every 10ms as shown in Fig. 5, so that features can be obtained in real time.

Fig 5. Multi-windowing feature extraction

The force ( F ) can be calculated as the variance of the last

20 acceleration data. ])19[],...,1[],[var(][ −−= kAkAkAkF

Touch area (TA ) is simply a sum of nine items of touch data, and an instance of contact (or no instance) ( C ) is determined depending on whether each touch area is 0.

∑=

=8

0][][

ifi kTkTA

ohterwisekTA

kC,0

0][,1][

≠=

The start point and end point ( ES , ) can be detected by the difference between ][kC and ]1[ −kC .

]1[][][ −−= kCkCkS ][]1[][ kCkCkE −−=

The contact time ( CT ) is the time difference between the current time and the start point time ( ST ).

ohterwiseSTkSk

ST,

1][, ==

ohterwisekCSTk

kCT,0

1][,][

=−=

No contact time ( NCT ) is the time difference between the current time and end the point time ( ET ).

ohterwiseETkEk

ET,

1][, ==

ohterwisekCETk

kNCT,0

0][,][

=−=

Repetition ( R ) is determined by a short no contact time at the start point.

427

ohterwisekSandTkNCT

kR REP

,01][][,1

][=<

=

Contact area changes ( CAC ) are calculated by the sum of the last four contact area differences ( CAD ) which is the sum of the touch differences (TD ) as follows:

]1[][][ −−= kTkTkTD fifii

∑=

=8

0][][

ii kTDkCAD

∑=

−=3

0][][

i

ikCADkCAC

There are three values of force, and two values each of contact time, repetition, and contact area changes in Table II. To determine the threshold values of the feature values, 15 touch patterns of subjects (13 men and 2 women ranging in age from 24 to 39) were investigated. The results are shown in Table III.

C. Temporal Decision Tree Classifier Table II shows that touch patterns can be classified by 4

feature values. However, a classification based on Table II cannot be implemented at every time step because a different time is required to make each feature available for use; moreover, each touch pattern can be classified at different time. For example, a hit can be detected directly after this type of touch occurs, while a rub and a push require some time to be detected. One means of solving this problem is to wait for the completion of all touch inputs and feature extractions before making the determination. For this reason, most classifications of human input were implemented offline.

One objective of this research is to classify touch patterns online so that a robot can recognize a human touch immediately without waiting until it terminates, even if a human touches a robot continuously and changes touch patterns.

A temporal decision tree is proposed to classify touch patterns online. A temporal decision tree, which is an extension of a decision tree as time flows, includes temporal information at each node so that it waits some time before it is split into the next branch according to the feature value [8].

It can be obtained from a temporal example table which describes the time flow of features according to each situation in which a specific touch occurs [9].

Table IV shows a temporal example table of four situations in which four touch patterns are input by a human.

TABLE IV

TEMPORAL EXAMPLE TABLE time sit1 sit2 time sit3 sit4

Force

0 N / H L / N 0 L L / N1 N / H L / N 1 L L / N... N / H L / N ... L L / N

endT N / H L / N LT L L / N

Contact

0 1 1 0 1 1 1 1 1 1 1 1 ... 1 1 ... 1 1

endT 0 0 LT 1 1

endT +5 0 RT 1 1

Repeat

0 0 1 0 0 0 1 0 1 1 0 0 ... 0 1 ... 0 0

endT 0 0 LT 0 0

Contact area change

0 NA NA 0 1 0 1 NA NA 1 1 0 ... NA NA ... 1 0

endT NA NA RT 1 0

CL Hit Beat Rub PushDL LT LT

endT : end time at which touch input terminates

LT : long time, boundary between short and long touch

RT : rub time from which rub can be recognized NA: Not Available

A temporal decision tree is composed based on Table IV

along with additional 3 situations for reliability in Fig 6.

Fig 6. Temporal Decision Tree

This temporal decision tree is executed once any contact

is detected and finishes when one of the pattern classes is recognized. When a human touch continues, even if the

TABLE III THRESHOLDS OF FEATURE VALUES

LNF NHF LT REPT cacP

5 20 240 480 0.1

LNF : Boundary between low and normal force

NHF : Boundary between normal and high force

LT : Boundary between short and long time

REPT : Boundary time between repeat or not

cacP : Boundary between contact area change or not

428

result is given, the tree is executed again according to the execution rule so that touch patterns can be classified online.

V. EXPERIMENTS AND RESULTS In order to validate the feasibility of the proposed touch

recognition system, experiments were conducted with 12 subjects (11 men and 1 woman between the ages of 24 and 38). They were asked to execute 10 touches for each touch pattern after reading the dictionary definition of each touch pattern.

Table V shows the result of the experiments. All of the results were obtained online so that one touch action could produce many classified results. Table V was compiled by counting all of the results and calculating recognition rate.

The system recognized the correct patterns 87.27% of the

time for a hit, 87.83% for beat, 79.62% for a rub, and 79.65% for a push.

The most frequently misclassified case was the recognition of rub as beat. This occured 13.27% of the time. This was caused when a human touch occurred out of the touch sensor range. The second most commonly misclassified case involved recognizing a hit as a push (at 9.90%). However, this occurred due to the unique touch pattern of one subject.

VI. CONCLUSION In this paper, a touch recognition system for a

hard-covered service robot was developed. A 3x3 charge-transfer touch sensor and an accelerometer were used to detect four features which can distinguish four human touch patterns of hit, beat, rub and push.

Multi-windowing feature extraction and a temporal decision tree classifier were introduced for online classification which is important for natural touch interaction between a human and a robot as opposed to an offline classification.

The experimental results showed acceptable recognition, rates of 87.2% for hit, 87.83% for beat, 79.62% for rub, and 79.65% for push. In addition, all of the results were obtained online.

The proposed touch recognition system can be applied to a service robot and can allow multi-modal human-robot interaction with vision and speech recognition.

REFERENCES [1] Michael A. Goodrich, Alan C. Schultz, “Human-Robot Interaction: A

Survey,” Foundations and Trends in Human-Computer Interaction Vol. 1, No. 3 (2007) 203-275.

[2] Sony, 1999, “Aibo”, Available: http://www.aibo.co.jp [3] W.D. Stiehl, J. Lieberman, C. Breazeal, L. Basel, L. Lalla, M. Wolf,

“Design of a therapeutic robotic companion for relational, affective touch,” Robot and Human Interactive Communication, 2005. ROMAN 2005. IEEE International Workshop on (2005) 408-415.

[4] K. Wada, T. Shibata, “Robot Therapy in a Care House-Its Sociopsychological and Physiological Effects on the Residents,” Robotics and Automation, 2006. ICRA 2006. Proceedings 2006 IEEE International Conference on (2006) 3966-3971.

[5] H. Iwata, S. Sugano, “Human-robot-contact-state identification based on tactile recognition,” Industrial Electronics, IEEE Transactions on 52 (6) (2005) 1468-1477.

[6] T. Shibata, T. Tashima, K. Tanie, “Emergence of emotional behavior through physical interaction between human and robot,” Robotics and Automation, 1999. Proceedings. 1999 IEEE International Conference on 4 (1999).

[7] http://dictionary.cambridge.org [8] L. Console, C. Picardi, D.T. Dupre, “Temporal Decision Trees:

Model-based Diagnosis of Dynamic Systems On-Board,” Journal of Artificial Intelligence Research 19 (2003) 469-512.

[9] L. Console, C. Picardi, D.T. Dupre, “Generating Temporal Decision Trees for Diagnosing Dynamic Systems,” Proc. DX 00, 11th Int. Workshop on Principles of Diagnosis.

TABLE V RECOGNITION RESULT

Recognized rate [%] Actual touch pattern Hit Beat Rub Push

Recognized touch

pattern

Hit 87.27 8.90 0.95 8.41 Beat 0.91 87.83 13.27 0.44 Rub 0.00 0.00 79.62 8.41 Push 9.90 0.00 1.90 79.65 N/T 2.72 3.26 4.27 3.10

429