Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
1
Sensor-based Body Gesture Tracking System for Natural User Interface in E-learning
Sunghyun Song
SungKyunKwan Univ.
Department of Human ICT Convergence
OUTLINE
• Introduction
• Related Work
• Proposed System
• Pilot Study
• Improvement
• Experimental Condition
• Result & Discussion (Data log, TLX, SUS, and interview)
• Conclusion
2
Introduction
• E-learning is a rapidly growing educational sector as much as 8% a year through 2016
T.Navio, “Global E-learning Market 2013-2016”, Infiniti Reserch Limited, (2013)
• The main advantage of audio-video class material is that students can flexibly play, pause,
rewind or skip around material according to their needs.
Bassili, J. N. and Joordens, S.: Media player tool use, satisfaction with online lectures and examination performance,(2008)
• According to these observations, researchers have suggested systems adjusting video
playback properties automatically. For example, reducing playback speed if a student is
taking notes.
Park, M., Kang, J., Park, S., Cho, K.: A Natural User Interface for E-learning Learners, (2014)
• However, there is no current video-playback system that can tracks user activities to
adjust dynamically video playback in order to support eLearning. A novel natural user
interface that tracks a user’s head movements to infer playback speed without explicit user
input is needed.
3
Related Work (1)
• Tracking head position to control aspects of computer use
- Lean and zoom: A system based on a camera calculates the distance between head
and screen in order to adjust the magnification of displayed contents.
Harrison, C., and Dey, A.K.: Lean and Zoom: proximity-aware user interface and content magnification (2008)
- Based on head movement, a system can adjust not only the zoom but also the location
and orientation of contents.
Yamaguchi, K., Komuro, T., Ishikawa, M.: Ptz control with head tracking for video chat (2009)
• Head-trackers used to extend the capabilities of pointer input.
- How camera-based system that tracks the user’s face can support pointing or selecting.
Kjeldsen, R.: Head Gestures for Computer Control (2001)
- Face-tracking system that allows users to quickly locate the cursor across multiple display.
Ashdown, M., Oka, K., Sato, Y.: Combining head tracking and mouse input for a GUI on multiple monitors (2005)
4
Related Work (2)
5
• Video browsing techniques
- System that lets users navigate the timeline of a video by directly dragging objects in the
displayed scene along their past or future visual path.
Dragiceive, P., Ramos, G., Bibliowitcz, J., Nowrouzezahrai, ..:Video browsing by direct manipulation (2008)
- SmartPlayer semi-automatically adjusts the video playback speed depending on the
complexity of the scene presented or user defined events.
Cheng, K., Luo, S., Chen, B., Chu, H.:SmartPlayer: user-centric video fast-forwarding (2009)
- Video control through gestures in an interface powered by the Microsoft Kinect.
Chu, T., Su, C.: A Kinect-based Handwritten Digit Recognition for TV Remote Controller (2012)
- Playful physical interface that lets children control video content through squeezing and
stretching bubble-like objects.
Ryokai, K. Raffle, H., Horii, H., Mann, Y.: Tangible video bubbles (2010)
Related Work (3)
6
• Natural User Interface (NUI)
- NUI means that expand beyond the mouse and keyboard and start incorporating more
natural forms of interaction such as touch, speech, gestures, handwriting, and vision.
Ballmer, S.: CES 2010:A Transforming Trend-The Natural User Interface (2010)
- NUIs are those that enable users to interact with computers in the way we interact with the
world.
Jain, J., Lund, A., Wixon, D.: The future of natural user interfaces., In CHI’11
Proposed System
7
• State & control
- A custom head-tracker system can
reliably detect where the user is looking
and discern among four different situations.
The video playback speed is then adjusted
to conform with the current activity.
- This system doesn’t request any additional
manipulation compared to previous research.
Proposed System
8
• Prototype HW
- six axis gyro+accelerometer sensor (MPU-6050)
mounted on a pair of headphones, and
connected with wires to a controlling unit.
(This sensor is capable of sensing head angle)
- Arduino Leonardo is micro controller serially
connected to a PC through USB port. This receive the value from sensor and process
pitch and yaw angle for decision of states.
- Calibration button on the mini bread board for setting standards of each state’s angle.
Pilot Study and Result
9
• Evaluation for the proper video playback speed
- 6 participants ( graduate students, 1 female, age 27-33 )
- Each student was presented with two tasks in random order following within-subject design
- First task : watch a short lecture video and take notes with only keyboard
- Second task : watch another video and take notes with my system and the keyboard.
- Result t(5)=4.03 , p<0.01
: keyboard input for automatic system = avg 1.3 (SD=1.2)
: keyboard-only input = avg 3.6 (SD=2.3)
• Semi-structure interview
- 0.6x is too slow and hard to understand
- 0.8x is enough for taking notes and slow condition. 0.6x 0.8x
System can reduce direct manipulation of user.
Improvement
10
• Observation from the pilot, 3 states are added
- Focus (⑦)
Context: see contents in detail / full of interest
Posture: lean forward to monitor
Control: pause
- Skimming (⑨)
Context: contents are easy or not important
Situation: reclining position
Control: fast forwarding (1.2x)
- Dwell (⑥)
Context: Student lose attention for a long time
Situation: turn head + stay in that more than 5 sec
Control: pause + rewind 15 sec
Improvement (2)
11
• Final System overview
- The seven different postures. States are
sensed by combined angular data from a head-
mounted six-axis IMU and distance data from
an infrared proximity sensor placed at the
bottom of the screen and facing the user.
Experimental Condition
12
• Material
- Video: 8min for each video clip “cold and warm fronts”, “the solar system”
• Condition
- 12 participants (7male and 5 female )
- Control condition: only keyboard input
- Tracked condition: both keyboard and tracking system
- The presentation of the videos is fully balanced.
User
Ex 1 Ex 2
Prototype Video Prototype Video
p1 O V1 X V2
p2 X V1 O V2
p3 O V2 X V1
p4 X V2 O V1
Experimental Condition
13
• Evaluation List
- Log data (e.g. Time for staying each states, The number of transition, and so on)
- TLX ( Task Load Index ) to check mental, physical load and so on.
- SUS ( System Usability Scale ) to check usability and learnability
- Semi-structured interview to check pros, cons and additional points
• Experiment order
1. Introduce our experiment
2. Practice using prototype ( 2min )
3 (6). Watch the video
4 (7). Test for understanding ( also, induce taking down a lecture )
5 (8). TLX
9. SUS
10. Interview
Repeat one more
J, Brooke.: SUS: a ‘quick and dirty’ usability scale (1996)
Experiment
14
• Raw data
Log data TEST, TLX, SUS and so on
15
Result & Discussion_ System Log
Size of the circles = Total time was spent in each states
Line = State Transition
• In tracked, most common transition took place btw ‘Normal’ and ‘Others’
• Transition btw ‘Normal’ and ‘Notes’ is occurred more frequently in tracked C.
than in control C.
• In tracked C, transition btw Normal and Distract is also many.
16
Result & Discussion _ TLX
From initial t-test
• Significantly lower Overall Workload(p=0.016) in the tracked condition
• Improvement in
Mental Demand(p=0.011), Temporal Demand (p<0.002), and Effort (p<0.04)
Development of NASA-TLX(Task Load Index): Results of empirical and theoretical Research (1988)
SG. Hart.: NASA-task load index (NASA-TLX); 20 years later (2006)
17
Result & Discussion _ SUS
J, Brooke.: SUS: a ‘quick and dirty’ usability scale (1996)
High Score Rank
• Q4: “Can learn by self”
• Q7: “Easy to learn”
• Q3: “Easy to use”
• Question
• Result in Likart 5 scale
learnability
18
Result & Discussion _ Semi-Structured Interview
• Question
(1) (Pros) Useful or comfortable point?
(2) (Cons) Inconvenience points or things to be improved?
(3) Is there any function to be added?
(4) Any other things?
• Answer_ key points
(1) “I want to get cue of transition... I felt anxiety because I don’t know which states I’m in”
=> People need feedback when transition be occurred. However, it should not disrupt user.
(2) “The speed of changing playback speed is too fast. I felt abnormal.”
=> System need delay btw 0.1x differency
(3) “It will be better, if I can customize the transition degree of each states.”
(4) “If the system provide the way of manipulating deliberately, will be better.”
=> For example, turn left your head two time quickly = 15 seconds rewind”
19
Conclusion
• Proposed Natural User Interface can
- be easy to learn and use based on SUS.
- lower user’s cognitive workload specifically in Mental, Temporal, and Effort based on TLX.
- be dynamically used during the studying based on Data Log.
• This system could be highly beneficial in learning scenarios.
• For better usefulness, below things could be added.
- Customizing the degree of transition
- Smooth change of playback speed
- Manipulating intentionally for more various control such as rewinding or skipping
- Feedback for transition
Reference
20
[1] T.Navio, “Global E-learning Market 2013-2016”, Infiniti Reserch Limited, (2013)
[2] Bassili, J. N. and Joordens, S.: Media player tool use, satisfaction with online lectures and examination performance, in J.
Distance Education, 22(2), (2008)
[3] Park, M., Kang, J., Park, S., Cho, K.: A Natural User Interface for E-learning Learners, in I.J. Multimedia and Ubiquitous
Engineering 9(7), (2014)
[4] Harrison, C., and Dey, A.K.: Lean and Zoom: proximity-aware user interface and content magnification in Proc. of CHI '08,
507-510 (2008)
[5] Yamaguchi, K., Komuro, T., Ishikawa, M.: Ptz control with head tracking for video chat, in CHI EA '09, 3919-3924. (2009)
[6] Kjeldsen, R.: Head Gestures for Computer Control, in IEEE ICCV Workshop on RATFG-RTS'01, 61-67 (2001)
[7] Ashdown, M., Oka, K., Sato, Y.: Combining head tracking and mouse input for a GUI on multiple monitors, in CHI EA '05,
1188-1191. (2005)
[9] Dragiceive, P., Ramos, G., Bibliowitcz, J., Nowrouzezahrai, ..:Video browsing by direct manipulation, in Proc. of CHI '08.
(2008)
[10] Chu, T., Su, C.: A Kinect-based Handwritten Digit Recognition for TV Remote Controller, in ISPACS'12, 414-419. (2012)
[11] Ryokai, K. Raffle, H., Horii, H., Mann, Y.: Tangible video bubbles, in CHI EA'10, 2775-2784. (2010)
[12] Cheng, K., Luo, S., Chen, B., Chu, H.:SmartPlayer: user-centric video fast-forwarding, in Proc of CHI '09, 789-798. (2009)
Reference
21
[13] Kim, J., Guo, P.J., Cai, C.J., Li, S., Gajos,K.Z., Miller, R.C.: Data-driven interaction techniques for improving navigation of
educational videos, in Proc. UIST’14, 563-572 (2014)
[14] Crossan, A., McGill, M., Brewster, S., Murray-Smith, R.” Head tilting for interaction in mobile contexts, in MobileHCI’09,
article 6 (2009)
[15] Kurihara, K.: CinemaGazer: a system for watching videos at very high speed, in AVI’12, 108-115 (2012)
[16] Smith, J.D, Graham, T.C.:Use of eye movements for video game control, in ACE 2006
[17] Hart, S. G., Staveland, L. E.: Development of NASA-TLX(Task Load Index): Results of empirical and theoretical Research,
Advances in psychology, 52, 139-183 (1988)
[18] Hart, S. G.: NASA-task load index (NASA-TLX); 20 years later, In Proc of the human factors and ergonomics society
annual meeting, Vol. 50, No. 9 (2006)
[19] Brooke, J.: SUS: a ‘quick and dirty’ usability scale, Usability evaluation in industry, 189-194 (1996)
[20] Ballmer, S.: CES 2010:A Transforming Trend-The Natural User Interface (2010)
[21] Jain, J., Lund, A., Wixon, D.: The future of natural user interfaces., In CHI’11. 211-214 (2011)
[22] Schilit, B., Adams, N., Want, R.: Context-Aware Computing appllication, In IEEE WMCSA’94, 85-90 (1994)
[23] Asteriadis, S., Tzouveli, P., Karpouzis, K., Kollias, S. : Estimation of behavioral user state based on eye gaze and head
pose—application in an e-learning environment. Multimedia Tools and Applications, 41(3), 469-493. (2009)
22
Thank you!
Sunghyun Song
23
Appendix_TLX
http://pro.sagepub.com/content/50/9/904.full.pdf
* Paper
24
Appendix_SUS
http://www.tbistafftraining.info/smartphones/documents/b5_during_the_trial_usability_scale_v1_09aug11.pdf
https://www.google.co.kr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CBwQFjAAahUKEwjk86Lp443JAhWmJ6YKHeWuBcQ&url=http%3A%2F%2Fwww.measuringux.com%2FSUS_Calculation.xls&usg=AFQjCNGLIUugJzqJtVqOZTDwkg-dt8Iv_w&sig2=5XV_IGss8e3VmllCHoKyBw&bvm=bv.107467506,d.dGY
* Paper
* Excel Sheet download (Ctrl+C and V)
25
•Context Aware Computing
- Head pose could be used for tracking student’s
attention
Asteriadis, S., Tzouveli, P., Karpouzis, K., Kollias, S. : Estimation of
behavioral user state based on eye gaze and head
pose—application in an e-learning environment. Multimedia Tools and
Applications, 41(3), 469-493. (2009)
- Context-triggered actions are simple IF-THEN rules
used to specify how context-aware
systems should adapt.
Bill, S., Norman, A., Roy, W.: Context-Aware Computing Appllication (1994)
Appendix_SUS
Concept idea
26
System will
(1) Notice Context from the posture of Students
(2) Control video playback speed automatically according to their posture.
( e.g If posture1 then playback speed1. )
As a result, system will be gesture-based user interface