Researcher: Zi Siang See
Research Presentation: Zi Siang See
Technical developmentEmail (University of South Australia): [email protected] (Sunway University): [email protected] (Industrial collaboration): [email protected] (University Tunku Abdul Rahman): [email protected] (University Tunku Abdul Rahman): [email protected]
Zi Siang See, Mark Billinghurst, Vimalan Rengganaten, Shelly Soo (2016)
SIGGRAPH Asia 2016 Mobile Graphics And Interactive Applications
Medical Learning Murmurs Simulation
with Mobile Audible Augmented Reality
Research &
collaboration
Zi Siang See, Mark Billinghurst, Vimalan Rengganaten, Shelly Soo (2016) Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality. SIGGRAPH Asia 2016 Mobile Graphics And Interactive Applications.
Researcher: Zi Siang See
Presentation Outline
1) Introduction, Background
2) Proposed Method & Apparatus
3) User Study
4) Discussion
Research &
collaboration
Researcher: Zi Siang See
Figure: augmented reality (AR), examples /preview from search engine
Introduction: Augmented Reality (AR)
• Displays, trackers, and AR systems in general need to become more accurate,
lighter, cheaper, and less power consuming (Azuma, 2001).
Researcher: Zi Siang See
Medical Student’s Life
• Educators and trainees work with expensive hardware
• Institutional-owned
• Hardware are not mobile
• Time and space constraints on fixed location-based environment
• Unable to include dynamic information on-demand
Figure: current medical learning laboratory, Sg Long, UTAR, MY.
Researcher: Zi Siang See
Medical Student’s Life
• Educators and trainees work with expensive hardware
• Institutional-owned
• Hardware are not mobile
• Time and space constraints on fixed location-based environment
• Unable to include dynamic information on-demand
Figure: current medical learning laboratory, Sg Long, UTAR, MY.SAM
Harvey
Harvey, Gordon Center for Research in Medical
Education, University of Miami
Student Auscultation Manikin, Cardionics.
Researcher: Zi Siang See
Medical Student’s Life
Researcher: Zi Siang See
Medical Student’s Life
Simulator provides murmurs audio
Instructor /students use stethoscope to listen and practice
Researcher: Zi Siang See
Conventional Mobile AR
• Almost everyone has a mobile device
• Mobile devices are internet-based + deliver 3D graphics
Researcher: Zi Siang See
AR for Medical Education
• AR has been used for a variety of different medical applications
• VR diagnostic imaging and patient treatment visualization (Sielhorst et al., 2008;
King et al., 2016).
• AR has also been used for medical education
• Showing virtual CT data onto the body of a trainee (Zhu et al., 2014)
• Viewing human anatomy with AR (Kamphuis et al., 2014, Bacca et al., 2014).
Researcher: Zi Siang See
AR for Clothes
• Augment alternative clothing /fashion experience (e.g. using Kinect)
• Additional AR content on markers imprinted on clothes for branding
• Previous research on using cloth for AR tracking
Trauman, 2015; Hilsmann and Eisert, 2009; Cushen and Nixon, 2012; Bradley et al., 2009.
Researcher: Zi Siang See
• Works on everyone’s mobile devices + AL-60 digital stethoscope
• Clothing system with imprinted markers (low-cost)
• Wearable /portable everywhere anytime
Murmurs AR from Aortic /Tricuspid /Pulmonary
/Mitral
Process AR tracking,
markers captured from mobile
Provides
murmur audio
Cloth with
imprinted marker
Server,
processor
Mobile audible AR interface +AL-60(optional graphic visual)
Mobile AR + Clothes + Medical Learning
Researcher: Zi Siang See
Mobile AR + Clothes + Medical Learning
Areas /position Possible murmurs detection
Aortic area Aortic valvular stenosisEarly diastolic aorticregurgitation
Pulmonary area Pulmonic valvular stenosis
Mitral area Acute mitral regurgitationMitral stenosis
Tricuspid area Tricuspid regurgitationTricuspid stenosisPericardial friction rubEarly diastolic aorticregurgitation
• Image-based tracking
• The prototype system is also transferable to other platforms
Researcher: Zi Siang See
• Different murmur digital wave patterns
• Medical trainees need to identify different types of murmurs
• With the AR murmurs simulator, digitized audio content can be
assigned or customized to suit various medical simulations.
Mobile AR + Clothes + Medical Learning
Researcher: Zi Siang See
Prototype Method and Apparatus
Researcher: Zi Siang See
Prototype Method and Apparatus
Mobile audible AR with murmur simulation
AL-60 stethoscope for listening and practicing (portable, low-cost, ease-of-use)
Researcher: Zi Siang See
User Study
• Measure usefulness of the prototype
• 15 medical students were involved in the test
• 9 female, and 6 male ranging in age between 20 to 23 years old (SD = 1.21).
Researcher: Zi Siang See
User Study
Setup (S1, S2)
• Each participant was asked to configure the AL-60 stethoscope with
mobile device with the AR application (S1).
• Put on the T-shirt imprinted with AR markers and become the subject
model for the next participant (S2).
Condition (C1, C2)
Run the prototype AR application twice, listening for audible murmurs:
• Once with the visual annotation aid (C1).
• Once without visual aid (C2).
Researcher: Zi Siang See
User Study
• Subject model was allowed to sit or lay down in a clinical
environment to mimic a mock patient visit session.
• Within-subjects experimental design, where each participant
experiences both of the AR visual display conditions.
Researcher: Zi Siang See
User Study
No. Questions
Q1 I found it easy to use
Q2 I found it natural to use
Q3 I found it reliable
Q4 I found it physically challenging
Q5 I found it mentally challenging
Q6 I found it useful
Researcher: Zi Siang See
User Study
No. Questions
Q1 I found it easy to use
Q2 I found it natural to use
Q3 I found it reliable
Q4 I found it physically challenging
Q5 I found it mentally challenging
Q6 I found it useful
* Perceived easiness and usefulness
Researcher: Zi Siang See
* Perceived difficulties
User Study
No. Questions
Q1 I found it easy to use
Q2 I found it natural to use
Q3 I found it reliable
Q4 I found it physically challenging
Q5 I found it mentally challenging
Q6 I found it useful
Researcher: Zi Siang See
: average results of C1 and C2 survey question.
Subjective Feedback
Figure: average results of C1 and C2 survey questions
ease of use
natural
reliable
Physically challenging
Mentally challenging
Usefullness
Researcher: Zi Siang See
User Study: Results
• Wilcoxon Signed Rank test was used to analyze the results– check for significant difference between using visual (C1) and non-visual interfaces (C2).
• For Q1, using a one-tailed test we found that participants felt that the visual
aid made the interface significantly easier to use than the interface with non-
visual cues, Z = -1.82, p = 0.03.
• For Q2, finding the interface natural to use, there was a near significant
difference between C1 and C2, with Z = -1.60, p = 0.06.
• There was no difference between conditions in terms of how reliable
participants felt each condition was (Q3), Z = -0.17, p = 0.44.
• In terms of the physical challenge, participants felt that the non-visual
condition (C2) was significantly more physically challenging (Q4) than the
visual condition (C1), Z = -2.191, p = 0.014.
• Similarly, C2 was felt to be more mentally challenging (Q5) than C1, Z = -
2.31, p = 0.01.
• C1 was viewed as being more useful (Q6) than C2, Z = -1.87, p = 0.03.
• Overall, these results show that the interface with visual aids was better
than the not using any visual aids.
Researcher: Zi Siang See
: average results of C1 and C2 survey question.
User Study: Setup
Figure: average results of S1 and S2 survey question.
ease of use
natural
reliable
Physically challenging
Mentally challenging
Usefullness
Researcher: Zi Siang See
User Study: Setup
• Users found the configuration of the stethoscope (S1) relatively easy
(Q1) and useful (Q6), but it was also rated above average in terms
of physical (Q4) and mental challenge (Q5).
• Similarly from S2 users found it easy to wear the tracking T-shirt
(Q1) and that it was useful (Q6), but it was an above average
physical challenge to use (Q4).
Researcher: Zi Siang See
User Study: Feedback
• Users said they “.. like the idea of it being portable.. ”,
• “The idea is pretty good, affordable and portable”,
• and “It is usually challenging to remember all murmurs and I hope
this new app can help me about it”.
• However users also felt that “.. it is quite difficult to use..”,
• “Markers are hard to detect under different lighting”,
• and users felt “.. quite exposed and uneasy wearing the shirt and
scanned by others”.
Researcher: Zi Siang See
Discussion
• Most users felt that having the interface with visual cues was more
useful as it provides clear visual feedback for an AR murmur audio is
being tracked.
• It can be confusing when the system is used without visual
feedback, as users could not be sure if an AR audio cue will be
played.
• If there is no audio feedback users move on to the next marker to
search for other murmurs to be heard, as a result users were
moving the mobile application around without getting much murmurs
being experienced (slow cloud-based AR loading, crease, etc).
• The use of AR and cloth tracking simulates the physical experience
of holding a digital stethoscope to check on a heart murmur.
• Audible AR also allows the users to be able to hear uncommon
heart murmurs that rarely occur during a diagnosis of a real patient.
Researcher: Zi Siang See
Conclusion
• This paper presents a medical education system teaching about
heart murmurs using mobile audible Augmented Reality.
• Existing mobile hardware using image-based tracking on clothes.
• Low cost and can be easily used in a wide variety of setting.
• Users felt that the system was natural and easy to use, and the use
of additional visual cues made the system much more useful.
• In the future, we would like to explore mobile audible AR for different
murmurs scenarios and the suitability of the proposed system being
tested in examination.
Murmurs AR from Aortic
/Tricuspid /Pulmonary /Mitral
Researcher: Zi Siang See
Further Development
• Smaller devices
• Macro lens (smaller markers)
• T-shirt different sizes
• Markers on clip-on badges
• Instructors use it for exam
• Offline /cloud-based?
Researcher: Zi Siang See
Demo
Researcher: Zi Siang See
Questions
• Name/Email (feedback/ discussion)
Contact
Zi Siang See, Mark Billinghurst, Vimalan Rengganaten, Shelly Soo (2016) Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality. SIGGRAPH Asia 2016 Mobile Graphics And Interactive Applications.
Paper