PowerPoint Presentation
HoloToolkit for UnityGaze & VoiceDmitri BouianovFounder @
Context VR
BackgroundComputer Science, Software Engineering
Unity + AR/VR/MR + full-stack development
Multiple startups, technical & product roles
Looking for experts: Unity, graphics, shaders
OverviewHoloToolkit is an open-source framework brought to you
by Microsoft HoloLens team
Encapsulates and simplifies key concepts of working with
HoloLens: Gaze, Gesture, Voice, Spatial Audio, Spatial
Perception
Today: Gaze & Voice Recognition
Gaze & Keyword ManagersHoloToolkit provides GazeManager
prefab for getting information about objects hit with users
gaze
HoloLens Unity SDK provides KeywordRecognizer API to register
voice commands and trigger event callbacks
Example: Bigger, Smaller to adjust object size
HoloToolkit adds KeywordManager prefab for registering keywords
and handlers in Unity Editor
ProblemScenario: Looking at objects instantiated at runtime and
sending voice commands to those objects
Problem: Objects need to exist in scene hierarchy to register
keywords and event handlers in KeywordManager
Workaround: OnSelect sent by GestureManager, OnGazeEnter &
OnGazeLeave sent by GazeManager + multiple dynamically created
KeywordRecognizers
Ideal: No boilerplate code when using voice commands + solution
that scales with large number of objects
Problem
SolutionFocusedObjectMessageSender component is a proxy that
sends keyword events to any currently focused object
FocusedObjectMessageSender
SolutionSelectedObjectMessageSender component adds persistent
selection to explicitly select objects for voice interaction
SelectedObjectMessageSender
OutcomeNow: Sending voice commands to focused and selected
objects is easy, no code required
Simple: Modular architecture decouples message senders from
message receivers
Next: Automatic registration of keywords from holographic UI
elements and other objects
Challenges: String message performance + type safety
Call to ActionDiscuss issues and challenges on HoloLens
Developer Forum & at WinHUGR meetups
Recognize common problems that can be solved for everyone with
some minimal code
Fork HoloTookit-Unity on Github, open issues, submit pull
requests, address code reviews
Spread the word and help grow the community
Thank You!@dbouianov
@contextvr
[email protected]
http://contextvr.com
ResourcesHoloToolkit for
Unityhttps://github.com/Microsoft/HoloToolkit-Unityhttps://github.com/Microsoft/HoloToolkit-Unity/issues/71https://github.com/Microsoft/HoloToolkit-Unity/pull/72https://github.com/Microsoft/HoloToolkit-Unity/pull/69
HoloLens Developer Forumshttp://forums.hololens.com/
WinHUGRhttp://www.meetup.com/Windows-Holographic-User-Group-Redmond/
