October 1, 2018October 1, 2018

Mark F. Bocko | ProfessorDepartment: Electrical and Computer EngineeringFocus: Spatial audio

Pilot Project: “Development of a quantitative framework for spatial audio characterization”

Project Goals• Develop quantitative methods to assess spatial audio rendering systems• Incorporate quantitative binaural hearing models into audio system design tools• Predict what listeners will report hearing (locations, spatial extent of sources, diffusiveness)

October 2018

Geunyoung Yoon | Professor

Department: Ophthalmology, The Institute of Optics, Center for Visual Science, Biomedical EngineeringFocus: Physiological Optics, Vision Correction, Visual Psychophysics, Optical Imaging, Biomechanics, Eye DiseasesLab website: http://www.cvs.rochester.edu/yoonlab/

OCULAR OPTICS & CUSTOMIZED VISION CORRECTION• Eye’s aberration and visual quality• Ocular wavefront sensing• Advanced ophthalmic lenses• Sport vision• Optical metrology

OCULAR OPTICS and VISION• Adaptive optics vision simulator• Adaptation to habitual optics• Neural processing and perception• Binocular integration• Neural plasticity• Stereopsis

ANTERIOR SEGMENT IMAGING• Mechanisms of pathologic cornea

diseases• Ocular surface diseases and dry eye• Corneal biomechanics• Multimodal high-resolution ocular

imaging• Advanced cataract surgery

RESEARCH TOPICS:

ACOOMMODATION & PRESBYOPIA• Vergence-Accommodation conflict

under VR/AR environments• Extended depth of focus technology• Accommodating intraocular lens• Peripheral vision and optics• Binocular accommodation• Emmetropization / Refractive error

October 2018

Zhiyao Duan | Assistant ProfessorDepartment: Electrical and Computer EngineeringFocus: Computer Audition, Music Information Retrieval, Audio-Visual Analysis, Audio for AR/VRLab: Audio Information Research (AIR) Lab

RESEARCH TOPICS:

MUSIC INFORMATION RETRIEVAL• Music transcription• Audio-score alignment• Music source separation• Music generation

AUDIO-VISUAL PROCESSING• AV analysis of music performances• Visually informed source separation• Music performance generation• Talking face generation from speech

ENVIRONMENTAL SOUND UNDERSTANDING• Sound search by vocal imitation• Sound event detection• Source localization and tracking

SPEECH PROCESSING• Speech emotion classification• Speaker recognition and diarization• Speech enhancement

October 2018

Andrew White| Assistant ProfessorDepartment: Chemical EngineeringFocus: Role of AR/VR in higher education, computational chemistryCourses: CHE 116 – Numerical Methods & Statistics | CHE 477 – Advanced Numerical Methods

Role of AR and VR in Higher Education• Collaborative project exploring the use of

AR and VR in STEM curriculum

• Emphasis on tactile, collaborative,interactive replacement for traditionalhands-on labs

• Intended for topics that are abstract orimpossible to have labs, e.g. quantummechanics or solving ODEs

• Provides a new tool for outreach togenerate enthusiasm for STEM careers

RESEARCH TOPICS:Computational Chemistry• Computer simulation of dynamics of

molecules at the level of atoms

• Provides insight at a length-scale inaccessibleto experiments

• Requires careful design of scientificallyaccurate, highly-parallel, algorithms and code

• Use techniques like multi-scale modeling tostudy complex phenomena like proteinadsorption

• High accuracy through novel methods toincorporate data derived from experiments insimulations

October 2018

Yuhao Zhu | Assistant ProfessorDepartment: Computer Science, Goergen Institute for Data ScienceFocus: Architecting next-generation computer hardware for an AR/VR-driven future!

RESEARCH TOPICS: Co-Design Computing Systems with Non-computing Systems for AR/VR

• Optical system + image sensor + imaging + computer vision (w/ Jannick Rolland)• Does the optimal design of optical system change with the specific vision task?• Does the optimal design of vision hardware change with optical systems?• How to design an end-to-end system with specific tasks and quality metrics in mind?• How to dynamically reconfigure both optical systems and computer systems on the fly?

October 2018

Michele Rucci | ProfessorDepartment: Brain & Cognitive SciencesFocus: Action & perception, visual perception in humans and machines, human behavior, sensory processing

COMPUTATIONAL MECHANISMS• Computational goals of visual

processing• Establishment of spatial

representations• Multimodal integration

HUMAN BEHAVIOR• Action properties • Identification of visuomotor

strategies• Limits of oculomotor control• Natural head-eye coordination

VISUAL DEVELOPMENT• Consequences of eye movements in

visual maturation• Abnormal eye movements• Myopia

RESEARCH TOPICS:

ACTIVE VISION • Vision as sensorimotor integrated

process• Dependence of visual functions on

eye movements• Disruption of oculomotor cycle via

gaze-contingent control

Building Virtual Concert Halls with Spatial AudioMing-Lun Lee, Matthew Brown, Zhiyao Duan, and Steven Philbert

Department of Electrical and Computer EngineeringEastman School of Music

Building Virtual Concert Halls with Spatial AudioMing-Lun Lee, Matthew Brown, Zhiyao Duan, and Steven Philbert

Department of Electrical and Computer EngineeringEastman School of Music

zbai@cs.rochester.edu http://zhenbai.io October 2018

Zhen Bai| Assistant ProfessorDepartment: Computer ScienceFocus: Human-Computer Interaction, Augmented Reality, Tangible User Interface, Embodied Conversational Agent, Education Technology, Computer-Supported Collaboration, Design for Diversity

RESEARCH TOPICS: Augmented Reality - Theory of Mind, Symbolic Play, Children with Autism Spectrum Condition

UR AR/VR Pilot Project: Real-time synthesis of a virtual talking face from acoustic speech

• Chenliang Xu, Assistant Professor of Computer Science• Collaborators:

• Ross Maddox, Assistant Professor of Biomedical Engineering • Zhiyao Duan, Assistant Professor of Electrical and Computer Engineering

[Chen, Li, Maddox, Duan, and Xu, ECCV 2018]

October 2018

Ross Maddox | Assistant ProfessorDepartment: Neuroscience & Biomedical EngineeringFocus: Audio-visual integration, selective attention, sound localization

AUDIO-VISUAL INTEGRATION• Multisensory binding and object

formation• Impact of “uninformative” visual

stimuli on auditory perception

VISUAL HEARING AID• Generate an artificial talking face

from speech audio in real-time• Improve listening abilities for

people with hearing impairments, attention disorders

RESEARCH TOPICS:

GAZE EFFECTS ON SPATIAL HEARING• Interaction between eye

movements and auditory spatial acuity

• Benefits of directed eye gaze on speech-in-noise understanding

BRAINSTEM CODING OF SPEECH• Use electroencephalography and

novel signal processing schemes to study how brainstem codes speech

• Investigate subcortical effects of attention and cognition

October 2018

Martina Poletti| Assistant ProfessorDepartment: NeuroscienceFocus: Visual perception, eye movements, attention, eyetracking.

RESEARCH TOPICS:

VISUOSPATIAL ATTENTION• Resolution of attention in the fovea• Attention contribution to fine

spatial vision• Pre-microsaccadic enhancements of

foveal vision

HIGH ACUITY VISION• Fine control of eye movements

during high acuity tasks• Distribution of high acuity

capabilities across the fovea• Oculomotor strategies in fine spatial

vision

SPATIAL REPRESENTATIONS• Multimodal integration• Spatial updating across saccades• Spatial updating during fixation

FOVEAL PRIORITY MAPS• Driving factors • Perceptual benefits• Spatiotemporal dynamics• Visual exploration at the foveal

scale

October 2018

River Campus LibrariesPresenter: Lauren Di Monte, Director of Research InitiativesFocus: Support AR/VR research, teaching, and learning

AR/VR Creation and Exploration Space

Enhance access and support Provide on-ramps Grow a community of practice

UR Medicine Health Lab (Hasselberg, Mitten, Dasilva)• Expertise in technology innovation to improve delivery of care

Department of Psychiatry (Cross, Hasselberg)• Expertise in cognitive behavior therapy, and implementation science

Eastman School of Music (Brown, Winders)• Expertise in visual and audio therapeutic functions

Art, Science, & Engineering (Luo)• Expertise in computer science and smartphone sensors

13

Cognitive Behavior Therapy Mobile App with Embedded Virtual Reality

October 2018

Edmund Lalor | Associate ProfessorDepartment: Biomedical Engineering and NeuroscienceFocus: Human neuroscience, sensory processing, perception, cognition

HUMAN SENSORY PROCESSING • Hierarchical processing of natural

audio and visual stimuli• The effects of knowledge and

prediction on early sensory processing

MULTISENSORY INTEGRATION• Audiovisual speech processing• The effect of visual input on

auditory scene segregation

NEURAL SIGNAL DECODING• Methods for decoding multivariate

neural data• Decoding representations of

acoustic space in the cortex• Decoding selective attention in real-

time

RESEARCH TOPICS:

ATTENTION• Visual spatial attention• Auditory selective attention –

particularly to speech (i.e., the cocktail party problem)

October 2018

Ania Busza | Assistant ProfessorDepartment: Neurology (Stroke division)Focus: Stroke Rehabilitation

KEY ISSUES IN STROKE REHABILITATION AND RECOVERY• What factors predict stroke recovery?• What is the best timing/dose for rehabilitation therapies?• How can we use new technologies to create more effective therapeutics?

RESEARCH TOPICS:

(1) Surface-EMG controlled Virtual Arm

(2) Using superficial sensors to quantify rehab “dose”

(3) EMG analysis of Motor System fatigue during learning

October 2018

Michael Jarvis| Associate ProfessorDirector, Digital Media Studies ProgramDepartments: History & ATHS

RESEARCH: VR, Visualization & Analysis of Cultural Heritage Sites; VR and Public History

UR/University of GhanaDigital Archaeology & Structural AnalysisField Research, 2017-present:VR applications

Aims 1 & 2: Self-regulation (executive functioning & autonomic regulation) à social motivation

Aim 3: Self-regulation (autonomic & behavioral reactivity/regulation) during immersive, 360 social filmà social motivation

Measuring: autonomic

reactivity/regulation & head orientation

• Custom-made

video (~6 min)

• Spatial audio

• Progressively

more directive

social overtures

made toward

participant

Self-regulation processes underlying social motivation in ASD: The influence of social context

1. Decision

Making

2. Inhibitory

Control

3. Cognitive

Flexibility

Social Motivation:Executive Functioning Tasks (done with and without social noise):

Collecting HRV & EDA during tasks

Jessica M. Keith| Doctoral Candidate Department: Clinical & Social Sciences in Psychology; Research Lab: Dev. Neuropsychology Lab (PI: Loisa Bennetto, PhD)

October 2018

Benjamin T. Crane| Associate ProfessorDepartment: Otolaryngology, Biomedical Engineering and NeuroscienceFocus: Human visual-vestibular multisensory integration, motion perception, rehabilitation of vestibular lesions

VISUAL-VESTIBULAR INTEGRATION• Common causality between visual

and inertial stimuli• Disambiguation of extern vs. self

motion

VISUAL-VESTIBULAR ADAPTATION• Use of visual-inertial heading offsets

to adapt heading• Exposure to a rotating environment

for heading adaptation

NEW METHODS OF VESTIBULAR REHABILITATION• Develop method for rehabilitation

of heading deviation• Decoding representations of

acoustic space in the cortex• Decoding selective attention in real-

time

RESEARCH TOPICS:

PATHOLOGICAL MOTION PERCEPTION• Rotation perception• Heading offset• Integration with visual motion• Effect of unilateral lesions• Migraine

Acute Chronic

IV

I

V

October 2018

Duje Tadin | ProfessorDepartment: Brain & Cognitive Science, Center for Visual Science, Neuroscience, Ophthalmology

Co-Director, Center for Augmented and Virtual Reality (Neuroscience)

Focus: Visual perception, Multisensory processing, Brain plasticityConflict of interest declaration: Senior Scientist, NeuroTrainer

VISUAL PERCEPTION & COGNITION• Motion perception• Visual attention & awareness• Binocular vision• Visual adaptation• Cognitive assessment in VR

BRAIN PLASTICITY• Perceptual learning• Cognitive training• Brain training in VR• Used in heathy adults, dementia,

stroke, corneal disease, concussion, schizophrenia.

METHODS• Psychophysics• Computational modeling• Eye tracking • Brain stimulation• Neuroimaging

RESEARCH TOPICS:

MULTISENSORY PROCESSING• Audiovisual interactions• Vision & proprioception• Synesthesia