1. ICCHP 2012Visual Nouns for Indoor/OutdoorNavigationEdgardo Molina, Dr. Zhigang Zhu, Dr. Yingli TianDept. of Computer Science and Dept. of Electrical EngineeringGrove School of Engineering, The City College of New YorkICCHP 2012 July 11, 20121

2. ICCHP 2012Motivation Assist the visually impaired in navigating new locations Provide guidance to facilities and destinations Restrooms Exits Elevators Water fountains How do the sighted recall a location? Visual cues? 2 3. ICCHP 2012What are visual nouns? Signs Text Icons 3 4. ICCHP 2012Using Visual Nouns Provide contextual information What is it telling us? Provide landmark markers Have I been here before? Camera pose estimation => User Orientation 4 5. ICCHP 2012Technology Alternatives RFID Embedded info Costly QR Codes / Bar Codes / Etc Embedded info Only meaningful to machines GPS Outdoors only Signs and Text are already in our environment5 6. ICCHP 2012Standardized Signage US DOT and AIGA signs Various Organizations have standard signage6 7. ICCHP 2012Visual Noun Project http://www.thenounproject.com 7 8. ICCHP 2012Localizing a user1. Capture video of surroundings2. Register video image frames to the first frame (will serve as reference)3. Generate a wide field-of-view panorama4. Extract and match visual-noun features5. Localize the user in 3D space relative to the visual nouns. 8 9. ICCHP 2012Framework 9 10. ICCHP 2012Input Video10 11. ICCHP 2012Panorama 11 12. ICCHP 2012Visual Noun Extraction Maximally Stable Extremal Regions (MSER) Performed on a sharpened image Use of EE-MSER could be better [Chen et. al. - ICIP11]12 13. ICCHP 2012Localization Localization done using PnP algorithm Compared number of features in a single view vs. apanorama 13 14. ICCHP 2012Localization Results 14 15. ICCHP 2012Summary We propose to use the signage thats already in ourenvironment to provide orientation and contextinformation to visually impaired users. Low cost to print out more signage Also beneficial to sighted users15 16. ICCHP 2012Thank you!Questions?Work is supported by US NSF Emerging Frontiers in Research and InnovationProgram under Award No. EFRI-1137172, and City SEEDs: City College 2011President Grant for Interdisciplinary Scientific Research Collaborations.16