View
432
Download
2
Embed Size (px)
DESCRIPTION
"There’s An App for That" -- but how do we actually develop them? While smartphones and tablets are even getting increasingly more popular and their application scenarios are growing, we still develop them using only a standard integrated development environment. As context-based services and apps do, next to network connectivity, require lots of sensor data, the tools for providing realistic sensor data during development are still immature. Developing, testing, debugging and evaluating those next-generation context-based apps require sensor data for the mobile device -- acceleration, motion, light, sound, camera and many more sensors are available. Though, the existing development tools do seriously limit application developers by not providing the data at all or only on a very limited scale. Especially for indoor environment with applications such as indoor navigation, seamless interaction between public and private displays and activity recognition and monitoring, realistic sensor data are needed and simulation support during the development phase is essential. In this paper, we present our work towards a holistic approach for mobile application development in intelligent environments, leveraging the existing development tool chain, facilitating more effective and realistic means for mobile application development at the example of the Android mobile device platform.
Citation preview
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Towards a Holistic Approach for Mobile Application Development
in Intelligent Environments
Stefan Diewald, Luis Roalter, Andreas Möller, and Matthias Kranz Technische Universität München
Beijing, December 9, 2011
10th International Conference on Mobile and Ubiquitous Multimedia (MUM 2011)
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
General Trends for Mobile Computing
§ “Always connected, personal and contextual” (finally Ubiquitous Computing!!!)
§ “Rapid development”: more and more apps being developed, but less and less time for it
§ “Rapid innovation”: apps are getting more and more complex and having more and more features
à better tools are needed to support better development in less time
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 2
“There’s an app for that.”
ü Source: Appcelerator and IDC: Mobile Developer Report, January 2011 (05.12.2011) ü Market numbers: http://appshopper.com/ and http://www.appbrain.com/stats/number-of-android-apps (08.12.2011)
over 512,000 apps approx. 18 billion D/Ls
over 340,000 apps over 10 billion D/Ls
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Motivation & Contribution
§ Improvement of the technical development process by extended debugging capabilities
§ Improvement of the application development process by extended prototyping capabilities
§ Improvement of the algorithm development process by extended simulation capabilities
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 3
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Outline & Structure § Introductory example: “MagicPhone” (Wu et al., 2010) as an example for a
mobile application for indoor interaction in intelligent environments § Introduction to ROS (http://www.ros.org - not our contribution, just our tool!)
and its key concepts – a robotics middleware, used also for intelligent environments (c.f. Roalter et al., 2011)
§ Current state of the art sensor simulation & tools for mobile application development – focus: sensor-data driven and event-driven applications and context-aware applications in indoor environments
§ Technical details of our approach § Examples for mobile interaction supported by the proposed approach
§ Outlook
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 4
ü Jiahui Wu, Gang Pan, Daqing Zhang, Shijian Li, and Zhaohui Wu. MagicPhone: Pointing & Interacting. In Proceedings of the12th ACM International Conference Adjunct Papers on Ubiquitous Computing (Ubicomp '10). ACM, New York, NY, USA, 451-452.
ü Luis Roalter, Andreas Möller, Stefan Diewald, and Matthias Kranz. Developing Intelligent Environments: A Development Tool Chain for Creation, Testing and Simulation of Smart and Intelligent Environments. In Proceedings of the 7th International Conference on Intelligent Environments (IE '11). pp. 214-221, Nottingham, UK, July 2011
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Modern Applications need Rich Sensor Data
§ Physical ubiquitous interaction § Context-based
Esp. indoor apps are heavily dependent on local and environment sensor data.
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 5
ü Image Source: MagicPhone: Pointing & Interacting: http://www.youtube.com/watch?v=6aVnSmY81KA (06.12.2011) ü Jiahui Wu, Gang Pan, Daqing Zhang, Shijian Li, and Zhaohui Wu. MagicPhone: Pointing & Interacting. In Proceedings of the12th
ACM International Conference Adjunct Papers on Ubiquitous Computing (Ubicomp '10). ACM, New York, NY, USA, 451-452.
§ Indoor localization / navigation § ...
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Testing & Development of Mobile Applications § Context-aware applications depend on
multiple sensors’ data at the same time (e.g. accelerometer, gyroscope, location, …)
§ Current limitations on multi-sensor simulation possibilities require that currently: • (much) testing is carried out on real devices • the tester has to repeat the physical actions
(incl. moving around, pointing, …) over and over again in the real world
§ Often functions are tested separately (e.g. only one sensor input at a time) which does not allow verification of the complete solution
§ Testing in artificial lab environments can only cover a very limited set of scenarios
Virtual Prototyping as potential approach for mobile application development in UbiComp
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 6 ü Image Source: MagicPhone: Pointing & Interacting: http://www.youtube.com/watch?v=6aVnSmY81KA (06.12.2011)
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
ROS as Middleware for Distributed Sensor-Actuator Systems § ROS = Robot Operating System, originates from the robotics domain, but it is
also used for intelligent and ambient assisted living environments (intelligent environment as “ImmoBot”, static distributed sensor-actuator system) http://www.ros.org/wiki/vmi-ros-pkg ß Our complete stacks avail. online
§ Provides hardware and software abstraction, device drivers, message passing, visualization and simulation tools (2D and 3D), (…)
Message Passing system (additionally also a service-oriented comm. system) § Event-based message generation & handling (somewhat similar to Android!) § Basic message types e.g. for strings, integers or floats § Special message types e.g. for images, point clouds or object poses § Publisher / Subscriber model allows fast interconnection of components § All kinds of messages can be recorded and replayed.
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 7 Publisher
Subscriber 1 Publisher Subscriber Subscriber n
Function e.g. add noise, rectify, …
msg
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Realization of Realistic Simulation with ROS (IE 2011 Paper)
Gazebo 3D Simulator (as part of ROS)
§ Goal: (more) realistic 3D environments with full texture and lighting support
§ Environments can be created by using a modified interior design application (‘SweetGazebo’)
§ Objects can be created with common 3D modeling tools such as Blender, Cinema4D or 3ds Max
§ Physics simulation: realistic behavior by defining mass, inertia and friction values for objects
§ Virtual sensors and actuators can be integrated in ROS via plugins and attached to any object in the simulation.
§ ROS connection allows reading out and setting objects’ states 09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 8
ü Luis Roalter, Andreas Möller, Stefan Diewald, and Matthias Kranz. Developing Intelligent Environments: A Development Tool Chain for Creation, Testing and Simulation of Smart and Intelligent Environments. In Proceedings of the 7th International Conference on Intelligent Environments (IE '11). pp. 214-221, Nottingham, UK, July 2011
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Current Situation w.r.t. the Development Support
Development Support for Sensor Data at the Example of Android-based Devices (fastest growing OS):
§ Position: input of GPS data via e.g. the NMEA protocol goal: make it easier to input (esp. for indoor env.)
§ Camera: test image or test video inclusion via a Client/Server solution – but: - How long would the user actually point a device in a certain direction? - How does this additional code influence the app during development, and when is it removed later? (not transparent to the application) à goal: make it part of the simulator/emulator!
§ (Acceleration/Pose/…) Sensor data simulation/generation: artificially generated with an input device with mismatching dimensions (2D mouse vs. 3D orientation), some support on replay/recording of data sets goal: make it more realistic
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 9
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Current Situation w.r.t. the Development Support
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 10
ü Left: Samsung Sensor Simulator, source: http://innovator.samsungmobile.com/image.view.do?serviceId=cmsCnts &attachmentId=11105 (05.12.2011)
ü Right: Live Camera Previews in Android, source: http://www.tomgibara.com/android/camera-source (05.12.2011)
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Current Situation – OpenIntents Sensor Simulator
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 11
ü Image Source: OpenIntents Sensor Simulator: http://code.google.com/p/openintents/wiki/SensorSimulator (05.12.2011)
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Issues and Problems with Current Development Support
§ Physicality of the human as user and of the device in space is often neglected, e.g. movement in an environment, pointing of a device at an object, scanning/touching of a QR or NFC marker, co-location of objects, relation of objects to each other, interrupts during application runtime as standard, not exception, …
§ Imperfection and variability of the real world are not really taken into account: à temporal and spatial instability/variability (of sensor data) “ignored” (“spice” of real world challenges – c.f. impressive UbiChallenge presentations!)
Examples of what is often neglected/not investigated during mobile app development:
• WLAN connection is ‘always bad’ at a given location (that is, the connection breaks, like in the real world, bursty location updates/jumpy positioning/suffering localization accuracy indoors; c.f. Kranz et al, 2010),
• places change frequently (like posters at the walls that influence tracking), …
• people/objects walk in the line of sight of a camera, …
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 12
ü M. Kranz, C. Fisher, A. Schmidt. A Comparative Study of DECT and WLAN Signals for Indoor Localization. 8th IEEE International Conference on Pervasive Computing and Communications (PerCom2010). pp. 235-243, Mannheim, Germany, March 2010
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Room for Improvement in the Development Process
§ Debugging & testing of interactive systems already during development phase (and not at the end) à not really available as of now (c.f. reviewed simulators)
§ Still no (little) realistic statements can be made on usage/performance of apps in real(istic) environments as they are (mostly) not part of the development process! à not really (quickly) testing possible in multiple environments before
deployment/extensive real world testing
§ Complex interactions such as touching objects or grabbing an image of a display can currently not directly be investigated in the development tools à not really available as of now
Challenges: § How to provide (better) data (or at least initial data at all)? § How to more efficiently develop those complex interactive systems (and not only
mobile phone software)? § How to more efficiently prototype multimodal and multimedia systems faster?
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 13
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Proposed Idea: Extended IDE and Development Tool Chain
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 14
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Example: Virtual Camera and Virtual Display Functionality
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 15
§ Linked ROS sensors and Android sensors in the example: camera and accelerometer.
§ Display: Android output that is pushed back to the simulated 3D environment all simulation no longer part of the application, but the emulator/simulator (question: how do we move/steer/control the device in the simulation? à later!)
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Supported Sensors and Actuators
Sensors § Accelerometer* and Gyroscope § Camera* and Depth Image Sensor § Light Sensor § Proximity Sensor § GPS § NFC Reader § Battery Status § … Actuators § Light Emitting Diodes and Displays* § Projectors § Speakers § …
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 16
A virtual Android device in a realistic 3d simulation of an office.
* Currently fully supported by the connected Android emulator, all others are already supported by ROS as well
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Example 1: Indoor Localization and Physical Interaction using NFC tags and QR codes
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 17
Real Simulated § Install changed version of app § Perform test: replay recorded data set
(even faster than real-time), variation can be added automatically, e.g. position offset, …
§ Connect device, install changed version of app, disconnect device
§ Perform test: walk several meters in corridor, take picture or touch NFC tag
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Ex 2: Indoor Localization/Tracking using Visual Information
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 18
Instead of taking various fixed and prerecorded videos, daylight, obstacles, and other properties are simulated in an extended IDE/tool chain. ü PTAM (Parallel Tracking and Mapping), source: http://www.robots.ox.ac.uk/~gk/PTAM/ (06.12.2011)
(markerless, feature-based tracking, no a-priori maps, no inertial measurement unit; test implementation)
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Example – Manual Testing of Mobile Applications in the Virtual Environment
§ 3d input devices like the 3Dconnexion SpaceNavigator or joysticks can be used for 6 degrees-of-freedom controlling
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 19
ü Image Sources: top: 3Dconnexion – SpaceNavigator: http://www.3dconnexion.de/products/spacenavigator.html (06.12.2011), bottom left: http://alwaysinternet.de/wp-content/uploads/2010/12/google-nexus-s-samsung.jpg (06.12.2011), bottom right: http://www.logitech.com (06.12.2011)
+
§ Mobile device’s accelerometer, gyroscope and magnetometer for 3D orientation + classic keyboard for 2D navigation (walking through the virtual scene by holding a real Android device for the pose information and steering by the keyboard for the 2D location info.)
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Example – (human-inspired robotic) Path Planner and Navigation Stack for Automated (Randomized) Testing
§ ROS offers several path planning and navigation stacks (spice for getting slightly different paths on each walk from A to B)
§ Autonomic ROS running robots (like the PR2) with laser scanner and/or depth cameras can be used for generating a map
§ Random paths can be created by imposing different constraints for the route
§ Mobile device could be steered along path e.g. having a fixed pose similar to holding a mobile phone in front user
§ Test how applications can adapt to disturbances, variations, …
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 20
ü Image Source: ROS package - base planner: http://www.ros.org/wiki/base_planner_cu (07.12.2011)
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Example – Testing with different Light Conditions
§ Light conditions can be varied in the 3d simulation.
§ Light direction, color and intensity can be adjusted.
§ Shadows and reflections are supported. § Additional distortion can be applied to the
virtual camera image.
Can be used for testing image/vision-based applications. Example: Checking under which conditions or distances a designed marker is still readable.
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 21
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Summary of the Potentials of the Proposed Approach
§ Multi-sensor simulation in a (more) realistic & easy-to-create 3d environment.
§ Sensor data can either be generated automatically, or by a human moving the mobile device in the intuitive 3D simulated virtual environment.
§ ‘Randomized’ automatic test cases can make use of available components (e.g. route planning algorithms) and data sets (biasing of positions and other sensor values)
§ Recording and replaying of all sensors and states (via ROS messages) allows detailed debugging.
§ Connection to a hardware emulator allows debugging end-user versions of mobile applications (no additional source code necessary, that is, no added code that changes the app’s behavior)
§ Pushing back the mobile’s displays’ content enables realistic usage of mobile applications in the simulated environment
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 22
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Outlook – To Dos for the Future
§ Verification and analysis of the tool chain / approach:
• Survey and test with professional developers is planned
• Compare development time within the scope of our research-oriented teaching: students will develop various indoor localizations systems with and without tool chain support
• Analyze and compare the types and amount of errors made with normal and proposed approach
§ Evaluation of realism of the simulated sensors: metrics need to be defined
§ Exploration of new interaction potentials: fast prototyping of mobile interaction with e.g. robots and intelligent environments (home automation systems), possible due to their ROS support
§ Extension for other (physical) interaction approaches, esp. Tangible User Interfaces
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 23
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Outlook – Further possible Application Areas
§ Public and private displays
§ 2d and 3d touch interfaces
§ Tangible User Interfaces (I like cubes)
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 24
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Outlook – 2D Tabletop Interactions with a Mobile Device
Mixed physical-digital interactions: § Displays: either external or mobile device as pocket table § Physical Objects (e.g. Tangible User Interfaces) § NFC (Near Field Communication) tagged objects
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 25
ü Image Source: reacTable: http://www.reactable.com/ (07.12.2011)
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Questions?
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 26
? ? ?
?
?
Thank you very much for your kind attention!
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz www.vmi.ei.tum.de
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 28
www.vmi.ei.tum.de
Thank you very much for your kind attention!
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Combination of the Real and Simulated World at the Example of Indoor Localization Systems
ROS can be used to connect real and simulated components.
09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 29
Environment
Real Simulated
Mobile Device
Real Real world test, most effort and time consuming
Ground truth, testing of controlled effects
Simulated Real effects, but no working prototpye necessary
Fast evaluation of algorithms, everything controllable
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Thank you for your attention! Questions?
? ? 04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 30
[email protected] www.vmi.ei.tum.de/team/matthias-kranz.html
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Paper Reference
• Please find the associated paper at: https://vmi.lmt.ei.tum.de/publications/2011/MUM11_preprint_v2.pdf
• Please cite this work as follows: • Stefan Diewald, Luis Roalter, Andreas Möller, Matthias Kranz. 2011.
Towards a Holistic Approach for Mobile Application Development in Intelligent Environments. In: Proceedings of the 10th International Conference on Mobile and Ubiquitous Multimedia (MUM 2011), Beijing, China, pp. 73-80, December 2011
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 31
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
If you use BibTex, please use the following entry to cite this work:
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 32
@InProceedings{MobileSimulation11, author = {Stefan Diewald and Luis Roalter and Andreas M{\"o}ller and Matthias Kranz}, title = {{Towards a Holistic Approach for Mobile Application Development in Intelligent Environments}}, booktitle = {Proceedings of the 10th International Conference on Mobile and Ubiquitous Multimedia}, series = {MUM '11}, location = {Beijing, China}, publisher = {ACM}, pages = {73--80}, numpages = {8}, month = dec, year = {2011}, address = {New York, NY, USA}, ISBN = {978-1-4503-1096-3}, url = {http://doi.acm.org/10.1145/2107596.2107604}, doi = {10.1145/2107596.2107604}, acmid = {2107604}, keywords = {Android, ROS, intelligent environment, middleware, mobile devices, simulation, smart phones, virtual environment}, }