The Convergence of Robotics, the Web, and the IoT

The Convergence of Robotics, the Web, and the Internet of Things

Michael McCool, Principal Engineer, Intel

IoT Dev Fest, June 4, 2017

http://inteliotfest.withthebest.com/

Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice

AbstractWhat do the web, the IoT, and robotics have in common? For one, they can all benefit from the network effect, where the whole is greater than the sum of the parts. For the Web, the network allows distribution of information between agents anywhere in the world. For the IoT, the focus is on the remote monitoring and coordination of physical sensors and actuators, and the analysis of data to provide actionable insight. Robots can be defined as autonomous cyberphysical systems that can sense and manipulate their environment. Although not essential to the definition, robots are often connected to the internet as well, and can use it to enhance or enable their function in various ways: supporting, for example, telepresence on the web, remote sensing integrated with the IoT, or coordination between multiple robots and IoT devices, or just to offload computation. In the past these three ecosystems had their own software frameworks and standards but there are indications they are converging: IoT systems often integrate with cloud services that have web APIs. Web API standards are being used to define the data models for IoT devices. Machine learning models can be trained with data from the Web and used for real-time perception in robots. IoT gateways can be used to command robots by voice. Robots can serve their own graphical user interfaces over the Web. Robots are adopting service-oriented distributed software architectures and programming models originally developed for web servers. IoT devices and radio networks can be used a navigation beacons and even raw sensory data for robots. In this talk, I will talk about a number of these convergences and what the implications are.

2


Outline and Agenda

• Ecosystem Overview• Web, IoT, and Robotics

• Requirements and technologies

• Commonalities and differences

• Trends and Technologies• Embedded HPC and AI

• Remote compute offload

• Real-time (sub)systems

• Semantic data modelling

• Emerging Standards• OSRF*/ROS2* and OMG*/DDS*

• OCF* and W3C* Web of Things

• OpenFog*3

Web

IoT Robotics


The World-Wide Web

• “Distributed information space with resources identified by URLs”

• Hyperlinked documents

• Delivered by Internet

• Displayed in browsers

• Many, many extensions…

• Search engines

• Web service RESTful APIs

• Applications as content

• Semantic markup

4https://en.wikipedia.org/wiki/World_Wide_Web

This Wikipedia and Wikimedia Commons image is from the user Chris 73 and is freely available at //commons.wikimedia.org/wiki/File:WorldWideWebAroundWikipedia.png under the creative commons cc-by-sa 3.0 license.

https://en.wikipedia.org/wiki/World_Wide_Web


Technologies Engendered by the Web

• Web Services

• RESTful: Set of properties that allows an online service to scale gracefully

• Stateless interface, small set of verbs, structured data

• Metadata: RAML, Swagger/OpenAPI

• Cloud Services: X-aaS, SD-X, virtualization…

• “Software applications as content”

• Containers and Microservices

• Docker, Snaps, etc.

• Compose services out of loose collections of other services

• Flexible structured data formats (and now efficient binary encodings…)

• JSON CBOR; metadata: JSON-Schema

• XML EXI; metadata: XSI

5


Semantic WebTo enable Web 3.0:

Allow the meaning (semantics) of data to be understood

Technologies to facilitate data interchange:

RDF: Resource Description Framework

OWL: Web Ontology Language

SPARQL: Protocol and RDF Query Language

Syntaxes: XML, N3, JSON-LD, etc.

Ontologies (Vocabularies): Haystack, Schema.org, etc.

6/37


RDF: Resource Description Framework

7/37

General approach for describing data as sets of “triples”

Subject-Predicate-Object

RDF triples are stored in a distributed fashion

Federated database

Subsumes relational databases

Standard terms and relations are defined in ontologies (“vocabularies”), including RDF and OWL

Socrates

ManIs-a

Mortal

IsIs


Internet of ThingsNetwork of devices, each capable of communication and associated with a set of sensors and/or actuators.

• Enable numerous applications

• Multiple verticals: home, industrial, city, building, vehicle, etc.

• Typically rely on backend systems for data analytics and services (eg web services)

• Edge + “infrastructure” nodes…8

https://commons.wikimedia.org/wiki/File:Nest_Learning_Thermostat_(cropped).JPG

https://commons.wikimedia.org/wiki/File:Nest_Learning_Thermostat_(cropped).JPG

Optimization Notice

Internet of Things (IoT) Systems

Gateway

Gateway

Analysis

(Server)

Human

Interface

Wi-Fi*

LTE

Wired

Endpoint

Endpoint


Robots

• Cyberphysical systems with the capability to autonomously interact with their environments

• Ideally capable of autonomous

• Perception

• Planning

• Mobility

• Manipulation

• Human-Robot Communication

• Note: typical “Industrial Robots” may only be capable of a subset of these

10


Typical Robot Software Stack

ServosWheel

Motors

RGB

Camera

Depth

CameraMotion Touch Mic Array

Discrete SensorsCamera Drivers Motor Drivers

Se

nso

r Fusio

n

Point Cloud

2DMap

Body Tracking

Face Tracking

Object Recognition

RGBD+IMU / SLAM

Speech RecognitionBehaviorControl

Reflex

Goals

Interact

Speech

OutputDialog

Control

User/ World

Knowledge

Gesture /

Action

World

View

Robot

State

Command

Inputs

Behavior Arbitrator

Path

Planning

Movement

External

Device

Control

Voice Command

External Controller

Gesture InputEnvironment

StimulusBehavior

BehaviorBehaviors

Applications

Acoustics

Op

tion

al In

pu

ts

Cloudoffload &control

IRUltra-

Sonic

Sense Plan Act

Sensor Hardware and Drivers Actuators

• Sense a potentially dynamic environment

• Determine current location

• Plan path and then navigate from one location to another

• Avoid obstacles

• Interact with humans as needed (avoid, follow, respond to commands, etc.)

Mobile Robotics

12

• Speech recognition

• Face recognition

• Body pose understanding

• Object recognition

• Dialogue management

• Context awareness

• Emotional state recognition

• Social relationship understanding

Social Robotics

13

Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice14


Robotics and the IoT

Robots are…

Cyberphysical systems with the capability to autonomously interact with their environments

Not just passive sensors

If connected to the internet, robots may also participate in the IoT ecosystem

Share experiences via the network

Use sensor data from other IoT devices

Offload work to external systems

Access large amounts of data for training data

Coordinate motion across multiple robots/devices

Next-generation connected robots are explicitly leveraging IoT technologies and middleware

But they also need to be autonomous15


Comparative Requirements

16

Web IoT Robots

Scalability

Connectivity

ScalabilityLow costLow powerFunctional safetyConnectivity

Functional safety

High performanceReal time

ReliabilityAvailabilityFlexibilitySecurity




Emerging Trends and Technologies

Artificial Intelligence

• Machine Learning

• Perception

• Semantic Processing

Embedded HPC (especially for Robotics)

• Computer Vision and Speech

• Localization and Mapping

Distributed computing

• Remote (cloud, fog) compute offload

• Service-oriented computing

Real-time (sub)systems

• Control systems

• Functional safety

17


Case Studies

ROS*: Robot Operating System

• Microservices architecture, Web integration, ROS2, OMG’s DDS, embedded perception

OCF*: IoT standardization effort

• Microservices architecture using CoAP/CBOR, IoTivityimplementation of Smart Home example system

W3C* Web of Things: IoT standardization effort

• Standardized metadata for microservices architecture

OpenFog*: Edge computing standardization effort

• Infrastructure support for microservices architecture

18


ROS*: Robot Operating System

• Supports distributed system based on a graph of communicating nodes

• Both pub-sub and request-response patterns

• Large community with many existing nodes for sensors, planning, navigation, etc.

• Various visualization tools and simulators are also provided


Autonomous Navigation ROS* Node Graph


Rviz: Robot Visualization


Robot Web Tools* and ROSWeb*

http://robotwebtools.org/ and https://github.com/EESC-LabRoM/rosweb22

ROSWeb Robot Manager

• Manage/control ROS robot via web console

• Uses rosbridge, which bridges ROS to a JSON-over-Websocketprotocol

• Javascript library can then access ROS functionality either from Node.js or a browser

• Can also use WebGL, TLS, D3S, Google maps, etc.

http://robotwebtools.org/

https://github.com/EESC-LabRoM/rosweb


ROS2*

Designed to resolve a number of issues with current ROS architecture:

• Real-time support

• Quality of service control in communications

• Distributed node management

• Lifecycle management

• Determinism

Built on top of OMG’s DDS standards, a middleware framework for industrial IoT systems, that provides real-time pub-sub

https://github.com/ros2/ros2/wiki and http://www.omg.org/spec/DDS/

https://github.com/ros2/ros2/wiki

http://www.omg.org/spec/DDS/


SAWR Project

Simple Autonomous Wheeled Robot

• Demonstration system for autonomous

robotics

• Can be built using either

• Intel® RealSense™ Robotic Development Kit,

which combines Aaeon’s UP* Board and an

Intel® RealSense R200 camera

• Intel® Joule™ compute module and an Intel®

RealSense ZR300 camera

• OSH design: laser-cut frame/BOM

https://github.com/01org/sawr

https://github.com/01org/sawr


Robotis* Turtlebot 3

25

Modular Robotics Research

and Training Platform

• Intel® Joule™ compute

module

• Intel® RealSense R200

camera

• Laser triangulation 360

degree rangefinder

• Real-time Arduino-based

subsystem

http://turtlebot3.robotis.com/en/latest/hardware.html

http://turtlebot3.robotis.com/en/latest/hardware.html

Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization NoticeIntel Confidential

Intel® RealSense™ SDK: Perceptual Capabilities

Objects Database

Library of pre-

trained

or customer

specific objects

Object Tracking

Track objects of

interest, even when

robot and/or object

are moving

Object Localization

Determine

where objects

are in the scene

Object Segmentation Pixels that are part of

the object for

manipulation or

removal



Person Interaction


OCF* & IoTivity*

Specification

Open Source Coordination

Business (Marketing, Strategy, Membership)

Open Source ProjectReference Implementation of OCF

(and Non-Spec Related Code)

Sponsored (funded) by OCF

Separate Governance

Coordination

April 2017

Open Connectivity Foundation Public Information

29

Certification

Data Modelling

Specs & Open Source ready simultaneously


OCF – Conceptual Framework

30 April 2017

Core Framework

ProfilesConsumer Enterprise Industrial Auto Education Health

Security, Identity & Permissions

DiscoveryData

Transmission

Data

ManagementDevice

Management

Transports(Smart)

RemoteAccess Cloud

Resource Model

Interaction / Data Model

Open Connectivit

y Foundation

Public Information


OCF – Resource Model

31 April 2017

Resource URI

rt: Resource Type

if: Resource Interface

prop: Policy

n: Resource Name

…

…

CommonProperties

ResourceSpecific

Identifies the type of resource

List of interfaces associated with the resource

Policy associated with resource: discoverable, observable, secure,

etc

Friendly name

§ The resource model, coupled with the common data model, provides the base interoperability of OCF

§ Any physical or software artifact on a device that needs to be manipulated or made visible across the network can be described via the resource model

§ A resource has a URI and a collection of Properties

links: Other Resource URI Resources can include links to

other resources

Open Connectivit

y Foundation

Public Information


Resource Model - examples

32 April 2017

14

Resource URI :/a/light1

rt: oic.ex.light

if: oic.if.rw

prop: discoverable,

observable

n: myHallWayLight

State: 0 (OFF)

Dim Level: 0

Resource URI :/a/fan1

rt: oic.ex.fan

if: oic.if.rw

prop: discoverable

n: myKitchenFan

State: 1 (ON)

Speed: 10

Open Connectivit

y Foundation

Public Information


CREATE Create a new Resource on the Server

RETRIEVE Get the current State or representation of a Resource from a Server

UPDATE Request a partial or complete update to the information stored in a Resource

DELETE Remove a Resource from the Server

NOTIFY Request asynchronous notification of state changes in a Resource

Special-cased to support OBSERVE for publish/subscribe support

Common Interaction Model – Transport Agnostic

April 201733

Open Connectivity Foundation

Public Inform


Let’s look deeper at a Resource …

34

Server - DevA

/dimming

/oic/d

/oic/res

Main Living Room Light

/switch

/oic/d

/oic/p

oic.r.switch.binary

oic.wk.d

oic.wk.p

false100

/switch true

/dimming oic.r.light.brightness

/oic/p LED-Light-Co, 40W Equivalent Bulb


There can be more Properties …

35

/oic/pmnml

mnmn

pi

http://ledlight.com/gg3-40

Led-Light-Co

GoGlow III - 40W

mnmo GG3-40


Resource Type - Data Modeling

36

JSON Schema*

• Describes the Properties and

Representation

ReSTful API Modelling Language

(RAML)*

• Describes the Request and Response

*JSON Schema and RAML are normative and also can be used for code generation

"oic.r.switch.binary": {

"type": "object",

"properties": {

"value": {

"type": "boolean",

"description": "Status of the switch"

}

}

}

get:

responses :

200:

body:

application/json:

schema: |

{

}

Defined by:

(see OIC_Resource_Type_Specification)


OCF/IoTivity Smart Home Demo

https://github.com/01org/SmartHome-Demo37

Prototype Smart Home

• Example architecture for Smart Home

• Based on OSS IoTivityimplementation of OCF

• Combines cloud services, gateways, and sensor nodes

• Technologies: Docker containers, Node.js, Zephyr RTOS, HTTPS, CoAP, WebGL, RAML, JSON/CBOR

https://github.com/01org/SmartHome-Demo


W3C Web of Things

https://www.w3.org/WoT/ and https://github.com/w3c/wot38

Thing

API A

Applications

Thing

API B

Applications

Interaction Model Interaction Model

Protocol Mappings Protocol Mappings

CoAPOneM2M

CoAP

OCFCoAP

BACnetBridging Gateway

Thing

Description

HTTP

W3C Working Group

• Standardize descriptions of data and interaction models for IoT devices (“Things”)

• Support semantic interoperability via linked data, JSON-LD, RDF, IoT ontologies

• Bridge multiple other standards

https://www.w3.org/WoT/

https://github.com/w3c/wot/


https://www.openfogconsortium.org/39

Description

• Reference architecture for distributed “fog” computation

• Fog: cloud computing, but supported throughout the network, both in data centers and at the edge

https://www.openfogconsortium.org/


Summary and Conclusions

• Web, IoT, and Robotics ecosystems to interoperate, but not merge

• Several key differences mean completely shared technology base is not feasible

• However, many core components can and should be shared

• Many key “IoT” services will depend on these systems interoperating

• New opportunities and challenges

• Moving cloud services to the edge:

• Embedded microservice architectures, fog computing, containerization

• Using web technologies and infrastructure in IoT and Robotics:

• RESTful APIs, cloud services, semantic interoperability

• Software development challenges:

• Data-centric, distributed computing, microservices, scaling, security

40


Legal Disclaimer & Optimization Notice

INFORMATION IN THIS DOCUMENT IS PROVIDED “AS IS”. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO THIS INFORMATION INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

Copyright © 2014, Intel Corporation. All rights reserved. Intel, Pentium, Xeon, Xeon Phi, Core, VTune, Cilk, and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries.

Optimization Notice

Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804

41

Technology

The Convergence of Robotics, the Web, and the IoT