ERO Technology Group

Programming RobotsProgramming with Visualisation, Concurrency and Coordination

Presented byAndreas Heilv-aheil@microsoft.com Date08. November 2006

Definition: Robot

A robot is an electro-mechanical device that can perform autonomous or preprogrammed tasks. A robot may act under the direct control of a human (eg. the Canadarm on the space shuttle) or autonomously under the control of a programmed computer.

Wikipedia1. Mechanical device programmed to perform tasks

any machine that can be programmed to carry out instructions and perform particular duties, especially one that can take over tasks normally done by people.

2. Imaginary machine like humana machine that resembles a human in appearance and can function like a human, especially in science fiction

3. Person like a machinesomebody who works or behaves mechanically, showing little or no emotion and often responding to orders without question

4. Traffic lightSouth Africa a set of automatic traffic light

Encarta Dictionary

Definition: Robot

A robot is a device, hard- or software with the capability of sensing and (re)acting.

Robotics Market PotentialService and consumer markets just emerging

• Remote assistance/presence• Assistive• Facilities maintenance• Security• Education• Entertainment

More applications are required to improve the market

• A lack of reusability keeps robot developers endlessly re-solving the same software problems

• Concentrating on the mechanics of robotics rather than the science of robotics

• Concurrency and complexity plague all software engineering and especially robotics

• Development requires too much effort and knowledge

0

10000000

20000000

30000000

40000000

50000000

60000000

70000000

Mar

ket

Siz

e $1

,000

s

1995 2000 2005 2010 2025

Year

Worldwide Robotics Market Growth

Home

Mediacal/Welfare

Public Sector

Bio-Industrial

Manufacturing

* Source Japan Robotics Association

Robotics Market Potential

0

10000000

20000000

30000000

40000000

50000000

60000000

70000000

Mar

ket

Siz

e $1

,000

s

1995 2000 2005 2010 2025

Year

Worldwide Robotics Market Growth

Home

Mediacal/Welfare

Public Sector

Bio-Industrial

Manufacturing

* Source Japan Robotics Association

Development Trend

2005

Rea

l vac

uu

m c

lean

er

2007 2010

2015 2020

Ro

bo

t w

ith

inte

gra

ted

Co

mm

un

icat

ion

Cap

abili

ties

1

Ro

bo

t lo

adin

g D

ish

was

her

2

To

urg

uid

e3

Ro

bo

t ad

apti

ng

to

ind

ivid

ual

Nee

ds

1 Filtering unnecessary Information2 Household Devices (Microwave, Fridge)3 Learning about new Environments by giving a Tour to it, after that it can re-give that tour.

Ro

bo

t h

elp

ing

han

dic

app

ed P

eop

le

Co

mp

anio

n f

or

eld

erly

Peo

ple

Gam

ing

Ro

bo

ts

Iro

nin

g R

ob

ot

Pro

acti

ve R

ob

ots

Ro

bo

ts u

nd

erst

and

ing

hu

man

Act

ivit

ies

24h

Ass

ista

nt

Increasing Software Challenges

* Source EURON Workshop 2005

Microsoft & Robotics

Systems

• PC Ecosystem (desktop, web, mobile, home)

• XP, XP Embedded, CE (real-time)

Development environment and tools

• Programming IDE• Debugging and Opmization

Existing applicable Technologies

• Speech SDK• Real-time Communications SDK• RFID SDK

Experience Groups

Microsoft Research

• Enabling Technologies• Human Robot Interaction• Personal Robotics• Educational Robotics• Programming Environments

Microsoft Robotics Groups

• Microsoft Robotics GroupRedmond

• Center for Innovative RoboticsCarnegie Mellon University

• External Research OfficeCambridge

New Requirements

• Ultra Heterogeneity• Distributed environments• Dynamic Configuration• Context-Awareness• Personalization• Extensibility• Reliability• Security• Privacy protection• Usability• Autonomy• …

Ubiquitous computing environments should be deployed incrementally

• Living space is not a demonstration room• We like to replace existing objects

How can software infrastructures help to decrease the complexities?

New Requirements

Challenges to Address

Future Applications• Support & Care, Education, Entertainment

Personal Robotics• Human-robotics Interaction• Mediator between digital/real world• Natural and affective interaction (speech,

gestures, emotions)

New programming paradigms• Body inspired software architecture (Pisa)• Meta-programming models (Berlin, Stuttgart)

From nature to software models

What is needed that robots could be successfully integrated in our everyday life? What are key technical issues?

A typical first Approach

• Build a (another) middleware

• Abstract from the underlying hardware (HAL)> i.e. classes for sensors, actuators, and communication

• Unique programming model > Can use every concept the

runtime provides

• Use the .NET CLR and APICLR

Robot API

Control Application

ControllerSensors and Actuators

This wont always work!

Things that work fine on a PC don’t scale down

• Thread switches on a smart-phone or PDA are costly

• Some hardware controller don’t support threads at all

• Hardware not capable enough for a middleware (CPU, memory, battery, …)

• Even though PDAs/Smartphones are fairly powerful, CF has some several restrictions (good for some reasons, but not for us)

Now think of even smaller devices

• Embedded controllers, wrist-watches, sensor networks, smart dust

VRDK – A Research Prototype

• Building a compelling & engaging programmable environment

• Innovative, very easy, flexible programming environment for Robot control applications

• Accessible to non technical audience (children, nurse, elderly, machine operator)

• Enables a ‘Path’ from very simple (beginner, child) to professional (using

Visual Studio)

Abstracting complexity on different levels

Source Code Visual

Model-driven Development Approach

Platform-independent Model of the Program(VRDK Language)

Same programming modelfor different devices

Transformation Transformation

GenerationGeneration

.NET .NET CF XYZPlatform-specific realization of the program

Executable Executable ExecutableExecutable or interpreteron target platform

Full Tool Chain

Deploy and Run

Deploy and Run

The gentle Shortcut

VRDK Editor

Code Generation

VS.NET

Target Devices

Demo

Domain Specific Language

Visual N#

• Domain-specific graphical language• Extension of VRDK• Easy to design

C# / VB.NET

• General purpose programming language• Don’t write code anymore

N#

• Domain-specific textual language• Easy to read and write

Users can switch betweenboth notations

Code Translator

N# - A Textual Notation for Visual N#

ambient MyAmbient @ Personwhere filter ($1.Company == „Microsoft Research“),

filter($1.Location == „Cambridge"), filter($1.SecurityLevel > 2) {

discover Lights @ Light where distance(a), filter($1.Color == „green");

process OnLampAdd @ l = Lights.Added {l.On();

}process OnLampRemove @ l = Lights.Removed {

l.Off();}

a = 100;}

Demo

The Microsoft Robotics Studio

A lightweight concurrency and services oriented runtime• Handling of sensory input and controlling actuators• Based on synchronous message passing• Decentralized System Services (DSS) facilitating tasks and

basic services such as debugging, logging, monitoring, security, discovery, and data persistence

Authoring/development tools• Visual programming editor• Simulation• Message debugging

Technology libraries and basic algorithms• Code samples and documentation

Open Platform

Technology librariesRobot models Device services(OEM, ISV, IRV)

Applications and tools

Runtime Environment

Robotic ApplicationRobotic Application

RuntimeRuntime

Authoring Tools

Visual Studio Visual Studio

Supported manufacturers

• Robotics Connection• Mobile Robots• Coroware• KUKA• LEGO• fischertechnik• Robosoft• WhiteBox Robotics• Lynxmotions• Phidgets• iRobot Roomba • …

• Community platforms

Development Model

Application model

• Build complex systems from smaller, simpler decentralized services

Distribution model

• Applications are a collection of distributed services

• Discovery of functionality via contract and categories

• Self-organizing • Interaction via Messaging

Programming model

• Coordination of messages • No manual creation of threads, locks,

semaphores

ProgrammingProgrammingModelModel

DistributionDistributionModelModel

Application ModelApplication Model

Robotics Studio Architecture

Orchestration ApplicationOrchestration Application

Decentralized System Services

Con

cu

rren

cy

Con

cu

rren

cy

an

d

an

d

Coord

inati

on

C

oord

inati

on

R

un

tim

eR

un

tim

e ActivationActivation DiscoveryDiscovery

DiagnosticDiagnosticss

TerminalTerminal

StorageStorage

UXUX

Robotic Services

Robot Robot Model Model

Vis/Sim Vis/Sim

Services

Device 1Device 1

Device 2Device 2

……

Device Services

VisionVision

SLAM SLAM

……

Library Services

Messaging TransportMessaging Transport

Signal Signal Processing Processing

Hardware Abstraction LayerHardware Abstraction Layer

HA

LS

PL

Serv

ice

sO

rch

estr

ati

on

Layered and Distributed Application

Behavior Control LayerBehavior Control Layer

Orchestration Layer Orchestration Layer

Services Layer Services Layer

Signal Processing Layer Signal Processing Layer

Hardware Abstraction Hardware Abstraction Layer Layer

“Learns” and begins to favors specific behavioral characteristics. Interacts with the orchestration layer

to achieve the favored patterns.

Defines and facilitates communication patterns that coordinate information processing amongst lower

level services.

Provides functionality abstraction, computational encapsulation, failure isolation, distributed and concurrent execution, via software services .

Executes algorithms that require near real-time computation and deterministic time control.

Directly interfaces with the physical robotics hardware - Sensors and Actuators.

Service Composition

• Services aren’t just for devices

• Services can be compose to form other services

• Services can represent passive or non-existent devices

> Fused sensory data can be re-exposed as an independent service

MotorMotor EncoderEncoder

DriveDrive

Steering Steering ServoServo

TrajectoryTrajectory

PosePose

WheelWheel Base GeometryBase Geometry

Rob

ot

Mod

el

Serv

ices

Robot Model ServiceRobot Model Service

Runtime Environment

An application is a composition of loosely-coupled services concurrently executing across scopes • Interactions described using service contacts• Components are networked• Provides isolation between components

> Reliability – ability to restart independently, replaceable/updatable> Parallelism

Application Boundary

Service

Main Port

Main Port

Service

State

Code

Private Port

State

Code

Service

Main Port

State

Code

Service

Main Port

Private Port

State

Code

H/WH/W

UI

Service

Main Port

Private Port

State

Code

* Example: Model car + sensors

Communication & Coordination

• Typed Ports

• Messages sent to ports

• Arbiter handle messages> Persistent> Batch> Choice> Join> …

Activate ( Arbiter.CreateReceiver(false, p, delegate(int i)

{ Console.WriteLine(i) ;

}) );

Port<int> p = new Port<int>();p.post(42) ;

Runtime Reliability

• Reliability> Reservations> Guarantueed CPU bandwith

• Number of fixed threads in dispatcher> Set during creation of dispatcher> Not using CLR threadpool> Less overhead

• No Priorities for threads> Hard to persuade developer not to use priorities> So, priorities introduced for dispatcher

• FIFO Ports> 90,000 SOAP messages/s within a node (=multiple services)> > 3,000 SOAP messages/s among nodes

Main Port

Service

State

Code

Private Port

Runtime Environment

Concurrency and Concurrency and Coordination Coordination

RuntimeRuntime

Decentralized Decentralized System ServicesSystem Services

Service Development vs. Application Development

• Done only once • Driver related• Done by manufacturer• Distribute & reuse

Authoring Environment

Visual StudioVisual Studio

Robot Package

Tool B

ox

Tool B

ox

Robot Model Designer Robot Model Designer Code GeneratorCode Generator

Application Designer Application Designer

Service Contract Service Contract DesignerDesigner

Debugging Debugging servicesservices

Packag

es

IDE

Deployment Designer Deployment Designer Code DownloaderCode Downloader

Visualization and Visualization and SimulationSimulation

Standalone & Visual Studio integrated

Scales with programmer skill

• Helps beginners with drag/drop programming

• Assists advanced developer with service orchestration

• Conceptually consistent with conventional programming languages

• Context based toolbox

Support Rapid Application Development via visual programming, simulation and visualization.

Supports programming of distributed scenarios Extensible to support other hardware

Visual Programming Language

Robots as first class objectsData flow and orchestration

• Easy for novices

• Rapid prototyping for experts

• Implicit parallelism

• Generates C# code

• Formal checking of protocol interactions

• Deadlock detection, state exploration

ActivityMessage link

Activity

Demo

Simulation Capabilities

Supported KUKA robot

Community: Robot Swarm Heading Target Area

Community: Robot Swarm Simulation

Community: Maze Simulator

Outlook

• VRDK> Research prototype – not developed anymore

• Microsoft Robotics Studio> Available as CTP (November CTP, today at 5.00 am)> Improved visualisation> Package based deployment> Wrap up for final version> CCR / DSS maybe available as separate DL> http://msdn.microsoft.com/robotics/

• N# > Ongoing research> Publications etc. > Visual Studio integration in progress

Thank you or your attention.