Prof. A. Taleb-Bendiab, talk: Whole Body Interaction Workshop07, Date: 03/06/2014, Slide: 1 Bio-Cybernetic Control for WBI: a birds eye view Prof. A. Taleb-Bendiab

Prof. A. Taleb-Bendiab, talk: Whole Body Interaction Workshop’07, Date: Prof. A. Taleb-Bendiab, talk: Whole Body Interaction Workshop’07, Date: 10/04/2310/04/23, Slide: , Slide: 11

Bio-Cybernetic Control for Bio-Cybernetic Control for WBIWBI: :

a bird’s eye viewa bird’s eye view

Prof. A. Taleb-BendiabProf. A. Taleb-BendiabSchool of ComputingSchool of Computing

Liverpool John Moores UniversityLiverpool John Moores Universityemail: email: [email protected]@livjm.ac.ukhttp://www.cms.livjm.ac.uk/talebhttp://www.cms.livjm.ac.uk/taleb

Bio-Cybernetic Control for Bio-Cybernetic Control for WBIWBI: :

a bird’s eye viewa bird’s eye view

Prof. A. Taleb-BendiabProf. A. Taleb-BendiabSchool of ComputingSchool of Computing

Liverpool John Moores UniversityLiverpool John Moores Universityemail: email: [email protected]@livjm.ac.ukhttp://www.cms.livjm.ac.uk/talebhttp://www.cms.livjm.ac.uk/taleb

mailto:[email protected]

mailto:[email protected]

Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 10/04/2310/04/23, Slide: , Slide: 22

OutlineOutline In the context of the Whole Body Interaction (WBI)In the context of the Whole Body Interaction (WBI)

DefinitionsDefinitions Biocybernetics, control, Augmented Reality, WBIBiocybernetics, control, Augmented Reality, WBI

Drivers for WBIDrivers for WBI Overview of emerging requirements for WBI, Overview of emerging requirements for WBI,

Technical challenges, requirements Technical challenges, requirements State-of-the-art relevant to bio-cybernetics control for State-of-the-art relevant to bio-cybernetics control for

WBI including: WBI including: Bio and socio-inspired computational modelsBio and socio-inspired computational models

ambient intelligence, ambient intelligence, Autonomic ComputingAutonomic Computing MiddlewareMiddleware body area sensor and actuator networks, body area sensor and actuator networks, cognitive systems cognitive systems

Underpinning technologies, Underpinning technologies, Conceptual frameworks for bio-cybernetics controlConceptual frameworks for bio-cybernetics control ConclusionsConclusions

Open questions research challenges relevant to WBIOpen questions research challenges relevant to WBI Q&AQ&A


Some Definitions #1Some Definitions #1 Biological Cybernetics (Bio-Cybernetics)Biological Cybernetics (Bio-Cybernetics)

… … is the application of cybernetics theory and is the application of cybernetics theory and systemicssystemics principles to further our understanding principles to further our understanding of systems biologyof systems biology

integrate different levels of information and information integrate different levels of information and information processing to understand biological systems processing to understand biological systems

function and development of cellular, multicellular systems, function and development of cellular, multicellular systems, organisms, complex living systems and ecosystems. organisms, complex living systems and ecosystems.

[[http://www.biological-cybernetics.de/wiki/index.php/Main_Pagehttp://www.biological-cybernetics.de/wiki/index.php/Main_Page]] [Source: Wikipedia[Source: Wikipedia]]

Cybernetics Cybernetics the study of control and communication in the the study of control and communication in the

animal and the machine including: animal and the machine including: organization, information flows, controlorganization, information flows, control

http://en.wikipedia.org/wiki/Systemics

http://www.biological-cybernetics.de/wiki/index.php/Main_Page


Some Definitions #2Some Definitions #2 Augmented RealityAugmented Reality: :

……. combination of real world and computer . combination of real world and computer generated datagenerated data

uses of live video imagery which is digitally uses of live video imagery which is digitally processed and "augmented" by processed and "augmented" by

the addition of computer generated graphics. the addition of computer generated graphics. Advanced research includes the use of motion tracking Advanced research includes the use of motion tracking

data, etc.., data, etc.., and the construction of controlled environments and the construction of controlled environments

containing any number of sensors and actuators.containing any number of sensors and actuators. [http://en.wikipedia.org/wiki/Augmented_reality][http://en.wikipedia.org/wiki/Augmented_reality]

Mixed reality:Mixed reality: refers to the merging of real and refers to the merging of real and virtual worldsvirtual worlds to produce new to produce new

environments and visualisations where physical and digital environments and visualisations where physical and digital objects co-exist and interact in real time.objects co-exist and interact in real time.

[http://en.wikipedia.org/wiki/Mixed_reality][http://en.wikipedia.org/wiki/Mixed_reality] Virtual Worlds:Virtual Worlds:

“… “… simulated environment intended for its users to inhabit simulated environment intended for its users to inhabit and interact via avatars …” and interact via avatars …”

[http://en.wikipedia.org/wiki/virtual_world][http://en.wikipedia.org/wiki/virtual_world]

http://en.wikipedia.org/wiki/Virtual_world


Socially Augmented Intelligent PICTSocially Augmented Intelligent PICT

HospitalNetwork General

PractitionerNetwork

Home Network

GymnasiumNetwork

Mobile DistrictNurse Network

Real-Time Communications




HospitalNetwork General

PractitionerNetwork

Home Network

GymnasiumNetwork

Mobile DistrictNurse Network






Convergence of Real and Virtual Convergence of Real and Virtual Worlds: Worlds: Second LifeSecond Life

Second LifeSecond Life Virtual WorldsVirtual Worlds hand-held and wearable hand-held and wearable

systems that act as systems that act as gateways between the gateways between the real and virtual worldsreal and virtual worlds

Eyewear, display badges Eyewear, display badges and speakers worn about and speakers worn about the neck will allow us to the neck will allow us to live more fully through live more fully through our avatarsour avatars

a wearable box that a wearable box that creates a 3D sound field creates a 3D sound field that allows the wearer to that allows the wearer to hear voices from the hear voices from the virtual world without virtual world without completely shutting out completely shutting out the real people around the real people around himhim


Vodafone in Second LifeVodafone in Second Life

More More recently recently Vodafone Vodafone now allows now allows you to you to interact with interact with your real your real phone via a phone via a Second Life Second Life avatar of the avatar of the phonephone

The real phone can be The real phone can be answered from within SLanswered from within SL

Text messages can be sent and Text messages can be sent and received between real-world received between real-world and SLand SL


Sensors that Control Second Sensors that Control Second Life ObjectsLife Objects

Sending external data into virtual worlds such as Sending external data into virtual worlds such as Second LifeSecond Life

Using Potentiometer the cube can be controlledUsing Potentiometer the cube can be controlled


Importance of WBIImportance of WBI WBIWBI

InterfaceInterface Transparency and virtualisationTransparency and virtualisation

Systems and Processes Systems and Processes InteractionInteraction Behaviour Behaviour Structure/organisationStructure/organisation

AffectiveAffective Sensing and Actuation Sensing and Actuation

Co-evolveCo-evolve Technical RequirementsTechnical Requirements

Protocols for WBIProtocols for WBI Standards and legacy systems Programming, Interaction and Control Models Usability Assurance

QoS Safety and Security, Etc.


Example: Bio-Cybernetics Example: Bio-Cybernetics ControlControl

RS232

TINI MicrocontrollerETS Service

802.11b

802.11b

Heart MonitoringService

RJ-45 Ethernet

RS232

RJ-45 Ethernet

BIOPAC MP150

RS232RS232

TINI MicrocontrollerETS Service

802.11b802.11b802.11b

802.11b802.11b802.11b



RJ-45 EthernetRJ-45 Ethernet

RS232RS232

RJ-45 EthernetRJ-45 Ethernet

BIOPAC MP150BIOPAC MP150

TIME

VOLTS

10

5

0 1 2 Sec

TIME

VOLTS

10

5

0 1 2 Sec


Virtual LabVirtual Lab


IBM Sensors Link Real and IBM Sensors Link Real and VirtualVirtual

Sensor data is visualised in real-timeSensor data is visualised in real-time Blue balls with white designs represent Blue balls with white designs represent

active Bluetooth devicesactive Bluetooth devices


The Story so Far The Story so Far #1#1 State-of-the-art in Situated AmI including;State-of-the-art in Situated AmI including;

autonomic software models and autonomic software models and architecture, standards architecture, standards

tools and techniques to support tools and techniques to support the design, modelling, analysis the design, modelling, analysis and evolution of autonomic software and evolution of autonomic software

Define models for their Define models for their programming, control programming, control interaction models with human and/or interaction models with human and/or

other non-AC systems (legacy). other non-AC systems (legacy). Delegation of authority and its Delegation of authority and its

adjustmentadjustment


The Story so Far The Story so Far #2#2 Has been informed by a set of design paradigms Has been informed by a set of design paradigms

Model-based vs self-organising systems design Model-based vs self-organising systems design modelsmodels

Top-down Top-down vsvs bottom-up bottom-up Applying and/or revisiting:Applying and/or revisiting:

cybernetic principlescybernetic principles control systems theory, regulation, reward control systems theory, regulation, reward

and sanctionsand sanctions Decision theory, DAI and CIDecision theory, DAI and CI

dynamic planning, deliberative models, MLdynamic planning, deliberative models, ML Middleware supportMiddleware support

self-awareness, reflection and deliberationself-awareness, reflection and deliberation etc.etc.


Research Challenges #1Research Challenges #1 Conceptual Design and Management modelsConceptual Design and Management models

“… “… how do we design, build, and evolve how do we design, build, and evolve such systems so that they can meet such systems so that they can meet given—and evolving—requirements ...” given—and evolving—requirements ...” Incremental deployment of SAmI Incremental deployment of SAmI

capabilities in legacy systems. capabilities in legacy systems. Hot-swapping, Dynamic AOP-based Hot-swapping, Dynamic AOP-based

evolution, Interoperationevolution, Interoperation Programming, control and Programming, control and

Interaction Models Interaction Models Usability and Interface Usability and Interface

Balancing and adjusting Balancing and adjusting governance and autonomygovernance and autonomy

Social and other requirementsSocial and other requirements Not covered hereNot covered here


Cybernetics: The Viable System Cybernetics: The Viable System Model Model

Beer’s VSM implements a Beer’s VSM implements a control & communication control & communication structure via hierarchies of structure via hierarchies of homeostats (feedback loops) homeostats (feedback loops) (1950) (1950)

Defines 6 major systems Defines 6 major systems ensure ‘viability’ of the systemensure ‘viability’ of the system ImplementationImplementation S1S1 MonitoringMonitoring S2S2 AuditAudit S3*S3* ControlControl S3S3 Intelligence Intelligence S4S4 PolicyPolicy S5S5

Offers an extensible, Offers an extensible, recursive, model-based recursive, model-based architecture, devolving architecture, devolving autonomy to sub-systemsautonomy to sub-systems

System 5Policy

System 4Intelligence

System 3Control

System 3*Audit

System One (S1)

5

4

3

2

2

2

2

3*

1

Org

aniz

atio

nal E

nviro

nmen

t

System 2Coordination


System One (S1)

5

3

43*

1


System One (S1)

5

3

43*

1


AutonomicSystems

CognitiveSystems


Situated AmI CapabilitiesSituated AmI Capabilities A software system is A software system is

autonomic, if it possesses autonomic, if it possesses the following capabilities:the following capabilities: Self-configuringSelf-configuring —— choosing a choosing a

suitable behaviour, based on suitable behaviour, based on user preferences, context, …user preferences, context, …

Self-tuningSelf-tuning —— choosing choosing behaviours that optimize behaviours that optimize certain qualities (performance, certain qualities (performance, year-end profits, …)year-end profits, …)

Self-repairingSelf-repairing —— shifting shifting execution to another behaviour, execution to another behaviour, given that the current one is given that the current one is failingfailing

Self-protectingSelf-protecting —— choosing a choosing a behaviour that minimizes risks behaviour that minimizes risks (attacks, viruses, …)(attacks, viruses, …)


Our Approaches Our Approaches #1#1 Related Work: Related Work:

IBMIBM blueprint blueprint (www-03.ibm.com/autonomic/pdfs/ACBP2_2004-10-(www-03.ibm.com/autonomic/pdfs/ACBP2_2004-10-04.pdf)04.pdf)

An autonomic manager contains a continuous control loop An autonomic manager contains a continuous control loop that monitors activities and takes actions to adjust the that monitors activities and takes actions to adjust the system to meet business objectivessystem to meet business objectives

Autonomic managers learn from past experience to build Autonomic managers learn from past experience to build action plansaction plans

Elements need to be instrumented consistently, based on Elements need to be instrumented consistently, based on open standardsopen standards

Our model – J ReferenceOur model – J ReferenceMonitor Execute

Analyze Plan

Knowledge

Managed Element

Sensors Effectors


Projects in the Networked Projects in the Networked Appliances Lab.Appliances Lab.

Composition of an Internet of Things Composition of an Internet of Things Infinitely Adaptable GamingInfinitely Adaptable Gaming Bridging between Physical and Bridging between Physical and

Virtual (Demo)Virtual (Demo) Managing 3D Multimedia StreamsManaging 3D Multimedia Streams


What It Looks LikeWhat It Looks Like


Prototype ImplementationPrototype Implementation

Paul Fergus, David Llewellyn-Jones, Amjad Shaheed, Madjid Merabti, Abdennour El Rhalibi, Networked Appliances: Manipulation of 3D Multimedia Streams, Accepted for publication in the 5th IEEE International Consumer Communications and Networking Conference: Workshop on Networking Issues in Multimedia Entertainment, 2008, Las Vegas, USA


Augmenting FramesAugmenting FramesNetwork Interface

Virtual Environment

Meta Data Frame Data

Augmented Frame Engine

Game Engine

Network Interface

Network

Television Scene Model JPEG Image

3D MetadataEngine

2D FrameEngine

Render 3D Model Render Image

Device

Augmented Frame Engine


Modelling AutonomyModelling Autonomy Our modelOur model

Using SSC used to formalizes the behaviour of Using SSC used to formalizes the behaviour of dynamically changing systems FOL (McCarthy, 1963).. dynamically changing systems FOL (McCarthy, 1963)..

Support concurrent actions and timing constraints. Support concurrent actions and timing constraints. Each situation can be viewed as a history of previous Each situation can be viewed as a history of previous

actions. actions. Action, guards and time can be modelled at Action, guards and time can be modelled at

deliberation points in an autonomic setting.deliberation points in an autonomic setting. M. Randles, A. Taleb-Bendiab, Philip Miseldine, Andy Laws, "M. Randles, A. Taleb-Bendiab, Philip Miseldine, Andy Laws, "

Adjustable Deliberation of Self-Managing SystemsAdjustable Deliberation of Self-Managing Systems", ", ECBS 2005ECBS 2005: 449-456. [: 449-456. [pptppt]]

Evolving and Adjustable AutonomyEvolving and Adjustable Autonomy Software governance (Control) viaControl) via

Formal modelling of norms, policiesFormal modelling of norms, policies Enactment support – from spec. to code Enactment support – from spec. to code

using using Neptune language Neptune language

http://www.cms.livjm.ac.uk/taleb/Publications/05/RM_Ease2005.pdf

http://ftp.informatik.uni-trier.de/%7Eley/db/conf/ecbs/ecbs2005.html#RandlesTML05

http://www.cms.livjm.ac.uk/taleb/Publications/05/MR_Ease%2705.ppt


Programming Autonomic SystemsProgramming Autonomic Systems Neptune Meta-Language Neptune Meta-Language

Integrated Development Integrated Development Environment:Environment:

Miseldine, P., Taleb-Bendiab A. A Programmatic Miseldine, P., Taleb-Bendiab A. A Programmatic Approach to Applying Sympathetic and Approach to Applying Sympathetic and Parasympathetic Autonomic Systems to Software Parasympathetic Autonomic Systems to Software Design. to appear in the 2005 International Design. to appear in the 2005 International Conference on Self-Organization and Adaptation Conference on Self-Organization and Adaptation of Multi-agent and Grid Systems (SOAS’2005). of Multi-agent and Grid Systems (SOAS’2005).


Conclusions and Future Conclusions and Future WorkWork Understanding WBIUnderstanding WBI

InterfaceInterface Transparency and virtualisationTransparency and virtualisation

Systems and Processes Systems and Processes InteractionInteraction Behaviour Behaviour Structure/organisationStructure/organisation

AffectiveAffective Sensing and Actuation Sensing and Actuation

Co-evolveCo-evolve Address the challenges including:Address the challenges including:

Protocols for WBIProtocols for WBI Standards and legacy systems Programming, Interaction and Control Models Usability Assurance

QoS Safety and Security, Etc.


AcknowledgementsAcknowledgements Acknowledgements

My thanks to All CMS Research Team


That’s the end – so I’m off !


Extra Slides … Extra Slides …

Documents

Prof. A. Taleb-Bendiab, talk: Whole Body Interaction Workshop07, Date: 03/06/2014, Slide: 1 Bio-Cybernetic Control for WBI: a birds eye view Prof. A. Taleb-Bendiab