LESSONS LEARNED ON - Alias RoboticsVilches, V.M. (2018). Time-Sensitive Networking for robotics. arXiv preprint arXiv:1804.07643. HARDWARE ETHERNET ETHERNET CABLING TCP/UDP/IP TCP/UDP/IP

LESSONS LEARNED ONREAL-TIME AND SECURITY

[email protected]

VÍCTOR MAYORAL VILCHES

Robotics engineer

www.aliasrobotics.com [email protected]

https://bit.ly/2pLNI4ILINK TO SLIDES

REMOVING 0-DAYS FROM ROBOTICS

1. REAL-TIME ORREAL-FAST?

REAL-TIME DEFINITION (control)

Real-time control system means that the control system must provide the control responses or actions to the stimulus or requests within specific

times, which therefore depend not just on what the system does but also on

how fast it reacts.

Zhang, P. (2008). Industrial control technology: a handbook for engineers and researchers. William Andrew.

…Real-time, zero-latency technologies capable of detecting attacks that target

running applications and protecting against those attacks.

Feiman, Joseph. “What Is Real-Time Security and Why It Is Needed.” Veracode, 25 May 2016, www.veracode.com/blog/

2016/01/what-real-time-security-and-why-it-needed.

REAL-TIME DEFINITION (security)

DEV. TOOLS

ROS API (C/C++ LIBRARIES)

APPLICATION

PRESENTATION

SESSION

TRANSPORT

NETWORK

DATA LINK

PHYSICAL

DDS

HARDWARE AND CONNECTIVITY (e.g. ETHERNET)

PROTOCOLS (UDP/IP)

ROS 2 APPLICATIONS

OS + DRIVERS

ROS MIDDLEWARE(RMW)

REAL-TIMEROBOT STACK

2. HARDWARERELEVANCEBOTTOM UP HARDWARE AND SOFTWARE COMPLIANCE WITH DEADLINES

ROS 2 APPLICATION

PRESENTATION

SESSION

TRANSPORT

NETWORK

DATA LINK

PHYSICAL

DDS

ETHERNET

UDP/IP

HIGH CRITICALITY

LOW CRITICALITY

MID. CRITICALITY

HIGH CRITICALITY

RTOS Bare metal

Real-Time core Safety core

Tightly Coupled Memory

On Chip Memory

LINUX RTOS Bare metal

HYPERVISOR

Application core

Application core

Application core

Application core

L2 CACHE

MEMORY CONTROLLER

EXTERNAL CONTROLLER

REAL-TIME INDUSTRIAL NETWORKING

TSN GATEWAY

SIL SUPERVISOR

SIL: Safety Integrity Level

STREAMING

HMI

HIGH LEVEL CONTROL

(e.g. ML-based)

PLCs

MOTION CONTROL

MOTOR CONTROL

SAFETY LOOP

3. HARDWARE COMM. LEVELREAL-TIME (CAPABLE) LINK LAYER

REAL-TIMEROBOT STACK ROS 2 APPLICATION

PRESENTATION

SESSION

TRANSPORT

NETWORK

DATA LINK

DDS

ETHERNET

UDP/IP

REAL TIME CAPABLE LINK LAYER

PHYSICAL

HARDWARECOMM. LEVELTOWARDS A REAL TIME CAPABLE LINK LAYER

ETHERNET

ETHERNET CABLING

TCP/UDP/IP TCP/UDP/IP

POWER LINK PROFINET RT

ETHERCAT SERCOS III

PROFINET IRT

ETHERNET ETHERNET MODIFIED ETHERNET

PROFINET ETHERNET/IP

DDS NON REAL-TIME PROTOCOLS


DDS REAL-TIME PROTOCOLS

BASED ON TCP/IP NON REAL-TIME PROTOCOLS

BASED ON TCP/IP REAL-TIME

STANDARD ETHERNET IEEE 802.3

MODIFIED ETHERNET MEDIA ACCESS

REAL-TIME ETHERNET SOLUTIONS

[3] Gutiérrez, C.S.V., Juan, L.U.S., Ugarte, I.Z., & Vilches, V.M. (2018).

Time-Sensitive Networking for robotics. arXiv preprint arXiv:1804.07643.

HARDWARE

ETHERNET

ETHERNET CABLING



ETHERCAT SERCOS III

PROFINET IRT













TOWARDS A REAL TIME CAPABLE LINK LAYER

COMM. LEVEL

HARDWARE

ETHERNET

ETHERNET CABLING



ETHERCAT SERCOS III

PROFINET IRT













TOWARDS A REAL TIME CAPABLE LINK LAYER

COMM. LEVEL

HARDWARE

NO REAL TIME CAPABLE LINK



LINK CAPACITY 1Gbps. NETWORK LOAD = 900Mbps

LAT

EN

CY

(µs)

HISTOGRAM COUNT

2500

2000

1500

1000

500

0250 200 150 100 50 0

TEST TIME (s)

2500

2000

1500

1000

500

00 2 4 6 8 10

LAT

EN

CY

(µs)

COMM. LEVEL

HARDWARE



IEEE 802.1QBV TIME-AWARE SCHEDULER

APPLICATION

MAC

PHY

i210

PHY

PORT 1 PORT 2

SWITCH

END POINT

TAS

PHY

TAS

MAC MAC

TAS

ROBOT CONTROLLER

APPLICATION

PHY

PORT 1 PORT 2

SWITCH

END POINT

TAS

PHY

TAS

MAC MAC

TAS

APPLICATION

ACTUATOR 1 ACTUATOR 2 SENSOR

APPLICATION

MAC

PHY

i210

COMM. LEVEL

HARDWARE



APPLICATION

MAC

PHY

i210

PHY

PORT 1 PORT 2

SWITCH

END POINT

TAS

PHY

TAS

MAC MAC

TAS

ROBOT CONTROLLER

APPLICATION

PHY

PORT 1 PORT 2

SWITCH

END POINT

TAS

PHY

TAS

MAC MAC

TAS

APPLICATION

ACTUATOR 1 ACTUATOR 2 SENSOR

APPLICATION

MAC

PHY

i210

REAL TIME CAPABLE LINK LAYER

TRx2 TRx1

FA1FA2FS TX1 TX2

Congestion Point

Congestion Point

COMM. LEVEL

HARDWARE

REAL TIME CAPABLE LINK



LINK CAPACITY 1Gbps. NETWORK LOAD = 900Mbps

LAT

EN

CY

(µs)

HISTOGRAM COUNT

00

TEST TIME (s)

00 2 4 6 8 10

LAT

EN

CY

(µs)

1500 1000 500

0.5

1

2

4

2.5

1.5

3.5

0.5

1

2

4

2.5

1.5

3.5

3 3

COMM. LEVEL

HARDWAREARCHITECTURE

PL (FPGA Logic)

PS (Processing System)

I/O Protocols ADC

MAC

Accelerator

multi core Cortex A9/A53

Real Time Processing

Unit

High Speed Connectivity

TSN

Sensor Fusion

User Function

PROCESSINGSYSTEM

PROGRAMMABLE LOGIC

TSN Switch

MAC MAC

TSN Endpoint

gPTP

Driver

Linux

API

TSN Subsystem LogiCORE IPYO

CTO

Single Chip with Application Processor and TSN

XILINX PROVIDE MIXED CRITICALITY SOLUTIONS THAT CONNECT TO TSN

4. RTOS AND NETWORKING STACK OPTIMIZED LINUX NETWORKING STACK

ROS 2 APPLICATION

PRESENTATION

SESSION

TRANSPORT

NETWORK

DATA LINK

PHYSICAL

DDS

ETHERNET

UDP/IP OPTIMIZED LINUX NETWORKING STACK


RTOS AND NETWORKING STACK OPTIMIZED LINUX NETWORKING STACK

[4] Gutiérrez, C. S. V., Juan, L. U. S., Ugarte, I. Z., & Vilches, V. M. (2018). Real-time Linux communications: an evaluation of the Linux

communication stack for real-time robotic applications. arXiv preprint arXiv:1808.10821.

ROUND-TRIP client

no-rt we use a vanilla kernel.

rt-normal we use a PREEMPT-RT kernel without binding the round-trip programs and network IRQs to any CPU.

rt-affinities we bind the IRQ thread of the priority queue and the client and server programs to CPU 1 of each device.

rt-isolation we run the roundtrip application in an isolated CPU.

ROUND-TRIP server

T1

T2

RTOS AND NETWORKING STACK OPTIMIZED LINUX NETWORKING STACK



TC CLASSIFICATION

QDISC QDISC QDISC

TX RING BUFFERS

TX QUEUES

… …

… …

… …

PROTOCOLS (TCP/IP,UDP/IP…)

QUEUE MAPPING

NETWORK DEVICE DRIVERS

QUEING DISCIPLINE

REAL-TIME TALKER

BEST-EFFORT TALKER

SOCKET

SOCKET

USERSPACE LINUX NETWORK STACK N.I.C.



NU

MB

ER O

F LA

TEN

CY

SA

MP

LES

100000

10000

1000

100

10

1

100 200 300 400 500 600 700 800

LATENCY (US)

IDLE

RTOS ANDNETWORKING STACKOPTIMIZED LINUX NETWORKING STACK



NU

MB

ER O

F LA

TEN

CY

SA

MP

LES

100000

10000

1000

100

10

1

100 200 300 400 500 600 700 800

LATENCY (US)

STRESS

RTOS ANDNETWORKING STACKOPTIMIZED LINUX NETWORKING STACK

RTOS ANDNETWORKING STACK

NU

MB

ER O

F LA

TEN

CY

SA

MP

LES

100000

10000

1000

100

10

1

100 200 300 400 500 600 700 800

LATENCY (US)

TX TRAFFIC 100MBPS



OPTIMIZED LINUX NETWORKING STACK

5. ROBOTICS FRAMEWORK &COMMUNICATION MIDDLEWAREROS 2.0 FOR REAL-TIME

ROS 2 APPLICATION

PRESENTATION

SESSION

TRANSPORT

NETWORK

DATA LINK

PHYSICAL

DDS

ETHERNET

UDP/IP

ROS 2.0 FOR REAL-TIME


[5] Gutiérrez, C. S. V., Juan, L. U. S., Ugarte, I. Z., & Vilches, V. M. (2018). Towards a distributed and real-time framework for robots: Evaluation

of ROS 2.0 communications for real-time robotic applications. arXiv preprint arXiv:1809.02595.

ROBOTICS FRAMEWORK &COMMUNICATION MIDDLEWAREROS 2.0 FOR REAL-TIME

ROBOT CONTROLLER




ROS 2.0 publisher

DDS publisher

ROS 2.0 subscriber

DDS subscriber




NU

MB

ER O

F LA

TEN

CY

SA

MP

LES

100

10

1

2000 4000 6000 8000

ROUND-TRIP LATENCY (US)

SYSTEM idle

10000

1000




NU

MB

ER O

F LA

TEN

CY

SA

MP

LES

100

10

1

2000 4000 6000 8000


SYSTEM under load (stress)

10000

1000




NU

MB

ER O

F LA

TEN

CY

SA

MP

LES

100

10

1

2000 4000 6000 8000


10000

1000

SYSTEM UNDER LOAD WITH RT SETTINGS




NU

MB

ER O

F LA

TEN

CY

SA

MP

LES

1000

100

10

1

2000 4000 6000 8000


SYSTEM UNDER LOAD WITH RT SETTINGS

10000




NU

MB

ER O

F LA

TEN

CY

SA

MP

LES

1000

100

10

1

2000 4000 6000 8000


SYSTEM UNDER LOAD WITH 1 Mbps CONCURRENT TRAFFIC AND RT SETTINGS

10000




NU

MB

ER O

F LA

TEN

CY

SA

MP

LES

1000

100

10

1

2000 4000 6000 8000



10000




NU

MB

ER O

F LA

TEN

CY

SA

MP

LES

1000

100

10

1

2000 4000 6000 8000



10000

REAL-TIMEREFERENCE DEMO

REAL-TIME PENDULUM

Gutiérrez, C. S. V., San Juan, L. U., ROS2, real-time, control, pendulum

https://github.com/ros2-realtime-demo/pendulum

6. TIME SYNCHRONIZATION IN ROBOT MODULES

ROS 2 APPLICATION

PRESENTATION

SESSION

TRANSPORT

NETWORK

DATA LINK

PHYSICAL

DDS

ETHERNET

UDP/IPSYNCHRONIZATION AT ALL LEVELS


TIME SYNCHRONIZATIONIN ROBOT MODULES

[6] Gutiérrez, C. S. V., Juan, L. U. S., Ugarte, I. Z., Goenaga, I. M., Kirschgens, L. A., & Vilches, V. M. (2018).

Time Synchronization in modular collaborative robots. arXiv preprint arXiv:1809.07295.

STATUS-MOTOR 1 (M1)

STATUS-MOTOR 2 (M2)

ROBOT CONTROLLER

start of period n

start of period n

start of period n+1

Δt PUB (M2)

ΔtSUB (M1)

Δt SUB (M2)

ΔtPUB (M1)

Δt SUB (M2)

ΔtPUB (M2)

Δt SUB (M1)

Δt PUB (M1)

ΔtPUB (M1)

motor1 publishing time offset

ΔtSUB (M1)

Motor1 arrival time offset

ΔtPUB (M2)

Motor1 publishing time offset

ΔtSUB (M2)

Motor1 arrival time offset




ROBOT CONTROLLER

NU

MB

ER O

F S

AM

PLE

S100

10

1

2000 4000 6000 8000

MICROSECONDS

ARRIVAL TIME OFFSET FROM THE EXPECTED PERIOD

10000




NU

MB

ER O

F S

AM

PLE

S100

10

1

2000 4000 6000 8000

MICROSECONDS

ARRIVAL TIME OFFSET FROM THE EXPECTED PERIOD

10000




NA

NO

SEC

ON

DS

2000

1500

1000

1

100 200 300 400

SAMPLES

PTP MAXIMUM ABSOLUTE OFFSET

500




500

600 700

7. REAL-TIMESECURITY

ROS 2 APPLICATION

PRESENTATION

SESSION

TRANSPORT

NETWORK

DATA LINK

PHYSICAL

DDS

ETHERNET

UDP/IPREAL-TIME RESILIENCE TO SECURITY BUGS?


REAL-TIMESECURITY

Kim, J., Smereka, J. M., Cheung, C., Nepal, S., & Grobler, M. (2018).

Security and performance considerations in ros 2: A balancing act.

arXiv preprint arXiv:1809.09566.

DiLuoffo, V., Michalson, W. R., & Sunar, B. (2018). Robot Operating System 2: The need for a holistic

security approach to robotic architectures. International Journal of Advanced Robotic Systems,

15(3), 1729881418770011.

Somehow understood as

UNDERSTAND THE ATTACK VECTORS THAT AFFECT REAL-TIME

THREATMODEL

CI PIPELINE

UPDATE DEPLOYMENT

SERVICE

OTASERVER

OTAUPDATE AGENT

Push

art

ifact

Push artifacts

Update

Enrollment request

ARTIFACT STORAGE

Cod

e Pu

sh

Sen

d co

mm

ands

Sen

d co

mm

ands

Cod

e co

mm

it

CODEREPOSITORY

ROBOTICS ENGINEER

ACTUATOR1

ACTUATOR2

IMU TEMPERATURESENSOR

TEMPERATURESENSORDRIVER

POWER SENSOR

POWER SENSORDRIVER

JOINTCONTROLTHREAD

Ret

urn

feed

back

Ret

urn

feed

back

Sen

d re

adin

gs

IMU DRIVER

Sen

d re

adin

gs

Sen

d re

adin

gs

Sen

d re

adin

gs

Sen

d re

adin

gs

Sen

d re

adin

gs

JOINTSTATE

MONITORTHREAD

JOINTMODEL

DELIVERYTHREAD

ROS2CONTROLNODES

Send commands

Send models

Send status

H-ROSAPI

CLIENT

H-ROSAPI

SERVER

NODELIFECYCLECONTROLER

Send commandsSend commands

UPDATEAGENT

Send commands

Send commands

ROBOT OPERATOR

UPDATEDEPLOYMENT

SERVICE

Push updates

ARTIFACT STORAGE

Fetc

h ar

tifa

ct

CI PIPELINE

Push artifact

Cod

e Pu

shC

omm

it c

ode

CODEREPOSITORY

ROBOTICS DEVELOPER

TECHNICALSUPPORT

TECHNICIAN

Fetc

h ar

tifa

ct

Perform maintainance

FIRMWARE UPDATES

OTASYSTEM

MARAROBOT

INDUSTRIALPC (ORC)

SOFTWARECONTROL

SECURITY SOLUTIONS

SERVICES

z

CODE TESTING

FORENSICS ROBOT SECURITY

VIRTUAL ROBOT HACKING

ROBOTIC SOFTWARE

DEVELOPMENT LIFECYCLE

THREAT MODEL

ANALYSIS

SECURITY STANDARDS

COMPLIANCE

PHYSICAL ROBOT HACKING

Robot security

ASSESSMENTS

Security

CONSULTING

RECORDS

™

Enables forensic investigation

ROBOT IMMUNE SYSTEM

Detects threats by learning usual comms

Non-intrusive. Real-time. No latencies

Hardware agnostic. Plug & Play

™

Records all robot data

PRODUCTS

ALIAS ROBOTICS

www.aliasrobotics.com [email protected]

REMOVING 0-DAYSFROM ROBOTICS

Documents

LESSONS LEARNED ON - Alias RoboticsVilches, V.M. (2018). Time-Sensitive Networking for robotics. arXiv preprint arXiv:1804.07643. HARDWARE ETHERNET ETHERNET CABLING TCP/UDP/IP TCP/UDP/IP