Tuning and Troubleshooting OpenSplice DDS Applications

Ope

nSpl

ice

DD

S

Angelo Corsaro, PhDChief Technology Officer OMG DDS Sig Co-Chair

PrismTechangelo.corsaro@prismtech.com

Tuning and Troubleshooting OpenSplice DDS Applications

Hans van’t Hag.Product Manager

OMG DDS Co-AuthorPrismTech

hans.vanthag@prismtech.com

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

US OSUM 2013

http://www.prismtech.com/events/opensplice-user-meeting-usa-2013

Ope

nSpl

ice

DD

S Part I -- Setting the Context

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Kinds of IssuesWhen building distributed applications there are several kinds of issues you can encounter. Some of the most common issues are often:

☐ Connectivity: “things” don’t seem to talk to each other

☐ Performance:the latency is too high or throughput is too low

☐ Scalability: your system does not scale as necessary

☐ Resource Utilization: your application is using way too much CPU, Memory, and/or Network

This webcast explores common issues in DDS applications and demonstrates how the tools provided by OpenSplice DDS can their diagnose and resolution

Ope

nSpl

ice

DD

S

Connectivity

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Connectivity Issues

☐ DDS provides dynamic discovery thus as there is network connectivity applications that are intended to communicate should discover each other and just work!

☐ What may be the cause of connectivity issues in DDS?☐ The network is not working (did you plug the wire?)☐ Your applications does not share the same domain (thus your mistake)☐ The QoS of the DataWriter and the DataReader don’t match

Ope

nSpl

ice

DD

S

QoS Mismatch?

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Understand the QoS Model☐ DDS defines 22 QoS

policies that can be applied to communication entities to control their local as well as end-to-end behaviour

Tip #8

HISTORY

LIFESPAN

DURABILITY

DEADLINE

LATENCY BUDGET

TRANSPORT PRIO

TIME-BASED FILTER

RESOURCE LIMITS

USER DATA

TOPIC DATA

GROUP DATA

OWENERSHIP

OWN. STRENGTH

LIVELINESS

ENTITY FACTORY

DW LIFECYCLE

DR LIFECYCLE

PRESENTATION

RELIABILITY

PARTITION

DEST. ORDER

RxO QoS Local QoS

Most of the QoS Policies that control an end-to-end property follow the so-called Request vs. Offered (RxO) Model based on which the QoS requested by the Consumer should not exceed the QoS Provided by the Producer. If there is an RxO mismatch no communication will take place

Ope

nSpl

ice

DD

S

Ope

nSpl

ice

DD

S

Performance

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Latency / Throughput☐ DDS provides several QoS Policies to control latency and

throughput

☐ OpenSplice provides extensions such as Streams to efficiently distribute small data (e.g. several millions samples per second)

Throughput

TimeBasedFilter

[Inbound]

[Outbound]Latency

Deadline

TransportPriority

LatencyBudget

☐ OpenSplice Tools such as the Tuner can help find the right QoS setting for achieving the desired level or latency/throughput

Ope

nSpl

ice

DD

S

Scalability

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Federated / Standalone Deployment

Federated Deployment Standalone Deployment

Operating Systems

Application

OpenSplice DDS

Application

OpenSplice DDSOpenSplice DDS

Application Application

Operating Systems

☐ Performance ☐ Scalability☐ Determinism

☐ Usability ☐ Simple Configuration☐ Simple Deployment

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Deployment Configuration

☐ The deployment mode can be changed by a simple configuration parameter. No recompilation or re-linking!☐ <SingleProcess>true</SingleProcess>

☐ The deployment options can be mixed at will (even within a single computing node)

☐ The same application can be deployed in federated and standalone mode (even on the same system)

“Simple when Sufficient.Performant when Required”

With OpenSplice DDS you can choose the

deployment architecture that makes

the most sense for your use case.

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Deployment Configuration

☐ The deployment mode can be changed by a simple configuration parameter. No recompilation or re-linking!☐ <SingleProcess>true</SingleProcess>

☐ The deployment options can be mixed at will (even within a single computing node)

☐ The same application can be deployed in federated and standalone mode (even on the same system)

“Simple when Sufficient.Performant when Required”

With OpenSplice DDS you can choose the

deployment architecture that makes

the most sense for your use case.

Ope

nSpl

ice

DD

S

Resource Utilization

Ope

nSpl

ice

DD

S

Shared Memory Sizing

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Shared Memory Size☐ OpenSplice DDS shared memory size is defined in its

configuration file. The size defined by the default configuration file is 10MBytes

☐ Beware that different OS have different limitations w.r.t. the maximum shared memory segment that can be allocated

☐ If you want to go beyond the OS limits you need to change the configuration of your kernel

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Linux☐ The default value for the maximum shared memory segment is

32MBytes

☐ This default can be changed in several ways

(1) Adding this line to your /etc/rc.d/rc.local file:

(2) Changing the settings for the sys-limits (save changes on /etc/sysctl.conf to maintain them across reboots):

echo “your_max_shared_memory_size” > /proc/sys/kernel/shmmax

$ sysctl -w kernel.shmmax=yourMaxValue

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Windows

☐ The default maximum size for Shared Memory segments on Windows is 2GB

☐ To exend it, say to 3GB, add the /3GB the boot.ini as shown below:

[boot loader]timeout=30default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS[operating systems]multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows NTWorkstation Version 4.00" /3GB

Ope

nSpl

ice

DD

S

Understand Topic Keys

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Topic Types & Keys

☐ Topic Types can define some of their attributes as keys

☐ Yet, even when a Topic type does not define a key the keylist directive has to be provided -- just to tell the IDL compiler that this is a topic

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Topic Types & Keysenum TemperatureScale { CELSIUS, FAHRENHEIT, KELVIN};

struct TempSensorType { short id; float temp; float hum; TemperatureScale scale;};#pragma keylist TempSensorType id

struct UniversalConstants { double PI; double e; // Nepero Number double g; // Gravitational Constant double NA; // Avogadro Number double F; // Faraday Constant double K; // Boltzman Constant double c; // Light Speed };#pragma keylist UniversalConstants

KeylessKeyful

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Topic Instances

Writer Reader

1 26.0 70.0 CELSIUS

2 26.0 70.0 CELSIUS 1 26.0 70.0 CELSIUS

2 26.0 70.0 CELSIUS

enum TemperatureScale { CELSIUS, FAHRENHEIT, KELVIN};

struct TempSensorType { short id; float temp; float hum; TemperatureScale scale;};#pragma keylist TempSensorType

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Topic Instances

Writer Reader

1 26.0 70.0 CELSIUS

2 26.0 70.0 CELSIUS1 26.0 70.0 CELSIUS 2 26.0 70.0 CELSIUS

enum TemperatureScale { CELSIUS, FAHRENHEIT, KELVIN};

struct TempSensorType { short id; float temp; float hum; TemperatureScale scale;};#pragma keylist TempSensorType id

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Instance Management☐ Each topic instance takes

resources (e.g. memory)

☐ If you don’t choose carefully your key, you may end-up creating an instance per sample...

☐ In addition, instances can be explicitly disposed in order to reclaim resources

Writer Reader

1 26.0 70.0 CELSIUS

2 26.0 70.0 CELSIUS1 26.0 70.0 CELSIUS 2 26.0 70.0 CELSIUS

Ope

nSpl

ice

DD

S

Example

struct Counter { short cid; long count;};#pragma keylist Counter ???

Ope

nSpl

ice

DD

S

Example

struct Counter { short cid; long count;};#pragma keylist Counter cid

Ope

nSpl

ice

DD

S

Example

struct SpatialSensor { short x; short y; float v;};#pragma keylist SpatialSensor ???

Ope

nSpl

ice

DD

S

Example

struct SpatialSensor { short x; short y; float v;};#pragma keylist SpatialSensor

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Topic Instances Lifecycle

com.myco.VPos

com.myco.VPos

struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition plate

struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Topic Instances Lifecycle

com.myco.VPos

com.myco.VPos

“A01” 100 200

“A01” 100 200

New

New

struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition plate

struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Topic Instances Lifecycle“A01” 100 200

“B41” 57 31

com.myco.VPos

“A01” 100 200“B41” 57 31

com.myco.VPos

New

New

struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition plate

struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

“A01” 100 200“B41” 57 31

com.myco.VPos

“A01” 110 210

Topic Instances Lifecycle“A01” 100 200

“B41” 57 31

com.myco.VPos

“A01” 110 210New

New

struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition plate

struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

“A01” 100 200“B41” 57 31

com.myco.VPos

“A01” 110 210

Topic Instances Lifecycle“A01” 100 200

“B41” 57 31

com.myco.VPos

“A01” 110 210“A01” 120 220

“A01” 120 220

New

New

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

“A01” 100 200“B41” 57 31

com.myco.VPos

“A01” 110 210

Topic Instances Lifecycle“A01” 100 200

“B41” 57 31

com.myco.VPos

“A01” 110 210“A01” 120 220

“A01” 120 220

“B41” 47 19

“B41” 47 19

New

New

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

“A01” 100 200“B41” 57 31

com.myco.VPos

“A01” 110 210

Topic Instances Lifecycle“A01” 100 200

“B41” 57 31

com.myco.VPos

“A01” 110 210“A01” 120 220

“A01” 120 220

“B41” 47 19

“B41” 47 19

New

New“B41” - -Disposed

Disposed

Ope

nSpl

ice

DD

S Part II -- Demoes

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

Content☐ Tools

☐ Configuration: OpenSplice Configurator☐ Black-box System (regression) Testing: OpenSplice Tester☐ White-box application Inspection & Tuning: OpenSplice Tuner☐ Recording & Replay: OpenSplice RnR☐ Shared-memory and Network monitoring: OpenSplice mmStat and ‘Wireshark’ ™

☐ Common Issues☐ System Configuration (deployment architecture, networking-setup)☐ System Deployment (resource usage/watermarks, statistics, reports, tracing)☐ Tool Deployment (intrusiveness: local versus remote operation)

☐ Demo’s☐ Tester: system browser (SHMEM vs. SP) and QoS conflict detection (Reliability)☐ Tuner: Statistics monitoring & QoS Tuning (Latency Budget)☐ Tester: 1-click capturing & real-time Charting (virtual attributes, scripting)☐ RnR: scripted recording & replay with RnR-service and Tester☐ RnR: graphical control with RnRManager

OpenSplice Tools

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

Total Control: OpenSplice Configurator™

☐ The reference tool for configuring OpenSplice DDS

☐ Rich online guide to configuration options

☐ Context help and parameter validation

☐ 100% Java

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

OpenSplice Tester™ ☐ “Black-Box” DDS system testing

☐ Automated testing of DDS systems☐ Dynamic Discovery of DDS Entities

☐ Domain-specific scripting languages

☐ Batch execution of regression tests

☐ Debugging of distributed DDS systems☐ System browser of DDS participants

☐ Connectivity & QoS conflict monitoring

☐ One-click definition of monitoring timeline☐ Analysis/comparison of topic data

☐ Virtual topic-attributes to ease analysis

☐ Statistics monitoring

☐ 1-click spawning of Tuner to ‘attach’ to a remote process / federation

☐ Integrated IDE☐ Syntax highlighting editor

☐ One-click relations between script/logs/timeline

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

☐ “Whitebox” debug/tuning Tool☐ Looking ‘inside’ a federation and/or

application

☐ Different perspectives (participant, topic, partition)

☐ Monitoring & Tuning☐ Inspect and Tune the app’s DDS entities

☐ Make snapshots of reader-caches

☐ Detect and resolve QoS Mismatch

☐ Inspect Statistics

☐ Reading & Writing☐ Read/Write data for arbitrary topics

☐ Import & Export☐ Inject Topic Definitions

☐ export and import XML-based reader/writer snapshots

Total Control: OpenSplice Tuner™

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

RnRService

OpenSplice DDS

RnRService(s)

Any topic Record/Replay command & status topics

RnRManager

OpenSplice RnR

Record/Replay command & status topics

☐ Dynamic recording of any topic-data in a DDS system

☐ Selective replay with variable speed

☐ Distributed control by topic-based API

☐ Seamless integration with OpenSplice Tester (topic-based API)

☐ Dedicated RnR-Manager graphical GUI for scenario-definition and data import/analysis

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

Wireshark ☐ WireShark Packet

Dissector

☐ Watch what goes on the wire

☐ Inspect DDSI-RTPS and/or RT-Networking packets being exchanged between applications

☐ Native RTnetworking dissector support available in source-distribution

☐ OpenSpliceDDS\V6.2.1\HDE\x86.win32\tools\wireshark-plugins\ospl (see README)

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

mmStat

mmstat -h mmstat [-M|m] [-e] [-a] [-i interval] [-s sample_count] [URI] mmstat [-t|T] [-i interval] [-s sample_count] [-l limit] [-f filter_expression] [URI]

Mode: -m Show memory statistics (default mode) -M Show memory statistics difference -t Show meta object references -T Show meta object references difference

-h Show this help -e Extended mode, shows bar for allocated memory -a Show pre-allocated memory as well. -i interval Display interval (in milliseconds) -s sample_count Stop after sample_count samples -l limit Show only object count >= limit -f filter_expr Show only meta objects which name passes the filter expression

Common Issues: Examples & Demo’s

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

11

☐ Selecting the proper deployment architecture☐ Federated / shared-memory (for scalability and data/application lifecycle decoupling)☐ Standalone / single-process (for simplicity yet at the cost of ‘network-scheduling determinism’)

☐ Selecting the proper networking architecture☐ RTNative networking: ‘push’ based architecture using pre-configured (multicast) network-partitions

☐ Pro : zero discovery times (optional topology-discovery only needed for fast disconnect-reaction times)☐ Con : push implies data might show-up at places where there’s no actual/current interest .. Yet tunable in OpenSplice

☐ DDSI networking: ‘pull’ based architecture using advertized locators☐ Pro : selective distribution to ONLY the set of discovered participants, using unicast whenever just 1 destination☐ Con : (standardized) discovery-protocol has scalability limitations (amount of heartbeats) .. Yet tunable in OpenSplice

☐ Configuring your domain☐ Using “OpenSplice Configurator” to aid in correct configurations☐ Assure system-wide consistent values

☐ Domain-ID’s and in case of RTNetworking the related port-numbers (channels and discovery)☐ RTNetworking only: NetworkPartition definitions (names and related [multicast-] addresses)

Common issues: System Configuration

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

13

☐ How to Prevent Issues☐ Using the Configurator to validate configurations [demo]☐ Assuring system-wide consistent networking/durability setup☐ Using ResourceLimits policies in your applications

☐ How to detect Issues☐ Setting resource-usage ‘watermarks’ with the configurator [demo]☐ Checking ospl_info.log and ospl_error.log files [demo]☐ Checking QoS RxO conflicts with the Tester and/or Tuner [demo]☐ Capturing behavior with Tester (and [auto-]creation of scripts to reproduce)☐ Recording of information using RnR (manually and/or scripted) [demo]

☐ How to resolve issues☐ Runtime Tuning of QoS policies with the Tuner [demo]☐ Exploring reader/writer/service statistics with Tester & Tuner [demo]

Common issues: System Deployment

Demo’s

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

Demo’s☐ Environment

☐ Perf1/2/3 reference-system (shmem versus SP)☐ Ishapes demo-application☐ Configurator / Tester / Tuner / RnR tools

☐ Configurator☐ Used configuration-files (SHMEM/RnR, SHMEM/Statistics, SP/Resource-limits☐ Demo-config: perf1: RnR added, perf2: statistics added perf3: resource-limits

☐ Tuner/Tester ‘ishapes’ demos☐ Tester: system browser (SHMEM vs. SP)☐ Tester: QoS conflict detection (Reliability)☐ Tuner: QoS Tuning (Latency Budget)☐ Tuner: Status & Statistics (Writer/Reader, perf2: Network-queues, perf3: Network receiving)☐ Tester: real-time Charting (virtual attributes, scripting)

☐ Recording & Replay☐ Scripting with Tester☐ Controlling with RnRManager

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

Perf3Perf2

Splice-Hans

Switch

Perf1

Switch

Router

Office-net

Test-net

Perf-net

Shared Memory

soap RT-Net RnR / Dur.

Tester-1/ RnRManager

DDS libs

IshapesstandaloneSOAP

NETW

Remote Tuner 1

remote Tester-2

Demo setup

Shared Memory

Soap RT-Net Durability

Ishapes Federated

Ishapes Federated

Remote Tuner 2

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

Teaser: ‘Streams’☐ Bundled Example

☐ Streaming 10 mln. Samples

☐ Throughput: up to 29 mln. Samples/s ! (shmem/i7)

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

Configuration

☐ Perf1☐ Ospl_shmem_native_rnr.xml

☐ Perf2☐ Ospl_shmem_native_statistics.xml

☐ Perf3☐ Ospl_sp_native_resource_limits.xml

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

QoS conflict detection

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

QoS Tuning

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

Statistics Monitoring

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

Data-Capture

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

Scripted RnR

Copyrig

ht  2013,  PrismTech  –    A

ll  Rights  Reserved.

Graphical RnR

Ope

nSpl

ice

DD

S

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

References

¥Fastest growing JVM Language¥Open Source¥www.scala-lang.org

¥ #1 OMG DDS Implementation¥ Open Source¥ www.opensplice.org

OpenSplice | DDS¥Scala API for OpenSplice DDS¥Open Source¥github.com/kydos/escalier

Escalier

¥Simple C++ API for DDS¥Open Source¥github.com/kydos/simd-cxx

¥DDS-PSM-Java for OpenSplice DDS¥Open Source¥github.com/kydos/simd-java

¥ DDS-based Advanced Distributed Algorithms Toolkit

¥Open Source¥github.com/kydos/dada

Ope

nSpl

ice

DD

S ¥@prismtech

¥@acorsaro

¥youtube.com/opensplicetube ¥slideshare.net/angelo.corsaro

¥ crc@prismtech.com¥sales@prismtech.com

¥opensplice.com ¥forums.opensplice.org

¥opensplice.org ¥opensplicedds@prismtech.com

:: Connect with Us ::