The DDS Benchmarking Environment

The DDS Benchmarking Environment

James Edmondsonjedmondson@gmail.com

Vanderbilt University Nashville, TN

2

Topics• What is the DBE?• What technologies does the DBE use?• How does the DBE setup the environment? • What problems had to be dealt with? • What features will be added? • Demonstration• Conclusion and Wrap Up

3

What is the DBE?• DDS Benchmarking Environment• Sets up the testing environment and framework• Configures and launches implementation specific services• Launches Publishers and Subscribers• Coordinates interaction, logging info, and stats collection• Allows for easily scaling the number of participants• Sets up the statistics for collection by Bagatel

Throu

ghpu

tPub

lishers

NetworkingSubscribers

4

What is the DBE?• May be used to test other middleware or technologies• Focus is on documented, reusable resources

Throu

ghpu

tPub

lishers

NetworkingSubscribers

Versatility Dependability

5

What does the DBE do?

Node 0

DBE

Node 1 Node 2 Node N

6

Node 0

DBE

Node 1 Node 2 Node N

What does the DBE do?

7

Node 0

DBE

Node 1 Node 2 Node N

Setup TestEnvironment

What does the DBE do?

8

Node 0

DBE

Node 1 Node 2 Node N

Setup TestEnvironment

What does the DBE do?

9

What does the DBE do?

Node 0

DBE

Node 1 Node 2 Node N

Setup TestEnvironment

ConfigureImplementation

10

What does the DBE do?

Node 0

DBE

Node 1 Node 2 Node N

Setup TestEnvironment

ConfigureImplementation

Start theSubscribers

11

What does the DBE do?

Node 0

DBE

Node 1 Node 2 Node N

Setup TestEnvironment

ConfigureImplementation

12

What does the DBE do?

Node 0

DBE

Node 1 Node 2 Node N

Setup TestEnvironment

ConfigureImplementation

13

What does the DBE do?

Node 0

DBE

Node 1 Node 2 Node N

Setup TestEnvironment

ConfigureImplementation

14

What does the DBE do?

Node 0

DBE

Node 1 Node 2 Node N

Subscriber Subscriber Subscriber

Setup TestEnvironment

ConfigureImplementation

15

What does the DBE do?

Node 0

DBE

Node 1 Node 2 Node N

Subscriber Subscriber Subscriber

Setup TestEnvironment

ConfigureImplementation

Start thePublisher

16

What does the DBE do?

Node 0

DBE

Node 1 Node 2 Node N

Subscriber Subscriber Subscriber

Setup TestEnvironment

ConfigureImplementation

17

What does the DBE do?

Node 0

DBE

Node 1 Node 2 Node N

Publisher Subscriber Subscriber Subscriber

Setup TestEnvironment

ConfigureImplementation

18

What technologies does the DBE use?

• Scripts are written in Perl and use PerlAce module• Remote execution is accomplished via SSH• Logs, stats and test info are stored on a NFS partition• All tests are currently ran on Fedora Core Linux machines• Configured to test NDDS, TAO DDS, and Splice• Synchronization between tests is maintained by using

lockfilesDBE

Linux SSH NFS Perl lockfiles

19

How does the DBE setup the environment?

• Most path information is updated using the ~/.ssh/environment file

• Some custom environment variables are set by DBE scripts on individual clients

• Individual Test IDs are assigned to each test• Directories are created on the NFS partition for stats and logs

DBE

20

How does the DBE setup the environment?

• Most path information is updated using the ~/.ssh/environment file

• Some custom environment variables are set by DBE scripts on individual clients

• Individual Test IDs are assigned to each test• Directories are created on the NFS partition for stats and logs

DBE

21

How does the DBE setup the environment?

• Most path information is updated using the ~/.ssh/environment file

• Some custom environment variables are set by DBE scripts on individual clients

• Individual Test IDs are assigned to each test• Directories are created on the NFS partition for stats and logs

DBE

Change $ENV

22

How does the DBE setup the environment?

• Most path information is updated using the ~/.ssh/environment file

• Some custom environment variables are set by DBE scripts on individual clients

• Individual Test IDs are assigned to each test• Directories are created on the NFS partition for stats and logs

DBE

Change $ENV

23

How does the DBE setup the environment?

• Most path information is updated using the ~/.ssh/environment file

• Some custom environment variables are set by DBE scripts on individual clients

• Individual Test IDs are assigned to each test• Directories are created on the NFS partition for stats and logs

DBE

Change $ENV Generate ID

24

How does the DBE setup the environment?

• Most path information is updated using the ~/.ssh/environment file

• Some custom environment variables are set by DBE scripts on individual clients

• Individual Test IDs are assigned to each test• Directories are created on the NFS partition for stats and logs

DBE

Change $ENV Generate ID

25

How does the DBE setup the environment?

• Most path information is updated using the ~/.ssh/environment file

• Some custom environment variables are set by DBE scripts on individual clients

• Individual Test IDs are assigned to each test• Directories are created on the NFS partition for stats and logs

DBE

Change $ENV Generate ID Create Directories

26

How does the DBE setup the environment?

• Publishers and subscribers are notified of test settings– Result file locations– QoS settings on a per subscriber basis– Network settings– Test specific settings (number of messages to send, etc.)

DBE

Repos Publishers Subscribers

Pass Test Info

27

What problems had to be dealt with?

• Lack of a terminal with remote command execution with SSH• TCP and UDP buffer sizes were too small

• Synchronization amongst publishers and subscribers

Subscriber

Subscriber

PublisherW

ait

PublisherBuffer

PublisherBuffer

Next Test

28

What problems had to be dealt with?

• Flock() doesn’t work as expected with stateless NFS

• Shared libraries are referenced locally• Block buffering of stdout

flock(“myfile”) touch(“myfile.lock”)

stdout

stderr

file

fileimmediate

block buffering

29

What problems had to be dealt with?

• More problems on request ;)

30

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

31

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Create Publisher

32

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Create Publisher

33

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher

Create Publisher

34

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher

35

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher

Create Subscriber

36

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher

Create Subscriber

37

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher Subscriber

Create Subscriber

38

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher Subscriber

39

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher Subscriber

Create Subscriber

40

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher Subscriber

Create Subscriber

41

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher Subscriber Subscriber

Create Subscriber

42

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher Subscriber Subscriber

43

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher Subscriber Subscriber

KillSubscriber 1

44

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher Subscriber Subscriber

KillSubscriber 1

45

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher Subscriber

KillSubscriber 1

46

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher Subscriber

KillSubscriber 1

47

What features will be added?

• New tests will require new features• One such feature is per node scripting of subscribers and

publishers• Feature will allow dynamic starting and stopping of

participantsDBE

Publisher Subscriber

48

What other additions are planned?

• The ability to launch logging services– Memory and CPU usage will need to be recorded

• The ability to test other middleware and applications

49

Demonstration

Primer messages = 100Stats messages = 1000Test type = throughputImplementation = NDDS

50

Conclusion and Wrap Up

• The DBE is an excellent suite of tools for automating DDS testing on a networked environment

• New test types will be added to gain further insight into the DDS implementations

• Many new features will also be added to better reveal problems and benefits of the various DDS products

• We would like to thank our AFRL and Vanderbilt Sponsors for their continued support