July 21, 2005 AESE 2005 1

Fast Hybrid Simulation with Geographically Distributed Substructures

Gilberto Mosqueda

Boza Stojadinovic

Jason P. Hanley (Presenter)

Andrei M. Reinhorn

Mettupalayam Sivaselvan

July 21, 2005 AESE 2005 2

Distributed Testing in NEES

• NEES is the Network for Earthquake Engineering Simulation– Utilize capabilities of geographically distributed

experimental facilities– Provide remote access to these facilities

• MOST was the first full scale test of the NEESgrid infrastructure

• Goal of research is to expand NEES infrastructure to:– Improve reliability of results– Increase speed of test

July 21, 2005 AESE 2005 3

Communications Protocol

NTCPNEESgrid Teleoperations Control Protocol

• A standardized protocol for exchange of control information

• Provides integrated authentication and authorization services

• Robust handling of network faults– Re-transmission after timeout– Ability to pause or restart an experiment

July 21, 2005 AESE 2005 4

Faster Continuous Testing

• Parallel communication with remote sites– Implement multi-threaded Simulation Coordinator

• Streamlined communications protocol (NTCP)– Minimize network communication– Reduce overhead caused by security protocols

• Event-driven controller– Reduce the time it takes to load experimental

substructures– Implement algorithms for continuous loading of

experimental substructures

July 21, 2005 AESE 2005 5

Multi-threaded Communication

Threads running in parallel

• In linear execution step time is the sum of communication times with each site

• In parallel execution step time is the maximum of the communication times with each site

MatlabPlugin

NTCPServer NTCP

Berkeley

MatlabPlugin

NTCPServer

BoulderMatlabPlugin

NTCPServer

UIUC

ScramnetPlugin

NTCPServer

Buffalo

MatlabPlugin

NTCPServer

Lehigh

NumericalSimulation

NTCPClient

Simulation Coordinator

NTCP

NTC

P NTC

P

NT

CP

July 21, 2005 AESE 2005 6

NTCP Improvements

• The original design of the protocol has too much overhead

• A simulation time step could involve up to 3 round-trip communicationsPropose > Execute > Query

• A new command (ProposeAndExecute) was created to do all this in one round-trip communication

• Improvements are backward compatible with any control system interfaced to NTCP

July 21, 2005 AESE 2005 7

Event-driven Controller

• Continuous loading of experimental substructures• Used polynomial extrapolation to predict load path and

corrects using interpolation when target becomes available

• Time out’s calibrated to network round-trip communication time

free_vibration

extrapolateD_update

D_target

interpolate

TimeOutD_update

TimeOut

LegendState:

State Transition Path:Event causing State Transition: Event/functionCall()

slow holdTimeOut

D_updatefree_vibration

extrapolateD_update

D_target

interpolate

TimeOutD_update

TimeOut

LegendState:

State Transition Path:Event causing State Transition: Event/functionCall()

LegendState:

State Transition Path:Event causing State Transition: Event/functionCall()

slow holdTimeOut

D_update

July 21, 2005 AESE 2005 8

Structural Model

6-span bridge model– Span and two column are numerical models– Other three columns are experimental models

Experimental Sites:BerkeleyBoulderBuffalo

Computational Sites:Buffalo (Span)UIUCLehigh

July 21, 2005 AESE 2005 9

Remote Substructures

Berkeley• Stiffness 0.49kN/mm• Yield disp. 16mm• Max. disp. 100mm

Buffalo• Stiffness 0.32kN/mm• Yield disp. 8mm• Max. disp. 25mm

The displacement and forces were scaled to match the initial stiffness of the numerical full-scale column.

July 21, 2005 AESE 2005 10

Numerical Simulation

• Bridge model was subjected to 15 seconds of ground shaking– Ground motion is an artificial record created from

spectral density

• Operator-splitting integration algorithm– Integration time step of 0.01 seconds– 1500 simulation steps

• Numerical simulation integrated with the Simulation Coordinator– Remove one round-trip communication per step

July 21, 2005 AESE 2005 11

Site Architecture

NumericalSimulation

NTCPClient

Simulation Coordinator

ControlPlugin

NTCPServer

NEESpop

A/D

Event-drivenController

ControlSystem

Internet

Instruments

Specimen

Experiment

Actuator

Remote SiteMaster Site

July 21, 2005 AESE 2005 12

Network Architecture

• All sites connected via Internet2 backbone

• Local NEESpop’s as close to network uplink as possible (1-2 hops)

Internet2 (Abilene)

Network Backbone

(10Gbps)

July 21, 2005 AESE 2005 13

Simulation Protocol

1. Local simulation of all sites

2. Local test of specimen with remote sites simulated

3. Distributed simulation of specimens

4. Distributed test with single specimen, others simulated

5. Distributed test

July 21, 2005 AESE 2005 14

Simulation Results

• Non-linear behavior is due to the actual measured behavior of the physical specimens (Buffalo & Berkeley)

• Remaining (3) column substructures were simulated numerically

July 21, 2005 AESE 2005 15

Timing Results

• Simulation took 1074 sec. (about 17 minutes)

• Average step time 0.66 sec.• Four substantial delays

– Four ~ 23 sec.

– One ~ 3 sec.

• Multiple recovery from network timeouts in the TCP and Application layers

July 21, 2005 AESE 2005 16

Lessons Learned

• Scheduling multi-site tests is hard– Getting 5 different sites equipment and personnel free

at the same time is not easy

• Usage of some communication channel is a must– Audio/Video conference– Chat room

• To address all problems computer scientists and structural engineers must work together– They don’t always speak the same language

July 21, 2005 AESE 2005 17

Conclusions

• A distributed control strategy was implemented into NEESgrid to support relatively fast and continuous hybrid simulation testing methods with geographically distributed substructures

• Considerable improvements to overall testing time and network robustness

• Reliability of the results was improved by the minimization of force-relaxation and other strain-rate related errors in the experimental substructures

July 21, 2005 AESE 2005 18

Thank you