Dr. Vijay Raghavan Defense Advanced Research Projects Agency Information Exploitation Office

Dr. Vijay RaghavanDr. Vijay RaghavanDefense Advanced Research Projects AgencyDefense Advanced Research Projects Agency

Information Exploitation OfficeInformation Exploitation Office

Network Embedded Systems TechnologyNetwork Embedded Systems Technology

(NEST)(NEST)

November 12, 2003November 12, 2003

Workshop on Extreme ScalingWorkshop on Extreme ScalingArlington, VAArlington, VA

2UNCLASSIFED – FOR OFFICIAL USE ONLY

Meeting Purpose

To discuss the NEST plan to create a robust 10,000 node autonomous ad hoc sensor network in an operationally relevant environment for demonstrating the capabilities in the monitoring and protection of long linear structures, by the end of FY ’04

Analysis:• To discuss the NEST plan Evaluate platform, identify technological challenges,

identify crucial tasks, formulate metrics, and create a schedule

• Robust 10,000 node autonomous ad hoc sensor network Not a toy demonstration, two orders of magnitude larger than before

• Operationally relevant environment Not a laboratory demonstration, not a simulation, it will be in a real system

• Demonstrating the capabilities Okay, not a real fieldable system, but having all the capabilities for the mission (TRL 4.5)

• Monitoring and Protection The network detects, tracks, and classifies intruders

• Long linear structures Pipelines, borders, …

• End of FY ’04 Okay, we can extend this a little, but …


Deliverables of the meeting

1. Initial specification of hardware

2. Identification of tasks to be performed

3. Schedule of activities for FY ’04


Agenda

TIME TOPIC

0830 – 0900 Breakfast

0900 – 0930 Welcome / Introductions / Meeting Purpose (DARPA)

0930 – 1000 Extreme Scaling Overview (Puritan Research)• Background• Operational Challenges• A 0th Order NEST Solution

1000 – 1045 Technical Challenges in Extreme Scaling (UC Berkeley)• Lessons learned• Preliminary Analysis of Challenges

1045 – 1100 Break

1100 – 1145 Preliminary Technical Approach (The Ohio State University)• Overview of “A Line in the Sand”• Sensor Requirements for the Extreme Scaling Experiment• System Architecture, Process

1200 – 1300 Lunch

1300 – 1330 Preliminary Program Plan (DARPA)

1330 – 1430 Extreme Scaling Planning Session I (DARPA)• Review Operational Scenario• Discussion of Technical Challenges

1430 – 1500 Break

1500 – 1630 Extreme Scaling Planning Session II (DARPA)• Refine Program Plan• Identify Issues/Action Items• Wrap Up / Next Steps (DARPA)


7/9/2003

Extreme Scaling - 2004

Securing a Large Distributed Perimeter

Norman Whitaker

NEST Extreme Scaling Workshop

November 12, 2003Cano-Limon PipelineColombia


N sensors

• Unit cost (N)• Information flow (N)• Unit Memory( N)• Network Bandwidth(N)

Exfiltration ~ N log N

Information Flow

Scaling



300 kmborder

IRAQ

SYRIA

AL QAIM

IRAQ / SYRIA Border October 2003


25 km

dirt road

500m

500m

Sensor node with 50m range

Exfiltration node (PDA)

400 nodes/km2

Both sides

1% of Total Laydown

Schedule

3 month6 month

9 month12 month

100030006000

Full systemdemonstration

Specs

• Dismounts – armed and unarmed• Vehicles• 2 second latency• < $150/node• Pfa < 1 alarm/day• Pd = 0.95 at walking speed

“Kansas Pipeline” – Canonical 10,000 Node Problem


The Big Three Problems - MITRE

Sensor Range• Require a magnetometer

• The sensor range needs to be ~1.3x the node spacing.In this case ~65 m.

• Magnetometer ranges of 65 m are beyond the state of the art.Earth’s magnetic “noise” exceeds the signal by ~10x.

Over-the-Air Loading• This is a complex service:

Reliable multicast (i.e., with local retry).

May need group service.

• Very strong motivation towards incremental loading.Some application may need to run while others load.

Implies dynamic linking.Requires OS-style security


The Big Three Problems - MITRE

Sentry Service (power)• The required “on” subset of the network will be too large.Alternative drives up required sensor range.

Extending system life by raising node density is inferior to just using bigger battery

• Need temporal sampling.

• Need power savings of ~100x.

• Need the almost always off communication model.Maybe comm. deadline and comm. scheduling.


1. Timeline is too aggressiveJune 04 100December 04 1000June 05 10,000

2. Sensor mix: a. Magnetics unlikely to detect weapons at ranges > 1mb. PIR unlikely to be a cure-all c. Consider McKuen radars

3. Comms range seen as problematic

4. Need 4 X more PDA’s – Pfa / lifetime / latency very aggressive

Feedback from 9-22-03 Conference Call


5. Need for rock-solid over air loading capability needed to demonstrate scalability – and this is not currently in hand.Architecture work needed.

6. Power is a huge problem for a demonstration- Always on needed to achieve Pfa- Debugging and software loads are battery intensive- Lifetimes are currently far too short

7. Solid packaging needed (“one touch only”)

8. Need a separate contractor to actually camp in desertand deploy sensors

9. Next tier PDA work needed , as well as highest tier (GUI)

Feedback from 9-22-03 Conference Call (cont.)


10.Dynamic tagging seen as an alternate approach that demonstrates scalability without problems in static sensor laydown

11. Rush to production seen as a costly mistake. A disciplined,Step-by-step process with formalized acceptance criteria at eachStage was recommended to avoid a failure.

Feedback from 9-22-03 Conference Call (cont.)


1. Self-contained sensors

2. Self-contained static sensor clusters

3. Sensor clusters with handoff interactions

Systems that Scale Well


4. Sensor clusters with dynamic assignments

5. Robust systems that adjust to sensor dropouts, etc.

6. Systems which maintain only local state

Systems that Scale Well (cont.)


1. Any system with a non-zero Pfa per sensor node

2. Any non-robust system that requires multiple touches per sensor

3. Any system with performance that can be degraded by a malfunctioning, poorly placed, or imperfect node.

4. Any system that creates a disproportionate response to stimuli

5. Any system that does not plug into umbrella information systems

Systems That Do Not Scale Well


Conclusion

For most applications sensor nets are inherently scalable, but their implementations are often not

The path to practical scalable systems lies through careful extensible architecture work and bulletproof hardware/software design

It may be possible to demonstrate (but not fully test) a scalable system with fewer than 10,000 nodes


Lessons from the UCB Wireless OEP Demo and Issues in

Scaling Sensor WebsCory Sharp, Robert Szewczyk, Massimo Franceschetti, Prabal Dutta, David

Culler, Shankar Sastry

NEST Extreme Scaling Workshop,Washington, DC11/12/2003


Lessons from UCB Demo

Scaling Issues we set out to tackle Things that proved very valuable Things that proved problematic Things we wished we had Monitoring and Restart from the Beginning Scalable Network Reprogramming Foundations of Scale Recommendations


Scaling Issues We Designed For


Key Pragmatic Issues

Solve the specific problem • within time and $ budget

Reliably

Time, Time, Time, Time

Time to design, prototype and test what you think is a working hardware & software solution

• Not at scale

Time to manufacture a large number of nodes• Only get one shot (be conservative)

Time to test-fix-test-integrate-test-fix-… • Don’t see scaling issues until late in the process

• ~1000 person-hours in last two weeks

Time to assemble, reassemble, …• Human scaling limits


Hardware

Driven by specific application plan• Not generic experimental like Mica• Usage factors

outside, robots running over it, in-sun, possibly wet, …• Specific sensing modalities

Board designs, Geometric orientation, … Mechanical Design is central (co-design)

• Appn => package => boards

Foresee the needs of dev, debug, deployment process• Field UI, reset• Assembly, re-assembly, recharging process

Minimize exposure Lift radio off the ground


PEG Mechanical Design

Exposed components

Watertight compartment

ultrasoundMica2 dot

mag sensepower

battery

reflector

Collision absorption

O-ring Seal

Good thermal characteristics


Software

NEST service architecture as a beginning Greater number => simpler function Complete modularity of each subsystem

• Node positioning• Sensing and report• Mobile-to-mobile routing• Pursuer hear and navigate

Not just source concept, but full interfaces to allow testing in the field of each without the others to blame

Delay Binding Wherever Possible• Complete parameterization

Thresholds, update rates, ranges, … Management services Assume irregular, lossy connectivity


Lowest-tier nodes only performed entity detection• Local processing, threshold detection, announce• Simple Leader Election• Simple aggregation for position estimation

Little calibration Simple mobile-mobile reliable routing

• Solid tree rooted at landmark Complexity consolidated in higher tier

• Disambiguation• Trajectory Estimation• Navigation• Noise-adaptive control loop

Minimalism pays• Nice to have solid theory to support doing very simple things

Algorithmic Simplicity by Decomposition


Steps that proved invaluable


Management Services

Ident• Ping, Version, Compile time

Command line scripting• Check all report in

• Check config values

Config• Set/Get each potential parameter

Automated behaviors on update

• Save/Restore to Flash

• Human override

Reset Sleep / Wakeup


Simple and Solid

Once a node is assembled, if it responds to Ping, it should work

• Reliability of quick test was key

Complete testing of each major component at near-scale in isolation

• Clean stimulus, observable response

• Combining was trivial

Powerful receiver to see what was going on in much of network


Multiple Fall Backs

Localization: • fix the position based on network address• Transform of address• Explicit Set• Ranging

Routing• Single hop to base• 802.11 back channel to pursuer• Evader-pursuer back channel• Landmark routing

Excellent signal-strength based slow tree build Sensing Used most of them in testing

• Kept them in the code Reduced stress along the way


Things that Proved Problematic


Hardware

Biggest Risk Factor• Getting boards done and software working in time• Budget allowed only one production shot• Scale was limited by slowest production step

Antenna • Internal coiled antenna was dismal => external whip• Proved to be time consuming and problematic in assembly• weak mount point (10% loss)

Recharging – required disassembly (see above) Power conditioning introduced noise issues & leakage Unforeseen interactions

• Radio + magnetometer• Used the wrong kind of wire (duh!)

No self-guided assembly Modularity, while has many values, has a significant cost Too hard to see LEDs no physical reset Multiple antennas on pioneers interfered Differential GPS black magic TIME, TIME, TIME

• You don’t know what is wrong till late in the development


Software

Network programming fell apart for large app on many nodes• Dictated the development cycle

• Config was life saver

• Got solid single hop just after it was needed

• Current design has deep shortcomings (below)

Software Knowledge bottleneck• Nice abstractions that too few people knew how to use

Underestimated the amount of glue code between the tiers Wished we had

• Better Regression suites

• Logged Everything

• Much better operational deployment protocol and bookkeeping


Towards the Extreme Scaling Effort


Monitoring and Management of the Network

Determine what went wrong and take action• Extremely simple and reliable

• Ensure liveness, preserve access to network nodes, help with fault diagnosis

Networking monitoring• Enforce minimum and maximum transmission rates

• Verify minimum reception rate

Node health monitoring• Beyond battery voltage – sensor data monitoring

• Failure of a sensor an indicator of node health

• Detectable failures impacting lifetime– GDI humidity and clock skew experience

Program integrity checks• Stack overflow

Watchdog / deadman timer • Require attention from different parts of the system, reset if abandoned

• Address many different time scales

• Test low-level system components – timers, and task queue to ensure basic system liveness

• First use: min reception rate

Partial system shutdown• Flash/EEPROM writing under low voltage conditions, broken sensors, etc.

Fault handling• Error logging and/or reporting


Scalable Network Programming

Out of core algorithms, epidemic algorithms Basic primitive – reliable page transfer

• Both broadcast and point-to-point transfer• Fragment and transmit packets per page, vector ack, selective fix up, snoop,

consistency checking• Constraints: fit in ram, match with flash transfer size

Dissemination algorithms• Fast push, pipeline series of pages in space• Epidemic maintenance

Eliminate the “wrong image” problem New version: propagate page diffs

• Potentially, multiple levels of pages to organize code and metadata• System image: version number + vector of page hashes• Organize image to reduce #pages that change

Linearization of functions, holes Blue sky ideas

• default TinyOS network reprogramming kernel as the system restore checkpoint


Conceptual Issues in Scaling


Percolation theory Random graphs Distributed computing Distributed control Channel physics Distributed sampling Network information theory Network coding

Connectivity Routing Storage Failures Packet loss Malicious behavior Remote operation

Theory Practice


Beyond the disc abstraction Random connection model Small world networks Effect of interference What is connectivity ?

Example: Percolation theory

Prob(correct reception)


Connectivity in Practice

Effective Region

Transitional Region

Clear Region

DeterminesNode

spacing

many


1

Connectionprobability

d

Continuum percolationContinuum percolation

2r

New modelNew model

d

1

Connectionprobability

Beyond the disc abstraction


CNP

Squishing and squashing Shifting and squeezing

(sporadic) long links help the connectivity process

CNP=average number of connections per node needed for percolation

What can we prove


CNP

Is the disc the hardest shape to connect overall?

What about non-circular shapes?


Routing with occasional long links Navigation in the small world Connection with social science !

What about routing ?

Prob(correct reception)


Regular RandomSmall World

Watts Strogatz (1998)

can we use the few long links for routing?

Small World Networks


Kleinberg (2000) New model (2003)

• Nodes on the grid• Fixed number of contacts• Probability scales with distance

• Nodes at random on the plane• Random number contacts in a given region • Density scales with distance

each node has only local information of the network connectivity

Routing in a small world


Long, Good Links Valuable


S

T

d

Connections of z are Poisson points of density

-delivery occurs when msg is delivered within to target

Theorem about Routing in a Small World


S

T

d

Build networks where the number of neighbors scales as 1/x2

to obtain efficient routing

Bottom line


Red, Yellow, Green Lights: The Systems View

Define the application in levels• Minimal demo

• Intermediate demo

• Ideal demo

Technical challenges required to reach a level determine the severity of the issue


Red Light Challenges

Application definition• Unchanging, realizable specifications• Subject to sensor limitations (if any)

Hardware design• For the specific application requirements• Must have minimal human per-node interaction• Requires multiple revisions – get it all done it time

Network reprogramming Any suitable power source

• Sufficient for development and debugging (De)composable services and architecture Timeline for scalable solution Manufacturing 10,000 nodes


Yellow light challenges

Sensor limitations, sensing at scale• Better sensors

• If no sensors: use active tagging

At-scale simulation Enclosure with conveniences Integration of heterogeneity Watchdog timers


Green light challenges

Security and demonstration Possible renewable / rechargeable energy source Regression suites Logging of all demo data


Long Poles in the Tent: Conceptual View

Architecture of the Networks: need to find stable routing trees for low latency comms using long links and short links. Heterogeneity with some less power constrained nodes, and many

Algorithms for sensor web localization poorTheory based on “one shot” random networks. Networks are time varying and need stable features for routing.

Tracking of multiple entitiesDistributed Control based on the quality of information


Anatomy of a Sensor Node

Mechanical: Spike to ensure positive coupling with ground

Recharge in situ w/ solar cell and/or provide contacts?

Antenna placement > 3” off ground.

SMA antenna connector and real antenna (or maybe planar)

Crazy idea: why not stick a Yagi on there (Szewczyk & Culler) to ensure a long link (but will it be bi-directional?)

Power switch puts into ultra low power mode

Upside down camera and conical optics shields sensor from rain

Antennas

Power Conv

End Cap

Solar Cell

SMA Ant Con

PIR (4)

Camera

Mic (4)

CPU,RF,Mag Lens (4)

BatteryHolder

Spike

Battery(2)

Base

PowerSwitch

Catadioptric sSta

nd

off

s &

Wir

egu

ides

GigaAnt


MobiLoc: Mobility Enhanced Localization

Some localization schemes• Need special hardware support (Cricket, Calamari, AHLoS/Medusa)

• Do not need special hardware support (RSSI, hop count)

• Use mobile beacon(s) (NCSU)

Different idea: use range/angle to target, as measured by default sensors, to localize nodes


Thinking Outside the Convex Lens: Catadioptrics

Most cameras have thousands of pixels – far too many for efficient processing on motes

Use optics to focus light and SW on area of interest

Research: Design a catadioptric mirror and lens (“magic”) that allows motes to focus on a very small number of pixels and follow tracks emerging from and terminating at those pixels

Use simple differencing type algorithms for detection

Only monitor pixels adjacent to current target position


PIR: Cheap, Small, Long Range, Low Current

Use 4 PIR sensors, each with 90 degree FOV

Cost: < $5 each in 1K quantity

Range: ~ 10m on most sensitive setting

Use for passive vigilance?• Duty cycle? Settling time?

• Draws 0.5mA @ 3V-9V

Problem: since it is a differential sensor, likely experiences washout in bright sunlight


Lessons from Lessons from

A Line In the SandA Line In the Sand

for Project Echelonfor Project Echelon

Anish AroraAnish Arora

Mohamed GoudaMohamed Gouda

Ted HermanTed Herman

Sandeep KulkarniSandeep Kulkarni

Mikhail NesterenkoMikhail Nesterenko

November 12, 2003November 12, 2003November 12, 2003November 12, 2003


Brief Recap of

A Line In The Sand


Put tripwires anywhere—in deserts, other areas where physical terrain

does not constrain troop or vehicle movement—to detect, classify &

track intruders

ALineInTheSand: ConOps


Hand-placed 90 pre-configured nodes at known locations, containing

magnetometer & MIR sensors

System detects each intruder

System classifies each intruder as one or none of known intruder

types, using distributed influence field concept

System tracks intruder as it moves past different nodes

System visualizes intruder on computer attached to relay

Scenarios:

Soldier Carrying “Gun”

• Walks/runs across line on trails

• With another soldier on different trail & after other soldier on same trail

Car• Runs across line on trails at 5-20mph

Civilian• Walks across line on trail

The Line under Attack: Reliability of Middleware Services

• Several motes in a region are turned off & then on again

• Some motes are moved around

• Timing & Routing services are shown to self-heal

25’

90 motes

Trails

65’

ObservationTent

Field Experiment Layout & Scenarios


Our Focus

Co-design of NEST-middleware & sensing/application

components that is

accurate (i.e., low false negatives & low false positives)

robust (i.e., tolerates uncertain environment)

cost effective (i.e., using middleware on dense set of cheap

sensors

vs. sparse set of resource rich sensors)


Lessons Learned from

A Line In The Sand


1. The Network Is Unreliable

The application generates relatively high message traffic

Message loss is a dominant problem• channel fading

• interference (interference range much larger than communication range)

• Problem: low end-to-end reliabilitye.g.: if each link has 90% reliability, success rate over 6-hop routes is 50%

Message corruption is also an issue• packets got corrupted in the air, even if CRC checking is used

• Problem: corrupt state of key middleware services, e.g. time sync

focus on dealing with problems resulting from network

unreliability


Impact on Application

Influence field concept for classification easily yields accuracy when

network is reliable

As network unreliability increases, accurate classification becomes harder

• fewer messages lesser separation between target classes

• lost messages object can be classified as multiple smaller objects

application design must handle network unreliability

• late, out-of-order messages, e.g. by maintaining network time at classifier

• network jitter, e.g. by tuning latency and classification thresholds

• false positives and outliers


Impact on Network Protocols

Added implicit-ACK based communication protocol to increase

reliability

• each message has ACK field, which presents “aggregated picture” of local queue

• ACK-based timeout & retransmit used to deal with loss

• ACK-based flow control used to reduce contention

• implicit ACKs save bandwidth & reduce contention as well as delay

increased message reliability yields tradeoff with increased delay

• parameter tuning for sweetspot was challenging: 50% delivery within 13s

• protocol choice still under study: TDMA experiments underway


Aside: TDMA

Developed TDMA schemes for grid based networks

Schemes are optimized for broadcast, convergecast, & local gossip

Simulation results show :

• 100% end-to-end delivery

• Slightly increased delay over CSMA based algorithms that suffer

significant end-to-end loss


Impact on Network Protocols (contd.)

Redesigned time synch to be robust wrt periods of poor connectivity

• used a structure-less approach, to :

avoid dealing with structure partitioning

exploit path diversity (converges fast too)

• used message redundancy & application level reasoning to deal with corruption

• side effect: can handle mobility of motes

increased synchronization robustness yields tradeoff with decreased

accuracy (~100us/hop)

• statistical skew estimates improve accuracy (~15us/hop) but computation is costly

• some timestamps still get corrupted in spite of replication


2. Unanticipated Bad Behaviors Occur

Unpredictable behavior of some nodes occurred (& was hard to

avoid), due to various reasons:• incorrect program download

• depleted battery levels

• debonded or overheated sensors

• …

Effect is often serious, e.g., drowns out “good” traffic or

compromises key services

detect & contain/correct Byzantine & transient misbehavior

• based on local invariants or contracts at node/service levels;

e.g. blocked nodes that communicated events at a rate higher than normal


3. Testing should be at Scale

Sevices/apps designed & tested for n nodes do not work for 10n nodes !

E.g., reliable communication (scaling in network traffic)

95% reliability when tested with 16 nodes

reduced to 50% at 100 nodes with increased background & application traffic

E.g., time synchronization (scaling in processing complexity)

improved accuracy when tested with 8-10 nodes, by handling in radio stack layer the

offset calculations for timesync messages

failed at higher scale : event loss → state corruption → arbitrary jumps in timesync

at design time, simulate at target scale

• develop traffic model generator, for each service, at target scale

btw, event loss is fundamental to platform !

• both node & network capacity pushed to limits → source of bad behavior


4. System Tuning should be Easy

Achieving co-design objectives involved tuning ~15 service

parameters• transmission power level, number of retransmissions, periodic service frequencies, application thresholds…

• reprogramming modes for each parameter change is overkill

Scalable, distributed service for over-the-network

monitoring/controlling of application parameters

• e.g., our walkabout ‘Commander/Query’ service

• automate addition of monitoring interfaces for parameters of each

component


5. Reprogramming

During reprogramming, loss of program capsules is random, not

bursty

• some capsules are lost due to source errors

all receivers miss them

• most other capsules are lost due to receiver errors

only a small subset of receivers miss them

• retransmission requests from motes often lost due to collision

Reduce reprogramming requests, especially when some capsules

are lost only by a small subset of motes

• e.g., FEC-based components for reducing retransmission → most lost

messages are recovered even while programming several motes at once

• currently evaluating several approaches for multihop case


6. Other Challenges

Micropower Impulse Radar (MIR) interfere with other MIRs

• no dithering on RF pulse timing or use of pseudo-random codes

• interference at close range (a few feet) Non-deterministic interference between magnetometer & MIR

• magnetometer failed to detect when MIR physically present (but not on)

• MIR eventually failed to detect in the presence of magnetometer

• did not use a schedule to coordinate different sensors Mechanical

• thermal effects severely reduced performance

• heavy rain took a mote’s life (and left a smoke trail)

• two screws is not enough Operating System

• felt the lack of static analysis tools for determining if tasks are “schedulable”


8. Signal Processing

Noise is not Gaussian, Laplacian, or IID Noise is correlated Noise mean and variance is time-varying (SNR is time-varying) Signal is not deterministic Signal has many nuisance parameters Signal processing capability severely limited on motes Difficult to distinguish signal from correlated noise Sensors were not calibrated prior to deployment Sensors drifted and became desensitized over time Signal bias and scale unknown MIR digital sensor outputs just plain don’t work and required statistical signal processing on analog signal to make usable

PFA PD; conversely PD PFA

sensing and signal processing require serious effort

node capacity is already being taxed


Need CFAR and Hysteresis to PFA, PD

Bottom Line: Need adaptive signal processing.


Echelon


Sensing

Sensors

• Magnetometer

• Passive Infrared

• Acoustic

• Seismic

• Radar


Sensing Options


2-Tier Architecture: Section Line

Lower Tier: Section Size: 100 motes

multihop connected (upto 5-7 hops) to 1-2 aggregation devices (iPAQ)

Sensing range: 10m

Comm. range: 20m or more (for mote) 1km (for iPAQ, 102.11g or a class protocol)

Capability: detect & classify targets at line periperhy possibly in conjunction with neighboring sections

iPAQ role in signal processing / application tasks a small number (e.g., 2) of individual targets

Robust wrt:• mote (&sensing ) loss

• path loss to ideal aggregation station


Higher Tier: PDA network

Multi-hop PDA network glues together sections into a line• it’s convenient to logically group 1-hop PDA-connected sections into a segment

• segment ~1km long

• segments participate in classification & tracking (of a somewhat larger number of targets)

• exfil to demonstrator node


Configuring the Line

Higher density at perimeter Simplest: Density only at perimeter

Grid: Sparse/Dense coverage Lattice:

ww

l

r


2003 2004

Jul

Aug

Sep

t

Oct

Mar

Apr

May

Jun

No

v

De

c

Jan

Feb

Mote Enhancement Design

Sensor Board Design

Packaging Design

Testbed Development

Sample Node Tests

Prototype Manufacture

Localization Testing

Middleware Experiments

Tier 2 Tests

Prototype Validation

Mote Redesign

Manufacture first article (1000)

Section (Tier 1) Testing

Tier 2 Interaction Testing

First Dry Run

Dry Run at Site

Jul

Aug

Sep

t

Oct

Mar

Apr

May

Jun

No

v

De

c

Jan

Feb

Milestones


Process

Always have a working prototype (continuous refinement)

Flexible, friendly, & organic collaboration

Regular interactions by phone & on some occasions in person

Open process, especially since zero-maintenance during

project lifetime design of packaging, sensing, & architecture

Single place to access & share materials

Liasons per team


Concerns

Volunteers for signal processing/application development ?

Underprovisioning

• of sensor density

• of cost for infrastructure, storing, transporting, testing, controlling access, maintenance, space, programming, management

• of time

Overprovisioning• of communication range

• current generation of motes will be pushed to its limits of sensing & communication capabilities


Preliminary Program Plan

Schedule/MilestonesRoles and ResponsibilitiesDeployment Considerations for Extreme Scaling ExperimentSystem MetricsTechnology Transition


Preliminary Program PlanSchedule and Performers

ActivityPrimary

Performer

System Analysis•Define Operational Scenario•Define Operational/Technical Rqmts.•Document System Architecture•Document Performance Metrics

DARPADARPAOSUOSU

System Design•Design System Hardware•Design System Software

CrossbowOSU

System Development•Enhance XSM Algorithms/Software•Enhance Relay Node Software•Enhance Display Unit Software

DevelopersDevelopersDevelopers

System Production•XSM Production•Relay Node Production•Display Unit Production

CrossbowCrossbowOSU

System Integration• Integrate XSM / Relay Node / Display Unit Components

OSU

System Testing/Demonstration•System Component Testing•Preliminary Integration Testing•Final Integration Testing•System Demonstrations

DevelopersOSUOSUOSU

Program Management•Extreme Scaling Workshop•IPT Meetings•Status Conference Calls

DARPADARPADARPA

Nov. ’03

Dec. ’03

Jan. ’04

Feb. ’04

Mar. ’04

Apr. ’04

May ’04

Jun. ’04

Jul. ’04

Aug. ’04

Sep. ‘04

Build 1 DeliveredBuild 1

DeliveredBuild 2

DeliveredBuild 2

DeliveredBuild 3

DeliveredBuild 3

Delivered

Component Demo

Integrated Demo #1

Final Demo

Integrated Demo #2


Technology DevelopmentTechnology DevelopmentTechnology DevelopmentTechnology Development

Preliminary Program PlanRoles and Responsibilities

Middleware ServicesMiddleware ServicesClock Sync (OSU, UCLA)Group Formation (OSU, UCB)Localization (UIUC)Remote Programming (UCB)Routing (OSU, UCB, UIUC)Sensor Fusion (OSU)Sentry (UCB, UVA)

Middleware ServicesMiddleware ServicesClock Sync (OSU, UCLA)Group Formation (OSU, UCB)Localization (UIUC)Remote Programming (UCB)Routing (OSU, UCB, UIUC)Sensor Fusion (OSU)Sentry (UCB, UVA)

Display UnitDisplay UnitOhio State

Display UnitDisplay UnitOhio State

GUI/Application LayerGUI/Application LayerOhio State

UC Berkeley

GUI/Application LayerGUI/Application LayerOhio State

UC Berkeley

MAC LayerMAC LayerUC Berkeley

MAC LayerMAC LayerUC Berkeley

Xtreme Scaling MoteXtreme Scaling MoteCrossbow Technology

Xtreme Scaling MoteXtreme Scaling MoteCrossbow Technology

Relay NodeRelay NodeCrossbow Technology

Relay NodeRelay NodeCrossbow Technology

Systems IntegrationSystems IntegrationOhio State

Systems IntegrationSystems IntegrationOhio State

Transition PartnersTransition PartnersUSSOUTHCOM, U.S. Customs & Border Protection, USSOCOM, AFRL

Transition PartnersTransition PartnersUSSOUTHCOM, U.S. Customs & Border Protection, USSOCOM, AFRL

Military ConsultantsMilitary ConsultantsCNS Technologies, DBBL, Select Graybeards

Military ConsultantsMilitary ConsultantsCNS Technologies, DBBL, Select Graybeards

SensorsSensorsVarious

SensorsSensorsVarious

Operating SystemOperating SystemUC Berkeley

Operating SystemOperating SystemUC Berkeley

Application ToolsApplication ToolsOhio State

UCLAUC Berkeley

Application ToolsApplication ToolsOhio State

UCLAUC Berkeley

Testing and EvaluationTesting and EvaluationOhio State, MITRE, CNS Technologies

Testing and EvaluationTesting and EvaluationOhio State, MITRE, CNS Technologies



Systems Integration• Define system architecture and hardware/middleware services required• Build and maintain integration testbed• Integrate system components from Technology Developers• Test, verify, and validate system components and overall system• Manage field testing/evaluation

Technology Development• Develop system components (both hardware and software)• Fabricate system hardware components (including packaging)• Develop and test algorithms and middleware services

Testing and Evaluation• Plan field testing/evaluation• Schedule testing facilities (e.g., MOUT site, other DoD facility)• Support system deployment at testing facilities• Conduct system training as necessary• Define system metrics and document system performance

Transition Partners• Support CONOP and TTP development• Facilitate meetings between DARPA and User Communities• Support technology transition activities (e.g., development of MOAs/MOUs)

Military Consultants• Support CONOP and TTP development• Interact with System Integrator, Technology Developers, and Transition Partners

Deployment Considerations for Extreme Scaling Experiment

Al SciarrettaCNS Technologies, Inc.

[email protected]

92

Nominal Assumptions

• Objective experiment:– Approximately 10,000 sensors – Sensors can be placed about 50m apart on

average (not a uniform distribution)– Sensor field: Approximately 1 km x 20 km– Hand emplacement

• There will be two smaller scale (proof of concept) experiments leading up to the objective experiment

93

Characteristics of Experiment Site

• Relatively flat area– Able to survey/mark off 1 km x 20 km site– Allow for observers to see large section of experiment site

• No large physical obstructions (e.g., buildings) to line-of-site communications– Small obstructions (e.g., small rocks) okay

• Should be easy to walk around area and set sensors on ground

• Consistently good weather (no rain, no strong winds, etc.)– Allowing sensors to stay out for days

• Located on military base– Site can be guarded

• Sensors deployed on day 1 and remain in place until end of experiment (days later)

94

Emplacement Rate

• One person: assume emplacement of 40 sensors/hour– Assuming each person can carry 40 sensors– Walking time of 4 km/hr (about 2.5 mph) 1.11 m/s

• Movement time ~ 1 min from sensor to sensor– 56 sec to move 50 m distance between sensors

– Add a few seconds to set sensor

– Allow approximately 20 minutes per hour for person to receive sensors, get to the area, get oriented throughout the emplacement area, change direction, etc.

95

Emplacement Time** & Number of Personnel

• Given: – 440 sensors per 1 km x 1 km square– 40 sensors/hour

• *Add 1 person for every 10 to manage the emplacement effort and distribute sensors

• ** These numbers and the ones on the previous page can be improved by taking into account that the average inter-sensor distance is much less (Vijay)

Emplacement Time

(hours)

Number of Personnel (1 km x 1 km)

Number of Personnel (1 km x 20 km)

1 11 220*

2 6 110*

3 4 74*

4 3 55*

96

Additional Considerations

• Site should be out of the way of normal base operations– No interference from base vehicles/personnel

• Environmental considerations (i.e., batteries) will dictate that every sensor be picked up at end of experiment – Sensor should have a bright, fluorescent color (e.g.,

orange) to assist in sensor pick-up.• Can color the case or add a small colored flag

– Flag could increase emplacement time

• On-off switch must be simple– Shouldn’t depend on emplacement personnel to turn

on the sensor• Sensors will have to self-localize

97

Additional Considerations(continued)

• Site should have personnel available to assist in emplacement of sensors and guard site– Tends toward active military base

• Experiment site will need some surveying– Sensor Field

• Mark outer boundaries• Mark 50 m intervals along boundaries• Mark boundary/intervals with tape, stakes, etc.• Survey in the “hub” sensors• Individual sensor site locations shouldn’t need to be surveyed

– Assumes self-localization– Intruder paths need to be surveyed

• Provides “ground truth”

• Sensor batteries should not require replacement for duration of experiment

98

Potential Sites

• Naval Air Weapons Facility, China Lake– 150 miles NE of Los Angeles– Encompasses 1.1 million acres of land in

California's upper Mojave Desert, ranging in altitude from 2,100 to 8,900 feet

• Varies from flat dry lake beds to rugged piñon pine covered mountains.

– Weather should be consistent• Issue: If experiment in summer, heat may be a

problem for sensors

99

Potential Sites(continued)

• Need further investigation– Fort Bliss, TX

• Home of U.S. Army missile test sites• Arid environment (heat in summer an issue)• Not flat – has dunes

– Nellis AFB (near Las Vegas, NV)• Large Air Force test site• Not sure of availability of large expanse of land

– Bombing sites may be off-limits

100

To Further Assess Sites

• Need – Experiment schedule

• Approximate times of technical tests and final experiment

– Characteristics of sensors• Heat tolerance• Weather tolerance• Capability of comms in flat to rugged/forested

terrain– Assume 50 m transmission distance


Preliminary Program PlanSystem Metrics

Operational MetricsOperational Metrics Latency

• <10 seconds (90% of the time) Probability of False Alarm (Pfa)

• <1 per day Probability of Detection (Pd)

• 0.99 for vehicles• 0.95 for dismounts

Classification• TBD

XSM Node Lifetime• TBD

Relay Node Lifetime• TBD

Network Deployment time• <1 hour

Network Uninstallation time• < 1 hour

Cost• Per node• Per unit area covered

Operational MetricsOperational Metrics Latency

• <10 seconds (90% of the time) Probability of False Alarm (Pfa)

• <1 per day Probability of Detection (Pd)

• 0.99 for vehicles• 0.95 for dismounts

Classification• TBD

XSM Node Lifetime• TBD

Relay Node Lifetime• TBD

Network Deployment time• <1 hour

Network Uninstallation time• < 1 hour

Cost• Per node• Per unit area covered

Technical MetricsTechnical Metrics

Robustness• Of node

• Of network

Sensor coverage Sensor density Effective bandwidth

• Of first-tier network

• Of second-tier network

Endurance• Of node

• Of network

Technical MetricsTechnical Metrics

Robustness• Of node

• Of network

Sensor coverage Sensor density Effective bandwidth

• Of first-tier network

• Of second-tier network

Endurance• Of node

• Of network


Preliminary Program PlanTechnology Transition

Transition Partners

Transition Process• Joint DARPA/Partner identification of operational challenges and potential solutions

• DARPA development of technical solution to operational challenges

• Joint evaluation of technical solution

• Joint Memorandum of Agreeement (MOA) / Memorandum of Understanding (MOU)


Extreme Scaling Planning Session

Technical challenges• Hardware issues

• Technical shortfalls/challenges in the extreme scaling experiment

• Remote programming

• Software Tools

Process• Tasks, Who does what

• Methodology, Tools for systems integration

Refine Program Plan and Schedule Action Items Other issues


Hardware Issues

Current limitations New features needed/desirable Packaging considerations Supermote and relay node requirements


Current Limitations

Larger SRAMFaster processorMore comms bandwidthRobust antenna solution (pre-soldered or on-board)ADC/radio modules freeze if a hardware interrupt is lostRace conditionsWider ADCSet/reset for magnetometerMore magnetometer rangeSoftware control for radioBattery power indicationMIR interference at close rangeRadio/MIR affected by ground absorptionOverheating effects on MIR and magnetometerSolar Cell


New Features

Sensor suite• Passive IR, magnetometer, acoustic/seismic

• Performance characteristics (range)

Switchless turn-on, visual/acoustic aids for deployment/recovery

Supermote• What kind of device (e.g., iPAQ, Stargate)

Relay nodes


Packaging Considerations

Ruggedness• Antenna

• Battery holder detaches from the board easily

Weather-proofing • Light rain

• Heat

Deployment• Stake or tripod to get stable connection to earth

Issues not considered• Stealth, camouflaging

• Physical security


Shortfalls and Challenges


Remote Programming

What are the desirable features?• Time to reprogram (order of a few minutes for a 100 nodes deployed in a 100m by 100m area)

• Area coverage

• Robustness

• Elementary security

• Issues – memory overrun, mote in infinite loop, …

• Multihop

• Ability to use the second-tier supermote (or not) for higher gain xmit


Tools

Extend Tossim and TinyvizScale-invariant Simulator

• What are the desirable features in this?

Tools for in-field deployment, testing, recoveryLoggersNetwork monitoringNode health monitoring, network health monitoring


Process

Tasks• Put time line on high-level tasks

• Assign POC for each

TBD by OSU (Anish) in consultation with DARPA



Systems Integration• Define system architecture and hardware/middleware services required• Build and maintain integration testbed• Integrate system components from Technology Developers• Test, verify, and validate system components and overall system• Manage field testing/evaluation

Technology Development• Develop system components (both hardware and software)• Fabricate system hardware components (including packaging)• Develop and test algorithms and middleware services

Testing and Evaluation• Plan field testing/evaluation• Schedule testing facilities (e.g., MOUT site, other DoD facility)• Support system deployment at testing facilities• Conduct system training as necessary• Define system metrics and document system performance

Transition Partners• Support CONOP and TTP development• Facilitate meetings between DARPA and User Communities• Support technology transition activities (e.g., development of MOAs/MOUs)

Military Consultants• Support CONOP and TTP development• Interact with System Integrator, Technology Developers, and Transition Partners


Refine Program Schedule

OSU (Anish) to refine program schedule in consultation with DARPA in time for the PI meeting


Action Items

Application Definition• OSU (Anish), Puritan Research (Norm), MITRE (Alex and Ken) are to create an application definition that will specify the extreme

scaling plan. (Dec PI meeting)• Anish will refine the schedule that Larry provides, update list of tasks, and provide metrics for dry runs as well as final field experiment

(Dec PI meeting) Power Management

• UCB is the lead on this component Localization

• UIUC is the lead on this component; will provide a demonstration of localization in an 100m x 100m grid with 100 nodes spread about 10m apart supporting a nominal accuracy of 1ft by Jan 31, 2004.

Deployment• CNS Technologies (Al Sciarretta) will make further plans after discussion with OSU and DARPA. (Dec PI meeting)

Clock Synchronization• UCLA is the lead on this component; they will collaborate with other efforts (such as Iowa, UCB, Vanderbilt) to deliver a component

that could possibly leverage the second-tier supermotes in the future, by Jan 31. Accuracy and further work to be determined following application definition.

Sensor Suite Study • OSU is to report on the use of PIR, magnetometer, and acoustic/sensor nodes as the appropriate complement of sensors for the

ESE; will interact with Crossbow; this report will influence the set of metrics, the deployment plan, localization requirements, and the tool development. (Dec PI meeting)

Remote Programming• UCB is the lead on this; they will provide a first-class remote programming tool that can be used for in-situ reprogramming of mote

fields (roughly a 100m x 100m grid of 100 nodes in a “few minutes”) by Jan 31, 2004. Support Infrastructure

• OSU and Berkeley will prepare items for further discussion of the support infrastructure needed for the ESE; the support infrastructure will include tools for simulation, testing, evaluation, …; other instrumentation needed for debugging and ESE development etc. (Dec PI meeting)

Sensor Fusion• OSU is the lead on this component; they will collaborate with UCB, UCLA, and other teams that can provide the expertise that’s

needed

Documents

Dr. Vijay Raghavan Defense Advanced Research Projects Agency Information Exploitation Office