Improving NEST Performance Using Surrogates

Improving NEST Performance

Using Surrogates

C I R LC I R L

David W. EtheringtonAndrew J. Parkes

Matt Ginsberg

Project Status: Dec 16, 2003

University of Oregon

C I R LC I R L

Administrative

• Project Title: Improving NEST Performance Using Surrogates

• Program Manager: Vijay Raghavan

• PI Names: David W. Etherington, Matthew L. Ginsberg, Andrew J. Parkes

• PI Phone Numbers: 541-346-{0472, 0471, 0434}

• PI E-Mail Addresses: {ether, ginsberg, parkes}@cirl.uoregon.edu

• Company/Institution: CIRL/University of Oregon

• Contract Number: F33615-02-C-4032

• AO Number:

• Award Start Date: 9/12/2002

• Award End Date: 9/11/2005

• Agent Name/Organization: Ed DePalma, AFRL

C I R LC I R L

Subcontractors and Collaborators

• Subcontractors– none

• Collaborators– none

Problem and Challenge New Ideas

FY04 Schedule

Application of surrogates to distributed reasoningeasy-to-measure standins for properties of interest

Transition behaviors can be used to justify use of surrogates for system behavior prediction/control

Extend notion of relaxation to approximate relaxationsExplore the interaction between structure and transitions

Improving NEST PerformanceUsing Surrogates

Avoidance of dangerous regions; develop tractable mechanisms for design/control/prediction

Find manageable surrogates for critical properties

1QFY04• Demonstrate use of local surrogates in random NESTs• Determine thresholds for static global surrogates

2QFY04• Discover local surrogates for static properties of interest• Use static global surrogates to predict/control structured NESTs

3QFY04• Develop temporally structured NEST testbed/generator• Determine thresholds for static local surrogates

4QFY04• Identify functional dynamic properties in temporally structured NESTs• Use static local surrogates to predict/control structured NESTs

Impact

Simplified design and control of performanceSimplified performance prediction/modeling with

uncertain configuration informatione.g., prediction of capabilities under various

attrition/failure modelsPredict/determine maintenance/replenishment

requirements under uncertain field conditionse.g., model tradeoffs of capabilities vs costs

Q1 Q2 Q3 Q4

Etherington, ParkesCIRL, University of Oregon

cost

quality

approx-imation

full propertyparam

estimation

pure surrogates

estimation + phase xition

C I R LC I R L

Problem Description/Objective

• Problem: NEST systems induce difficult design and control problems– these impair our ability to achieve expected

benefits like robustness and speed

• Goal: develop the ability to predict, analyze, and bound NEST performance “in the large”

• Approach: use surrogates: properties that are easy to measure/control yet strongly correlated with real objectives– identify “hard” properties of interest– exploit structure to find surrogates

C I R LC I R L

Current Technical Approach

• Identify hard problems:– model technical problems of interest

• experiment to determine transition points/control variables• discrete/continuous analysis of identified parameters

– predict scaling behavior

• Apply appropriate tools to find surrogates– simple control behavior

• thresholds & experimental bounds on control variables

– complex behavior• group theory (ZAP)• constraint weakening/strengthening

C I R LC I R L

Changes in Technical Approach

• Deëmphasized development of synthetic generators/surrogates to more quickly connect to NEST demonstration platforms.

• Reëmphasized computational hardness of target problems, as well as utility for NEST.

• Reëmphasized general modeling as opposed to search for specific surrogates.

C I R LC I R L

Progress Since Last PI Meeting

• Built NEST-generating system– parameterized distribution, connectivity, etc.

• Identified “pseudo-density” connectivity surrogate in structured NESTs– covers realistic distributions, interesting properties– identified thresholds controlling connectivity

• ‘Twisted tree’ disjoint spanning tree surrogate– demonstrated on various, non-idealized, distributions

• Discovered group-theoretic structure exploitation– potential exponential compaction of discovery process

Details in technical presentation, following.

C I R LC I R L

Deliverables and Publications

Generalizing Boolean Satisfiability I: Background and Existing Work. Heidi E. Dixon, Matthew L. Ginsberg, and Andrew J. Parkes: to appear in JAIR, 2004.

Generalizing Boolean Satisfiability II: Theory. Heidi E. Dixon, Matthew L. Ginsberg, and Andrew J. Parkes: in preparation, to be submitted to JAIR.

Generalizing Boolean Satisfiability III: Implementation. Heidi E. Dixon, Matthew L. Ginsberg, and Andrew J. Parkes: in preparation, to be submitted to JAIR.

Scaling Properties of Pure Random Walk on Random 3-SAT. Andrew J. Parkes. Proceedings of the Eighth International Conference on Principles and Practice of Constraint Programming (CP2002). Published in Lecture Notes in Computer Science, LNCS 2470. Pages 708--713.

Easy Predictions for the Easy-Hard-Easy Transition. Andrew J. Parkes. Eighteenth Nat’l Conference on Artificial Intelligence (AAAI-02)

Likely Near-Term Advances in SAT Solvers. Heidi E. Dixon, Matthew L. Ginsberg, Andrew J. Parkes, at MTV-02.

Inference methods for a pseudo-Boolean satisfiability solver. Heidi E. Dixon and Matthew L. Ginsberg. AAAI-02.

Six further papers are currently in preparation for submission to AAAI-03, titles will be added after the blind review period has expired.

C I R LC I R L

Success Criteria

• Metrics– predictive accuracy of discovered surrogates – utility of the control surrogates that are discovered– design process simplification through surrogates– generality/reusability of methodology

• Decision points– ability to generate predictive/control models for static

problems of interest– ability to demonstrate transitions implying existence

of, and to identify, surrogates for those problems

C I R LC I R L

Project Plans: Collaboration

• Collaborate in extreme scaling effort– help identify hard, make-or-break, issues– model problematic aspects of “ideal” approach– develop surrogates that enable minimal functionality

Metric: impact of models, problems, surrogates found

• Assist in exploitation of existing surrogates– twisted-path implementation, etc.

Metric: degree of successful exploitation

• Develop new surrogates– exfiltration, sentry/relay/sleep issues, scalability, etc.

Metric: utility for design or modeling

C I R LC I R L

Project Plans: Modeling

• Study problem-distribution’s influence on surrogates– Payoff: help find useful, reliable, surrogates– Risk: open ended; difficult to determine right distributions

• Strengthen/weaken constraint-set to identify surrogates– Payoff: provide systematic methods– Risk: unproven; may not provide effective mechanism

• Off-line search (characterize search space, and identify useful gradient indicators that aid convergence)– Payoff: improved ability to find surrogates automatically– Risk: required understanding of the underlying search space

may be slow in coming

Metrics: in all of these, the metrics will be predictive accuracy and the ability to find useful surrogates

C I R LC I R L

Random NESTs

Project Schedule and Milestones

Structured NESTs

Temporally structured NESTs

Upcoming milestones:• demonstrate use of local, static surrogates in structured NESTs• discover local, static surrogates in structured NESTs• determine local and global static thresholds• elaborate network generator to produce more sophisticated NESTs

C I R LC I R L

Specific Milestones

1. Random NESTs/synthetic properties1. develop parameterized network

generator

2. Identify static properties of interest

3. Discover surrogatesa) Identify globally observable surrogates

b) Identify local surrogates

4. Determine thresholdsa) for global surrogates

b) for local surrogates

5. Build demonstration systemsa) use global surrogates to predict

behavior

b) use of local surrogates

Legend:

On schedule

Partially completed

Delayed

2. Structured NESTs/synthetic properties1. Develop network generator based on

community specifications/needs

2. Identify functional properties

3. Discover surrogatesa) Discover global surrogates

Notes: • As described previously, efforts on

1.3.b, 1.4.b, and 1.5.b were suspended in order to more quickly move the project to domains of practical interest to the NEST community.

• Continuing work is expected to be done to make the system developed in 2.1 more widely useful to the community.

C I R LC I R L

Technology Transition/Transfer

• N/A

C I R LC I R L

Program Issues

• N/A

C I R LC I R L

Technical Progress Summary

• Modeling of realistic sensor deployments– developed simulator for experiments– identified transition behavior in big-brother problem

• Surrogates for connectivity– pseudo-density controls many measures of interest

• Fairly robust routing– twisted trees: minimal disjoint spanning trees surrogate

• Group-theoretic structure exploitation (ZAP)– exponential simplification of certain search problems– potential for automatic generation of surrogates

• by simplification of generators for relevant groups

C I R LC I R L

The Overall Plan

• Surrogates provide computationally effective alternates to operational features that are impractical to evaluate directly

C I R LC I R L

The Overall Plan


• Example: satisfiability in random 3-SAT– Must be possible to evaluate surrogate quickly

C I R LC I R L

The Overall Plan


• Example: satisfiability in random 3-SAT– Must be possible to evaluate surrogate quickly– Quality/feature being modeled must have real

operational use

C I R LC I R L

The Overall Plan



operational use– Feature being modeled must be beyond the reach

of existing capabilities

C I R LC I R L

The Overall Plan





C I R LC I R L

The Overall Plan


• Example: satisfiability in random 3-SAT– Surrogate is clause/variable ratio– Quality/feature being modeled must have real



C I R LC I R L

The Overall Plan


• Example: satisfiability in random 3-SAT– Surrogate is clause/variable ratio– Satisfiability is the coin of the realm in this domain

ffff– Feature being modeled must be beyond the reach


C I R LC I R L

The Overall Plan


• Example: satisfiability in random 3-SAT– Surrogate is clause/variable ratio– Satisfiability is the coin of the realm in this domain

ffff– 3-SAT is NP-complete

C I R LC I R L

Multipath Routing: July





C I R LC I R L



• Example: biconnectedness– Must be possible to evaluate surrogate quickly– Quality/feature being modeled must have real



C I R LC I R L



• Example: biconnectedness– Density as a surrogate (V2/N)– Quality/feature being modeled must have real



C I R LC I R L



• Example: biconnectedness– Density as a surrogate (V2/N)– Quality/feature being modeled must have real

operational use– Not hard! Vijay: linear-time algorithms exist

C I R LC I R L



• Example: biconnectedness– Density as a surrogate (V2/N)– Not useful, either!

– Not hard! Vijay: linear-time algorithms exist

C I R LC I R L



• Example: biconnectedness– Density as a surrogate (V2/N)– Not useful, either! Long paths


C I R LC I R L



• Example: biconnectedness– Density as a surrogate (V2/N)– Not useful, either! Long paths, impossible to find


C I R LC I R L



• Example: biconnectedness– Density as a surrogate (V2/N)– Not useful, either! Long paths, impossible to find

and who cares about disjointedness anyway?– Not hard! Vijay: linear-time algorithms exist

C I R LC I R L

Multipath Routing: December


• Example: ability to find multiple, short, nearly disjoint routings

C I R LC I R L



• Example: ability to find multiple, short, nearly disjoint routings– computationally viable surrogate

C I R LC I R L



• Example: ability to find multiple, short, nearly disjoint routings– density (V2/N) still works

C I R LC I R L



• Example: ability to find multiple, short, nearly disjoint routings– density (V2/N) still works– problem has to be hard

C I R LC I R L



• Example: ability to find multiple, short, nearly disjoint routings– density (V2/N) still works– problem is known to be NP-hard (disjoint case)

C I R LC I R L



• Example: ability to find multiple, short, nearly disjoint routings– density (V2/N) still works– problem is known to be NP-hard (disjoint case)– has to be useful

C I R LC I R L



• Example: ability to find multiple, short, nearly disjoint routings– density (V2/N) still works– problem is known to be NP-hard (disjoint case)– clear impact on both robustness and power

consumption

C I R LC I R L


• Biconnectedness is too easy• Robust routing is too hard• Fairly robust routing is just right

• And curiously enough, V2/N is a surrogate for all of them

C I R LC I R L

Fairly Robust Routing

• Goal: “approximately disjoint” spanning trees– low cost to construct (time/communication)– short– well dispersed

• Approach: exploit weak localization information– augment standard flood fill with directional bias– produce a loose “spiral” in toward root

• Results: near-shortest-path trees– good spatial dispersion– low cost to construct– no spanning tree needed (extreme scalability)

C I R LC I R L

Twisted Trees

• Angular bias applied to flood fill algorithm– based on angle between arcs to neighbor and root– for a 30º left bias, choose neighbor bearing closest

to 30º left of the bearing to the root

• Works in high surrogate density (V2/N) regions• Serial complexity is O(V+E)

– makes efficient use of distribution of sensors– conjecture: O(log(V)) parallel complexity

• Experimentally validated, proofs in progress– good spatial separation (not perfect!)– short paths (small multiplier of optimal)

C I R LC I R L

Gauss network graph

C I R LC I R L

Shortest path routing tree

C I R LC I R L

Boundary edges for router

C I R LC I R L

Twisted routing tree

C I R LC I R L

Multipath route

C I R LC I R L

Crop-duster network graph

C I R LC I R L

Shortest-path routing tree

C I R LC I R L

Boundary edges for router

C I R LC I R L

Twisted routing tree

C I R LC I R L

Multipath route

C I R LC I R L

A Common Surrogate

• Biconnectedness• Robust routing• Fairly robust routing

• V2/N is a surrogate for all of them

C I R LC I R L

A Common Surrogate


• V2/N is a surrogate for all of them• How come?

C I R LC I R L

• Continuous analysis yields order parameters:– critical clustering density:

– fraction of reachable nodes:

– surrogate density that controls reachability:

Predicting Coverage

€

€

ρC

= 1− e−R2 /2V 2 ⎛

⎝

⎜ ⎜ ⎜

⎞

⎠

⎟ ⎟ ⎟

€

f = 1 −2 πρCV

2

N

€

V 2

N

C I R LC I R L

Experimental Validation

Connected fraction tracks prediction well

C I R LC I R L

Sensor coverage predictions

C I R LC I R L

Sensor coverage predictions

C I R LC I R L

A Common Surrogate


• V2/N is a surrogate for all of them• Why?

C I R LC I R L

A Common Surrogate


• V2/N is a surrogate for all of them• Because Gaussian distributions are benign

C I R LC I R L

A Common Surrogate


• V2/N is a surrogate for all of them• Because Gaussian distributions are benign• Suggests existence of other surrogates here:

C I R LC I R L

A Common Surrogate



– what is needed for exfiltration?

C I R LC I R L

A Common Surrogate



– what is needed for exfiltration?– how to manage sentry/relay tradeoff?

C I R LC I R L

A Common Surrogate



– what is needed for exfiltration?– how to manage sentry/relaysleep tradeoff?– extreme scalability: what will work?

C I R LC I R L

A Common Surrogate



– what is needed for exfiltration?– how to manage sentry/relay tradeoff?– extreme scalability: what will work?– surrogates avoid guesswork

C I R LC I R L

Next Steps

• Assist in exploitation of existing surrogates– twisted path implementation, etc.

• Participate in the extreme scaling effort– develop new surrogates in specific areas as needed– exfiltration, sentry/relay/sleep issues, scalability, etc.

• Develop tools to find surrogates in general case– as originally proposed, but harder than expected– general tools for exploiting structure (ZAP)

Documents

Improving NEST Performance Using Surrogates