Resilience, Survivability, Heteorgeneity in Postmodern ... file14 October 2009 Resilience, Survivability, Heterogeneity in Postmodern Internet 7 ITTC Sterbenz, et al. Resilient Networks

Sterbenz, et al.ITTCResilience, Survivability,and Heterogeneity

in the Postmodern Internet

14 October 2009

James P.G. Sterbenz*†

Джеймс Ф.Г. Стербэнз 제임스스터벤츠 司徒傑莫David Hutchison,

Abdul Jabbar, Justin P. Rohrer, Egemen Çetinkaya

*Department of Electrical Engineering & Computer ScienceInformation Technology & Telecommunications Research Center

The University of Kansas†Computing Department, Infolab 21

Lancaster University

[email protected]://www.ittc.ku.edu/~jpgs

http://wiki.ittc.ku.edu/resilinets© 2009 Sterbenz

14 October 2009 Resilience, Survivability, Heterogeneity in Postmodern Internet 2

Sterbenz, et al.ITTC

Where is Kansas?Geography Lesson

Denver

Seattle

Portland

Salt Lake City

Las Vegas

Los Angeles

PhoenixSan Diego

St Louis

Minneapolis

Kansas City

Houston

Dallas

New Orleans

Cleveland

Boston

New York

Miami

DetroitChicago

Milwaukee

Philadelphia

Atlanta

San Francisco Washington DC

KANSASKU – Lawrence



Resilience and HeterogeneityOutline

• Resilience strategy and principles• Postmodern Internet Heterogeneity• Example realms

– WDTN– highly mobile airborne ad-hoc networking

• Evaluation Methodology– simulation– experimentation



Resilient Networks Motivation: Reliance

• Increasing reliance on network infrastructure– consumers– commerce & financial– government and military

⇒ Increasingly severe consequences of disruption⇒ Increasing attractiveness as target from bad guys



Resilient Networks Motivation: Consequences

• Increasing reliance on network infrastructure⇒ Increasingly severe consequences of disruption

– threat to life and quality of life– threat to financial health economic stability– threat to national and global security

⇒ Increasing attractiveness as target from bad guys



Resilient Networks Motivation: Attractiveness

• Increasing reliance on network infrastructure⇒ Increasingly severe consequences of disruption⇒ Increasing attractiveness as target from bad guys

– recreational and professional crackers– industrial espionage and sabotage– terrorists and information warfare



Resilient NetworksResilience Definition

• Resilience– provide and maintain acceptable service– in the face of faults and challenges to normal operation

• Challenges– faults– unintentional misconfiguration or operational mistakes– large scale natural disasters– malicious attacks from intelligent adversaries– environmental challenges– unusual but legitimate traffic– service failure at a lower level



Scope of Resilience Relationship to Other Disciplines

TrustworthinessRobustnessComplexity

Security nonrepudiabilityconfidentiality

availability integrity

AAA

authenticity

authorisabilityauditability

Dependability

reliability maintainability safety

Performability

QoS measures

Challenge Tolerance

Traffic Tolerance

legitimate flash crowd

attack DDoS

Disruption Tolerance

energy

connectivity

delay mobility

environmental

Survivability

Fault Tolerance(few ∧ random)

many ∨ targetted failures



Resilient NetworksFault → Error → Failure Chain

• Challenge– may trigger

dormant fault

• Active fault– may cause

error(activation)

• Error– may be

observedas failure(propagation)



ResiliNets AxiomsIUER: Inevitable

A0. Faults are inevitable• Not possible to construct perfect system

– internal faults will exist

• Not possible to prevent challenges and threats– external faults will occur



ResiliNets AxiomsIUER: Understand

A1. Understand normal operations• Normal operation : no adverse conditions present

– loosely corresponds to PSTN and Internet design• e.g. PSTN: traffic engineering for Mother’s Day, not 9/11

• Leads to understanding when network is:– challenged– threatened



ResiliNets AxiomsIUER: Expect

A2. Expect inevitable adverse events and conditions• Adverse event/condition : challenge normal operation• Anticipated : predictable based on

– past events– threat analysis

• Unanticipated : not predictable with specificity– still need to be prepared in general sense

• Motivates part of D2R2 + DR strategy– defend– detect



ResiliNets AxiomsIUER: Respond

A3. Respond to adverse events and conditions• Respond to adverse events by remediation

– graceful service degradation if necessary• permit timing failure

– ensure correct operation• prevent content failure

• Motivates part of D2R2 + DR strategy– remediate (immediately)– recover– diagnose and refine (long term)



ResiliNets StrategyD2R2 + DR

• Real time control loop: D2R2

– defend• passive• active

– detect– remediate– recover

• Background loop: DR– diagnose– refine Refine

Diagnose

Remed

iate

Detect

Recover

Defen

d



ResiliNets StrategyD2R2 + DR: Defend (Passive)

S1a. Defend against challenges to normal operation• Reduce the probability of a fault

leading to a failure• Reduces the impact of

an adverse event• Analogy

– thick outer and inner castle walls

• Examples– spatially diverse redundant paths Refine

Diagnose

Remed

iate

Detect

Recover

Defend

Defen

d



ResiliNets StrategyReal-Time Control Loop D2R2 + DR

• Real-time control loop: D2R2

– real-time with respect tonetwork operation

– many simultaneousindependent loops

• Background loop: DR

Refine

Diagnose

Remed

iate

Detect

Recover

Defen

d

Defend



ResiliNets StrategyD2R2 + DR: Defend (Active)

S1b. Defend against challenges to normal operation• Reduce the probability of

a fault leading to a failure• Reduces the impact of

an adverse event• Analogy

– lookout guard on castle wall

• Examples– filtering traffic for

known attack signaturesRefine

Diagnose

Remed

iate

Detect

Recover

Defen

d

Defend



ResiliNets StrategyD2R2 + DR: Detect

S2. Detect when an adverse event or condition occurs• Determine when defenses

– have failed andremediation needs to occur

– need to be strengthened

• Analogy– detect hole in wall from trebuchet

• Example– detection of behavioural anomaly

• traffic load or pattern• protocol control messages

Refine

Diagnose

Remed

iate

Detect

Recover

Defen

d

Defend



ResiliNets StrategyD2R2 + DR: Remediate

S3. Remediate during adverse condition• Do the best possible

– after adverse event / during adverse condition

• Corrective action at all levels– graceful degradation– correct operation

• Analogy– send subjects to inner wall;

pour boiling oil over hole

• Examples– reroute network traffic

Refine

Diagnose

Remed

iate

Detect

Recover

Defen

d

Defend



ResiliNets StrategyD2R2 + DR: Recover

S4. Recover to normal operations• Return to original state

once adverse condition over– redeploy infrastructure– restore normal control

and management

• Analogy– repair hole in wall

• Example– restore original network routing

Refine

Diagnose

Remed

iate

Detect

Recover

Defen

d

Defend



ResiliNets StrategyBackground Control Loop: D2R2 + DR

• Real time control loop: D2R2

• Background loop: DR– out-of-band analysis of the

reaction to adverse events– increase resilience of system

Refine

Diagnose

Remed

iate

Detect

Recover

Defen

d

Defend



ResiliNets StrategyD2R2 + DR: Diagnose

S5. Diagnose fault that lead to error or failure• Root cause analysis to

discover design flaws– faults not directly detectable

• Analogy– analyse wall-thickness to

trebuchet projectile weight ratio

• Example– analyse packet traces to determine

protocol vulnerabilityRefine

Diagnose

Remed

iate

Detect

Recover

Defen

d

Defend



ResiliNets StrategyD2R2 + DR: Refine

S6. Refine behaviour for the future• Learn from past D2R2 cycles

– better defense, detection,remediation next time

• Analogy– build thicker castle wall– build watch tower for

advance warning

• Example– enrich network topology– redesign protocols

Refine

Diagnose

Remed

iate

Detect

Recover

Defen

d

Defend



ResiliNets PrinciplesHigh Level Grouping

• Prerequisites: to understand and define resilience• Tradeoffs: recognise and organise complexity• Enablers: architecture and mechanisms for resilience• Behaviour: require significant complexity to operate

prerequisites tradeoffs enablers behaviour

servicerequirements

normal behaviour

threat and challenge models

metrics

heterogeneity

resourcetradeoffs

statemanagement

complexity

redundancy

diversity

context awareness

self-protection

translucency

multilevel

self-organisingand autonomic

adaptable

evolvable

connectivity

Resilience



Resilience and HeterogeneityPostmodern Internet Heterogeneity






Heterogeneous Networks Motivation

• ARPANET included heterogeneous subnets – leased lines, packet radio, satellite links

• Internet evolved as mostly reliable wired links– TCP assumes all losses are congestion– routing protocols need relative stability to converge

• New applications and usage scenarios challenge this– untethered access– mobility– energy-constrained networks (e.g. wireless sensor networks)

• Internet hourglass constrains progress– forces an unacceptable lowest-common denominator



Internet HourglassTheory

• Internet “hourglass”• “anything over

IPover anything”

• IP is narrow waist– global addressing– end-to-end forwarding

• Assumption– IP is the right network layer for all scenarios

IP+ICMP

TCP UDP

Ethernet SONET



Internet HourglassRouting and Identifier Reality

• Internet “hourglass”• IP is narrow waist

– addressing– forwarding

• Routing needed– BGP is routing waist– any IGP in AS

• DNS required for human-friendly identifiers

IP+ICMP

TCP UDP

Ethernet SONET

BGP IGPsDNS



Internet HourglassTransport Protocol Reality

• Internet “hourglass”• Transport hourglass

– only widelydeployedtransport protocolsare TCP, UDP(and RTP)

IP+ICMP

TCP UDP

Ethernet SONET

BGP IGPsDNS



Internet HourglassIncreasing Complexity

• Internet “hourglass”– TCP+UDP

– IP+ICMP– BGP+DNS– DHCP+NAT+…

• The waist is no longer thin– but still constrains behavior of subnetwork realms– TCP/IP inappropriate for many emerging scenarios

• WSNs, WMNs, MANETs, DTNs

IP+ICMP

TCP UDP

802.x SONET

BGP IGPsDNSNAT DHCP



Internet ArchitectureCurrent Situation and Implications

• Internet is not resilient– globally resilient to random failures– not resilient to a variety of attacks

• BGP, DNS, DDoS, infrastructure attacks

• Internet does not support heterogeneity well– heavyweight gateways or– least-common denominator solutions

• Need a new internetworking layer– new thin waist of the hourglass– embraces heterogeneity– includes resilience as fundamental design characteristic



Postmodern Internet Overview

• PoMo Principles– new internetworking layer– strict separation of concerns– heterogeneous realms :

mechanism, policy, trust– include mechanisms supporting all foreseeable policies

• Novel characteristics– separate path determination from forwarding: source routes– unforgeable record of path traversed by each packet– separate customer-provider relationship from topology– support cross-layer knobs and dials



Postmodern Internet Architecture Header3

• Knobs: how KU– advice to network layer (and below) from above

• Dials: what– instrumentation from realms and inter-realm paths

knobsaccountabilitymotivationforwardingdirective dials

payload



End-to-End Communication Redundancy and Diversity

• E2E transport over multiple diverse paths– that have minimal (if any) shared fate

• Diversity in– service provider: resilience to contract and peering disputes

(e.g. Cogent vs. Level3)– underlying technology: resilience to medium challenge

• e.g weather disruption of wireless mesh links

– path geography: resilience to natural disaster and attacks• e.g. Baltimore tunnel fire, Hinsdale central office fire• fault tolerance necessary but not sufficient for survivability

• Diversity at all layers



End-to-End Communication Example Scenario

backboneprovider 2

backboneprovider 1 wireless

access net

opticalaccess net

opticalaccess net

wirelessaccess net

• Realm path choices explicitly available to end user– spreading (e.g. erasure coding) or hot standby– service tradeoffs: optical when available, fail-over to wireless– cheapest path under dynamic pricing



Resilience and HeterogeneityWeather Disruption-Tolerant Networking






MotivationBroadband Wireless Links

• Wireless alternative to traditional fiber-optic links– point-to-point wired link replacement in MANs and WANs– mesh between base stations and cell towers

• Potential use– backhaul of 3G (and eventually 4G) data– Internet expansion to new areas, such as rural– alternative to add capacity in congested areas, such as cities



MotivationMillimeter-Wave Wireless Links

• Millimeter-wave communication links– exploit recently available commercial radios

• frequency band: 70 – 90 GHz• lightly licensed by FCC in US

– higher data rate potential than microwave links• very-high data rate: 1 – 10 Gb/s• potential replacement for 1/10 GbE and OC-48/192

– significantly cheaper to deploy than fiber



MotivationMillimeter-Wave Wireless Links

• Disadvantages of millimeter-wave links– highly susceptible to weather

• significant rain-based attenuation

⇒unreliable high-speed links• do not meet carrier requirements for backhaul and distribution• reliability, delay, 50 ms restoration



Millimeter-Wave Mesh Networks Architecture

• Mesh architecture– high degree of connectivity– alternate diverse paths

• mm wave link to CO/POP• alternate mm links

802.163–4G

CO/POP



Millimeter-Wave Mesh Networks Architecture

• Mesh architecture– high degree of connectivity– alternate diverse paths

• severely attenuated mm wave• alternate mm links• alternate lower-freq. RF• fiber bypass (competitor)

• Solution– route around failures

• before they occur

– avoid high error links

802.163–4G

CO/POP



Impact of Weather DisruptionsGeometric Storm Modeling

• Storm model is abstraction of precipitation intensity– cells modelled as ellipses moving along a trajectory– links modelled as line segments– effective attenuation calculated based on link & cell overlap

A B

C

green

red

red

yellow

red

Rs

R1

R2 R3

R4

A B

Cgreen

red

yellow



Weather Disruption-Tolerant Routing New Algorithms

• P-WARP: predictive weather-assisted routing prot.– uses weather-radar data to forecast future link conditions– precipitation modelled as {none/light, moderate, heavy}– effective BER calculated and used to adjust link cost

• XL-OSPF: cross-layered OSPF– uses radar to instantaneously estimate attenuation– conventional OSPF but without dead interval detection



SimulationsObserved Storm in Northeast Kansas

• Millimeter-wave grid location– 38.8621N, 95.3793W

• Storm observed at: – 20:39:26Z 30 Sep 2008



Observed StormPerformance Analysis: Packet Loss

40s OSPF dead interval

node out

link out

40s OSPF dead interval

10s XL-OSPF HELLO interval



Resilience and HeterogeneityHighly-Mobile Airborne Ad Hoc Networking


– WDTN– highly-mobile airborne ad hoc networking




Airborne Telemetry NetworkingScenario and Environment

• Very high relative velocity– Mach 7 ≈ 10 s contact– dynamic topology

• Communication channel– limited spectrum– asymmetric links

• data down omni• C&C up directional

• Multihop– among TAs– through relay nodes

GSGS

RN

TATA

TAs

Internet

GWGW

TA – test articleRN – relay node

GS – ground stationGW – gateway



Highly-Mobile Airborne NetworkingLink Stability and Contact Durations

ScenarioTransmit Range

[nmi]Relative Velocity Contact Duration

[sec]

Single-Hop Best Case

GS – TA 140 400 knots 2520

TA – TA 15 800 knots 135

Single-Hop Worst Case

GS – TA 100 Mach 3.5 300

TA – TA 10 Mach 7.0 15

• Multihop case significantly harder– probability of stable end-to-end path very low



Airborne Network Protocol SuiteProtocol Stack and Interoperability

TmNS

GW TA

iNET MAC

iNET PHY

AeroTP

AeroNP

iNET MAC

iNET PHY

AeroN|RP

iNET MAC

iNET PHY

AeroTP

AeroNP

link/MAC

PHY

TCP

IP

link/MAC

PHY

TCP

IP

RN or TA

TA peripherals

HMI appsgNET

• AeroTP: TCP-friendly transport• AeroNP: IP-compatible forwarding• AeroRP: routing



time [s]

pack

ets

rece

ived

[p

kt/s

]

50 150 250 350 450 550 650 750 850 950 0

400

800

1200

1600 AeroRP

DSDV

AODV

AeroRPPerformance Comparison (preliminary)

• 60-node ns-2 simulation in 150×150 km2 test range• TA tx range = 15 nmi; v = [200 knot, Mach 3.5]• CBR traffic = 200 kb/s per TA [MILCOM 2008]

0

0

0

0

0

0

0

0

0



Resilience and HeterogeneityEvaluation Methodology: Simulation



• Evaluation methodology– simulation– experimentation



Evaluation MethodologyFlexible and Realistic Topology Generation

• Hierarchical topology generation– evaluation of PoMo mechanisms– network engineering for resilience

• Level 1: backbone realms– nodes distributed based on location constraints– links generated using various models under cost constraints

• Level 2: access network realms– distributed around backbone nodes– access network connectivity: ring, star, mesh

• Level 3: subscribers– distributed around access network node



Evaluation MethodologyChallenge Simulation Module

• Separate challenge from network simulation• Simulate challenges to any network over time interval

– natural disaster destroy network infrastructure (e.g. Katrina)or large-scale grid failure (e.g. Northeast US 2003)

– attack: {node|link} down, wireless link attenuated

××




# of nodes down

aggr

egat

e pa

cket

del

iver

y ra

tio




• KU-CSM Challenge Simulation Module– challenge specification describes challenge scenario– network coordinates provide node geo-locations– adjacency matrix specifies link connectivity– input to conventional ns-3 simulation run– generates trace to plot results

adjacencymatrix.txt 0 0

1 0

simulation description.cc

challenge specification.txt

node coordinates.txt

ns-3 simulator

(C++)

plotted trace

KU-LoCGen



Resilience MeasureTowards a Resilience Metric

• Need: analyse and understand level of resilience• Problem: how to measure

– ideal: ℜ = [0,1]– but too many metrics to combine into a single number

• perhaps a small set of objective functions

– need to separate service spec. from operational parms.

• Model resilience as two-dimensional state space– operational state– service level



Resilience MetricsMultilevel Approach

• Resilience defined at any layer boundary– operational state describes system below layer boundary– service states represents behaviour above boundary

• Operational range arbitrarily divided into 3 regions– normal operation– partially degraded– severely degraded

• Service delivered arbitrarily divided into 3 regions – acceptable service– impaired service– unacceptable service



Resilience MetricsStates

• State of the system represented by (N, P) tuple– multivariate operational state N– multivariate service state P

• Initial state described by– acceptable service– during normal operations

• Challenges perturb N and P– resilience trajectory



Resilience State SpaceOperational Resilience

NormalOperation

PartiallyDegraded

SeverelyDegraded

Operational State N

S

• Operationalresilience– minimal

degradation– in the face of

challenges

• Resilience state– remains in

normal operation



Resilience State SpaceService Resilience

NormalOperation

PartiallyDegraded

SeverelyDegraded

Acceptable

Impaired

Unacceptable

Operational State N

Ser

vice

Par

amet

ers

P

S

• Serviceresilience– acceptable

service– in the face of

degraded operation

• Resilience state– remains in

acceptable service

S



Resilience State SpaceResilience Trajectories

• Choose scenario– network– application

• Metrics– choose– aggregate

• Observe– under challenge

Unacceptable

Impaired

AcceptableP [d

elay

, pkt

del

iver

y ra

tio]

Normal operation

Partially degraded

Severely degraded

S5

S1

S4

p1 > 10 secp2 > 0.85

p1 < 10 secp2 > 0.50

p1 < 1 secp2 < 0.50

n1 ≤ 1.5ρ0n2 ≥ 3.0

n1 ≤ 3ρ0n2 ≥ 1.0

n1 > 3ρ0n2 < 1.0

N [load, node degree]

S3

S2



Evaluation MethodologyExample: Sprint Synthetic Fragile Topology



Resilience EvaluationPreliminary Results: Sprint Case Study

normalc = 0

partially degraded0 < c ≤ 5

severely degraded5 < c ≤ 68

Unacceptabled < 2 & lc < 0.95

Impaired4 > d ≥ 2 & 1 > lc ≥ 0.95

Acceptabled ≥ 4 & lc = 1

# of link cuts

aver

age

node

deg

ree,

com

pone

nt s

ize



Resilience and HeterogeneityEvaluation Methodology: Experimentation



• Evaluation methodology– simulation– experimentation



GENIOverview

• GENI: Global Environments for Network Innovation– funded by the US NSF– managed by the GPO (GENI Project Office – BBN)

• Goal: new experimental network infrastructure• 1st solicitation: 29 projects funded

– grouped into 5 control framework clusters (PlanetLab, …)– including 2 regional testbeds (GpENI and MANFRED)

• 2nd solicitation: 33 projects funded



GpENIOverview

• GpENI [dʒɛ’pi ni] Great Plains Environment for Network Innovation

• Regional network part of Cluster B in GENI Spiral 1– exploiting new fiber infrastructure in KS, MO, and NE

KSUKU

UMKC

UNL

MOREnetKanREN

GPN



GpENIProject Goals

• Collaborative research infrastructure in Great Plains• Flexible infrastructure to support GENI program • Open environment for network research community• Outreach to grow GpENI infrastructure

– Great Plains region– internationally including EU FIRE



GpENI Physical Topology and Network Infrastructure

KSU – KS KU – KS

UMKC – MO

UNL – NE

GpENI CienaCoreDirector

GpENI CienaCN4200

CCD FlarsheimHall

AveryHall

C-bandn λs

KU/Qwestfiber

NicholsHall

RathboneHall

GpENI nodecluster

WTC fiber

KU/Qwestfiber

SFBB fiberEllsworth

HallPowerPlant

QwestPOP

KC MO

KU/Qwestfiber

Internet2POP

KC MO

MOREnetfiber Newcomb

Hall

Ethernet

ScottCenter

UNL (L3)fiber

splicepatch

CC

to Smith Ctr. KS (eventual link to CO)

dark fiber

2 λs

4 λs

C-bandn λs

C42

GpENI

CCD

GpENI

C42

GpENI

C42

GpENI

C42

GpENI

• Physical topology– multiwavelength optical backbone

• current or imminent deployment

– 4 universities in 3 states• 1 switch/year with current funding

new nodes



KSUKU UMKC

UNL

GpENIGPN Midwest Expansion

DSU

USD

SDSMT

UMCIU

Europe

Asia

• Regional US GpENI partners– South Dakota: 3 universities– Missouri: 1 university– GMOC at Indiana University



• European GpENI partners– 12 nations– 20 research institutions– 100 nodes– more under discussion

Bucharest

Wien

Warszawa

ISCTE

SICS

HelsinkiSimula

Bilkent

GpENIEuropean Expansion

KSUKU UMKC

UNL

DSU

USD

SDSMT

UMC IUAsia

Internet2

GÉANT2

JANET

DANTE

NORDUnet

SWITCH

Passau

UPC Barcelona

Cambridge

Bern

ETH

U-ZürichKonstanz

Karlsruhe

München

Lancaster

KTH

Karlstads



GpENIAsian Expansion

KSUKU UMKC

UNL

DSU

USD

SDSMT

UMC IU

Europe

POSTECH

IIT Mumbai

IISc Bangalore

IIT Guwahati

Internet2

ERNET

APAN

• Asian GpENI partners– 2 nations– 4 research institutions– 20 nodes– more under discussion



GpENINode Cluster

• GpENI cluster• 5–10 PCs

– GpENI mgt.– L4: PlanetLab– L3: prog. routers

• GbE switch– arbitrary interconnection– VLAN connectivity to GENI– SNMP cluster monitoring

• Ciena optical switch– L1 GpENI interconnection

Ciena optical

GpENImanagement& control

Campus net

GpENI optical backboneto Internet2 and KC SPPthen to Mid-Atlantic Net

Internet

GbEnetGENIVLANs

PlanetLab GENIwrap

control framework

prog. routersVINI,

XORP, click,…

� �site specificKUAR,

sensor, …



End

Documents

Resilience, Survivability, Heteorgeneity in Postmodern ... file14 October 2009 Resilience, Survivability, Heterogeneity in Postmodern Internet 7 ITTC Sterbenz, et al. Resilient Networks