Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
UW EE Colloquium Feb. 2009 1
Network Simulation and Emulationin a Testbed Era
Tom HendersonUniversity of Washington
The Boeing Company
UW EE ColloquiumFebruary 24, 2009
Engineering, Operations & Technology | Boeing Research & Technology
Copyright © 2009 Boeing. All rights reserved.
Networked Systems Technology
Information Assurance
Digital Communications &
Network Technology
Information Management and Transformation
Applied Mathematics
Architecture and Networked Systems
Technology
Contract Research and Development
Programs
Boeing Research & Technology organization
• Development of high assurance multi-level security technologies
– EAL7 common criteria evaluation experience
– Formal methods in support of EAL 7 evaluation
• High assurance guard solutions for edge, core, & tactical environments
• Security modeling & formal method analysis of end to end systems
• Policy-based secure computing– Systems of Systems
IA approach• Intrusion detection correlation
technologies• Secure workflow and SOA
technologies• Information operations• System security engineering• Advanced trusted system
implementations• Text analysis for information
assurance• Proprietary program support in IA
technologies• Secure RFID infrastructure• Anti-tampering technologies
Networked Systems Technology
• Phased array antennas• Free air laser
communications• High capacity RF
communications• Bandwidth efficient
modulation• On board wireless
communications• Network processors• AFRL RAPID• Integrated communications
systems• MANET protocols• Reliable routing• Network management for
tactical environments• Auto-configuration• Network gateways• End to end QoS• Network security• UAV network router
• Data & Text Mining• Human Language technologies• Decision support information• Information visualization• Machine translation• Knowledge representation• Data integration• Simulation visualization• 3D visualization• Model-based collaboration• Social networks analysis• Intelligent graphics & docs.• Collab. human robot systems• Autonomous systems
– Architectures– World modeling– Multi-vehicle behaviors
• Reasoning• Adaptive planning• Learning • Distributed multi-agents • Global information sharing• RFID info. mgmt. services• Adaptive intelligent info. integration• Semantic modeling, ontologies,
adaptive workflow modeling• Quality of Service info. algorithms• Wireless mobile dynamic info. mgmt.• Sensor networks• Data & information fusion• Info. assessment aids• Trends analysis/intent inferencing• Situation assessment• Resource allocation techniques• Deconfliction algorithms• Course of action analysis/generation
• Variation analysis & reduction• Reliability & time dependent data
analysis• Analysis of large datasets• Financial modeling & decisions• Survey & test design• Statistical training• Quality assurance & Lean+• Design of Experiments• Computer-aided geometric design• Numerical optimization• Design space exploration• Multidisciplinary Design
Optimization • Geometric modeling• Multivariate data fitting• Optimal control• Engineering modeling & analysis• Computational mechanics• Sensor Fabric design & operation• System dynamics & control
modeling• Data structures for complex
product families• Kalman filter applications• Operations research• Systems of Systems modeling &
analysis• Discrete/network/stochastic/
Systems of Systems optimization• Data fusion• Discrete-event simulation• Transportation, manufacturing,
business, communication & military system modeling
• UML architecting• Arch. analysis• CONOPS development• System requirements• NR-KPP compliance• SoSE• Systems engineering• NCO demo.• LVC technologies• LVC tools• DIS router• Data link translation• Gateways• Networked manufacturing• Location services• Planning / visibility tools• Functional integration• High performance computing• GRID computing• Resource management• Functional integration• CINS lab• System/network mgmt.• QoS• Mobile agents• Situation awareness• Reconfigurable computing• S/W architecture• S/W processes• Arch. evaluation (ATAM)• Performance assessment
(PAPM)• QoS
• Real-time system development• Tactical info. mgmt. with an
emphasis in C2/tactical collaboration
• Quality of Service network utilization
• Software enabled control for UAV automation
• Dynamic network management• Affordable validation &
verification of flight critical software & systems
• Collaborative anticipatory systems
• Cyberwarfare / Cybercraft• Effects-based operations• System & vehicle health
management• Diagnostics• InfoCenter & information
management• Database technology &
implementation• Distributed sensor system
technology
Homeorganization
UW EE Colloquium Feb. 2009
What is a network simulator?
• A software tool and set of models used to evaluate computer network behavior
• A discrete-event network simulator describes networks in terms of a sequence of discrete events– The simulation kernel temporally orders and
dispatches events to models– Models generate output or more events– Time advances in discrete steps
UW EE Colloquium Feb. 2009 4
Application
The basic model
Application
Protocolstack
Node
NetDeviceNetDevice
ApplicationApplication
Protocolstack
Node
NetDevice
Channel
Packet
Simul
UW EE Colloquium Feb. 2009
The basic model
// Define your topologyNode n0 = new Node;Node n1 = new Node;AddInternetStack (n0, n1);Channel c0 = new Channel;.Connect (n0, n1, c0);Application a0 = new TrafficGenerator;
// Configure thingsa0.SetDataRate (1Mb/s);a0.Start (10.0 seconds);
// Define outputsWriteTraceFile (“outfile”);
// Run the simulatorSimulator::Run();
UW EE Colloquium Feb. 2009 6
Network simulation remains popular
• Many commercial products– e.g. OPNET, QualNet, Shunra VE, Matlab Simulink,
OMNet++, NCTUns, SSFNet, Extend
• A few free, open source projects– ns-2 is the most widely used such tool– roughly 10,000 downloads/month
• Widely used in research
Source: Search of ACM Digital Library papers citing simulation, 2001-04
UW EE Colloquium Feb. 2009 7
Skepticism in the research community
“For years, the community had to rely on simulators, which now seem a little dated, and it’s not clear who was convinced to adopt anything new based on ns2 simulations;”
Nick McKeown, VINI public review, ACM Sigcomm 2006
UW EE Colloquium Feb. 2009
What has changed?
• Simulators have historically been used for experiments in place of real machines– A tradeoff of realism for scale
But now...• Cost of test hardware has plummeted
• Virtualization technologies proliferate
UW EE Colloquium Feb. 2009
Emulab (University of Utah)
• Source: http://www.emulab.net
UW EE Colloquium Feb. 2009
Emulab
• Original Emulab allowed experiments of roughly ~100 nodes– FreeBSD Dummynet and VLANs to interconnect the
nodes
• Emulab now offers virtualization– Experiments of up to ~2000 (virtual) nodes– Requires new approaches for experimental
configuration
• Also, some 802.11 wireless nodesSource: M. Hibler et al., “Large-scale Virtualization in the Emulab
Network Testbed,” Proc. 2008 Usenix Technical Conference.
UW EE Colloquium Feb. 2009
PlanetLab: Global-scale Internet testbed
• Source: http://www.planet-lab.org
PlanetLab “slices”
UW EE Colloquium Feb. 2009
PL-VINI and Trellis
• PL-VINI adds virtual networking capabilities (via network overlay) to PlanetLab– A. Bavier et al, “In VINI Veritas: Realistic and
Controlled Network Experimentation,” Proc. ACM Sigcomm, 2006
• Trellis is a kernel-based implementation of the same concept– S. Bhatia et al, “Trellis: A Platform for Building
Flexible, Fast Virtual Networks on Commodity Hardware,” ROADS Workshop, Dec. 2008
UW EE Colloquium Feb. 2009
ORBIT (Rutgers WINLAB)
• Source: http://www.winlab.rutgers.edu/docs/focus/ORBIT.html
• 802.11a/b/g radios
• Bluetooth
• GNU/USRP radios
• Noise generators
UW EE Colloquium Feb. 2009 14
Simulation “out of the loop”
• Source: NSF GENI project: http://www.geni.net/docs/GENIOvrvw092908.pdf
UW EE Colloquium Feb. 2009 15
Skepticism in the research community
“For years, the community had to rely on simulators, which now seem a little dated, and it’s not clear who was convinced to adopt anything new based on ns2 simulations;”
Nick McKeown, VINI public review, ACM Sigcomm 2006
UW EE Colloquium Feb. 2009 16
Overheard* on e2e-interest mailing list
“...Tragedy of the Commons...”“...around 50% of the papers appeared to be...
bogus...”“Who has ever validated NS2 code?”“To be honest, I'm still not sure whether I will use a
simulation in a paper.”“...I will have a hard time accepting or advocating
the use of NS-2 or any other simulation tool”...
* September 2005 archives of the e2e-interest mailing list
UW EE Colloquium Feb. 2009
Questions
• Do we need to simulate networks anymore?
• What kind of network simulator is needed?
• Why should you believe network simulation results?
UW EE Colloquium Feb. 2009
Why simulate?
• Wireless– Reproducibility– Fidelity (especially, real-time constraints)– Radios may not exist or be available– Field tests in realistic conditions cost $$
• Scalability– 10,000+ nodes?– For smaller configurations, execution time
Simulation is often an appropriate tool for the job
UW EE Colloquium Feb. 2009 19
Challenges for ns (and simulators)
• Align with how research is now conducted
• Improve credibility
Can a new simulator help with these problems?
UW EE Colloquium Feb. 2009 20
What is ns-3?
An open source project building a new network simulator to replace ns-2
UW EE Colloquium Feb. 2009 21
ns-3 project goal
Develop a preferred, open simulation environment for networking research1) a tool aligned with the simulation needs of
modern networking research2) an open-source project that encourages
community contribution, peer review, and validation of the software
UW EE Colloquium Feb. 2009 22
Relationship to ns-2
ns-3 is a new simulator, without backward compatibility
Similarities to ns-2:• C++ software core• GNU GPLv2 licensing• ported ns-2 models: random variables, error models,
OLSR, Calendar Queue scheduler, (more planned)
Differences: • Python scripting (or C++ programs) replaces OTcl• most of the core rewritten• also based on the yans and GTNetS simulators• new animators, configuration tools, etc. are in work
UW EE Colloquium Feb. 2009 23
Funding support
ns-3 project is supported by:• NSF CISE CRI awards (2006-10)
– PIs: Tom Henderson and Sumit Roy (University of Washington), George Riley (Georgia Institute of Technology)
– NSF CNS-0551686 (University of Washington)
• INRIA Planete and the French governmentWith additional support from:• 2008 Google Summer of Code• University of Washington• Georgia Institute of Technology
‘
UW EE Colloquium Feb. 2009 24
Challenges for ns (and simulators)
• Align with how research is now conducted
• Improve credibility
UW EE Colloquium Feb. 2009
Test and Evaluation Options
• When mathematical analysis alone won’t do
Increasing realism
Increasing complexity
Puresimulation
Simulationcradles
Virtual/Physicaltestbeds
Fieldexperiments
Livenetworks
Test and evaluation options
Can we develop tools to span this space?
UW EE Colloquium Feb. 2009
Some ns-3 design goals
1. Realism2. Software reuse3. Emulation4. Composition
UW EE Colloquium Feb. 2009 27
1) Attention to realism
Research should often involve a mix of simulations and testbed or live experiments
• If the simulator cannot be made to closely model a real system:– hard to compare results or validate the model– hard to reuse software between the two domains
When in doubt, do what implementations do
UW EE Colloquium Feb. 2009 28
Attention to realism (example)
An ns-3 Node is a husk of a computer to which applications, stacks, and NICs are added
ApplicationApplication
Application
“DTN”
UW EE Colloquium Feb. 2009 29
Realism (cont.)
• Align ns-3 to be more faithful representation of real computers– sockets API– packets are “packed” binary representations
ready for direct serialization to/from network devices
– alignment with Linux architecture– multiple interfaces handled properly
UW EE Colloquium Feb. 2009 30
ns-3 follows a Linux architecture
classNetDevice
classIpv4Protocol
• Linux corollaries– each NetDevice is modelled
like struct net_device
– each layer-3 protocol has a struct in_devicecomponent for each NetDevice
– this interface (transmit side) is aligned with Linux dev_queue_xmit()
– on receive side, demux is modelled after Linux protocol handlers
UW EE Colloquium Feb. 2009 31
2) software integration
Do not reimplement models and tools for which open-source implementations abound
• ns-3 conforms to standard input/output formats so that other tools can be reused.– e.g., pcap trace output, ns-2 mobility scripts
• ns-3 is adding support for running implementation code– Network Simulation Cradle integration has met with
success: Linux TCP code– ns-3 “process” API
UW EE Colloquium Feb. 2009
ns-3 simulation cradle
• Port by Florian Westphal of Sam Jansen’s Ph.D. work
Figure reference: S. Jansen, Performance, validation and testing with the Network Simulation Cradle. MASCOTS 2006.
UW EE Colloquium Feb. 2009
ns-3 simulation cradle
For ns-3:• Linux 2.6.18• Linux 2.6.26• Linux 2.6.28
Others:• FreeBSD 5• lwip 1.3• OpenBSD 3
Other simulators:• ns-2• OmNET++
Figure reference: S. Jansen, Performance, validation and testing with the Network Simulation Cradle. MASCOTS 2006.
UW EE Colloquium Feb. 2009 34
ns-3 “processes” and Posix API
Support for a synchronous Posix socket API• each Posix type and function is redefined in the simulator• processes get their own private stack
– somewhat like a lightweight virtual machine
• Benefits:– makes porting real world application code much easier– makes writing applications easier because the BSD socket API is
faithfully followed
• see the “mathieu/ns-3-simu” code repository
UW EE Colloquium Feb. 2009 35
3) emulation support
Support moving between simulation and testbeds or live systems
• A real-time scheduler, and support for two modes of emulation
UW EE Colloquium Feb. 2009 36
ns-3 emulation modes
virtualmachine ns-3
virtualmachine
1) ns-3 interconnects real or virtual machines
realmachine
ns-3
TestbedTestbed
realmachine
ns-3
2) testbeds interconnect ns-3 stacks
real machine
Various hybrids of the above are possible
UW EE Colloquium Feb. 2009 37
Example: ORBIT and ns-3
• Support for use of Rutgers WINLAB ORBIT radio grid
UW EE Colloquium Feb. 2009
4) Composition
ns-3 is a component of an overall workflow
Define ns-3 withinterfaces to supportintegration with othertools
• Topology generators• Traffic generators• Data analysis frameworks• Configuration tools• Animators
Engineering, Operations & Technology | Boeing Research & Technology
Copyright © 2009 Boeing. All rights reserved.
Networked Systems Technology
Common Open Research Emulator
Scalable Network Emulator• Network lab “in a box”
• Efficient and scalable• Easy-to-use GUI canvas
• Kernel-level networking efficiency• Reference passing packet sending
• Runs real binary code• No need to modify applications
• Connects with real networks• Hardware-in-the-loop• Distributed - runs on multiple servers• Virtual nodes process real packets
• Fork of the IMUNES project• University of Zagreb
• Open Source• http://cs.itd.nrl.navy.mil/work/core
Engineering, Operations & Technology | Boeing Research & Technology
Copyright © 2009 Boeing. All rights reserved.
Networked Systems Technology
netgraphsystem
Tcl/Tk GUIcore_wlan
core_spanCOREAPI
ng_wlan
NIC
FreeBSDkernel
virtual images(vimages)
tunnels
userspace
Overview of CORE Components
Three key components
1. virtual machines
3. graphical user interface
2. networking subsystem
Engineering, Operations & Technology | Boeing Research & Technology
Copyright © 2009 Boeing. All rights reserved.
Networked Systems Technology
CORE features
IMUNES base:• Flexible GUI-based management of FreeBSD lightweight
virtual containers• Virtual topology can be hooked to real devicesCORE extensions:• GUI enhancements, kernel enhancements, wireless
networking, Linux support (OpenVz and NetEm), distributed emulation, and packaging.
• Software leads: Jeff Ahrenholz and Claudiu Danilov, Boeing• http://cs.itd.nrl.navy.mil/work/core
Engineering, Operations & Technology | Boeing Research & Technology
Copyright © 2009 Boeing. All rights reserved.
Networked Systems Technology
CORE and ns-3 integration
ns-3 tap-wifi-dumbbell.cc program
UW EE Colloquium Feb. 2009 43
ns-3 and research alignment
In summary, make it easier to move from simulation to emulation to experiments
• Realism and alignment with popular interfaces• Support use of real code• Emulation capabilities• Combine ns-3 with other tools
UW EE Colloquium Feb. 2009 44
Challenges for ns (and simulators)
• Align with how research is now conducted
• Improve credibility
UW EE Colloquium Feb. 2009 45
Background on simulation credibility
[1] “Why We STILL Don’t Know How to Simulate Networks”– Mostafa Ammar, Georgia Institute of Technology, Annual
Simulation Symposium 2005
[2] “Maintaining a Critical Attitude Towards Simulation Results”– Sally Floyd, WNS2 Workshop Keynote, October 2006
[3] “MANET Simulation Studies: The Incredibles”– Kurkowski, Camp, and Colagrosso, ACM Sigmobile, MC2R,
Volume 9, Issue 4, October 2005
[4] “An Integrated Approach to Evaluating Simulation Credibility”– Muessig, Laack, and Wrobleski, U.S. Naval Air Warfare Center,
August 2001
UW EE Colloquium Feb. 2009 46
Criteria for Credibility
• Repeatable• Unbiased• Realistic Scenarios• Statistically Sound• Model Accuracy• Results Accuracy (Validation)• Data Accuracy• Usability
from [3]
from [4]
UW EE Colloquium Feb. 2009 47
Repeatability
A hallmark of the scientific method...
• Papers should identify simulator, version, operating system, parameters, etc.
• Better yet, make code and configuration scripts available to the community– Yet, 0 out of 84 ACM Mobihoc MANET
simulation papers (2000-2004) referenced publicly available code (from [3])
UW EE Colloquium Feb. 2009
ns-3 approach
• We are adding support to systematically document how simulations were conducted– A configuration subsystem that dumps
simulation parameters to a configuration file (input and output)
• We will host code, publications, validation results, etc.
UW EE Colloquium Feb. 2009 49
Model accuracy
• “error-free-ness” of software and models
• ns-3 goals here:– Support real code where possible– Open source models– Maintainers
UW EE Colloquium Feb. 2009 50
Open source models
• “Given enough eyeballs, all bugs are shallow”– Eric Raymond, “The Cathedral and the Bazaar”
• ns-3 needs ways to certify models, too– capture level of community acceptance– publication lists, cross-reference– need to identify maintainers, or state the
absence of a maintainer– validation techniques and results
UW EE Colloquium Feb. 2009
Example: ns-3 Wifi development
Several research groups are maturing the original INRIA model:• Karlsruhe Institute of Technology: 802.11 PHY, 802.11e
– Equalizing PHY models including capture effects, user-definable coding rates (e.g. 5.9 GHz from 802.11p), EDCA QoS extensions of 802.11e, Nakagami/Rayleigh propagation loss model
• University of Florence: 802.11n features – Frame Aggregation, Block ACK, HCF (EDCA and support for
HCCA),TXOP, HT terminal (also with protection modes), MIMO
• Russian Academy of Sciences: 802.11s– a complete model of IEEE802.11s D2.0 Draft Standard
• Deutsche Telekom Laboratories in Berlin: 802.11 PHY• Boeing: 802.11b channel models, validation• (and others...)
UW EE Colloquium Feb. 2009
Pledge break
• ns-3 needs participation from the research community1) improving simulation credibility2) contributed and supported models3) maintainers
Please support your open source projects!
UW EE Colloquium Feb. 2009 53
Summary of simulation credibility
• Learn from good and bad examples of simulation research, to produce credible simulations
• Consider open source (or publishing of models and scripts) to be integral part of your research
• Please give back to the simulators that you use
UW EE Colloquium Feb. 2009 54
Acknowledgments
Thanks to:• the core development team and research project leads
– Raj Bhattacharjea, Gustavo Carneiro, Walid Dabbous, Craig Dowell, Joe Kopena, Mathieu Lacage (software lead), George Riley, Sumit Roy
• 2008 Google Summer of Code mentors and students• many code authors and testers• the ns-2 PIs and developers for creating ns-2 and for
supporting ns-3 activities• USC ISI for hosting project mailing listsContact: [email protected] (office M354)