Upload
robyn-dalton
View
224
Download
1
Embed Size (px)
Citation preview
NOC Services and Applications 1
NOC Services and Applications
Sunday FolayanNishal Goburdhan
Isatou Jah
NOC Services and Applications 2
What is Network Management?
“In order to operate a reliable service, the network must be managed according to a determined discipline, using a coherent structure of information management.”
Geoff Huston, ISP Survival Guide
NOC Services and Applications 3
What is a NOC?
Network Operations Centre (NOC) Monitors and manages a service provider’s
network• Information about current, historical and planned
availability of systems• Network status and operational statistics• Fault monitoring and management
Engineers can coordinate their work through the NOC
NOC Services and Applications 4
Network Management - Components
Parts of Network Management
• Configuration/Change management• Performance/Accounting management• Fault management • Security management
NOC Services and Applications 5
Configuration Management
Maintaining information relating to the design of the network and its current configuration
Network State• Record of network topology
– Static what is deployed where it is deployed how it is attached Who is responsible for it How do I contact them
– Dynamic operational status of the network elements
NOC Services and Applications 6
Configuration Management
inventory management• database of network elements• history of changes & problems
directory maintenance• all hosts & applications• nameserver database
host and service naming coordination• "Information is not information if you can't find it"
NOC Services and Applications 7
Configuration Management
Operational Control of network Start/stop individual components Alter configuration of devices Load and save config versions Hardware/Software upgrades Methods of access
• SNMPGet / SNMPSet• Out-of-Band access
NOC Services and Applications 8
RANCID
RANCID - Really Awesome New Cisco confIg Differ
Also works for IOS/CatOS/JunOS/...Open SourceRuns on FreeBSD, Linux, OSX, even MS-
Windowshttp://www.shrubbery.net/
(lots of other useful tools here too!)
NOC Services and Applications 9
RANCID
Collections of scripts that run from cron and automate • logging into routers• capturing configuration• highlighting configuration ‘differences’• emailing the ‘diffs’ to a mail list• installing ‘diffs’ into CVS
NOC Services and Applications 10
RANCID
• Track config changes– Normal day-to-day
• Track hardware changes– Where’s that spare…?
• Track (I)OS changes• Malicious changes ?
– What did your NOC do last night?
• Retrieve dead router configs.• Track router crashes!!
NOC Services and Applications 11
RANCID aka Big Brother
• Announce changes to entire team - everybody starts looking out for anyone making random changes!
• If it’s broken, what’s changed?• Make it user friendly - CVSWeb
NOC Services and Applications 12
RANCID Sample Output
!Slot 2/MBUS: hvers 1.1 !Slot 2/MBUS: software 01.36 (RAM) (ROM version is 01.33) !Slot 2/MBUS: 128 Mbytes DRAM, 16384 Kbytes SDRAM !- !Slot 6: 1 Port Gigabit Ethernet- !Slot 6/PCA: part 73-3302-03 rev C0 ver 3, serial CAB031216OL- !Slot 6/PCA: hvers 1.1- !Slot 6/MBUS: part 73-2146-07 rev B0 dev 0, serial CAB031112SB- !Slot 6/MBUS: hvers 1.2- !Slot 6/MBUS: software 01.36 (RAM) (ROM version is 01.33) !Slot 7: Route Processor !Slot 7/PCA: part 73-2170-03 rev B0 ver 3, serial CAB024901SI !Slot 7/PCA: hvers 1.4 !Slot 7/MBUS: part 73-2146-06 rev A0 dev 0, serial CAB02060044
NOC Services and Applications 13
RANCID Demo
Demo of live RANCID system
NOC Services and Applications 14
RANCID Re-use
More than configuration management.• Cheap Asset Tracker/NMS
UNIX script - easily extendible to other applications.• Re-use login scripts• Manage configuration changes
Correlate syslog and RANCID using Simple Event Correlator (SEC)
– http://threebit.net/mail-archive/cisco-nsp/msg00053.html
NOC Services and Applications 15
RANCID - Even More Uses
Looking Glass software
See Joe Abley and Stephen Stuart NANOG presentation:
• http://www.nanog.org/mtg-0210/abley.html
• Consistency/Audit checks• Generate DNS zone files• Create Topographic maps
NOC Services and Applications 16
What is SNMP?
Simple Network Management Protocol query - response system
• can obtain status from a device• standard queries• enterprise specific
uses database defined in MIB• management information base
NOC Services and Applications 17
What do we use SNMP for?
query routers for:• in and out bytes per second• CPU load• uptime• BGP peer session status
query hosts for:• network status• Message queues• Web traffic• Squid proxy load
NOC Services and Applications 18
SNMP Exercise
NOC Services and Applications 19
Configuration Management
nnhvd
husc6
harvard
geo
oitgw1
mghgw
sphgw1
wjhgw1
wjh12
generali
talcott
harvisr
huelings
pitirium
nngw
lmagw1
dfch tch tch
SNMP driven display
NOC Services and Applications 20
Performance Management
A Consistent level of network performance Data collection
– interface stats– throughput– error rates– usage– percent availability
Data analysis for performance metrics and trends
Establishment of performance thresholds Capacity planning and deployment
NOC Services and Applications 21
Importance of Network Statistics
Accounting Troubleshooting Long-term trend analysis Capacity Planning Two different types
• active measurement• passive measurement
Management Tools have statistical functionality
NOC Services and Applications 22
MRTGSystem: bb-rtr.ws.afnog.org inMaintainer:Description: FastEthernet0/0.67-802.1Q-vLAN-subif Upstream LinkifType: Layer 2 Virtual LAN using 802.1Q (135)ifName: Fa0/0.67Max Speed: 12.5 MBytes/sIp: 196.216.67.254 ()
NOC Services and Applications 23
MRTG and MRTG Exercise
NOC Services and Applications 24
Netflow
Cisco developed - 1996 Initially a mechanism for forwarding
packetsNo longer - Now, primarily used for
• Accounting/Billing• Network planning• Peering arrangements• Traffic engineering• Security monitoring
NOC Services and Applications 25
Netflow
Netflow packet typically contains• IP SRC+DST• Port SRC+DST• Protocol information• TOS byte (DSCP)• Input logical interface (ifIndex)
Extendible (IOS capable) •AS / VRF / ...
NOC Services and Applications 26
Netflow
Uses CPU and memory!Export Netflow to external collector (or
use online on router)• http://www.splintered.net/sw/flow-tools/
Router summarisation possibleNetflow V5 is most commonly usedhttp://www.cisco.com/go/netflow
NOC Services and Applications 27
Netflow
Only works on inbound trafficUnidirectional flowShows transit (traffic through) and to the
router.Enabled by:
•ip route-cache flow•ip flow ingress (new syntax)
Output seen with: •show ip cache [verbose] flow
NOC Services and Applications 28
Netflow Example
From your workstation: ping 196.200.220.1
On your router:router# conf t
router(config) int fa0/0
router(config-if)# ip flow ingress
router# show ip cache flow
NOC Services and Applications 29
Netflow Example (cont).
What’s missing?(Why are the flows only in 1 direction?)
How do you fix it ?
Now repeat the BCP38 packet spoofing exercise, but track the bogus packets with Netflow. Pay attention to what happens when uRPF is enabled.
NOC Services and Applications 30
Netflow examples
Top ten lists (or top five) ##### Top 5 AS's based on number of bytes #######srcAS dstAS pkts bytes 6461 237 4473872 3808572766 237 237 22977795 3180337999 3549 237 6457673 2816009078 2548 237 5215912 2457515319
##### Top 5 Nets based on number of bytes ######Net Matrix----------number of net entries: 931777 SRCNET/MASK DSTNET/MASK PKTS BYTES 165.123.0.0/16 35.8.0.0/13 745858 1036296098 207.126.96.0/19 198.108.98.0/24 708205 907577874 206.183.224.0/19 198.108.16.0/22 740218 861538792 35.8.0.0/13 128.32.0.0/16 671980 467274801 ##### Top 10 Ports ####### input outputport packets bytes packets bytes119 10863322 2808194019 5712783 42730455680 36073210 862839291 17312202 138781709420 1079075 1100961902 614910 627542687648 1146864 419882753 1147081 41466321225 1532439 97294492 2158042 722584770
NOC Services and Applications 31
Accounting Management
What do you account for?• Use of the network and the services it provides
Types of accounting data• RADIUS/TACACS accounting data from Access
servers• Interface statistics• Protocol statistics
Accounting Data affects Business Models• Bill on usage?• Flat-rate billing?
NOC Services and Applications 32
Fault Management
Identify the fault• Regular polling of network elements
Isolate the fault• Diagnosis of the network components
Respond to the fault• Allocate resources to resolve the fault• Priority scheduling• Technical/management escalation
Resolve the fault• notification
NOC Services and Applications 33
Fault Management - systems
reporting mechanism• link to NOC• notify on-call personnel
setup & control alarm procedures repair/recovery procedures ticket system
NOC Services and Applications 34
Fault Management - Fault Detection
Who notices a problem with the network?• Network Operations Center w/ 24x7 operations staff
– open trouble ticket to track problem– preliminary troubleshooting– Assign engineer to problem or escalate ticket status
• Customer call• Other ISPs
NOC Services and Applications 35
Fault Management - Fault Detection (con)How can you tell if there is a problem with the
network?• Network Monitoring Tools
– common utilities ping Traceroute Ethereal Snmp
– Monitoring Systems NOCol Big Brother Nagios HP Openview, etc…
• Report state or unreachability– detect node down– routing problems
NOC Services and Applications 36
Fault Management - Ticket System
Very Important! Need mechanism to track:
• failures• current status of outage• carrier tickets
NOC Services and Applications 37
Fault Management:Ticket System
system provides for:• short term memory & communication• scheduling and work assignment• referrals and dispatching• oversight• statistical analysis• long term accountability
NOC Services and Applications 38
Fault Management - Ticket Usage
create a ticket on ALL calls create a ticket on ALL problems create a ticket for ALL scheduled events copy of ticket mailed to reporter and mailing
list(s) all milestones in resolution of problem maintain
the same ticket # ticket stays "open" until problem resolved Ticket reporter determines that ticket should be
closed.
NOC Services and Applications 39
Fault Management - Ticket Example
Sample opening ticketSubject Serial Number Fix sshd on E2 instructor machines 6
Area Queue none afnog-noc
Requestors Owner [email protected] inst
Status Last User Contact resolved Wed May 10 17:02:21 2006 (12 hr ago)
Current Priority Final Priority 1 1 Due No date assigned
Last Action Wed May 10 17:02:21 2003 (12 hr ago) Created Mon May 8 14:08:08 2003 (2 days ago)
NOC Services and Applications 40
Exercise: Ticket System•RT is already installed on http://e2-noc.ws.afnog.org
•Create tickets to track network occurrences as they occur - network failures will be provided ;-)
NOC Services and Applications 41
Fault Management - typical failures
• Node unpingable• no ip connectivity to router• possible reasons:
– serial link downcall telco
– router down/hardware problemcall engineer
– routing problem troubleshoot with tracerouterouteviews machine
NOC Services and Applications 42
Security Management: Do’s & Don’t’s
Dont’ leave things that are likely to be interesting to mice lying on the kitchen table overnight
Plug the holes that mice are using to get into the house Don’t provide places within the house for mice to build nests Set traps along walls where you often see mice out of the corner
of your eye Check the traps daily to rebait them and to dispose of squashed
mice. Full traps don’t catch mice, and they smell Avoid using commercial bait-and-kill poisons. Traditional snap
traps are best. Get a cat!
NOC Services and Applications 43
Security Management - Tools
security tools• cops - host configuration checker (www.cert.org)• swatch - email reports of activity on machine• Tcpwrappers – log connections, restrict access• ssh/skey – crypto authentication and communications• Tripwire – monitor changes to system files
Keep up to date with security information• bug reports
– CERT advisories mailing list: http://www.cert.org./contact_cert/certmaillist.html
• bug fixes• intruder alerts
NOC Services and Applications 44
Security Management – Good Practice
reporting procedure for security events• e.g. break-ins• abuse email address for customers to report
complaints ([email protected]) control internal and external gateways
• control firewalls (external and internal) security log management
• centralized logging host• Stealth logger, so it cannot be compromised
NOC Services and Applications 45
How do I manage my network?
Which tools should I use? What do I really need?• Keep it simple!• Need to consider engineers working remotely• Don’t want to spend too much time maintaining the
tool (it should be helping you!)• Different tools for NOC and engineers• Different tools for statistics• RELIABILITY!
NOC Services and Applications 46
References http://www.merit.edu/ipma/docs/isp.html http://www.nanog.org http://www.caida.org http://www.nlanr.net http://www.cisco.com http://www.amazing.com/internet/ http://www.isp-resource.com/ http://www.merit.edu/ipma http://www.ripe.net
NOC Services and Applications 47
More Tools!
http://www.caida.org/Tools/• OC3Mon/Coral
http://www.merit.edu/~ipma– RouteTracker
– IRRj
– ASExplorer
http://www.geektools.com/ http://www.merit.edu/ipma/tools/other.html
NOC Services and Applications 48
SNMP Tool references
• MON - http://www.kernel.org/software/mon/• NOCol - ftp://ftp.navya.com/pub/vikas/nocol.tar.gz• Sysmon - ftp://puck.nether.net/pub/jared• Rover - http://www.merit.edu/~rover• Concord - http://www.concord.com• http://www.merit.net/~netscarf