Embed Size (px)
Citation preview
http://www.grnet.gr
GRNET NOC In-house management tools
TF-NOC
George Kargiotakis ([email protected])
Andreas Polyrakis ([email protected])
Leonidas Poulopoulos ([email protected])
TF-NOC 2 GRNET NOC in-house management tools
Why???
• Automation
Time
PEBKACs
Share
• “Where there an API there is a way”
• No API ???
Try to “create” one
TF-NOC 3 GRNET NOC in-house management tools
Tools
TF-NOC 4
• MuPy • DCmon • Optical Network Mon • Servermon
GRNET NOC in-house management tools
MUPY
TF-NOC 5 GRNET NOC in-house management tools
TF-NOC 6 GRNET NOC in-house management tools
Why? •Munin: POWERFUL •Deployed in GRNET
•Server monitoring
BUT •No AAI scheme •No info aggregation •No groupings •No saved queries “Show all disk IOs graphs” (http://munin.ping.uio.no/) “Show disk graphs from gud and knuth”
What?
TF-NOC 7 GRNET NOC in-house management tools
•Parser •BeautifulSoup lib
•Wrapper •Python Django
•Saved queries •Open source https://code.grnet.gr/projects/mupy
•Demo: http://mupy-demo.grnet.gr/
DATACENTER POWER MONITORING
TF-NOC 8 GRNET NOC in-house management tools
Datacenter Power-monitoring
TF-NOC 9 GRNET NOC in-house management tools
•Datacenter Ministry of Education •Appliance
•Collects data •Graphs •Weekly
•FTP •CSV
Why?
TF-NOC 10 GRNET NOC in-house management tools
•Appliance web server is slooooowwww •Live queries cause load •No combination of metrics •No PUE calculation •No public web interface •But
•Provides data in html table/csv •Every 5 mins
How?
TF-NOC 11 GRNET NOC in-house management tools
•HTML results parser •Store nodes/metrics in db •Store data in rrd files •Combine data to calculate PUE •Provide web interface •Beta internal •Source code soon available
Sneak Peek-Dashboard
TF-NOC 12 GRNET NOC in-house management tools
Sneak Peek-Nodes
TF-NOC 13 GRNET NOC in-house management tools
Sneak Peek-Graphs
TF-NOC 14 GRNET NOC in-house management tools
OPTICAL NETWORK MONITORING
TF-NOC 15 GRNET NOC in-house management tools
Optical Network Monitoring
TF-NOC 16 GRNET NOC in-house management tools
•Alcatel 1626LM, 1696MS •3 months ago:
•Alcatel NMS (PM module) •No attenuation (manual) •No triggers •Fiber Cuts
•Manual filters(Alarm Surveillance) •Manual triggers
TIME TIME TIME TIME
Alcatel tools
TF-NOC 17 GRNET NOC in-house management tools
•Veeeryyy old UI •No graphing
Optical NetAdmin… 3 months ago
TF-NOC 18 GRNET NOC in-house management tools
Requirements
TF-NOC 19 GRNET NOC in-house management tools
•Harvest •Element info •Performance data
•Topology map •Graphs •Alerting
•Visual, Mail •Open data •Web
Architecture
TF-NOC 20 GRNET NOC in-house management tools
Optical Network
Elem
en
t G
rap
hs
IOO client
RRD
NMS
GENOS IOO API
Django
Memcached
Bootstrap UI
RRD Analyser
Top
olo
gy
Att
enu
atio
n
Att
enu
atio
n
Dif
f
Fib
er C
uts
Elem
en
t In
fo
Optical Mon
TF-NOC 21 GRNET NOC in-house management tools
•http://mon.grnet.gr (Optical Network Mon) •IOO Client: http://code.grnet.gr •Source code soon open •Open access •Frequent feature updates
Anomaly detection
TF-NOC 22 GRNET NOC in-house management tools
•On RRD files •Attenuation (diff of counters)
•Iterate measurements •Pattern matching
•If att==0 for x minutes then fiber_cut •If |att_diff|>a for x minutes then anomaly •More simple yet smart checks
•Show visual alerts •Send mail to optnetadmins
Anomaly detection improvements
TF-NOC 23 GRNET NOC in-house management tools
•RRD Holt-Winters Time Series Forecasting
Optical NetAdmin… now
TF-NOC 24 GRNET NOC in-house management tools
SERVERMON
TF-NOC 25 GRNET NOC in-house management tools
Sneak Peek
TF-NOC 26 GRNET NOC in-house management tools
Sneak Peek
TF-NOC 27 GRNET NOC in-house management tools
Sneak Peek
TF-NOC 28 GRNET NOC in-house management tools
Sneak Peek
TF-NOC 29 GRNET NOC in-house management tools
Source Code
• https://code.grnet.gr/
TF-NOC 30 GRNET NOC in-house management tools
Thank you
TF-NOC 31 GRNET NOC in-house management tools
Questions ?
https://twitter.com/grnetnoc
https://www.facebook.com/noc.grnet.gr
