View
6.352
Download
2
Tags:
Embed Size (px)
DESCRIPTION
The Assimilation Monitoring Project is an open source project providing both continuous Stealth Discovery(TM) and extreme scale monitoring.
Citation preview
GRAPHCONNECT
Modeling IT Infrastuctureusing
The Assimilation Project
#AssimProj @OSSAlanR
http://assimproj.org/
http://bit.ly/AssimGC2013
Alan Robertson <[email protected]>Assimilation Systems Limited
http://assimilationsystems.com
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 2/25
GRAPHCONNECT
Upcoming Events
GraphConnect San Francisco (today!)
Open Source Monitoring Conference - Nürnberg
NSA / Homeland Security Assimilation Technical Talk
Large Installation System Administration Conference - DC
Colorado Springs Open Source User’s Group
linux.conf.au – Awesome Australian Linux Conf - Perth
Details on http://assimilationsystems.com/
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 3/25
GRAPHCONNECT
Talk Outline
● Project Overview● Infrastructure Schema● Python Object-Graph-Mapping API
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 4/25
GRAPHCONNECT
Assimilation Project History
● Inspired by 2 million core computer (cyclops64)
● Concerns for extreme scale● Topology aware monitoring● Topology discovery w/out security issues
=►Discovery of everything!
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 5/25
GRAPHCONNECT
Project Scope
Zero-network-footprint continuous Discoveryintegrated with extreme-scale Monitoring
● Continuous extensible discovery– systems, switches, services, dependencies
– zero network footprint
● Extensible exception monitoring– more than 100K systems
● All data goes into central graph database
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 6/25
GRAPHCONNECT
Discovery
Discovering● systems you've forgotten● what you're not monitoring● whatever you'd like● without setting off security alarms
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 7/25
GRAPHCONNECT
Why Discovery?
● Documentation: incomplete, incorrect● Dependencies: unknown● Planning: Needs accurate data● Best Practices: Verification needs
data● ITIL CMDB (Configuration Mgmt
DataBase)
Our Discovery: continuous, low-profile
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 8/25
GRAPHCONNECT
Why Neo4j (graph db)?
● Dependency & Discovery information: graph● Speed of graph traversals depends on size
of subgraph, not total graph size● Root cause queries graph traversals –
notoriously slow in relational databases● Visualization of relationships● Schema-less design: good for constantly
changing heterogeneous environment● Graph Model === Object Model
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 9/25
GRAPHCONNECT
Assimilation Communication Neighbor-Rings
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 10/25
GRAPHCONNECT
Ring Representation Schema
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 11/25
GRAPHCONNECT
ssh -> sshd dependency graph
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 12/25
GRAPHCONNECT
Switch Discovery Datafrom LLDP (or CDP)
CRM transforms LLDP (CDP) Data to JSON
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 13/25
GRAPHCONNECT
OGM – Object Graph Mapping
● Managing the Graph Nodes “disappears”● The Object Model is the Graph Model● Significant Improvement in Thinking
Clarity– The “mud” is gone
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 14/25
GRAPHCONNECT
OGM rules
● Don't use Constructors● Relationships replace hash tables and
object references and so on● Constructor parameter names match
attribute names
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 15/25
GRAPHCONNECT
OGM sample
@RegisterGraphClassclass Person(GraphNode): 'A Person - or at least an electronic representation of one' def __init__(self, firstname, lastname, dateofbirth=None): GraphNode.__init__(self, domain='global') self.firstname = firstname self.lastname = lastname if dateofbirth is not None: self.dateofbirth = dateofbirth else: self.dateofbirth='unknown' @staticmethod def __meta_keyattrs__(): 'Return our key attributes in order of significance' return ['lastname', 'firstname']
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 16/25
GRAPHCONNECT
OGM sample
@RegisterGraphClassclass TestSystem(GraphNode): 'Some kind of semi-intelligent system' def __init__(self, designation, roles=[]): GraphNode.__init__(self, domain) self.designation = designation.lower() if roles == []: roles = [''] self.roles = roles
def addroles(self, role): if self.roles[0] == '''': del self.roles[0] if isinstance(role, list): for arole in role: self.addroles(arole) elif role not in self.roles: self.roles.append(role)
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 17/25
GRAPHCONNECT
OGM sample @staticmethod def __meta_keyattrs__(): 'Return our key attributes in order of significance' return ['designation']
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 18/25
GRAPHCONNECT
OGM sample
# (seven)-[:formerly]->(Annika)
Annika = store.load_or_create(Person, firstname='Annika', lastname='Hansen')
seven = store.load_or_create(Drone, designation='SevenOfNine', roles='Borg')
store.relate(seven, 'formerly', Annika)store.commit()
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 19/25
GRAPHCONNECT
OGM sample
@RegisterGraphClassclass TestDrone(TestSystem): def __init__(self, designation, roles=[]): TestSystem.__init__(self, designation=designation) if isinstance(roles, str): roles = [roles] roles.extend(['host', 'Drone']) System.__init__(self, designation, roles=roles)
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 20/25
GRAPHCONNECT
Current State
● First release was April 2013
● Great unit test infrastructure
● Nanoprobe code – works well
● Service monitoring works● Lacks digital signatures, encryption, compression
● Reliable UDP comm code working
● Several discovery methods written
● CMA and database code restructuring near-complete
● UI development underway
● Licensed under the GPL, commercial license available
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 21/25
GRAPHCONNECT
Future Plans● Production grade by end of year
● Purchased support
● “Real digital signatures, compression, encryption
● Other security enhancements
● Much more discovery
● GUI
● Alerting
● Reporting
● Add Statistical Monitoring
● Best Practice Audits
● Dynamic (aka cloud) specialization
● Hundreds more ideas
– See: https://trello.com/b/OpaED3AT
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 22/25
GRAPHCONNECT
Get Involved!
Powerful Ideas and Infrastucture
Fun, ground-breaking project
Looking for early adopters, testers!!
Needs for every kind of skill● Awesome User Interfaces (UI/UX)● Evangelism, community building● Test Code (simulate 106 servers!)● Python, C, script coding● Documentation● Feedback: Testing, Ideas, Plans● Many others!
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 23/25
GRAPHCONNECT
Resistance Is Futile!
#AssimProj @OSSAlanR
#AssimMon
Project Web Sitehttp://assimproj.org
Blogtechthoughts.typepad.comlists.community.tummy.com/cgi-bin/mailman/admin/assimilation
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 24/25
GRAPHCONNECT
The Elder GeekGirl
GraphConnect4 October
2013
© 2013 Assimilation Systems Limited 25/25
GRAPHCONNECT
Younger GeekGirl's Computer
Running LinuxOf Course!