Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
presented by
Unstructured Data To Interactive Graph
Jacob KomarovskiXpoLog, Turn Data Into Action
presented by
XpoLog – End to end log management solution• Collection of logs data from different sources • Automated analytics engine utilizing NLP• Visualize logs data using dashboards and widgets• Continues monitoring and alerting• Enterprise ready
XpoLog standard cluster• Collects 1.5+ TB of data per day• Analyze 1000’s of different logs
XpoLog – Log Management
presented by
• DevOps and Operations– Log Analysis Platform
– Search events and patterns
– Troubleshoot and Analyze IT and Apps
– Reporting and Monitoring
• IT Data– ML and NLP Powered Search– Identify bottlenecks and faults– Analyze Audit Data– Quickly identify risk
XpoLog – Use Cases
presented by
Why visualize log data as a graph?• Easy exploration
– Show the big picture– Visualize cross logs transactions– Fast root cause analysis
• Data Augmentation – Paths coloring– Easy visualization of enhanced data
• Rules• Tags• AI\ML engines• Risk
Why Graph
presented by
Explored and tested several technologies• Main feature is UX
– Performance – in memory, uses sparse adjacency matrices– Attributes on Nodes and Edges – Labeled Nodes – Consolidate different types of data sets
• On Premise solution – Easy deployment– Industry standard– Easy to grow with the data - Clustering
• First implementation of 3rd party graph solution– Because of YOU – great community
Why Redis Graph
presented by
XPLG Data Flow
Data Collection
Connectors• Push• Pull• Cloud
Automated Pattern recognition
Running Processing Engines
• Index
• Analytics
• Rules
• Graph
UI Visualization
• Search console
• Dashboards
• Graph
Automation
• Complex Algorithms• AI and NLP• Graph theory
applications
• Monitor & Act • Rest API calls• Scripts
• Alert• Slack, Teams…• Mail
presented by
Graph Data Flow
Log4J
ParserNLP and Statistical Analysis
XplgGraph
Redis Graph
UI Model
D3 Visualization
Monitoring & Alerting
Data Augmentation
Configuration FileStructured DataRule Engine
presented by
Add ability to create graph model on any type of log in different contexts• XpoLog Parsers results in structed data• Graph structure defined using a configuration file
– Nodes structure and representation in the DB– Connections between nodes– UI representation attributes
Generic Graph Parser and Engine
presented by
Graph Structure• Every real world entity (e.g. user, server….) is
represented by one node in the graph• Every real world transaction is represented by
one edge in the graph
Our use case• Initially, mostly new nodes• After a while, small number of new nodes and
mostly new edges
Adding New Data To The Graph
presented by
Merging a small graph into a large graph• Merge every node• Create all edges
Observations• Nodes should be cached (using Redis for example)• After a while, almost no nodes will need to be
merged• Graph merging is challenging …
Adding New Data To The Graph
presented by
Query• Is a Graph object• Same as xplgGraph• Every node and edge have a filter that we apply on the graph
Response• Is also an xplgGraph• Has the same structure as the query graph
Transforming Java object to Cypher query
presented by
Java to Cypher• An AST is built from the query graph and then transformed into a Cypher query
compatible with RedisGraph• Achieved by building on top of jCypher (https://github.com/Wolfgang-
Schuetzelhofer/jcypher)• JRedisGraph (https://github.com/RedisGraph/JRedisGraph) is used to
communicate to Redis
Transforming Java object to Cypher query
presented by
Data visualization• Bi directional data flow
– Graph model translated to json structure with UI properties– User interactions translated to a query graph and then transformed into a Cypher query
compatible with Redis Graph
• Visualize using D3 library– Wrap D3 inside XpoLog product
Graph Model -> UX
presented by
Goal - combine graph representation in XpoLog Dashboards, running over 1TB data per day for up to 5 years• Split stored graph – creating a forest• Create pre-processed aggregated graphs
– Aggregate overtime in advance
• Using dumps for hot/cold data
Future Challenges - Dealing With Big Data and UX
presented by
Running continues testing from scripts and XpoLog application• First graph building from existing environment
– 10#$ of events from different logs normalized to graph structure• Insert in bulks• Side process• Hot/Cold storage
– Many new nodes and edges
• Continues graph maintenance – Simulate data collection process
• Non stop update of nodes• Adding new edges
– Maintain pre-processed aggregations
Testing Benchmarks
presented by
DEMO
presented by
Thank you!