-
APPLY BIG DATA ON STATISTICS AND MONITOR QUERY TO NATIONAL DNS
OF VIETNAM
Danang City | 24 August 2018
-
Content
1. Overview
2. Apply Big Data on log analyze
3. Deploy the new monitoring system using Big Data
4. Results and conclusion
-
National DNS of Vietnam
• Most national critical information system.
• Established, managed, operated and secured by VNNIC.
• Distribution: – 5 domestic clusters in
Hanoi, HCM City and Danang City
– 2 oversea clusters
-
Query .VN domain
Query log: – Format: ISC BIND
– Rate: 4000 qps
– Velocity: 60GB/day
– Volume: • 20TB minimum (for 1
year)
• Longer for report and statistics
-
DNS Log Monitoring System
-
Traditional approach
Database Server
SAN Storage
Stats and monitor Server
Syslog server
DNS Server
DNS Server
DNS Server
Send log
Send log
Send log
Insert data
Read & write
Return data
Process data
Query data
Log analyser
User request
Send result
-
The limitations of traditional approach
• Limitations: – Storage capacity and speed – Data processing
performance – Scalability – High cost
• Solution with traditional approach: – Reduce the sample time –
Eliminate information – Less monitor and statistics cretiria
Reduce the value of information!
-
New approaches
• Main goal: horizontal scability in data storage and
processing. – Distributed storage with able to parallel access
(read and
write). – Log collecting can perform independently and parallel.
– Massive parallel in data processing.
• Best solutions: – ELK Stack – Splunk – Hadoop (Big data)
-
Content
1. Overview
2. Apply Big Data on log analyze
3. Deploy the new monitoring system using Big Data
4. Results and conclusion
-
Big Data Hadoop
Hadoop Distributed File System (HDFS): • Filesystem written in
Java ( based on Google ‘s GFS) • Sit on top of a native filesystem
(ext3, ext4, nfs,…) • Provides redundant storage for massive
amounts of
data. • Data is distributed across all nodes.
Yet Another Resource Nagotiator (YARN): • Platform for managing
resources in Hadoop cluster. • Allocate and schedule resources
(CPU, memory)
usage. • Run the distributed data processing application . •
Monitor the application process.
-
HDFS
Master/Slave architecture: • Datanode: read and write blocks of
large files as request from client. • Namenode: holds all metadata
(file locations, file ownership and
permissions, name of individual blocks, locations of
blocks).
-
YARN
• Resource Manager (RM): – Manage Node Managers (track heart
beat) – Determine how resources are allocated – Create Application
Master and track heart beat – Create containers (CPU, Memory) as
request
from Application Master. Deallocate containers when expired or
application completed.
• Node Manager (NM): – Register and provide information on
node
resources to Resource Manager – Launch Application Masters on
request from
Resource Manager. – Launch application processes – Monitor
resouces usage by containers.
• Job History Server (JHS): – Archives MapReduce jobs ‘s metric
and
metadata
-
Hadoop ecosystem
-
Applying Big Data on log analyse
Data ingestion
Data storage
Data processing
Data source
-
Deploy Hadoop Big Data Cluster with Cloudera
-
Content
1. Overview
2. Apply Big Data on log analyze
3. Deploy the new monitoring system using Big Data
4. Results and conclusion
-
Architecture of DNS Log Monitoring System
DNS SERVER
DNS SERVER
DNS SERVER
DNS SERVER
FLUME COLLECTOR
MONITORING(BANANA)
STATISTICSAPPLICATION (JDBC)
FLUME COLLECTOR
-
DNS Query Log ingestion with Flume
-
Deploy Flume with HA
DNS Server
Source
Channel
AvroSink
FLUME AGENT
DNS Server
Source
Channel
AvroSink
FLUME AGENT
DNS Server
Source
Channel
AvroSink
FLUME AGENT
AvroSource
Channel
Sink
FLUME AGENT
DNS Server
Source
Channel
AvroSink
FLUME AGENT
AvroSource
Channel
Sink
FLUME AGENT
HDFS
BIG DATA CLUSTER
-
DNS Log Analyze
AvroSink
AvroSink
AvroSink
AvroSource
Channel 2Solr Sink
FLUME AGENT
Channel 1HDFS Sink
AvroSource
Channel 2Solr Sink
FLUME AGENT
Channel 1HDFS Sink
HDFS
SOLR
• Perform in Flume Collector • Using Morphlines:
• Interceptor • MorphlineSolrSink
• Output: • HDFS Sink: Hive table data
(Data Warehouse), can query by Impala. Use for BI, Machine
Learning.
• Solr Sink: Search engine for monitoring system’s query.
-
DNS Log Analyze with Morphline
• Morphlines support: – Grok – GeoIP (Maxmind2
database) – Translate – Conditional (IF THEN
ELSE) – Java code – Load solr
• Parse unstructured log data into fields: client ip, server ip,
query time, domain, query type, record type,…
• Enrich the information: • TLDs, type of domain (ASCII, IDN) •
Version of IP Address (v4 or v6) • Location of client • Query
parameter (DNSSEC,
TCP/UDP, EDNS,…)
-
Data visualization with Banana
Lucidworks Banana: • Similar to Elastic’s Kibana • Use Solr for
data analysis and
display • Based on dashboards, which
contain rows of panels • Supported panels:
– Histogram Panel – Heatmap Panel – Table Panel – Term Panel
-
Content
1. Overview
2. Apply Big Data on log analyze
3. Deploy the new monitoring system using Big Data
4. Results and conclusion
-
Dashboard of new DNS Log Monitoring System
-
Features of new DNS Log Monitoring System
• Near realtime monitoring system. • Time picker: relative,
absolute. • Flexible filters:
– Field’s value as click on panel. – Advandce filtering.
• Monitoring criteria: – Number of query in DNS Clusters, DNS
Servers – Top client IP Addresses – Top domain (.VN, other TLDs) –
Client location – Record type – IPv4/IPv6 query – DNSSEC or non
DNSSEC query – Query protocol (TCP/UDP). – Recent DNS queries.
-
Conclusions
• Tradional approach for log analyse has many limitations:
performace, scability.
• Big Data can solve this problem and bring more valuable
information from raw data sources.
• Our solution can apply on: – ISP DNS Server’s query log. –
Other IT systems’ log.
• Future plan: – Apply machine learning on data warehouse to
secure the National
DNS System, .VN domain names. – Apply BI for operating the
National DNS System and making policy on
Internet Resources management.
-
Thank you!
