BigData Analytics On Log File And UI Design

Introduction:........................................................................................................................................1

Specific Focus:.....................................................................................................................................1

Generic Focus:.....................................................................................................................................1

UI Design:............................................................................................................................................2

Introduction:In below section briefly explained how big data analytics can apply for log files and what kind of user interface is appropriate for each case.

Specific Focus:1. Identify security vulnerabilities

a. Anomaly Detection: Use k-means clustering algorithm to detect anomalies, outliers, exceptions, malwares and so forth.

b. Identify DoS attacks (simulate more users than website can handle): Normally result in an abnormal number of requests (hits) in a short period of time and create firewall rules to block a specific IP Address to avoid DoS.

2. Host/Date based event statisticsa. The process for returning all the events for a particular date.b. All events in the collection for a particular host on a particular date. This kind of analysis

may be useful for investigating suspicious behavior by a specific user.3. Predictive analysis

a. Categorizing the errors, events and co relating the errors would provide predictive analysis on system down.

4. Automated route cause analysisa. E.g.: Why the memory utilization is high at specific time? Correlate system log with web

log with specific time and provide the memory utilization is high because of high volume of user logged in as RCA.

Generic Focus:

Web Logs:5. Visitors activity statistics:

a. Monthly, Weekly, Daily, Hourly visitors, b. Monthly, Weekly, Daily, Hourly Hits,c. Monthly, Weekly, Daily, Hourly Bandwidth,d. Monthly, Weekly, Daily, Hourly Visit Duratione. Monthly, Weekly, Daily, Hourly Page Views per Visitorf. Monthly, Weekly, Daily, Hourly Pages by View Timeg. How long they staying in.

Big data Analytics On Log Files And UI Design Page 1

6. Visitors system statisticsa. Operating Systems used,b. Browsers used,

7. Geo location statisticsa. N/W traffic by county/countriesb. Visitorsc. Hitsd. Bandwidthe. Time served

8. Error statistics

System Logs (iostat, vmstat, netstat):9. Predict future resource needs based on long-term predictive reports

a. These reports generate a long-term trend line for performance. The data from these reports is often used with a linear regression to predict when additional RAM memory or CPU power is required for the server.

10. Capacity planning predictions a. Use graph algorithms to predict to evaluate hardware resources required for the specific environment.

UI Design:1. Monthly calendar heat map,

Bandwidth, Hits, Visitors counts per month in calendar view.

2. Sparkline charts order by most logged in for that week.

Big data Analytics On Log Files And UI Design Page 2

Col 1 : User id Col 2 : No.Of logins for that week Col 3 : Monthly logins for the user group by 4 weeks Col 4 : No.Of page views for that week Col 5 : Monthly page view for the user group by 4 weeks Col 6 : No.Of error codes returned for that week Col 7 : Monthly error codes view for the user group by 4 weeks

3. Daily activities table and chart Drilldown links for each hits, page views, visitors columns. Drilldown is user based for that day.

Big data Analytics On Log Files And UI Design Page 3

4. Popular pages in table and chart format Drilldown links for each hits, visitors columns.

5. No of hits distributed by module Claims, Provider, Member, Operations and so on

Big data Analytics On Log Files And UI Design Page 4

6. Infographic type display

Big data Analytics On Log Files And UI Design Page 5

Big data Analytics On Log Files And UI Design Page 6