Charles Wheelus - .conf2017 | The 8th Annual Splunk ... · Charles Wheelus, MSCS • Senior Data Scientist, Cequint ... research interests: Data Mining and Machine Learning • 2012

SLA Compliance Assurance

Charles WheelusSenior Data Scientist, Cequint

Splunk .conf 2013October 2nd, 2013

1

Thursday, October 3, 13

About me:Charles Wheelus, MSCS

2



• Senior Data Scientist, Cequint

2



• Senior Data Scientist, Cequint• Ph.D. Candidate, Florida Atlantic University

2



• Senior Data Scientist, Cequint• Ph.D. Candidate, Florida Atlantic University research interests: Data Mining and Machine Learning

2



• Senior Data Scientist, Cequint• Ph.D. Candidate, Florida Atlantic University research interests: Data Mining and Machine Learning• 2012 Splunk Ninja Revolution award recipient

2



• Senior Data Scientist, Cequint• Ph.D. Candidate, Florida Atlantic University research interests: Data Mining and Machine Learning• 2012 Splunk Ninja Revolution award recipient• Splunk Certified Architect

2



• Senior Data Scientist, Cequint• Ph.D. Candidate, Florida Atlantic University research interests: Data Mining and Machine Learning• 2012 Splunk Ninja Revolution award recipient• Splunk Certified Architect • Technology consultant for 20 years

2



• Senior Data Scientist, Cequint• Ph.D. Candidate, Florida Atlantic University research interests: Data Mining and Machine Learning• 2012 Splunk Ninja Revolution award recipient• Splunk Certified Architect • Technology consultant for 20 years• Splunk user and evangelist for three years

2



• Senior Data Scientist, Cequint• Ph.D. Candidate, Florida Atlantic University research interests: Data Mining and Machine Learning• 2012 Splunk Ninja Revolution award recipient• Splunk Certified Architect • Technology consultant for 20 years• Splunk user and evangelist for three years• Started with version 4.3

2


3


About

3


Cequint provides handset and Carrier data services to most major wireless carriers in the U.S.

About

3


Cequint provides handset and Carrier data services to most major wireless carriers in the U.S.

About

http://cequint.com

3


http://cequint.com

http://cequint.com

About

4


About

4


About

5


About

5


Service Level Agreement (SLA)Compliance Assurance

Charles Wheelus October 2nd, 20136


...or

How to kill a flock of birds with one stone



Disclaimer: No birds were injured during the production of this presentation. :)



SLA Compliance


(on a Wireless Carrier network)

9


SLA Compliance


The project:


9


SLA Compliance


The project:


Develop a system that provides proof of our SLA compliance with our carrier customer

9


SLA Compliance


The project:


Develop a system that provides proof of our SLA compliance with our carrier customer

Time is of the essence!

9


SLA Compliance



SLA Compliance


What is the significance of a Service Level Agreement?

10

degradedperformance


SLA Compliance



10

+

degradedperformance

extendedperiod


SLA Compliance



10

unhappycustomer


SLA Compliance



10

happycustomer


SLA Compliance



SLA Compliance


Step 1: Determine the Key Performance Indicators

11


SLA Compliance


What are Key Performance Indicators (KPI)?


11


SLA Compliance


What are Key Performance Indicators (KPI)?

Metrics used to evaluate factors that are critical to the optimal performanceof a organization, project or system


11


SLA Compliance


Challenges

12


SLA Compliance


Challenges

• Numerous subsystems

12


SLA Compliance


Challenges


• Different development teams

12


SLA Compliance


Challenges



• Different programming languages

12


SLA Compliance


Challenges




• Different operating systems

12


SLA Compliance


Challenges




• Different operating systems

• Wide variety of hardware types

12


SLA Compliance


The “Cat Herder”

13


SLA Compliance


Determine what data to get

14


SLA Compliance



Study the SLA

14


SLA Compliance



Study the SLA

Engage others in the process

14


SLA Compliance



Study the SLA


• Developers

14


SLA Compliance



Study the SLA


• Developers• Management

14


SLA Compliance



Study the SLA


• Developers• Management• Product team

14


SLA Compliance



Study the SLA


• Developers• Management• Product team• Operations

14


SLA Compliance


Determine the best place(s) to get the data from

15


SLA Compliance


Determine the best place(s) to get the data from

15


SLA Compliance


Establish best practice for data input

16


SLA Compliance



What simple step can you take in the beginning that will save time later?

16


SLA Compliance




Best practices document

16


SLA Compliance




Best practices document

Verify the data is in the expected format!

16


SLA Compliance


Determine transport method for getting the data into Splunk

17


SLA Compliance



syslog

17


SLA Compliance



syslogUDP

17


SLA Compliance



syslogUDP

17


SLA Compliance



18


SLA Compliance



UniversalForwarder

18


SLA Compliance



19


SLA Compliance



19


SLA Compliance



19


SLA Compliance



19


SLA Compliance


KPI Data flow diagram

20


SLA Compliance


SLA report (RECAP):

21


SLA Compliance


SLA report (RECAP):

• Establish KPI

21


SLA Compliance


SLA report (RECAP):

• Establish KPI

• Get KPI data into Splunk

21


SLA Compliance


SLA report (RECAP):

• Establish KPI


• KPI counter aggregation and reconciliation

21


SLA Compliance


SLA report (RECAP):

• Establish KPI



• Use Splunk REST API to build the report

21


SLA Compliance


SLA report (RECAP):

• Establish KPI



• Use Splunk REST API to build the report

21


SLA Compliance







“Black-box” testing





The problem:

24




The problem:

Performance information about the Carrier’s self provisioning gateway is unavailable. We have to run our own tests to determine the expected performance

24




The problem:

Performance information about the Carrier’s self provisioning gateway is unavailable. We have to run our own tests to determine the expected performance


24


















Black Box Testing



Black Box Testing



Black Box Testing



Black Box Testing









Load test results analysis





The problem:

28




The problem:

We need a quick way to evaluate the results of load testing.

28




The problem:

We need a quick way to evaluate the results of load testing.


28


















Event Reporting



Event Reporting


The problem:

32


Event Reporting


The problem:

Thousands of subsystem events may be generated into the log files, some events are inter-dependent. We need a comprehensive and robust system for detecting, correlating, and reporting these events to the correct development team.

32


Event Reporting


The problem:

Thousands of subsystem events may be generated into the log files, some events are inter-dependent. We need a comprehensive and robust system for detecting, correlating, and reporting these events to the correct development team.


32


Event Reporting



Event Reporting


The solution:

33


Event Reporting


The solution:

Splunk saved and scheduled searches!

33


Event Reporting


The solution:

Splunk saved and scheduled searches!

With very brief training, the developers are building their own queries, saving and scheduling

33


Event Reporting



Event Reporting



Event Reporting








Event Monitoring and Alarming

35



The problem:


35



The problem:

The operations team requires that the KPI produce alarm output into their pre-existing monitoring and alarm system


35



The problem:

The operations team requires that the KPI produce alarm output into their pre-existing monitoring and alarm system



35





• Operations has pre-existing alarming software




• Operations has pre-existing alarming software

• Splunk was connected to OPS alarm system using the Splunk API















Performance Analysis

38



The problem:


38



The problem:

The entire team needs to have up to the minute business intelligence.


38



The problem:

The entire team needs to have up to the minute business intelligence.



38




39



The answer:


39



The answer:

Splunk Dashboards and Apps!


39






• Customized tools for Developers





• Dashboards for Operations






• Trouble shooting for Developers and Operations






• Trouble shooting for Developers and Operations

• Business Intelligence for Management



41


41


41


41


41


41


41



Cut to the chaseSplunk’s greatest benefits:

42


•Time savings



42


•Time savings•Ability to react quickly (SPL)



42


•Time savings•Ability to react quickly (SPL)•Real time analytics



42


•Time savings•Ability to react quickly (SPL)•Real time analytics•Rapid dashboard production



42


What’s next?



What’s next?


• Deeper analytics

43


What’s next?


• Deeper analytics• New metrics & dashboards

43


What’s next?


• Deeper analytics• New metrics & dashboards• Modular inputs

43


What’s next?


• Deeper analytics• New metrics & dashboards• Modular inputs• More use of Splunk Apps

43








Charles Wheelus

http://about.me/charleswheelus

http://facebook.com/charleswheelus

[email protected]

45






mailto:[email protected]

mailto:[email protected]

Documents

Charles Wheelus - .conf2017 | The 8th Annual Splunk ... · Charles Wheelus, MSCS • Senior Data Scientist, Cequint ... research interests: Data Mining and Machine Learning • 2012