Download pptx - Optimizing IT Costs & Services With Big Data (Little Effort!) - Case Studies - #Pink13

Copyright © 2012 TeamQuest Corporation. All Rights Reserved.

TeamQuest and the TeamQuest logo are registered trademarks in the US, EU and elsewhere.All other trademarks and service marks are the property of their respective owners.

Optimizing IT Costs & Services with Big Data, Little

Effort…David Wagner

TeamQuest Advocate


• Why?• Big Data: conceptual overview • 2013 Capacity Management 101: – History– Goals– Obstacles

• New “Big Data” approaches– Concepts– Case Study Value

Agenda


• We passionately believe always having and using the right amount of resources is a societal imperative – Anything less is failure– Anything more is wasteful

• 20+ years sole focus– ensuring our customers can continuously and

automatically perform at their utmost level of efficiency

– ensuring business service performance, conserving scarce resources, saving money and improving productivity

• We call this: IT Service Optimization

Why does TeamQuest Exist?


• IS:– Applying Big data approaches to Capacity

Management• Faster and larger value• More scalable

– New ways to think about optimization beyond ITIL Capacity Management• Include ITIL Service Management and Delivery• Not just technology anymore

• Is NOT!– A Primer on Big Data or a Big Data “how to”

Presentation• Hadoop ecosystem deep dive, etc…

What this Presentation is… and is not!


• Big Data is about: data actionable information– Plethora of existing sources

• Technology• Business (Sales, Marketing, …)• Service (Transactions, SLAs, …)

– Learning new insights from “old” data– Key is Analytics

• Deep• Wide• Adaptable

• But… Optimizing costs with Capacity Management?

Big Data at 50,000 feet…


• Data Access and Aggregation– Build huge “data marts” (aka: Data Warehousing)– Integrate with multiple different data sources

• Technology (e.g. Server, Network, Storage, etc.)• Service (Catalog, Metrics, Tickets, etc.)• Business (KPIs, Plans, Transactions, etc.)

• Implement Analytics against/across– Flexible and adaptive– Turn data within, into actionable information across

• But… Capacity Management???

Technology Approaches


• Answering “what if” questions…– Change in technology, demand, etc… impact?– Focus on Optimizing Server Cost versus

Performance• Extremely Technology-centric– Servers, Mainframes– Occasionally Storage or Network – in isolation

• Big Value and Return, but also effort– Highly trained staff– Required building a central, massive datamart

(CMIS)– Scalability of Staff, Tools, …, Politics

2013 Capacity Management 101 - History


• Maintain traditional value, and add– Optimize– Amplify– Accelerate

• Increase Business relevance– Valuable predictive analytics in business and

service context– Optimize Efficiency

• Virtualization and Cloud Scale to everything– Many to many inter-relationships; Capacity critical

2013 Capacity Management – Goals: What


• Integrate and Analyze across multiple sources– Technology (e.g. Server, Network, Storage, etc.)– Service (Catalog, Metrics, Tickets, etc.)– Business (KPIs, Plans, Transactions, etc.)

• Single pane of “Analytic Glass”– Ability to tie together, correlate, and operate across

• Tear down the wall!– Don’t force reinvention… or data duplication!– Flexible and adaptive– Turn data within, into actionable information across

2013 Capacity Management 101 – Goals: How


• Data Access and Aggregation– Building huge “data marts” (fka: Data

Warehousing)• Complexity = (data ETL) x (# sources) x (maintenance

effort)• Compliance: Data duplication, privacy, audit, etc…• Costly and time consuming

• Implementing Analytics against/across– General purpose BI Analytics for Capacity?– Traditional Performance/Capacity for General

Purpose?• “Big Data” + ITIL = Optimized Capacity

Management?

2013 Capacity Management 101 - Obstacles


• Service Strategy– Financial management

• Service Design– Service Level and Availability management

• Service Transition– Asset, Change and Configuration Management

• Service Operations– Service Desk– Application and IT operations– Event, Incident, Problem

• Or, in simpler terms… – Integrate Capacity across ITIL V2: Service Support and Service Delivery!

Capacity Management with ITIL 2011


• Leverage the data (and tools) you have!– Don’t reinvent or reimplement

• Quickly and easily with True Federation– Use existing data/tools already in place– Don’t force data duplication, ETL– Capacity Analysis across data sources

• Key ITIL discipline metrics amplify Capacity Management Value– Strategy factor financials– Design factor Service Levels, technology performance– Transition track business and technology changes– Operations factor Service risks, multiple technologies

Optimized Capacity Management


• ITIL: Strategy– Capacity Management integrated with Financial

costing/reporting• ITIL: Design

– Capacity Management integrated with Risk Registry• ITIL: Transition

– Includes integration with Asset and Configuration Management

• ITIL: Operations– Integration with Service Desk– Operations factor Service risks, multiple technologies

Integrated Case Study Walkthrough


Very Large BankAs an IT Shop:• Operate tens of thousands of servers• Every server platform under the sun• Manage dozens of data centers• Huge mainframe with many thousands

of MIPS• Thousands of VMs• Thousands of VDIs & Citrix• Many Petabytes of storage

14


Seamless data integration & analysis

1. All capacity/performance data2. All platforms, OS’s, …3. Configuration data4. Change records5. Risk registry

15


Deliverable: Fully Automated Application Report

We need:1. Risk detection and tracking2. Risk reporting3. Actionable information

Reporting has to be:• Automated• Repeatable• Human-readable – financials, business terms, not “speeds and

feeds”

16


Analysis Overview

17

Application and Configuration from Service Catalog and CMDB


Usage Patterns

18

Time Series data from Performance and Event

Management


Service Desk and Risk management

19


Scal

eabl

y ID

Pos

sible

Ex

istin

g Ca

pacit

y Iss

ues

20


ID P

ossib

le Fu

ture

Ca

pacit

y Ri

sks

21


Fixed Costs / Variable Costs - Method

22

Source: wikipedia.org

Variable Costs

Fixed Costs


Capacity Management + Strategy (Financials)

Fixed/Variable Cost

23

server0009b01a - Excess Capacity ReportProduced by the Server Capacity & Performance Management (SCPM) Team

Analysis Period: August 01 2010 to August 31 2010Run Time: 4:09 PM September 27 2010 (8 seconds)

Purpose: To analyze the system's current resource consumption and compute the available headroom based on a fixed/variable costs methodology and our rules-of-thumb. This report also attempts to determine the nearest bottlenecks, from a consumption perspective.

server0009b01a: Maximum Growth Capability by ResourceName Growth Vaule

CPU RunQ Length Growth 2.15Disk - 0 4.41Memory Utilization Growth 5.48FS - C: 10.61Virtual Memory Growth 20.27CPU Growth 38.10Net In 100MB - NIC1 260.82Net Out 100MB - NIC1 349.39Net In 1GB - NIC1 2608.18Net Out 1GB - NIC1 3493.92

server0009b01a: Top 10 PIDsNAME PIDGROWTH SLOPE MINCPU AVGCPU MAXCPU

System:4 17.54 0.00 0.06 0.09 5.18NTRtScan:1660 29.88 -0.00 0.00 0.02 3.01beasvc:1080 47.72 -0.00 0.00 0.14 1.89svchost:840 184.41 -0.00 0.03 0.07 0.52svchost:872 213.22 0.00 0.05 0.08 0.47TmListen:2160 763.86 -0.00 0.00 0.01 0.12python:1788 848.86 0.00 0.00 0.03 0.11wmiprvse:268 987.95 -0.00 0.00 0.04 0.09wmiprvse:2228 1322.98 0.00 0.02 0.04 0.09wmiprvse:2044 1328.88 -0.00 0.02 0.05 0.09






VM Optimization Analysis• Thousands of VMs• Some too small• Some too big• Some idle• Which ones?• What size should they be?

26


Physical to Virtual Analysis

27


28

Total Virtual Machines Idle Virtual Machines Oversized Virtual Machines22 3 13

vCPUs Max % CPU Max % CPU Ready

Avg % CPU

Avg % CPU

ReadyTotal

MemoryMax

Memory Used

Max % Memory

Util

Average Memory

Used

Avg % Memory

UtilRecommended SSO vCPUs

Recommended SSO

Memory in GB

CLUSTER0019V019 4 0 0 0 0 4096 0 0 0 0 1 2CLUSTER0019V024 2 0 0 0 0 2000 0 0 0 0 1 2CLUSTER0019V029-OLD_DO_NOT_USE 4 0 0 0 0 4096 0 0 0 0 1 2

vCPUs Max % CPU Max % CPU Ready

Avg % CPU

Avg % CPU

ReadyTotal

MemoryMax

Memory Used

Max % Memory

Util

Average Memory

Used

Avg % Memory

Util

Recommended SSO vCPUs

Recommended SSO Memory in

GBCLUSTER0019V001 2 40 5 9 1 2044 1921 94 1729 85 1 4CLUSTER0019V003 2 30 10 7 2 2048 1895 93 1737 85 1 4CLUSTER0019V004 2 45 51 3 3 2048 1914 93 1333 65 2 4CLUSTER0019V005 2 41 17 8 2 2048 1955 95 1716 84 2 4CLUSTER0019V006 2 45 41 2 2 2048 1963 96 1510 74 2 4CLUSTER0019V008 2 27 23 2 2 2048 1860 91 1232 60 1 4CLUSTER0019V013 2 30 40 3 3 2048 1845 90 1326 65 1 4CLUSTER0019V014 2 30 36 3 2 2048 1834 90 1286 63 1 4CLUSTER0019V018 2 32 30 3 2 2000 1843 92 1581 79 1 4

CLUSTER0019V029-REAL 4 42 30 5 4 4096 3612 88 2951 72 4 8

CLUSTER0019V030 2 47 23 2 2 2048 1881 92 1387 68 2 4CLUSTER0019v009 2 43 14 2 1 4096 3117 76 2258 55 2 4CLUSTER0019v010 2 30 18 2 1 4096 3102 76 2151 53 1 4

Capacity Optimization Candidates

Idle Virtual Machines

Oversized Virtual Machines


Delivered:• Repeatable processes• Quicker analysis• Powerful • Flexible

29