Upload
softwarecentral
View
399
Download
1
Embed Size (px)
DESCRIPTION
Citation preview
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 1
StickyMinds.com and Better Software magazine presents…
Whose Environment is It Anyway?
Performance Tuning in a Virtual Environment.
Sponsored by Empirix
Non-streaming participants should call 1-866-761-8643International Non-streaming participants should call 1-904-596-2362
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 2
Scott Barber, (A.K.A. “The Perf Guy”)[email protected]
Chief Technologist, PerfTestPlus, Inc.www.perftestplus.com
Vice President & Executive Director, Association for Software Testingwww.associationforsoftwaretesting.org
Co-Author, Performance Testing Guidance for Web Applications www.amazon.com/gp/product/0735625700
A Heuristic Approach
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 3
Developers write code
Developers release code for test
Testers test code, often in various environments
Testers assess system performance in not very production-like environments
Managers make decisions based on test results
Some defects get fixed before release
The team gets surprised by defects in production
The team puts in lots of overtime to fix the defects
Testing in a Non-Virtual World
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 4
Wikipedia lists 8 categories of platform virtualization and 4 categories of resource virtualization
Platform Virtualization• Software simulates a computer environment on a host
machine, usually called “virtual machines”
• Applications run on “virtual machines” as if on a stand-alone hardware platform
• As if stand-alone ≠ Isolated
• Platform virtualization software has overhead
(http://en.wikipedia.org/wiki/Virtualization)
What is Virtualization?
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 5
Resource Virtualization• Identify a resource and exclusively assign it’s use to a
specific application
• Sometimes partitions single resources, sometimes spans multiple resources
• Resources may or may not be co-located on a physical machine
• Exclusively assigned ≠ Isolated
• Resource virtualization has overhead
(http://en.wikipedia.org/wiki/Virtualization)
What is Virtualization?
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 6
Developers write code
Developers release code for test
Testers test code, often in various configurations
Testers assess system performance in not very production-like environments
Managers make decisions based on test results
Some defects get fixed before release
The team gets surprised by defects in production
The team puts in lots of overtime to fix the defects
Testing in a Virtual World
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 7
Virtualization, if done well, can:• Dramatically reduce the overhead involved in managing
test environments
• Create opportunities for unique, targeted, ad-hoc environments without additional hardware costs
• Simplify configuration and release management
• Put the test team in control of their test environments
But what about performance testing?
What’s the Difference?
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 8
“With an order of magnitude fewer variablesperformance testing could be a science,
but for now,
performance testing is at best a scientific art.”
--Scott Barber
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 9
Test environments are still test environments
Virtual test environments are less likely to match virtual production environments
There are more places for performance issues to hide in virtual environments• Exclusively assigned ≠ Isolated
• As if stand-alone ≠ Isolated
• Virtualization adds overhead
Study: “virtualized servers can handle 14% lower [load] than traditional server configurations” (http://www.sys-con.com/read/503412.htm)
Testing Virtual Performance?!?
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 10
“Linear extrapolation of performance test results is,
at best, black magic.
Don’t do it.
(Unless you learned from Connie Smith, PhD or Daniel Menasce, PhD)”
--Scott Barber
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 11
If done well, there can be benefits:• Dramatically reduce the overhead involved in managing
test environments
• Easier for performance testing to have it’s own test environment
• Enable collaborative tuning without having to “borrow” hardware resources
• Put the test team in control of their test environments
Virtualization does not change the core principles of successful performance testing
Testing Virtual Performance?!?
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 12
“Only performance testing at the conclusion of system or functional testing
is like
ordering a diagnostic blood test after the patient is dead.”
--Scott Barber
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 13
Core Principles
Project context is central to successful performance testing.
Business, project, system, & user success criteria.
Identify system usage, and key metrics; plan & design tests.
Install & prepare environments, tools, & resource monitors.
Script the performance tests as designed.
Run and monitor tests. Validate tests, test data, and results.
Analyze the data individually and as a cross-functional team.
Consolidate and share results, customized by audience.
"Lather, rinse, repeat" as necessary.
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 14
Contact Info
Scott Barber
Chief TechnologistPerfTestPlus, Inc
E-mail:
[email protected] Web Site:www.PerfTestPlus.com
Whose Environment Is It Anyway?Whose Environment Is It Anyway?Dan Koloski
Web BU CTO and Director of Strategy
Empirix
IMAGE
Empirix – At a GlanceEmpirix – At a Glance
Testing & Monitoring Solutions for:
Privately held, founded 2000
380+ employees
Worldwide presence; HQ in Boston & offices in Japan, UK, Germany and South Korea
Patents in test, monitoring and VoIP diagnostics methodology
Web Applications & SOA
Web Applications & SOA
Contact Center & CRMContact Center & CRM
Carrier-Class VoIP & TelcoCarrier-Class VoIP & Telco
Empirix in the Quality EcosystemEmpirix in the Quality Ecosystem
Empirix Offerings Delivered OutcomeEmpirix Offerings Delivered Outcome
PRODUCTSPRODUCTS
MANAGED SERVICESMANAGED SERVICESMANAGED SERVICESMANAGED SERVICES
EXPERT SERVICESEXPERT SERVICESEXPERT SERVICESEXPERT SERVICESOnsite Consulting, Training, Mentoring
e-TEST suite™OneSight™
e-LoadExpert™
Do-it-yourself
Remote ValidationPerformance Tuning
Supplement Your TeamWe Do it for You
A few points to add to Scott’s presentation…A few points to add to Scott’s presentation…
Scott said…– Virtualization does not change the core principles of
successful performance testing
Why?– Hardware is not virtual
– Whose environment is it, anyway?
– Rapid Bottleneck Identification Tuning a Virtualized Context
Hardware is not virtualHardware is not virtual
Virtualization is app/OS or hardware emulation specific
Underneath it all is a fixed pool of HW resources!
Performance Testing is designed to understand the impact on the hardware – as seen from the perspective of the user
Common Virtualization Types. Source: QAZone Whitepaper, Virtualization from the perspective of a QA/IT Operation Professional, http://qazone,empirix.com.
The Best Performance Data in a Virtual EnvironmentThe Best Performance Data in a Virtual Environment
Understand the impact on the hardware…– Requires accurate workload stimulus representing activity on and
through all VMs associated with that device
– Requires measurement of hardware characteristics at the base layer as well as OS and app-level characteristics in each VM
– Requires real-time collaboration during load testing to differentiate intra-VM bottlenecks (business logic) vs Hardware bottlenecks (CPU, Memory, NIC)
…as seen from the perspective of the user!– Your users don’t care about Virtualization
– Metrics that matter are end-user perceived performance
– Performance testing from inside the VM helps create stimulus but doesn’t report on end-user-perceived performance
Rapid Bottleneck Identification Load Testing in a Virtual EnvironmentRapid Bottleneck Identification Load Testing in a Virtual Environment
What is Load testing?
“Testing conducted to isolate and identify the system and application issues (bottlenecks) that will keep the application from scaling to meet its performance requirements”
What is a Bottleneck?What is a Bottleneck?
Any resource (hardware, software or bandwidth) that places defining limits on data flow or processing speed: your application is only as efficient as its least efficient element
On the Web, bottlenecks directly affect performance and scalability
Most untested systems have more than one bottleneck, but they can only be identified and resolved one at a time.
Where are the Bottlenecks?Where are the Bottlenecks?
The vast majority of bottlenecks are
caused by throughput
limitations – not concurrency.
Throughput80%
Concurrency20%
Source: Empirix
Sample throughput limitations in virtual environmentsSample throughput limitations in virtual environments
Disk I/O– Multiple applications accessing shared disk resources
– Virtual Memory Paging on shared disk resources
NIC– Multiple VMs shipping data through the same physical NIC
CPU– Processor-intensive (i.e., database) applications will compete for shared
CPU time.
Memory– Depends on type of VM configuration how much of an issue this can be
Whose Environment Is It Anyway?Whose Environment Is It Anyway?
Virtualization blurs the line between Dev/Test/Ops environments and ownership– Can exacerbate “silo” problems between Dev/Test/Ops
However, there is an opportunity…
…A common set of measurements can un-blur the line– Test data should inform production SLAs
– Common measurement/metrics can bridge the gap
– Measurement consistency is the cornerstone of ALM
Load Testing Tools DON’T CollaborateLoad Testing Tools DON’T Collaborate
Load Testing Tools Apply Load and Make Measurements
People (and their expertise & experience) Find Bottlenecks & Resolve Them
Collaboration With Experts Resolves Issues Faster & Cheaper
Key to Successful Collaboration = Access to People & Access to Measurements
Tools Should Support Collaboration
e-TEST suite is designed for collaboratione-TEST suite is designed for collaboration
Distributed, high-scale load controlMulti-user Web UI for real-time individual analysis
Each user can view results from the same load test as it runs
SummarySummary
Hardware is not virtual – virtualization does NOT change core performance test best practice
However…– Virtualization introduces new test planning challenges to cover
– Workload/stimulus
– Measurement of impact on end-user
In a world of scarce time, focus efforts on throughput-related workload to identify most common bottlenecks
Visit QAZone for the slidesVisit QAZone for the slides
http://qazone.empirix.com
Free resource, open to all (Empirix & non-Empirix users)– Forums (both QA-centric and
Product related discussions)
– Knowledge Base
– Resource Center
– Events, Announcements, Community & Industry News
Allows members to interact with peers, industry experts and a larger Empirix audience
www.PerfTestPlus.com
© 2008 PerfTestPlus, Inc. All rights reserved.
Performance Tuning in a Virtual Environment Page 30
Have a question for the speakers? Ask now.
Q & A