Upload
geoffrey-joseph
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
PROPRIETARY & CONFIDENTIALPROPRIETARY & CONFIDENTIAL
Best PracticesBest Practicesin Performance Testingin Performance TestingBest PracticesBest Practicesin Performance Testingin Performance Testing
Jennifer Turnquist
Storage Service Line Director
Lionbridge Technologies
Jennifer Turnquist
Storage Service Line Director
Lionbridge Technologies
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
Lionbridge Profile Lionbridge Profile
Public Company (Nasdaq: LIOX)- Nearly $400M+ in revenues
- Profitable
Deep expertise across the application life cycle
- Application Development & Maintenance - Testing (Independent V&V)- Content Development, Conversion & Enhancement
- Globalization
Worldwide scale and capability- Over 4000 employees operating in 25 countries (Scale)
- SEI CMM Level 5 certified process model (Quality)
8 of the world’s 10 most valuable companies are Lionbridge customers BusinessWeek Global 1000, July 2004
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
Services Designed around our Client’s NeedServices Designed around our Client’s Need
LionbridgeLionbridgeA Trusted Partner Around the WorldA Trusted Partner Around the World
Global Development & Testing Solutions
Global Language & Content Solutions
Interps
Software Development LifecycleSoftware Development Lifecycle
Application Development Testing &
Certification
Maintenance & Support
Full Content LifecycleFull Content Lifecycle
Localization/Translation
Technical Publications
eLearning Courseware
Off-shore platforms leverage Off-shore platforms leverage more than staff in China, India, more than staff in China, India, and Eastern Europeand Eastern Europe
Global footprint enables local Global footprint enables local interaction and facilitates interaction and facilitates worldwide release and supportworldwide release and support
Trusted, US-based public Trusted, US-based public company protects against IP losscompany protects against IP loss
Localization services spanning more than Localization services spanning more than 80 languages80 languages
Proprietary web-architected TM and Proprietary web-architected TM and terminology solution accelerates terminology solution accelerates production and improves consistencyproduction and improves consistency
Authoring and eLearning development Authoring and eLearning development services integrate seamlessly with services integrate seamlessly with localization to address global demandlocalization to address global demand
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
VeriTest: Setting the Standard in Testing Since 1987VeriTest: Setting the Standard in Testing Since 1987
World’s largest independent testing company
- Over 400 test architects, engineers, and analysts in 11 labs across US, Europe, Asia
- Rapid expansion in VeriTest India
- From PDAs and PCs to 32-way servers
- Data center class storage lab
Industry leader
- Exclusive provider and architect of industry-leading certification programs
- Developer of PC Magazine benchmarks
- Test and publish industry standard ISP benchmarks
- Operate globally-networked onsite to offshore model
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
The Lionbridge TeamThe Lionbridge TeamLocal Connections, Global EfficiencyLocal Connections, Global Efficiency
4,000+ Worldwide StaffExperience and Efficiency
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
Today’s AgendaToday’s Agenda
• Why Test Performance?
• Different Types of Performance Testing
• Performance Testing Roadmap
• Choosing the Right Testing Tools
• Top 10 performance testing pitfalls
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
"The standard philosophy of 'test to destruction'... will probably give you an idea of roughly how many users your site can handle at once, but it won't always tell you why the site fails to function properly. And without knowing why, you're not likely to be able to do much about it..."
--Extreme Tech
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
Why Test Performance?Why Test Performance?
The internet and IT infrastructure crucial to business
Users—employees, business partners, customers—rely on portals, applications, and data to do their jobs
Cost of failure can be devastating
Performance testing in the enterprise is intermittent, cyclical, often prompted by upgrades
Testing is highly specialized
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
The high cost of The high cost of not not conducting performance testingconducting performance testing
Performance testing overlooked until disaster strikes
Lost and abandoned sales - most visible result of poor performance testing but…
Efficiency of mission-critical systems directly impacts business productivity
Preventing problems—lost productivity, lost business, lost reputation, and even injury or death—is a major incentive
Knowing the vital performance metrics = ammunition to IT departments when planning and justify purchasing decisions
Provides the ability to demonstrate to investors and other critical stakeholders that the company’s infrastructure is adequate
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
Events that trigger performance testingEvents that trigger performance testing
Build vs. buy
Evolving requirements
Technology due diligence
Consolidating servers
Deploying a SAN
Deploying or upgrading enterprise application
Migrating to a new platform
Addition of features
Response to public critique
Enhancements due to buying trends
Acquiring or merging a business
Launching new product
Enhancing web application
Promoting an offering
Doing any of the above globally
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
Performance testing is not a
one time event.
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
Conduct the right test to get the right resultsConduct the right test to get the right results
• Load Testing- Determines the response time and throughput during typical user load
• Stress Testing - Determines the peak user load
• Volume Testing - Determines the problems that occur during long-term user activity
• Component Testing - Determines the performance and behavior of a specific component
• Benchmark Testing- Measures the performance of a system or component relative to a
standard
• Transaction Cost Analysis- Determines the system resources consumed by a single transaction
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
Performance Testing RoadmapPerformance Testing Roadmap
Define Communicate
TestScript
Plan
•Identify stakeholders•Agree on goals of testing•Determine budget•Determine schedule constraints•Agree to promotion strategy•Outline resources available •Determine staffing plan•Engage test lab (if needed)
•Verify basic functionality•Generate use cases•Capture user activity logging information•Analyze user activity profile•Model user activity•Choose the tool(s)•Identify re-usable script components•Assign resources neededfor scripting, testing•Create test environment
•Design scripts•Create scripts•Validate scripts•Build script library
•Execute tests •Collect data•Analyze test results•Run possible iterations•Troubleshoot bottlenecks•Tune system•Retest•Log non-performance failures
•Outline context•Draft results•Provide feedback to stakeholders•Deliver action items•Finalize report(s)•Promote results
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
An overview of the performance testing processAn overview of the performance testing process
After initiating the test, the load generator systems to begin accessing the system under test using the designed usage patterns.
Depending on whether the test is a global, local, or isolated configuration, the load generators may be located worldwide or completely contained within a test lab.
The one critical configuration requirement for the load-generating systems is that they have adequate network bandwidth throughput capability to access the system under test in a realistic manner without bandwidth constraints.
If bandwidth constraints become a problem, adding additional load generators to the pool of load generators will typically fix this problem.
If the test is global or local, the Internet will be an important factor in the configuration. For an isolated configuration, the Internet is not a factor.
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
An overview of the performance testing processAn overview of the performance testing process
Once a performance test is initiated, it can run for several minutes to several days, depending on the test goal.
During the test time, the test tool monitors and collects performance data from all of the components within the system under test, such as the Web server, application server, or database server.
All of the monitor data along with the performance test data collected at the generating client end to determine the overall performance as well as the potential system bottlenecks.
In a typical performance test cycle, the performance bottlenecks are located, fixed, and iteratively retested to ensure that they are fixed as designed.
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
High Level Picture of the ProcessHigh Level Picture of the Process
Overcome resource limitations• Replace testers with “Virtual Users”• Run many Virtual Users on few machines• Controller manages Virtual Users• Run repeatable tests with scripted actions• Get meaningful results with analysis tools
System under TestLoad Generation
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
You don’t have to go it aloneYou don’t have to go it alone
Define Communicate
TestScript
Plan
Build and train internal resourcesHire contractors
Utilize service offerings from test tool vendorsRely on application provider
Engage with a consulting firm or SIPartner with an independent testing company
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
Important considerations for choosing the resourcesImportant considerations for choosing the resources
Deadlines
Testing skills and experience
Technology and/or application expertise
Frequency and scale of testing requirements
Infrastructure requirements
Risk assessment
Market factors
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
The vast number of performance testing tools can be The vast number of performance testing tools can be overwhelmingoverwhelming
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
Important considerations for choosing the right toolImportant considerations for choosing the right tool
Do you already own the license?
Do you have the internal resources to script and execute?
Will it meet the test objectives?
Is it compatible with your technology objectives?
Does it fit within your budget constraints?
Do you have the training and expertise to analyze the results?
Does it match the frequency of your testing needs?
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
Leading performance toolsLeading performance tools
Benchmark Tool Pros Cons
Mercury Interactive LoadRunner
•Compatible with numerous protocols
•Excellent data analysis tools
•WAN emulation
•Web transaction breakdown monitor
•Expensive license
•Requires a unique license for each protocol type
Segue SilkPerformer •Offer “Lite” version for reduced price
•Excellent data analysis tools
•Root cause analysis tools included
•Windows only
•Expensive license
RadView WebLoad •Inexpensive license
•Good data analysis tools
•Covers few protocols; primarily Web-based automation
Spirent Avalanche/Reflector •Uses SST TracePlus to provide record and playback feature
•No additional hardware required
•Expensive hardware to purchase
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
Popular Benchmark ToolsPopular Benchmark Tools
Benchmark Tool Workload SimulatedVeriTest Tools VeriTest NetBench
VeriTest WebBench
CIFS file / network traffic
•HTTP / Web traffic
Microsoft Tools WCAT
WMLS
Microsoft LOADSIM
DBHammer
HTTP(S) traffic
streaming media
Exchange email traffic
SQL database traffic
SPEC benchmarks
SPECjbb
SPECsfs
SPEC Java Business Benchmark
Exercises the CPU
NFS File / Network Traffic
Other Industry Standard Tools
IOMeter
IOZone
TSScaling
block level data transfer, OLTP(database) traffic
block level data transfer
terminal services traffic
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
Manual testing may be your best tool Manual testing may be your best tool
• Frequency of testing requirements
• Rate of change
• Limitations of available tools
PROPRIETARY & CONFIDENTIALPROPRIETARY & CONFIDENTIAL
Top 10 performance Top 10 performance testing pitfallstesting pitfallsTop 10 performance Top 10 performance testing pitfallstesting pitfalls
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
Top 10Top 10
1. Not Testing.
2. Lack of clearly defined test objectives and poor planning.
3. Relying exclusively on beta customers to find performance issues.
4. Using the wrong tool for the job.
5. Introducing too many variables simultaneously into a test.
6. Failing to test how your product or system is actually used.
7. Conducting load testing in a vacuum.
8. Treating performance testing like a one-time event.
9. Assume that the scripting effort will be short and simple.
10. Finding functional bugs during performance runs.
© 2003 Lionbridge Technologies, Inc. © 2003 Lionbridge Technologies, Inc.
ConclusionConclusion
Companies rely on systems to conduct business efficiently and effectively
Performance testing ensures that your users are getting reliable and timely access to the resources they need
Performance testing mitigates the risk of lost time and money due to poor performance
A fully integrated performance testing program is the preventative medicine that keeps your system from becoming an inaccessible and costly resource.
Though it may seem counterintuitive at first to slow your deployment for performance test planning and execution, the payoff in time, money, and quality will be big and will come soon.
PROPRIETARY & CONFIDENTIALPROPRIETARY & CONFIDENTIAL
Thank YouThank YouThank YouThank You