Upload
compuware-apm
View
3.236
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Dollar Thrifty Automotive Group, Inc. (DTG) customers are increasingly choosing the Internet as the primary way to rent vehicles from the Dollar Rent A Car and Thrifty Car Rental Web sites. Recently DTG undertook a significant redesign initiative for its two Web sites to optimize customer experience ahead of its busiest summer travel season and used Gomez’s web load and performance testing solution to validate their efforts. Attendees of this hands-on Webinar will see a Gomez Reality Load product demonstration and learn about the steps DTG took to validate peak performance for all internal and external components including Content Delivery Networks (CDNs), ads, analytics and ecommerce platforms, delivered across the Internet to its customers’ browsers. This Webinar will cover: •How Dollar Thrifty geared up for their peak summer season. •How a new style of load testing enables organizations to “walk in their customer’s shoes” and find problems before end-users find them. •Best practices for identifying and resolving Web performance issues across the entire Web application delivery chain, inside and outside the firewall. •Testing approaches that don’t require costly hardware or software investments. •How to uncover geographical response time discrepancies that may surface under load.
Citation preview
Best Practices for Identifying Web Performance Issues Before Your Customers Do
A Gomez/Dollar Thrifty Web Performance Testing Case Study:
Imad Mouline, CTO, GomezEmanuel Daniele, Sr. Solution Engineer, GomezJim Arrowood, DTAG, Director, Web Development and Architecture
Agenda
Reality Load Overview – Imad Mouline
Dollar Thrifty Case Study – Jim Arrowood
Product Demo – Emanuel Daniele
Internet
Content DeliveryNetworks
MajorISP
Local ISP
Mobile Carrier
3rd Party/Cloud Services
Browsers and devices Users
The Challenge of Delivering Quality Web Experiences
Storage
Load Balancing
Web Servers
App Servers
DB Servers
Mobile Components
Storage
Load Balancing
Web Servers
App Servers
DB Servers
Mobile Components
Users
The Web Application Delivery Chain
Traditional zone of control
Traditional testing: “OK” …user is
NOT happy
Internet
Content DeliveryNetworks
MajorISP
Local ISP
Mobile Carrier
Storage
Load Balancing
Web Servers
App Servers
DB Servers
Mobile Components
Users3rd Party/
Cloud ServicesBrowsers
and devices
The Web Application Delivery Chain
• Network peering problems
• Outages
• Inconsistent geo performance• Bad performance under load• Blocking content delivery• Incorrect geo-targeted content
• Configuration issues• Oversubscribed POP• Poor routing optimization• Low cache hit rate
• Network peering problems
• Bandwidth throttling• Inconsistent
connectivity• Configuration
errors• Application
design issues• Code defects• Insufficient
infrastructure
• Poorly performing JavaScript
• Browser/device incompatibility
• Page size too big
• Too many objects
• Low cache hit rate
• Network resource shortage
• Faulty content transcoding
• SMS routing / latency issues
The Challenge of Ensuring Quality Web Experiences
Zone of customer expectationZone of customer expectationTraditional zone of control
Traditional testing: “OK” …user is
NOT happy
Internet
Content DeliveryNetworks
MajorISP
Local ISP
Mobile Carrier
Storage
Load Balancing
Web Servers
App Servers
DB Servers
Mobile Components
Users3rd Party/
Cloud ServicesBrowsers
and Devices
BackboneHigh volume load (HTTP, Browser)
Find infrastructure breaking points Define capacity headroom
100,000+ consumer- grade desktops
168+ countries
2,500+ ISPs
Major mobile carriers around the globe
Gomez Load Testing: On-Demand Realistic Load Testing from Browser to Data Center
Last Mile Real-world load
Find user experience breaking points Accurately measure response time
100+ commercial-grade nodes & data centers
Gomez Network: The World’s Most Comprehensive Performance and Testing Network
Web Performance Management and Load Testing
100+ locations
Backbone
Cross-Browser Testing 500+ browser/OS combo’s5,000+ supported devices
Virtual Test Bed
Web Performance Management and Load Testing
100,000+ locations
Gomez Last Mile
Reality Load XF – Overview & Demo
SaaS with no investment or maintenance costs and rapid paybackSelf-service, full turnkey solution, or tailored to meet your needs with Gomez Professional Services offerings
Tests from an “Outside-in” customer point of view, with drill down to all web application components
Full desktop browser testing across globally distributed geographies
Expanded Load Generation
• Additional Geographies for Based Load high volume testing
• More Last Mile peer populations
Simplified Test Provisioning
• More easily schedule complex test scenarios
• Save and re-use test scenarios
• Last Mile UI scheduling improvements
Enhanced Reporting
• Detailed reporting for complex scenarios
• Filter results by load origin or time
• Streamlined data and chart exports
New in July 2009
Dollar Thrifty Case Study
Jim Arrowood, DTAGDirector -Web Development and Architecture
8
Who is Dollar Thrifty?
• Driven by the mission “Value Every Time,” the Company’s brands, Dollar Rent A Car and Thrifty Car Rental, serve value-conscious travelers in over 70 countries.
• Dollar and Thrifty have over 600 corporate and franchised locations in the United States and Canada, operating in virtually all of the top U.S. and Canadian airport markets.
• The Company’s approximately 6,400 employees are located mainly in North America, but global service capabilities exist through an expanding international franchise network.
9
Dollar and Thrifty.com
10
The Dollar Thrifty Environment
Dollar.comMiddle-Tier
Thrifty.comMiddle-Tier
Rate EngineMiddle Tier
Reservation Middle Tier
Database
11
The Problem
• In a highly, horizontally scaled environment, development, test, and staging environments rarely 100% match production
• Dollar Thrifty significantly redesigned both of our websites in 2008 leaving question as to how the sites will perform in the peak of the 2009 summer
• Internal load tests using traditional methods in test and staging environments did not provide the needed confidence
12
The Gomez Relationship
• Dollar Thrifty has utilized the Gomez ActiveXF platform for several years– Synthetic Tests of Key Business Processes
• Monitors for response time (Home Page Load & Reservation Process)
• Monitors for successful execution (Home Page & Reservation Process)
13
Gomez Reality Load
• In Early 2009, Dollar Thrifty teamed with Gomez to launch the first external load test of the company’s infrastructure and key software platforms utilizing Gomez Reality Load
14
The Goals
• Regardless of how much load, where is our weakest point?
• Can we handle our previous year’s peak +25% traffic?
15
Identify Core Business Processes
• How do the vast majority of the consumers interact with the site?
• Use 80-20 rule– Shop for Rates– Make Reservation– Modify Reservation– Cancel Reservation
80%
20%
Core Business ProcessesOther Processes
16
Distribute Core Business Processes
• By percentage, how do consumers interact with the site?– Shop for Rates – 70%*– Make Reservation – 15%*– Modify Reservation – 10%*– Cancel Reservation – 5%*
70%
15%
10%5%
Shop for RatesMake ReservationModify ReservationCancel Reservation
*Hypothetical 17
Forecast Visitors
• Based on historical web analytic data and the goal of the test, compute the intended visitor capacity of the site within an hour– 10,000 visitors per hour at peak historically*– If we intend to support peak +25% then our visitor
forecast is 12,500 visitors
*Hypothetical
0
2000
4000
6000
8000
10000
12000
Jan Feb Mar-Apr May-Jun Jul-Aug Sep-Oct Nov-Dec
Vis
ito
rs P
er
Ho
ur
18
Identify the Needed Scripts
• Throughput Tests – Where are we weakest?– One script per core business process with no
“think-time”• Capacity / Load Tests – How much load can we
legitimately handle?– One script per core business process with “think-
time”
19
Develop Needed Scripts
• Gomez Script Recorder allows the client to create of the script, inject look-up data, and replay the script
• RealityLoad utilizes the same scripting engine as ActiveXF
• The base of our scripts existed as we already had them built to execute in the ActiveXF platform– Added logic to pull from look-up data – Added think time
• Once, complete the tests are loaded into the Gomez portal
20
Configure Tests
• Using the Gomez portal, tests are configured and scheduled for execution
21
Execute
• Execute tests at off peak time – We chose 2 AM to 4 AM CST
• Establish war room with key system owners• Establish bridge line with key infrastructure and
monitoring personnel• As tests are executing ensure all internal monitoring
tools in all tiers of the application are capturing as much detail as possible without skewing results– CPU, Memory, Disk IO, Network Throughput, etc.
22
Analyze Results
• Each system owner and infrastructure owner collects key metrics and findings and submits to the test leader
23
Discuss Key Findings
• Thrifty.com front-end servers ran hot while the middle-tier was cold and overpowered
• Dollar.com front-end servers ran cool while the middle-tier was running on target
• Discovered excessive calls for validation to legacy reservation system
• Thrifty.com middle-tier had sticky sessions enabled causing load balancing to become less ineffective
24
Make Corrective Actions & Retest
• Mitigate the discovered issues and risks– We were able make a simple configuration change to
resolve the “sticky sessions” issue– We were able to move Thrifty middle-tier boxes into
the Thrifty.com front-end and Dollar.com middle-tier to better distribute our load with no additional expense
• When time and budget allows, retest to ensure mitigating actions resolve the issue and ensure no additional bottlenecks have appeared
25
Demo
Emanuel Daniele Gomez Reality Load Demo
26
Wrap Up
Questions & Answers Check back on QA Forums
To find out more about Reality Load:
[email protected]@gomez.com
Product Informationhttp://www.gomez.com/products-solutions/products/load-testing/2 Minute Explainerhttp://www.gomez.com/resources/video-library/gomez-reality-load-testing-two-minute-explainer/
Company: Online presence for a popular TV show
• Following episodes of the TV show the web site sees high traffic spikes
• Goal was to achieve 1500 logins per minute
• Load tested DB to improve performance in anticipation of another traffic spike
Ensuring Performance Of Login Process
1
Internet
Content DeliveryNetworks
MajorISP
Local ISP
Mobile Carrier
3rd Party/Cloud Services
Browsers and devices
Storage
Load Balancing
Web Servers
App Servers
DB Servers
Mobile Components
Storage
Load Balancing
Web Servers
App Servers
DB Servers
Mobile Components
Users
Application Bottleneck Causes Immediate Response Time Issue
• As users were added, the response time of step 3 (the login) climbed immediately
• The test bottlenecked at 160 logins per minute (Goal 1500)
• But quickly dropped off as users received server errors
• New login query was not optimized and was bottlenecking the database servers’ CPUs
Application Bottleneck – Re-test
30
• Following the tuning effort the application performance was improved.
• This time the bottleneck was at 1300 logins per minute.
• A bandwidth limit was reached at just under 90 Mbps, resulting in an overall slowdown as users were added.
• This highlights:• The importance of re-testing
following each change.• The fact that applications often
have many bottlenecks, that can only be uncovered one at a time.
Company: Online Retailer
• Several 3rd Parties now involved in serving up key content
• Goal was to validate performance of entire application
Ensuring Performance of All 3rd Party Components
Internet
Content DeliveryNetworks
MajorISP
Local ISP
Mobile Carrier
3rd Party/Cloud Services
Browsers and devices
Storage
Load Balancing
Web Servers
App Servers
DB Servers
Mobile Components
Storage
Load Balancing
Web Servers
App Servers
DB Servers
Mobile Components
Users2
The load increases throughout the test
The transaction rate increases and then falls off as response times climb
Errors are seen, all on a 3rd party object
Response Times Rise Due To 3rd Party Object Error
• 3rd party hardware was insufficient for overall demands on application
• Based on SLAs 3rd party had to improve performance to get paid
Company: Regional Online News Source
• Began testing for the 2008 election season
• Goal was to validate overall performance focusing in 2 key regions
Ensuring Performance in Key Markets
Internet
Content DeliveryNetworks
MajorISP
Local ISP
Mobile Carrier
3rd Party/Cloud Services
Browsers and devices
Storage
Load Balancing
Web Servers
App Servers
DB Servers
Mobile Components
Storage
Load Balancing
Web Servers
App Servers
DB Servers
Mobile Components
Users
3
By traditional test standards the test passed
Page response times stayed under 4 seconds, outside of one brief blip
There was only 1 page error and 11 errors total out of 60000+ transactions
Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99%
No Performance Issues Detected From Data-Center
Last Mile data showing substantial number of measurements greater than 4 seconds
Performance Issues Detected From Real User Desktops
Key geographies for this customer are New York and Pennsylvania
Last Mile Case Study: Primary Geographies
36
The response time never met the 4 second average
goal.
Availability was Less than 99%.
Key geographies for this customer are New York and Pennsylvania.
By these standards the test failed.