Performance Testing Crash Course

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Performance Testing Crash CourseBy Dustin Whittle


The performance of your application affects your business more than you might think. Top engineering organizations think of performance not as a nice-to-have, but as a crucial feature of their product. Those organizations understand that performance has a direct impact on user experience and, ultimately, their bottom line. Unfortunately,

most engineering teams do not regularly test the performance and scalability of their infrastructure. Join this session to find out about the latest performance testing tools and why your team should add

performance testing to your agile development process.

2


Agenda

• Why performance matters? • Case studies on business impact of performance

• Tools of the trade • MultiMechanize • Bees with Machine Guns • Google PageSpeed • AppDynamics

• Best practices • Server Side • Client Side

• Integrating into your process • Develop -> Test -> Deploy -> Repeat • What was the performance impact the last release? • What about the impact of that configuration change or package upgrade?

3

• dustinwhittle.com

• @dustinwhittle

• San Francisco, California, USA

• Technologist, Traveler, Pilot, Skier, Diver, Sailor, Golfer

Dustin Whittle

• Developer Evangelist @

• Consultant & Trainer @

• Developer Evangelist @

What I have worked on


Why does performance matter?

6


Microsoft found that Bing searches that were 2 seconds slower resulted in a 4.3% drop in

revenue per user

7


When Mozilla shaved 2.2 seconds off their landing page, Firefox downloads increased

15.4%

8


Making Barack Obama’s website 60% faster increased donation conversions by 14%

9


Amazon and Walmart increase revenue 1% for every 100ms of improvement

10


11


Performance directly impacts the bottom line

12


13


14

HealthCare.gov


An example spring app to test performance

16


git clone https://github.com/cloudfoundry-samples/hello-spring-cloud

19

https://github.com/cloudfoundry-samples/hello-spring-cloud


mvn package

20


cf push

23


http://springone-demo.cfapps.io/

24



25


Tools of the trade for performance testing

26


Understand your baseline performance

27


Static

vs

Hello World

vs

Applications

28


Apache Bench

29


ab -c 1 -t 10 -k http://springone-demo.cfapps.io/

30



Benchmarking springone-demo.cfapps.io (be patient)Finished 187 requests

Server Software: Apache-Coyote/1.1Server Hostname: springone-demo.cfapps.ioServer Port: 80

Document Path: /Document Length: 5217 bytes

Concurrency Level: 1Time taken for tests: 10.039 secondsComplete requests: 187Failed requests: 0Keep-Alive requests: 187Total transferred: 1021768 bytesHTML transferred: 975579 bytesRequests per second: 18.63 [#/sec] (mean)Time per request: 53.687 [ms] (mean)Time per request: 53.687 [ms] (mean, across all concurrent requests)

31


ab -c 10 -t 10 -k http://springone-demo.cfapps.io/

32



Benchmarking springone-demo.cfapps.io (be patient)Finished 659 requests

Server Software: Apache-Coyote/1.1Server Hostname: springone-demo.cfapps.ioServer Port: 80

Document Path: /Document Length: 5217 bytes

Concurrency Level: 10Time taken for tests: 10.015 secondsComplete requests: 659Failed requests: 0Keep-Alive requests: 659Total transferred: 3600776 bytesHTML transferred: 3438003 bytesRequests per second: 65.80 [#/sec] (mean)Time per request: 151.970 [ms] (mean)Time per request: 15.197 [ms] (mean, across all concurrent requests)

33


Siege

34

siege -c 10 -b -t 10S http://springone-demo.cfapps.io/


** SIEGE 3.0.6** Preparing 10 concurrent users for battle.The server is now under siege...Lifting the server siege... done.

Transactions: 623 hitsAvailability: 100.00 %Elapsed time: 9.57 secsData transferred: 3.10 MBResponse time: 0.15 secsTransaction rate: 65.10 trans/secThroughput: 0.32 MB/secConcurrency: 9.91Successful transactions: 623Failed transactions: 0Longest transaction: 0.30Shortest transaction: 0.10


Crawl the entire app to discover all urls

37

sproxy -o ./urls.txt

SPROXY v1.02 listening on port 9001...appending HTTP requests to: ./urls.txt...default connection timeout: 120 seconds

wget -r -l 0 -t 1 --spider -w 1 -e "http_proxy = http://127.0.0.1:9001" "http://acmedemoapp.com/"

sort -u -o urls.txt urls.txt

http://acmedemoapp.com/


http://acmedemoapp.com/ http://acmedemoapp.com/about http://acmedemoapp.com/cart http://acmedemoapp.com/currency/change/EUR http://acmedemoapp.com/currency/change/GBP http://acmedemoapp.com/currency/change/USD http://acmedemoapp.com/login http://acmedemoapp.com/register/ http://acmedemoapp.com/t/brand/bookmania http://acmedemoapp.com/t/category/books http://acmedemoapp.com/t/category/mugs http://acmedemoapp.com/t/category/stickers http://acmedemoapp.com/terms-of-service

41


Benchmark traffic across all unique urls with siege

42

siege -v -c 50 -i -t 3M -f urls.txt -d 10


Apache JMeter

44


46


Multi-Mechanize is an open source framework for performance and load

testing

47

pip install multi-mechanize

multimech-newproject demo

import requests

class Transaction(object): def run(self): r = requests.get(‘http://acmedemoapp.com/) r.raw.read()

import mechanizeimport time

class Transaction(object): def run(self): br = mechanize.Browser() br.set_handle_robots(False)

start_timer = time.time() resp = br.open(‘http://acmedemoapp.com/) resp.read() latency = time.time() - start_timer self.custom_timers['homepage'] = latency

start_timer = time.time() resp = br.open(‘http://acmedemoapp.com/cart') resp.read() latency = time.time() - start_timer self.custom_timers['cart'] = latency

assert (resp.code == 200)

http://acmedemoapp.com/cart'

[global]run_time = 10rampup = 5results_ts_interval = 1progress_bar = onconsole_logging = offxml_report = on

[user_group-1]threads = 1script = demo.py

multimech-run demo


What about when you need more than one machine?

55


Who lives in the cloud?

56

Bees with Machine Guns


A utility for arming (creating) many bees (micro EC2 instances)

to attack (load test) targets (web applications)

58

pip install beeswithmachineguns

# ~/.boto

[Credentials]

aws_access_key_id=xxx aws_secret_access_key=xxx

[Boto]

ec2_region_name = us-west-2 ec2_region_endpoint = ec2.us-west-2.amazonaws.com

bees up -s 2 -g default -z us-west-2b -i ami-bc05898c -k appdynamics-

dustinwhittle-aws-us-west-2 -l ec2-user

Connecting to the hive.Attempting to call up 2 bees.Waiting for bees to load their machine guns.......Bee i-3828400c is ready for the attack.Bee i-3928400d is ready for the attack.The swarm has assembled 2 bees.

bees report

Read 2 bees from the roster.Bee i-3828400c: running @ 54.212.22.176Bee i-3928400d: running @ 50.112.6.191

bees attack -n 1000 -c 50 -u http://springone-demo.cfapps.io/


Read 2 bees from the roster.Connecting to the hive.Assembling bees.Each of 2 bees will fire 500 rounds, 25 at a time.Stinging URL so it will be cached for the attack.Organizing the swarm.…Offensive complete. Complete requests: 1000 Requests per second: 306.540000 [#/sec] (mean) Time per request: 163.112000 [ms] (mean) 50% response time: 151.000000 [ms] (mean) 90% response time: 192.000000 [ms] (mean)Mission Assessment: Target crushed bee offensive.The swarm is awaiting new orders.

bees attack -n 100000 -c 1000 -u http://springone-demo.cfapps.io/


Read 2 bees from the roster.Connecting to the hive.Assembling bees.Each of 2 bees will fire 50000 rounds, 500 at a time.Stinging URL so it will be cached for the attack.Organizing the swarm.…Offensive complete. Complete requests: 100000 Requests per second: 502.420000 [#/sec] (mean) Time per request: 360.114000 [ms] (mean) 50% response time: 451.000000 [ms] (mean) 90% response time: 402.000000 [ms] (mean)Mission Assessment: Target crushed bee offensive.The swarm is awaiting new orders.

Read 2 bees from the roster.Connecting to the hive.Assembling bees.Each of 2 bees will fire 50000 rounds, 500 at a time.Stinging URL so it will be cached for the attack.Organizing the swarm.Bee 0 is joining the swarm.Bee 1 is joining the swarm.Bee 0 is firing his machine gun. Bang bang!Bee 0 lost sight of the target (connection timed out).Bee 1 lost sight of the target (connection timed out).Offensive complete. Target timed out without fully responding to 2 bees. No bees completed the mission. Apparently your bees are peace-loving hippies.The swarm is awaiting new orders.

bees down

Read 2 bees from the roster.Connecting to the hive.Calling off the swarm.Stood down 2 bees.


What about the client side?

72


In modern web applications more latency comes from the client-side than the server-

side.

73


Google PageSpeed

74


Google PageSpeed Insights

75

curl "https://www.googleapis.com/pagespeedonline/v1/runPagespeed?

url=http://dustinwhittle.com/&key=xxx"


WBench

79

gem install wbench

wbench http://dustinwhittle.com/


Automate client-side performance testing with Grunt

83


Use Bower (for dependencies), Grunt (for automation),

and Yeoman (for bootstrapping)

84


How many people understand exactly how fast their site runs in production?

87


Track performance in development and production

88


Instrument everything = code, databases, caches, queues, third party services, and infrastructure.

89


Chef / Sensu

90

http://sensuapp.org/

Statsd + Graphite + Grafana


Track performance of end users

94

Episodes / Boomerang


webpagetest.org

99


SiteSpeed.io

103


106


Load testing services from the cloud

112


Test for failures

• NetFlix Simian Army + Chaos Monkey

• What happens if you lose a caching layer? • What happens if dependencies slow down?

116


Best Practices

• Capacity plan and load test the server-side • Optimize and performance test the client-side • Understand your starting point • Instrument everything • Measure the difference of every change • Automate performance testing in your build and

deployment process • Understand how failures impact performance

117


Integrate automated performance testing into continuous integration for server-side

and client-side

118


Understand the performance implications of every deployment and package upgrade

119


Monitor end user experience from end to end in production

120


Questions?

121


Find these slides on SpeakerDeck

https://speakerdeck.com/dustinwhittle

122

https://speakerdeck.com/dustinwhittle

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 123

Learn More. Stay Connected

Test and monitor your applications! Performance Testing Crash Course

@springcentral | spring.io/video