See the latest (interactive) version at /talks/squad/2015/09/09/automated-load-testing-with-tsung/

Load Testing withTsung

SQuAD 2015, Sept. 14 , 2015Ben Taitelbaum

@coshxben@coshx.com

    

Storytime

th

Idea

Congratulations?

Worries

Tue Sep 15 201517:46pm

Simple Test - ab

apache bench

ben@too$ ab -n 100 -c 10 http://localhost:3000/

Simple Test - ab results

Time taken for tests: 19.810 seconds

Complete requests: 100

Failed requests: 0

Requests per second: 5.05 [#/sec] (mean)

Min request time: 198ms

Max request time: 2154ms

Mean request time: 1893ms

Simple Test - ab

CacheOther Assets (js, img, css)Actual Use-Case?

Issues

Simple Test - tsung

1. Record2. Update Config File

3. Playback

Recording - BrowserSetup

Set up browser

ben@too$ firefox -ProfileManager

Recording - Proxy Setup

Preferences ↦ Advanced ↦ Network, Configure

HTTP Proxy to localhost on port 8090

Record

Record - stop

ben@too$ tsung-recorder stop

Create config file

<!DOCTYPE tsung SYSTEM "/usr/share/tsung/tsung-1.0.dtd" [

] >

<tsung loglevel="info"

dumptraffic="false">

<clients>...</clients>

<servers>...</servers>

<monitoring>...</monitoring>

<load>...</load>

<options>...</options>

<sessions>...</sessions>

</tsung>

~/.tsung/tsung.xml (default)

tsung.xml - clients

<clients>

<client host="localhost"/>

</clients>

tsung.xml - servers

<servers>

<server host="localhost" port="3000"

</servers>

tsung.xml - monitoring

<monitoring>

<monitor host="localhost" type="erlang"

</monitoring>

tsung.xml - load

<load>

<arrivalphase phase="1" duration="60"

<users maxnumber="100" arrivalrate=

</arrivalphase>

</load>

tsung.xml - options

<options>

<option type="ts_http" name="user_agent"

<user_agent probability="100">

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36

(KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36

</user_agent>

</option>

</options>

tsung.xml - sessions

can paste in recording file, or…

<!DOCTYPE tsung

SYSTEM "/usr/share/tsung/tsung-1.0.dtd" [

<!ENTITY fetch_homepage

SYSTEM "/home/me/.tsung/tsung_recorder20150914-0438.xml">

]>

<sessions>

&fetch_homepage;

</sessions>

Open Recording File

<session name='YYYYmmdd-HHMM' probability=

<request>

<http url='http://localhost:3000/'

</http>

</request>

<request>

<http url='/assets/application.css'

</http>

</request>

<!-- also .js and .png -->

</session>

~/.tsung/tsung_recorderYYYYmmdd-

HHMM.xml

Edit Recording File

Remove outside assets like

<request>

<http url='self-repair.mozilla.org:443'

</http>

</request>

Playback

ben@too$ tsung start

visitlocalhost:8091/es/ts_web:status

Terms - User

comes in at specific phase/ratecompletes entire session

user

Terms - Page

set of requests on same domainpage

Terms - Session

set of requestssupport for conditionals, loops, andmore

session

Terms - Transaction

grouping of requestswithin a session

transaction

Stats & Graphs

1. Install gnuplot2. Install any missing perl

dependencies ( Template)3. ben@too$ cd~/.tsung/log/YYYYmmdd-HHMM

4. ben@too$/usr/lib/tsung/bin/tsung_stats.pl

5. open report.html or graph.html

Stats

Main Statistics

Namehighest10secmean

lowest10secmean

Highest RateMeanRate

MeanCount

connect0.979msec

0.979msec

1.7 / sec0.57/ sec

0.932msec

100

page22.69sec

9.83sec

3.33333333333333/ sec

3.24/ sec

12.07sec

80

request0.86sec

0.51sec

75.3 / sec72.51/ sec

0.64sec

1860

session22.70sec

9.83sec

3.33333333333333/ sec

3.24/ sec

12.08sec

80

Stats

HTTP return code

CodeHighestRate

Mean RateTotalnumber

200 75.3 / sec 72.51 / sec 2300

Users (arrival &departure)

Simultaneous Users

Transaction Time

Throughput

Compare Graphs

local (dev.) vs. heroku (prod.)

ben@too$ tsplot "Local" 20150914-0644/tsung.log

Warning: ensure consistent

primed/unprimed state

Compare Graphs

Compare Graphs

Increase Load

<users maxnumber="1000" arrivalrate="100"

Compare Loads

heroku 10 users/second vs. heroku100 users/second

ben@too$ tsplot "10 users/sec" 20150914-1115/tsung.log

Compare Loads

Compare Loads

Why?

Errors

Name Highest Rate Total number

error_abort_max_conn_retries0.2 / sec 2

error_connect_nxdomain2.9 / sec 81

error_connection_closed10.7 / sec 979

Stuck

Try something

Did it Work?

Did it Work?

Did it Work?

Nope

Check logsbottleneck at database

▮▶

Record RealisticSession

1. tsung-recorder start2. open up firefox (select the right

profile)3. use the site

Realistic Session

Visit Homepage & Click Order Now

Realistic Session

Fill in Order & Click Create Order

Realistic Session

Order created

New Directives

thinktime

<thinktime random='true' value='5' />

New Directives

HTTP POST

<request>

<http url='/orders'

contents='order%5Bname%5D=Ben+Tester&amp;authenticity_token=%2F5Shnt%2BQr...'

content_type='application/x-www-form-urlencoded'

method='POST'>

</http>

</request>

Issue?

Dynamic Data

grab it from an input field

<input type="hidden" name="authenticity_token"

<request>

<dyn_variable name="authenticity_token"

<http url='/orders/new' method='GET'></http>

</request>

Dynamic Data

or from a meta tag

<meta name="csrf-token" content="%2F5Shnt..."

<request>

<dyn_variable name="authenticity_token"

re="name=&quot;csrf-token&quot; content=&quot;(.*)&quot;"

<http url='/orders/new' method='GET'></http>

</request>

Dynamic Data

substitute into later request

<request subst='true'>

<http url='/orders' method='POST'

contents='authenticity_token=%%_authenticity_token%%&amp;...'

</http>

</request>

Dynamic Data

Dynamic Data + Erlang

stick into the sessionthen use %%_encoded_csrf%%instead

<setdynvars sourcetype="eval"

code="fun({Pid,DynVars})->

{ok,Val}=ts_dynvars:lookup(authenticity_token,DynVars),

edoc_lib:escape_uri(binary_to_list(Val)) end.">

<var name="encoded_csrf" />

</setdynvars>

Just scratching the

surface

Analyzing Results

what’s going on here?

Scale

4-computer cluster (2003)3,000 requests / sec

75-computer cluster> 1 million requests / sec

12,000 simultaneous users (4-computercluster in 2003)

10 million simultaneous users

HTTP / HTTPS

Part of Automated QA

1. Jenkins Job to start load ( tsungplay)

2. Run manual/automated tests

Debugging

change loglevel to debug

add protocol=dumptraffic

In tsung.xml

See File structure for more details

<tsung loglevel="debug"

dumptraffic="protocol">

...

</tsung>

Error Messages - Good

2542- fatal: {endtag_does_not_match,{was,arrivalphase,should_have_been,users}}

Config Error, aborting ! {fatal,

{{endtag_does_not_match,

{was,arrivalphase,should_have_been,users}},

{file,"/home/btaitelb/.tsung/tsung.xml"},

{line,20},

{col,6}}}

Error Messages - Bad

596- fatal: {failed_validation,

{element_seq_not_conform,{wait,sessions},{is,option}}}

Config Error, aborting ! {fatal,

{{failed_validation,

{element_seq_not_conform,

{wait,sessions},

{is,option}}},

{file,"/home/btaitelb/.tsung/tsung.xml"},

{line,33},

{col,1}}}

the file is only 32 lines long

Error Messages - Ugly?

596- fatal: {failed_validation,

{element_seq_not_conform,{wait,sessions},{is,option}}}

Config Error, aborting ! {fatal,

{{failed_validation,

{element_seq_not_conform,

{wait,sessions},

{is,option}}},

{file,"/home/btaitelb/.tsung/tsung.xml"},

{line,33},

{col,1}}}

References

https://commons.wikimedia.org/wiki/File:Light

https://commons.wikimedia.org/wiki/File:Scarlet

https://www.flickr.com/photos/30099605@N04/3421554349

CC0 from pixabay.com

lightbulb image:

scarlet pimpernel image:

plaid image:

images of sticks, dandelions,clovers:

References

http://tsung.erlang-projects.org/user_manual/index.html

Tsung User Manual:

Questions?