See the latest (interactive) version at /talks/squad/2015/09/09/automated-load-testing-with-tsung/
Load Testing withTsung
SQuAD 2015, Sept. 14 , 2015Ben Taitelbaum
Storytime
th
Idea
Congratulations?
Worries
Tue Sep 15 201517:46pm
Simple Test - ab
apache bench
ben@too$ ab -n 100 -c 10 http://localhost:3000/
Simple Test - ab results
Time taken for tests: 19.810 seconds
Complete requests: 100
Failed requests: 0
Requests per second: 5.05 [#/sec] (mean)
Min request time: 198ms
Max request time: 2154ms
Mean request time: 1893ms
Simple Test - ab
CacheOther Assets (js, img, css)Actual Use-Case?
Issues
Simple Test - tsung
1. Record2. Update Config File
3. Playback
Recording - BrowserSetup
Set up browser
ben@too$ firefox -ProfileManager
Recording - Proxy Setup
Preferences ↦ Advanced ↦ Network, Configure
HTTP Proxy to localhost on port 8090
Record
Record - stop
ben@too$ tsung-recorder stop
Create config file
<!DOCTYPE tsung SYSTEM "/usr/share/tsung/tsung-1.0.dtd" [
] >
<tsung loglevel="info"
dumptraffic="false">
<clients>...</clients>
<servers>...</servers>
<monitoring>...</monitoring>
<load>...</load>
<options>...</options>
<sessions>...</sessions>
</tsung>
~/.tsung/tsung.xml (default)
tsung.xml - clients
<clients>
<client host="localhost"/>
</clients>
tsung.xml - servers
<servers>
<server host="localhost" port="3000"
</servers>
tsung.xml - monitoring
<monitoring>
<monitor host="localhost" type="erlang"
</monitoring>
tsung.xml - load
<load>
<arrivalphase phase="1" duration="60"
<users maxnumber="100" arrivalrate=
</arrivalphase>
</load>
tsung.xml - options
<options>
<option type="ts_http" name="user_agent"
<user_agent probability="100">
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36
</user_agent>
</option>
</options>
tsung.xml - sessions
can paste in recording file, or…
<!DOCTYPE tsung
SYSTEM "/usr/share/tsung/tsung-1.0.dtd" [
<!ENTITY fetch_homepage
SYSTEM "/home/me/.tsung/tsung_recorder20150914-0438.xml">
]>
<sessions>
&fetch_homepage;
</sessions>
Open Recording File
<session name='YYYYmmdd-HHMM' probability=
<request>
<http url='http://localhost:3000/'
</http>
</request>
<request>
<http url='/assets/application.css'
</http>
</request>
<!-- also .js and .png -->
</session>
~/.tsung/tsung_recorderYYYYmmdd-
HHMM.xml
Edit Recording File
Remove outside assets like
<request>
<http url='self-repair.mozilla.org:443'
</http>
</request>
Playback
ben@too$ tsung start
visitlocalhost:8091/es/ts_web:status
Terms - User
comes in at specific phase/ratecompletes entire session
user
Terms - Page
set of requests on same domainpage
Terms - Session
set of requestssupport for conditionals, loops, andmore
session
Terms - Transaction
grouping of requestswithin a session
transaction
Stats & Graphs
1. Install gnuplot2. Install any missing perl
dependencies ( Template)3. ben@too$ cd~/.tsung/log/YYYYmmdd-HHMM
4. ben@too$/usr/lib/tsung/bin/tsung_stats.pl
5. open report.html or graph.html
Stats
Main Statistics
Namehighest10secmean
lowest10secmean
Highest RateMeanRate
MeanCount
connect0.979msec
0.979msec
1.7 / sec0.57/ sec
0.932msec
100
page22.69sec
9.83sec
3.33333333333333/ sec
3.24/ sec
12.07sec
80
request0.86sec
0.51sec
75.3 / sec72.51/ sec
0.64sec
1860
session22.70sec
9.83sec
3.33333333333333/ sec
3.24/ sec
12.08sec
80
Stats
HTTP return code
CodeHighestRate
Mean RateTotalnumber
200 75.3 / sec 72.51 / sec 2300
Users (arrival &departure)
Simultaneous Users
Transaction Time
Throughput
Compare Graphs
local (dev.) vs. heroku (prod.)
ben@too$ tsplot "Local" 20150914-0644/tsung.log
Warning: ensure consistent
primed/unprimed state
Compare Graphs
Compare Graphs
Increase Load
<users maxnumber="1000" arrivalrate="100"
Compare Loads
heroku 10 users/second vs. heroku100 users/second
ben@too$ tsplot "10 users/sec" 20150914-1115/tsung.log
Compare Loads
Compare Loads
Why?
Errors
Name Highest Rate Total number
error_abort_max_conn_retries0.2 / sec 2
error_connect_nxdomain2.9 / sec 81
error_connection_closed10.7 / sec 979
Stuck
Try something
Did it Work?
Did it Work?
Did it Work?
Nope
Check logsbottleneck at database
▮▶
Record RealisticSession
1. tsung-recorder start2. open up firefox (select the right
profile)3. use the site
Realistic Session
Visit Homepage & Click Order Now
Realistic Session
Fill in Order & Click Create Order
Realistic Session
Order created
New Directives
thinktime
<thinktime random='true' value='5' />
New Directives
HTTP POST
<request>
<http url='/orders'
contents='order%5Bname%5D=Ben+Tester&authenticity_token=%2F5Shnt%2BQr...'
content_type='application/x-www-form-urlencoded'
method='POST'>
</http>
</request>
Issue?
Dynamic Data
grab it from an input field
<input type="hidden" name="authenticity_token"
<request>
<dyn_variable name="authenticity_token"
<http url='/orders/new' method='GET'></http>
</request>
Dynamic Data
or from a meta tag
<meta name="csrf-token" content="%2F5Shnt..."
<request>
<dyn_variable name="authenticity_token"
re="name="csrf-token" content="(.*)""
<http url='/orders/new' method='GET'></http>
</request>
Dynamic Data
substitute into later request
<request subst='true'>
<http url='/orders' method='POST'
contents='authenticity_token=%%_authenticity_token%%&...'
</http>
</request>
Dynamic Data
Dynamic Data + Erlang
stick into the sessionthen use %%_encoded_csrf%%instead
<setdynvars sourcetype="eval"
code="fun({Pid,DynVars})->
{ok,Val}=ts_dynvars:lookup(authenticity_token,DynVars),
edoc_lib:escape_uri(binary_to_list(Val)) end.">
<var name="encoded_csrf" />
</setdynvars>
Just scratching the
surface
Analyzing Results
what’s going on here?
Scale
4-computer cluster (2003)3,000 requests / sec
75-computer cluster> 1 million requests / sec
12,000 simultaneous users (4-computercluster in 2003)
10 million simultaneous users
HTTP / HTTPS
Part of Automated QA
1. Jenkins Job to start load ( tsungplay)
2. Run manual/automated tests
Debugging
change loglevel to debug
add protocol=dumptraffic
In tsung.xml
See File structure for more details
<tsung loglevel="debug"
dumptraffic="protocol">
...
</tsung>
Error Messages - Good
2542- fatal: {endtag_does_not_match,{was,arrivalphase,should_have_been,users}}
Config Error, aborting ! {fatal,
{{endtag_does_not_match,
{was,arrivalphase,should_have_been,users}},
{file,"/home/btaitelb/.tsung/tsung.xml"},
{line,20},
{col,6}}}
Error Messages - Bad
596- fatal: {failed_validation,
{element_seq_not_conform,{wait,sessions},{is,option}}}
Config Error, aborting ! {fatal,
{{failed_validation,
{element_seq_not_conform,
{wait,sessions},
{is,option}}},
{file,"/home/btaitelb/.tsung/tsung.xml"},
{line,33},
{col,1}}}
the file is only 32 lines long
Error Messages - Ugly?
596- fatal: {failed_validation,
{element_seq_not_conform,{wait,sessions},{is,option}}}
Config Error, aborting ! {fatal,
{{failed_validation,
{element_seq_not_conform,
{wait,sessions},
{is,option}}},
{file,"/home/btaitelb/.tsung/tsung.xml"},
{line,33},
{col,1}}}
References
https://commons.wikimedia.org/wiki/File:Light
https://commons.wikimedia.org/wiki/File:Scarlet
https://www.flickr.com/photos/30099605@N04/3421554349
CC0 from pixabay.com
lightbulb image:
scarlet pimpernel image:
plaid image:
images of sticks, dandelions,clovers:
References
http://tsung.erlang-projects.org/user_manual/index.html
Tsung User Manual:
Questions?