View
4.550
Download
5
Category
Preview:
DESCRIPTION
Lessons learned from Rest Performance Testing at Constant Contact
Citation preview
Copyright © 2012 Constant Contact Inc. 1
Performance Testing REST APIsSelenium Conference 2013Jason Weden @jweden
Principal Quality Engineer
Constant Contact, WebServices
Copyright © 2012 Constant Contact Inc. 2
Who We Are
What do we do? Expose our internal web services externally to developers and
partners via an API for their own applications.
http://developer.constantcontact.com Fun Digression: Very easy to get up and running with signing into API using
Mashery and using our console to hit the REST API
Copyright © 2012 Constant Contact Inc. 3
Problem Domain
System Under Test: HTTP REST APIs Hitting the REST API with concurrent connections Defining your Performance Test approach Getting & Sharing results – performance metrics
Copyright © 2012 Constant Contact Inc. 4
Lesson 1: Be DRY
Why in the heck repeat yourself? You already have the logic in your functional tests for REST calls so reuse
that logic for your performance tests http://techblog.constantcontact.com/api/what-lego-blocks-teach-us-ab
out-rest-api-testing/ Reuse your data-seeding (e.g. for tests that have GET calls) Tools like LoadRunner and Jmeter have their place but
They’re creating blocking connections that reduce scalability You are wet – you’re rewriting logic you already wrote in your functional
tests You confine the creation, execution, and general knowledge of the
performance testing arena to one or more toolsmiths
Copyright © 2012 Constant Contact Inc. 5
Lesson 2: Choose Carefully
Choose your http client library with care Non-blocking Scale-up Seriously consider Akka (if on the JVM)
Non-blocking but with high throughput Scales to multiple processors and prevents need to deal with shared state More on this later
Other Examples on the JVM of non-blocking http clients http://spray.io (Scala) – Spray – Akka-based http://www.eclipse.org/jetty/ (Java) -- Jetty 9
https://github.com/timboudreau/netty-http-client (Java) -- Netty
Copyright © 2012 Constant Contact Inc. 6
Lesson 2: Choose Carefully (cont.) – Our Choice…
Written in Java but takes advantage of Akka for concurrency Integrated into test framework for everyone to use 3rd Party libraries
Apache HttpClient library Apache HttpCore library Akka concurrency framework Apache Commons Math3 library
High performance Easy to use programmatic API – small learning curve
Developers can go in and create performance tests Useful for any http functional OR performance testing
Even functional tests have performance metrics logged
Copyright © 2012 Constant Contact Inc. 7
Lesson 2: Choose Carefully (cont.) – Our Choice…
Found errors & significant bottlenecks Comparison with JMeter and LoadRunner
Preliminary tests show faster than JMeter (even with our 8 threads vs JMeter's 1500)
All performance tests can be checked into source control No dependency on toolsmiths for HTTP performance testing Performance testing experience shared across the team Likely faster than JMeter and LoadRunner -- still to be
scientifically determined Ability to tweak our home-grown solution vs. closed source
purchased solutions. (e.g. lots of concurrency properties to tweak)
Copyright © 2012 Constant Contact Inc. 8
Lesson 3: Devise a Performance Test Plan
What is your SLA? (e.g. avg. response time, avg. requests/sec) What kind of performance test? Options:
Profile of production performance. Do the following concurrently: X amount of POSTs to endpoint 1 Y amount of GETs to endpoint 2 (data seeding needed) Z amount of DELETEs to endpoint 3 (data seeding needed)
One type of Request: x amount of GETs Longevity – run test above for x amount of time
What metrics will be recorded (client vs server)? How will results be captured and reported? Have sign-offs on your test plan (e.g. in a wiki section)
Copyright © 2012 Constant Contact Inc. 9
Lesson 4: Capture Perf. Test Results in a Database
Capture: Performance Metrics
Client: min/max/avg response time, standard deviation, avg requests/second, percentiles, request failures
Server: cpu, memory, open db connections, thread count, consider app. profiling
Test tool used and version Test name and description Freeform comments Concurrency level In what environment was test performed (server hostnames) When it was performed Version of Test tool and application under test
Allows for posterity Allows for trending
Copyright © 2012 Constant Contact Inc. 10
Lesson 5: Run Performance Tests Continuously
Lobby for a dedicated performance environment
Run tests in non-production CI environments if only for trending relative measurements
Run daily
Copyright © 2012 Constant Contact Inc. 11
Lesson 6: Create Visualizations for Perf Test Results
Charts, Charts, Charts: Have a webpage dashboard Preferably having ability to query or filter to obtain customized results
Charts, Charts, Charts: Email results of each performance test run
Visualization Research: Splunk w/ DB Connect D3: http://shop.oreilly.com/product/0636920026938.do
Fun Digression: playing with this at: http://codepen.io
Copyright © 2012 Constant Contact Inc. 12
References – The end…
http://akka.io/
http://commons.apache.org/math/
http://hc.apache.org/httpcomponents-client-ga/
Slides available at: https://www.slideshare.net/jweden/rest-performance-testing
Recommended