Embed Size (px)
Citation preview
Low Cost Load and Performance Testing
Example Test
Why Test?
Performance is a direct reflection on your companyYour client does not care about the ‘cause’
Whose Fault Is It?
Cause can come from many sources◦Poor architecture◦Poor configuration◦ Inadequate hardware◦ Insufficient network bandwidth
Why Isn’t Performance Testing Done
Takes timeTakes production or production- like
environmentTakes a special skill setTakes specialized tools
Why Isn’t Performance Testing Done
Can take lots of $$$$$!◦Proprietary tools – can cost 100’s of
thousands of dollars to buy and maintain
So What Are We Supposed to Do?
Option 1 – Do nothingOption 2 – Keep buying hardwareOption 3 – Buy an expensive load and
performance tool. Hire a tool expert to use it.Option 4 – Allocate in-house personnel to
build a custom solution
So What Are We Supposed to Do?
Option 5 – Pray
Low Cost Options
Big tool… little price◦Utilize an industry recognized best-of-
breed tool at a greatly reduced cost
◦Pros Access to functionality that normally
costs lots of $$ Good for client server or web
applications
◦Cons Proprietary languages. Need ‘tool guru’ Need hardware, bandwidth to run it
Low Cost Options
Open Source◦40+ options - JMeter, OpenSTA,
WebLoad, Grinder◦Pros
Good price Can extend and adapt to your needs Good for simple tests
◦Cons Not as easy to use. Lacks functionality Only for web based products Need hardware, bandwidth to generate load
Low Cost Options
Cloud Based◦ Rapidly growing offering
BrowserMob, LoadStorm◦ Pros
Low price Can easily scale to thousands of users Can completely eliminate the need for hardware
◦ Cons Early in the development phase Only for web-based products Analysis is slowly coming up to speed
Cost Comparison
How Do I Choose?
How do I know what is best for me?
Understand Your True Needs
Performance, Load, Stress, Reliability◦All unique areas of testing
Performance - not to find functionality bugs , but to eliminate system bottlenecks and establish a baseline for future regression tests Takes a white-box approach. The system is inspected
and monitored "from the inside out" and from a variety of angles. Measurements are taken and analyzed and tuning is done accordingly .
Understand Your True Needs
Load - process of testing the system by feeding it the largest tasks it can operate withGoals
expose bugs such as memory management bugs, memory leaks, buffer overflows, etc
Ensure that the application meets the performance baseline established during performance testing.
Understand Your True Needs
Stress - one primary objective, and that is to determine the load under which a system fails, and how it fails. The main purpose is to make sure that when
the system fails, it recovers gracefully. i.e. – no locked sessions, no corrupted data, if load is
removed does it return to normal working condition
Understand Your True Needs
Reliability – determine application performance over a prolonged period of time. Load can vary between very high and very lowThe main purpose is to make sure that the
system performance does not degrade over time. i.e. – slow memory leaks, stale sessions, other
processes interfering with off hour activities
Understand Your True Needs
Understand your user scenarios◦ Analyze traffic patterns
Google Analytics, application logs◦User types
Administrative, buyer, seller, poster, responder, browser, etc…
◦User activity Downloading, uploading, purchasing, saving cart,
submitting,◦User ‘load’
Mix of user type and activity. How many of each
Don’t Go It Alone
May not do load and performance testing for months/years◦ Difficult for most companies to have SME
Reach out to colleagues, user groups, consultants◦ Can help ‘reduce the noise’◦Guide you through the process
Keep Customers Happy
Performance is something customers notice when it is poor
Don’t give customers a reason to look at other options
Questions/Discussion Points
Questions?
Lawrence Nuanez – Senior Consultant – ProtoTest
[email protected] – ext 109www.ProtoTest.com
