View
224
Download
0
Category
Preview:
Citation preview
8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
1/15
A Creative Intellect Consulting Thought Leadership PaperCreative Intellect Consulting UK is constantly monitoring adoption of softwaretesting tools and best practices. In this paper we examine the benets and
challenges of early lifecycle testing for a cluster of technical test types,commonly known as performance testing. The aim is to bring managers up todate on the latest thinking on the subject and practice in the eld, with a viewto cutting away the myths and removing obstacles blocking adoption.
The Benets and Practice of EarlyLifecycle Performance Testing
An Opportunity for Optimizing
Software Delivery Processes
Paul Herzlich
Research Analyst,
Creative Intellect Consulting
EuroSTARSoftware TestingC o m m u n i t y
8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
2/15
The Benets and Practice of Early Lifecycle Performance Testing
1
PA
GE
SummaryMessages
Early Testing Reserved forFunctional Testing
Early testing delivers business benets by
enabling IT organizations to deliver more
at lower cost. However, organizations have
concentrated on introducing early functional
testing and have largely ignored the potential
of early performance testing.
The dynamics of early testingapply equally or more toperformance testing
Early testing delivers benets in broadly
two ways:
It saves money on rework
It mitigates the risks of surprises late in a
project.
These dynamics are extremely relevant to
performance issues, which perhaps even
more than functional errors hold the potential
of sending a project back to the drawing
board if components or infrastructure dont
perform as required.
Practitioners believe there are twounsolvable challenges to earlyperformance testing
There are a whole host of potential difculties
to introducing early performance testing.
Most of them are the same as those that
impede the adoption of early functional
testing. They are largely soft, people issues,
centered around the roles and psychology
of software coders. There are fairly well
understood change management techniques
for dealing with them.
However, to most testers and QA managers,
there are two aspects of performance testing
that render it unsuitable for early testing. The
beliefs are:
You can only test performance on a fully
built system which doesnt exist early on.
You dont have suitable component level
performance requirements.
The solutions to these objections require
a ner look at what performance testing
consists of and a pragmatic approach to
achieving your goals without necessarilyachieving textbook perfection.
Introducing early performancetesting is a change managementproject
Starting down the road towards early
performance testing will involve developing
motivation among those who will be
responsible for doing it. You will need to
make controlled changes to all your existing
development processes, enlist the help of
specialists including your existing late cycle
performance testers and adopt tools that
give you the kind of automated support that
works best alongside existing development
tools. You should only undertake this within
the context of a properly conceived change
management project.
The BusinessImperative
Most software delivery organisations accept
the need for performance testing, but it is still
commonly done very late in the applicationlifecycle. Some even wait until after software
has been released in production.
8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
3/15
The Benets and Practice of Early Lifecycle Performance Testing
2
PA
GE
We know from studies over the last 30 years
that testing early in the lifecycle yields big
benets. Although some resistance in the
eld still remains, the argument has been
largely won for early functional testing.
However, despite this general acceptance,
the practice lags far behind for performance
testing. Although QA managers at the leading
edge of process maturity are beginning to
adopt it, there is a widespread perception
that early performance testing, even if
desirable, is impractical or unfeasible.
The challenges to adopting early functionaltesting were soft issues mainly people
and process. Once the benets of early
functional testing were understood, it
became a matter of building the practice
into development processes and winning the
hearts and minds of developers to adopt it.
Developers traditionally resisted the notion
that formalised testing was part of their job.
Textbook unit testing was largely a myth.However, Agile methods, powerful IDEs and
unit test frameworks have been signicant
in bridging the gap between testing and
development activities. Early functional
testing increasingly has traction throughout
the industry.
Early performance testing is a practice
that presents not only the same set of soft
behavioural challenges as early functional
testing, but also a set of hard, technical
constraints. We know how to solve the soft
challenges, but the technical constraints
seem intractable to all but a few very
experienced specialists in test processes.
An example of a widely held view, stated
in the extreme, is that you can only test
performance meaningfully when you
have a fully assembled system in a nearlyproduction-like environment, things you
dont have early in the lifecycle. If that truly
were the case, early performance testing
would indeed be impossible. In fact, these
are not technical constraints at all but simply
intellectual obstacles that can be addressed.
Creative, innovative and forward-thinking
test practitioners have found solutions.
If early performance testing is so unintuitive
and counter to the received wisdom, then why
is it worth pursuing? The answer is that we
have an obligation to make software delivery
more responsive to the perennial business
needs to deliver more and cost less. Test,
QA and development managers aligned with
these business goals understand that early
performance testing is yet one more tool toachieving this.
How Early TestingDelivers Benets
There are two ways in which early testing
delivers benets:
cost and timescale reduction and
increased project predictability.
Both of these are extremely important in a
commercial environment where business
success depends on IT to minimize costs
and meet changing market demands quickly.
Businesses cannot afford cost overruns anddelayed product releases. The days when
an estimated 60% of all projects get canned
usually for not delivering the right systems
at the right time are over. Project budgets
must be controlled, timescales managed
and uncertainty kept to a minimum.
Rework reducing the costs of going
backwards
Many studies have been performed over
the years that conclude that testing early
has nancial benets. The National Institute
8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
4/15
The Benets and Practice of Early Lifecycle Performance Testing
3
PA
GE
of Standards & Technology (NIST) study
from 2002 on The Economic Impacts of
Inadequate Infrastructure for Software
Testing is comprehensive and accessible
for understanding the whole picture. It
demonstrates that bugs are signicantly
more expensive to correct after code leaves
the developer for system and performance
testing. It states that, it is costlier to repair
a bug that is created in the unit stage in the
component or system development stage
than it is to remedy the same bug in the unit
stage when it was introduced.
The report estimates that a bug introduced in
the coding stage is 10 times more costly to
locate and correct in system testing phase,
20 times more costly in pre-production
or beta stages and 30 times more costly
post-release than it would have been at
the unit stage when the bug was created.
Programmers create the code and their code
contains bugs. Getting them to nd and x
them is not an unreasonable demand.
For bugs in the initial requirements the
picture is equally as dramatic. This is worthy
of attention since most bugs emanate from
badly captured or ill-dened requirements. If
the cost of nding a requirements bug during
the requirements capture stage is x, it costs
5x in coding and unit test stages, 10x in later
test stages, 15x in pre-production and 30x
in the released system.
If youre skeptical, think about the processes
required to locate and x a bug. When a bug
is found in a stage later than coding and unittest, it has to be documented and returned to
a developer. A developer has to establish an
environment to reproduce the bug. This may
entail re-creating test data and restoring an
earlier version of source code. Locating the
bug the error that is the source of a fault is
complicated by the fact youre now dealing
with a complex assembly of components
rather than simple units.
Attending to a bug report comes with an
immediate disruption to the developers other
work, which in turn has a cost to activities
wholly unrelated to this project. When
the bug is xed, code must be promoted
and retested at later stages, with all theheadaches that entails for source code and
build management.
1
10
20
30
Coding System Test Pre-producon Post-live
Chart 1: Relative cost of xing a coding bug at later lifecycle stages
8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
5/15
The Benets and Practice of Early Lifecycle Performance Testing
4
PA
GE
It is clear that if there is any possibility of
producing and running a test while a unit
is in the original coding environment, any
bug that is found is signicantly cheaper to
locate and correct.
Predictability mitigating risks and
avoiding surprises
Although no one asserts that early lifecycle
testing would ever nd all bugs, when the
practice is deployed effectively, companies
benet from shorter project timescales and
lower project costs. Early lifecycle testinghas another benecial effect. It enhances the
predictability of software delivery, a benet
that managers cant quantify easily, but miss
sorely when they get unwelcome surprises
in the late stages of a project.
Early testing contributes to greater
predictability by providing better information
about the true state of a project in the earlystages. As work is passed from coding to
integration or system testing (or iteration to
iteration), there is a great deal more certainty
about the codes readiness when it has been
unit tested by developers (real unit testing,
not the mythical or ad hoc testing developers
have historically performed.) A manager has
a great deal more condence that coding is
truly complete for 10 programs handed over
with a full suite of passed unit tests than 10
programs without documented tests.
This condence works not only at handovers
within a project, but also outwards from the
project. Without completion evidenced by
passed tests, all but the most experienced
managers fall into the 95% complete
syndrome, and pass that misinformation to
their managers, the sponsor and end users.Such self-delusion means that everyone is
in for an unwelcome surprise not just a
surprise about the software quality but also
about the project progress.
Another way of understanding the non-
nancial benet of early testing is to consider
it as a risk mitigation technique. In fact, all
testing is about risk mitigation. The earlier you
can close down a risk determine that the
risk is mitigated the more condently you
can proceed to later stages and the better
you can focus your late cycle activities.
Early testing principles applied to
performance testing
The experience of those doing early functional
testing is that it brings clear business benetsand better project management. But are
there potential advantages from detecting
and correcting performance problems at an
early stage? You bet, and in spades.
Performance problems need to be treated as
bugs. Whether performance requirements
are explicit or implicit, if someone in
a project is capable of declaring thatperformance is inadequate, the discrepancy
is a potential bug just like any other gap
between requirement and implementation.
It needs the same scrutiny that a functional
failure undergoes. Ultimately there will be a
determination whether the failure is indeed
a bug.
When you nd a performance failure at
a late stage in the lifecycle, all the costs
of identifying the cause and reworking
the solution are magnied. Functional
code bugs (when they are not caused by
requirements errors) usually relate back
to individual, small sequences of code.
However performance bugs found late in
the lifecycle can call into question a whole
architecture or implementation. The source
of the bug could be a code sequence, or itcould be the conguration or environment.
It is hugely helpful if coding bugs that hit
performance have been eliminated by the
time you do a full-scale performance test on
8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
6/15
The Benets and Practice of Early Lifecycle Performance Testing
5
PA
GE
a complete application. By doing this, you
enter your late cycle performance testing
with more certainty about code, giving you a
chance to concentrate on infrastructure and
conguration.
What We Mean ByPerformanceTesting
Up to now, we havent been precise
about what we mean when we talk about
performance testing. However in order
to uncover the possible approaches and
opportunities for early testing of code that
can impact on performance we need to be
more specic.
Performance testing is commonly used as
a term for several related test types whose
goals are to establish that an end user will
get a functionally correct response from
an application and that the timing of the
response will be acceptable. Some of the
constituent test types are load, volume,
stress, concurrency and soak. We need to
look more closely at the individual test types
to understand which are most suitable for
early testing.
Straddling the functional and performance
domains is concurrency testing. Running
more than one instance of a process
concurrently can raise functional problems
like inconsistencies from simultaneous
database updates and performance issues
like badly formed locks on resources, which
slow things down.
In the narrow sense, performance testing is
a measurement of response time or latency
and the resources consumed to achieve it. To
measure meaningfully requires tests under
conditions that represent, approximate
or can be mapped onto the production
environment. So software developers and
testers most commonly test for responsetime under production-like loading conditions
with production-size data volumes load
testing and volume testing, respectively.
Typical performance, load and volume
tests are measuring against targets based
on normal and peak workloads executed
over timescales of less than one working
day. When a test is run to ramp a system tobreaking point, it becomes a stress test and
when the test execution spans days, weeks
or even months, it becomes a soak test.
Its important to note that most performance
testing is impossible without tools. Tools
are needed for driving execution, measuring
activity and reporting results. Even crowd-
source approaches for driving tests
require tooling. Load and stress testing are
particularly tool intensive. But there are two
subtly different approaches to using a load
test tool.
Most commonly in load testing, testers
attempt to create realistic transaction
scenarios through a load test tool and
drive them through large numbers of virtual
users operating against production-likevolumes of data. During the course of the
test, measurements are gathered across
the software components and infrastructure
under test. This is a major logistical exercise
that characterizes what most software
engineers understand as performance
testing.
However, load can be used in a muchsimpler way, which well call background
load testing. Without attempting to mimic
realistic user activity, virtual users are used
8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
7/15
The Benets and Practice of Early Lifecycle Performance Testing
6
PA
GE to create a background of infrastructure
activity disk access, CPU, network trafc that represents a particular normal loading
of the production environment. Then user
transactions are driven manually and
performance is measured of the manually
entered workload.
When we look at how to do early performance
testing, we need to keep in mind all the
various test types available and not getlocked into the image of the set-piece, large-
scale performance test.
The Case AgainstEarly PerformanceTesting
With so much to be gained, why is the
practice of early performance testing still
relatively rare the domain of a few visionary
test and QA managers plus specialist test
service vendors?
History and tradition in software testing
practice, and more widely in development,
have a lot to answer for. The Waterfall model
for development and, to a lesser extent, the
V model widely known among testers, both
convey a picture in which technical tests such
as performance are relegated to late in the
lifecycle. That is, at least, the most commoninterpretation, even if it was not strictly the
intention. Testing tool vendors have also
contributed. They have bought into the late
cycle testing vision for commercial reasons
and, in turn, the available tools have shaped
(or misshaped) current practice.
We alluded earlier to the soft issues
surrounding people and process that are
used to undermine early functional testing.
Those issues are still present for performance
testing: developers dont like testing and are
generally not accountable for it. Testing is
Chart 2: Relationships among various performance-family test types
8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
8/15
The Benets and Practice of Early Lifecycle Performance Testing
7
PA
GE
considered a lesser technical skill (although
this is patently untrue for performance
or security testing), confers lower status,
is inherently less interesting, less sexy
and probably a bad career move. No onehas made a lm called The Social Testing
Network.
Agile processes have fewer such biases.
Cross-disciplinary teams are the norm, and
it is common for developers to perform
testing. With Agile processes and good
understanding of the benets of early
testing for functional testing, and with betterIDEs and unit test frameworks, it has been
possible to win the hearts and minds of
experienced developers (although newbies
may still need to fail once or twice before
they get the idea) to take some responsibility
for functional testing.
However, adoption of Agile does not totally
solve the problem. Early performance testing
poses two key intellectual challenges that
experienced developers on both Agile and
Waterfall projects have trouble seeing their
way through.
Challenge 1: You can only test performance
on a fully built system in a production-like
environment.
Challenge 2: It is impossible to set and you
almost never have meaningful performance
requirements at the level of individual
components.
It takes a highly motivated and creative
test or QA practitioner to see how these
challenges can be met.
PragmaticApproach To Early
PerformanceTesting
You need to have a pragmatic mindset to
make benecial use of early performance
testing. If you demand complete knowledge
for example of requirements or total
realism for example, of testing environment
you will never get started and never getthe benet of all the positive things you can
do early in the lifecycle to verify that your
software will perform as required. Your
objective is to close down as many doubts
about performance as you can with the
resources you have.
Were going to take a closer look at what
you can do to make early performance
testing a reality. There is no one-size-
ts-all prescription. Our aim is to open up
the possibilities and move away from the
intimidating complexities of the big event
late cycle load test.
Choosing what to test tackling the
risks
You could attempt to do performancetesting on every unit within the development
environment, but when that is not practical
(forget about possible) concentrate on the
riskiest components. Risky components
would include elements that are:
Resource intensive in bandwidth,
CPU, disk access, etc.
Time-critical latency within thecomponent is critical.
Database routines a perennial source
of performance bottlenecks.
8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
9/15
The Benets and Practice of Early Lifecycle Performance Testing
8
PA
GE
The developer is the person most ideally
placed to evaluate the technical risks of the
components they are building.
Selecting test types
No one type of performance-family testing
is totally unsuitable for early testing. It
depends on the architecture of the system
under test and where the risks lie. However,
in developing a general early performance
testing process for a typical commercial
software delivery organisation, the following
types are arranged from the most to least
well suited.
Stress tests are unlikely candidates for early
performance testing. Load (as opposed to
background load) and soak are possibleprimarily as part of continuous integration. The
others are good candidates for developers
alongside functional unit testing.
Strategies for absent performance
requirements
Most performance requirements are cast in
business terms as end-to-end transaction
responses, database capacities, and
operational SLAs. Developers can easily
argue that performance testing of individual
components is impossible when they dont
have performance requirements at the
individual component, method or unit level.
The obvious solution is to analyse the high-
level performance requirements and create
requirements for components. If no one in
a team not even an architect or designer
is capable of doing that, you have to
conclude there is no certainty that the built
application will perform adequately when
the components are assembled into a whole
application. In which case, the need for
more granular performance requirements
and early testing is even greater.
Chart 3: Test types by suitability for early performance testing
8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
10/15
The Benets and Practice of Early Lifecycle Performance Testing
9
PA
GE
In fact, we nd that adopters of early
performance testing can routinely
create reasonable working performance
requirements for components. They
sometimes do this in two distinct ways,
depending on whether they have a reference
point or not.
If there is no existing system or similar
components, they develop low-level
performance requirements in a simplistic,
uncomplicated way taking high-level
requirements and using crude averages
for, for example, numbers of methods perbusiness transaction and an assumption
about client-end and network latency.
If there are existing systems, they use
past performance as a baseline. Agile
developments have the advantage that
they quickly produce baselines from prior
iterations.
Crude as they are, these two approachesproduce reasonable results, especially
coupled with a developers experience and
insight into the code. The information you
get from such tests may not demonstrate
denitively that your codes performance is
adequate. But when a test like this fails, it is
a useful signal to a developer to investigate
more closely.
None of this is intended to remove theneed for good performance requirements
for risky components, but these methods
serve a useful purpose when you need to be
pragmatic.
Strategies for working with non-
production environments
As a rule, developers have neither a complete
application nor a production-like environmenton which to perform early performance
testing, but, actually, sometimes they do.
We tend to look at changes in software
delivery processes in the context of brand
new systems, but they represent the
minority of software delivery projects. When
enhancing existing systems, there is often
a complete application to test, one which
is built with one or more new components.
So the objection that you dont have a fullapplication at the early development stages
turns out to be less true than imagined.
The objection that the development
infrastructure is not sufciently close to
production is also surmountable. The
analysis to compare a development
systems capabilities to a production system
is feasible. Capacity planners do this sort of
calculation all the time.
Putting aside for the moment the process
management issues for maintaining a
performance testing environment for
developers, some organizations do indeed
have available test environments run as
labs with known relationships to production.
While these are generally utilized for late
cycle performance tests, they could be
made available for early cycle testing too.
Testing outside the coding environmentmay entail changes to source management
and build procedures, but these are not
insurmountable obstacles either.
In other cases, the capacity and scale of
the development environment compared
to the production environment are well
understood and can be used to calibrate
results. Once again, Agile iterations make
such comparisons easier.
In our pragmatic approach to early testing, we
can work with ballpark gures. Development
environments are less powerful than
production environments but they are also
less loaded. A component that performs
badly in an early cycle performance test is
an early warning, and a signal to investigate
further.
You still need the big event near the end
Just as you must still do system testing after
early functional testing, you must still expect
a full late cycle performance test. However,
you should enter the late test phase with
8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
11/15
The Benets and Practice of Early Lifecycle Performance Testing
10
PA
GE
fewer uncertainties and more condence in
code performance. You will nd fewer code-
related performance bugs and save time
and costs from less rework.
Introducing EarlyPerformanceTesting Practices
Introducing early performance testing should
be treated like any change managementproject. Good communications and
sensitivity to people are always essential.
However, here are some specic points to
keep in mind.
Developing motivation
You need to convey two distinct messages:
that the benets of early cycle performance
testing are worth it and that it is feasible to
do.
The starting point has to be a clear
explanation of how nding defects early
has both a nancial benet to the business
and improves project manageability and
predictability. It must be made clear that
project delivery schedules will be adjusted
to take into account the additional up frontwork, and that any learning curve is built-in.
Whats in it for those who will adopt the
practice? It depends on your organisations
culture. The certain benet is that for an
additional effort in development, there will
be far fewer awkward disruptions later for
debugging and rework.
Beyond the benets, developers needto be reassured of the feasibility. For this
you will need to ensure that the change
management project includes a skilled
champion or evangelist someone who
instils condence.
In return, coders should be clear that
they become as accountable for early
performance tests whatever scope you
have chosen as they are for code and
functional unit tests.
Integration with existing processes
A software delivery organisations process
maturity matters. You need to be already
operating at a mature level of testing process
before you undertake early performancetesting.
We dont see any intrinsic unsuitability for
early performance testing within Waterfall
or Agile projects, but Agile does have some
characteristics that make introduction of a
new test type a little easier, for example:
The cross-disciplinary nature of Agileteams means they are used to coders
and testers working together
Agile iterations quickly begin to
provide the information needed to
dene component level requirements
and map development environment
performance onto production
performance
The Agile culture is generally more
dynamic and accepting of change and
uncertainty. Agile developers are more
pragmatic about incomplete
requirements.
We recommend that your process include
risk-based analysis for deciding whether
to do early performance testing on a
component, how much and what type (based
on the types we outlined earlier.)
There are several existing processes that are
likely to be touched by early performance
testing:
8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
12/15
The Benets and Practice of Early Lifecycle Performance Testing
11
PA
GE
Coding stage testing
Build process and/or continuous
integration
Source code management.
We would expect to see processes for the
test types we identied earlier assigned
to the coding stage. Mature integration
and build processes, with integrated
functional unit test suites, can be extended
to included performance tests as well.
Those performance tests could include
any component performance tests, but
also further performance tests on the builtassembly of components. It is at this stage
that some early load testing becomes
feasible in addition to the early background
load testing you can do on components.
Changes to source code management
and build procedures may be needed if
performance tests are executed outside the
coding environment.
The pragmatic approach should be
encouraged across the board to ensure
exibility. It is essential that imperfect
knowledge of requirements and environmental
behaviour not be used as an excuse not to
test, where experience and common sense
could with the right attitude plug the
gaps.
Specialists
You will need the buy-in of those who are
usually involved in late cycle performance
testing, including any specialist performance
testers, architects, infrastructure experts
(network, DB, service orchestration). They
will need to change their own processes to
include support for the early performancetesting. This is sometimes viewed as an
opportunity because they have been at the
sharp end of the problems inherent in leaving
all the testing towards the end. However, as
the industry has developed with its bias
for late-cycle testing you may nd that
performance testing specialists may feel
threatened that their specialism is being
undermined.
Tools
Performance testing of all types requires
tools. For early performance testing, you
need tools that t in with coding and
integration environments and that dont
require all the scaffolding and preparation
of the set piece late cycle performance test.They need to support middleware and back-
end components as easily as they support
client-end scripts.
One commonly held idea is that functional
unit tests can form the basis of performance
tests. More often than not, this proves not to
be true, so dont rely on it.
It is worth pointing out that for background
load testing where you are just trying to
create a background of machine loading
against which you will run manual transactions
you dont need as ne a control of the
timing and ramp up of automated scripts
as you need for full load testing. Virtual user
scripts and orchestration by a virtual user
controller can be relatively simple. You are
not concerned about the performance of the
individual virtual user transactions; you are
interested in the load they place overall on
resources.
A key factor in the cost-benet equation
of early performance testing is the cost of
tools. When they are available in developers
IDEs the marginal cost is low. However,
the cost of virtual users for driving loadtests can become a signicant expense. It
is important to make a reasonable stab at
estimating how many are really required.
On one hand, there is a tendency to over-
8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
13/15
The Benets and Practice of Early Lifecycle Performance Testing
12
PA
GE
specify the number, perhaps in the interests
of expediency. On the other hand, it can be
a mistake to scrimp and attempt to make up
for the shortage by distorting the content of
the scripts. The essential point is that you
want the right number of virtual users to
achieve your goals, whether that is 25, 1000
or more.
And one nal soft issue: you will need to
develop the skills of tool specialists who can
mentor others.
Conclusion
Early performance testing may not be
the rst test process improvement a
development organization chooses to make,
but there is no good reason why it should be
as overlooked as it is. Agile processes and
powerful development environments with
integrated testing capabilities are conducive
to a pragmatic approach. Determination to
obtain the benets of lower rework costs
and more manageable projects provides the
motivation.
Microsofts ALM Assessment has been
designed by expert ALM practitionersto help you understand how well your
organisation currently approaches
ALM and where and how you can make
improvements.To nd out more, click here.
http://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessment8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
14/15
Dont forget to join us for EuroSTAR 2012 in
Amsterdam from the 5 8 November.
Its our 20th anniversaryand we are going to
celebrate in style!
EuroSTARSoftware TestingC o n f e r e n c e
http://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessment8/2/2019 The Benefits and Practice of Early Lifecycle Performance Testing
15/15
w w w . e u r o s t a r c o n f e r e n c e s . c o m
Join the EuroSTAR CommunityAccess the latest testing news! Our Community strives to provide testprofessionals with resources that prove benecial to their day-to-day roles.These resources include catalogues of free on-demand webinars, ebooks,
videos, presentations from past conferences and much more...
Follow us on Twitter @esconfs
Remember to use our hash tag #esconfs when tweetingabout EuroSTAR 2012!
Become a fan of EuroSTAR on Facebook
Join our LinkedIn Group
Contribute to the EuroSTAR Blog
Check out our free Webinar Archive
Download our latest ebooks
The EuroSTAR BlogWritten by Testers for Testers
http://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.microsoft.com/visualstudio/en-gb/strategies/almassessmenthttp://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/Recommended