Top Challenges Performance Engineers Face in 2013

7/27/2019 Top Challenges Performance Engineers Face in 2013

http://slidepdf.com/reader/full/top-challenges-performance-engineers-face-in-2013 1/14

January 2013



What are the top concerns and challenges performance engineers face in 2013? In which

stages of the software development life cycle (SDLC) should you engineer for performance?

Which teams should focus on performance? What are the expected top challenges of your

2013 performance strategy? Our respondents answered these questions and more.

This report reveals findings from a Shunra

survey about the top concerns and

challenges performance engineers face in

2013. The results were released during a

webcast on January 14, 2013. Based on the

responses from 316 IT specialists,

managers, architects, developers, and

engineers, the survey uncovered howperformance engineers plan to include

performance in application development

and deployment plans, important

elements to include in 2013 performance

strategies, and how to account for

production network conditions before

deployment. Confirming a growing trend

regarding an increased business focus on network virtualization and application

performance engineering, 17.7% of the 316 IT job titles included the word “performance.”

The following pages detail the findings from each of the questions posed to our

participants.

Analyst

4%

Architect,

Developer,

Engineer

27%

IT Executive

4%

IT Manager

41%

IT Specialist

16%

QA

4%

Other

4%

Watch a webcast summary of the report findings at: http://www.shunra.com/resources/on-demand-webinars.

http://www.shunra.com/resources/on-demand-webinars






Responses to this question were categorized across three

core areas of the SDLC: Design/Development, QA/Test and

Staging/Production. Respondents could select multiple stages

of the application life cycle.

61% of the survey participants do not design or develop for

performance today, but do test for performance in the later

stages of the SDLC. This demonstrates recognition of the need

to ensure performance in deployed applications, but a

persistent failure to consider performance requirements inthe earliest stages of the application life cycle.

Conversely, 7.91% of survey participants only consider

performance in the Design and/or Development stages. These

particular companies show an understanding of the need to

consider performance early, but do not take the necessary

steps to validate those efforts.

Do not design or develop for

performance

Not at all

All stages

Staging or Production only

During QA/Test only

Design or Development only

61.00%

5.06%

8.80%

10.12%

16.13%

7.91%



Only 8.8% of companies surveyed showed the maturity to consider performance

throughout the entire SDLC.

Less than 10% of companies surveyed have made performance a policy and built

performance into their development and deployment best practices. This represents a

significant and costly gap which is demonstrated by recent media and research indicatingthat applications continue to fail to live up to end user expectations.

Application performance is the number one factor impacting application success and end

user adoption. It also remains the number one cost when it comes to developing, managing

and maintaining applications.

When asked when performance is

considered in different types of testing, not surprisingly, load

testing was most frequently

selected. However, demonstrating

the growing trend to incorporate

performance throughout testing

methodologies, over 90% consider

performance in at least two

different test types.

Of concern, especially consideringthe impact of performance on

mobile app users, is that less than half of survey participants consider performance with

functional testing. Traditional testing methodologies are incomplete if conditions affecting

application performance are not incorporated. They do not account for the dynamic nature

of today’s global networks, and are therefore not reflective of real end-user experience,

thus accounting for the large number of applications that fail in production but “work ed

fine” in test .

For example, load testing in a pristine test environment will provide some insight into

system scalability, but without an understanding of the network conditions experienced by

different (virtual) user groups, assumptions made regarding capacity requirements will be

misleading, as will test results. If one of these virtual user groups is a mobile audience, it is

28.20%

42.70%

83.90%

75.30%

Single unit test

Functional testing

Load testing

Capacity/Scalability



imperative to account for the longer mobile user sessions that result from the inherent

latency of mobile networks and how system resources will be affected.

In the same manner, functional testing in a pristine test environment is not indicative of

how a user will experience an application or website. Functional test results may confirm

that button A leads to transaction B, but if that transaction takes 30 seconds or longer inthe real world due to network constraints, end users may not wait long enough to see the

results. Performance thresholds must be set, and violations of those thresholds should be

considered a functional fail.

Industry experts tell us that testing in all four categories is the best practice, but only

12.65% chose all four types.

In order to create a reliable testing environment, a“three-legged stool” approach is often employed. The

seat of the stool represents any test environment –

load, functional, single user, etc. The test platform is

held up by three legs:

User virtualization: Test environments should

emulate the users or load that will be

accessing the system. This is accomplished via

load generators and load testing tools. Some

organizations apply single user test automation or even have manual users to

perform tests as real users in the production

environment.

Services virtualization: In production, applications often rely on third party

resources or data feeds. These services can number in the hundreds for larger and

more complex applications. It is not always practical, possible or affordable to bring

those services into the test environment, yet performance of those services is

critical to end user experience. Service virtualization emulates the behavior of

specific application dependencies that developers or testers need to exercise inorder to complete end-to-end transactions in test environments. Rather than

virtualizing entire systems, it virtualizes only specific slices of dependent behavior

critical to the execution of development and testing tasks.

S e r v i c e s

Test Platform



Network virtualization: As with services, network conditions play a critical role in

user experience. With mobile apps, for example, up to 70% of end user experience is

dependent on network conditions. Network virtualization is a pre-production

process for recreating network conditions from the production environment for use

within the test environment. Network conditions such as latency, upstream and

downstream bandwidth, packet loss, and jitter, are all critical factors that must be

taken into account when testing or validating application performance. Network

virtualization enables connections between applications, services, dependencies and

end users to be accurately emulated in the test environment.

This combination of “legs” stands up any test environment, improving the accuracy of test

results. What happens if you try to rest your test platform on this stool without all three of

the legs in place? You fall over. A test environment without all three aspects is

incomplete.

Only 37.97% of survey respondents apply the industry best practice of accurately

emulating the production environment by virtualizing all three legs of the stool.

Showing some promise, 36.70% of organizations surveyed address two of the three legs.

However, over 20% address just one leg of the stool, and an unfortunate 4.11% do not

address even a single leg.

Users, Services,

and Network

38%

At least two legs

37%

At least one leg

21%

No legs

4%

Users, Services, and Network Virtualization

Users and Services – 18.67%

Users and Network – 14.87%

Services and Network – 3.16%

Users – 9.81%

Services – 6.33%

Networks – 5.06%



Regardless of the number of “virtualized legs” you use, the large majority understands the

importance of load testing with 81.30% factoring users/load testing into their

environments today. Also worth noting from this question is that less than half (45.30%)

consider testing the end user device.

38.29% do not plan to test for network conditions or are going to test in production only.

16.4% do not plan to test for networks conditions at all in 2013 and 21.89% are going to

wait until they deploy. This is a top challenge for performance engineers in 2013. Testing in

production only is not enough.

Industry best practices dictate that all teams have a focus on performance. However, only

19% of our respondents have a best practice in place regarding team focus on

performance. One out of five respondents considers performance across the board

(choices included architect, developers, QA/test, operations, line of business owners). Over

95% of organizations have assigned performance responsibility to at least one team. More

often than not, that team was QA/Testers who comprised nearly 65% of all responses.

Only 24.4% of businesses identified line of

business owners as having a focus onperformance. A higher percent was expected

as line of business owners have a

responsibility for their business units’ bottom

line which is heavily influenced by application

adoption and use. As the mobile wave

continues to grow, and as mobile apps

continue to be initiated by individual business

units, a larger percentage of line of business

owners is expected to be assigned applicationperformance responsibilities.



Understanding internal and/or external constraints was selected by 81.01%. Of the

responses related to these constraints, having the ability to understand internal system

constraints, such as contention, memory usage, database, and storage, was selected by

72.80% of respondents; in addition, the ability to understand external system constraints,

such as use of external web services, CDNs, and cloud-based services was deemed

important to 58.90% of respondents.

Having the ability to understand inefficient use of network resources (chatty latency-

dependent applications, uncompressed content, duplicated/redundant requests, etc.) was

chosen by 68.40% of participants. Knowing how applications scale during peak usage

periods is important to 71.20% of our respondents, and assessing real-world network

conditions and the impact on their performance is important to 72.20% of participants.

Approximately one-third (35.44%) chose all the answers as important to their 2013

performance strategies. Which of these did respondents feel would be challenges in 2013?

All areimportant

35.44%

Understandinginternal and/orexternal system

constraints

81.01%

Assessing real-world, end-user

networkconditions

72.20%

Understanding

inefficient use of network resources

68.40%

Understandingof how appsscale duringpeak usage

periods

71.20%



The top two responses are related to the network, which could be related to the maturation

of the mobile market. A year ago, performance engineers focused more on the device and

the functional aspects of the application, and now the growing realization is that functional

testing will only capture part of the problem and the impact of the network must be

considered.

66.13% chose assessing real-world, end user network conditions and

their impact on application performance and/or understanding the

inefficient use of network resources as top challenges.

Why should you engineer for performance?

• Identifying and remediating production issues prior to deployment saves hundreds

of thousands, if not millions, of dollars.

• Reliably knowing how an application will behave empowers IT professionals with

the foresight and knowledge needed to make informed decisions about application

deployments.

• Accurate performance testing enables organizations to meet and validate SLAs and

SLOs.

25.10%

29.60%

30.20%

40.80%

45.30%

Understanding of how apps scale during peak usage…

Understanding internal system constraints

Understanding external system constraint

Understanding inefficient use of network resources

Assessing real-world, end-user network conditions

Top Challenges Performance Engineers Face in 2013



Investments in performance

engineering are shifting left in the

development life cycle, as

organizations continue to better

understand the critical importance

of designing and developing for

performance. In this survey,

56.96% have increased budgets or

their performance budget has

remained intact from 2012. A verysmall amount, 6.01%, had a budget

decrease from 2012 to 2013.

37.02% do not have budgets in

2013.

The Cost of Poor Performance

Industry analysts, like Forrester, EMA and others, agree that the average cost to a

business of a production incident can exceed $45,000 per hour . This cost includes

multiple factors that impact the business, including the cost of resources to resolve

the issue along with the impact of the issue on business factors like revenue,

employee productivity and customer satisfaction. The impact to the business’

bottom line varies on the function and criticality of the affected application. In

another Shunra survey, our customers estimated the cost per incident at $80,000

for remediation, but this did not consider the impact on the business like brand

damage, revenue loss, or customer dissatisfaction. Based on ROI input from

customers and the average cost of a production incident as reported by industry

analysts, we know that an incident that requires one day to remediate can cost abusiness $360,000. A three-day remediation quickly turns into more than $1M, and

a six-day remediation can cost a business $2.1 million.

Yes, and

budget has

increased.

25.63%

Yes, and

budget has

remained the

same. 31.33%

Yes, and

budget has

decreased.6.01%

No budget for

designing for

performance.

37.03%



Mobile is rapidly becoming important to employee productivity. A sign of inefficient testing

practices and performance assurance, 67.8% of responding organizations reported

experiencing problems related to the performance of critical business applications

accessed by employees via mobile devices. More troubling is the fact that only 44.3% have

a 2013 plan in place to proactively assess the performance of these applications.

While concerning, this data is also not entirely

surprising. Testing mobile apps and ensuringmobile performance is challenging. When you

add a requirement to have a mobile load

scalability test, for example, there may be

disparity between the recognition of the need,

the expertise required and availability of

necessary tools.

So why are less than half of survey respondents

doing something about this issue? The World

Quality Report supports these findings.

Capgemini, one of the world’s foremost

providers of consulting, technology and

outsourcing services, and Sogeti, its local professional services division, recently released

the findings of the fourth annual World Quality Report. The report, published in

conjunction with HP, revealed that organizations are struggling to manage the challenges of

the mobile era, with only one-third of those surveyed currently formally testing their

mobile applications.

Only 6% of respondents reported “critical problems,” though over 60% cited the

occurrence of at least a few problems. Even these few problems are a concern when it

comes to mobility. Just a fraction of a second delay in transaction response times means

increased abandonment of mobile users, resulting in decreased revenue (in the case of m-

commerce apps) or loss of productivity (for internal app use). This also brings the light the

No

problems

32.30%

A few

problems

41.80%

Multiple

problems

20.30%

Critical

problems

5.70%



interpretation of severity. What may seem to be a trivial problem, or minor nuisance to an

organization, may actually be much more meaningful, and frustrating, to an end user.

Studies have shown that if you have a revenue-generating application, a failure can impact

up to 10% of your revenue. Furthermore, according to a survey from CA Technologies, the

loss associated with IT issues has been quantified as a 63% reduction in productivity,

resulting in an average of 552 people hours lost per year per company.

Most performance engineers

are juggling several projects in

any given year, so it’s not

surprising respondents chose

multiple projects of concern

for 2013. Mobile projects are of highest concern for

performance engineers in

2013, with 65.5% of

participants, again reflecting a

likely expertise or tool gap that

exists with mobile testing.

Other projects of concern

include Cloud migrations and third-party software deployments. Of note, data center

relocations were among the top three project concerns. Shunra research shows that up to39% of applications fail to meet user expectations after a data center move or

consolidation, underscoring the need to consider performance not just with application

deployments but also with infrastructure change.

Mobile

Cloud

DCR

SAP deployment

CRM deployment

65.5%

60.4%

42.0%

16.4%

17.1%



The largest response to this question indicated that a limited to no budget is the hindrance,

with 57.10% choosing this response as a reason. The impact and cost of failure forperformance issues actually far outweighs the budget required for proactive performance

testing. According to National Institute of Standards and Technology, approximately 80%

of the total cost of ownership of an application is spent and directly attributable to finding

and fixing performance issues post-deployment, costing businesses $60 billion annually in

the United States alone; one-third of those costs could be avoided with better testing

practices.

The second most popular reason survey respondents are not proactively testing for

performance is due to an inadequate testing environment at 49.20%. Lack of management

support came in third with 26.70% as the reason companies are not proactively testing forperformance. The gap which exists between knowing what must be done to ensure end

user experience and actually employing best practices must be closed. This can only be

accomplished by both a top-down and bottom-up approach, with all areas of the business

recognizing the financial implication of poor performance and an enterprise-wide

commitment to end user experience.

Agile methodologies continue to gain traction with development teams. Of the 316 survey

participants, 57.9% are using While Agile adoption is still growing, much has been written

about the challenges of Agile and how performance fits into an Agile environment, if at all.

According to the survey, only 25.6% have Agile organizations have performance as part of

their requirements. Just 15.5% integrate performance acceptance tests into automated

tests. Less than 14% fully automate performance and can compare each build to prior

builds for response time and resource utilization. For more insight and explanation around

the Agile-performance dilemma, please view the Shunra webcast:

How does performance testing fit into Agile?

http://ape.shunra.com/PerfTestandAgile.html






Application performance is a competitive differentiator and a clear indicator of business

success. However, 80% of the costs associated with application development occur in

remediating failed or underperforming applications after deployment, when the ineffective

application has already had a negative impact on the end user or customer experience.

One-third of those costs could be recovered with better testing practices.

Shifting left in the software development lifecycle is the key – testing early and often will

help organizations avoid the negative impact of performance failure while affording the

opportunity to capitalize on end user demands for immediate and ubiquitous access to

data, creating a significant competitive advantage for those organizations that are most

successful in building performance in to their application design, development and

deployment efforts.

Network conditions such as latency, limited bandwidth, packet loss, and jitter, are all

critical factors that must be taken into account when testing or validating application

performance, especially when considering the multiple network connections required to

support 3rd party services and external resources. With Network Virtualization, you

discover the real-world network conditions affecting end users and application service,

virtualize network impairments (such as high jitter rate and limited available bandwidth)

in the test lab so that the effect of the network on transaction response time can be reliably

and accurately measured, and then remediate or optimize performance.

Shunra has the benefit of working with more than 2,500 organizations globally and with

some of the most complicated application infrastructures in the world. For over a decade,

Shunra has witnessed a dramatic shift in awareness and focus on application performance

and seen this gap continue to shrink. Great progress is being made, and Shunra has been

fortunate to help our customers implement network virtualization solutions for software

testing and incorporate application performance engineering policies that encompass best

practices and proven strategies for improving and ensuring application performance. New

methodologies and approaches like Agile and DevOps continue to move companies towards

performance optimization, but greater awareness and performance policy across the

enterprise are still needed in order to truly achieve performance Zen.

More than 2,500 organizations use Shunra Network Virtualization solutions to proactively

test and validate application performance with real-world, virtualized network conditions

before deploying to production. Visit www.shunra.com for more information.

http://www.shunra.com/




Documents

Top Challenges Performance Engineers Face in 2013