Top Performance Land MinesAnd how to address them

Andreas Grabner, Lead of Center of Excellence @ dynaTrace

WHY do WE care?

OnlineGraduatePrograms.com

“Instant America”

OnlineGraduatePrograms.com

“Instant America”

WHAT do WE miss?

• Deployment mistakes

• Traffic estimates incorrect

• Poorly provisioned hardware

• No consideration of end user devices or latency

• Misconfigured CDN

• Load Balancing Issues

• Networking/infrastructure issues

• Problematic 3rd parties

• Bloated Web Frontends

• Too many Database Statements

• Too heavy logging and tracing

• Synchronized to Death

• Too chatty on remoting channels

• Wrong use of O/R Mappers

• Memory Leaks

Performance Land Mines found in Production

Problem #1: Size

Watch your Weight(Content Size)

May '11

Jun '11

Jul '11 Aug '11

Sep '11

Oct '11

Nov '11

Dec '11

Jan '12

Feb '12

Mar '12

Apr '12

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

0

5

10

15

20

25

30

35

40

45

50

Avg Bytes Avg GPSI # of Objects

Byte

s

# of

Obj

ects

/Avg

GPS

I

Where did all of this stuff come from?(Application Complexity)

29Average number of hosts accessed by the browser across the Gomez US Sports Web Performance Benchmark

Where did all of this stuff come from?(Application Complexity)

12Average number of hosts accessed by the browser across the Gomez US Sports Mobile Website Performance Benchmark

Cost of Complexity – 1

Page Size

As the page gets larger, it gets slower and more

complex

Cost of Complexity – 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54

0

2

4

6

8

10

12

14

16

0

500000

1000000

1500000

2000000

2500000

3000000

Response Time and Page Size by Hosts

Geometric Mean BYTES

Hosts

Seco

nds

Byte

s

Hosts

As the page gets more complex, it gets slower

and larger

Tips for Testing: Analyzing Page KPIs

• Analyze Key Performance Indicators (KPIs) for every tested page

Tested Pages

KPIs

Tips for Testing: Comparing Test Runs

Tips for Testing

• Free Tooling– dynaTrace AJAX Edition for IE and FF

– YSlow for FF

– SpeedTracer for Chrome

– ShowSlow as performance repository

Problem #2: Deployment Issues

Top Deployment Problems

• Missing Files

• Incorrect Access Settings

• Slow Web Server Modules

(1) Missing Files: Impact User Experience

Enter Here

PopUp Here

(1) Missing Files: Lead to JavaScript Errors

Errors caused by missing files

(1) Missing Files: How to identify

• Watch out for HTTP 4xx

(1) Missing Files: Tips for Testing

• Test with real browsers– HTTP-only testing might not download dynamically loaded JS files

– Catch browser-specific implementations

• Test from the In- and Out-side– Files might be deployed on AppServer but blocked on Web Server or

Load Balancer

– Certain files might come from a CDN. Testing the CDN settings is important

(2) Incorrect Access Settings: Impact User Experience and Performance

14k Errors in 5 minutes after deploying in Production

(2) Incorrect Access Settings: Impact User Experience and Performance

Deployment causes extra roundtrips for each restricted resource

(2) Incorrect Access Settings: Tips for Testing

• Test with real browsers– HTTP-only testing might not download dynamically loaded JS files

– Catch browser-specific implementations

• Test from the In- and Out-side– Files might be accessible on AppServer but blocked on Web Server

– Watch out for missing JS, CSS and Image files

(3) Slow Web Server Modules: Performance Impact

• 90% of ASP.NET Request Time spent in IIS

90%

(3) Slow Web Server Modules: Tips for Testing

• Test with all Modules used in Production

• Analyze Time spent in Modules vs. Application Code

Problem #3: Overhead through

Logging/Tracing

Logging is important!!

• As otherwise we are really blind – but – it comes with a cost

?

(1) Excessive use of Exceptions

• Tomcat Connection Pool with logAbandoned=true

A debug flag causes many exception objects to be created

(1) Excessive use of Exceptions

• HUNDREDS of exceptions per request

(1) Excessive use of Exceptions

• Getting Stack Trace that many times is a performance risk

(2) Too granular logging

• Too much logging doesn’t provide much value

Tips for Testing

• What to watch out for during a test– Number of log entries written

– Size and growth of log files

– Usage of Stack Traces in Logs -> Usage of Exception Objects

• Test Production Readiness by– Watch out for DEBUG, FINE, FINEST, … Log Messages -> should not be

logged

– Show produced log files to developers and ask whether they are useful

Other Problems: Excessive Database

Common Database Access Problem Patterns

Application

Database

select ... where id=1

select ... where id=1

select ... where id=100

Database

select ... from master

select ... from detail

select ... from detail

select ... from detail

Database

select ... from master,detail where ..

Application

Application

N+1 Query/Too many SQLs

Lazy vs. Eager Loading

Application Connection

getConnection()

closeConnection()

Connection locked

Application Connection

getConnection()

closeConnection() Connection lockedCode requiring DB access

Code NOT requiring DB access

Unnecessary Resource Usage

Loading too much data

(1) N+1 Query Problem

(2) Loading too much data

• Example from SharePointfor (int itemIx=0;itemIx< SPContext.Current.List.Items.Count;itemIx++) {

SPListItem listItem = SPContext.Current.List.Items[itemIx];

// do something ...

}

Every access to Count and Items Property queries the whole SharePoint list

We end up with 202 SQL Executions with a total exec time of > 1s

Tips for Testing

• Watch out for– Slow SQL Statements

– Number of SQL Statements per Request

– Same SQL executed many times per request

– Prepared vs. Unprepared Statements

– Connection Pool Usage

Other Problems: Synchronization

Identifying Synchronization Problems

BAD

GOOD

Response Time

CPU

Tips for Testing

• Use Performance Management Tool

• Take Continuous Thread Dumps

Other Problems: 3rd Party Content

3rd Party Content is important

Testing 3rd Party Content

• Watch out for Size and Number of Resources# of Domains # of Resources Total Bytes DNS [ms] Connect [ms]

With Third Party Content 26 176 2856 Kb 1286,82 1176,09

Without Third Party Content 2 59 897 Kb 0,91 22,25

• Remember: Watch this over time

Testing 3rd Party Content Performance Impact

Testing 3rd Party Content Performance Impact

Resources

• Free Online Java Enterprise Performance Book– http://book.dynatrace.com

• Our Performance Blog– http://blog.dynatrace.com

• Contact– andreas.grabner@compuware.com

– @grabnerandi

THANK YOU