Tech Talk: Performance benchmark

Tech Talk: Performance benchmark

Key benchmark deliverables➢ Performance benchmark report

➢ Issues detected/fixed in each

iteration

➢ Optimizations performed in

each iteration

Figure: sample performance benchmark report

Number of

times executed

Number of times

executed

Number of times

executed

Avg. Test Time

(sec)Avg. Test Time (sec) Avg. Test Time (sec)

(throughput for

iteration 1)

(throughput for

iteration 2)

(throughput for

iteration 3)(iteration 1) (iteration 2) (iteration 3)

994 executions 1470 executions 194 seconds = ~3 minutes 1 minute 52 seconds

From iteration 1: +190%From iteration 2:

+47%From iteration 1: -64% From iteration 2: -42%

Total gain: +329% Total gain: -79%

951 executions 1471 executions 187 seconds = ~3 minutes1 minute 44

 seconds

From iteration 1: +182%From iteration 2:

+54%From iteration 1: -65% From iteration 2: -44%

Total gain: +336% Total gain: -81%

890 executions 1430 executions 180 seconds = 3 minutes 1 minute 44 seconds

From iteration 1: +142%From iteration 2:

+60%From iteration 1: -62% From iteration 2: -42%

Total gain: +289% Total gain: -78%

935 executions 1497 executionsSubmit: 2.0

secondsSubmit: 2.0 seconds End-to-end flow: 9 seconds

From iteration 1: +161%From iteration 2:

+60%

End-to-end flow:

21.0 secondsEnd-to-end flow: 12.0 seconds From iteration 2: -25%

Total gain: +318% From iteration 1: -42% Total gain: -57%

N/A N/A N/A N/A

(run in isolation) (run in isolation) (run in isolation) (run in isolation)#9 Price verification 1461 5 seconds End-to-end flow: 45,7 seconds

#12 Sales order (flow A-1) 368 executions 3 minutes 469 = ~8 minutes

#10 Stock verification 358 executions 5 seconds

541 seconds = ~9

minutes

#8 Sales order normal flow 337 executions 3 minutes 534 = ~9 minutes

Scenario

IDScenario

Maximum

acceptable target

from business

#11 Sales order (flow A-2) 342 executions 3 minutes

Tech Talk: Performance benchmark

- Focus on the performance

requirements

- Keep the objectives reasonable

- Document in BRD/FRD

1. Define

performance

objectives

2. Define

scenarios

3. Design and

develop

system

6. Configure

test

environment

5. Build test

simulations

and data

4. Design

tests

7. Execute

tests

8. Tune and

Retest

iteratively

9. Report

- Focus on all critical components

- Keep them close to reality

- Document in BRD/FRD

- Keep goals and objectives in

mind

- Analyze traces and fix issues

- Maintain constant checks on

progress

User1 loginSO

Headerwait Line wait Post

User2 loginPO

Headerwait Line wait Post

User3 login WaitGL

Headerwait Line wait Post

User4 login waitPO

Headerwait Line wait Post

User5 loginSO

Headerwait Line Line Line Line

User6 login waitGL

Headerwait Line wait Post

User7 loginSO

Servicewait

SO

Servicewait SO Service

ConcurrencyLoad pattern

Transaction Mix

Pacing

Network/Client

Narrow it

down

Troubleshoot

Develop the

solution

Test the

solution

Success or

next iteration

ACTIVITY PARTNER (sample) CUSTOMER (sample)

Define the target/projected business goals I AR

Define the detailed benchmark scenarios RIC A

Take task recordings and document the reproduction steps I AR

Provide the environment artefacts (code build and database to use) I AR

Build the benchmark environment R A

Create test scripts and data scripts R A

Execute the performance benchmark R A

Deliver the performance benchmark report R A

In case of bugs in the standard solution, open a support request to Microsoft C AR

Tune performance (outside Microsoft's scope, e.g. configuration and

customizations)R A

CAPTION R: Responsible A: Accountable C: Consulted I: Informed

Tech Talk: Performance benchmark

the environments

planning Tech Talk

Tech Talk: Performance benchmark

1. Define and

capture scenarios

2. Environment

preparation

3. Create and

validate scripts with

single user

4. Convert to multi-

user tests

5. Performance test

with incremental

workload

6. Conclude and

report

Tech Talk: Performance benchmark

Main article on Docs for PerfSDK

Tech Talk on the performance testing approach

Tech Talk on the troubleshooting tools

Tech Talk on the patterns / anti-patterns

Tech Talk on the RSAT

Considerations for retail performance testing