Exactpro: Non-functional testing approach

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com

1

Non-functional testing approach

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 2460, +1 415 830 38 49www.exactpro.com

http://www.exactpro.com/

http://exactpro.com/


2

Contents

• NFT Team structure• Tests preparation • Test tools• Latency measurements• Capacity tests• DLC test• Failover tests• Other NFT tests



3

NFT team structure



4

LSEG Non-Functional Testing Team



5

Test plan

Sample of an Non-Functional Test plan



6

Load Generators Classification

Measurement-Based Model-Based

Closed-Cycle Open-Cycle



7

Asymmetric Response Model



88 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com

Item Description

Exactpro Test Strategy Step(s)

Load generation in trading systems

Testing Type Active Load and Non Functional Testing

Target SUT Trading Platforms, Market Data Delivery and Post-Trade Systems and their combinations

SUT Interface Back-end (typically connected to message gateways / APIs; data streams generation: mcast/ucast); GUI Testing Capabilities not supported

SUT Interaction Method

Inputs and outputs are generated based on the configured load shapes, parameters and templates. Captured messages can be viewed and analyzed post-factum using the DB queries (Shsha) or/and performance calculator tool (also developed by Exactpro)

Protocols Extant plug-ins for Industry-standard (FIX and dialects, FAST, ITCH, etc.) and Proprietary (MIT, SAIL, HSVF, RTF, RV, Quant House, etc.) protocols. New plug-ins for additional protocols developed by request

Test Scripts Capable to stress the system with high rate of transactions including microbursts. Used for Throughput, Bandwidth, Latency tests. Can be used for support of fault tolerance (Failover) tests

Test Management, Execution and Reporting

Simulation of multiple client connections with specified load shape for each connection or group of connections (configure number of connections, messages templates, Load Shape for each connection or group of connections, messages distribution for each connection or group of connections), throughput up to 75,000 msg per core per secondSimulation of market data streams with required SLAs

Platform requirements Linux on 64-bit platform

Primary Competitor VeriFix TestPilot, HP Load Runner, IBM Rational Performance Tester, JMeter, Yandex.Tank

Load Injector test tool



9

Exchange testing common scheme



10

Tests preparation

• System configuration:- Production setup (Instruments, trading parameters, etc)

• Data for building test load:



11

Tests preparation. Load injector configuration files

Load injector configuration files structure



12

Tests preparation

Load shape:



13

Latency results End-to-end

Do we actually send what we thought we send?

• Evaluation of message rate ‘per millisecond’ unit and order mix balance:

Number of msgs per millisecond

% SamplesInbound (into MIT)

Outbound (from MIT)

<5 55.64% 55.01%5-8 3.67% 4.05%8-10 2.60% 2.77%10-15 5.32% 5.39%15-20 5.88% 5.95%20-80 26.85% 26.78%>80 0.05% 0.05%

Partition 1 Message

TypeME cores

Total0 1 2 3Order 3.74% 3.02% 2.00% 4.14% 12.89%Cancel 3.56% 2.89% 1.93% 4.02% 12.39%Amend 0.60% 0.53% 0.34% 0.68% 2.16%Quote 0.32% 0.11% 0.16% 0.27% 0.85%Trades 0.24% 0.18% 0.13% 0.29% 0.84%

Order mix and load balance across ME cores:Message rate per millisecond:

• Internal system stats arbitration (rates per second and total amount of transactions)



14

Latency results End-to-end

%Native

avg max100 82 518

99.99 82 40899.9 82 139

99 80 103

12 31 48 76 320

687

2527

4005

1874

237

869 30 45 62 181

546

1299

3009

1292

829

990 29 32 61 77 436

791

2545

4977

2540

4

0

100

200

300

400

500

600

max_100max_99_9max_99_0

rate, mps

Late

ncy,

us

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101

106

111

116

121

126

131

136

141

146

151

156

161

166

171

176

050000

100000150000200000250000300000350000400000450000500000

Totalnew_oamend_onew_qcancel_o

Latency, us

Num

ber o

f sam

ples

Latency distribution:

Latency percentiles:



15

Other NFT tests

• Capacity test

– Check all external interfaces:• Traiding FIX• Trading Native• PostTrade• DropCopy• ITCH Replay/Recovery

– Check hardware resources consumption by system components:• CPU• Memory• Disk• Network



16

Other NFT tests

• DLC test– Pass system through Production like schedule:

• All trading cycles• All scheduled sessions

– Apply appropriate load during various cycles to trigger extensions, special sessions, etc– Perform some functional tests under load

– Data consistency check • All expected messages are disseminated via ITCH during session transitions• All Downstream systems connected and confirmed daily checks



17

Other NFT tests

• Failover tests– Application failovers– Hardware outages

• Application failovers– Simulation of various outage types (SIGKILL, SIGSTOP, Stop via SysGuard)– Failovers during different system state (at startup / trading day / EOD)

• Hardware failovers– Network equipment failovers (Switches, Ports, NICs) – impact on Application– Server isolations– Oracle instances



18

Other NFT tests

• Rapid user actions tests (connect-disconnect, logon-logout)– System should sustain against such user behavior– HW resources consumption should not grow up

• Slow consumer tests– System should handle such users and should has a protection against them– HW resources consumption should not grow up

• Intensive usage of recovery channels–System should be able to handle high number of requests on recovery channels and

should be able satisfy them

• Massive actions from Market Operations (mass order cancels, mass trade cancels, mass instrument halts)– System should handle Market operations’ actions like mass cancel of 10k active orders

or trades



19

Other NFT tests

TQ Special latency test – Main market Reference prices processing delay for TQ dark book:



20

Thank You!


Technology

Exactpro: Non-functional testing approach