Spring Batch - Lessons Learned out of a real life banking system

2013 © Trivadis

BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA

2013 © Trivadis

Spring Batch Lessons Learned out of a real life banking system.

Michael Beer

Senior Consultant

Raffael Schmid

Consultant

December 3rd, 2013

Lessons Learned out of a real life banking system

1

2013 © Trivadis

About us

Michael Beer

Senior Consultant

December 3rd, 2013


2

Trivadis since 2001

design and development of

web based applications

part of the Trivadis APM

team

Trivadis since 2010

design and development of

web based applications

interested in performance

related things on the JVM

Raffael Schmid

Consultant

2013 © Trivadis

AGENDA

December 3rd, 2013


3

Initial

position

Requirements

and Topics

Lessons

learned Conclusion

2013 © Trivadis

Initial position

December 3rd, 2013


4

Initial

position

Requirements

and Topics

Lessons

learned Conclusion

2013 © Trivadis

common usage

Initial and target system context

Automatic load out of thirdparty

systems

multiple times per day

export / import as CSV file

Manual load by user

at irregular time intervals

Migration load due to thirdparty

system decommissioning

run once

different volumes

December 3rd, 2013


5

Web Interface

Rule Engine

File Interface

Thirdparty system

Thirdparty system

2013 © Trivadis

Why we used Spring Batch?

It was already part of the technology stack within the customers

environment.

There were no real (free) alternatives at that time (or at least we didn‘t

know any).

Exceptional permit was needed for Spring Integration because it was

shipped with Spring Batch Admin.

December 3rd, 2013


6

2013 © Trivadis

Terminology of Spring Batch jobs

December 3rd, 2013


7

Job

Job Instance

Job Execution

Step

Step Execution

*

*

*

*

*

Job Execution

Context

Step Execution

Context

* Job Parameter

Tasklet *

2013 © Trivadis

Requirements and Topics

December 3rd, 2013


8

Initial

position

Requirements

and Topics

Lessons

learned Conclusion

2013 © Trivadis

Requirement: Performance

parallelized on multiple threads

order of execution matters

number of threads can be changed before or after job execution

December 3rd, 2013


9

Control load Deactivate jobs Inter-job

dependencies Trigger jobs

Reprocess

failed items Gather detailed

job information Summary mail Performance

2013 © Trivadis

Requirement: Reprocess failed items

rerunning a job processes failed items (only)

process state of item therefore needs to be maintained

December 3rd, 2013


10



Reprocess



2013 © Trivadis

Requirement: Summary mail

write a detailed execution log that contains

Exceptions

messages out of the Rule Engine

aggregate to summary

Number of errors, warnings, etc.

worst status level

send summary to submitter and operator

December 3rd, 2013


11



Reprocess



2013 © Trivadis

Requirement: Gather detailed job information

rerun failed single item in trace mode

collect diagnostic information

e.g. out of the Rule Engine

December 3rd, 2013


12



Reprocess



2013 © Trivadis

Requirement: Trigger jobs

periodically

either fixed delay or fixed rate

on event

e.g. new data arrived in database

December 3rd, 2013


13



Reprocess



2013 © Trivadis

Requirement: Control load

prevent too many jobs in parallel

conditions might prevent job execution

requests should prevent system shutdown

December 3rd, 2013


14



Reprocess



2013 © Trivadis

Requirement: Deactivate jobs

deactivate job execution

set job execution on hold

December 3rd, 2013


15



Reprocess



2013 © Trivadis

Requirement: Inter-job dependencies

finished jobs might trigger dependencies

December 3rd, 2013


16



Reprocess



2013 © Trivadis

Requirements grouped into five different topics

December 3rd, 2013


17



Reprocess



Sourcing

Job Control

Error

Handling Monitoring & Tracing Partitioning

2013 © Trivadis

Lessons learned

December 3rd, 2013


18

Initial

position

Requirements

and Topics

Lessons

learned Conclusion

2013 © Trivadis

Topics we will cover

December 3rd, 2013


19

Error

Handling

Monitoring

& Tracing

Job Control Sourcing Partitioning

depends on

depends on

depends on

firing trigger

supervising

2013 © Trivadis

Sourcing

December 3rd, 2013


20

Error

Handling

Monitoring

& Tracing


depends on

depends on

depends on

firing trigger

supervising

2013 © Trivadis

Load data into staging table

December 3rd, 2013


21

CAT_ID | PERSON_ID | AMOUNT | CURRENCY | VALID_FORM | LAST_CHANGE|

310 | 2908 | 200000 | USD | 03.12.2013 | 02.02.2013 |

310 | 1608 | 100000 | CHF | 03.12.2013 | 02.01.2013 |

311 | 1608 | 100000 | CHF | 03.12.2013 | 02.01.2013 |

PARTITION LINE TYPE STATE FIELD01 FIELD02 FIELD03 FIELD04

PT_000122 171 D 02 441 1804 400000 EUR

PT_000122 172 D 02 441 1002 221000 EUR

PT_000123 1 H 01 CAT_ID PERSON_ID AMOUNT CURRENCY

PT_000123 2 D 01 310 2908 200000 USD

PT_000123 3 D 01 310 1608 100000 CHF

PT_000123 4 D 01 311 1608 100000 CHF

Sourcing

2013 © Trivadis

Allows partitioning, single record execution

Partitioning out of the box

December 3rd, 2013


22

PARTITION LINE TYPE STATE FIELD01 FIELD02 FIELD03 FIELD04

PT_000123 1 H 01 CAT_ID PERSON_ID AMOUNT CURRENCY

PT_000123 2 D 01 310 2908 200000 USD

PT_000123 3 D 01 310 1410 100000 CHF

SELECT * FROM LOAD

WHERE PARTITION =‘PT_000123’

AND LINE BETWEEN 0 AND 99;

SELECT * FROM LOAD


AND STATE = ‘01’

Reexcution of failed records

Execution of single record

SELECT * FROM LOAD


AND LINE = 8

Sourcing

2013 © Trivadis

Job Control

December 3rd, 2013


23

Error

Handling

Monitoring

& Tracing


depends on

depends on

depends on

firing trigger

supervising

2013 © Trivadis

Trigger

Simplest way of triggering a job

December 3rd, 2013


24

0 0/5 * * * ? Spring Batch

(SimpleJobLauncher) job-requests

Job Control

<int:inbound-channel-adapter method="create" channel="job-requests"

ref="requestFactory">

<int:poller cron="0 0/5 * * * ?" />

</int:inbound-channel-adapter>

Request Factory

<int:inbound-channel-adapter method="create" channel="job-requests"

ref="requestFactory">

<int:poller fixed-rate="300000"/>

</int:inbound-channel-adapter>

Or even simpler (PeriodicTrigger)

Define polling channel adapter (CronTrigger)

2013 © Trivadis

Persist job launch requests into database

December 3rd, 2013


25

job launch

request job-

requests

Data import job

(database)

Periodic jobs 0 0/5 * * * ?

*JobQueueMessageHandler job queue

*JobLaunchingMessageHandler

update status

save job dependencies

Guard

verification *JobLauncher

Job Control

persist

2013 © Trivadis

Partitioning

December 3rd, 2013


26

Error

Handling

Monitoring

& Tracing


depends on

depends on

depends on

firing trigger

supervising

2013 © Trivadis

Performance

December 3rd, 2013


27

Type Local/Remote Description

Multi-threaded Step Local A step is multithreaded (TaskExecutor)

Parallel Steps Local Executes steps in parallel using multithreading

Partitioning Step Local

Remote Partitions data and splits up processing

Remote Chunking Remote Distributed chunk processing to remote nodes

Partitioning

2013 © Trivadis

Partitioning overview

December 3rd, 2013


28

Job

Master

Step

Step Slave

Slave

Slave

Slave

Slave

Partitioning

2013 © Trivadis

Partitioning detail

December 3rd, 2013


29

StepScopes

Master

Slave [1]

Slave [2]

Slave [3]

Slave [4]

Slave [0]

Execution

Context [1]

Execution

Context [2]

Execution

Context [3]

Execution

Context [4]

Execution

Context [0]

rowId % 5

Partitioning

2013 © Trivadis

Characteristics of data

There might be dependencies between records. The order of execution

matters at some point.

December 3rd, 2013


30

CAT_ID | PERSON_ID | AMOUNT | CURRENCY | VALID_FORM | LAST_CHANGE|

310 | 2908 | 200000 | USD | 03.12.2013 | 02.02.2013 |

310 | 1608 | 100000 | CHF | 03.12.2013 | 02.01.2013 |

311 | 1608 | 100000 | CHF | 03.12.2013 | 02.01.2013 |

312 | 1608 | 100000 | CHF | 03.12.2013 | 02.01.2013 |

313 | 1608 | 100000 | CHF | 03.12.2013 | 02.01.2013 |

310 | 1410 | 100000 | CHF | 03.12.2013 | 02.01.2013 |

390 | 1108 | 100000 | CHF | 03.12.2013 | 02.01.2013 |

310 | 2908 | 500000 | USD | 04.12.2013 | 03.02.2013 |

Partitioning

Insert

Update

Read Write

Process / Write

2013 © Trivadis

Partitioning detail – Spring Batch Admin

December 3rd, 2013


31

Partitioning

2013 © Trivadis

Performance – Read / Write / Process

December 3rd, 2013


32

Open

Transaction

Begin Step

Commit

Transaction ItemReader

Item read()

ItemProcessor

Process(Item)

ItemWriter

Write(List<Item>)

Finish Step

Rollback

Transaction

RuntimeException RuntimeException RuntimeException

Partitioning

For each item For each item

2013 © Trivadis

Performance

December 3rd, 2013


33

No error Error on each item

• ~ 22 sec

• 0 Rollbacks

• ~ 50 sec

• 1‘100 Rollbacks

• 5‘500 Filter

• 1‘000 Write Skips

Partitioning

2013 © Trivadis

Error Handling

December 3rd, 2013


34

Error

Handling

Monitoring

& Tracing


depends on

depends on

depends on

firing trigger

supervising

2013 © Trivadis

Restartability

December 3rd, 2013


35

Job 1

Instance 1

Execution 1

Job 1

Instance 1

Execution 2

read

new/error read

new/error

Error

Handling

regular start restart

2013 © Trivadis

Restartability configuration

December 3rd, 2013


36

Error

Handling

Job

Step

initialization

Step

process

Step

finalize

initial (failed) execution

Job

Step

initialization

Step

process

Step

finalize

restarted execution

Execution

Context

cleanup

serialized

context

2013 © Trivadis

Item based error handling

December 3rd, 2013


37

Feature When? What? Where?

Skip For nonfatal

exceptions

Keeps processing for an

incorrect item Chunk-oriented step

Retry For transient

exceptions

Makes new attemps on an

operation

Chunk-oriented step,

application code

Error

Handling

2013 © Trivadis

Monitoring

December 3rd, 2013


38

Error

Handling

Monitoring

& Tracing


depends on

depends on

depends on

firing trigger

supervising

2013 © Trivadis

Monitoring – Spring Batch Admin

December 3rd, 2013


39

Monitoring &

Tracing

2013 © Trivadis

Monitoring – logging to a database

December 3rd, 2013


40

Step

Read

Process

Write

LoggerService @Transactional(propagation =

Propagation.REQUIRES_NEW)

2 calls on error

Listener

Before

OnSkip

[WARN]

OnError

[ERROR]

After

[INFO]

Monitoring &

Tracing

2013 © Trivadis

Monitoring - Mail

December 3rd, 2013


41

Step

ReadProcessWrite

Job

FAILED

Error Mail OK Mail

Monitoring &

Tracing

2013 © Trivadis

Tracing – Explain plan

December 3rd, 2013


42

Step

ReadProcessWrite

Partitioner

1 Record

Read

1 Record

Process

Debug mode

Step

…

Writer

does nothing

Operator

explicit job start

Monitoring &

Tracing

2013 © Trivadis

Monitoring / Tracing summary

December 3rd, 2013


43

Mail

OK/ERROR Mail

ERROR

UI

Log

Spring Batch

Admin

UI

Explain plan User Operator

Monitoring &

Tracing

2013 © Trivadis

Conclusion

December 3rd, 2013


44

Initial

position

Requirements

and Topics

Lessons

learned Conclusion

2013 © Trivadis

Challenges – Error Handling

December 3rd, 2013


45

Fire and forget

Commit until first error

- Skip

- Retry

Depending on the characteristic of data

Rollback

- Whole job

- Partition

- Chunk

Job flow

Deciders

Operator takes care about the errors

2013 © Trivadis

Why we still use Spring Batch

Out of 155 SE tests in the JSR-352 TCK, Spring Batch 3.0 (milestone 1)

passes 70.

The terminology stays more or less the same: Job, Step, Chunk, Item,

ItemProcessor, JobInstance, JobExecution.

December 3rd, 2013


46

Spring Batch JSR 352 Comments

Tasklet Batchlet

ItemReader / ItemStream ItemReader JSR-352‘s ItemReader includes Spring Batchs

ItemStream capabilities

ItemWriter / ItemStream ItemWriter JSR-352‘s ItemReader includes Spring Batchs

ItemStream capabilities

JobExecutionListener JobListener

StepExecutionListener StepListener

2013 © Trivadis

Suitable for different applications

It’s very easy to customize existing parts at your needs. Different level of

customization is possible:

December 3rd, 2013


47

Spring Batch supports implementing reliable batch jobs with small effort.

In few cases even no Java code is necessary.

configuration partial

implementation job control

2013 © Trivadis

Questions and answers ...

2013 © Trivadis

BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA

Michael Beer

[email protected]

+41 58 459 51 90

Raffael Schmid

[email protected]

+41 58 459 52 34

December 3rd, 2013


48

Career

Spring Batch - Lessons Learned out of a real life banking system