© T

ieto

Cor

pora

tion

Pub

lic

Shared Database Concurrency

Aivars Kalvāns

Lead Software ArchitectTietoaivars.kalvans@tieto.com

© Tieto Corporation

Pub

lic

2

Who am I• Working at “Tieto Cards” > 11 years

• Junior developer → Lead Software Architect• C and C++, Python• Oracle database

• C++ library for working with Oracle OCI• MongoDB for Developers certificate

2013-09-09

© Tieto Corporation

Pub

lic

3

About Tieto Cards• Payment card systems

• Both on-line and batch processing• Linux, AIX, HP-UX, Solaris, Windows• Oracle Tuxedo middleware (C++)• Oracle database (PL/SQL, PRO*C, OCI)

2013-09-09

© Tieto Corporation

Pub

lic

4

Largest customers• 31,000,000 cards• 1,000,000,000 financial transactions/year• 160,000 POS terminals• 3,700 ATM

• 250 financial transactions/second on-line• More for batch processing: 24h data within 8h• Thousands in case of spherical cards in a vacuum

2013-09-09

© Tieto Corporation

Pub

lic

5

Terms• Concurrency• Consistency• Locks• Blocking

2013-09-09

© Tieto Corporation

Pub

lic

6

Shared memory concurrencyConcurrent components communicate by altering the contents of shared memory locations

• Threads (C++, Java, C#, …)• Processes (Oracle on Unix)• Popular topic at developer conferences

• Lock-free• Actor model, …

2013-09-09

© Tieto Corporation

Pub

lic

7

Shared database concurrencyConcurrent components communicate by altering the contents of shared memory locations

Concurrent components communicate by altering shared data in a database

2013-09-09

© Tieto Corporation

Pub

lic

8

Why is it important• Concurrency in application server →

Concurrency in database server• Database is the bottleneck• Mixing functionality of batch and online processing• Design for concurrency

• Hard to fix afterwards

2013-09-09

© Tieto Corporation

Pub

lic

9

Concurrently shared data• Accounts and limits

• Card• Merchant• Bank

• Real-time statistics for fraud detection• Country• Merchant category (hotel, casino, escort service)

2013-09-09

© Tieto Corporation

Pub

lic

10

Locking in a database• Pessimistic locking and Optimistic locking• http://www.orafaq.com/papers/locking.pdf

More challenging than the technology is overcoming resistance from seasoned development professionals who have been using the trusted SELECT… FOR UPDATE for all of their Oracle careers.These individuals may need to be convinced of the benefits of using optimistic and on large development projects their support will be crucial.

2013-09-09

© Tieto Corporation

Pub

lic

11

What has been tried• Use the best practices, architecture, patterns for the

application

• Leave database concurrency to optimistic locking

2013-09-09

© Tieto Corporation

Pub

lic

12

• Does not work as expected*• Acceptable until x transactions per second• System gets slower when concurrency increases

• Optimistic = hope nobody modifies the same data

* for our use case, your experience may differ

2013-09-09

© Tieto Corporation

Pub

lic

13

Oracle 101• Row-level (TX) locks• LOCK TABLE …• Locks are held until COMMIT or ROLLBACK

• Not in case of ROLLBACK TO SAVEPOINT

• Writes don’t block reads• Reads don’t block writes

2013-09-09

© Tieto Corporation

Pub

lic

14

Placing a row-level lock• UPDATE, DELETE, SELECT … FOR UDATE [NOWAIT]• INSERT

• Primary key or unique constraint violation

• Every modification implies locking• To guarantee ACID properties• Let’s call it “implicit locking”

2013-09-09

© Tieto Corporation

Pub

lic

15

Few years later…

2013-09-09

© Tieto Corporation

Pub

lic

16

Pessimistic vs. OptimisticTime Pessimistic Optimistic

t1 SELECT …FOR UPDATE

SELECT version, …

t2 Modify data Modify data

t3 UPDATE … UPDATESET version = :next_versionWHERE version = :known_version

t4 Retry if ROWCOUNT=0

t5 COMMIT COMMIT

2013-09-09

LOC

K

LOC

K

© Tieto Corporation

Pub

lic

17

“Implicit” vs. OptimisticTime “Implicit” Optimistic

t1 SELECT … SELECT version, …

t2 Modify data Modify data

t3 UPDATE … UPDATESET version = :next_versionWHERE version = :known_version

t4 Retry if ROWCOUNT=0

t5 COMMIT COMMIT

2013-09-09

LOC

K

LOC

K

• Additional consistency control!

© Tieto Corporation

Pub

lic

18

When to control consistency?There are 10 kinds of updates

2013-09-09

Don’t need it Need consistency

Relative update

balance = balance + :deposit balance = balance - :withdraw

Absolute update

name = :new_name balance = :new_balance

© Tieto Corporation

Pub

lic

19

Conclusions about locking• If you don’t have concurrent updates on the same data

• Does not matter which locking you use• Every modification places row-level locks

• If you do have concurrent updates on the same data• There is nothing “optimistic” about optimistic locking in Oracle

database• Pessimistic is better• Optimistic locking burns more cycles due to retries

2013-09-09

© Tieto Corporation

Pub

lic

20

How to choose locking• “Implicit locking” by default

• Relative updates• Last update wins for absolute updates

• Pessimistic locking• To prevent concurrency

• Optimistic locking• Stateless applications• “Conditional consistency”

2013-09-09

© Tieto Corporation

Pub

lic

21

How we have chosen locking• Cards – pessimistic locking

• Historic reasons, unlikely concurrent updates• Statistics – “implicit locking”• Accounts

• Optimistic locking for “conditional consistency”• Ensure consistency sometimes

• for withdrawals and complex interest calculations• “Implicit locking” otherwise

• …and few more tricks

2013-09-09

© Tieto Corporation

Pub

lic

22

How to increase concurrencySame as with shared memory concurrency• Reduce the time locks are held• Reduce lock granularity

Not an option• Lock-less …• NoSQL

2013-09-09

© Tieto Corporation

Pub

lic

23

Reduce (b)locking time• Do as little work as possible between locking DML and

COMMIT• Reorder to perform locking DML last

• UPDATE, INSERT, COMMITvs.• INSERT, UPDATE, COMMIT• Up to 50% improvement

• Not all DML have equal possibility of blocking• Update card account – 10 times / month• Update bank account – 10 times / second

2013-09-09

© Tieto Corporation

Pub

lic

24

Increase performance• Bulk operations

• Multiple rows with one DML• SELECT … WHERE id IN (:id1, :id2, :id3)

• ROWID within transaction boundaries• Physical address of the row

• Index maps values to ROWIDs• Except:

• Index-Organized Tables• Table management (shrink, flashback, partition key change)

• SELECT ROWID, …• UPDATE / DELETE … WHERE ROWID = :rowid

2013-09-09

© Tieto Corporation

Pub

lic

25

Increase performance• Faster servers

• Faster CPU (memory, disk) → shorter locking time• Few fast CPUs are better than many slow CPUs

• Locking time can be reduced by faster not more processes

2013-09-09

© Tieto Corporation

Pub

lic

26

Do more with less DML• INSERT … RETURNING … INTO …

• UPDATE and DELETE as well• INSERT ALL … / INSERT WHEN …• MERGE …

• USING (SELECT … FROM DUAL)

2013-09-09

© Tieto Corporation

Pub

lic

27

Release locks faster• Avoid distributed transactions

• Often too easy to use• Two-phase commit: prepare and commit• Extra step and coordination overhead before locks are released

• COMMIT on DML success• DML+COMMIT in single round-trip• INSERT, COMMIT vs. INSERT+COMMIT – 10% improvement• Up to 50% improvement in case of blocking

2013-09-09

Processes TPS / Distributed TPS / Local

4 836 1070 (+28%)

8 972 1260 (+29%)

© Tieto Corporation

Pub

lic

28

Reduce lock granularity• “Partition” rows

• Multiple rows instead of one• Update row based on random or hash• Aggregate rows when reading

2013-09-09

id balance

13 42

id balance partition

13 15 0

13 7 1

13 11 2

13 9 3

© Tieto Corporation

Pub

lic

29

Reduce lock granularity• INSERT instead of UPDATE

• Create a change journal• INSERT offsets• Aggregate and UPDATE once in a while

2013-09-09

© T

ieto

Cor

pora

tion

Pub

lic

Thank youand

a happy programmers’ day!

2013-09-0930

© T

ieto

Cor

pora

tion

Pub

lic

Aivars Kalvāns

Lead Software ArchitectTietoaivars.kalvans@tieto.com