Tech Talks_04.07.15_Session 2_Danail Branekov_Avoiding And Diagnosing Deadlocks In Java Multithreaded Applications

Avoiding and Diagnosing

DEADLOCKS

Danail Branekov

July 4, 2015

2CONFIDENTIAL

CREDITS

Dr Heinz Kabutz

• Born in Cape Town, South Africa, now lives in Greece

• Has a PhD in Computer Science

• Created the Java Specialists’ Newsletter

• One of the first Sun Java Champions

• Carries out Java training courses all over the world

• Regular speaker at all the major Java conferences

This presentation is based on materials from Heinz’ Extreme Java – Concurrency and

Performance training with his explicit permission

http://www.javaspecialists.eu/archive/archive.html

https://java-champions.java.net/

http://www.javaspecialists.eu/courses/xj-conc-j8.jsp

3CONFIDENTIAL

FROM THE CLASSIC

• Classic problem is that of the dining philosophers which we change to drinking

philosophers

• This is where the word symposium comes from:

• sym – together, such as in symphony

• poto - drink

• Ancient Greek philosophers used to get together to drink & think

• In our example a philosopher needs two glasses to drink

• First he takes the right one, then the left one

• When he finishes drinking he returns them and carries on thinking

4CONFIDENTIAL

WORST CASE SCENARIO

Table is ready, all philosophers are thinking

1

5 2

4 3

5CONFIDENTIAL

WORST CASE SCENARIO

Philosopher 5 wants to drink, takes right cup

1

5 2

4 3

6CONFIDENTIAL

WORST CASE SCENARIO


5 2

4 3

1

7CONFIDENTIAL

2

WORST CASE SCENARIO


5

4 3

1

8CONFIDENTIAL

3

2

WORST CASE SCENARIO


5

4

1

9CONFIDENTIAL

4 3

2

WORST CASE SCENARIO


5

1

10CONFIDENTIAL

4 3

2

WORST CASE SCENARIO

DEADLOCK

• All philosophers are waiting

for their left cups but they

will never become available

5

1

11CONFIDENTIAL

GLOBAL ORDER WITH BOOZING

PHILOSOPHERS

We can solve the deadlock with the drinking philosophers by requiring that locks are

always acquired in a set order

• For example, we can make a rule that philosophers always first take the cup with the

largest number

• And return the cup with the lowest number first

12CONFIDENTIAL

GLOBAL LOCK ORDERING

Table is ready, all philosophers are thinking

1

5 2

4 3

1 2

3

4

5

13CONFIDENTIAL

5


Philosopher 5 takes cup 5

• remember our rule!1

2

4 3

1 2

3

4

5

14CONFIDENTIAL

1

5



• remember our rule!

2

4 3

1

3

4

5

2

15CONFIDENTIAL

2

1

5



4 3

1

4

5

2

3

16CONFIDENTIAL

3

2

1

5



4

1

5

2

3

4

17CONFIDENTIAL

3

2

1

5


Philosopher 1 takes cup 1 - Drinking

4

5

2

3

4

1

18CONFIDENTIAL

1

3

25


Philosopher 1 finished drinking, returns cup 1

• Cups are returned in the opposite

order to what they are

acquired

4

5

2

3

4

1

19CONFIDENTIAL

1

3

25



4

5

2

3

4

1

20CONFIDENTIAL

5

1

3

2


Philosopher 5 returns cup 1

4

5

2

3

4

1

21CONFIDENTIAL

5

1

3

2



4

5

2

3

4

1

22CONFIDENTIAL

5

1

3

2



4

5

2

3

4

1

23CONFIDENTIAL

5

1

3

2



4

5

2

3

4

1

24CONFIDENTIAL

4

5

1

3

2



5

2

3

4

1

25CONFIDENTIAL

2

4

5

1

3



5

2

3

4

1

26CONFIDENTIAL

2

4

5

1

3



5

2

3

4

1

27CONFIDENTIAL

2

4

5

1

3



5

2

3

4

1

28CONFIDENTIAL

3

2

4

5

1



5

2

3

4

1

29CONFIDENTIAL

3

2

4

5

1



5

2

3

4

1

30CONFIDENTIAL

3

2

4

5

1



5

2

3

4

1

31CONFIDENTIAL

4 3

25

1



5

2

3

4

1

32CONFIDENTIAL

4 3

25

1



5

2

3

4

1

33CONFIDENTIAL

LOCK ORDERING AS JAVA CODE

public class Symposium {

private final Cup[] cups;

private final Thinker[] thinkers;

public Symposium(int delegates) {

cups = new Cup[delegates];

thinkers = new Thinker[delegates];

for (int i = 0; i < cups.length; i++) {

cups[i] = new Cup();

}

for (int i = 0; i < delegates; i++) {

Cup right = cups[i];

Cup left = cups[(i + 1) % delegates];

thinkers[i] = new Thinker(i, left, right);

}

}

34CONFIDENTIAL


public class Symposium {

. . .

public ThinkerStatus run() throws InterruptedException {

ExecutorService exec = Executors.newCachedThreadPool();

for (Thinker thinker : thinkers) {

exec.submit(thinker);

}

. . .

}

}

35CONFIDENTIAL


public class Thinker implements Callable<ThinkerStatus> {

private final int id;

private final Cup smaller, bigger;

private int drinks = 0;

public Thinker(int id, Cup right, Cup left) {

this.id = id;

this.bigger = right.compareTo(left) > 0 ?

right : left;

this.smaller = this.bigger == right ?

left : right;}

public ThinkerStatus call() throws Exception {

for (int i = 0; i < 1000; i++) {drink(); think();}return drinks == 1000 ? ThinkerStatus.HAPPY_THINKER :

ThinkerStatus.UNHAPPY_THINKER;

}

. . .

public void think() { . . . }

}

36CONFIDENTIAL



private final Cup smaller, bigger;

. . .

public Thinker(int id, Cup right, Cup left) {

this.bigger = right.compareTo(left) > 0 ?

right : left;

this.smaller = this.bigger == right ?

left : right;

}

public void drink() {

synchronized (bigger) {

synchronized (smaller) {

drinking();

}

}

}

private void drinking() { drinks++; }

}

37CONFIDENTIAL

LOCK ORDERING AS JAVA CODE (VER 2)


private static final Object TIE_LOCK = new Object();

. . .

public void drink() {

long leftHashCode = System.identityHashCode(left);

long rightHashCode = System.identityHashCode(right);

if (leftHashCode > rightHashCode) {

synchronized (left) {

synchronized (right) { drinking(); }

}

} else if (leftHashCode < rightHashCode) {

synchronized (right) {

synchronized (left) { drinking(); }

}

} else {

synchronized (TIE_LOCK) {

synchronized (right) {

synchronized (left) { drinking(); }

}

}

}

38CONFIDENTIAL

DEADLOCKING COOPERATING OBJECTS

In this example, the deadlock is more subtle

• Taxi is an individual taxi with a location and destination

• Dispatcher represents a fleet of taxis, capable of providing an image representing the

current location of taxis

Spot the deadlock

39CONFIDENTIAL

TAXI, REPRESENTING AN INDIVIDUAL VEHICLE

public class Taxi {

private Point location, destination;

private Dispatcher dispatcher;

public void setDispatcher(Dispatcher dispatcher) {

this.dispatcher = dispatcher;

}

public void setDestination(Point destination) {

this.destination = destination;

}

public synchronized Point getLocation() {

return location;

}

public synchronized void setLocation(Point location) {

this.location = location;

if (location.equals(destination))

dispatcher.notifyAvailable(this);

}

}

}

40CONFIDENTIAL

DISPATCHER, MANAGING A FLEET OF TAXIS

public class Dispatcher {

private final Set<Taxi> taxis;

private final Set<Taxi> availableTaxis;

public Dispatcher(Taxi... taxis) {

this.taxis = new HashSet<Taxi>(Arrays.asList(taxis));

this.availableTaxis = new

HashSet<Taxi>(Arrays.asList(taxis));

}

public synchronized void notifyAvailable(Taxi taxi) {

availableTaxis.add(taxi);

}

public synchronized Image getImage() {

Image image = new Image();

taxis.forEach(taxi -> image.drawMarker(taxi.getLocation()));

return image;

}

}

41CONFIDENTIAL

HOW TO DEADLOCK THE TAXI INDUSTRY

AsetLocation()

synchronized(taxi)

dispatcher.notifyAvailable(taxi)

synchronized(dispatcher)

BgetImage()


taxi.getLocation()

synchronized(taxi)

BLOCKED

BLOCKED

42CONFIDENTIAL

OPEN CALLS

• Calling an alien method with a lock held is difficult to analyze and therefore risky

• Both Taxi and Dispatcher violate this rule

• Calling a method with no locks held is called an open call

• Makes it much easier to reason about liveness

43CONFIDENTIAL

REFACTORED Taxi.setLocation()

• We should not call alien methods whilst holding locks

• Here we split the method up into parts that need the lock and those that call alien

methods

public void setLocation(Point location) {

boolean reachedDestination;

synchronized (this) {

this.location = location;

reachedDestination = location.equals(destination);

}

if (reachedDestination) {

dispatcher.notifyAvailable(this);

}

}

44CONFIDENTIAL

REFACTORED Dispatcher.getImage()

• We make a copy of the taxis set to prevent race conditions

public Image getImage() {

Set<Taxi> copy;


copy = new HashSet<>(taxis);

}

Image image = new Image();

copy.forEach(taxi -> image.drawMarker(taxi.getLocation()));

return image;

}

45CONFIDENTIAL

DEADLOCK SOLVED

AsetLocation()

synchronized(taxi)



BgetImage()


taxi.getLocation()

synchronized(taxi)

46CONFIDENTIAL

ANOTHER SOLUTION

• An alternative solution would be to make the Taxi.location field volatile which allows

making the getLocation method not synchronized

public class Taxi {

private volatile Point location;

. . .

public Point getLocation() {

return location;

}

}

AsetLocation()

synchronized(taxi)



BgetImage()


taxi.getLocation()

synchronized(taxi)

47CONFIDENTIAL

BENEFITS OF OPEN CALLS

• Strive to use open calls throughout your program

• Programs that rely on open calls are far easier to analyze for deadlock-freedom than

those that allow calls to alien methods with locks held

• Alien method calls with lock held are probably the biggest cause of deadlocks in the

field

48CONFIDENTIAL

OPEN CALL IN Vector

• In Sun Java 6 Vector.writeObject() is synchronized

private synchronized void writeObject(java.io.ObjectOutputStream s)

throws java.io.IOException {

s.defaultWriteObject();

}

• The reason to have the method synchronized is to provide thread safety during writing

• However, since it calls the alien s.defaultWriteObject() it can deadlock

• http://www.javaspecialists.eu/archive/Issue184.html

http://www.javaspecialists.eu/archive/Issue184.html

49CONFIDENTIAL

OPEN CALL IN Vector

IBM avoids this problem with an open call

private void writeObject(ObjectOutputStream stream) throws IOException {

Vector<E> cloned = null;

// this specially fix is for a special dead-lock in customer

// program: two vectors refer each other may meet dead-lock in

// synchronized serialization. Refer CMVC-103316.1


try {

cloned = (Vector<E>) super.clone();

cloned.elementData = elementData.clone();

} catch (CloneNotSupportedException e) {

// no deep clone, ignore the exception

}

}

cloned.writeObjectImpl(stream);

}

private void writeObjectImpl(ObjectOutputStream stream) throws IOException {

stream.defaultWriteObject();}

50CONFIDENTIAL

OPEN CALL IN Vector

OpenJDK 7 also uses an open call

private void writeObject(ObjectOutputStream s) throws IOException {

final ObjectOutputStream.PutField fields = s.putFields();

final Object[] data;


fields.put("capacityIncrement", capacityIncrement);

fields.put("elementCount", elementCount);

data = elementData.clone();

}

fields.put("elementData", data);

s.writeFields();

}

51CONFIDENTIAL

RESOURCE DEADLOCKS

• We can also cause deadlocks waiting for resources

• For example, you may have two DB connection pools (D1 and D2)

• Some tasks might require connection to both databases

• Thus thread A might hold a semaphore for D1 and wait for D2, while thread B

might hold a semaphore for D1 and be waiting for D2

public class DatabasePool {

private final Semaphore connections;

public DatabasePool(int connections) {

this.connections = new Semaphore(connections);

}

public void connect() {

connections.acquireUninterruptibly();System.out.println("connect");

}

public void disconnect() {

System.out.println("disconnect");connections.release();

}

}

• NOTE: Thread dump and ThreadMXBean do not show this as deadlock!

52CONFIDENTIAL

ANTI-DEADLOCK BEST PRACTICES

• If you only ever acquire one lock, you cannot get a lock-ordering lock

• This is the easiest way to avoid deadlocks but it is not always practical

• If you need to acquire multiple locks, include lock ordering in your design

• It is important to specify and document possible lock sequences

• Identify where multiple locks could be acquired

• Do a global analysis to ensure that lock ordering is consistent (could be

extremely difficult on large programs)

• Use open calls when possible

• Do not call alien methods whilst holding a lock

53CONFIDENTIAL

UNIT TESTING FOR LOCK ORDERING

DEADLOCKS

• Code typically has to be called many times before a deadlock occurs

• How many times do you need to call it to prove that there is no deadlock?

• Nondeterministic unit tests are bad, a unit test should either always fail, or always

pass

54CONFIDENTIAL


DEADLOCKS

Consider the following example

public class Bank {

public boolean transferMoney(Account from,

Account to, DollarAmount amount) {

synchronized (from) {

synchronized (to) {

return doActualTransfer(from, to, amount);

}

}

}

}

In the transferMoney method a deadlock occurs if after the first lock (from) is granted,

the first thread is swapped out and another thread requests the second lock (to)

55CONFIDENTIAL


DEADLOCKS

• We can force the deadlock by sleeping a short while after requesting the first lock

public class Bank {

public boolean transferMoney(Account from,

Account to, DollarAmount amount) {

synchronized (from) {

sleepAWhileForTesting();

synchronized (to) {

return doActualTransfer(from, to, amount);

}

}

}

protected void sleepAWhileForTesting() {}

}

• The empty sleepAWhileForTesting method will be optimized away by the HotSpot

compiler

56CONFIDENTIAL


DEADLOCKS

• In our unit test we extend the class and use it instead

public class SlowBank extends Bank {

private final long timeout;

private final TimeUnit unit;

public SlowBank(long timeout, TimeUnit unit) {

this.timeout = timeout; this.unit = unit;

}

@Override

protected void sleepAWhileForTesting() {

try {

unit.sleep(timeout);

} catch (InterruptedException e) { . . . }

}

}

• We can then implement a parallel task which checks for thread deadlocks and fail the

test in case of one

57CONFIDENTIAL

FINDING DEADLOCKED THREADS

• ThreadMXBean has two methods for finding deadlocks

• findMonitorDeadlockedThreads()

• Include only “monitor” locks, i.e. synchronized

• The only way to find deadlock in Java 5

• findDeadlockedThreads()

• Include “monitor” and “owned” (Java 5) locks

• Preferred method to test for deadlocks

• Does not find deadlocks between semaphores

• Both methods are designed for troubleshooting use but not for synchronization control

• Finding deadlocked threads might be an expensive operation

• http://www.javaspecialists.eu/archive/Issue130.html

http://www.javaspecialists.eu/archive/Issue130.html

58CONFIDENTIAL

FINDING DEADLOCKED THREADS

private static final ThreadMXBean tmb =

ManagementFactory.getThreadMXBean();

public Collection<Thread> findDeadlockedThreads() {

long[] ids = tmb.findDeadlockedThreads();

if (ids == null) { return Collections.emptyList();}

return findThreadsByIds(ids);

}

. . .

private Thread findThreadById(long threadId) {

for (Thread thread : Thread.getAllStackTraces().keySet()) {

if (thread.getId() == threadId)

return thread;

}

return null;

}

59CONFIDENTIAL

DEADLOCK ANALYSIS WITH THREAD DUMPS

• We can also analyze deadlocks via looking at the thread dump

• Thread dump can be caused in many ways:

• Ctrl-Break on Windows or Ctrl-\ on Unix

• Invoking kill -3 on the process id

• Calling jstack on the process id (only shows deadlocks since Java 6)

• It is useful to have unique thread names

60CONFIDENTIAL


Found one Java-level deadlock:

=============================

"pool-1-thread-2":

waiting to lock monitor 0x0000000002ae5bb8 (object 0x000000076b814b08, a

taxi.deadlock.Dispatcher),

which is held by "pool-1-thread-1"

"pool-1-thread-1":

waiting to lock monitor 0x000000001e08c458 (object 0x000000076b812b88, a

taxi.deadlock.Taxi),

which is held by "pool-1-thread-2"

61CONFIDENTIAL


Java stack information for the threads listed above:

===================================================

"pool-1-thread-2":

at taxi.deadlock.Dispatcher.notifyAvailable(Dispatcher.java:17)

- waiting to lock <0x000000076b814b08> (a taxi.deadlock.Dispatcher)

at taxi.deadlock.Taxi.setLocation(Taxi.java:30)

- locked <0x000000076b812b88> (a taxi.deadlock.Taxi)

at taxi.deadlock.TaxiDriver.run(TaxiDriver.java:25)

...

"pool-1-thread-1":

at taxi.deadlock.Taxi.getLocation(Taxi.java:19)

- waiting to lock <0x000000076b812b88> (a taxi.deadlock.Taxi)

at taxi.deadlock.Dispatcher.lambda$0(Dispatcher.java:22)

at taxi.deadlock.Dispatcher$$Lambda$1/710209934.accept(Unknown Source)

at java.lang.Iterable.forEach(Iterable.java:75)

at taxi.deadlock.Dispatcher.getImage(Dispatcher.java:22)

- locked <0x000000076b814b08> (a taxi.deadlock.Dispatcher)

at taxi.deadlock.TaxiObserver.run(TaxiObserver.java:14)

...

Found 1 deadlock.

62CONFIDENTIAL


ThreadMXBean (and JConsole) do not detect lock order deadlocks caused by acquiring

semaphoresSemaphore s1 = new Semaphore(1);

Semaphore s2 = new Semaphore(1);

ExecutorService execService = Executors.newCachedThreadPool();

execService.submit(new TestTask(s1, s2));

execService.submit(new TestTask(s2, s1));

static class TestTask implements Runnable {

public TestTask(Semaphore first, Semaphore second) {

this.first = first;

this.second = second;

}

public void run() {

try {

first.acquire();

second.acquire();

. . .

second.release();

first.release();


}

}

63CONFIDENTIAL


ThreadMXBean (and JConsole) do not detect lock order deadlocks caused by acquiring

semaphores

"pool-1-thread-1" #15 prio=5 os_prio=0 tid=0x000000001ddb2800 nid=0x1f78 waiting on

condition [0x000000001e69e000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for <0x000000076b9637a8> (a

java.util.concurrent.Semaphore$NonfairSync)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

at

java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Abstract

QueuedSynchronizer.java:836)

. . .

64CONFIDENTIAL

DEADLOCK RECOVERY

Can I recover from a deadlock?

• DISCLAIMER: Always prefer deadlock prevention via implementation best practices

over recovering from deadlocks which already took place. Recovering from deadlocks

should be only used as an extreme measure

• NO, if you deadlocked with synchronized

• synchronized goes into BLOCKED state

• YES, with ReentrantLock

• ReentrantLock.lock() and ReentrantLock.lockInterruptibly() go into WAITING

state

• Deadlocked threads must make sure that they release locks held when

interrupted or stopped

• We can implement a DeadlockArbitrator which interrupts or stops one of the

threads

65CONFIDENTIAL

DEADLOCK RECOVERY VIA INTERRUPT

ReentrantLock first = new ReentrantLock();

ReentrantLock second = new ReentrantLock();

private class TestTask implements Runnable {

public void run() {

try {

first.lockInterruptibly();try {

// do some work

second.lockInterruptibly();try {

// do more work

} finally {

second.unlock();

}

} finally {

first.unlock();

}


}

66CONFIDENTIAL

DEADLOCK RECOVERY VIA INTERRUPT

public class ThreadInterruptingDeadlockArbitrator {

public void tryResolveDeadlock() throws InterruptedException {

Thread t = findDeadlockedThread(); // via ThreadMXBean

if (t != null) {

t.interrupt();

}

}

}

67CONFIDENTIAL

DEADLOCK RECOVERY VIA STOP

ReentrantLock first = new ReentrantLock();

ReentrantLock second = new ReentrantLock();

private class TestTask implements Runnable {

public void run() {

first.lock();try {

// do some work

second.lock();try {

// do more work

} finally {

second.unlock();

}

} finally {

first.unlock();

}

}

}

68CONFIDENTIAL

DEADLOCK RECOVERY VIA STOP

public class ThreadInterruptingDeadlockArbitrator {

public void tryResolveDeadlock() throws InterruptedException {

Thread t = findDeadlockedThread(); // via ThreadMXBean

if (t != null) {

t.stop();

}

}

}

69CONFIDENTIAL

LINKS

• Java Concurrency in Practice by Brian Goetz

• The Java Specialists’ Newsletter

• Sample source code

http://www.amazon.com/Java-Concurrency-Practice-Brian-Goetz/dp/0321349601/ref=sr_1_1?s=books&ie=UTF8&qid=1435570055&sr=1-1&keywords=java+concurrency+in+practice

http://www.javaspecialists.eu/archive/

https://github.com/danail-branekov/java-deadlocks-sample

70CONFIDENTIAL

THANK YOU!

Documents

Tech Talks_04.07.15_Session 2_Danail Branekov_Avoiding And Diagnosing Deadlocks In Java Multithreaded Applications