Operating Systems - Concurrency

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Operating SystemsCMPSCI 377

Concurrency PatternsEmery Berger

University of Massachusetts Amherst

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Finishing Up From Last Time

Avoiding deadlock: is this ok?

2

lock (a);

lock (b);

unlock (b);

unlock (a);

lock (b);

lock (a);

unlock (a);

unlock (b);

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Finishing Up From Last Time

Not ok – may deadlock.

Solution: impose canonical order (acyclic)

3

lock (a);

lock (b);

unlock (b);

unlock (a);

lock (b);

lock (a);

unlock (a);

unlock (b);

lock (a);

lock (b);

unlock (b);

unlock (a);

lock (a);

lock (b);

unlock (b);

unlock (a);

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Motivating Example: Web Server

Client (browser)

Requests HTML, images

Server

Caches requests

Sends to client http://server/Easter-bunny/200x100/75.jpg

not found

client

webserver

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Possible Implementation

while (true) {

wait for connection;

read from socket & parse URL;

look up URL contents in cache;

if (!in cache) {

fetch from disk / execute CGI;

put in cache;

}

send data to client;

}

5

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Possible Implementation

while (true) {

wait for connection; // net

read from socket & parse URL; // cpu

look up URL contents in cache; // cpu

if (!in cache) {

fetch from disk / execute CGI;//disk

put in cache; // cpu

}

send data to client; // net

}

6

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Problem: Concurrency

Sequential fine until:

More clients

Bigger server Multicores, multiprocessors

Goals:

Hide latency of net & disk I/O Don’t keep clients waiting

Improve throughput Serve up more pages

clients

webserver

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Building Concurrent Apps

Patterns / Architectures

Thread pools

Producer-consumer

“Bag of tasks”

Worker threads (work stealing)

Goals:

Minimize latency

Maximize parallelism

Keep progs. simple to program & maintain

8

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 9

Thread Pools

Thread invocation & destruction relatively expensive

Instead: use pool of threads

When new task arrives, get thread from pool to work on it; block if pool empty

Faster with many tasks

Limits max threads

( ThreadPoolExecutor class in Java)

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Producer-Consumer

Can get pipeline parallelism:

One thread (producer) does work

E.g., I/O

and hands it off to other thread (consumer)

10

producer consumer

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Producer-Consumer

Can get pipeline parallelism:

One thread (producer) does work

E.g., I/O

and hands it off to other thread (consumer)

11

producer consumer

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Producer-Consumer

Can get pipeline parallelism:

One thread (producer) does work

E.g., I/O

and hands it off to other thread (consumer)

12

producer consumer

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Producer-Consumer

Can get pipeline parallelism:

One thread (producer) does work

E.g., I/O

and hands it off to other thread (consumer)

13

producer consumer

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Producer-Consumer

Can get pipeline parallelism:

One thread (producer) does work

E.g., I/O

and hands it off to other thread (consumer)

14

producer consumer

LinkedBlockingQueueBlocks on put() if full, poll() if empty

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Producer-Consumer Web Server

Use 2 threads: producer & consumer

queue.put(x) and x = queue.poll();

15

while (true) {

do something…

queue.put (x);

}

while (true) {

x = queue.poll();

do something…

}

while (true) {

wait for connection;

read from socket & parse URL;

look up URL contents in cache;

if (!in cache) {

fetch from disk / execute CGI;

put in cache;

}

send data to client;

}

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Producer-Consumer Web Server

Pair of threads – one reads, one writes

16

while (true) {

wait for connection;

read from socket & parse URL;

queue.put (URL);

}

while (true) {

URL = queue.poll();

look up URL contents in cache;

if (!in cache) {

fetch from disk / execute CGI;

put in cache;

}

send data to client;

}

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Producer-Consumer Web Server

More parallelism –optimizes common case (cache hit)

17

while (true) {

wait for connection;

read from socket & parse URL;

queue1.put (URL);

}

while (true) {

URL = queue1.poll();

look up URL contents in cache;

if (!in cache) {

queue2.put (URL); return;

}

send data to client;

}

while (true) {

URL = queue2.poll();

fetch from disk / execute CGI;

put in cache;

send data to client;

}

1

2

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

When to Use Producer-Consumer

Works well for pairs of threads

Best if producer & consumer are symmetric

Proceed roughly at same rate

Order of operations matters

Not as good for

Many threads

Order doesn’t matter

Different rates of progress

18

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Producer-Consumer Web Server

Have to be careful to balance load across threads

19

while (true) {

wait for connection;

read from socket & parse URL;

queue1.put (URL);

}

while (true) {

URL = queue1.poll();

look up URL contents in cache;

if (!in cache) {

queue2.put (URL);

}

send data to client;

}

while (true) {

URL = queue2.poll();

fetch from disk / execute CGI;

put in cache;

send data to client;

}

1

2

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Bag of Tasks

Collection of mostly independent tasks

20

worker worker worker worker

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Bag of Tasks

Collection of mostly independent tasks

21

worker worker worker worker

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Bag of Tasks

Collection of mostly independent tasks

22

worker worker worker worker

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Bag of Tasks

Collection of mostly independent tasks

23

worker worker worker worker

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Bag of Tasks

Collection of mostly independent tasks

24

worker worker worker worker

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Bag of Tasks

Collection of mostly independent tasks

25

worker worker worker worker

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Bag of Tasks

Collection of mostly independent tasks

Bag could also be LinkedBlockingQueue(put, poll)

26

worker worker worker worker

addWork

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Bag of Tasks Web Server

Re-structure this into bag of tasks:

addWork & worker threads

t = bag.poll() or bag.put(t)

27

while (true) {

wait for connection;

read from socket & parse URL;

look up URL contents in cache;

if (!in cache) {

fetch from disk / execute CGI;

put in cache;

}

send data to client;

}

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Bag of Tasks Web Server

Re-structure this into bag of tasks:

addWork & worker

t = bag.poll() or bag.put(t)

28

addWork:

while (true) {

wait for connection;

bag.put (URL);

}

Worker:

while (true) {

URL = bag.poll();

look up URL contents in cache;

if (!in cache) {

fetch from disk / execute CGI;

put in cache;

}

send data to client;

}

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Bag of Tasks Web Server

Re-structure this into bag of tasks:

t = bag.poll() or bag.put(t)

29

worker worker

addWork: while (true){

wait for connection;

bag.put (URL);

}

worker: while (true) {

URL = bag.poll();

look up URL contents in cache;

if (!in cache) {

fetch from disk / execute CGI;

put in cache;

}

send data to client;

}

worker

addWork

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Bag of Tasks vs. Prod/Consumer

Exploits more parallelism

Even with coarse-grained threads

Don’t have to break up tasks too finely

Easy to change or add new functionality

But: one major performance problem…

30

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

What’s the Problem?

31

worker worker worker worker

addWork

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

What’s the Problem?

Contention – single lock on structure

Bottleneck to scalability

32

worker worker worker worker

addWork

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Work Queues

33

executor

Each thread has own work queue (deque)

No single point of contention

Threads now generic “executors”

Tasks (balls): blue = parse, yellow = connect…

executor executor executor

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

executor executor executor executor

Work Queues

34

Each thread has own work queue (deque)

No single point of contention

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Work Queues

35

Each thread has own work queue (deque)

No single point of contention

executor executor executor executor

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

executor executor executor executor

Work Queues

36

Each thread has own work queue (deque)

No single point of contention

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Work Queues

37

Each thread has own work queue

No single point of contention

Now what?

executor executor executor executor

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Work Stealing

38

worker

When thread runs out of work,steal work from random other thread

worker worker worker

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Work Stealing

39

worker

When thread runs out of work,steal work from top of random deque

Optimal load balancing algorithm

worker worker worker

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Work Stealing Web Server

Re-structure:readURL, lookUp, addToCache, output

myQueue.put(new readURL (url))

40

while (true) {

wait for connection;

read from socket & parse URL;

look up URL contents in cache;

if (!in cache) {

fetch from disk / execute CGI;

put in cache;

}

send data to client;

}

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Work Stealing Web Server

Re-structure:readURL, lookUp, addToCache, output

myQueue.put(new readURL (url))

41

readURL(url) {

wait for connection;

read from socket & parse URL;

myQueue.put (new lookUp (URL));

}

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Work Stealing Web Server

Re-structure:readURL, lookUp, addToCache, output

myQueue.put(new readURL (url))

42

readURL(url) {

wait for connection;

read from socket & parse URL;

myQueue.put (new lookUp (URL));

}

lookUp(url) {

look up URL contents in cache;

if (!in cache) {

myQueue.put (new addToCache (URL));

} else {

myQueue.put (new output(contents));

}

}addToCache(URL) {

fetch from disk / execute CGI;

put in cache;

myQueue.put (new output(contents));

}

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

Work Stealing

Works great for heterogeneous tasks

Convert addWork and worker into units of work (different colors)

Flexible: can easily re-define tasks

Coarse, fine-grained, anything in-between

Automatic load balancing

Separates thread logic from functionality

Popular model for structuring servers

43

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

The End

44