1

PSS Guest Lecture

Asynchronous IO Programming

20th April 2006

by Henrik Thostrup Jensen

2

Outline

● Asynchronous IO– What, why, when– What about threads

● Programming Asynchronously– Berkeley sockets, select, poll– epoll, KQueue– Posix AIO, libevent, Twisted

3

What is Asynchronous IO

● Also called event­driven or non­blocking● Performing IO without blocking

– Changing blocking operations to be non­blocking– Requires program reorganization

● The brain must follow● The program must never block

– Instead ask what can be done without blocking

4

Why Asynchronous IO

● Concurrency is really difficult– Coordinating threads and locks is non­trivial– Very non­trivial– Don't make things hard

● Threads and locks are not free– They take up resources and require kernel switches

● Often true concurrency is not needed● Asynchronous programs are race free

– By definition

5

Drawbacks

● The program must not block– Only a problem when learning– And with crappy APIs

● Removes “normal” program flow– Due to mixing of IO channels– Requires state machinery or continuations

● Hard to use multiple CPU cores– But doable

6

Program Structure

● One process must handle several sockets– State is not shared between connections– Often solved using state machines– Makes program flow more restricted

● Debugging is different – but not harder– Threads and locks are darn hard to debug

7

Avoiding Asynchronous IO

● Not suitable for everything● When true concurrency is needed

– Very seldom– Utilizing several CPUs with shared data

● Or when badly suited– Long running computations

● Can be split, but bad abstraction

8

Are Threads Evil

● No ­ just misunderstood– Used when inappropriate

● Coordination is the problem– Often goes wrong

● Not free– Spawning a thread for each incoming connection– 500 incoming connections per second => problem– Thread pools used for compensating

9

Setting Things Straight

● Async. IO and threads can do the same– Probably includes performance

● It's about having the best abstraction– Or: Making things easy– Concurrency is not known to simplify things

10

Async. Programming

● First introduced with Berkeley sockets● Then came select, later poll● Recent trends

– epoll, KQueue, Posix AIO, Twisted

11

Berkeley Sockets

● Unsuitable for many sockets– Many kernel switches– The try/except paradigm is ill suited for IO

int s = socket(family, type, protocol);fcntl(s, F_SETFL, O_NONBLOCK);// bind+listen or connect (also async.)void *buffer = malloc(1024);retval = read(s, buffer, 1024);if (retval == -EAGAIN) {

// Reschedule command}

12

Select

● Monitors three sets of file descriptors for changes– Tell which actions can be done without blocking

● pselect() variant to avoid signal race● Has a limit of of maximum file descriptors● Availability: *nix, BSD, Windows

– Reasonably portable

int select(int n, fd_set *readfds, fd_set *writefds, fd_set *exceptfds, struct timeval *timeout);

13

Poll

● Basically a select() with a different API● No limit on file descriptors● Availability: *nix, BSD

int poll(struct pollfd *ufds, unsigned int nfds, int timeout);struct pollfd { int fd; /* file descriptor */ short events; /* requested events */ short revents; /* returned events */};

14

State vs. Stateless

● select() and poll() is stateless– This forces them to be O(n) operations

● The kernel must scan all file descriptors– This tend to suck for large number of file descriptors

● Having state means more code in the kernel– Continuously monitor a set of file descriptions– Makes an O(n) operation to O(1)

15

Epoll 1/2

int epfd = epoll_create(EPOLL_QUEUE_LEN);int client_sock = socket();static struct epoll_event ev;ev.events = EPOLLIN | EPOLLOUT | EPOLLERR;ev.data.fd = client_sock;int res = epoll_ctl(epfd, EPOLL_CTL_ADD, client_sock, &ev);// Main loopstruct epoll_event *events;while (1) { int nfds = epoll_wait(epfd, events, MAX_EVENTS, TIMEOUT); for(int i = 0; i < nfds; i++) { int fd = events[i].data.fd; handle_io_on_socket(fd); }}

16

Epoll 2/2

● Has state in the kernel– File descriptors must be added and removed– Slight more complex API

● Outperforms select and poll– When the number of open files is high

● Availability: Linux only (introduced in 2.5)

17

KQueue

● We've had enough APIs for now● Works much like epoll● Also does disk IO, signals, file/process events● Arguably best interface● Availability: BSD

18

libevent

● Event based library– Portable (Linux, BSD, Solaris, Windows)– Slightly better abstraction

● Can use select, poll, epoll, KQueue, /dev/poll– Possible to change between them

● Can also be used for benchmarking :­)

19

libevent benchmark 1/2

20

libevent benchmark 2/2

21

Asynchronous Disk IO

● What about disk IO– Often blocks so little time that it can be ignored– But sometimes it cannot

● Databases are the prime target● Sockets has been asynchronous for many years

– With disk IO limping behind● Posix AIO to the rescue

22

Posix AIO

● Relatively new standard– Introduced in Linux in 2005

● Was in several vendor trees before that– Often emulated in libc using threads

● Also does vector operations● API Sample:int aio_read(struct aiocb *aiocbp);int aio_suspend(const struct aiocb* const iocbs[], int niocb, const struct timespec *timeout);

23

Twisted

● An event­driven framework● Lives at: www.twistedmatrix.com● Started as game network library● Rather large

– Around 200K lines of Python– Basic support for 30+ protocols

● Has web, mail, ssh clients and servers● Also has its own database API● And security infrastructure (pluggable)

24

Twisted vs. the world

● Probably most advanced framework in its class● Only real competitor is ACE 

– Twisted borrows a lot of inspiration from here– ACE is also historically important

● Some claim that java.nio and C# Async are also asynchronous frameworks

25

Twisted Architecture

● Tries hard to keep things orthogonal● Reactor–Transport–Factory–Protocol­Application

– As opposed to mingled together libraries● Makes changes very easy (once mastered)

– Often is about combining things the right way– Also means more boilerplate code

26

Twisted Echo Serverfrom twisted.internet import reactor, protocol

class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data)

factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()

● Lots of magic behind the curtain– Protocols, Factories and the Reactor

27

The Twisted Reactor

● The main loop of Twisted● Don't call us, we'll call you (framework)

– You insert code into the reactor● Also does scheduling

– Future calls, looping calls● Interchangeable

– select, poll, WMFO, IOCP, CoreFoundation, KQueue– Integrates with GTK, Qt, wxWidgets

28

Factories and Protocols

● Factories produces protocols– On incoming connections

● A protocol represents a connection– Provides method such as dataReceived– Which are called from the reactor

● The application is build on top of protocols– May use several transports, factories, and protocols

29

Twisted Deferreds

● The single most confusing aspect in Twisted– Removes the concept of stack execution

● Vital to understand as they are the program flow– An alternative to state machines

● Think of them as a one­shot continuation– Or: An object representing what should happen– Or: A promise that data will be delivered– Or: A callback

30

Deferred Example

● Fetching a web page:def gotPage(page): print "I got the page:", page

deferred = twisted.web.getPage("http://www.cs.aau.dk/")deferred.addCallback(gotPage)return deferred

● The getPage function takes time to complete– But blocking is prohibited

● When the page is retrieve the deferred will fire– And callbacks will be invoked

31

Deferreds and Errors

def gotPage(page): print "I got the page:", page

def getPage_errorback(error): print "Didn't get the page, me so sad - reason:", error

deferred = twisted.web.getPage("http://www.cs.aau.dk/")deferred.addCallback(gotPage)deferred.addErrback(getPage_errorback)

● Separates normal program flow from error handling

32

Chaining Callbacks

● Often several sequential actions are needed● The result from gotPage is provided as argument 

to changePage● The execution stack disappears

– Causes headaches during learning– Coroutines can make program more stack­like

deferred = twisted.web.getPage("http://www.cs.aau.dk/")deferred.addCallback(gotPage)deferred.addCallback(changePage)deferred.addCallback(uploadPage)

33

Deferreds and Coroutines

● Notice: yield instead of return– The function is reentrant

@defer.deferredGeneratordef myFunction(self): d = getPageStream("http://www.cs.aau.dk") yield d ; stream = d.getResult() while True: d = stream.read() yield d ; content = d.getResult() if content is None: break print content

34

Why Twisted

● Grasping the power of Twisted is difficult– Also hard to explain

● Makes cross protocol integration very easy● Provides a lot of functionality and power

– But one needs to know how to use it● Drawbacks

– Steep learning curve– Portability: Applications cannot be ported to Twisted

35

Summary

● Asynchronous IO– An alternative to threads– Never block– Requires program reorganization

● Asynchronous Programming– Stateless: Berkeley sockets, select, poll– State: epoll, Kqueue– Disk: Posix AIO– libevent and Twisted

● Exercises?