Asynchronous Programming FTW! 2 (with AnyEvent)

AsynchronousAsynchronous

ProgrammingProgramming

FTW!FTW!

(Sawyer X)

@PerlSawyer

Theory

Practice

TheoryTheory

We think in non-blocking terms

We act in non-blocking terms

We program in blocking terms

An example? Sure!

The exampleThe example

use LWP::UserAgent;

my @urls = ( 'https://duckduckgo.com', 'http://cluj.pm', 'http://telaviv.pm.org',);

my $ua = LWP::UserAgent->new;foreach my $url (@urls) { my $res = $ua->get($url); say $res->decoded_content;}

Problem?Problem?

Wait on each URL

Ineffecient waste of time

Pointless

Senseless

CRIME AGAINST HUMANITY

(maybe just ineffecient)

How we thinkHow we think

Nothing to work on? Defer to later!

Like cooking!

Making dinnerMaking dinner

Boil water. Wait? No! Salad time!

(water gets deferred to later)

Cook potatoes. Wait? No! Sauce time!

(potatoes get deferred to later)

We think and act non-blocking

TIMTOWTDITIMTOWTDI

Forks

Threads

Event-loops <- this talk

Event loop?Event loop?

Single thread, single process

Like a big while(1) {}Heavy usage of code references

No real race conditions

No locking, semaphores, etc.

Can yield better speed

But... nothing can be blocking! (I/O)

Event loops in PerlEvent loops in Perl

POE

Reflex

IO::Async

IO::Lambda

Qt, GTK, Tk, Glib, Cocoa::EventLoop, EV, FLTK, Irssi

AnyEvent <- this talk

AnyEventAnyEvent

Light-weight

Fast

Very fast

Slim interface

Good support for other event loops

Lots of modules

PracticePractice

Cooking with AnyEventCooking with AnyEvent

(timers)(timers)

use AnyEvent;say '>> Water boiling...';my $water = AnyEvent->timer( after => 5*60, cb => sub { say '<< Water boiled!'; say '>> Potatos being prepared...'; my $potatoes; $potatoes = AnyEvent->timer( after => 10*60, cb => sub { undef $potatoes; say '<< Potatos ready!'; } );} );

say '>> Salad being prepared...';my $salad = AnyEvent->timer( after => 6*60, cb => sub { say '<< Salad ready!'; say '>> Sauce being prepared...'; my $sauce; $sauce = AnyEvent->timer( after => 12*60, cb => sub { undef $sauce; say '<< Sauce ready!'; } );} );

Water - 5 minutes

Potatos - 10 minutes after water

Salad - 6 minutes

Sauce - 12 minutes after salad

Water, salad, potatoes, sauce

>> Water boiling...>> Salad being prepared...<< Water boiled!>> Potatos being prepared...<< Salad ready!>> Sauce being prepared...<< Potatos ready!<< Sauce ready!

Condition variablesCondition variables

A condition waiting to be fulfilled

Starting at false, hoping to get to trueHelps waiting for other events to finish

Call ->recv() to wait for stuff to finish

Call ->send() when finished doing your stuff

Without condition variables, we reach the end of programs

We reach the end == program closes

Program closes == our code doesn't get executed

Our code doesn't get executed == we get executed!

Simple exampleSimple example

use AnyEvent;

my $count = 1;my $cv = AnyEvent->condvar;my $timer = AnyEvent->timer( interval => 1, cb => sub { say "Count: $count"; $count++ == 5 and $cv->send; },);

$cv->recv;say 'We counted to 5!';

Bigger exampleBigger example

my $cv = AnyEvent->condvar;my $water = AnyEvent->timer( after => 5*60, cb => sub { my $potatoes; $potatoes = AnyEvent->timer( after => 10*60, cb => sub { undef $potatoes; } );} );

my $salad = AnyEvent->timer( after => 6*60, cb => sub { my $sauce; $sauce = AnyEvent->timer( after => 12*60, cb => sub { undef $sauce; $cv->send; } );} );

$cv->recv;

Better example :: LWPBetter example :: LWP

use LWP::UserAgent;

my @urls = ( 'https://whatismyip.com', 'http://checkip.dyndns.org', 'http://wtfismyipaddress.org',);

my $ua = LWP::UserAgent->new;foreach my $url (@urls) { my $res = $ua->get($url); say $res->decoded_content;}

Better example :: AnyEventBetter example :: AnyEvent

use AnyEvent;use AnyEvent::HTTP;

my $cv = AnyEvent->condvar;foreach my $url (@urls) { $cv->begin; http_get $url, sub { my $body = shift; say $body; $cv->end; }}

$cv->recv;

POP QUIZ TIME!POP QUIZ TIME!

A few examples

Spot the error(s) and get a prize

Available prizes: handshake, high five, hug, kiss

(not necessarily from me)

Round 1Round 1

use AnyEvent;use AnyEvent::HTTP;

http_post $url, $body, sub { say 'Successfully made an API call to do something cool!';};

# wait for all events to finishAnyEvent->condvar->recv;

Round 2Round 2

use AnyEvent;

my $timer = AnyEvent->timer( interval => 5, cb => sub { sleep 1; say 'Bwahaha!'; });

Round 3Round 3

use AnyEvent;my $cv = AnyEvent->condvar;my $player1 = AnyEvent->timer( after => 3.7, cb => sub { say 'Player 1 joins the game!'; $cv->send; },);my $player2 = AnyEvent->timer( after => 7.2, cb => sub { say 'Player 2 joins the game!'; $cv->send; },);$cv->recv;

Round 4Round 4

use AnyEvent; use HTTP::Tiny;my $cv = AnyEvent->condvar;$cv->begin for 1 .. 2;my $player1 = AnyEvent->timer( after => 3.7, cb => sub { say 'Player 1 joins the game! Alerting server!'; my $res = HTTP::Tiny->new->get('http://server:3030/add/1'); $res->{'success'} or die 'Failed to insert player 1 to game!'; $cv->end; },);my $player2 = AnyEvent->timer( after => 7.2, cb => sub { say 'Player 2 joins the game!'; $cv->end; },);$cv->recv;

Stuff to do with AnyEventStuff to do with AnyEvent

xkcd interface

WWW::xkcdWWW::xkcd

use WWW::xkcd;my $xkcd = WWW::xkcd->new;

$xkcd->fetch( sub { my ( $img, $comic ) = @_; say "Today's comic is titled: ", $comic->{'title'}; ...} );

Stuff to do with AnyEventStuff to do with AnyEvent

xkcd interface

PSGI/Plack server

TwiggyTwiggy

use Twiggy::Server;my $app = sub { # PSGI app};

my $server = Twiggy::Server->new(...);$server->register_service($app);

AnyEvent->condvar->recv;

Stuff to do with AnyEventStuff to do with AnyEvent

xkcd interface

PSGI/Plack server

Async AWS EC2 interface

AnyEvent::EC2::TinyAnyEvent::EC2::Tiny

use AnyEvent::EC2::Tiny;my $ec2 = AnyEvent::EC2::Tiny->new(...);my $xml = $ec2->send( 'RegionName.1' => 'us-east-1', Action => 'DescribeRegions', success_cb => sub { my $xml = shift; # prints ec2.us-east-1.amazonaws.com say $xml->{'regionInfo'}{'item'}[0]{'regionEndpoint'}; }, fail_cb => sub { my $error = shift; ... },);

Stuff to do with AnyEventStuff to do with AnyEvent

xkcd interface

PSGI/Plack server

Async AWS EC2 interface

Monitoring framework

JunoJuno

use Juno;my $juno = Juno->new( hosts => [ 'server1', 'server2' ], interval => 10, checks => { HTTP => { headers => { { 'Host', 'example.com' }, },

on_result => sub { my $result = shift; ... }, }, },);

Stuff to do with AnyEventStuff to do with AnyEvent

xkcd interface

PSGI/Plack server

Async AWS EC2 interface

Monitoring framework

Real-time network path resolution

SIP RouteSIP Route

use AnyEvent;use Anyevent::Util 'fork_call';use AnyEvent::Socket;use AnyEvent::Handle;use Data::Dump 'dump';

my $tcp_server_guard = tcp_server '127.0.0.1', 2020, sub { my ($fh) = @_; my $handle = AnyEvent::Handle->new( fh => $fh ); $handle->push_read( line => sub {...} );};

my ( $hdl, $target ) = @_;

if ( ! defined $target ) { $hdl->push_write($default_route); $hdl->destroy; return;}

chomp $target;

if ( ! exists $data{$target} ) { $hdl->push_write($default_route); $hdl->destroy; return;}

my @sorted = sort { $data{$target}{$a} <=> $data{$target}{$b} } keys %{ $data{$target} };

$hdl->push_write( $sorted[0] );

$hdl->destroy;

foreach my $target (@targets) { foreach my $iface (@interfaces) { my $cmd = "sipsak --sip-uri=sip:$target -D $timeout -X $iface"; push @timers, AE::timer $start_after, $check_every, sub { my $time = AE::now; fork_call { system "$cmd >/dev/null 2>&1"; return $? >> 8; } sub { my $exit = shift; ( $exit == 0 || $exit == 1 ) or return delete $data{$target}{$iface}; $data{$target}{$iface} = AE::now - $time; }; }; }}

AE::cv->recv;

if ($verbose) { push @timers, AE::timer 5, 5, sub { dump \%data };}

Stuff to do with AnyEventStuff to do with AnyEvent

xkcd interface

PSGI/Plack server

Async AWS EC2 interface

Monitoring framework

Real-time network path resolution

API interfaces to services

AnyEvent::RiakAnyEvent::Riak

use AnyEvent::Riak;

my $riak = AnyEvent::Riak->new( host => 'http://127.0.0.1:8098', path => 'riak',);

$riak->list_bucket( 'mybucket', {props => 'true', keys => 'false'}, sub { my $res = shift; ... });

AnyEvent::SNMPAnyEvent::SNMP

use AnyEvent::SNMP;use Net::SNMP;

my $cv = AnyEvent->condvar;Net::SNMP->session( -hostname => "127.0.0.1", -community => "public", -nonblocking => 1)->get_request( -callback => sub { $cv->send (@_) } );

my @result = $cv->recv;

AnyEvent::XMLRPCAnyEvent::XMLRPC

use AnyEvent::XMLRPC;

my $serv = AnyEvent::XMLRPC->new( methods => { 'echo' => \&echo, },);

AnyEvent::DNSAnyEvent::DNS

use AnyEvent::DNS;

my $cv = AnyEvent->condvar;AnyEvent::DNS::a "www.google.ro", $cv;

# later...my @addrs = $cv->recv;

AnyEvent::TwitterAnyEvent::Twitter

my $ua = AnyEvent::Twitter->new( consumer_key => 'consumer_key', consumer_secret => 'consumer_secret', access_token => 'access_token', access_token_secret => 'access_token_secret',);

my $cv = AnyEvent->condvar;

# GET request$cv->begin;$ua->get( 'account/verify_credentials', sub { my ($header, $response, $reason) = @_;

say $response->{screen_name}; $cv->end;} );

AnyEvent::GraphiteAnyEvent::Graphite

my $graphite = AnyEvent::Graphite->new( host => '127.0.0.1', port => '2003',);

$graphite->send("a.b.c.d", $value, $timestamp);

Stuff to do with AnyEventStuff to do with AnyEvent

xkcd interface

PSGI/Plack server

Async AWS EC2 interface

Monitoring framework

Real-time network path resolution

API interfaces to services

Find matching special actors in TV series

Actor matchesActor matches

use AnyEvent; use AnyEvent::HTTP;my $base = 'http://www.imdb.com/title';my %titles = ( tt1196946 => { name => 'Mentalist, the' }, tt1219024 => { name => 'Castle' }, tt0773262 => { name => 'Dexter' }, tt0491738 => { name => 'Psych' }, tt0247082 => { name => 'CSI Las Vegas' }, tt0458253 => { name => 'Closer, the' },);

# DON'T PARSE HTML WITH REGEX!!!!1127my $episode_regex = qr{...};my $cast_regex = qr{...};

# Fetching number of seasons for each titlemy $cv = AE::cv;foreach my $title_id ( keys %titles ) { my $title = $titles{$title_id}{'name'}; print "[$title] Fetching number of seasons.\n";

my $url = "$base/$title_id/"; my $regex = qr{/title/$title_id/episodes\?season=([0-9]+)}; $cv->begin; http_get $url, sub { my ( $body, $hdr ) = @_; my @found = $body =~ /$regex/mg; $titles{$title_id}{'episodes'}{$_} = [] for @found; $cv->end; };}$cv->recv;

# fetching the list of episodes for each seasonforeach my $title_id ( keys %titles ) { my $cv = AE::cv; foreach my $season ( keys %{ $titles{$title_id}{'episodes'} } ) { my $title = $titles{$title_id}{'name'}; print "[$title] [Season $season] fetching episodes\n";

my $season_url = "$base/$title_id/episodes?season=$season"; $cv->begin; http_get $season_url, sub { my ( $body, $hdr ) = @_; my @found = $body =~ /$episode_regex/mg;

while (@found) { my $id = shift @found; my $name = shift @found;

foreach my $title_id ( keys %titles ) { my $title = $titles{$title_id}{'name'};

foreach my $season ( keys %{ $titles{$title_id}{'episodes'} } ) { my @episodes = @{ $titles{$title_id}{'episodes'}{$season} };

print "[$title] [$season] Fetching cast for season $season\n"; my $cv = AE::cv; foreach my $episode_id (@episodes) { print " -> [$title] [$season] $episode_id\n"; my $url = "$base/$episode_id/fullcredits";

$cv->begin; http_get $url, sub { my ( $body, $hdr ) = @_; my @found = $body =~ /$cast_regex/mg;

print "Populating actors\n";my %actors = ();foreach my $title_id ( keys %titles ) { my $title = $titles{$title_id}{'name'};

foreach my $cast_sets ( @{ $titles{$title_id}{'cast'} } ) { my ( $name, $role ) = @{$cast_sets}; $actors{$name}{$title} = $role; }}

print "Cross referencing\n";foreach my $name ( keys %actors ) { scalar keys %{ $actors{$name} } > 1 or next;

my @where = (); foreach my $title ( keys %{ $actors{$name} } ) { my $role = $actors{$name}{$title}; push @where, "$title ($role)"; }

printf "Actor $name appeared in %s\n", join ', ', @where;}

Some dataSome data

6 TV series

42 total seasons

1,510 total episodes

6,484 total actors

7,493 total roles

380,520 total requests (6 * 42 * 1,510)

~ 2s/req = 761,040s = 12,684m = 211h = 8.8 days

Time to run (incl. analysis) with AnyEvent: 6 minutes

Matches found... 1,095!

Actor Michael Patrick McGill appeared in:

CSI Las Vegas (Ruben Caster)

Dexter (Troy)

Actor Robert Esser appeared in:

CSI Las Vegas (Waiter)

Castle (Officer #2 (uncredited))

Actor Andre Alexsen appeared in:

Closer, the (Adam)

Mentalist, the (CA State Govenor (uncredited))

Actor Becky O'Donohue appeared in:

CSI Las Vegas (Ava Rendell)

Mentalist, the (Sasha)

Psych (Molly Gogolack)

Mark PellegrinoMark Pellegrino

Tom Dempsey / Tom Dempsey III in Castle

Gavin Q. Baker III in The Closer

Bruno Curtis in CSI Last Vegas

Paul Bennett in Dexter

Von McBride in The Mentalist

Most accomplishedMost accomplished

cross-actorcross-actor

Stuff to do with AnyEventStuff to do with AnyEvent

xkcd interface

PSGI/Plack server

Async AWS EC2 interface

Monitoring framework

Real-time network path resolution

API interfaces to services

Find matching special actors in TV series

Whole lot of other awesome stuff

(which I'm not gonna show you now)

Thank youThank you