Writing a Cassandra client in Perl
Stefano [email protected]
@larsen
• High Write Throughput
• Large Tables
• Reports
• Analytics
Cassandra
Bigtable: A distributed storage system for structured data, 2006
Dynamo: Amazon's highly available keyvalue
store, 2007
Dynamo partitioning and
replication
ColumnFamily data model similar
to Bigtable's
High availability
Incremental scalability
Eventually consistent
Tunable tradeoffs between consistency
and latency
Minimal administration
(not quite so)
No SPOFp2p distribution model
Writing
Writinginsert(key, {column_name => value})
☺
Writing
R
☺
Writing
R
☺
Writing
17%
17%
17% 17%R
☺
Writing
R
☺
Writing
R
☺
Writing a Cassandra client in Perl
Talking with a server
Cassandra::CassandraClient
Thrift::BinaryProtocol
Thrift::FramedTransport
Thrift::Socket
Cassandra::CassandraClient
Thrift::BinaryProtocol
Thrift::FramedTransport
Thrift::Socket
Cassandra::CassandraClient
Thrift::BinaryProtocol
Thrift::FramedTransport
The Transport layer provides a simple abstraction for reading/writing from/to the network. This enables Thrift to decouple the underlying transport from the rest of the system (serialization/deserialization, for instance).
Cassandra::CassandraClient
Thrift::BinaryProtocolThe Protocol abstraction defines a mechanism to map in-memory data structures to a wire-format. In other words, a protocol specifies how datatypes use the underlying Transport to encode/decode themselves.
Cassandra::CassandraClient"High level" interface: login, reading & writing data, …
1) Install Cassandra2) Install Thrift
thrift -‐gen perl cassandra.thrift
Login
thriftservice Cassandra { # auth methods void login(1: required AuthenticationRequest auth_request) i throws (1:AuthenticationException authnx, 2:AuthorizationException authzx), …
Perl (generated code)package Cassandra::CassandraClient;…
sub login{ my $self = shift; my $auth_request = shift;
$self-‐>send_login($auth_request); $self-‐>recv_login();}
thriftservice Cassandra { # auth methods void login(1: required AuthenticationRequest auth_request) i throws (1:AuthenticationException authnx, 2:AuthorizationException authzx), …
Perl (generated code)package Cassandra::CassandraClient;…
sub login{ my $self = shift; my $auth_request = shift;
$self-‐>send_login($auth_request); $self-‐>recv_login();}
thrift
Perl (generated code)
struct AuthenticationRequest { 1: required map<string, string> credentials}
package Cassandra::AuthenticationRequest;use base qw(Class::Accessor);Cassandra::AuthenticationRequest-‐>mk_accessors( qw( credentials ) );
sub new { …}
my $ks = 'TestIPW';
my $socket = Thrift::Socket-‐>new($server_name, $server_port);my $transport = Thrift::FramedTransport-‐>new($socket, 1024, 1024);my $protocol = Thrift::BinaryProtocol-‐>new($transport);$client = Cassandra::CassandraClient-‐>new( $protocol );
eval { $transport-‐>open; my $auth = Cassandra::AuthenticationRequest-‐>new( { credentials => {} } ); $client-‐>login($auth);
$client-‐>set_keyspace($ks);};if ( $@ ) { die Dumper( $@ );}
my $ks = 'TestIPW';
my $socket = Thrift::Socket-‐>new($server_name, $server_port);my $transport = Thrift::FramedTransport-‐>new($socket, 1024, 1024);my $protocol = Thrift::BinaryProtocol-‐>new($transport);$client = Cassandra::CassandraClient-‐>new( $protocol );
eval { $transport-‐>open; my $auth = Cassandra::AuthenticationRequest-‐>new( { credentials => {} } ); $client-‐>login($auth);
$client-‐>set_keyspace($ks);};if ( $@ ) { die Dumper( $@ );}
Cassandra::CassandraClient
Thrift::BinaryProtocol
Thrift::FramedTransport
Thrift::Socket
my $ks = 'TestIPW';
my $socket = Thrift::Socket-‐>new($server_name, $server_port);my $transport = Thrift::FramedTransport-‐>new($socket, 1024, 1024);my $protocol = Thrift::BinaryProtocol-‐>new($transport);$client = Cassandra::CassandraClient-‐>new( $protocol );
eval { $transport-‐>open; my $auth = Cassandra::AuthenticationRequest-‐>new( { credentials => {} } ); $client-‐>login($auth);
$client-‐>set_keyspace($ks);};if ( $@ ) { die Dumper( $@ );}
thriftservice Cassandra { # auth methods void login(1: required AuthenticationRequest auth_request) i throws (1:AuthenticationException authnx, 2:AuthorizationException authzx), …
Reading
thriftlist<ColumnOrSuperColumn> get_slice( 1:required binary key, 2:required ColumnParent column_parent, 3:required SlicePredicate predicate, 4:required ConsistencyLevel consistency_level=ConsistencyLevel.ONE) throws (1:InvalidRequestException ire, 2:UnavailableException ue, 3:TimedOutException te)
Perl (generated code)package Cassandra::CassandraClient;…sub get_slice{ my $self = shift; my $key = shift; my $column_parent = shift; my $predicate = shift; my $consistency_level = shift;
$self-‐>send_get_slice($key, $column_parent, $predicate, $consistency_level); return $self-‐>recv_get_slice();}
thriftlist<ColumnOrSuperColumn> get_slice( 1:required binary key, 2:required ColumnParent column_parent, 3:required SlicePredicate predicate, 4:required ConsistencyLevel consistency_level=ConsistencyLevel.ONE) throws (1:InvalidRequestException ire, 2:UnavailableException ue, 3:TimedOutException te)
Perl (generated code)package Cassandra::CassandraClient;…sub get_slice{ my $self = shift; my $key = shift; my $column_parent = shift; my $predicate = shift; my $consistency_level = shift;
$self-‐>send_get_slice($key, $column_parent, $predicate, $consistency_level); return $self-‐>recv_get_slice();}
thrift
struct ColumnParent { 3: required string column_family, 4: optional binary super_column,}
Perl (generated code)
package Cassandra::ColumnParent;use base qw(Class::Accessor);Cassandra::ColumnParent-‐>mk_accessors( qw( column_family super_column ) );
thriftlist<ColumnOrSuperColumn> get_slice( 1:required binary key, 2:required ColumnParent column_parent, 3:required SlicePredicate predicate, 4:required ConsistencyLevel consistency_level=ConsistencyLevel.ONE) throws (1:InvalidRequestException ire, 2:UnavailableException ue, 3:TimedOutException te)
Perl (generated code)package Cassandra::CassandraClient;…sub get_slice{ my $self = shift; my $key = shift; my $column_parent = shift; my $predicate = shift; my $consistency_level = shift;
$self-‐>send_get_slice($key, $column_parent, $predicate, $consistency_level); return $self-‐>recv_get_slice();}
thrift
struct SlicePredicate { 1: optional list<binary> column_names, 2: optional SliceRange slice_range,}
Perl (generated code)
package Cassandra::SlicePredicate;use base qw(Class::Accessor);Cassandra::SlicePredicate-‐>mk_accessors( qw( column_names slice_range ) );
thriftlist<ColumnOrSuperColumn> get_slice( 1:required binary key, 2:required ColumnParent column_parent, 3:required SlicePredicate predicate, 4:required ConsistencyLevel consistency_level=ConsistencyLevel.ONE) throws (1:InvalidRequestException ire, 2:UnavailableException ue, 3:TimedOutException te)
Perl (generated code)package Cassandra::CassandraClient;…sub get_slice{ my $self = shift; my $key = shift; my $column_parent = shift; my $predicate = shift; my $consistency_level = shift;
$self-‐>send_get_slice($key, $column_parent, $predicate, $consistency_level); return $self-‐>recv_get_slice();}
thrift
struct SliceRange { 1: required binary start, 2: required binary finish, 3: required bool reversed=0, 4: required i32 count=100,}
Perl (generated code)
package Cassandra::SliceRange;use base qw(Class::Accessor);Cassandra::SliceRange-‐>mk_accessors( qw( start finish reversed count ) );
thriftlist<ColumnOrSuperColumn> get_slice( 1:required binary key, 2:required ColumnParent column_parent, 3:required SlicePredicate predicate, 4:required ConsistencyLevel consistency_level=ConsistencyLevel.ONE) throws (1:InvalidRequestException ire, 2:UnavailableException ue, 3:TimedOutException te)
Perl (generated code)package Cassandra::CassandraClient;…sub get_slice{ my $self = shift; my $key = shift; my $column_parent = shift; my $predicate = shift; my $consistency_level = shift;
$self-‐>send_get_slice($key, $column_parent, $predicate, $consistency_level); return $self-‐>recv_get_slice();}
thriftenum ConsistencyLevel { ONE = 1, QUORUM = 2, LOCAL_QUORUM = 3, EACH_QUORUM = 4, ALL = 5, ANY = 6, TWO = 7, THREE = 8,}
Perl (generated code)
package Cassandra::ConsistencyLevel;use constant ONE => 1;use constant QUORUM => 2;use constant LOCAL_QUORUM => 3;use constant EACH_QUORUM => 4;use constant ALL => 5;use constant ANY => 6;use constant TWO => 7;use constant THREE => 8;
package Cassandra::ConsistencyLevel;use constant ONE => 1;use constant QUORUM => 2;use constant LOCAL_QUORUM => 3;use constant EACH_QUORUM => 4;use constant ALL => 5;use constant ANY => 6;use constant TWO => 7;use constant THREE => 8;
package Cassandra::ConsistencyLevel;use constant ONE => 1;use constant QUORUM => 2;use constant LOCAL_QUORUM => 3;use constant EACH_QUORUM => 4;use constant ALL => 5;use constant ANY => 6;use constant TWO => 7;use constant THREE => 8;
my $cf = '20121011.larsen.tweets';
my $columnParent = Cassandra::ColumnParent-‐>new({column_family => $cf});my $sliceRange = Cassandra::SliceRange-‐>new({ start => '', finish => '' });
my $predicate = Cassandra::SlicePredicate-‐>new({ slice_range => $sliceRange });
$res = $client-‐>get_slice( $key, $columnParent, $predicate, Cassandra::ConsistencyLevel::ONE);
my $cf = '20121011.larsen.tweets';
my $columnParent = Cassandra::ColumnParent-‐>new({column_family => $cf});my $sliceRange = Cassandra::SliceRange-‐>new({ start => '', finish => '' });
my $predicate = Cassandra::SlicePredicate-‐>new({ slice_range => $sliceRange });
$res = $client-‐>get_slice( $key, $columnParent, $predicate, Cassandra::ConsistencyLevel::ONE);
thrift
struct SliceRange { 1: required binary start, 2: required binary finish, 3: required bool reversed=0, 4: required i32 count=100,}
my $cf = '20121011.larsen.tweets';
my $columnParent = Cassandra::ColumnParent-‐>new({column_family => $cf});my $sliceRange = Cassandra::SliceRange-‐>new({ start => '', finish => '' });
my $predicate = Cassandra::SlicePredicate-‐>new({ slice_range => $sliceRange });
$res = $client-‐>get_slice( $key, $columnParent, $predicate, Cassandra::ConsistencyLevel::ONE);
thriftlist<ColumnOrSuperColumn> get_slice( 1:required binary key, 2:required ColumnParent column_parent, 3:required SlicePredicate predicate, 4:required ConsistencyLevel consistency_level=ConsistencyLevel.ONE) throws (1:InvalidRequestException ire, 2:UnavailableException ue, 3:TimedOutException te)
Writing
thrift
void insert(1:required binary key,· 2:required ColumnParent column_parent, 3:required Column column, 4:required ConsistencyLevel consistency_level=ConsistencyLevel.ONE) throws (1:InvalidRequestException ire, 2:UnavailableException ue, 3:TimedOutException te)
Perl (generated code)
sub insert{ my $self = shift; my $key = shift; my $column_parent = shift; my $column = shift; my $consistency_level = shift;
$self-‐>send_insert( $key, $column_parent, $column, $consistency_level); $self-‐>recv_insert();}
thrift
void insert(1:required binary key,· 2:required ColumnParent column_parent, 3:required Column column, 4:required ConsistencyLevel consistency_level=ConsistencyLevel.ONE) throws (1:InvalidRequestException ire, 2:UnavailableException ue, 3:TimedOutException te)
Perl (generated code)
sub insert{ my $self = shift; my $key = shift; my $column_parent = shift; my $column = shift; my $consistency_level = shift;
$self-‐>send_insert( $key, $column_parent, $column, $consistency_level); $self-‐>recv_insert();}
thrift
Perl (generated code)
package Cassandra::Column;use base qw(Class::Accessor);Cassandra::Column-‐>mk_accessors( qw( name value timestamp ttl ) );
struct Column { 1: required binary name, 2: optional binary value, 3: optional i64 timestamp, 4: optional i32 ttl,}
eval { my $cf = 'TestCF'; my $key = 'bar';
my $columnParent = Cassandra::ColumnParent-‐>new({ column_family => $cf });
my $ts = sprintf "%.6f", Time::HiRes::time; $ts =~ s/\.//; say $ts;
my $column = Cassandra::Column-‐>new({ name => 'test', value => 'a test value', timestamp => $ts });
$client-‐>insert( $key, $columnParent, $column, Cassandra::ConsistencyLevel::ONE);};if ( $@ ) { die Dumper( $@ );}
eval { my $cf = 'TestCF'; my $key = 'bar';
my $columnParent = Cassandra::ColumnParent-‐>new({ column_family => $cf });
my $ts = sprintf "%.6f", Time::HiRes::time; $ts =~ s/\.//; say $ts;
my $column = Cassandra::Column-‐>new({ name => 'test', value => 'a test value', timestamp => $ts });
$client-‐>insert( $key, $columnParent, $column, Cassandra::ConsistencyLevel::ONE);};if ( $@ ) { die Dumper( $@ );}
thrift
struct Column { 1: required binary name, 2: optional binary value, 3: optional i64 timestamp, 4: optional i32 ttl,}
CPAN?
☹
Net::Cassandra ✘Net::Cassandra::Easy ✘
Cassandra::Lite ✘AnyEvent::Cassandra ✘
☺
Questions?