©2012 DataStax

CQL3 in depth

Yuki MorishitaSoftware Engineer@DataStax / Apache Cassandra Committer

1

Cassandra Conference in Tokyo, 11/29/2012

©2012 DataStax2

• Why CQL3?• CQL3 walkthrough• Defining Schema• Querying / Mutating Data• New features

• Related topics• Native transport

Agenda!

©2012 DataStax

Why CQL3?

3

©2012 DataStax

Cassandra Storage

4

create column family profileswith key_validation_class = UTF8Typeand comparator = UTF8Typeand column_metadata = [ {column_name: first_name, validation_class: UTF8Type}, {column_name: last_name, validation_class: UTF8Type}, {column_name: year, validation_class: IntegerType}];

nobu first_name Nobunaga

last_name Oda

year 1582

row key columns

columns are sortedin comparator order

values are validated by validation_class

©2012 DataStax

Thrift API

• Low level: get, get_slice, mutate...• Directly exposes internal storage

structure• Hard to change the signature of API

5

©2012 DataStax6

Column col = new Column(ByteBuffer.wrap("name".getBytes()));col.setValue(ByteBuffer.wrap("value".getBytes()));col.setTimestamp(System.currentTimeMillis());

ColumnOrSuperColumn cosc = new ColumnOrSuperColumn();cosc.setColumn(col);

Mutation mutation = new Mutation();mutation.setColumn_or_supercolumn(cosc);

List<Mutation> mutations = new ArrayList<Mutation>();mutations.add(mutation);

Map<String, List<Mutation>> cf = new HashMap<String, List<Mutation>>();cf.put("Standard1", mutations);

Map<ByteBuffer, Map<String, List<Mutation>>> records = new HashMap<ByteBuffer, Map<String, List<Mutation>>>();records.put(ByteBuffer.wrap("key".getBytes()), cf);

client.batch_mutate(records, consistencyLevel);

Inserting data with Thrift

©2012 DataStax

... with Cassandra Query Language

7

INSERT INTO “Standard1” (key, name)VALUES (“key”, “value”);

• Introduced in 0.8(CQL), updated in 1.0(CQL2)

• Syntax similar to SQL• More extensible than Thrift API

©2012 DataStax

CQL2 Problems

• Almost 1 to 1 mapping to Thrift API, so not compose with the row-oriented parts of SQL

• No support for CompositeType

8

©2012 DataStax

CQL3

• Maps storage to a more natural rows-and-columns representation using CompositeType• Wide rows are “transposed” and unpacked

into named columns• beta in 1.1, default in 1.2• New features• Collection support

9

©2012 DataStax

CQL3 walkthrough

10

©2012 DataStax

Defining Keyspace

• Syntax is changed from CQL2

11

CREATE KEYSPACE my_keyspace WITH replication = { 'class': 'SimpleStrategy', 'replication_factor': 2};

©2012 DataStax

Defining Static Column Family

12

• “Strict” schema definition (and it’s good thing)• You cannot add column arbitrary• You need ALTER TABLE ... ADD column

first• Columns are defined and sorted using

CompositeType comparator

©2012 DataStax

Defining Static Column Family

13

CREATE TABLE profiles ( user_id text PRIMARY KEY, first_name text, last_name text, year int)

nobu

first_name: Nobunaga

last_name: Oda

year: 1582

user_id

columns are sortedin comparator order

values are validated by type definition

CompositeType(UTF8Type)

user_id | first_name | last_name | year---------+------------+-----------+------ nobu | Nobunaga | Oda | 1582

:

©2012 DataStax

Defining Dynamic Column Family

• Then, how can we add columns dynamically to our time series data like we did before?• Use compound key

14

©2012 DataStax

Compound key

15

CREATE TABLE comments ( article_id uuid, posted_at timestamp, author text, content text, PRIMARY KEY (article_id, posted_at))

550e8400-..

1350499616:author yukim

1350499616:content blah, blah, blah

1368499616:author

article_id

columns are sortedin comparator order,first by date, and thencolumn name

values are validated by type definition

CompositeType(DateType, UTF8Type)

1368499616:content well, well, well...

yukim

1350499616:

1368499616:

©2012 DataStax

Compound key

16

cqlsh:ks> SELECT * FROM comments;

article_id | posted_at | author | content--------------+--------------------------+--------+------------------ 550e8400-... | 1970-01-17 00:08:19+0900 | yukim | blah, blah, blah 550e8400-... | 1970-01-17 05:08:19+0900 | yukim | well, well, well

cqlsh:ks> SELECT * FROM comments WHERE posted_at >= '1970-01-17 05:08:19+0900';

article_id | posted_at | author | content--------------+--------------------------+--------+------------------ 550e8400-... | 1970-01-17 05:08:19+0900 | yukim | well, well, well

©2012 DataStax

Changes worth noting

• Identifiers (keyspace/table/columns names) are always case insensitive by default• Use double quote(“) to force case

• Compaction setting is now map type

17

CREATE TABLE test ( ...) WITH COMPACTION = { 'class': 'SizeTieredCompactionStrategy', 'min_threshold': 2, 'max_threshold': 4

};

©2012 DataStax

Changes worth noting

• system.schema_*• All schema information are stored in system

Keyspace• schema_keyspaces, schema_columnfamilies,

schema_columns• system tables themselves are CQL3 schema

• CQL3 schema are not visible through cassandra-cli’s ‘describe’ command.• use cqlsh’s ‘describe columnfamily’

18

©2012 DataStax

More on CQL3 schema

• Thrift to CQL3 migration• http://www.datastax.com/dev/blog/thrift-to-cql3

• For better understanding• http://www.datastax.com/dev/blog/whats-new-in-cql-3-0• http://www.datastax.com/dev/blog/cql3-evolutions• http://www.datastax.com/dev/blog/cql3-for-cassandra-experts

19

©2012 DataStax

Mutating Data

20

INSERT INTO example (id, name) VALUES (...)

UPDATE example SET f = ‘foo’ WHERE ...

DELETE FROM example WHERE ...

• No more USING CONSISTENCY• Consistency level setting is moved to protocol

level

©2012 DataStax

Batch Mutate

• Batches are atomic by default from 1.2• does not mean mutations are isolated

(mutation within a row is isolated from 1.1)• some performance penalty because of batch

log process

21

BEGIN BATCH INSERT INTO aaa (id, col) VALUES (...) UPDATE bbb SET col1 = ‘val1’ WHERE ... ...APPLY BATCH;

©2012 DataStax

Batch Mutate• Use non atomic batch if you need

performance, not atomicity

• More on dev blog• http://www.datastax.com/dev/blog/atomic-batches-in-cassandra-1-2

22

BEGIN UNLOGGED BATCH ...APPLY BATCH;

©2012 DataStax

Querying Data

23

SELECT article_id, posted_at, authorFROM commentsWHERE article_id >= ‘...’ORDER BY posted_at DESCLIMIT 100;

©2012 DataStax

Querying Data

• TTL/WRITETIME• You can query TTL or write time of the column.

24

cqlsh:ks> SELECT WRITETIME(author) FROM comments;

writetime(author)------------------- 1354146105288000

©2012 DataStax

Collection support

• Collection• Set• Unordered, no duplicates

• List• Ordered, allow duplicates

• Map• Keys and associated values

25

©2012 DataStax

Collection support

• Collections are typed, but cannot be nested(no list<list<text>>)

• No secondary index on collections26

CREATE TABLE example ( id uuid PRIMARY KEY, tags set<text>, points list<int>, attributes map<text, text>);

©2012 DataStax

Collection support

27

INSERT INTO example (id, tags, points, attributes)VALUES ( ‘62c36092-82a1-3a00-93d1-46196ee77204’, {‘foo’, ‘bar’, ‘baz’}, // set [100, 20, 93], // list {‘abc’: ‘def’} // map);

©2012 DataStax

Collection support

28

UPDATE example SET tags = tags + {‘qux’} WHERE ...UPDATE example SET tags = tags - {‘foo’} WHERE ...

UPDATE example SET points = points + [20, 30] WHERE ...UPDATE example SET points = points - [100] WHERE ...

UPDATE example SET attributes[‘ghi’] = ‘jkl’ WHERE ...DELETE attributes[‘abc’] FROM example WHERE ...

• Set

• Map

• List

©2012 DataStax

Collection support

• You cannot retrieve item in collection individually

29

SELECT tags, points, attributes FROM example;

tags | points | attributes-----------------+---------------+-------------- {baz, foo, bar} | [100, 20, 93] | {abc: def}

©2012 DataStax

Collection support

• Each element in collection is internally stored as one Cassandra column

• More on dev blog• http://www.datastax.com/dev/blog/cql3_collections

30

©2012 DataStax31

Related topics

©2012 DataStax

Native Transport

• CQL3 still goes through Thrift’s execute_cql3_query API

• Native Transport support introduces Cassandra’s original binary protocol• Async IO, server event push, ...• http://www.datastax.com/dev/blog/binary-protocol

• Try DataStax Java native driver with C* 1.2 beta today!• https://github.com/datastax/java-driver

32

©2012 DataStax33

Question ?

Or contact me later if you have one yuki@datastax.com yukim (IRC, twitter) Now

Hiringtalented engineers from all

over the world!