N-O-SQL, new database technologies on the rise

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org

N-O-SQLnew database technologies on the rise

http://www.flickr.com/photos/wolfgangstaudt/2215246206/

Who am I

» Steven Noels - stevenn@outerthought.org

»Outerthought : scalable content applications

»makers of Daisy and Lily open source CMS

Agenda

» raison d’être: what brought us here

» concepts: required theory readings

»market overview: trees & the forest

» experiences and (h)in(d)sights

Raison d’être

History

hierarchical databases

OODBMS

XMLDB RDBMS

1. standardization

2. simplification

Inconsistency through slave lag

Scaling writes (1)

Scaling writes (2)

Issues with partitioning

» lose the ability to make arbitrary queries

» have to predict data access patterns when formulating partitioning strategy

» complex and fragile systems

Replication complexity

Scaling relational systems

»When scaling relational systems you loose their advantages but retain their overhead

History

RDBMS NOSQL

cachingdenormalisationshardingreplication ...

3. scaling

4. rethinkingthe problem

Moore vs Kryder

» seek time isconstant (networklatency as well?)

» transfer rate ! spindles !

» as a principle, writes arehard to scale

Cambrian Explosion

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 15

?Buzz-oriented development

Cambrian Explosion

N-O-SQL

The Perspective of Cost

Common themes

» SCALE SCALE SCALE

» new datamodels

» devops

»N-O-SQL

»The Cloud :technology is of no interest anymore

Numbers of scale

http://qos.doubleclick.net/counters/

Types of scaling

» scaling for usage» volume of users

» volume of data

availabilityreplication

» scaling types of ops» concurrent read

» concurrent write

partioningconsistency

distribution

Distributed systems are hard !

8 fallacies of distributed computing» The network is reliable.

» Latency is zero.

» Bandwidth is infinite.

» The network is secure.

» Topology doesn't change.

» There is one administrator.

» Transport cost is zero.

» The network is homogeneous.

New Data

» sparse structures

»weak schemas

» graphs

» semi-structured

» document-oriented

N-O-SQL =not only SQL !

The NOSQL footprint

free-structured or sparse data

referential integrity,typed data

ly scalable an

davailab

le (com

Cassandra

CouchDB

MongoDB

NOSQL, if you need ...

» horizontal scaling (out rather than up)

» unusually common data (aka free-structured)

» speed (especially for writes)

» the bleeding edge

SQL/RDBMS, if you need ...

» SQL

»ACID

» normalisation

» a defined liability

Theory

Robust systems

Academic background

»Amazon Dynamo

»Google BigTable

» Eric Brewer CAP theorem

Amazon Dynamo

» coined the term ‘eventual consistency’

» consistent hashing

Consistent hashing

http://horicky.blogspot.com/2009/11/nosql-patterns.html

Consistent hashing

- node C+ node D

http://www.lexemetech.com/2007/11/consistent-hashing.html

»multi-dimensional column-oriented database

» on top of GoogleFileSystem

» object versioning

Google BigTable

CAP theorem

strong consistency

highavailability

partition-tolerance

»Strong Consistency: all clients see the same view, even in the presence of updates

»High Availability: all clients can find some replica of the data, even in the presence of failures

»Partition-tolerance: the system properties hold even when the system is partitioned

Consistency

»Where is my data I just updated?

» Ideal world :

The result of every write-operation is reflected by subsequent read-operations.

Consistency

Sunny-day scenario

Network partioning

Culture Clash

»Classic distributed systems: focus on ACID

» atomic

» consistent

» isolated

» durable

»Modern internet systems: focus on BASE

» basically available

» soft-state (or scalable)

» eventually consistent

Culture Clash

»ACID» highest priority: strong

consistency for transactions

» availability less important

» pessimistic

» rigorous analysis

» complex mechanisms

» BASE» availability and scaling

highest priorities

» weak consistency

» optimistic

» best effort

» simple and fast

spectrum

Building for failure

» defensive programming

» creating replicas

» disk flushing

»watch out for failure of utility infrastructure

» conscious sync/async decisions

Possible storage failures

» Application errors

» Repeatable DB failures

» Unrepeatable DB failures

» OS errors

» Local cluster HW failure

» Local cluster network partitioning

» Disaster

» WAN network failure between remote clusters Mic

Availability ≠ total async !

The Enterprise Service Bus

✘bus =

congestion

Bus systems

» objects don’t fit in a pipe

» object ➙ message

» serialization / de-serialization cost

»message size

» queuing = cost

Use a mixture of both

»async + sync

stuff which matters !

Numbers of scale

http://qos.doubleclick.net/counters/

Processing large datasets :

Map/Reduce

Smart Data

» sparse as a feature

»weak schemas

» ad-hoc indexing

» organic analytics

» near-data processing

» live(ly) datawarehouse

» distribution ➙ parallellization ➙ performance

Hadoop: HDFS + MapReduce» single filesystem + single execution-space

MapReduce example: WordCount

MapReduce

MapReduce and HDFS

Physical architecture

Processing large datasets with MR

»Benefit from parallellisation

» Less modelling upfront (ad-hoc processing)

»Compartmentalized approach reduces operational risks

»AsterData et al. have SQL/MR hybrids for huge-scale BI

Market overview

Categories

» key-value stores

» column stores

» document stores

» graph databases

Key-value stores

»Redis

»Voldemort

»Tokyo Cabinet

»REmote DIctionary Server

» http://code.google.com/p/redis/

» vmware

Redis Features» persisted memcache, ‘awesome’

» RAM-based + persistable

» key ➙ values: string, list, set

» higher-level ops

» i.e. push/pop and sort for lists

» fast (very)

» configurable durability

» client-managed sharding

Voldemort

» http://project-voldemort.com/

» LinkedIn

Voldemort

» persistent

» distributed

» fault-tolerant

» hash table

Voldemort

API: GET, PUT,DELETE

Voldemort

routing logic moving up the stack,smaller latency

Voldemort data format

» key+values = arrays of bytes

» So how do we objects ⬌ bytes ?

» json

» string

» java-serialization

» protobuf

» identity

Tokyo Cabinet

» http://1978th.net/tokyocabinet/

»mixi.jp (i.e. Facebook Japan)

Product Family

Tokyo Cabinet

»memory or filesystem

» hash, b-tree, fixed-length, table

Column stores

»BigTable

»HBase

»Cassandra

BigTable

» http://labs.google.com/papers/bigtable.html

»Google

» layered on top of GFS

» http://hadoop.apache.org/hbase/

» StumbleUpon / Adobe / Cloudera

» sorted» distributed» column-oriented»multi-dimensional» highly-available» high-performance

» persisted» storage system

» adds random access reads and writes atop HDFS

HBase data model

»Distributed multi-dimensional sparse map

»Multi-dimensional keys:(table, row, family:column, timestamp) → value

»Keys are arbitrary strings

»Access to row data is atomic

Storage architecture

Cassandra

» http://cassandra.apache.org/

»Rackspace / Facebook

Cassandra

»Key-value store (with added structure)

»Reliability (identical nodes)

» Eventual consistent

»Distributed

»Tunable

» Partitioning

» Replication

Cassandra write pattern

Cassandra applicability

» Scalable reliability (through identical nodes)» Linear scaling»Write throughput» Large Data Sets

NO FIT

» Flexible indexing»Only PK-based

querying»Big Binary Data» 1 Row must fit in

RAM entirely

Document stores

»CouchDB

»MongoDB

»Riak

»MarkLogic

CouchDB

» http://couchdb.apache.org/

» couch.io

CouchDB

» fault-tolerant

» schema-free

» document-oriented

» accessible via a RESTful HTTP/JSON API

CouchDB documents

{ “_id”: ”BCCD12CBB”, “_rev”: ”AB764C”, “type”: ”person”, “name”: ”Darth Vader”, “age”: 63, “headware”: [“Helmet”, “Sombrero”], “dark_side”: true }

CouchDB REST API

»HTTP

» PUT /db/docid

»GET /db/docid

» POST /db/docid

»DELETE /db/docid

CouchDB Views»MapReduce-based

» Filter, Collate, Aggregate

» Javascript

function (Key, Values) { var sum = 0; for(var i in Values) sum += Values[i]; return sum; }

function (doc) { for(var i in doc.tags) emit(doc.tags[i], 1); }

map reduce

CouchDB

» be careful on semantics

» replication ≠ partioning/sharding !

» distributed database = distributable database

» sharded / distributed deploymentrequires proxy layer

MongoDB

» http://www.mongodb.org/

» 10gen

MongoDB

» cfr. CouchDB, really

» except for:

» performance focus

» runtime queries (mapreduce still available)

» native drivers (no REST/HTTP layering)

» no MVCC: update-in-place

» auto sharding (alpha)

» http://riak.basho.com/

»Basho Technologies

» buckets/keys, links

» values/content = bucket + metadata

» pluggable storage engines (fs, (D)ETS, InnoDB)

»HTTP/REST API

» automatic distribution

»mapreduce using Javascript

Jackrabbit

» http://jackrabbit.apache.org/

»Day Software

Jackrabbit

» reference implementation for JSR 170 & 283

» remoting: WebDAV & RMI

» persistence: RDBMS, fs, memory

Jackrabbit

» Java-centric (duh)

» complex repository model (nodes+properties)

»mixins, inheritance

»workspaces

» query language

» no partioning/sharding

JCR API levels

Graph databases

»Neo4j

»AllegroGraph (RDF)

» http://neo4j.org/

»Neo Technology

Neo4j» data = nodes + relationships + key/value properties

»many language bindings, little remoting

» ‘whiteboard’ friendly

» scaling to complexity (rather than volume?)

» lots of focus on domain modelling

» SPARQL/SAIL impl for triple geeks

»mostly RAM centric (with disk swapping & persistence)

Experiences & (h)in(d)sights

NOSQL applicability

»Horizontal scaling

»Multi-Master

»Data representation

» search of simplicity

» data that doesn’t fit the E-R model(graphs, trees, versions)

» Speed

Tools for the trade

» non-relational data: Couch, Mongo, Riak

»massive quantities: Cassandra, HBase

» persistent caching: Redis, Voldemort

» graphs: neo4j

Tool selection

» be careful on the marketeese:smoke and mirrors beware!

»monitor dev list, IRC, Twitter, blogs

»monitor project ‘sponsors’

»mix-and-match

»DON’T NOSQL WITHOUT INTERNAL SYS ARCHS & DEV(OP)S !

aptness

complexity

NOSQL}S

Our NOSQL-based project: Lily

» (open source)

» scalable store (Apache HBase)

» and search (Apache SOLR)

» content repository

»α due mid 2010

»www.lilycms.org or @outerthought

Lily architecture

Lily client

client

Lily store node

store node

distributed process coordination

and configuration (ZooKeeper)

query indexerupdate

WAL M/RMQ

documents2ary

indexes

Lily Store Server

HBase Region Server

Hadoop DFS

index replica

replica replica

} SOLR

inverted index

When combining store and search, make sure your (search) index doesn’t become the store.

Key lessons learned

» importance of keyspace design

» secondary indexing

» data de-normalization

» schema vs. code flexibility?

» distribution is everywhereand you shouldn’t forget about it

Reading material

»Amazon Dynamo, Google BigTable, CAP

» http://nosql.mypopescu.com/

» http://nosql-database.org/

» http://twitter.com/nosqlupdate

» http://highscalability.com/

Questions?

http://www.flickr.com/photos/leehaywood/4237636853/

» stevenn@outerthought.org

» @stevenn

Thanks for your attention !

N-O-SQL, new database technologies on the rise

Technology

Moving MS SQL Express Database to MS SQL Server. SQL Express Database to SQL Server.pdf · Moving MS SQL Express Database to MS SQL Server. Install Application Software & SQL Server

Azure SQL Database

SQL Database Project

Azure SQL Database Basics - files.meetup.com SQL Database.pdf · Azure SQL Database Basics Intro: Timothy P. McAliley ... Unit of Focus: SQL Server Managed Relational Database as

Database, SQL, & ADO.Net

SQL DML database

Sql database object

advanced sql(database)

Introduction to Database SQL & PL/SQL

Database, Database Objects and SQL

SQL Database Backup

SQL Database

Using SQL Standards? Database SQL comparition

Introducing Windows Azure - David Chappell€¦ · SQL Database For relational storage, Windows Azure provides SQL Database. Formerly called SQL Azure, SQL Database provides all of

CHAPTER 8 Database: SQL, MySQL. Topics Introduction Relational Database Model Relational Database Overview: Books.mdb Database SQL (Structured

Using a Database Advanced SQL SQL

SQL Azure Database

Go database/sql

Database SQL

Oracle Database SQL Quick Referencesidhanti/tutorials/oracle/B... · 2009-07-15 · Oracle Database SQL Language Reference Oracle Database PL/SQL Language Reference SQL*Plus User's