Introducing N1QL: Couchbase Connect 2015

INTRODUCING N1QL:QUERY WITHOUT COMPROMISEGerald Sangudi | @sangudiChief Architect, Query | CouchbaseTeam @N1QL

©2015 Couchbase Inc. 2

Agenda

The Nature of Data

Why Bring SQL to NoSQL

The N1QL Language

The Query Service

Benefits of N1QL for your Application and

Enterprise

Community and Participation

Getting Started

Q & A

The Nature of Data

©2015 Couchbase Inc. 4

Data Comes from the Real World

©2015 Couchbase Inc. 5

Properties of Real-World Data

Rich structure Attributes, Sub-structure

Relationships To other data

Value evolution Data is updated

Structure evolution Data is reshaped

Person

Name

DOB

Billing

Connections

Purchases

Jane Smith

Jan-30-1990

So Why Bring SQL to NoSQL?

©2015 Couchbase Inc. 7

Models for Representing Data

Data Concern Relational Model JSON Document Model (NoSQL)

Rich Structure Multiple flat tables Constant assembly /

disassembly

Documents No assembly required!

Relationships Represented Queried (SQL)

Represented Queried? Not until

now…

Value Evolution

Data can be updated Data can be updated

Structure Evolution

Uniform and rigid Manual change

(disruptive)

Flexible Dynamic change

©2015 Couchbase Inc. 8

For the Best of Both, Bring SQL to NoSQL

From JSON Rich structure, no assembly

Schema flexibility

From SQL General, proven, and pervasive query capabilities Querying across relationships

The Result: N1QL: SQL for JSON

The N1QL Language

©2015 Couchbase Inc. 10

The N1QL Language

Goal

Give developers and enterprises an

expressive, powerful, and complete

language for querying, transforming,

and manipulating JSON data.

N1QL Features from SQL

©2015 Couchbase Inc. 12

SELECT Statement

SELECT [ DISTINCT ] …

FROM … JOIN …

WHERE …

GROUP BY … HAVING …

ORDER BY …

LIMIT …

OFFSET …

( UNION | INTERSECT | EXCEPT )

[ ALL ] …

©2015 Couchbase Inc. 13

SELECT Statement Highlights

Querying across relationships JOINs

Subqueries

Aggregation

MIN, MAX

( SUM, COUNT, AVG, ARRAY_AGG ) [ DISTINCT ]

Combining result sets using set

operators

( UNION, INTERSECT, EXCEPT ) [ DISTINCT ]

©2015 Couchbase Inc. 14

Data Modification Statements

UPDATE … SET … WHERE …

DELETE FROM … WHERE …

INSERT INTO … ( KEY, VALUE ) VALUES …

INSERT INTO … ( KEY …, VALUE … ) SELECT …

MERGE INTO … USING … ON …

WHEN [ NOT ] MATCHED THEN …

Note: Couchbase Server provides per-

document atomicity.

©2015 Couchbase Inc. 15

Index Statements

CREATE INDEX ON …

DROP INDEX …

EXPLAIN …

Highlights

Functional indexes

on any data expression

Partial indexes

©2014 Couchbase, Inc. ©2015 Couchbase Inc. 16

N1QL Expressions from SQL

16

Literals• Primitives [ 0, ‘hello’, TRUE ]

• NULL

Operators

• Arithmetic [ +, -, *, /, % ]

• Logical [ AND, OR, NOT ]

• Comparison [ <, <=, =, !=, >=, >, BETWEEN, IS NULL ]

• Pattern matching [ LIKE ]

• Conditional [ CASE ]

Scalar functions

• Numeric [ trigonometric, ROUND, TRUNC, … ]

• String [ UPPER, LOWER, TRIM, SUBSTR, … ]

• Date [ string and numeric dates, NOW, date arithmetic, …

]

Aggregate functions

• MIN, MAX, SUM, AVG, COUNT [ DISTINCT ]

Subqueries • Subqueries are full expressions

N1QL Extensions to SQL

©2015 Couchbase Inc. 18

N1QL Query Operators [ 1 of 2 ]

USE KEYS …

Direct primary key lookup bypassing index scans

Ideal for hash-distributed datastore

Available in SELECT, UPDATE, DELETE

JOIN … ON KEYS …

Nested loop JOIN using key relationships

Ideal for hash-distributed datastore

©2015 Couchbase Inc. 19

N1QL Query Operators [ 2 of 2 ]

NEST Special JOIN that embeds external child documents

under their parent Ideal for JSON encapsulation

UNNEST

Flattening JOIN that surfaces nested objects as top-

level documents

Ideal for decomposing JSON hierarchies

JOIN, NEST, and UNNEST can be chained in any

combination

©2014 Couchbase, Inc. ©2015 Couchbase Inc. 20

N1QL Expressions for JSON

20

Ranging over collections

• WHERE ANY c IN children SATISFIES c.age > 10 END

• WHERE EVERY r IN ratings SATISFIES r > 3 END

Mapping with filtering • ARRAY c.name FOR c IN children WHEN c.age > 10 END

Deep traversal, SET, and UNSET

• WHERE ANY node WITHIN request SATISFIES node.type = “xyz”

END

• UPDATE doc UNSET c.field1 FOR c WITHIN doc END

Dynamic Construction

• SELECT { “a”: expr1, “b”: expr2 } AS obj1, name FROM … //

Dynamic object

• SELECT [ a, b ] FROM … // Dynamic array

Nested traversal • SELECT x.y.z, a[0] FROM a.b.c …

IS [ NOT ] MISSING • WHERE name IS MISSING

©2015 Couchbase Inc. 21

N1QL Data Types from JSON

N1QL supports all JSON data types

Numbers

Strings

Booleans

Null

Arrays

Objects

©2015 Couchbase Inc. 22

N1QL Data Type Handling

Non-JSON data types

MISSING

Binary

Data type handling

Date functions for string and numeric

encodings

Total ordering across all data types

Well defined semantics for ORDER BY and comparison

operators

Defined expression semantics for all input data

types

No type mismatch errors

The Query Servicein Couchbase Server 4.0

©2015 Couchbase Inc. 24

Topology of Couchbase Server 4.0

Client SDK

App

Index

CB Node

ODBC / JDBC

App

Query

Manager

Data

Index

CB Node

Query

Manager

Data

Index

CB Node

Query

Manager

Data

Index

CB Node

Query

Manager

Data

Index

CB Node

Query

Manager

Data

Index

CB Node

Query

Manager

Data

Multi-dimensional scaling

Query throughput scales with nodes

Multiple connectivity options

ODBC / JDBC

BI Tool

©2015 Couchbase Inc. 25

Query Engine

Query Processor

Inside a Query Engine

Service Listener

Parser Optimizer

Data Stores

Execution Engine

Couchbase Server

Auth DataIndexers

GSIView

s

Others…

8093/18903

File systemData Service

Index Service

......

Cluster Manager

Bucket#1

Bucket#2

Index#2

Index#1

Admin Listener

©2015 Couchbase Inc. 26

Query Request Processing

Index Services

Clients

Data Services

(1) Request (7) Response

Query Services

(3) Plan: AST to execution plan

(6) Evaluate: Documents to results

(4) Scan: Attributes to keys (5) Fetch: Keys to documents

- Prepared

statements

- Streaming, no

cursors (2) Parse: Query to AST

©2015 Couchbase Inc. 27

Query Execution

Client

FetchParse Plan Join FilterPre-Aggregate

Offset Limit ProjectSortAggregateScan

Query ServiceInde

x Servi

ce

Data Servi

ce

Data-parallel — Query latency scales with cores

In-memory, streamed

Pluggable architecture — datastore, indexer, client, …

Benefits of N1QLfor your Application and Enterprise

©2015 Couchbase Inc. 29

Benefits for your Application and Enterprise Model your Data Cleanly

Model once, query Use both relationships and embedding

Query your Data with Flexibility Query across structure, relationships, and datasets Query across change and heterogeneity

Develop Rich Applications with Agility Deliver features that transform, combine, and aggregate data Use your favorite clients, frameworks, and interfaces

Integrate your Applications and Data Leverage ecosystem: Connectivity, Analytics, Cloud and Packaged Apps

Community and Participation

©2015 Couchbase Inc. 31

Community and Participation

Enterprise and Community Beta Programs

Community Beta Work with CB engineers and influence the product Showcase your use case

Community Ecosystem Build complementary tools, products, and drivers Build and integrate via our open APIs

©2015 Couchbase Inc. 32

Community and Participation in Action

Couchbase N1QL Query Runner by Warren Postma, Beta customer 30-minute quick start with Python’s Tkinter GUI package

Getting Started

©2015 Couchbase Inc. 34

Getting Started

Test drive Couchbase Server 4.0 Beta

Visit query.couchbase.com

Play with the online tutorial

Ask us questions Couchbase forums, Stack Overflow, @N1QL

N1QL Sessionsat Couchbase Connect ‘15

©2015 Couchbase Inc. 36

N1QL Customer Sessions

Wed June 3 Thurs June 4

9:30amKeynote

9:45amKeynote

3:45pmDeveloper

1:45pmDeveloper

©2015 Couchbase Inc. 37

N1QL Partner Sessions

Wed June 3 Thurs June 4

1:00pmBig Data

1:45pmBig Data

3:45pmBig Data

3:45pmBig Data

4:30pmBig Data

©2015 Couchbase Inc. 38

Wed June 3 Thurs June 4

1:45, 3:45pmArchitecture

1:00pmDeveloper

2:30pmArchitecture

1:45pmArchitecture

4:30pmDeveloper

4:30pmBig Data

Sampling of N1QL Sessions by Couchbase

Indexing:Intro & Deep Dive

N1QL: Internals & Power FeaturesN1QL in Dev SDKs:Java, .NET, Node.js

Migrating: MySQL to N1QL

Tuning Query Performance

Big DataQuery Landscape

Q & A

Thank you.

Gerald Sangudi, Couchbase@sangudi

©2015 Couchbase Inc. 41