Tech Talk_25.04.15_Session 2_Martin Toshev_KDB database

1

KDB Database

Martin Toshev 25.04.2015

2

Agenda

KDB Database Overview

Q Lang Basics

3

Agenda

KDB vs classical RDBMS

KDB Java API

KDB Use Cases

4

KDB Database Overview

5

KDB database overview

KDB:

• an in-memory column-based database

• can be used to handle large volumes of real-time or historical data

• provides OLTP and OLAP capabilities

6


KDB:

• provides much of the same functionality of an RDBMS

• supports SQL and KSql (for column-based queries)

7


Used by almost any large financial institution …

• Goldman Sachs

• Morgan Stanley

• J.P. Morgan

• Deutsche Bank

• Commerzbank AG

8


KDB+:

• 64bit release of KDB

• OLTP: 100 000 – 1 000 000 records per second per CPU

• OLAP: 1 000 000 – 10 000 000 records per second per CPU

• Backed by the Q general purpose programming language

9


KDB+:

• multi-threaded and multi-process (scales vertically)

• can be deployed in a distributed environment (scales horizontally)

• provides multiple APIs for C++, Java, Perl, Python …

• and provides support for JDBC …

10

Q Lang Basics

11

Q Lang Basics

Q language:

• vector processing language

• allows for complex calculations across a large volume of data

• merges the K and ksql languages

12

Q Lang Basics

Q language:

• Interpreted (a mixture of APL and Scheme)

• simple rules of precedence (right -> left evaluation of all expressions)

• functions are first-class citizens (can be assigned to variables)

13

Q Lang Basics

Q language data types:

• Integer

• floating point

• char

• symbol

• lists

• dictionaries

• time • datetime • date

• others …

14

Q Lang

Demo

15


16


KDB is column-oriented while RDBMS are row-oriented

Both support SQL for querying the database

17


Classical RDBMS are in control of the persistence

In KDB+ the user is in control of the persistence

18


No concept of transactions in KDB+

Queries in KDB+ are always SERIALIZABLE in terms of

isolation levels

19


KDB provides utilities for dealing with time-series data:

• time joins (af – asof join, wj – window join)

20

KDB Java API

21

KDB Java API

Extremely simple …

A single class c that represents a KDB+ connection

22

KDB Java API

c.ks() – for executing an operation without waiting for a

response

c.k() – for execution an operation and waiting for a response

c.k() – for waiting for a message from the KDB+ server

(naming is somewhat dummy … )

23

KDB Java API

// getting the 'tbl' table

c connection = new c("localhost", 5000);

Flip tbl = (Flip) connection.k("tbl");

for(String key : tbl.x) {

System.out.println(key + " " +

tbl.at(key).getClass().getSimpleName());

}

24

KDB Java API

// inserting a record in the 'tbl‘ table

connection.ks("`tbl insert (`x;10)");

25

KDB Java API

Demo

26

KDB Use Cases

27

KDB Use Cases

Capturing, storing and analyzing massive volumes of data … (kdb+tick)

Kdb+tick tickerplant

Market data feeds

Other feeds

KDB+ log files

Kdb+tick rdb

Java/C/ Python

Kdb+tick hdb

28

KDB Use Cases

Analyzing FIX messages for post trade analytics and compliance (kdb+tick)

Kdb+tick tickerplant

Market data feeds

Other feeds

KDB+ log files

Kdb+tick rdb

Java/C/ Python

Kdb+tick hdb

OMS (Order Management

System)

29

KDB Use Cases

Importing and analyzing massive volumes of data from an exchange … (kdb+taq)

Kdb+tick

taq

NYSE

30

Thank you

31

Resources

Wikipedia’s entry on KDB

http://en.wikipedia.org/wiki/KDB

KDB wiki

http://code.kx.com/wiki/Main_Page

KDB-tick overview

http://kx.com/papers/Kdb+tickDiagram.pdf

KDB guides

http://www.timestored.com/kdb-guides/











32

Resources

K language

http://www.math.bas.bg/bantchev/place/k.html

KDB for Gods

http://www.firstderivatives.com/lecture_series.asp

KDB+ function reference

http://code.kx.com/wiki/Reference







33

Resources

Kdb+ Database and Language Primer

http://kx.com/q/d/primer.htm

KDB+ Database Reference Manual

http://kx.com/q/d/kdb+1.htm

Q language reference manual

http://kx.com/q/d/q1.htm

Abridged KDB+ Database Manual

http://kx.com/q/d/kdb+.htm

Abridged Q Language Manual

http://www.kx.com/q/d/q.htm











34

Resources

Kdb database tutorial

http://www.thalesians.com/finance/index.php/Knowledge_Base/Databases/Kdb

A shallow introduction to the K language

http://www.kuro5hin.org/?op=displaystory;sid=2002/11/14/22741/791

K as a prototyping language

http://www.cs.nyu.edu/courses/fall02/G22.3033-007/kintro.html









Technology

Tech Talk_25.04.15_Session 2_Martin Toshev_KDB database