BIG DATA IN FINANCETomasz Bednarz

���1

OVERVIEW OF THIS TALK

���2

• Flavors of big data in finance • Technology map for big data in finance • Wheel of reincarnation • A touch of history • Role of visualisation • Frameworks • Real Time HFT

BIG DATA AGAIN• Big data is not a “crystal ball”

• The value from big data can only be extracted when there is precise business problem to be addressed

• Business problem has to be understood and well defined at the first place

• Business to data mapping • Better business predictions = difficult process

• Analysing big data • Data scientists to examine the data and extract critical information such as customers buying habits - need for experts

• Catalogue data assets (10% of data may actually mean something)

• Privacy • No clear access controls to access data?

Volume

VelocityVarietyBig Data

SURVEY

http://strata.oreilly.com/2012/01/enterprise-big-data-survey-results.html

FLAVORS OF BIG DATA IN FINANCE

• Structured • Market data feeds

• Big and fast • Order flows • Execution info

• Unstructured • News & • Research

Dad Bailey’s news stand in 1939. Comics are on the top left.

���5

TECHNOLOGY MAP FOR DATA

Information Retrieval Analytics

Loosely Structured Information

Highly Structured Information

Human Computer

Market Interface

= high frequency ticks/trading & order flow

= text and relationships

���6

TECHNOLOGY MAP FOR DATA

Information Retrieval Analytics

Loosely Structured Information

Highly Structured Information

Human Computer

Market Interface

= text and relationships

= high frequency ticks/trading & order flow

fast: trading

���7

TECHNOLOGY MAP FOR DATA

Information Retrieval Analytics

Human Computer

Market Interface

= text and relationshipsLoosely Structured Information

Highly Structured Information = high frequency ticks/trading & order flow

slow: investment research, portfolio management

���8

SKILL ZONES - HUMAN VERSUS MACHINE

���9

Time to exploit

information

Data Subtlety & Complexity. Language & Concept Importance.

ms

months

low highpure HPC (HFT)

collaborative hybrid generation

smart market

monitoringalgo

gurus

Visualizing Marathon 2011CSIRO GPU Cluster

Reinvent tech/self every ~three years!!!

WHEEL OF REINCARNATION

���10

T. H. Myer and I. E. Sutherland. On the design of display processors. Communications of ACM, June 1968, 410-414.

STRUCTURED DATA

���11

���12

NYSENew York Stock Exchange started as very low-tech place. In 1779, the NYSE was a bunch of guys standing around a buttonwood tree at 68 Wall Street shouting at each other on days when it didn’t rain or snow.

In 1794 we see the first big technological solution: the roof.

NYSEEverybody moved inside, to the Tontine Coffee House at the corner of Water and Wall streets. Hands, roofs, chalk (oh technology)!

1n 1823, the Difference Engine was invented by Charles Babbage “I wish to

God these calculations had been executed by steam.”

TECHNOLOGICAL INVASION - SIMPLER ERA

���14

Before telegraphy, in 1850s, the sky over Wall Street was open and clear

It took only a short time for telegraphy’s compression of time and space to transform

the scenery. Everybody had to have it!

BUT you had to know Morse Code to participate in the market! (Morse, 1837)

TICKER TAPEA huge success (as important as the roof, hand signals, and the telegraph). People use jumbo magnifying lenses - people traded faster than the machines, so delay meters were installed.

All that ticker tape also made for nice parades. Here group of specialists

celebrating the one-millionth bagging of a buy-side trader.

AND MORE TECHNOLOGYTicker tapes (an enduring market visualisation) are still with us today.

Type 80 Card Sorter IBM 1925

Our CSIRAC 1 operation per second - 1949

ROLE OF VISUALISATION

Dynamic market view: Oculus Info’s Visible Marketplace - portfolio visualisation. !Not very dynamic on paper. !Another reason to be glad we have web browsers.

MAP OF THE MARKET

Map of the Market , the classic big picture of whole market visualization. It was invented by Marten Wattenberg, now at IBM Many Eyes.

Cultural Finance

Finance Cover Letter

UNSTRUCTURED DATA

���20

TECHNOLOGY MAP FOR DATA

Information Retrieval Analytics

Human Computer

Market Interface

= text and relationshipsLoosely Structured Information

Highly Structured Information = high frequency ticks/trading & order flow

slow: investment research, portfolio management

���21

INFORMATION SOURCES• Specialised Industry Media • Local and International Media • Direct Corporate Communications • Research Labs • Government Agencies • Social Media • Crowd Sourcing !

• HTML / Text Feeds / XML

BIG DATA AT NICTA• Developed to deliver customized business intelligence and advanced data mining to the Enterprise marketplace

• Machine Learning to obtain deep insights in real time

• Build upon a scalable open source software platforms

• See http://www.youtube.com/watch?v=TxQPdUt_x3c • Scoobi - a Scala productivity framework for Hadoop (allows you to write

what you want rather than how to do it). • Source code: https://github.com/nicta/scoobi

• Demo of Scoobi tomorrow at the hands-on afternoon session (by Piotr)

http://www.ambiata.com/index.html

���24

http://www.zdnet.com/how-is-big-data-faring-in-the-enterprise-7000002404/

REAL-TIME REAL-TIME REAL-TIME

• Everything is moving to real time

• Everything is moving towards continuous time

• Everything is moving towards mobility (anywhere, anytime)

• Also GPU for high-frequency trading (HFT)

• GPU alongside FPGAs receiving stream of market data to increase the accuracy of the strategy, or even to suggest change in the algorithm used as market conditions evolve

• GPU Direct enables a GPU to talk directly to the FPGA, receiving data from it into a kernel, which may be preloaded to perform its magic once the data is available

• Hadoop + GPUs

ACKNOWLEDGEMENTS

• Andrew Sheppard, Fontainhead

• David Leinweber, The Lawrance Berkley National Lab, author of Nerds on Wall Street NOWS

• Colleagues from Westac and Commonwealth banks

• Craig Mudge for organising Big Data workshop, and always great discussions

���27

THANK YOU