Upload
mongodb
View
2.875
Download
4
Embed Size (px)
DESCRIPTION
Learn how MongoDB can help ingest high volume financial services messages. See how we can consume FIX, FpML and Swift message formats and handle their ever-changing nature. We will be delivering this session with our partner C24.
Citation preview
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
Title Text
• John Davies (@jtdavies) | CTO - C24• Trisha Gee (@trisha_gee) | 10gen
•Webinar• 4th December 2012
Processing High Volume Data Feeds with MongoDB & C24
1Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
Agenda...
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
• C24 iO - Introduction
• C24 iO - Demo
2Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 TechnologiesConfidential Information of C24 Technologies Ltd.
Typical Usage
© 2012 C24 Technologies Confidential Information of C24 Technologies7
Import or model any data formatsXML Schema, DTD, XML, Relax-NG, RDBMS, Java Classes, Text Files, Tag/Value, Delimited (CSV), Fixed, Binary, Proprietary Canonical Formats
Choose from hundreds of pre-modelled Standards SWIFT-MT, SWIFT-MX, ISO-20022, SEPA, FIX, FpML, DTCC, MINOS, CBI
Create and manage syntactic & semantic validation rulesFlexible technology, Java-based, Full XPath 2.0 support, Scala, Groovy, Clojure etc.
Compare models & manage versionsGraphical hierarchical comparison
For DevelopersBound Java ObjectsFully Documented APIGetters / SettersXPath & XQuery APIThousands of JUnit testsANT & Maven Build for C.I.Integration with Camel / Fuse, Spring Integration / Batch, Mule, GemFire, GigaSpace, Coherence, Oracle, MongoDB and others
ProductionFully tested JAR
For Business AnalystsA Powerful GUI for managing, modelling and transformation
Export Documents & Reports
3Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
SWIFT Example (MT541)
{1:F01DRESGB2LAXXX0548034693}{2:I541DRESDEFFXXXXN2}{3:{108:invalid:c3}}{4::16R:GENL:20C::SEME//FRTJ123REC2:23G:NEWM:99B::TOSE//100:16S:GENL:16R:TRADDET:98A::TRAD//20000519:98A::SETT//20000524:90A::DEAL//PRCT/101,001283:35B:ISIN GB0987654321:16S:TRADDET.........:95P::PSET//CRSTGB22:16S:SETPRTY:16R:AMT:19A::SETT//GBP4047151,3:16S:AMT:16S:SETDET-}
4Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
Transformation Example
5Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
Model Visualisation
•Our old friend the MT541 - Receive Against Payment
6Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
Graphical Navigation
• Some more complex than others• A Fix 5.0 SP2 Trade Capture Report (One of the
simpler messages)
7Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
So why are we here?
• Take any one of these technologies...•MongoDB, Neo4J, Coherence, HazelCast, Redis, GemFire, GigaSpaces,
Terracotta, MemcacheD etc.• Even a classic RDBMS, in fact “sadly”, usually a RDBMS
• A bunch of complex messages...• XML Schema, DTD, XML, Relax-NG, RDBMS, Java Classes, Text Files, Tag/
Value, Delimited (CSV), Fixed, Binary, Proprietary Canonical Formats• SWIFT MT/MX, ISO-20022/SEPA, FIX, FpML, DTCC, MINOS, CBI etc.
• And you have an integration problem...• How, for example, do you get your internal canonical messages into
MongoDB?• How do you store your front office Fix messages, in any way?
8Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
It’s a Very Simple Solution
• To load a CSV into an RDBMS is simple - 2D data anyway• There are loads of tools to do the job, usually from the database vendor• ETL (Extract, Transform, Load) is the classic tool for the RDBMS
• Complex (Hierarchical or 3D) data is difficult to map to a RDBMS•Minor changes in the data require awkward table redesign every time
•MongoDB is to complex data what an RDBMS is to a CSV•We are no longer restricted by table schema, multi-versions possible• But we need new ETL tools
•We could chuck everything in as a String or byte array• But we still need to understand the content if we are to be able to build
queries and extract what interests us
9Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
Technical Overview
• As I said, it’s very simple with this technology combination...• Load the messages into a pre-built C24 model and convert into JSON• The JSON format is built in•
Message queueMQ Series / JMS
Files Reader
C24 Parse
C24 Validate
Save to MongoDB
10Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
Fix
• Take a FIX message...
8=FIXT.1.1 9=1 35=AE 1128=7 49=STRING 56=STRING 90=1 91=D 34=1 50=STRING 142=STRING 57=STRING 143=STRING 144=STRING 145=STRING 52=20020101-00:00:00.000 122=20020101-00:00:00.000 212=1 213=D 347=ISO-2022-JP 369=1 627=1 628=STRING 629=20020101-00:00:00.000 630=1 571=STRING 487=0 568=STRING 150=0 572=STRING 17=STRING 527=STRING 378=0 570=Y 55=STRING 65=CD 48=STRING 22=1 454=1 455=STRING 456=1 460=1 461=STRING 167=FAC 200=200201 541=20020101 224=20020101 225=20020101 239=CORP 226=1 227=1 228=1 255=STRING 543=STRING 470=AF 471=GB 472=STRING 240=20020101 202=1 206=0 231=1 223=1 207=XLON 106=STRING 348=1 349=D 107=STRING 350=1 351=D 32=1 31=1 15=AFA 120=AFA 194=1 195=1 30=XLON 75=20020101 60=20020101-00:00:00.000 768=1 769=20020101-00:00:00.000 1033=A 1034=1 1035=E.W 63=0 64=20020101 573=0 574=A1 552=1 54=1 453=1 448=STRING 447=B 452=1 802=1 523=STRING 1=STRING 581=1 81=0 575=Y 576=1 577=0 578=STRING 579=STRING 376=STRING 377=Y 582=1 336=STRING 625=STRING 12=1 13=1 479=AFA 497=Y 157=1 230=20020101 158=1 159=1 238=1 237=1 118=1 119=1 155=1 156=M 77=O 58=STRING 354=1 355=D 518=1 519=1 520=1 521=AFA 136=1 137=1 138=AFA 139=1 37=STRING 198=STRING 11=STRING 38=1 468=0 469=1 528=A 529=1 483=20020101-00:00:00.000 381=1 10=000
• Relatively simple as they go but to get this into a database in a useful format is not trivial
A reminder that it’s not flat, even Fix has hierarchy...
11Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
Fix
• Take a FIX message...
8=FIXT.1.1 9=1 35=AE 1128=7 49=STRING 56=STRING 90=1 91=D 34=1 50=STRING 142=STRING 57=STRING 143=STRING 144=STRING 145=STRING 52=20020101-00:00:00.000 122=20020101-00:00:00.000 212=1 213=D 347=ISO-2022-JP 369=1 627=1 628=STRING 629=20020101-00:00:00.000 630=1 571=STRING 487=0 568=STRING 150=0 572=STRING 17=STRING 527=STRING 378=0 570=Y 55=STRING 65=CD 48=STRING 22=1 454=1 455=STRING 456=1 460=1 461=STRING 167=FAC 200=200201 541=20020101 224=20020101 225=20020101 239=CORP 226=1 227=1 228=1 255=STRING 543=STRING 470=AF 471=GB 472=STRING 240=20020101 202=1 206=0 231=1 223=1 207=XLON 106=STRING 348=1 349=D 107=STRING 350=1 351=D 32=1 31=1 15=AFA 120=AFA 194=1 195=1 30=XLON 75=20020101 60=20020101-00:00:00.000 768=1 769=20020101-00:00:00.000 1033=A 1034=1 1035=E.W 63=0 64=20020101 573=0 574=A1 552=1 54=1 453=1 448=STRING 447=B 452=1 802=1 523=STRING 1=STRING 581=1 81=0 575=Y 576=1 577=0 578=STRING 579=STRING 376=STRING 377=Y 582=1 336=STRING 625=STRING 12=1 13=1 479=AFA 497=Y 157=1 230=20020101 158=1 159=1 238=1 237=1 118=1 119=1 155=1 156=M 77=O 58=STRING 354=1 355=D 518=1 519=1 520=1 521=AFA 136=1 137=1 138=AFA 139=1 37=STRING 198=STRING 11=STRING 38=1 468=0 469=1 528=A 529=1 483=20020101-00:00:00.000 381=1 10=000
• Relatively simple as they go but to get this into a database in a useful format is not trivial
12Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
Using C24-iO
• Select or import the model (Fix in this example)
•Deploy the code (one click)• This generates a Fix endpoint jar that is capable of reading, validating and
writing out Fix messages - in multiple formats• Plus of course a rich API, XPath 2.0 and Spring Integration• XPath, like QL offers a single language to filter/query/sort any data structure
• From a small snippet of code we can read in a Fix message (into our Jar) and write it directly into MongoDB• It’s even simpler with Spring Integration & Spring Data - no code (just XML)
•With Spring Integration / Data we can load Fix messages into MongoDB without any code at all
13Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
Seconds later...
• Select the output format as JSON and..."StandardHeader":{ "SendingTime":{ "withMilliseconds":"Tue Jan 01 00:00:00 GMT 2002" }, "OrigSendingTime":{ "withMilliseconds":"Tue Jan 01 00:00:00 GMT 2002" }, "XmlData":{ "XmlDataLen":1, "XmlData":"D" }, "MessageEncoding":"ISO-2022-JP", "LastMsgSeqNumProcessed":1, "HopGrp":{ "NoHops":1, "HopGrpFields":{ "HopCompID":"STRING", "HopSendingTime":{ "withMilliseconds":"Tue Jan 01 00:00:00 GMT 2002"
14Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
Fix into MongoDB
• In code (for the techies)...
15Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
Or if you prefer in Spring...
•We literally can take Fix (SWIFT, FpML, CSV etc.) messages from a queue/file and write them to MongoDB with NO CODE!
16Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
Some Important Links...
• The C24 iO and MongoDB page•White Paper, Log4J & Fix examples
• C24-iO• http://www.c24.biz• John Davies (@jtdavies) | CTO - C24
• 10gen• http://www.10gen.com• Trisha Gee (@trisha_gee) | 10gen
http://www.c24.biz/io-mongodb
17Tuesday, 4 December 12
© 2012 C24 Technologies Confidential Information of C24 Technologies Ltd.
Thank you
© 2012 C24 Technologies Confidential Information of C24 Technologies18Tuesday, 4 December 12