View
3.553
Download
3
Category
Tags:
Preview:
Citation preview
WEBINAR:
Comparing NoSQL Databases
for Real-time Bidding
Presented by
Sergey Zhemzhitsky,
CTO of CleverDATA
cleverdata.ru | info@cleverdata.ru
International market business development since 2012
One of three leading IT companies in Russia
43 branches in Russia and abroad
+5500 employees
100K projects for 10K customers
Data management innovative
platform (Data Exchange Service)
Cloud Service
In-house development
Internet advertising solutions
Data Management Platforms
Customers Base Management
Web Analytics
Marketing automation
Big DataData Mining
Digital IntelligenceOperational Intelligence
Low Latency and NoSQLCloud Computing
cleverdata.ru | info@cleverdata.ru
Agenda
• RTB intro;• Challenges;• Choice difficulties;• Results;• Do’s and Don’ts.
cleverdata.ru | info@cleverdata.ru
Real Time Bidding (RTB)
publishers
AD NETWORKAD NETWORK
AD NETWORK
AD NETWORKAD NETWORK
AD NETWORK
advertisers
cleverdata.ru | info@cleverdata.ru
How RTB works
publishers
AD NETWORKAD NETWORK
AD NETWORK
AD NETWORKAD NETWORK
AD NETWORK
advertisers
cleverdata.ru | info@cleverdata.ru
Demand Side Platform (DSP)
advertiserspublishers
AD NETWORKAD NETWORK
AD NETWORK
AD NETWORKAD NETWORK
AD NETWORK
DSP
cleverdata.ru | info@cleverdata.ru
Supply Side Platform (SSP)
publishers
AD NETWORKAD NETWORK
AD NETWORK
AD NETWORKAD NETWORK
AD NETWORK
advertisers
SSP
TRACKING DATA
cleverdata.ru | info@cleverdata.ru
Data Management Platform (DMP)
publishers
COOKIE SYNCs
ACCESS LOGS
PARTNERS’ DATA
3rd PARTY DATA
CLICK STREAMS
advertisers
SSP
DSP
WHATEVER ELSE DATA
cleverdata.ru | info@cleverdata.ru
RTB challengesvisitors publishers
advertisers
10
0 m
s
20 ms
10.000+ rps
TRACKING DATA
cleverdata.ru | info@cleverdata.ru
publishers
COOKIE SYNCs
ACCESS LOGS
PARTNER’S DATA
3rd PARTY DATA
CLICK STREAMS
advertisers
SSP
DSP
DMP in the RTB ecosystem
DMP
cleverdata.ru | info@cleverdata.ru
Choice criteria
• Linear scalability;• Sharding “out of the box”;• Distributed by design;• Data redundancy & replication;• Low latency.
cleverdata.ru | info@cleverdata.ru
Aerospike Cassandra Mongo Redis
Scalability
Sharding
Redundancy
Latency
SPoF
Maintainability
Monitoring tools
Features
cleverdata.ru | info@cleverdata.ru
nginx 1.2.xngx-http-redislua-nginx-modulelua-resty-mongolngx-aerospike
aerospike 2.xredis 2.6.xmongodb 2.4.xwrkiperfnmon
• Intel® Core™ i7-920 Quad-Core• 48 GB RAM • 1 Gbit/s NICs (*)• 20 * 106 msgs• 512 bytes msg
* 200 Mbit/s guaranteed
Tools & Materials
cleverdata.ru | info@cleverdata.ru
• No logging;• No plugins;• CPU Affinity.
NGINX
WRK
Measuring Nginx
cleverdata.ru | info@cleverdata.ru
Nginx results. Base line
L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s
1.35 1.46 1.64 1.90 68851
cleverdata.ru | info@cleverdata.ru
Measuring Redis
NGINX
WRK
Node 1Shard 1
Shard 2
Slave 3
Slave 4
Node 2Shard 3
Shard 4
Slave 5
Slave 6
Node 3Shard 5
Shard 6
Slave 2
Slave 1
cleverdata.ru | info@cleverdata.ru
Redis results
L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s
2.68 3.10 3.25 3.90 34769
cleverdata.ru | info@cleverdata.ru
Measuring Mongo
NGINX :: MONGOS
WRK
Node 1mongod :: cfg
mongod :: repl
mongod :: repl
mongod :: repl
Node 2mongod :: cfg
mongod :: repl
mongod :: repl
mongod :: repl
Node 3mongod :: cfg
mongod :: repl
mongod :: repl
mongod :: repl
Shard 1Replica Set 1
Shard 2Replica Set 2
Shard 3Replica Set 3
cleverdata.ru | info@cleverdata.ru
Mongo results
L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s
6.70 8.22 10.22 15.46 14220
cleverdata.ru | info@cleverdata.ru
Node 1
Chunk 1
Replica 2
Node 2
Chunk 2
Replica 3
Node 3
Chunk 3
Replica 1
NGINX
WRK
Measuring Aerospike
cleverdata.ru | info@cleverdata.ru
L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s
8.93 14.99 26.83 106.48 3402
Aerospike results
cleverdata.ru | info@cleverdata.ru
Wks L, 50% L, 75% L, 90% L, 99% T, r/s Ngx, CPU Ngx, p/s As, CPU As, p/s
4 8.22 10.22 15.46 97.3 3402 7 8 K 1 1 K
8 7.89 15.16 92.4 89.8 6028 10 13 K 1 2 K
12 4.76 8.74 103.1 121.6 10233 15 20 K 2 3 K
16 3.91 6.21 99.6 111.3 13178 22 26 K 3 6 K
24 2.13 2.87 4.68 76.81 25744 21 60 K 5 10 K
32 2.01 2.60 4.42 81.29 28925 25 70 K 6 11 K
64 2.54 3.66 112.4 118.1 26468 27 70 K 6 11 K
… more Aerospike results
cleverdata.ru | info@cleverdata.ru
Libevent to the rescue (2nd attempt)
Node 1
Chunk 1
Replica 2
Node 2
Chunk 2
Replica 3
Node 3
Chunk 3
Replica 1
LIBEVENT :: LIBEVHTP
WRK
cleverdata.ru | info@cleverdata.ru
Finally…
L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s
2.64 3.09 3.27 3.95 35746
cleverdata.ru | info@cleverdata.ru
Nginx Redis Mongo Aerospike, b Aerospike, nb
Latency, 50% 1.35 2.68 6.70 8.93 2.64
Latency, 75% 1.46 3.10 8.22 14.99 3.09
Latency, 90% 1.64 3.25 10.22 26.83 3.27
Latency, 99% 1.90 3.90 15.46 106.48 3.95
Throughput, m/s 68851 34769 14220 3402 35746
CPU, Http, % 29 20 70 7 33
Network, Http, p/s, K 71 71 37 8 71
CPU, Db, % - 6 25 1 6
Network, Http, p/s, K - 12 21 1 12
Unofficial results
cleverdata.ru | info@cleverdata.ru
• Do use non-blocking I/O;
• Do computations locally;
• Be lazy. Don’t do more than needed;
• Don’t afraid of using other’s good ideas;
• Don’t trust anyone.
Do’s & Don’ts
info@cleverleaf.co.uk :: info@cleverdata.ru
cleverleaf.co.uk :: cleverdata.ru
1dmp.io/en :: crawler.1dmp.io/en
facebook.com/CleverData :: +7 (495) 967-66-50
Recommended