The OLX data theory of everything - aws-de-media.s3 ...aws-de-media.s3. © 2018, Amazon Web Services,

  • View
    212

  • Download
    0

Embed Size (px)

Text of The OLX data theory of everything - aws-de-media.s3 ...aws-de-media.s3. © 2018, Amazon Web...

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Caspar Schönau

    Head of Global BI

    Jakub Orłowski

    Data engineering manager

    The OLX data theory of everything

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    ‘The biggest internet company that you have

    never heard of’

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Founded 1915 South-Africa

    Market cap: $100B

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    43 Countries

    +350M MAU

    +5,000 Employees

    35 Offices

    +4B events / day

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    The OLX challenge:

    ‘Give everybody the data that he or she

    needs’ (but also not much more)

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Halleluja! We are a data-driven company!

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    What decisions are you really taking on a daily basis? And how does data play a role?

    Do I need move to the next valley to

    survive the winter?

    Can I win the javelin throw competition at the next Olympics?

    Can I buy a higher desk, so I don’t destroy my back while

    working?

    • Kg of food in the valley • # of apes in the valley • Kg of food / ape needed

    per day • # days of left this winter

    • WR javelin throw in m • PB javelin throw in m • Time between date of

    PB and next Olympics

    • $ in the bank • Price of a decent

    desk in $

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    The same goes for an organization like OLX. Which data points are really influencing your decisions?

    CEO GM CS Manager

    months secondsurgency

    Infra engineerBusiness analyst

    Do I launch a new car portal

    in Mexico?

    • Size of the prize of online cars market in Mexico

    • Cost of success • Chance of success • Available war chest

    Shall I invest more in online

    or in offline marketing?

    • ROI of a typical offline marketing campaign

    • ROI of continuous online marketing

    • Expected reach of both offline and online marketing

    Should I fire CS agent #253?

    • Average # of ads moderated by agent #6 in the last month

    • Average # of ads moderated by the team in the last month

    • Error rate agent #6 • Error rate of the team

    Is the platform still online?

    Can I predict which listings

    have the highest probability to sell?

    • Requests per second • Post per second

    • Detailed properties of all listings (# pictures, attributes, length of the the title, etc)

    • All individual replies, including related buyers and seller data

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Disc

    42 is broken

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

  • The OLX data iceberg model

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    OLX data lake

    The old school way of providing data…

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Platform A Platform B Any other stack

    Top management

    Business owner

    Data analyst

    Global reporting

    Data self service tool

    Operational dashboards

    Data warehouses

    Data reservoir A Data reservoir B

    Data reservoir C Data reservoir D Data reservoir E

    Global BI

    Local reporting

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Platform A Platform B Any other stack

    Top management

    Business owner

    Global reporting

    Data self service tool

    Operational dashboards

    Data reservoir A Data reservoir B

    Data reservoir C Data reservoir D Data reservoir E

    Global BI

    Local reporting

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Platform A Platform B Any other stack

    Top management Global reporting

    Data self service tool

    Operational dashboards

    Data reservoir A Data reservoir B

    Data reservoir C Data reservoir D Data reservoir E

    Global BI

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Platform A Platform B Any other stack

    Data self service tool

    Operational dashboards

    Data reservoir A Data reservoir B

    Data reservoir C Data reservoir D Data reservoir E

    Global BI

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Platform A Platform B Any other stack

    Top management

    Business owner

    Data analyst

    Global reporting

    Data self service tool

    Operational dashboards

    Data warehouses

    Data reservoir A Data reservoir B

    Data reservoir C Data reservoir D Data reservoir E

    Global BI

    Local reporting

    Operations/ data scientist

    Designated reservoirs

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Platform A Platform B Any other stack

    Top management

    Business owner

    Data analyst

    Global reporting

    Data self service tool

    Operational dashboards

    Data warehouses

    Data reservoir A Data reservoir B

    Data reservoir C Data reservoir D Data reservoir E

    Global BI

    Local reporting

    Operations/ data scientist

    Designated reservoirs

  • 4 “V”s of Big Data top challenges of data processing

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Volume data at rest

    Amazon S3

    Amazon Redshift

    Amazon EMR

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Velocity data in flight

    Amazon Kinesis

    Amazon SQS

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Variety data in many shapes

    AWS Glue

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Veracity data quality

    Amazon Athena

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Volume

    Data at rest

    Terabytes of existing, historical data that

    needs to be stored for extended period

    Velocity

    Data in flight

    Streaming, near real-time data sources, short

    reaction and computation time required

    Variety

    Data types

    Platform databases, user behaviour, infrastructure

    monitoring, pictures

    Veracity

    Data quality

    Tracking coverage, downtime,

    implementation errors, schema changes

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Data democratization

    Data understandingData access

    Volume Velocity Variety Veracity

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    producers

    complianceconsumers

  • Data democratization at OLX architecture overview

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    Platform A Platform B Any other stack

    Top management

    Business owner

    Data analyst

    Global reporting

    Data self service tool

    Operational dashboards

    Data warehouses

    Data reservoir A Data reservoir B

    Data reservoir C Data reservoir D Data reservoir E

    Global BI

    Local reporting

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    producers

    complianceconsumers

  • © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    The OLX challenge:

    ‘Give everybody