Big DataBig Dataecosystemecosystem

Presentation about Espeo's BigData ecosystem based on 1000simulated drones flying around

Poznan city

Drones produce and collect realtime sample data like:

latitude,longitude,height(m),temp(C),wind(m/s),humidity,air-polution

Drone soft written in Scala language

Drone soft written in Scala languageIn real time it streams data to the server

Drone soft written in Scala languageIn real time it streams data to the serverUsing Kafka

Drone soft written in Scala languageIn real time it streams data to the serverUsing Kafka

On a server, data is read by Spark Streaming.

It allows us to:

It allows us to:

save data to Cassandrasend calculated data to browser through websocketsend it to another Kafka consumersave the whole log to Hadoop cluster

On a server, data is read by Spark Streaming.

By saving logs to Hadoop cluster, we can lateraccess those logs, if we didn't save somethingin Cassandra

By sending data to the browser throughwebsocket, we can see where our drones are inrealtime, monitor sensors and much more

By using Cassandra and Apache Spark datascientists can analyze given data later,by using:

1. Apache Zeppelin - Apache Spark(df, RDD) + Scala - Apache Spark MLLib2. Azure Machine Learning

We prefer to use Azure Machine Learninginstead Spark MLLib because it is much easierto understand - and design new predictions

Read our blog post about Azure ML:http://espeo.eu/blog/azure-machine-learning-predictions/

CompleteCompleteecosystem diagramecosystem diagram

Drones

Wifi

Drones

Wifi

Drones

Wifi

websocket

Drones

Wifi

websocket

Drones

Wifi

websocket

Drones

Wifi

websocket

API

Drones

Wifi

websocket

API