Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
IoT REALIZED CONNECTED CAR Ian Huston Senior Data Scientist @ianhuston
https://github.com/pivotal/IoT-ConnectedCar
B A C K T O T H E BEGINNING
REALLY COOL IoT PROJECT
COVERED BY AN
NDA
CONNECTED
CAR
PIECES ARE THE
SAME
P R E D I C T T H E DESTINATION
PREDICT THE RANGE
TCP SOCKET OVER
BLUETOOTH HTTP POST
OVER CELLULAR
HOW DOES THIS
WORK?
START IN THE
CAR
ON BOARD DIAGNOSTICS
OBD II
PHONE PROVIDES CONNECTIVITY
127.0.0.1:9000
{ ! "vehicle_speed":103, ! "obd_standards":2, ! "intake_manifold_pressure":"", ! "accelerator_throttle_pos_e":14, ! "engine_load":89, ! "maf_airflow":33, ! "latitude":"32.897554", ! "vin":"1HGCM82633A004352", ! "bearing":"343.922580", ! "catalyst_temp":779, ! "relative_throttle_pos":12, ! "fuel_level_input":89, ! "fuel_system_status":[2,0], ! "accelerator_throttle_pos_d":29, ! "acceleration":"0.953", ! "throttle_position":21, ! "barometric_pressure":97, ! "control_module_voltage":13, ! "longitude":"-96.810236", ! "distance_with_mil_on":0, ! "coolant_temp":94, ! "intake_air_temp":34, ! "rpm":1593, ! "short_term_fuel":-2, ! "time_since_engine_start":4054, ! "absolute_throttle_pos_b":38, ! "long_term_fuel":3 !} !
Pivotal λ-Architecture
Spring XD
Pivotal HD
Pivotal GemFire
Speed Layer Serving Layer
Batch Layer
Pivotal CF ®
What is Cloud Foundry? http://cloudfoundry.org
Open Source
Multi-Cloud Platform
Simple App Deployment, Scaling & Availability
Cloud Applications Haiku Here is my source code Run it on the cloud for me I do not care how.
- Onsi Fakhouri @onsijoe
Same applications running across different cloud providers
Private Public Hosted
Multi-Cloud
Pivotal Cloud Foundry for Open Cloud Computing
$ cf push
Pivotal λ-Architecture
Spring XD
Pivotal HD
Pivotal GemFire
Speed Layer Serving Layer
Batch Layer
Pivotal CF ®
O N T H E SERVER
SPRING XD EXTREME DATA
Spring XD Unified, distributed, and extensible open-source system for data ingestion, real time analytics, batch processing, and data export. o Data Ingestion and Pipeline Processing
o Kafka, RabbitMQ, MQTT, JMS, HTTP, GPDB, HAWQ
o Partition, Filter, Transform, Split, Aggregate
o Real Time Analytics and Complex Event Processing o Spark Streaming, RxJava, JPMML Scoring
o Redis, GemFire, Cassandra, etc..
o Rapid Dashboarding
o Batch Workflow Orchestration + ETL o Map Reduce, HDFS, PIG, Hive, GPDB, HAWQ, Spark
o RDBMS, FILE, FTP, Log, Mongo, Splunk Spring XD
SPRING
Ingestion
Orchestration
Extraction
Real-time Analytics
D I S T R I B U T E D RUNTIME
STREAMING BATCH &
http | filter | acmeenrich | shell | hdfs!
// Ingestion Stream Definitions !
IoT-HTTP.shell > typeconversiontransformer | !gemfire-server !
// Tap Definitions !
job create pi --definition "sparkapp --name=PI \ ! --master=local[8] \ ! --appJar=path/lib/spark-examples-1.4.0-hadoop2.6.0.jar \ ! --mainClass=org.apache.spark.examples.SparkPi \ ! --programArgs=100" –deploy !!job launch pi !
// Spark Example Job !
TCP SOCKET OVER
BLUETOOTH HTTP POST
OVER CELLULAR
Spring XD
HTTP transformers hdfs
gemfire tap
python
hdfs
GemFire
REST API
PySpark
Spring XD
HTTP transformers hdfs
gemfire tap
python
hdfs
GemFire
REST API
PySpark
Pivotal λ-Architecture
Spring XD
Pivotal HD
Pivotal GemFire
Speed Layer Serving Layer
Batch Layer
Pivotal CF ®
PivotalBig Data Suite
Advanced Analytics
Pivotal HAWQ
Pivotal Greenplum Database
Pivotal GemFire
RabbitMQ
Redis
Apps at ScaleData Processing
Big Data Suite Services on Pivotal Cloud Foundry
Spring XD
Spark
Pivotal HD
Pivotal Big Data Suite
Pivotal HAWQ RabbitMQRedisSpring XD Pivotal HD
Pivotal HD
Spring XD
Realtime Evaluation Batch Training
Data Persistence
REALTIME DATA SCIENCE
1 PREDICT JOURNEY
HOW DOES I T
WORK?
STORE SENSOR
DATA
R E C O R D E D
JOURNEYS
OFFLINE BATCH TRAINING
J O U R N E Y CLUSTERS
INITIAL PREDICTION
DRIVING HOME TO WORK
DRIVING WORK TO HOME
ONLINE PREDICTION
2 PREDICT RANGE
… !"Predictions": { ! "ClusterPredictions": { ! "0": { ! "EndLocation": [ ! 32.98525175453122, ! -96.70940837440399 ! ], ! "MPG_Journey": 25.60900810315203, ! "Probability": 0.63736 ! }, !… !
HORIZONTAL SCALABILITY
100,000’s OF CLIENTS
MILLIONS OF MESSAGES
PER MINUTE OVER
100 GB
IoT
ISSUES
COMPATIBILITY
CONNECTIVITY
SECURITY
LOOKING INTO THE
FUTURE
A D D I T I O N A L USE CASES
F L E E T MANAGEMENT
P R E D I C T I V E MAINTENANCE
A C C I D E N T ASSISTANCE
Ian Huston @ianhuston Michael Natusch @cumulyst