Upload
bigdataeurope
View
1.411
Download
0
Embed Size (px)
Citation preview
BIG DATA EUROPE
Integrating Big Data, Software & Communities for Addressing Europe’s Societal Challenges
negative conotation
Big Data has often a
6-oct.-15www.big-data-europe.eu
Big Data in Marketing
6-oct.-15www.big-data-europe.eu
Big Data in Intelligence
6-oct.-15www.big-data-europe.eu
BigDataEurope aims to help maximizing
the societal value of Big Data
Health, demographic change and wellbeing;
Food security, sustainable agriculture and forestry, marine and maritime and inland water research, and the Bioeconomy;
Secure, clean and efficient energy;
Smart, green and integrated transport;
Climate action, environment, resource efficiency and raw materials;
Europe in a changing world - inclusive, innovative and reflective societies;
Secure societies - protecting freedom and security of Europe and its citizens.
6-oct.-15www.big-data-europe.eu
The three Big Data „V“ – Variety
is often neglected
Quelle: Gesellschaft für Informatik
© Fraunhofer-Allianz Big Data 7
Proactive Maintenance at Rolls Royce
New Business Model integrating Sensor Data & Big Data Analytics
Dr. Dirk Hecker
Condition Monitoring, Proactive maintenance, „Power-by-the-hour“,
as-a-service Business Model – payment modell by flight hours
Quelle: www.springboeck.ch/SR_Technics.htm
© Mark Hillary | Flickr
© Fraunhofer-Allianz Big Data 8
The rolling Smartphone
New Business Models for the Automotive Industrywith Data Value Chains
Dr. Dirk Hecker
Windshield wiper as rain sensors for micro wether prognosis
Automotive industry can become data provider for other industries
Qu
elle
: G
TÜQ
uelle
: ww
w.fa
rmin
g-sim
ula
tor.co
m
© Fraunhofer-Allianz Big Data 9
Predictive Analytics
Dr. Dirk Hecker
From Business Intelligence to Big Data Analytics
Business Intelligence Monitoring Predictive Analytics
What happenedbefore?
What happensnow?
What will happen soon?
What shouldhappen?
Prescriptive Analytics
„the last Mile“
“prescriptive analytics suggests decision options on how to take advantage of a future opportunity”
Quelle: BMW Quelle: www.7-forum.com Quelle: BMW Quelle: Volvo
BigDataEurope Rationale
Show societal value of Big Data
Lower barrrier for using big data technologies
o Required effort and resources
o Limited data science skills
o Lack of Generic Architectures, components
Help establishing cross-
lingual/organizational/domain Data Value Chains
o Multiple Data Sources
o Required: Integration, Harmonisation
6-oct.-15www.big-data-europe.eu
BigDataEurope: Objectives
6-oct.-15www.big-data-europe.eu
COORDINATION
Stakeholder Engagement
(Requirements Elicitation)
SUPPORT
Design, Realise, Evaluate
Big Data Aggregator
Platform
Create and Manage
Societal Big Data
Interest Groups
Cloud-deployment
ready
Big Data Aggregator
Platform
CSA
Measures
Results
Orthogonal Dimensions of Big Data Ecosystems
Generic Big Data Enabling Technologies
Data Value Chain
Data Generation & Acquisition
Data Analysis & Processing
Data Storage & Curation
Data Visualization &
Usage
Data-driven Services
So
cie
tal
Ch
all
en
ge
s
Do
ma
in S
pe
cifi
c D
ata
Ass
ets
& T
ech
no
log
y
Healthcare
Food Security
Energy
Intelligent Transport
Climate & Environment
Inclusive & Reflective Societies
Secure Societies
Stakeholder Engagement Cycle
© Fraunhofer-Institut für Intelligente Analyse- und Informationssysteme IAIS
Open-Source Technologies for Big Data Apps (small selection :-)
14
© Fraunhofer-Institut für Intelligente Analyse- und Informationssysteme IAIS
Big Data - Technologies
Volume
VelocityVariety
Storm
15
© Fraunhofer-Institut für Intelligente Analyse- und Informationssysteme IAIS
Groups of Technologies
Big Data Technolog
ies
Data Storage Technologies
Data Processin
g
Workflow Coordinati
on
Querying/
Processing
Search
Data Export/ Import
Data Analysis
Statistics
Text Mining
16
© Fraunhofer-Institut für Intelligente Analyse- und Informationssysteme IAIS
Big Data Requirements
Analysis of historical dta
Millions of entries
Varying analysis quesitions
Years of input data
=> Big Data Batch Processing
Interactive analysis by online queries
Thousands of users online
Extremely fast response time
Super high availability
=> Big Data Databases
Analysis of actual data with low latency in
"real-time"
React to newest trends
Low-Latency change detection
Real-time online monitoring
=> Big Data Stream Processing
But how to put it together ?17
© Fraunhofer-Institut für Intelligente Analyse- und Informationssysteme IAIS
The „traditional“ Hadoop Ecosystem + NoSQL
components
a Big Data Management System
ZooKeeper
askaban
Kafka
cassandra
voldemort
MongoDB
CouchDB
elastic search
solr
lucene
© Fraunhofer-Institut für Intelligente Analyse- und Informationssysteme IAIS
Batch
Function
Speed
Function
Data Storage
pages with
postings
Batch View
Realtime
View
me
ssa
ge
pa
ssin
g
message passing
Application
Horizontal Scalability in the Lambda Architecture
19
> volume
> users
> users, volume
> velocity> volume, velocity
Blueprint of the Data Aggregator Platform
Follows typical Lambda Architecture
Integrated on top of existing Big Data distribution
+ Semantic Layer (Retaining Semantics using LD approach )
Batch Layer
Speed Layer
Data Storage
Real-time data &
Transactions …
Batch View
Real-time
View
mess
ag
e p
ass
ing
message passing
Applications & Showcases
Real-time dashboards
Domain-specific BDE apps
Big Data Analytics
In-stream Mining
BD
E P
latfo
rm&
Intellig
ence
Input data
Stream
Spatial
Social
Statistical
Temporal
Transactional
Imagery
BDE Platform based on BigTop
Packaging Smoke testing Virtualization
Package RPMs and DEBs, so
that you can manage and
maintain your own cluster.
Integrated smoke testing
framework
Vagrant recipes, raw images,
and docker recipes for
deploying BigData
infrastructures from zero.
6-oct.-15www.big-data-europe.eu+ Semantic Layer - Retaining Semantics using Linked Data
Data Aggregator Platform Challenges
Ingest semantic (RDF) and non-semantic (CSV,
JSON, XML, …) data
o Integrate various mapping techniques (R2RML, CSV on
the Web, JSON-LD)
preserve semantics, provenance and metadata in
Big Data processing chains
o Preserve URI/IRIs
o Preserve triples
Exploit semantics for aggregations
6-oct.-15www.big-data-europe.eu
Current Activities – Year#1
2015 BDE Societal Workshops (7) Planned
o Schedule on Website
7 W3C Interest Groups set up: Please Join!o SC1: HEALTH https://www.w3.org/community/bde-health/join
o SC2: FOOD & AGRICULTURE https://www.w3.org/community/bde-food/
o SC3: ENERGY https://www.w3.org/community/bde-energy/
o SC4: TRANSPORT https://www.w3.org/community/bde-transport/
o SC5: CLIMATE & ENVIRONMENT https://www.w3.org/community/bde-climate/
o SC6: SOCIETIES https://www.w3.org/community/bde-societies/
o SC7: SECURITY https://www.w3.org/community/bde-secure-societies/
www.big-data-europe.eu
BDE Partners
Sören Auer
Big Data Europe Coordinator
Fraunhofer IAIS & University of Bonn
Thanks
6-oct.-15www.big-data-europe.eu
Energy/Climate Example: Greenshifting
6-oct.-15www.big-data-europe.eu