Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
BigFoot project
GRIDPOCKET 2012 1
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Filip GLUSZAK [email protected]
+33 6 79 73 90 52
Paris - Sophia Antipolis 20/03/2012
Contents � SmartGrid applications for BigData
� Grid monitoring � Smart metering � Intelligent home
� BigFoot research project � Consortium � Platform � Objectives
� Conclusions
GRIDPOCKET 2009 CONFIDENTIAL 2
3
Provider of value added services platforms for energy
• GRID = ‘electric network’, POCKET = ‘personal’,
• established in 2009, 10 employees
About GridPocket
3
GridPocket Systems SA R&D Center Poland, Gdansk
Head-Quarters France, Sophia-Antipolis
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
4
« kWh » to « Energy Services » paradigm shift
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
New needs and opportunities:
Data Services for Energy Environmental
constraints
Market deregulation
Technology disruption
GRIDPOCKET 2012
Smart Grid data analytics market will generate $11.3 billion cumulative from
2011 to 2014 (Pike Research 2011)
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Data sources in SmartGrid
� Grid monitoring
� Smart Meters � Intelligent home / M2M
5 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
6 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Synchrophasors data � Phaser Measurement Units provide
phaser information (magnitude and phase angle) in real time
� Reporting rate of up to 60 frames per second to Phaser Data Concetrators
7 Source : Siemens.com
Source : Siemens.com
GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Example projects � Tennessee Valley Authority
(TVA) Hadoop � OpenPDC (data from Phaser
Measurement Units) standard � project done by Claudera
� Data volumes : � Serving 9 million consumers � Planning 1000 PMDs � 30 readings per second � 4.2 billion samples per day � Need 500TB storage
� Application � MapReduce oscillation scan :
detection of power grid anomalies, creating power grid maps and evaluating power consumption history
8
Source : claudera.com GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
9 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
10
Smart metering deployments
Electricity, Gas and Water markets 270 millions smart meters in 2013 (ABI, Gartner)
$200 billion investment by 2015 (Pike)
Smart metering projects world-wide (Engage Consulting)
GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Regulations supportin smart metering � EU energy-climate package
20/20/20 � EU directive supporting smart
metering (2009/72/CE) – 80% in 2020
� Directive EU – obligation for energy reduction of 1,5% per year
� Building energy performance directive (2002/91/EG)
� RT 2012 – new constructions regulation
11 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Commercial Billing � Valuation, Estimation and Editing (VEE) – prepare data for
billing � Application of complex pricing methods, rate schedules
� Time of use (TOU), Critical peak pricing (CPP), Peak day pricing (PDP)
� Production of billing data, settlement statements, transaction reports, production and distribution billing
12 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Consumer applications � Energy consumption visualization � Bill analysis (where the usage comes from, reduce call center
load) � “Rebound effect” management, behavioral influencing � Social applications
13
0 10 20 30 40
Indirect
Direct display
PC displays
Ambient (alerts)
Social impact
Variable pricing
% d’économie d’énergie Source: Oxford 2006 report based
on peaks
GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Demand-side management � Grid assets usage analysis and forecasting and backcasting � Load research and trend analysis � Remote load control – demand – response mangement � Outage management and distribution optimization
14
http://www.moorsydeactiongroup.org.uk/windpower.html Source; Eurpean Energy Exchange (28 Dec 2009) http://www.eex.com/en/Market%20Data/Trading%20Data/Power/Intraday%20|%20Spot/Intraday%20Chart%20|%20Spot/spot-intra-chart/2009-12-28/1d
GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Revenue management � De-regulation of energy markets
○ High utilities consumers churn at 10-15% annual levels � Product differentiation, customers segmentation � Customer retention (loyalty) tools, behavioral analysis � Revenue loss detection (customer accounts analysis)
15
Source Poweo financial report Sept 2009
Source CACI Scottish and Southern Energy Case 2008
GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Volumes of data � For 35 million meters (France), data
reading every 10,30, or 60 minutes � Generates : 40-200 TB per year (depending
on reading frequency and format) � Gas and water grid under deployment
16 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Source : ERDF
!Source : ERDF
Example projects with metering data � Ontario Power Authority (OPA) experimentation planned 4.5
million meter collecting data every hour – system wide analysis
� Southern California Edison with Teradata – 100TB warehousing for user metering data
� Itron partners with IBM, SAP and Teradata to form Active Smart Grid Analytics – enabling applications such as Settlements, Network Loss, and Revenue Protection
� Opower - 56 billions meter reads per year, planning to use Hadoop
17
External data sources
OperationalDatabases
Interactive Query Engine
DistributedData Store
DistributedFileSystem
Batch Analytics Engine
Dashboard
Q Q Q
Ad hoc Queries
Data MiningAlgorithms
StatisticalPatternsAnalysis
Applications:* Front-end* Data analysis
BigFoot System:* DataStores* Data Engines
Data Sources
GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
18 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Intelligent home applications
19
• Security (detec,on, camera)
• Smart metering
• Ligh,ng control • Smart appliances
• Comfort control / HVAC
• Mul,media
• And more…
GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Volumes of data � Multiple standards for sensors inter-
connectivity � ETSI M2M framework � For 35 million meters,
� 4-20PB per year in the future (100 sensors per HH)
20 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
21 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
BIG Data Analytics of Digital FOOTprints
BigFoot project consortium
� Eurecom (France) Project leader � Ecole Polytechnique Fedérale de
Lausanne � TU Berlin / Deutsche Telecom Lab
(T-Lab) (Germany) � Symantec (Ireland) � GridPocket (France)
� Projet Proposal to EU FP7 Call
22 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Eurecom experimental platform
� 100 nodes � single ARM v5, 1.2GHz,
512MB RAM, 64GB
� Power efficient � 5 watts per node
� Simple network � GB Ethernet
23 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Eurecom platform - research results � Benchmarking platform for performance evaluation
and comparative analysis of the inner components in the DFS and parallel processing framework
� System optimization for map reduce, using synthetic workloads (eg. load similar to Facebook jobs or others)
� 2 papers published � « ACM SIGOPS LADIS : Hbase, bulk data input »
- resultas integrated in the Hbase source (v0.9.x) � Data analysis for a telecom operator (A1 Austria)
– log files (billing and anomaly detection) algorithm
24 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
BigFoot fundamental objectives � Cross-layer approach system
optimization � Service-oriented query engine - transform
high level declarative query language into parallel programs
� Novel data placement mechanisms for storage layer optimization - physical optimization (layout, typically data is sorted in time, optimize the way data is written)
� Go beyond best-effort on virtualized clusters (Performance guarantee) � Currently virtualization servers are
applications agnostic, (eg. If several VM share same I/O there is no benefit)
GRIDPOCKET 2009 CONFIDENTIAL 25
VIRTUALIZATION
APPLICATIONS: DATA ANALYTICS SCALABLE ALGORITHM DESIGN
HIGH-PERFORMANCE BATCH ANALYTICS ENGINE
DATA STORES
HIGH-LEVEL QUERY LANGUAGE
LATENCY-SENSITIVE QUERY ENGINE
Distributed File System
Distributed Data Base
BigFoot experimental objectives � Experimental test beds
� Fine-grained measurements on all system layers
� Design, implementation and evaluation of algorithms for large data processing � IT Logs Cyber Attacks analysis
(Symantec) � Grid data analysis (GridPocket)
GRIDPOCKET 2009 CONFIDENTIAL 26
BigFoot open-source objectives
� Contribution to the existing open-source projects (eg. Apache Hadoop)
� Establishment of new open-source projects for BigFoot when relevant
GRIDPOCKET 2009 CONFIDENTIAL 27
BigFoot - SmartGrid data processing tasks � Data analysis tasks apply, for example, to the
following cases: � Consumer billing: a monthly scan of all customer data to
produce consumption reports; � Consumer Dashboard Web applications: analysis of the
whole consumption data to create personalized customer reports;
� Load analysis: non intrusive detection of the loads mix, detection of anomalies, consumer behavioral analysis
� Consumer segmentation: execution of complex algorithms to classify consumers based on their consumption patterns and produce, for example, personalized contract;
� Assets usage analysis: analysis of geographical consumption patterns and design of predictive algorithms to help operators in provisioning of their electric network.
28 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
BigFoot SmartGrid scenario
29 GRIDPOCKET 2012
BigFoot Platform SmartGrid Apps
Conclusions � Smart Grid and big data
� Generates significant amounts of data � New needs and applications are coming
� Hadoop technology � Promising, however still need to gain on maturity � Complex setup and maintenance � Interoperability � Limited skills and support available in Europe
� Next steps � BigFoot and other BigData projects w � GridPocket and BigFoot are open for cooperation � Looking for comments and suggestions
30 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
31 GRIDPOCKET 2012