S.2.f Specifications for Data Ingestion via Green Button

WP4 – Integration of SUNSHINE pilot smart urban services

D4.5 Meter Data Management service #2

CIP-ICT-PSP-2012-6 – 325161

Page 1 of 45

"This project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the

Competitiveness and Innovation Framework Programme by the European Community"

(http://ec.europa.eu/ict_psp).

Annex to D4.5 - Meter Data Management Service #2

Specifications for Data Ingestion

via Green Button

WP 4 – Integration of SUNSHINE pilot smart urban services

Task 4.9 – Integration of new smart services

with existing service infrastructures

Task 4.2 – Meter data management service

Revision: v1.1

http://ec.europa.eu/ict_psp



CIP-ICT-PSP-2012-6 – 325161

Page 2 of 45




REVISION HISTORY AND STATEMENT OF ORIGINALITY

Revision Date Author Description

v. 0.1 3rd September 2014 Luca Giovannini Document created.

v. 0.3 4th September 2014 Rossana Bambili First draft of guidelines

v. 0.9 18th September 2014 Luca Giovannini Final draft

v. 1.0 14th November 2014 Stefano Pezzi Release

v. 1.1 7th January 2015 Stefano Pezzi Revision after pilot installation

v.1.2 16th January 2015 Stefano Pezzi Added some information on the pre-population of the datacustodian tables

Statement of originality:

This deliverable contains original unpublished work except where clearly indicated otherwise.

Acknowledgement of previously published material and of the work of others has been made through

appropriate citation, quotation or both.

Moreover, this deliverable reflects only the author’s views. The European Community is not liable for any

use that might be made of the information contained herein.




CIP-ICT-PSP-2012-6 – 325161

Page 3 of 45




Table of contents

1 Purpose ............................................................................................................................... 5

1.1 Green Button in a nutshell ....................................................................................................... 5

1.1.1 Green Button & Sunshine ..................................................................................................... 6

1.2 OAuth ..................................................................................................................................... 9

1.2.1 OAuth v.1.0 vs v.2.0 .......................................................................................................... 14

1.3 OAuth & Green Button ......................................................................................................... 16

1.4 Green Button in Sunshine scenario ........................................................................................ 17

2 Software Installation ......................................................................................................... 20

2.1 Software Requirements (DataCustodian) ............................................................................... 20

2.2 Installation instructions ......................................................................................................... 20

2.2.1 Java JRE ............................................................................................................................. 21

2.2.2 Apache Tomcat .................................................................................................................. 21

2.2.3 MySQL ............................................................................................................................... 22

2.2.4 DataCustodian application ................................................................................................. 27

3 Configuration and Use ....................................................................................................... 29

3.1 datacustodian DB structure ................................................................................................... 29

3.2 Populating the datacustodian database ................................................................................. 31

3.2.1 ‘retail_customers’ table ..................................................................................................... 31

3.2.2 ‘application_information’ table .......................................................................................... 32

3.2.3 ‘application_information_scopes’ table .............................................................................. 32

3.2.4 ‘time_configurations’ table ................................................................................................ 32

3.2.5 ‘usage_points’ table ........................................................................................................... 33

3.2.6 ‘reading_types’ table ......................................................................................................... 34

3.2.7 ‘meter_readings’ table ....................................................................................................... 34

3.3 Import data service ............................................................................................................... 35

3.3.1 Service url ......................................................................................................................... 36




CIP-ICT-PSP-2012-6 – 325161

Page 4 of 45




3.3.2 Consumption data XML ...................................................................................................... 36

3.4 Export data ........................................................................................................................... 38

3.5 Security underneath the application layer .............................................................................. 39

3.6 Generation of the access token .............................................................................................. 39

Annex 1 – DST rules .................................................................................................................. 42

Annex 2 – Codelists .................................................................................................................. 44

ServiceCategoryKind ................................................................................................................................ 44

AccumulationBehaviourType .................................................................................................................. 44

CommodityType ...................................................................................................................................... 44

TimeAttributeType .................................................................................................................................. 45

UomType ................................................................................................................................................. 45

CurrencyCode .......................................................................................................................................... 45




CIP-ICT-PSP-2012-6 – 325161

Page 5 of 45




1 Purpose

The purpose of this guideline is to support the pilots in implementing at their own premises a Green Button

web service, the DB and all the backend supporting services behind it. Through this web service, they will

be able to expose their consumption data to the central Sunshine platform.

1.1 Green Button in a nutshell

Green Button is an initiative of the U.S. Government that aims to allow consumers to access their own

meter data (called Energy Usage Information, EUI) about energy consumption. In the original context, the

name Green Button (in the rest of the document we’ll use GB) denotes the whole initiative, but in our

project we will call GB only the suite of web services that makes the data access possible.

There are three actors involved in the scenarios of making meter data available:

Retail Customer (RC) = is the person (a real one or an entity) that consumes energy and is the actual

owner of the data.

Data Custodian (DC) = is typically the utility that delivers the energy and measures it with meters in

order to get it paid by the retail customer. To do this, the data custodian has to

record consumptions data and store them for its operational purposes.

Third Party (TP) = is a service provider that offers value-added services to the retail customer,

services that elaborate his/her meters data

These three actors interact among them usually in these two alternative ways:

1. Retail Customer directly accesses his/her own data by using data custodian services.

2. Retail Customer delegates a Third Party to access his/her own data. The Third Party uses the Retail

Customer’s data to provide some sort of special services to the Retail Customer, probably using

some other information available from further data providers (meteorological data, global energy

consumption data …). Some examples of services could be: advanced reporting, hints about energy

saving, etc.

In GB jargon, the first is called “GB Download My Data” and the second “GB Connect My Data”.




CIP-ICT-PSP-2012-6 – 325161

Page 6 of 45




1.1.1 Green Button & Sunshine

The Sunshine scenario can be described by the latter of the two GB interactions (GB Connect My Data) with

some peculiarity. In Sunshine the actors are the following:

Pilot = is the owner or conductor of a building that consumes energy and is the actual

owner of the consumption data.

Pilot technical partner = is a subject that acts on behalf of the utility company in the sense that it is in

charge of gathering the consumption data in various manner (in real time or off

line). It is an intermediary of the utility company because this one could not be

active part of the project: by means of smart meters, retrofitted meters, or

simply reading the old bills, consumption data are retrieved and made available.

The Pilot Technical Partner is strictly related to the Pilot and sometimes it can

overlap completely with it.

Sunshine platform = is the platform of services, data and software components that Sunshine has put

in place.

Let see know the correspondence with GB actors:

GB actor Sunshine actor

Retail Customer Pilot

Data Custodian Pilot Technical Partner

Third Party Sunshine platform

Table 1 - Correspondence between GB and Sunshine actors

Like sketched in Figure 1, the Retail Customer is actually the owner of the building where the energy is

consumed and measured. In Sunshine’s consortium, the building owners are either the Pilot Technical

Partners (see HEPESCO) or the Pilot (i.e. Municipality of Ferrara buildings).




CIP-ICT-PSP-2012-6 – 325161

Page 7 of 45




Utility company

Pilot Technical Partner

Sunshine

Pilot

EUI EUI (GB)

Energy

Data Custodian & Retail Customer

Third Party

Figure 1 - Energy Usage Information flow in Sunshine

The real Data Custodian should be the utility company that distributes the energy, but, due various

technical and “political” problems, it’s quite impossible to get the data directly from the utility company. So

we can consider the pilot’s technical partner as a surrogate of a Data Custodian, since it manages to get the

meter data either by asking them to the utility in a non-automatic manner or by adding its own meter

reading infrastructure to the utility’s one.

The Third Party is the Sunshine platform that needs to access the consumption data to process them and

provide value added services.

Figure 2 - The GB scenario




CIP-ICT-PSP-2012-6 – 325161

Page 8 of 45




The first two actors are almost the same and this is the peculiarity of the Sunshine scenario: Retail

Customer could easily access directly its data, because it is very close to the Data Custodian or sometimes

it’s the Data Custodian itself, but requires the services that Sunshine platform can offer.

The original context of the GB scenario is the one depicted in Figure 2. Here we can see the flows between

the 3 actors. Notice the two button icons that indicate the respective flows of data: in the diagram red

arrows show the “GB Download My Data” flow, while the blue ones are involved in the “GB Connect My

Data” use case.

The Sunshine’s context is a profile of the “GB Connect My Data” context of GB scenario, and we can

simplify the diagram in that of Figure 3, taking away the “GB Download My Data”.

Figure 3 – The Sunshine scenario

The dotted line arrows represent the authorization flows, in particular those of the delegated access

implemented by OAuth. The term “One-Time Authorization” is used to highlight the fact that once has

happened it may last for a long time in order to avoid a new authorization delegation exchange every time

a data access has to be performed.

The solid blue line arrow is the real data exchange between Third Party and Data Custodian, the one that

interests most.

The green line arrow is an interaction between Third Party and Data Custodian that aims to register the

Third Party in the Data Custodian system as a subject that is being given a generic pre-authorization to use

the Data Custodian APIs or services. This registration operation allows creating a trust between DC and TP,




CIP-ICT-PSP-2012-6 – 325161

Page 9 of 45




and letting the DC know all the details of the actions that the TP could execute on the RU resources (i.e.

read and add documents from a web drive) once obtained the authorization from the owner. The

registration operation can take place offline although GB services allow to do it dynamically; in order to

simplify the exchange of data in Sunshine scenario, since the DC and the TP are well known from the start,

we will consider only the offline registration.

The OAuth protocol, on which GB bases its security features, relies also on HTTP redirection because the

main use case is the one in which the user access the Third Party services by means of his/her browser (in

the diagrams it’s called “agent”) and the first redirection is from TP’s web portal to DC’s web portal, where

the RU can authenticate and delegate access to the TP services.

In Sunshine the access delegation is like the TP registration: it can be done once and offline for the same

reasons; furthermore the DC web portal and the TP web portal, in the sense of the web portal functionality

involved in the authorization delegation operations, does not play any roles. So we can achieve a further

simplification of the diagram as we show in the Figure 4.

Figure 4 - Further simplification of Sunshine context

1.2 OAuth

As we said before, the three actors’ scenario of GB recalls the more generic one of delegated access: this is

an increasingly widespread situation in which a web user is involved almost every day. The user owns some

resources (emails, images, documents, address book, path recordings, bank …) that are maintained by

some provider (Google, Flickr, bank …). Other service providers offer more advanced services on these data




CIP-ICT-PSP-2012-6 – 325161

Page 10 of 45




(professional printing, special visualization, instant messaging, spatial analysis …), but, of course, they need

access to the original data do some work on them. This kind of interaction strongly involves security aspects

like granting access to other subjects to own protected resources without having to share own

authentication credential. OAuth1 protocol is about this problem and points out a particular authorization

flow among the involved subjects in order to fulfil the descripted objective.

In the OAuth terminology, the actors involved in the delegated authorization scenario are:

Resource Owner = the owner of the resources that we want to secure.

Client = an application making protected resource requests on behalf of the Resource

Owner and with its authorization. The term “client” does not imply any

particular implementation characteristics (e.g., whether the application executes

on a server, a desktop, or other devices).

Resource Server = The server hosting the protected resources, capable of accepting and

responding to protected resource requests using access tokens, that is the

artefact that substitutes the Resource Owner credential.

Authorization Server = the server issuing access tokens to the client after successfully authenticating

the Resource Owner and obtaining authorization.

User-agent = typically a web browser or other software under the control of the Resource

Owner.

We are talking of a three actors’ scenario, but here we’ve just listed five subjects: some of these actors

actually overlap in a higher level architecture and their roles are split only from an implementation and

technological point of view. In particular:

Resource Server ≡ Authorization Server

Resource Owner ≡ User-agent

Other terms used in the OAuth protocols are the following2:

protected resources = an access-restricted resource that can be obtained from the server using an

OAuth-authenticated request.

1 OAuth v.1.0 is based on two proprietary protocols used by Google and Flickr: “AuthSub” and “API Auth”,

respectively.

2 http://tools.ietf.org/html/rfc5849




CIP-ICT-PSP-2012-6 – 325161

Page 11 of 45




credentials = Credentials are a pair of a unique identifier and a matching shared secret. OAuth

defines three classes of credentials: client, temporary, and token, used to identify

and authenticate the client making the request, the authorization request, and

the access grant, respectively.

token = A unique identifier issued by the server and used by the client to associate

authenticated requests with the resource owner whose authorization is requested

or has been obtained by the client. Tokens have a matching shared-secret that is

used by the client to establish its ownership of the token, and its authority to

represent the resource owner.

In the sample flow below, let’s imagine an user Bob that wants to use some professional printing services

offered by a web company PrintEveryThing that advertises its capacity of printing Facebook, Flickr, Google

Drive, Picasa … web stored photos. The roles are the followings:

Bob = Resource Owner

Flickr = Resource Server

PrintEveryThing = Client

The flow is described by the following steps; step # 0 represents the preconditions of the flow:

0. Bob owns some photos at Flickr. Flickr implements OAuth mechanism and has an Authentication

Server and an Authorization Server as well. The PrintEveryThing is a registered Client at Flickr. This

means that PrintEveryThing has been given an ID by Flickr and shares a secret for encrypting its

messages to assure they come from it.

1. Bob accesses the PrintEveryThing web portal in order to use some of its services.

2. At a certain point of the interaction with the PrintEveryThing’s portal, Bob confirms that he wants

to print his photo album at Flickr. This action makes PrintEveryThing communicates with Flickr

asking for an unauthorized request token; the release of this token means that Flickr has

recognized PrintEveryThing as a registered Third Party, but without an explicit authorization from

the user, the token can’t be used to access the photos.

3. Bob’s browser gets redirected by PrintEveryThing’s portal to the Flickr portal. The redirection

contains a call-back URL and the unauthorized request token, plus some other information.

4. Flickr’s portal asks for Bob’s credentials and authenticates him by means of its Authentication

Server.




CIP-ICT-PSP-2012-6 – 325161

Page 12 of 45




5. Flickr’s portal recognizes the unauthorized request token and, retrieving the information stored in

the TP registration step (not shown in this flow), asks Bob what kind of permissions to give to

PrintEveryThing and for which protected resources (photos).

6. Bob chooses the privileges and resources and then confirms his intention to let PrintEveryThing

access them. This action makes Flickr produce a request token that is returned to PrintEveryThing

by means of the next step. We can say that the user granting operation transforms the

unauthorized request token into a request token.

7. Flickr redirects back BOB’s browser to the call-back URL at PrintEveryThing’s site bearing some

other information: the request token. This token contains the information that the user has granted

the Third Party the privileges to his resources, but can’t be used to directly access the resource yet.

8. Finally PrintEveryThing exchanges with Flickr its request token with an access token3

9. PrintEveryThing uses the access token to retrieve the protected photos from Flickr using its APIs

and then does its job on them.

In the diagram of Figure 5 the OAuth flow is shown with only the Client and the Resource Server

highlighted; user (the Resource Owner) or user’s browser is involved in the passages represented with a

solid line.

Actually, the OAuth flow that we exemplified here is one of the possible because several variants can be

sketched: for instance, the user could start interaction at the Resource Server portal instead of the Client’s

and by means of some redirections that go the other way around, the same steps can be performed and

the same result obtained. Instead of starting from the client’s portal, choosing the Resource Server

3 The access token is the final result of the OAuth flow: it is the string representing an access authorization issued to

the client, rather than using the resource owner's credentials directly.




CIP-ICT-PSP-2012-6 – 325161

Page 13 of 45




Client Resource Server

Requests unauthorized request

token

Grantsunauthorized request

token

Redirects user to resource server

Authenticates user

Obtains userauthorization

Redirects userto Third Party

RequestsAccess token

Unauth. req token

Request token

Grantsaccess token

Request token

Unauth. req token

AccessesProtected resources

Access token

Protected resources

Access token

User browses portal

Figure 5 - OAuth flow

The three fundamental interchanges can be further simplified in the schema of Figure 6.




CIP-ICT-PSP-2012-6 – 325161

Page 14 of 45




Client Resource Server

Authorization request

Authorization grant

Authorization grant

Access Token

Access Token

Protected Resource

Figure 6 – High level OAuth interchanges

1.2.1 OAuth v.1.0 vs. v.2.0

The OAuth authorization flow described up to here refers to version 1.0 that was first published at the end

of 2007 and that became a standard (RFC 5849) in mid-2010 (after an update release 1.0a fixed a security

leak). A subsequent version of OAuth, the v.2.0, has been released afterwards, the first draft in the very

same period of the 2010; this version is not backwards compatible, but it maintains the overall architecture

and the approach descripted above. Now the stable version4 is in a proposal state under the code RFC

6749.

This new release has been also a source of controversy: some of the original creators of OAuth abandoned

the project because some vulnerability has been introduced in exchange for more simplicity at client side,

but here we don’t want to discuss this aspect. The fact is that the major web companies have adopted

OAuth v.2.0 even if many features are still changing and being added.

Green Button relies on v.2.0, so it’s better that we take a quick look at the most important differences;

some of these are both simplifying the protocol and making it less secure, as critics says:

4 https://tools.ietf.org/html/rfc6749




CIP-ICT-PSP-2012-6 – 325161

Page 15 of 45




1. Cryptography. Client applications do not have to use HMAC (keyed-hash message authentication

code) to encrypt tokens and request string anymore, but they can send plaintext token over an

https secure channel.

2. Signature. A simpler signature is used using only one secret5 instead of two.

3. Bearer token. It is a special type of token that can be used instead of an access token. Every

request that has a bearer token with it, can access the protected resources for which it has been

issued, no matter who is the requestor. No need of the “proof-of-possession” that is proving that

who is sending the request is in possess of cryptographic key material. An access token instead is

used along with a signature that identifies the requestor.

4. Access token refresh. Access tokens can be long-lived and even have unlimited lifetime. This could

be a security issue in case an access token (or better, a bearer) is stolen. In OAuth v.2.0 server can

issue short-lived access tokens in order to minimize the time windows in which is possible to use

the access token fraudulently. The short-lived access token is released together with long-lived

refresh tokens that can be used to reissue the access token. The difference is that to use a refresh

token you need to use the secret and the client ID.

5. Separation of roles. OAuth v.2.0 separates the roles of Authorization server from the one of

Resource Server. Authorization server is responsible for obtaining user authorization and issuing

the relative tokens. Resource Server checks handles the API calls to the protected resources and

enforces the authorization policies. The original flow of Figure 6 can be re-drawn like in Figure 7. In

this diagram the client request to the resource owner is usually made indirectly via the

authorization server as an intermediary.

5 In cryptography a secret is a data known only by the two parts involved in a secure communication. This data, usually

a string or a big number, can be shared before the communication starts or at the start of it. The secret is used to

encrypt the messages exchanged between the parts.




CIP-ICT-PSP-2012-6 – 325161

Page 16 of 45




ClientAuthorization request

Authorization grant

Authorization grant

Access Token

Access Token

Protected Resource

Resource Owner

Authorization server

Resource server

Figure 7 - OAuth v.2.0 interchanges

In the Sunshine scenario we don’t have a user interacting with either the Third Party’s portal or the

Resource Server’s one. So, a part from the overlap of the Resource Owner with the Resource Server, we

have this further characteristic of an absence of a human interaction. This means that all the outcomes of

the interaction must be produced in another way, in advance or off-line.

An example is given by the transformation of the “unauthorized request token” in a “request token” that is

the result of the user granting authorization to the third party.

1.3 OAuth & Green Button

The Green Button and the OAuth scenarios resembles very much. But these are not only analogies: as a

matter of fact GB services are strongly based on the OAuth protocol, in particular OAuth v.2.0.

In OAuth, the actors’ names are slightly different from the GB ones, so it’s better to remind the

correspondence:

GB actor OAuth actor

Retail Customer Resource Owner (RO) or user




CIP-ICT-PSP-2012-6 – 325161

Page 17 of 45




Data Custodian Resource Server6 (RS) or server or service provider

Third Party Client or consumer

Table 2 - Correspondence between GB and OAuth actors

So the use cases faced by GB can be classified in two main groups: the first that deals with the authorization

interchanges and that resembles the one of OAuth and the second that handles the real data transmission.

GB provide for dynamic authorization phase, in which actors “meet” them for the very first time and

registration, trust release and specific authorization for the resources are all managed using the OAuth

protocol and the OAuth interchanges. In Sunshine the parties involved are known from the start and some

passages can be done offline in order to simplify the operations.

In particular in the scenario “GB connect my data” the consumer is involved only in approving access to the

data, but is not (necessarily) involved in the movement of the data from the Data Custodian to a third party

service provider, and it results in a machine-to-machine communication.

In particular we’ll see in the next chapter how to install and configure an open source implementation of

Green Button services that offers a wide range of web APIs that can manage all the possible communication

between the parties. We won’t use all these APIs precisely because we want to keep the system simple.

1.4 Green Button in Sunshine scenario

In the Sunshine scenario, pilots that want to transmit data by means of the GB mechanism will have to:

1. Act like a retail customer and give Sunshine platform the authorization to access its data; this

action will be executed by a human user that interacts with the software capable of generating an

OAuth access token. The user will deal with both the Data Custodian and the Third Party software

that will interchange messages and redirect the user’s browser like in the interactions described

above. The access token will be finally stored in the Sunshine platform and used to access the

resources.

2. Expose the EUI by means of a GB service; therefore the data shall be ingested from the original

format into the database structures known by the service.

6 The Resource Server plays several roles that in OAuth can be highlighted as separated because they actually can be

implemented by different components of the architecture: Authentication server and Authorization server. OAuth

v.2.0 explicitly introduces this separation.




CIP-ICT-PSP-2012-6 – 325161

Page 18 of 45




To achieve these results, Sunshine is going to reuse the software developed as a reference implementation

by the initiative EnergyOS7 (Open Source for Energy Infrastructure) and the private company Pivotal Labs8,

called OpenESPI.

In particular, the open source software packages implemented are the

1. DataCustodian module, that is the component that publishes the energy consumption data and the

2. ThirdParty module that communicates with it for the authorization steps.

A third component, developed in the Sunshine project, is the GreenButton Client, that is the module that,

once obtained the authorization token, will handle the mere EUI transfer operation.

Another function that the DataCustodian offers is the ingestion of the EUI data from original sources. A

service is exposed to facilitate the ingestion of the readings in the EUI database: the pilot can either

activate the “B” flow using this service (encoding the readings in a XML request) or the “A” flow loading the

data directly in the database with some custom scripts (see the blue arrows in Figure 8).

Like written above, in Green Button jargon, Pilot plays the role of Data Custodian and Sunshine platform

the Third Party one. Therefore the software module DataCustodian must be installed at the Pilot’s side that

whilst the ThirdParty module will be installed on the Sunshine platform.

DataCustodianThirdParty

EUI internal flow

EUI

Pilot

Sunshine

datacustodianDB

(EUI)

Token DB

Original consumption

data

Oauth flow

SOSGreenButton

Client

A

B

B

Figure 8 – The interactions between Pilot and Sunshine platform

7 http://www.energyos.org/

8 http://pivotallabs.com/




CIP-ICT-PSP-2012-6 – 325161

Page 19 of 45




In the next chapter only the instruction to install the DataCustodian module will be given. Then, we’ll

describe the procedure to assign to the Sunshine platform a token to allow the EUI access and finally how

to feed the GB service with the consumption data.




CIP-ICT-PSP-2012-6 – 325161

Page 20 of 45




2 Software Installation

The DataCustodian is available as a web application packed in “war” file that is downloadable at the

address:

http://sunshine.sinergis.it/download/greenbutton/1.1/DataCustodian.war

**The package is based on the version 1.1-SNAPSHOT released on 4th September 2014 and has been

adapted specifically for Sunshine needs. The current version available at the link above and on which are

shaped the current guidelines differs from the one that was available for download at the time of release of

version v0.9 of these guidelines, so pilots are invited to download and use the current WAR version.

2.1 Software Requirements (DataCustodian)

The installation of the following software is required:

Java JRE (Java Run-Time Engine) 1.7.0 or higher (downloadable from http://www.java.sun.com)

Servlet container Apache Tomcat v. 7.0.54 or higher (downloadable from

http://jakarta.apache.org/tomcat)

Moreover, the availability of a mySQL relational database is required9 to host the token store. The EUI database can be implemented either by a mySQL DB or by a PostgreSQL.

2.2 Installation instructions

The deployment diagram of the DataCustodian application is shown in Figure 9.

The DB Server and the Application Server may be one and the same server.

The two databases maintain the consumption data that are published by the DataCustodian services, but

also support the OAuth mechanism implemented by GB. They could be two instances installed on separate

DB servers, but to keep it simple, consider them two SCHEMAs (same as two DATABASEs) in the same

server and in the same mySQL instance.

9 Next release should support also PostgreSQL and HSQL


http://sunshine.sinergis.it/download/greenbutton/1.1/DataCustodian.war

http://www.java.sun.com/

http://jakarta.apache.org/tomcat



CIP-ICT-PSP-2012-6 – 325161

Page 21 of 45




Figure 9 - Deployment diagram of DataCustodian application

2.2.1 Java JRE

The following instructions refer to a MS Windows environment (64bit), but are valid also for a Unix/Linux

environment, provided that the relevant path adjustments are applied:

1. Create a subfolder named greenbutton in the root folder of your system.

2. Download and execute jdk-7u65-windows-x64.exe.

3. Select as installation folder: C:\ greenbutton\jre1.7.0_45.

4. Set the environment variable JAVA_HOME to JAVA_HOME = C:\ greenbutton\ jre1.7.0_45.

Even if a JRE is already present, check that the installation path (step 3) has no blank spaces and that the

environment variable JAVA_HOME (step 4) is properly set.

2.2.2 Apache Tomcat

The following instructions refer to a MS Windows environment, but are valid also to a Unix/Linux


1. Create a subfolder named greenbutton in the root folder of your system.

2. Download apache-tomcat-7.0.54.zip.

3. Unpack the package apache-tomcat-7.0.54.zip in the greenbutton folder.

4. Set the environment variable CATALINA_HOME to CATALINA_HOME = C:\ greenbutton\apache-

tomcat-7.0.54.




CIP-ICT-PSP-2012-6 – 325161

Page 22 of 45




5. Open file <CATALINA_HOME>/bin/startup.bat and set memory allocation parameters for Java Virtual

Machine (JVM) as follows: set CATALINA_OPTS=-server –Xms2048m –Xmx2048m -XX:PermSize=256m

-XX:MaxPermSize=256m.

Even if Apache Tomcat is already installed in the server, check that the installation path has no blank spaces

and that the environment variable CATALINA_HOME (step 4) is properly set.

2.2.3 MySQL

The following instructions refer to a MS Windows environment, but are valid also to a Unix/Linux


1. Download MySQL Installer from http://dev.mysql.com/downloads/installer/ and execute it.

2. Choose the appropriate Setup Type for your system (developer or custom).

3. Follow the wizard’s instructions.

2.2.3.1 DataCustodian DB (EUI)

The datacustodian DB is meant to host the readings (EUI). It is automatically created by the application

DataCustodian when it is run for the very first time with some particular configuration settings. See the

paragraph 2.2.4.1 for the details; here the instruction for mySQL DB are illustrated, but the postgreSQL case

is different just for the connection parameters and the configuration file name. Here and in the rest of the

document the EUI database has been named “datacustodian”, but the pilot can choose whatever name and

modify consequently the configuration files.

2.2.3.2 Token DB

The Token DB is just for the authorization phase. It must be created manually running the SQL creation

scripts that you’ll find in

http://sunshine.sinergis.it/download/greenbutton/1.1/createSunshineTS_mySQL.sql and listed hereinafter.

CREATE DATABASE IF NOT EXISTS `tokenstore` /*!40100 DEFAULT CHARACTER SET utf8 */;

USE `tokenstore`;

-- MySQL dump 10.13 Distrib 5.5.35, for debian-linux-gnu (x86_64)

--

-- Host: 127.0.0.1 Database: tokenstore

-- ------------------------------------------------------

-- Server version 5.5.32

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;

/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;

/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;

/*!40101 SET NAMES utf8 */;

/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;

/*!40103 SET TIME_ZONE='+00:00' */;

/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;


http://dev.mysql.com/downloads/installer/

http://sunshine.sinergis.it/download/greenbutton/1.1/createSunshineTS_mySQL.sql



CIP-ICT-PSP-2012-6 – 325161

Page 23 of 45




/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;

/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;

/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;

--

-- Table structure for table `clientdetails`

--

DROP TABLE IF EXISTS `clientdetails`;

/*!40101 SET @saved_cs_client = @@character_set_client */;

/*!40101 SET character_set_client = utf8 */;

CREATE TABLE `clientdetails` (

àppId` varchar(255) NOT NULL,

`resourceIds` varchar(256) DEFAULT NULL,

àppSecret` varchar(256) DEFAULT NULL,

`scope` varchar(256) DEFAULT NULL,

`grantTypes` varchar(256) DEFAULT NULL,

`redirectUrl` varchar(256) DEFAULT NULL,

àuthorities` varchar(256) DEFAULT NULL,

àccess_token_validity` int(11) DEFAULT NULL,

`refresh_token_validity` int(11) DEFAULT NULL,

àdditionalInformation` varchar(4096) DEFAULT NULL,

àutoApproveScopes` varchar(256) DEFAULT NULL,

PRIMARY KEY (àppId`)

) ENGINE=InnoDB DEFAULT CHARSET=utf8;

/*!40101 SET character_set_client = @saved_cs_client */;

--

-- Table structure for table òauth_access_token`

--

DROP TABLE IF EXISTS òauth_access_token`;



CREATE TABLE òauth_access_token` (

`token_id` varchar(256) DEFAULT NULL,

`token` mediumblob,

àuthentication_id` varchar(256) DEFAULT NULL,

ùser_name` varchar(256) DEFAULT NULL,

`client_id` varchar(256) DEFAULT NULL,

àuthentication` mediumblob,

`refresh_token` varchar(256) DEFAULT NULL



--

-- Table structure for table òauth_approvals`

--

DROP TABLE IF EXISTS òauth_approvals`;



CREATE TABLE òauth_approvals` (

ùserId` varchar(256) DEFAULT NULL,

`clientId` varchar(256) DEFAULT NULL,


`status` varchar(10) DEFAULT NULL,

èxpiresAt` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,




CIP-ICT-PSP-2012-6 – 325161

Page 24 of 45




`lastModifiedAt` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00'



--

-- Table structure for table òauth_client_details`

--

DROP TABLE IF EXISTS òauth_client_details`;



CREATE TABLE òauth_client_details` (

`client_id` varchar(255) NOT NULL,

`resource_ids` varchar(256) DEFAULT NULL,

`client_secret` varchar(256) DEFAULT NULL,


àuthorized_grant_types` varchar(256) DEFAULT NULL,

`web_server_redirect_uri` varchar(256) DEFAULT NULL,

àuthorities` varchar(256) DEFAULT NULL,

àccess_token_validity` int(11) DEFAULT NULL,

`refresh_token_validity` int(11) DEFAULT NULL,

àdditional_information` varchar(4096) DEFAULT NULL,

àutoapprove` varchar(256) DEFAULT NULL,

PRIMARY KEY (`client_id`)



--

-- Table structure for table òauth_client_token`

--

DROP TABLE IF EXISTS òauth_client_token`;



CREATE TABLE òauth_client_token` (


`token` mediumblob,

àuthentication_id` varchar(256) DEFAULT NULL,

ùser_name` varchar(256) DEFAULT NULL,

`client_id` varchar(256) DEFAULT NULL



--

-- Table structure for table òauth_code`

--

DROP TABLE IF EXISTS òauth_code`;



CREATE TABLE òauth_code` (

`code` varchar(256) DEFAULT NULL,

àuthentication` mediumblob



--

-- Table structure for table òauth_refresh_token`

--




CIP-ICT-PSP-2012-6 – 325161

Page 25 of 45




DROP TABLE IF EXISTS òauth_refresh_token`;



CREATE TABLE òauth_refresh_token` (


`token` mediumblob,

àuthentication` mediumblob



/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;

/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;

/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;

/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;

/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;

/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;

/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;

/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;

Download the SQL script

http://sunshine.sinergis.it/download/greenbutton/1.1/createSunshineTS_mySQL.sql and run inside the

“mySQL workbench” or some other SQL console connected to the mySQL instance that will host the

tokenstore DB .

2.2.3.3 Populating the token DB

The tokenstore DB should be prepopulated with some records in order to enable the application to create

the access token for the Sunshine platform.

A SQL script containing the insert statements can be downloaded at the address

http://sunshine.sinergis.it/download/greenbutton/1.1/prepopulateSunshineTS_mySQL.sql.

For the data exchange the needed record is the first one that describes the third party user (Sunshine

Platform). The other records are respectively:

a user that the pilot can use to write and read his own data;

a user that the pilot can use to read all the authorizations released to the third party applications.

In all these records a string “secret” is used as a default value for the field “client_secret”, but this should

be changed: they are the string that are used to encrypt the messages exchanged between the parties

involved in the OAuth flow. Obviously the three strings can be different: so change them in the SQL file


http://sunshine.sinergis.it/download/greenbutton/1.1/createSunshineTS_mySQL.sql

http://sunshine.sinergis.it/download/greenbutton/1.1/prepopulateSunshineTS_mySQL.sql



CIP-ICT-PSP-2012-6 – 325161

Page 26 of 45




created from the below snippet. This string must be pre-shared in some way with the third party (Sunshine

Platform) 10 .

The value of “scope” is a coded string that specifies the parameters of the reading that Sunshine Platform

(the third party) is interested in.

The “access_token_validity” is expressed in seconds and is set to one year, while the

“refresh_token_validity” is set to five years.

Very important is the value of the call back URL that is set to

http://sunshine.sinergis.it/ThirdParty/espi/1_1/OAuthCallBack. It’s the address of the endpoint

of a ThirdParty service exposed at the Sunshine Platform and necessary for the authorization token release.

INSERT INTO òauth_client_details` VALUES

('sunshine_platform', NULL, 'secret', 'read', 'authorization_code,refresh_token',

'http://sunshine.sinergis.it/ThirdParty/espi/1_1/OAuthCallBack', 'ROLE_USER', '31536000',

'157680000', NULL, 'FALSE');


('data_custodian_admin', NULL, 'secret', 'DataCustodian_Admin_Access',

'client_credentials', NULL, 'ROLE_DC_ADMIN', '31536000', NULL, NULL, 'FALSE');


('third_party_admin', NULL, 'secret', 'ThirdParty_Admin_Access', 'client_credentials',

NULL, 'ROLE_TP_ADMIN', '31536000', NULL, NULL, 'FALSE');

Download the SQL script located at

http://sunshine.sinergis.it/download/greenbutton/1.1/prepopulateSunshineTS_mySQL.sql.

Change the “secret” strings contained in the script.

Run the script inside the “mySQL workbench” or some other SQL console connected to the mySQL

instance that hosts the tokenstore DB .

10 Instead of using unsecure communication like mail or document transfer, one could use the web site

https://onetimesecret.com/ that allows exchanging privately small amount of data in a very simple manner.


http://sunshine.sinergis.it/ThirdParty/espi/1_1/OAuthCallBack

http://sunshine.sinergis.it/download/greenbutton/1.1/prepopulateSunshineTS_mySQL.sql

https://onetimesecret.com/



CIP-ICT-PSP-2012-6 – 325161

Page 27 of 45




Communicate the “secret” string and the endpoint of your DataCustodian installation to the Sunshine

Platform administrators in order to let them configure the ThirdParty.

2.2.4 DataCustodian application

After downloading the war file, it is necessary to modify some configuration files inside it. It is important to

do it inside the zipped war so use an editor that can do it or extract the file, modify it and then re-insert it in

the war file.

These are configurations that Tomcat must read the first time it is run, that is when unpacks and deploys

the war file content in its folder

<CATALINA_HOME>/webapps/

Then Tomcat is started for the first time and in this run it will do several one-shot actions; after this first run

it must be stopped, some other configurations must be done (this time in the deployed unzipped files) and

then Tomcat finally restarted to have the application working.

2.2.4.1 Pre-install configurations

The files to be configured tell the application which is the database to connect to, the behaviour to adopt

with the DB, and which tables create; the files that need to be modified are the following:

1. The first file is a “.properties”:

WEB-INF\classes\spring\mysql-data-access.properties

and must be modified setting the DB connection parameters:

jdbc.url=jdbc:mysql://<dbserver>:<dbport>/datacustodian

jdbc.username=<username>

jdbc.password=<password>

2. Then there are two XML files; the first is :

WEB-INF\classes\spring\business-config.xml

and must be modified inserting the “create-drop” value in the following element:

<prop key="hibernate.hbm2ddl.auto">create-drop</prop>

This tells the application to create the tables needed eventually dropping the existing ones.

In the same file, insert the file name mysql-data-access.properties in the location attribute of the

element context:property-placeholder as follows:

<context:property-placeholder location="classpath:spring/mysql-data-

access.properties" system-properties-mode="OVERRIDE"/>

The second XML file is:




CIP-ICT-PSP-2012-6 – 325161

Page 28 of 45




WEB-INF\classes\spring\datasource-config.xml

and in this file we must do the last same modification:

<context:property-placeholder location="classpath:spring/mysql-data-

access.properties" system-properties-mode="OVERRIDE"/>

3. The third configuration regards another XML file:

WEB-INF\classes\spring\oauth-AS-config.xml

Here find and edit the element “property” with attribute “baseURL” with the appropriate URL for the

DataCustodian application:

<property name="baseURL" value="http://<pilot DataCustodian server>/DataCustodian"/>

Inside this file there are also a lot of references to the “token DB” and the jdbc-style connection URL with the user and relative password; these credentials must be changed to match the ones of the installed instance of the mySQL DB and the connection URL must be set to the real one if it is not the same server of the DataCustodian application (it’s default setting is jdbc:mysql://localhost:3306/tokenstore).

2.2.4.2 Installation and first run

Copy the modified war file into folder:

<CATALINA_HOME>/webapps/

when Apache Tomcat is not running. Then start Tomcat and check if it connects to the database and creates

some tables in the “datacustodian” database inside the mySQL instance.

2.2.4.3 Post-installation configurations

Stop Apache Tomcat again; the datacustodian tables structure has been created during the first run and

now we must instruct the application not to drop and recreate the tables each time it is started. To do this,

we must modify the file business-config.xml anew, but this time the unzipped one in the Tomcat’s folder:

<CATALINA_HOME>\webapps\DataCustodian\WEB-INF\classes\spring\business-config.xml

Open it and insert the value “update” in the following element, overwriting the previous value:

<prop key="hibernate.hbm2ddl.auto">update</prop>

You can also drop the war file from the webapp folder and finally start Tomcat again and the DataCustodian application should be up and running.




CIP-ICT-PSP-2012-6 – 325161

Page 29 of 45




3 Configuration and Use

In this chapter we will see how to set up the EUI database, feed it with the data coming from the meters

and let them be harvested by the Sunshine platform. In order to do so, the pilot should set up a mechanism

that feed the datacustodian DB ingesting the data from the original source (the meters loggers, the AMI,

the files coming from the utilities company …) to the standard tables of the datacustodian database and

expose the service that enables the transmission of data to Sunshine.

DataCustodian application provides two services (import & export) that allow these operations: the former

can be used by the pilot back-end procedures and the latter will be used by the Sunshine platform. Both are

protected by OAuth mechanism so they can be exposed on the internet.

3.1 datacustodian DB structure

The structure of the datacustodian DB that is relevant to the implementation of the Web Service and deals

with the consumption data is described in Figure 10.

Figure 10 - Database structure

Table retail_customers stores the authentication information of the service’s users. The users to set up are

two: the service administrator, who is represented by the pilot technical partner, and the “retail user”, that

is the owner of the data.




CIP-ICT-PSP-2012-6 – 325161

Page 30 of 45




Table usage_points stores the meter information. Every meter is a usage point and one or more usage

points can be associated to a single retail customer.

Table time_configuration stores the information about time zone and daylight saving time rules for the

area where the usage point is located. Consumption timestamps in the DB are stored in UTC format and

this type of information is used in order to convert data into local time frame, if needed. For the purposes

of Sunshine project conversion into local time is not needed, so the time configuration can be set to UTC

(see the following sub-section).

Table meter_reading lists the available types of reading coming from the usage points (the meters).

Examples of meter readings form the same usage point (i.e. a gas meter) can be “the set of readings for a

certain period” or “the readings at an interval of 4 hours” to be distinguished from “the readings at an

interval of 15 min”. Typically however every usage point is associated to just one meter reading.

Table reading_types describes the properties of the readings. Every meter reading is associated to a

reading type describing i.e. the type of energy measured, the frequency, the nature of the measurement

(absolute, integrated, etc).

Table interval_blocks stores the information about the regular data feeds into the DB. Each ingestion of

consumption data (be it a single reading or a chunk of readings) correspond to an interval block and is

associated to a specific meter reading.

Table interval_readings stores the actual data about consumption and cost. Each entry corresponds to a

single reading (energy consumption and its related cost, if available) and is linked to the interval block it

was part when it was imported in the DB.

As Figure 10 shows, part of the tables require only a one-time set-up, while tables interval_blocks and

interval_readings require regular updates as they store the actual dynamic consumption data. Section 3.2

describes how one-time tables can be set-up via direct SQL INSERT calls, while section 3.3 describes how to

insert dynamical consumption data into the DB and section 3.4 illustrates a procedure to extract the data

from the DB to verify its content.




CIP-ICT-PSP-2012-6 – 325161

Page 31 of 45




3.2 Populating the datacustodian database

This section is a collection of template INSERT statements to populate the one-time tables of datacustodian

DB. Important note: each entry of every table has to have a unique UUID11 which has to be generated by

the pilots.

The insert statements are gathered in a SQL file available for download at the address

http://sunshine.sinergis.it/download/greenbutton/1.1/prepopulateSunshineDC_postgreSQL.sql.

Note that some information is duplicated and are present in the tokenstore DB as well as in the

datacustodian DB.

As a general consideration, all these tables contain fields with names “self_link_href” or “up_link_href”

coupled with fields “self_link_rel” and “up_link_rel”. The values for these fields are respectively the URLs of

the resource itself (self_link_href) or of the resource of the upper hierarchical level (up_self_link) and the

qualifiers of these links. This information is used to form the response of the REST services following the

principle of the HATEOAS constraint of the REST architecture.

As a common characteristic, records have a UUID as external identifier, so this UUID must be regenerated

and substituted before running the script.

3.2.1 ‘retail_customers’ table

This table contains the users of the datacustodian application: the first user represents the retail customer

and will have a role that allows him or her to give a third party the authorization to access the data;

another role of this user is the feeding of the EUI data, using a specific import service as we’ll see in § 3.3.

Another user is the data custodian itself that is enabled for some administration activities that are not

important in this context.

The ids of a resource (in this case of a retail customer) are very important as in REST syntax they are used to

form the URIs that allow manipulating the resource itself: here we are using a progressive.

Retail user (building owner)

INSERT INTO retail_customers(

11 http://en.wikipedia.org/wiki/Universally_unique_identifier. An online UUID generator derived from the current

time can be found here (just for testing purposes): http://www.famkruithof.net/uuid/uuidgen


http://sunshine.sinergis.it/download/greenbutton/1.1/prepopulateSunshineDC_postgreSQL.sql

http://en.wikipedia.org/wiki/Universally_unique_identifier

http://www.famkruithof.net/uuid/uuidgen



CIP-ICT-PSP-2012-6 – 325161

Page 32 of 45




id, description, published, self_link_href, self_link_rel, up_link_href,

up_link_rel, updated, uuid, enabled, first_name, last_name, password,

role, username)

VALUES (

1,'Ferrara municipality','2014-12-31','/RetailCustomer/1','self',

'/RetailCustomer','up','2014-12-31','106E8B03-0299-467E-972A-

A883ECDCC598','1','Servizio energia','Ferrara','F3rr@r@','ROLE_USER',

'ferrara');

Data custodian (Pilot admin)

INSERT INTO retail_customers(

id, description, published, self_link_href, self_link_rel, up_link_href,

up_link_rel, updated, uuid, enabled, first_name, last_name, password,

role, username)

VALUES (

2, 'Pilot Ferrara Data Custodian', '2014-12-31', '/RetailCustomer/2',

'self','/RetailCustomer','up','2014-12-31','106E8B03-0299-467E-972A-

A883ECDCC599','1','Sinergis', 'Sinergis', '21n3r612','ROLE_CUSTODIAN',

'admin');

3.2.2 ‘application_information’ table

This table contains information about the applications that are registered in order to use the datacustodian

services. Of particular importance is the application that has the attribute “kind” with value

“THIRD_PARTY”, that is the Sunshine platform.

3.2.3 ‘application_information_scopes’ table

This table contains the scopes related to the applications listed in the previous table.

3.2.4 ‘time_configurations’ table

Here we insert the two records that represent the UTC and CET Summer time zones codifications.

Time configuration: UTC




CIP-ICT-PSP-2012-6 – 325161

Page 33 of 45




INSERT INTO time_configurations(

id, description, uuid, dstoffset, tzoffset)

VALUES (

1, 'UTC', '/LocalTimeParameters/1', 'self', '/LocalTimeParameters', 'up',

'106E8B03-0299-467E-972A-A883ECDCC593', 0, 0);

Time configuration: Central European (Summer) Time

Annex 1 – DST rules describes how to compute the values of dststartrule and dstendrule in order to define a

Daylight Saving Time rule.

INSERT INTO time_configurations(

id, description, uuid, dstendrule, dstoffset, dststartrule, tzoffset)

VALUES (

2, 'Central European (Summer) Time', '/LocalTimeParameters/2', 'self',

'/LocalTimeParameters', 'up', '106E8B03-0299-467E-972A-A883ECDCC595',

decode('AE0E2000', 'hex'), 3600, decode('3E0E2000', 'hex'), 3600);

If you use a time zone different from UTC or CET summer time, please add it.

3.2.5 ‘usage_points’ table

A “usage point” in GreenButton is roughly equivalent to a single meter. It has a relation 1:n with the

“retail_customers” table that is a retail customer can be linked to more than one usage point. The code list

for the servicecategory_kind attribute is described in Annex 2 – Codelists.

INSERT INTO usage_points(

id, description, self_link_href, self_link_rel, up_link_href,

up_link_rel, uuid, servicecategory_kind, localtimeparameters_id,

retail_customer_id)

VALUES (

1, 'FER-001', '/RetailCustomer/1/UsagePoint/1', 'self',

'/RetailCustomer/1/UsagePoint', 'up', '106E8B03-0299-467E-972A-

A883ECDCC594', 1, 1, 1);




CIP-ICT-PSP-2012-6 – 325161

Page 34 of 45




3.2.6 ‘reading_types’ table

A reading type is a group of readings that share the same characteristics in terms of:

type of commodity (gas, electricity …)

unit of measure

the type of measure (incremental or cumulative)

the currency for eventual costs

and other more detailed and not mandatory features. A reading type is related to 1 or more meter reading.

The codelists for the attributes accumulationbehaviour, commodity, currency, timeattribute and uom are

described in Annex 2 – Codelists.

Just as examples the accumulationbehaviour code indicates if a measure of energy regards an interval of

time of the cumulative measure since the start of measuring; the commoditytype is the kind of energy

mean; currency is always euro; timeattribute is the reading frequency and finally the uom is the unit of

measure of the reading.

INSERT INTO reading_types(


up_link_rel, uuid, accumulationbehaviour, commodity, currency,

timeattribute, uom)

VALUES (

1, 'GASR_MCU_1h', '/ReadingType/1', 'self', '/ReadingType', 'up',

'106E8B03-0299-467E-972A-A883ECDCC593', 4, 7, 0, 7, 42);

Here is shown a gas meter reading, with a frequency of an hour.

3.2.7 ‘meter_readings’ table

A meter reading is not the real reading coming from the meter, but a way to group them in order to

simplify the relation with the usage point and the characteristics of the reading stored in the reading_types

table. In fact the real readings are stored in the interval_block table where only the relation with the

meter_reading table is necessary.

INSERT INTO meter_readings(




CIP-ICT-PSP-2012-6 – 325161

Page 35 of 45





up_link_rel, uuid, reading_type_id, usage_point_id)

VALUES (

1, 'FER-001_GASR_MCU_1h',

'/RetailCustomer/1/UsagePoint/1/MeterReading/1', 'self',

'/RetailCustomer/1/UsagePoint/1/MeterReading', 'up', '106E8B03-0299-467E-

972A-A883ECDCC592', 1, 1);

Here is shown the gas meter reading type is associated with the usage point #1.

Download the SQL script located at

http://sunshine.sinergis.it/download/greenbutton/1.1/prepopulateSunshineDC_postgreSQL.sql.

(If you installed the DataCustodian DB in a mySQL instance, just add single quotes to table and field names

and to values to get the mySQL syntax.

Change the UUID, the date values, the client secret and the DataCustodian endpoint. These changes

are indicated in the script file by comments. Even the retail user’s credential should be changed.

Run the SQL script inside a SQL console connected to the datacustodian DB.

Communicate to the Sunshine Platform admin the name of the registered retail users12

3.3 Import data service

In order to populate the datacustodian DB with the dynamical consumption data coming from the energy

meters, the DataCustodian application is provided with an import service. Consumption readings can be

imported one by one or in a contiguous group (as meters may not always be capable of sending reading in

real time, but instead may store them temporarily and send them in bundles) and each data import

corresponds to a new interval block item in the DB.

Consumption data has to be formatted in XML and posted to the import service whose URL specifies the

RetailCustomer, UsagePoint and MeterReading the data refers to. The service is a RESTlike service whose

12 At the moment the ThirdParty application has not a self registration functionality.


http://sunshine.sinergis.it/download/greenbutton/1.1/prepopulateSunshineDC_postgreSQL.sql



CIP-ICT-PSP-2012-6 – 325161

Page 36 of 45




URL indicates the resource to be modified (the IntervalBlock); a POST to this URL should be invoked with a

payload coded as an XML. Service URL and XML template are described below.

3.3.1 Service url

The URL of the import service is like this:

http://<pilot-greenbutton-server>/DataCustodian/espi/1_1/resource/<interval-block-up-link>

where <interval-block-up-link> has the form:

RetailCustomer/X/UsagePoint/Y/MeterReading/Z/IntervalBlock

Labels X, Y and Z are respectively the IDs of the RetailCustomer whose data are being uploaded and the

relative UsagePoint and the MeterReading they refer to.

Important note: generally speaking, different interval blocks of consumption data will correspond to

different UsagePoints/MeterReadings (as each pilot have more than one meter), so be careful to set X, Y

and Z values accordingly each time you call the import service.

i.e. http://sunshine-fe.sinergis.it/DataCustodian/espi/1_1/resource/RetailCustomer/1/UsagePoint/3/

MeterReading/2/IntervalBlock

Where "sunshine-fe.sinergis.it" is the base URL for the server exposing consumption data for the pilot in

Ferrara, and RetailCustomer id = 1 ("Ferrara municipality"), UsagePoint id = 3 (gas meter in kindergarten

Rampari) and MeterReading id = 4 (hourly reading of total cubic meters of consumed gas).

3.3.2 Consumption data XML

The XML-formatted data provided below is a template for the format consumption data that has to be

provided in to the data import service:

<ns3:entry xmlns:espi="http://naesb.org/espi"

xmlns:ns3="http://www.w3.org/2005/Atom">

<ns3:id>urn:uuid:3061204d-0148-1000-0000-000000000318</ns3:id>

<ns3:link href="http://sunshine-fe.sinergis.it/DataCustodian/espi/

1_1/resource/RetailCustomer/1/UsagePoint/3/MeterReading/4/IntervalBlock"

rel="up"/>

<ns3:content>

<espi:IntervalBlock>

<espi:interval>

<espi:duration>7200</espi:duration>

<espi:start>975628800</espi:start>


http://sunshine-fe.sinergis.it/DataCustodian/espi/1_1/resource/RetailCustomer/1/UsagePoint/3/



CIP-ICT-PSP-2012-6 – 325161

Page 37 of 45




</espi:interval>

<espi:IntervalReading>

<espi:timePeriod>



</espi:timePeriod>

<espi:value>1023</espi:value>

<espi:cost>1987</espi:cost>

</espi:IntervalReading>


<espi:timePeriod>



</espi:timePeriod>

<espi:value>1107</espi:value>

<espi:cost>2139</espi:cost>

</espi:IntervalReading>

</espi:IntervalBlock>

</ns3:content>

<ns3:published>2000-12-01T02:00:00+01:00</ns3:published>

<ns3:updated>2000-12-01T02:00:00+01:00</ns3:updated>

</ns3:entry></ns3:entry>

<ns3:id>

this will be the UUID of the interval block we are importing. As for all the items in the DB, it has to be

unique and has to be generated by the user13.

<ns3:link >

this is exactly equal to the import service URL specified in the previous sub-section and is likewise

dependent on the specific IDs of the RetailUser, UsagePoint and MeterReading the imported consumption

data refer to.

<espi:interval>

This block describes the overall start and duration of the uploaded group of consumption readings.

Duration is expressed in seconds while the start date is expressed in Unix time14, that is defined as the

number of seconds that have elapsed since the midnight UTC of January 1st 1970, not counting leap

seconds. Important note: This means that start date is expressed in UTC time reference frame.

13 See footnote 11

14 http://en.wikipedia.org/wiki/Unix_time. An online conversion tool (just for the sake of testing) can be found here:

http://www.onlineconversion.com/unix_time.htm


http://en.wikipedia.org/wiki/Unix_time



CIP-ICT-PSP-2012-6 – 325161

Page 38 of 45





This block describes start date and duration of the single reading, following the same format of the

espi:interval tag, as well as the measured consumption value and the related cost.

Consumption value has to be expressed as thousandth (10-3) of the measuring unit (i.e. 1 kWh has to be

written as 1.000).

Cost has to be expressed as hundred-thousandth (10-5) of the currency unit (i.e. 1 Euro has to be written as

100.000). Cost can be omitted if not available.

<ns3:published>

Both tags ns3:published and ns3:updated have to be populated and they have to report the same value,

corresponding to the timestamp of the end of the interval block, that is to say the start time of the

espi:interval plus its duration.

The format of the timestamp is the ISO Time15 format. Time can be specified in any time zone reference,

but UTC reference is preferred (i.e. 2000-12-01T01:00:00Z [UTC] and 2000-12-01T02:00:00+01:00 [CET] are

equivalent and are both accepted).

3.4 Export data

Consumption data properly uploaded into the DB will be then periodically harvested by the central

sunshine platform in order to feed the central sensor DB and be from there exposed to the final sunshine

user.

In order to allow sunshine central platform to properly ingest data coming from GreenButton export

service, each pilot has to fill a mapping file along the lines of the following template. Pilots should also

specify the ID of the RetailUser associated with the sunshine platform (recommended ID is ID = 1).

Meter identification

(from meter_mapping file)

Meter Reading ID Reading Type ID Usage Point ID

FER-003_GASA_MCU_1m 4 1 3

[…]

15 http://en.wikipedia.org/wiki/ISO_8601




CIP-ICT-PSP-2012-6 – 325161

Page 39 of 45




Pilots can test the export service by calling the URL

http://<pilot-greenbutton-server>/

DataCustodian/espi/1_1/resource/Batch/RetailCustomer/X/UsagePoint/Y

where labels X and Y are respectively the IDs of the RetailCustomer (has to be the “sunshine platform” user

defined in the previous section, typically id = 1) and UsagePoint of interest.

The export service must be exposed on the internet and should be reachable with HTTP protocol.

3.5 Security underneath the application layer

OAuth works at the same OSI layer of HTTP (application layer or layer 7). In order to achieve a complete

security of the data flow in GB “Connect My Data” scenario, OAuth is not enough and Transport Layer

Security (TLS) is required for all exchanges. Despite its name, TLS works at presentation layer (OSI 6 layer)

and during the negotiation is on OSI layer 5, session layer. The developers of the protocol recommend that

all implementations move to TLS version 1.2. However, most client web browsers are still at TLS 1.0 or at

best TLS 1.1. So, the following levels of TLS are required for the different actors:

Data Custodian – Must implement TLS 1.2

Third Party – Must Implement TLS 1.1 or higher

Retail Customer – Can Implement TLS 1.0, 1.1, or 1.2

Each party must choose the highest level of TLS mutually supported during the TLS negotiation. Thus in the

rest of the document whereas in an URL an http: schema is written, in production you should use the

https: one.

3.6 Generation of the access token

The generation of the access token is done interactively by the retail customer accessing the GUI of the

DataCustodian web application installed at the pilot’s server and through a redirection to the GUI of the

ThirdParty web application installed on the Sunshine Platform, reproducing the interaction of the OAuth

protocol.

The generation of the access token involves the active participation of two actors over the web: the pilot

(DataCustodian) and the Sunshine Platform (ThirdParty). Thus they must know of each other

and must see each other over the internet. In other word their configuration must provide the

URL reference of the necessary service, the must share the same “secret”, the human user that

does the operation must be registered on both platforms.




CIP-ICT-PSP-2012-6 – 325161

Page 40 of 45




The steps are the following:

1. Access the DataCustodian GUI at the address:

http://<pilot-greenbutton-server>/DataCustodian/

(i.e. http://sunshine-fe.sinergis.it/DataCustodian/ )

2. Login the application

Here the retail customer should authenticate him/herself with the credentials that have been assigned to

him/her by the administrator of the DataCustodian application.

3. The application will show the user a welcome page where a menu bar allows to activate the following

functions:

a. Usage Points (list the configured usage points whose data are maintained in the DB)

b. Third Parties (list the configured third parties known to the Data Custodian and that can be

authorized to access the retail customer’s EUI data).

c. Logout

4. Click the “Third Parties” menu item and a radio button list (probably formed by just one element) of

organizations that can play the third party role will be shown. Aside the third party, there will be the

URL to which the retail user will be redirected for the scope selection phase. This is an address of the

ThirdParty application installed in the Sunshine platform.

5. Choose the “Sunshine platform” item and click next.

6. You will be redirected to the Third Party login page since you’re not authenticated in the Sunshine

platform, but only at the pilot’s side.

7. After the login, a radio button list with the type of authorizations that can be granted (scope) will be

shown. For our purposes, only a scope “read” will be in the list: choose it and click next.

8. The ThirdParty application will therefore redirect the user’s browser to a DataCustodian page (no

authentication needed this time) to confirm the intention of giving “read” authorization access to

“Sunshine platform”. Choose “Approve” and click “Submit”.

9. Now, another redirection back to ThirdParty application, to a page where the Access Token (and its

relative Refresh Token) generated by the sequence of operations just concluded are shown.

The generated tokens are also stored in the ThirdParty DB to be used by the Sunshine platform for

accessing the EUI data.


http://sunshine-fe.sinergis.it/DataCustodian/



CIP-ICT-PSP-2012-6 – 325161

Page 41 of 45




Generate the access token for the Sunshine Platform by following the steps above. Before beginning,

assure that the retail user has been registered on the pilot’s DataCustodian application and on the Sunshine

platform ThirdParty application. There’s no self-registration, so these operations must be done by the

administrators. Take contact with the Sunshine Platform administrator to share the user name and his/her

password.




CIP-ICT-PSP-2012-6 – 325161

Page 42 of 45




Annex 1 – DST rules

Daylight Saving Time (DST) is defined by two rules, one identifying the start date of DST within a year and the second identifying the end date. Each country or group of countries define its own rules, for instance North America and EU use two slightly different set of rules. Each rule can be encoded in an 8-digit hex string following the following schema.

The bit encoding:

Bits 0 - 11: seconds 0 - 3599

Bits 12 - 16: hours 0 - 23

Bits 17 - 19: day of the week 0 = not applicable, 1 - 7 (Monday = 1)

Bits:20 - 24: day of the month 0 = not applicable, 1 - 31

Bits: 25 - 27: operator (detailed below)

Bits: 28 - 31: month 1 - 12

Rule value of 0xFFFFFFFF means rule processing/DST correction is disabled.

The operators:

0: DST starts/ends on the Day of the Month

1: DST starts/ends on the Day of the Week that is on or after the Day of the Month

2: DST starts/ends on the first occurrence of the Day of the Week in a month

3: DST starts/ends on the second occurrence of the Day of the Week in a month

4: DST starts/ends on the third occurrence of the Day of the Week in a month

5: DST starts/ends on the forth occurrence of the Day of the Week in a month

6: DST starts/ends on the fifth occurrence of the Day of the Week in a month

7: DST starts/ends on the last occurrence of the Day of the Week in a month

An example:

EU DST rule for CET (UTC+01:00)

Start: Last Sunday in March

End: Last Sunday in October

Time: 1.00 am (01:00) Greenwich Mean Time (GMT)

http://wwp.greenwichmeantime.com/time-zone/rules/eu/




CIP-ICT-PSP-2012-6 – 325161

Page 43 of 45




The hex strings:

<dstEndRule>AE0E2000</dstEndRule>

<dstStartRule>3E0E2000</dstStartRule>

The conversion:




CIP-ICT-PSP-2012-6 – 325161

Page 44 of 45




Annex 2 – Codelists

The codelists reported in this annex are limited to the values that are used for the consumption meters in

the project sunshine. The following subsections will provide the mapping between the meter properties

already described by each pilot in the meter_mapping file into the proper value of the relevant codelist.

Each meter listed in the meter_mapping file is associated with a meter identification string:

<meterID>_<type><abs/rel>_<unit>_<freq> (i.e. FER-003_GASA_MCU_1m)

Where the following relations with GreenButton DB schema apply:

<meterID> UsagePoint description FER-003

<type> ServiceCathegory/Commodity GAS

<abs/rel> AccumulationBehaviour A

<unit> Uom MCU

<freq> TimeAttribute 1m

The following sections describe the mapping between the codelist values defined in the meter_mapping file

and the corresponding relevant values of the GreenButton codelists.

ServiceCategoryKind

Meter_mapping ServiceCategoryKind (GreenButton) Value

ELE 0 electricity

GAS 1 gas

THE 5 heat

AccumulationBehaviourType

Meter_mapping AccumulationBehaviour (GreenButton) Value

A 3 Cumulative

R 4 DeltaData

CommodityType

Meter_mapping CommodityType (GreenButton) Value

ELE 1 Electricity secondary

GAS 7 NaturalGas

THE 12 HeatingFluid




CIP-ICT-PSP-2012-6 – 325161

Page 45 of 45




TimeAttributeType

Meter_mapping TimeAttributeType (GreenButton) Value

15 2 15-minute

1h 7 60-minute

1d 11 Daily

1w 24 Weekly

1m 13 Monthly

1y 0 Not Applicable

ir 0 Not Applicable

UomType

Meter_mapping UomType (GreenButton) Value

KWH 72 Real energy (Watt-hours)

MCU 42 m3 (Cubic Meter)

LTR 134 Liter

CurrencyCode

Follows codes defined in ISO 421716.

CurrencyCode (GreenButton) Value

0 Not Applicable

840 US Dollar

978 Euro

16 http://en.wikipedia.org/wiki/ISO_4217


Technology

S.2.f Specifications for Data Ingestion via Green Button