ESSnet SCFE DELIVERABLE D5-3 · This document is licensed under a Creative Commons License: Attribution-ShareAlike 4.0 International ESSnet SCFE DELIVERABLE D5-3 Assessing open source

This document is licensed under a Creative Commons License: Attribution-ShareAlike 4.0 International

ESSnet SCFEDELIVERABLE D5-3Assessing open source software solutions for statistical production and fortheir capability to become available in the ESS as a CSPA compliantstatistical service

Project acronym:

SCFE

Project title:

“Sharing common functionalities in the ESS”

Name(s), title(s) and organization or the auhor(s):

Joaquim Machado, Dr. ([email protected])José Carlos Martins, Eng. ([email protected])

Instituto Nacional de Estatítica

Tel: +351 218 426 100Fax: +351 218 454 083e-mail: [email protected]

Date: 21 Dec. 2017

https://creativecommons.org/licenses/by-sa/4.0/legalcode

Table of Contents

Introduction.............................................................................................................................1

Methodology...........................................................................................................................3

Discussion..............................................................................................................................8

Recommendations...............................................................................................................11

Design the service.........................................................................................................12

Be consistent and standard...........................................................................................13

Keep the service simple and task-oriented...................................................................15

Test and maintain the service........................................................................................15

Document the service....................................................................................................16

Register the service.......................................................................................................17

Conclusions..........................................................................................................................18

Annexes...............................................................................................................................19

Annex A: CBA – Statistical Products.............................................................................20

References...........................................................................................................................37

Assessing open source software solutions for statistical production and fortheir capability to become available in the ESS as a CSPA compliantstatistical service

Introduction

The majority (58,82%) of NSIs1 that participated in the D5-1 survey conduct

research/benchmark/comparative studies to include open source products when

evaluating solutions for statistical products. These studies are mainly economic and

financial studies. Although these studies are performed, no NSI declares to have an open

source strategy.

Although all NSIs are familiar with "CSPA Application Architecture", only one potentially

reusable/shareable statistical product is already aligned with CSPA Application

Architecture2: RMéS.

Only one reported statistical product does not support GSBPM sub-process (CZSO-

INPUT). All NSIs are familiar with this concept.

Historically, little relevance has been given to "development interaction" and "license type"

when choosing statistical products, since only one reported statistical product (PC-Axis

family) has open/community development interaction, and half of the statistical products

have proprietary license type. This finding can be possibly explained by the fact that NSIs

do not have a well defined open source policy. The same is true when considering support

since the majority of the reported statistical products only have "Best effort support" and

none has "Community based support".

A large number of NSIs are interested in being community contributors and active

supporters of the community.

1Bulgaria, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Italy, Lithuania,Malta, Portugal, Slovenia, Spain, Sweden, Switzerland.2 http://www1.unece.org/stat/platform/display/CSPA/CSPA+Application+Architecture

1


The above mentioned reality and all information collected on D5-1 and D5-2 enabled us to

provide hints and guidelines on how to implement, or improve, an open source web

service platform for statistical production.

2


Methodology

The assessment of open source software solutions for statistical production and their

capability to become available in the ESS as a CSPA compliant service is based on key

findings of D5-1 survey and the criteria defined in D5-2 for assessing/bechmarking such

solutions and projects for statistical production.

In alignment with the task force on VIP.SERV ESSnet, it was decided in D5-2 that the cost

benefit analysis of open source software solutions for statistical production and their

capability to become available in the ESS as a CSPA compliant service should be derived

from Attractiveness, Achievability and Affordability (AAA) method.

Choosing criteria/sub-criteria and metrics requires considerable thought and care to

support specific business goals as they play an important role in providing guidelines and

hints on software components needed to implement an open source platform.

The selected sub-criteria for Attractiveness were:

• GSBPM Sub-Process supported

• Number of users

• Current development activity

• Development interaction

• License type

• Support

The selected sub-criteria for Achievability were:

• Status of software

• Aligned with CSPA Application Architecture

• Has Web-service

3


The selected sub-criteria for Affordability was:

• Estimated cost of implementing non existing features

The assigned weight to the three criteria of CBA were:

Attractiveness: 35%

Achievability: 50%

Affordability: 15%

Achievability was considered the most important criteria due the fact that a statistical

product may be attractive and affordable, but if it fails to achieve/support desired/specific

business goals it is nearly useless.

Although affordability is an important factor, it was considered that in the context of

“Sharing Common Functionalities in the ESS”, attractiveness has a greater impact on

decision makers and software users.

Amongst the above mencioned criteria and subcriteria greater relative weight were

assigned to those sub-criteria considered meaningful according to Open Source Solutions

(OSS) and projects for statistical production in the European Statistical System (ESS)

scope, namely:

Attractiveness: GSBPM Sub-Process supported, Development interaction, License type.

Achievability: Aligned with CSPA Application Architecture

Affordability: Estimated cost of implementing non existing features

4


The resulting CBA is:

5

Cost Benefit Analysis

Multi-Criteria Analysis (MCA): Attractiveness/Achievability/Affordability (AAA)

Relative Weight (%)Attractiveness 35Achievalibility 50Affordability 15

Attractiveness

Sub-Criteria OptionsChoice/Score Lookup TableChoice %

(1) No 1 030,000

(2) Yes 2 100

Number of users

1 0

10,0002 253 504 755 1001 0

10,0002 253 504 100

Development interaction1 0

20,0002 100

License type

1 0

20,000(2) Other 2 25(3) GPL 3 50(4) EUPL 4 75(5) Permissive (Apache, BSD, MIT, other) 5 100

Support

1 0

10,0002 253 504 100

100,000

Achievability

Status of software1 0

30,0002 503 100

(1) No 1 040,000(2) Can be 2 50

(3) Yes 3 100

Has Web-service(1) No 1 0

30,000(2) Can have 2 50(3) Yes 3 100

100,000

Affordability

1 0

100,0002 253 504 755 100

100,000

Relative Weight(%)

GSBPM Sub-Process supported

(1) Less than 10(2) Between 10 and 99(3) Between 100 and 1000(4) More than 1000(5) Internet wide

Current development activity

(1) Dead/Legacy(2) Maintenance(3) Moderately Active(4) Very Active(1) Private(2) Open/Community(1) Proprietary software

(1) No support (As is)(2) Best effort(3) Commercial paid support(4) Community based support

(1) Experimental/Testing(2) Early adoption(3) Well established (Production)

Aligned with CSPA Application Architecture

Estimated cost of implementing non existing features

(1) Not sure(2) More than 100000€(3) Between 25000€ and 100000€(4) Between 5000€ and 25000€(5) Less than 5000€


Two important strands should be taken into account when planning Open Source Solutions

(OSS) and projects for statistical production in the European Statistical System (ESS) and

their implementation as an open source platform:

- Software metrics;

- Business metrics.

Although software metrics are important a special attention should be given to business

metrics. Choosing metrics requires considerable thought and care to support specific

business goals. This is critical: Measurements should be designed to answer business

questions [1].

Because everything in software development is unique, nothing we can measure has any

predictive value in isolation. Trends (not fluctuations) in objective metrics remain useful,

but only when they support a specific business goal. This is why modern software

development focuses on subjective metrics, attached to certain features and supporting

specific business goals, thus allowing us to use metrics as an effective tool for continuous

learning and improvement [1].

Some of the selected criteria can belong to both strands. One possible categorization is:

Business metrics (Alphabetic order):

• Aligned with CSPA Application Architecture

• Estimated cost of implementing non existing features

• GSBPM Sub-Process supported

• Has Web-service

6


Software metrics (Alphabetic order):

• Current development activity

• Development interaction

• License type

• Number of users

• Support

• Status of software

As a future evolution of the model, we can consider other metrics, like business-

methodology metrics, namely: “Implements a methodology which is a standard/can be a

de facto standard”.

7


Discussion

The supported “GSBPM Sub-Process” of the eight collected statistical products on D5-1 is

presented on the next image. These eight statistical products cover different stages of the

production process.

Note: The reported "GSBPM Sub-Process" are highlighted with a grey elipse.

Next, we identify the GSBPM Sub-process and the statistical products.

GSBPM Sub-Process Statistical product

Metadata Management(*) RMéS

3.1 Build collection instrument Coltrane

4.3 Run collection Coltrane, IDEV, eSTATISTIK.core

5.1 Integrate data CDA

8


GSBPM Sub-Process Statistical product

7.1 Update output systems GENESIS

7.2 Produce dissemination products GENESIS

7.3 Manage release of dissemination products PC-Axis family (e.g. PX-web)(*)Over-arching process.

On Table 1 we present the Attractiveness, Achievability, Affordability scores (0-100%) of all

collected Statistical products on the “D5-1 Survey”. All scores are weighted (Attractiviness:

35%, Achievability: 50%, Affordability: 15%)1.

Attractiveness Achievability Affordability AAACDA 14,875 25,000 11,250 51,125

Coltrane 25,375 32,500 7,500 65,375

CZO-INPUT 7,875 15,000 0,000 22,875

eSTATISTIK.core 15,750 30,000 0,000 45,750

GENESIS 18,375 30,000 0,000 48,375

IDEV 15,750 15,000 0,000 30,750

PC-Axis 23,625 40,000 0,000 63,625

RMéS 15,750 50,000 7,500 73,250Table 3 - Attractiveness, Achievability and Affordability scores of all collected Statistical

products on the “WP5 – D1 Survey”, based on the tuned CBA. All scores are weighted.

Half of the collected statistical software have a weighted AAA score greater than 50%

(CDA, Coltrane, PC-Axis and RMéS).

As noted above, all NSIs are familiar with "CSPA Application Architecture", but only one

potentially reusable/shareable statistical product is already aligned with CSPA Application

Architecture: RMéS. From amongst the seven Statistical products than are not aligned with

CSPA, or can be aligned with CSPA, three have web-service (GENESIS, PC-Axis family,

1 Detailed results are presented on Annex A.

9


eSTATISTIK.core), two can have (CZSO-INPUT, Coltrane) and two don’t have (CDA,

IDEV). This finding

The existence of a web service is one of the recommendations of the “CSPA Application

Architecture”, and a missing key feature of half of the statistical products. Therefore, it is

very important to see this software component in detail and to provide some guidelines on

how to implement, or improve, an open source web service platform for statistical

production.

10


Recommendations

Web services represent the current generation of Internet technologies. They allow

computer applications to exchange data directly over the Internet, essentially allowing

modular or distributed computing in a more flexible fashion than ever before. In order to

allow web services to function, however, many standards are required: for requesting and

supplying data; for expressing the enveloping data which is used to package exchanged

data; for describing web services to one another, to allow for easy integration into

applications that use other web services as data resources [2].

Many web-services standards already exist, however, and there is no need to re-invent

them for use specifically within the statistical community. Specifically, SOAP (which

originally stood for the “Simple Object Access Protocol”) and the Web Services Description

Language (WSDL) can be used by “statistical products”1 to complement the data and

metadata exchange formats they are standardizing. In the web services world, the REST

(“Representational State Transfer”) protocol is also often used, relying on a URL-based

syntax to invoke web services. Such REST-based services can be described in a standard

fashion using WADL (“Web Application Description Language”), in the same way that XML-

invoked web services based on SOAP can be described using WSDL [2].

Despite the promise of SOAP and WSDL, it became evident from early implementations

by vendors that these were not, in fact, interoperable. It was for this reason that the Web

Services - Interoperability (WS-I) initiative was started2. This consists of a group of vendors

who have all implemented the same web-services standards the same way, and have

verified this fact by doing interoperability tests. They publish profiles describing how to use

web services standards interoperably [2].

1 Original text: “SDMX”.2 Note: WS-I is now part of OASIS (http://www.oasis-ws-i.org/).

11


Conventional applications and services traditionally expose their functionality through

application programming interfaces (APIs). Web services are no different – they provide a

public version of the function calls which can be accessed over the web using web-

services protocols (SOAP or REST). In order to make a set of web services interoperate, it

is necessary to have a standard abstraction, or model, on which these public functions are

based [2].

Taking in account the goal of developing an open source web service platform statistical

community and avoid common pitfalls, namely the lack of interoperability, a well planed

web service project should follow specific guidelines and phases, namely [3]:

• Design the service;

• Be consistent and standard;

• Keep the service simple and task-oriented;

• Test and maintain the service;

• Document the service;

• Register the service.

Design the service

Web services are nearly always implemented as an afterthought. Service providers usually

already have some local code that they want to expose as a public service. So they

generate an interface to it automatically in the "hack and publish" approach. The interface

ends up being cumbersome and hard to use and understand, tied to the underlying local

implementation and configuration, exposing internal ids, class names and formats.

Design services with compatibility and interoperability in mind, namely: internationalization

(translation), portability to other technical infrastructures, etc.

12


Reliability and stability of services must be considered. Users value services that are

reliable and stable, but services decay over time if they are not maintained properly.

Reliability and stability increase users' confidence and rating of services.

It is extremely useful for service consumers or service registries to be able to test whether

a service is available and whether it is running correctly. It is good practice for service

providers to consider how their service can be tested by external bodies. For example, a

"ping" operation to check that a WSDL service is "alive".

Services should allow to be used by consumers who know minimal information about the

service. It is easy for the service provider to assume that service consumers will know

almost as much about the service as they do. This assumption leads to lack of

documentation, strange features (parameters that change meaning or that should not be

used together), changes in the semantics of output and input data. The service provider

should assume that the service consumer's knowledge of the service is minimal. The

service consumer merely wants to achieve the task exposed via the service.

Services should expose consumer-oriented tasks i.e. what a service consumer wants to

do. Services should not expose provider-implemented operations i.e. what the service

provider's code does to perform a task.

Be consistent and standard

There are a number of standards for services in particular domains, such as the TAPIR

protocol, Web Map Service and the IVOA services. In the domain of statistical products,

some efforts are being made, namely in SDMX. [2]

Specify the format. If a service is returning XML then the service provider should specify

the xsd to which the data conforms. Similarly, if the service takes XML then the xsd of the

13


data should be documented. If the service is a WSDL service, then the data format should

be specified via inclusion and referencing of the xsd within the WSDL. It is not correct to

just specify any.

Return data in the format requested by the client

(http://www.xml.com/pub/a/2004/08/11/rest.html). A resource may have more than one

representation. It should never return data in a default format that does not match that

specifically requested. Incorrectly returning data in a default format prevents the client from

being notified of the format problem and also causes the client to process data in the

wrong format. REST GET services are very similar to fetching pages for a web browser.

Some service providers simply deliver an HTML document corresponding to what they

would return when the corresponding web page is fetched. Unless the service is

requesting a HTML document, the service provider should not return an HTML document.

The service should return the data that underlies the HTML document.

A service must be consistent with what is returned. Some REST services return JSON for

certain resources, a choice of JSON or XML for other resources, and XML for other

resources. This is very confusing for service consumers. Services should be consistent in

the formats in which they can return data. When a service can return a resource in

alternate formats, the data returned must (as far as the formats allow) contain the same

information. For example, if the JSON that is returned contains the name, address and

telephone number of a person, so should the XML.

Return the correct error codes. Service providers should read the HTTP Status Code

definitions and familiarize themselves with them.

(http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html).

Services should only return error codes as specified in the HTTP specification. Creating

error codes that are specific to a service breaks the HTTP specification and prevents

clients from using the service.

14


Services themselves should never intentionally return any error code in the 500 family -

these are reserved for use by the harness within which the service is running (eg Apache).

This way you know that if you see a 500 error code then it is not your service that is

broken but something further down the stack.

Service providers should not only document what errors are returned, but also check (as

far as feasible) what happens when unexpected requests are received.

Comply with Web Services Interoperability Standard (WS-I). "WS-I profiles define how

existing WS (Web Services) specifications should be used in order to achieve maximum

interoperability. The profiles effectively clarify the way existing standards should be used

because the final documents were not clear in places or the flexibility they allowed was

leading to interoperability nightmares. If the rules of WS-I profiles are followed, it is

expected (but not guaranteed) that the resulting deployments would interoperate (at least

as far as the underlying infrastructure is concerned)."

Keep the service simple and task-oriented

Never expose implementation details: The internal details of the service implementation

should be hidden, as far as possible. For example, just returning a serialization of a Java

object is not a sensible or usable response to a service call.

A service should do one thing, not many.

Test and maintain the service

Predicting performance bottlenecks for service can be difficult. There are however tools

such as Apache JMeter (http://jmeter.apache.org/) which can be used to assess the

15


behaviour of services in a structured way. Detected performance limitations should then be

included in the service documentation.

Many service implementations lack a clear use case and are just exposed in addition to a

human-readable portals for an unknown future application. To ensure that service provide

the expected functionality and performance developers should try to use them as much as

possible in their own systems such as web-portals using these services rather than local

APIs.

Document the service

Many services are poorly described. Even when the services are registered, only minimal

effort is put into documenting them. An undocumented service is an unusable service. The

documentation should include, at least,

• Description of the task that the service performs - what it does from a service

consumer's point of view

• What can be used as input, with example values and description of what happens if

the value is not specified

• What will be returned as output, including example values

• Possible error messages, including what they mean from the point of view of the

input data and the intended task. The error message should not be described by relation to

what has gone wrong in the provider's code

It is very useful if the documentation includes:

• Example programs, scripts and/or workflows (including example input data and

results) that use the service

• Other services that work well with the service

16


Register the service

Unless the NSI intend to keep the service private, it is very good practice to register it. In

the case of the statistical products and the scope of share and reuse of services, the CSPA

catalogue1.

1 https://webgate.ec.europa.eu/fpfis/mwikis/cspacatalogue/index.php/CSPA_catalogue

17


Conclusions

Choosing criteria/sub-criteria and metrics requires considerable thought and care to

support specific business goals as they play an important role in providing guidelines and

hints on software components needed to implement an open source platform.

A large number of NSIs are interested in being community contributors and active

supporters of the community.

The existence of a web service is one of the recommendations of the “CSPA Application

Architecture”, and a missing key feature of half of the statistical products reported in D5-1.

Therefore, it is very important to take action in order to implement, or improve, an open

source web service platform for statistical production. One potential limitation in the

implementation of the required web services for statistical production, components is the

fact that NSIs mentioned that they are not sure about the “estimated cost of implementing

non existing features” for the majority of Statistical products.

Taking in account the goal of developing an open source web service platform statistical

community and avoid common pitfalls, namely the lack of interoperability, a well planed

web service project should follow specific guidelines and phases. Many web-services

standards already exist, however, and there is no need to re-invent them for use

specifically within the statistical community.

18


Annexes

19


Annex A: CBA – Statistical Products

20


Statistical Product: CDA

21



Attractiveness

Sub-Criteria Options Choice

(1) No2 100 30,000

(2) Yes

Number of users 2 25 2,500

3 50 5,000

Development interaction 1 0 0,000

License type 2 25 5,000(2) Other(3) GPL(4) EUPL(5) Permissive (Apache, BSD, MIT, other)

Support 1 0 0,000

ScoreUnweighted (0..100) 42,500Weighted (“AAA Score”) 14,875

Achievability

Status of software 3 100 30,000

(1) No2 50 20,000(2) Can be

(3) Yes

Has Web-service(1) No

1 0 0,000(2) Can have(3) Yes


Affordability4 75 75,000


Attractiviness/Achievalibility/Affordability Score (0..100) 51,125

Choice Score(0..100)

Choice Score(Relative)











Statistical Product: CDA

22

Attractiveness

Achievability Affordability

0,000

50,000

100,000

Cost Benefit Analysis - Unweighted

Attractiveness


0,000

25,000

50,000

Cost Benefit Analysis - Weighted


Statistical Product: Coltrane

23


Multi-Criteria Analysis (MCA): Attractiveness/Achievability/Affordability (AAA

Attractiveness


(1) No2 100 30,000

(2) Yes


4 100 10,000



Support 2 25 2,500


Achievability


(1) No2 50 20,000(2) Can be

(3) Yes


2 50 15,000(2) Can have(3) Yes

















Statistical Product: Coltrane

24

Attractiveness


0,000

50,000

100,000


Attractiveness


0,000

25,000

50,000



Statistical Product: CZSO-INPUT

25



Attractiveness


(1) No1 0 0,000

(2) Yes


3 50 5,000



Support 3 50 5,000


Achievability


(1) No1 0 0,000(2) Can be

(3) Yes


2 50 15,000(2) Can have(3) Yes

















Statistical Product: CZSO-INPUT

26

Attractiveness


0,000

50,000

100,000


Attractiveness


0,000

25,000

50,000



Statistical Product: eSTATISTIK.core

27



Attractiveness


(1) No2 100 30,000

(2) Yes


2 25 2,500



Support 2 25 2,500


Achievability


(1) No1 0 0,000(2) Can be

(3) Yes


3 100 30,000(2) Can have(3) Yes

















Statistical Product: eSTATISTIK.core

28

Attractiveness


0,000

50,000

100,000


Attractiveness


0,000

25,000

50,000



Statistical Product: GENESIS

29



Attractiveness


(1) No2 100 30,000

(2) Yes


4 100 10,000



Support 2 25 2,500


Achievability


(1) No1 0 0,000(2) Can be

(3) Yes


3 100 30,000(2) Can have(3) Yes

















Statistical Product: GENESIS

30

Attractiveness


0,000

50,000

100,000


Attractiveness


0,000

25,000

50,000



Statistical Product: IDEV

31



Attractiveness


(1) No2 100 30,000

(2) Yes


2 25 2,500



Support 2 25 2,500


Achievability


(1) No1 0 0,000(2) Can be

(3) Yes


1 0 0,000(2) Can have(3) Yes

















Statistical Product: IDEV

32

Attractiveness


0,000

50,000

100,000


Attractiveness


0,000

25,000

50,000



Statistical Product: PC-Axis

33



Attractiveness


(1) No2 100 30,000

(2) Yes


3 50 5,000



Support 2 25 2,500


Achievability


(1) No2 50 20,000(2) Can be

(3) Yes


3 100 30,000(2) Can have(3) Yes

















Statistical Product: PC-Axis

34

Attractiveness


0,000

50,000

100,000


Attractiveness


0,000

25,000

50,000



Statistical Product: RMéS

35



Attractiveness


(1) No2 100 30,000

(2) Yes


4 100 10,000



Support 2 25 2,500


Achievability


(1) No3 100 40,000(2) Can be

(3) Yes


3 100 30,000(2) Can have(3) Yes

















Statistical Product: RMéS

36

Attractiveness


0,000

50,000

100,000


Attractiveness


0,000

25,000

50,000



References

[1] Steven A. L. (2016, June 1). "Why metrics don't matter in software development

(unless you pair them with business goals)". [Online]. Available at:

https://techbeacon.com/why-metrics-dont-matter-software-development-unless-you-pair-

them-business-goals

[2] SDMX (2013, April). “SDMX Standards: Section 7. Guidelines for the use of web

services”. [Online]. Available at: http://sdmx.org/wp-content/uploads/SDMX_2-1-1-

SECTION_07_WebServicesGuidelines_2013-04.pdf

[3] Alan Williams and Robert Haines (2013, November, 25). “Web services guidelines”.

[Online]. Available at: http://dev.mygrid.org.uk/wiki/display/scrap/Web+services+guidelines

37

http://dev.mygrid.org.uk/wiki/display/scrap/Web+services+guidelines

http://sdmx.org/wp-content/uploads/SDMX_2-1-1-SECTION_07_WebServicesGuidelines_2013-04.pdf

http://sdmx.org/wp-content/uploads/SDMX_2-1-1-SECTION_07_WebServicesGuidelines_2013-04.pdf

https://techbeacon.com/why-metrics-dont-matter-software-development-unless-you-pair-them-business-goals

https://techbeacon.com/why-metrics-dont-matter-software-development-unless-you-pair-them-business-goals

Documents

ESSnet SCFE DELIVERABLE D5-3 · This document is licensed under a Creative Commons License: Attribution-ShareAlike 4.0 International ESSnet SCFE DELIVERABLE D5-3 Assessing open source