Managing the evolution of F/OSS with Model Driven Techniques

Università degli Studi dell’AquilaUniversità degli Studi dell’Aquila

Dipartimento di InformaticaUniversità degli Studi dell’Aquila (Italy)

[email protected]

Alfonso Pierantonio, Davide Di Ruscio

2

→ More than 29,000 interdependent packages

GTTSE 2011 Alfonso Pierantonio – Managing the evolution of F/OSS with Model-Driven Techniques

→ Linux distributions are among the most complexsoftware ecosystems

→ Keynote at SLE11 by Krzysztofca. 10,000 features only in the kernel

3


explicit dependencies among linux packages

4

Free and Open Source Software

→ From tarballs to distributions

→ Package technology

Problem: upgrade failures

Abstracting the System

→ Modeling languages for describing the System

→ Maintainer Script Analysis

→ Injectors

Analyzing the System

→ Upgrade simulator

→ Fault detector

Conclusions


5

TodayAlfonso Pierantonio

→ A gentle introduction

→ Models of Complex Systems

July 6, 2011 – 14:45Davide Di Ruscio

→ Concrete Artifacts

→ Demo


6


FOSS distributions are among the most complex software systems known, being made of thousands components evolving rapidly without centralized coordination

Large numbers (components/developers)

→ SourceForge.net contains more than 123,736 projects, 1,342,153 “users”

→ Debian GNU/Linux is distribuited with more than29,000 pre-compiled packages


7


Numbers like that have the potential to radically change the way we produce and study software, in particular the complex systems of the future

complexity dependencies

8

In the beginning it was the tarball

Before the advent of distributions, the peculiar way to install free software on client machine was:

userinstallations

serverside

clientside

proj 1

proj 2

proj n

bazaar

GTTSE 2011Alfonso Pierantonio – Managing the evolution of F/OSS with Model-Driven Techniques

9To answer these problems, GNU/Linux distributions have born as intermediaries between FOSS projects and their users

userinstallations

serverside

clientside

proj 1

proj 2

proj n

FOSSbazaar

package repository

meta-installer

distributioneditors

packagemanagement

10


FOSS distributions are among the most complex software systems known, being made of thousands components evolving rapidly without centralized coordination

FOSS components are provided in packaged form by distribution editors

Packages define the granularity at which components are managed using package manager applications


11

GNU/Linux distributions

Central notion in distributions (to abstract over the complex underlying infrastructure):

→ package, together with package management software

package repository

distributioneditors

packagemanagement

GTTSE 2011Alfonso Pierantonio – Managing the evolution of F/OSS with Model-Driven Techniques

12

GNU/Linux distributions

Central notion in distributions (to abstract over the complex underlying infrastructure):

→ package, together with package management software

package repository

distributioneditors

packagemanagement

And yet, doing things right can be extremely difficult!


13

Installation intricacies

In state of the art distributions:

→ knowing if the installation process terminates without errors is undecidable (reason: configuration scripts) … today

→ there is no guarantee that distribution repositories contains only installable packages

→ a large number of errors faced by users are dependency resolution errors

─ more than 29,000 packages (and 10,000 sources) in Debian

─ more than 200.000 inter-package relationships


14

PROBLEM: UPGRADE FAILURES

15

→ each phase can fail – it actually happens quite often!

→ efforts should be made to identify errors as early as possible, if not predict

16

Upgrade failures

Essentially they can fail for the following classes of problems

→ Consistency breach due to administrator operations at a finer level of granularity than that of packages

─ eg. needed resources are manipulated or removed from the command-line

→ Implicit dependencies are not considered in the installation/removal script in the upgraded packages

→ Script erroneous behavior


17

> apt-get install libapache-mod-ssl

Configuration 1

Configuration 2

Installing packagelibapache-mod-ssl

+

18

19

20

21

once mod_ssl is installed, it is enabled in

apache


22

> apt-get remove libapache-mod-ssl

Configuration n+1

Configuration n

removing packagelibapache-mod-ssl

-

23


24

It disables mod_ssl in apache.

What happens if the maintainer does not write this statement ?


25

An “inconsistent” configuration is reached and it is detected only at run-time:

the package dependency metadata are not enough !


26

Problem: Upgrade failures

Current tools are able to predict a limited set of upgrade failures before deployment

When trying to predict upgrade failures, existing tools only consider static package metadata and the behaviour of the maintainer scripts is completely ignored

→ This leaves a wide range of failures unpredicted


27

Upgrade failures classification

Static deploy-time occur when a static requirement is violated during the upgrade.

→ the low-level package manager fails at deploy-time, aborting the upgrade process

Dynamic deploy-time occur when a maintainer script fails

→ such failures can originate from errors ranging from syntax to failures in external tools

→ they are not addressed by state of the art package managers

Undetected failures remain undetected through upgrade deployment

→ the upgrade has been completed successfully, but the obtained system configuration contains undetected incoherences


28

IDEA : ABSTRACTING AND ANALYZING THE SYSTEM

29

Idea

Extracting relevant information from a linux box («snapshot») at a given moment of time

Analysis and upgradesimulation on the «snapshot»

injectors

Linux box

«snapshot»

30

Ingredients 1

Modeling languages for describing the several aspects of a linux distribution

→ Packages

─ including maintainer scripts

→ System Configuration

─ Installed packages

─ Configuration files

─ MIME-type handlers

─ Alternatives

─ etc

Injectors for harvesting the system and building the models

→ collection of injectors


31

Ingredients 2

Modeling language for the maintainer scripts

→ scripts are written in POSIX languages whose semantics is far from being simple, although

→ maintainer scripts does not harness the full expressivity ofsuch languages (template-based “macro-language”)

Maintainer scripts asmodels

→ which semantics ?


32

Ingredients 3

Transformational semantics for simulating the behavior of the maintainer scripts on the system «snapshot»

→ M2M transformations obtained by «compiling» the maintainer scripts into ATL

Fault detector, a general mechanism for performingqueries over the «snapshot» for digging the model and search for inconsistencies


33

Configuration 1

«snapshot» 1

Configuration 2

?

«snapshot» 2

system injectionpackage injection

M2Mtransformation M2M transformation

34

A model-based approach is introduced to support the package upgrades and enhance the failure detection possibilities:

→ the simulator is used to predict the effect of maintainer script executions (deploy-time failures)

→ the fault detector is used to deal with undetected failures

35

A model-based approach is introduced to support the package upgrades and enhance the failure detection possibilities:

→ the simulator is used to predict the effect of maintainer script executions (deploy-time failures)

→ the fault detector is used to deal with undetected failures

EVOSS

36

MAINTAINER SCRIPT ANALYSIS

37

Rigourous maintainer script analysis (1/2)

Script analysis for Debian and RPM based distributions

To support the analysis and represent the results a database of scripts has been developed

→ Already known templates have been identified─ By taking into account annotations in the scripts (when available,

eg. in case of Debian scripts)

─ Exact match of existing templates in the scripts

→ New templates have been identified─ By using an “exact match” between scripts

─ By analyzing block statements (if, while, case, for, etc.)

─ by using a similarity based approach


38

Rigourous maintainer script analysis 2/2

http://evoss.di.univaq.it

39

INJECTORS

40

Used to represent in terms of models packages and system configurations


41

Configuration and package injectors

By means of the model injection, given a linux system a corresponding model is obtained

Injectors inspect software artifacts and inject relevant information into corresponding models

The process is driven by the metamodels


42

Configuration injector

cfginject [-d <distribName>] [-full]

→ It creates a system configuration model starting from a real system. It ispossible :

─ to specify the distribution name─ to force the complete regeneration of the model. By default incremental

changes of the model are performed in order to keep the system and the corresponding model synchronized

Mancoosi Model Management

Mancoosi

Model Injection Infrastructure

Debian

Model Injector

Mandriva

Model Injector

Caixa Magica

Model Injector

Ubuntu

Model Injector…

DEMO

43

Package injector

pkginject [--installed] [--package <package file name>][--cachedir]

→ It creates package models. By default, a model for each installedpackage is created (--installed)

→ It is possible to inject only one package by specifying the package file name (--package)

→ It is also possible to inject all the packages which are in the cachedir (--cachedir)

DEMO

44

UPGRADE SIMULATOR

45

Upgrade Simulator

Used to simulate system configuration upgrades


46

ConfigurationModel

Upgrade Simulator

Upgrade plan:

{(p1,u1),(p2,u2)…,(pn,un)}New

ConfigurationModel

Package Injection

Package Model1

Package Model2

Package Modeln

…

{p1, p2, …, pn}

InputOutput

ErrorModel


47

For 1≤i ≤ n : retrieve the state of pi

For each pi simulate upgrade scenario

Upgrade Simulator

ConfigurationModel

pi stateui

Package Model1

Package Model2

Package Modeln

…

Upgrade plan:

{(p1,u1),(p2,u2)…,(pn,un)

}

New Configuration

Model

ErrorModel

valid simulation

error found


48

Not installedinstall

preinst install

postinst configure postrm abort-install

New Configuration

Model

ErrorModel

Upgrade scenario

OK

Installed

Half-configured

Not installed

Half installed

“Reinst required”

FAILED

FAILED

OK

FAILED

OK

“Files are unpacked”

ConfigurationModel Package Modeli


49

PackageModel

ConfigurationModel

Script Simulator

ScriptScriptScript

St1

St2

…Stn

Modeltransformationorchestration

execution

Model transformationorchestration

generation

Retrieval of Modeltransformations

Repository ofModel

transformations

New Configuration

Model

ErrorModel


50

Script

ConfigurationModel

Tst2

New Configuration

Model

CM1

CM2

…

CMn-1

st1

st2

stn

…

Tst1

Tstn

ErrorModel

Orchestrationof model

transformations


51


52


53

<?xml version="1.0" encoding="UTF-8"?><selectionStates xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><selectionState type="install">

<param name="package" value="swi-prolog"/><param name="version" value="5.6.58"/><param name="architecture" value="all"/>

</selectionState><selectionState type="remove">


</selectionState><selectionState type="install">


</selectionState><selectionState type="install">


</selectionState></selectionStates>

Sample upgrade plan

swi-prolog5.6.58

[Not-Installed]

swi-prolog5.6.58

[Installed]

swi-prolog5.6.58

[ConfigFiles]

swi-prolog5.6.58

[Installed]

swi-prolog5.7.59

[Installed]

install

remove

install

upgrade

DEMO

54

FAULT DETECTOR

55

Used to check system configurations for faults that might give place to failures in the real system


56

Fault detectorDiscovering implicit dependencies among packages: we are able to discover dependencies that are not declared into the package's meta-information

Discovering missing configuration files: according to the system configuration model, some configuration files are required but they are not available in the system

Discovering Mime-type dangling handlers: according to the available information, the considered system should be able to manage a mime type, but the corresponding handler is missing in the system

Discovering missing services: the init.d file contains services that should start at the system start-up; by querying the configuration model, it is possible to detect missing services

…


57

Fault detector architecture

-Libraries (.jar files) and OCL queries-Catalogue of faults-Catalogue of solutions

Web Portal

FD-Server

MANCOOSI Server

User machine

FD-Client

User

DEMO

58

Conclusions 1

The prediction of the package upgrade impact has been enhanced: a larger number of cases which lead to faulty behaviors can be detected

The core ingredients have been defined

→ Modeling languages for the different aspects

→ Injectors: system configuration and packages

→ Transformational Script Semantics─ based on ATL

→ Fault detection: query-based global knowledge base─ OCL and JAR queries to detect faults

─ Contribution from the community


59

Conclusions 2

The main difficulty is in building the model injectors

→ labor intensive, deep platform knowledge required, ad-hoc techniques

→ Interesting (academic) tools available─ GRA2MOL – grammarware / modelware bridging

─ WIRES* – Model transformation orchestration

Monolithic real scale metamodels

→ Maintenance of the developed metamodels has been an issue: interdependencies, ripple effects

→ Difficult stabilization of the support tools

Model comparison for validating the simulation

→ EMF Compare


60

Conclusions 3

This work has been done within the

→ EU FP7 ICT STREP MANCOOSIhttp://www.mancoosi.org

Industrial partners

→ Mandriva

→ Caixa Magica

→ IBM Ilog


61

62

Università degli Studi dell’AquilaUniversità degli Studi dell’Aquila

Questions?

63

64

PackageModel

PackageModel

Packages

Upgrade

Planner

Upgrade plan:

{(p1,u1),…,(pn,un)

} CM1

(p,u)

PackageModel

Upgrade

Upgraded Real

System

Real

System

CM2

CM3

Upgrade

Simulator

Configurations

compare

System

InjectorPackage

Injector

System

Injector

DeltaModel

1.2. 3.

4.

5.

6.

7.

65

Education

Managing the evolution of F/OSS with Model Driven Techniques