Part I Part II Part III Part IV - echtzeitsysteme.github.ioechtzeitsysteme.github.io/fosd2017/themes/yellow-swan/... · An Analysis of the Variability in Forty Preprocessor-Based

Why Should We Care about Similarity of SatisfiabilityProblems?

Part IPart IIPart IIIPart IV

Part I - Why Does Configurable Software NeedCompositionality?

Highly-Configurable Systems

Feature Models

A Matter of Size

Why to Solve Lots of Similar SAT Problems?

Part II - Compositional Analyses with Feature-ModelInterfaces

Compositionality Principle

Feature-Model Interfaces

Reduced Feature-Model Size

Reduced Effort in Type Checking Linux?

Part III - Solving Similar SAT/BDD/SMT Instances

Profiling the SAT4J Implementation

Effect of Our SAT4J Optimization

The Influence of Large Submodels

Cumulative Solver Times with Threshold

Solving Similar SAT/BDD/SMT Instances

Conclusion

Part IV - Backup Slides

Decomposition on Demand?

Make Implicit Constraints Explicit

Decomposition with Feature-Model Interfaces

Compositionality with Feature-Model Interfaces

Reasoning Strategies

Setup for Measurements

Time to Solve SAT Queries

Time vs. Number of Selected Submodels

Threats to Validity

Why Should We Care about Similarity ofSatisfiability Problems?

Thomas Thüm, Frederik Kanning, Stephan Mennicke, Ina Schaefer, . . .FOSD Meeting 2017 in Grasellenbach, March 16, 2017

Part I

Why Does Configurable SoftwareNeed Compositionality?

Highly-Configurable Systems

An Analysis of the Variability in Forty Preprocessor-BasedSoftware Product Lines

Jörg Liebig, Sven Apel,and Christian Lengauer

University of Passau{joliebig,apel,lengauer}@fim.uni-

passau.de

Christian Kästner and Michael SchulzeUniversity of Magdeburg

{ckaestne,mschulze}@ovgu.de

ABSTRACTOver 30 years ago, the preprocessor cpp was developed toextend the programming language C by lightweight metapro-gramming capabilities. Despite its error-proneness and lowabstraction level, the preprocessor is still widely used inpresent-day software projects to implement variable software.However, not much is known about how cpp is employedto implement variability. To address this issue, we haveanalyzed forty open-source software projects written in C.Specifically, we answer the following questions: How doesprogram size influence variability? How complex are exten-sions made via cpp’s variability mechanisms? At which levelof granularity are extensions applied? Which types of ex-tension occur? These questions revive earlier discussions onprogram comprehension and refactoring in the context of thepreprocessor. To provide answers, we introduce several met-rics measuring the variability, complexity, granularity, andtypes of extension applied by preprocessor directives. Basedon the collected data, we suggest alternative implementa-tion techniques. Our data set is a rich source for rethinkinglanguage design and tool support.

Categories and Subject DescriptorsD.2.3 [Software Engineering]: Coding Tools and Tech-niques; D.2.8 [Software Engineering]: Metrics; D.3.4 [Pro-gramming Languages]: Processors—Preprocessors

General TermsEmpirical Study

KeywordsSoftware Product Lines, C Preprocessor

1. INTRODUCTIONThe C preprocessor (cpp) is a popular tool for implement-

ing variable software. It has been developed to enhance C by

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ICSE ’10, May 2-8 2010, Cape Town, South AfricaCopyright 2010 ACM 978-1-60558-719-6/10/05 ...$10.00.

lightweight metaprogramming capabilities and is commonlyused to merge files, make arbitrary textual substitutions, anddefine conditional code fragments (a.k.a. conditional inclu-sion) [21]. As the cpp tool is line-based, it can be used withany text artifact including other programming languagessuch as Java or C#. In the past, it has been observed thatthe use of cpp causes various problems: (1) the occurrenceof syntactic and semantic errors during the generation ofsoftware products [23]; (2) code pollution due to scatteredand tangled #ifdefs (a.k.a. #ifdef hell) [32]; (3) a decrease inmaintainability and in ability to evolve [13].

The implementation of variable software is also a majorgoal of software product line engineering. A software productline (SPL) is a set of software-intensive systems sharing acommon, managed set of features that satisfy the specificneeds of a particular market segment or mission and thathave been developed from a common set of core assets in aprescribed way [8]. Software product line engineering aims atmanaging variability among the different software products(a.k.a. variants) of a product line and facilitating reuse in thatthe products of the product line share as many common coreassets as possible, such as source code artifacts [8, 11, 27].

It is widely assumed that the variability mechanism of thecpp tool is used quite frequently in the implementation ofSPLs [1, 12, 18, 33]. We believe this assumption to be trueand contribute to the discussions on the connection betweenpreprocessors and SPLs started in prior work with a substan-tial set of case studies. We answer the following questions:How does program size influence variability of SPLs? Howcomplex are the extensions applied by features via cpp’svariability mechanisms? At which level of granularity areextensions applied? Which types of extension occur? Weargue that insights into the implementation problems of fea-tures solved with cpp help to judge whether cpp usage causesproblems with regard to code pollution, error-proneness, andreduced maintainability and ability to evolve. An analysis ofcpp usage also allows developers to estimate the effort andbenefits of migrating to other, more well-founded implemen-tation techniques for SPL engineering, such as aspects [17,24] or various flavors of feature modules [5, 6].

To answer the questions above, we analyzed forty open-source software systems of different sizes (thousands to mil-lions lines of code) taken from different domains includingoperating systems, database systems, and compilers. Wepropose a set of metrics that allow us to infer and classifycpp usage patterns and to map the patterns to common SPLimplementation concepts. Our analysis reveals that cpp isused to a large extent to implement and control variability

1

Liebig et al. @ ICSE’10

Thomas Thüm Why Should We Care about Similarity of Satisfiability Problems? Slide 3

Feature Models

Base

BAR

F8F7FOO

F1

F6F5F4


A Matter of Size

Feature models with thousands of features are challenging


A Matter of Size

Feature models with thousands of features are challenging


Why to Solve Lots of SAT Problems?

#ifdef BAR

int x = 0;

#endif

// ...

#ifdef FOO

x++;

#endif

FOO ⇒ BAR ?

Idea: reduce size of feature model

Type checking

Parsing

Dataflow analysis

Model checking

Deductive verification

Refactoring

Feature-model analysis

Configuration process



#ifdef BAR

int x = 0;

#endif

// ...

#ifdef FOO

x++;

#endif

FOO ⇒ BAR ?


Type checking

Parsing

Dataflow analysis

Model checking


Refactoring





#ifdef BAR

int x = 0;

#endif

// ...

#ifdef FOO

x++;

#endif

FOO ⇒ BAR ?


Type checking

Parsing

Dataflow analysis

Model checking


Refactoring





#ifdef BAR

int x = 0;

#endif

// ...

#ifdef FOO

x++;

#endif

FM �FOO ⇒ BAR ?


Type checking

Parsing

Dataflow analysis

Model checking


Refactoring





#ifdef BAR

int x = 0;

#endif

// ...

#ifdef FOO

x++;

#endif

¬SAT (FM ∧ (FOO ⇒ BAR)) ?


Type checking

Parsing

Dataflow analysis

Model checking


Refactoring





#ifdef BAR

int x = 0;

#endif

// ...

#ifdef FOO

x++;

#endif



Type checking

Parsing

Dataflow analysis

Model checking


Refactoring





#ifdef BAR

int x = 0;

#endif

// ...

#ifdef FOO

x++;

#endif



Type checking

Parsing

Dataflow analysis

Model checking


Refactoring





#ifdef BAR

int x = 0;

#endif

// ...

#ifdef FOO

x++;

#endif



Type checking

Parsing

Dataflow analysis

Model checking


Refactoring





#ifdef BAR

int x = 0;

#endif

// ...

#ifdef FOO

x++;

#endif



Type checking

Parsing

Dataflow analysis

Model checking


Refactoring




Part II

Compositional Analyseswith Feature-Model Interfaces


(FOO ⇒ X)∧ Y ∧ Z ∧ (X ⇒ BAR) � FOO ⇒ BAR

(FOO ⇒ X)∧ (X ⇒ BAR) � FOO ⇒ BAR










Feature-Model Interfaces [ICSE’16]

For scalability: features eliminated from propositional formulas withexistential quantification


Feature-Model Interfaces [ICSE’16]

For scalability: features eliminated from propositional formulas withexistential quantification


Reduced Feature-Model Size [ICSE’16]

Given four monthly snapshots of an automotive feature model growingfrom 14k to 18k featuresRQ1: How small can interfaces be compared to submodels?

RQ2: How often are feature-model interfaces compatible?

Features

in V1

14,010

0%

20%

40%

60%

80%

100%

Features

in V2

17,742

Features

in V3

18,434

Features

in V4

18,616


Reduced Feature-Model Size [ICSE’16]

Given four monthly snapshots of an automotive feature model growingfrom 14k to 18k featuresRQ1: How small can interfaces be compared to submodels?RQ2: How often are feature-model interfaces compatible?

Features

in V1

14,010

0%

20%

40%

60%

80%

100%

V1↓

V2

19

Features

in V2

17,742

V2↓

V3

14

Features

in V3

18,434

V3↓

V4

13

Features

in V4

18,616



Family-based type checking with TypeChef

Linux kernel version 2.6.33.3 with 11,000 features

1,363 out of 7,760 files

173,845 queries

90ms per query

12s reasoning time per file

Reasoning: 60% of type checking phase






173,845 queries

90ms per query








173,845 queries

90ms per query




Literals and Distinct Features per Query

1 28 55 82 109

136

163

190

217

244

271

298

325

352

379

406

433

460

487

514

100

101

102

103

104

105

Literals


Literals and Distinct Features per Query

1 28 55 82 109

136

163

190

217

244

271

298

325

352

379

406

433

460

487

514

100

101

102

103

104

105

Literals

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28

Features


Apply Linux Queries to Automotive Feature Model

Problem: no decomposition for Linux feature model andno domain artifacts/queries for automotive

Idea: use queries from Linux evaluation as templates(¬A ∧ B)∨ A y (¬P ∧ Q)∨ P

Snapshot Features Constraints Clauses Submodels

1 14,010 666 237,706 442 17,742 914 342,935 453 18,434 1,300 347,557 464 18,616 1,369 350,287 46


(No) Reduced Effort for Product-Line Analysis?

1 2 3 4 5

0

1,000

2,000

3,000

4,000

Literals in query

Average

time(m

s)

fullreduced


Part III

Solving SimilarSAT/BDD/SMT Instances





public void remove(T elem) {

int j = 0;

for (; this.myarray[j] != elem; j++) {

if (j == size ())

throw new NoSuchElementException ();

}

System.arraycopy(this.myarray , j + 1,

this.myarray , j, size() - j - 1);

this.myarray[--this.nbelem] = null;

}


Effect of Our SAT4J Optimization

1 2 3 4 5

0

1,000

2,000

3,000

4,000

Literals in query

Average

time(m

s)

fullreduced

full (opt.)reduced (opt.)


Cumulative Solver Time with Optimization

Snapshot1

Snapshot2

Snapshot3

Snapshot4

0

20,000

40,000

60,000

80,000

2,462

3,255

3,466

3,468

52,373

79,666 79,997 79,132

Cumulative

solver

time(m

s) fullreduced



100 101 102 103 104 105

Clauses

Snapshot1

Snapshot2

Snapshot3

Snapshot4

0

5,000

10,000

15,000

20,000

11,100

15,698

14,818

15,866

11,453

11,961

11,486

12,378

Cumulative

solver

time(m

s) fullreduced



100 101 102 103 104 105

Clauses

Snapshot1

Snapshot2

Snapshot3

Snapshot4

0

5,000

10,000

15,000

20,000

11,100

15,698

14,818

15,866

11,453

11,961

11,486

12,378

Cumulative

solver

time(m

s) fullreduced


Time vs. Added Submodel Clauses

0 0.2 0.4 0.6 0.8 1 1.2

·104

0

5

10

15

20

Added clauses

Tim

e(m

s)

fullreduced

full (approx.)reduced (approx.)


Cumulative Solver Times with Threshold

Snapshot1

Snapshot2

Snapshot3

Snapshot4

0

5,000

10,000

15,000

20,000

11,100

15,698

14,818

15,866

11,453

11,961

11,486

12,378

10,083

11,750

11,341

12,105

Cumulative

solver

time(m

s) fullreducedadaptive


Solving Similar SAT/BDD/SMT Instances

SAT: similar instances not part of satisfiability contests⇒ not optimized for this purpose

BDD: no BDD for Linux, hard to join BDDs with different variableorderings

SMT: recent developments for lightweight removal of formulas


How Should Superheroes Eat Cake?


We Should Care about Similarity of SAT Problems

Configurable software requires to solve many similar SATinstances: consumes 60% of type checking (without parsing)

Feature-model interfaces can significantly reduce the similar part

Performance gain for reasoning 10–24%

Open question: What are good decompositions of feature models?

Open question: Are solvers ready to solve similar instances?


Experience Report: How NOT to Travel to FOSD’17

Forest + Darkness= Very Late Dinner


Decomposition on Demand? [SPLC’16]

Removing x % of 18k features of an automotive feature model:


Make Implicit Constraints Explicit [FOSD’16]

What are implicit constraints?

C ⇒ D ¬C ∨ ¬D




C ⇒ D

¬C ∨ ¬D




C ⇒ D

¬C ∨ ¬D




C ⇒ D ¬C ∨ ¬D



RQ1: How many implicit constraints exist in subtrees?Automotive feature model with 2,513 features and 2,833 constraints

Depth 1: 6 features and 12 implicit constraintsDepth 2: 25 features and 186 implicit constraints



RQ2: What is the structure of implicit constraints?

RQ3: What is the number of involved subtrees?Depth 1: max. 4/6 partial modelsDepth 2: max. 5/25 partial models



Base

BAR

F8F7FOO

F1

F6F5F4



Base

BAR

F8F7FOO

F1

F6F5F4

select subtrees



Base

BAR

F8F7FOO

F1

F6F5F4

select subtrees



Base

BAR

F8F7FOO

F1

F6F5F4

identify local features & local constraints



Base

BAR

F8F7FOO

F1

F6F5F4

identify local features & local constraints



Base

BAR

F8F7FOO

F1

F6F5F4

decompose model by removing local features



Base

BAR

F8F7FOO

F1

F6F5F4

decompose model by removing local features



Base

BAR

F8F7

F1

F4

Γ

F1

F6F5F4

∆1

BAR

F8F7FOO

∆2

Γ ∪ ∆2 �

FOO ⇒ BAR ?

X

Γ ∪ ∆1 �F6 ⇒ Base ?X

Γ ∪ ∆1 ∪ ∆2 2F1 ∧ F6 ∧ FOO ?X



Base

BAR

F8F7

F1

F4

Γ

F1

F6F5F4

∆1

BAR

F8F7FOO

∆2

Γ ∪ ∆2 �

FOO ⇒ BAR ?

X

Γ ∪ ∆1 �F6 ⇒ Base ?X

Γ ∪ ∆1 ∪ ∆2 2F1 ∧ F6 ∧ FOO ?X



Base

BAR

F8F7

F1

F4

Γ

F1

F6F5F4

∆1

BAR

F8F7FOO

∆2

Γ ∪ ∆2 �FOO ⇒ BAR ?

X

Γ ∪ ∆1 �F6 ⇒ Base ?X

Γ ∪ ∆1 ∪ ∆2 2F1 ∧ F6 ∧ FOO ?X



Base

BAR

F8F7

F1

F4

Γ

F1

F6F5F4

∆1

BAR

F8F7FOO

∆2

Γ ∪ ∆2 �FOO ⇒ BAR

?

X

Γ ∪ ∆1 �F6 ⇒ Base ?X

Γ ∪ ∆1 ∪ ∆2 2F1 ∧ F6 ∧ FOO ?X



Base

BAR

F8F7

F1

F4

Γ

F1

F6F5F4

∆1

BAR

F8F7FOO

∆2


?

X

Γ ∪ ∆1 �

F6 ⇒ Base ?

X

Γ ∪ ∆1 ∪ ∆2 2F1 ∧ F6 ∧ FOO ?X



Base

BAR

F8F7

F1

F4

Γ

F1

F6F5F4

∆1

BAR

F8F7FOO

∆2


?

X

Γ ∪ ∆1 �F6 ⇒ Base ?

X

Γ ∪ ∆1 ∪ ∆2 2F1 ∧ F6 ∧ FOO ?X



Base

BAR

F8F7

F1

F4

Γ

F1

F6F5F4

∆1

BAR

F8F7FOO

∆2


?

X

Γ ∪ ∆1 �F6 ⇒ Base

?

X

Γ ∪ ∆1 ∪ ∆2 2F1 ∧ F6 ∧ FOO ?X



Base

BAR

F8F7

F1

F4

Γ

F1

F6F5F4

∆1

BAR

F8F7FOO

∆2


?

X

Γ ∪ ∆1 �F6 ⇒ Base

?

X

Γ ∪ ∆1 ∪ ∆2 �

F1 ∧ F6 ∧ FOO ?

X



Base

BAR

F8F7

F1

F4

Γ

F1

F6F5F4

∆1

BAR

F8F7FOO

∆2


?

X

Γ ∪ ∆1 �F6 ⇒ Base

?

X

Γ ∪ ∆1 ∪ ∆2 �F1 ∧ F6 ∧ FOO ?

X



Base

BAR

F8F7

F1

F4

Γ

F1

F6F5F4

∆1

BAR

F8F7FOO

∆2


?

X

Γ ∪ ∆1 �F6 ⇒ Base

?

X

Γ ∪ ∆1 ∪ ∆2 2F1 ∧ F6 ∧ FOO

?

X



¬φ ∧

φ Adaptive:m(φ) < τ ?

SAT solver

¬φ ∧ ∧

Full

Reduced

CNF

CNF

CNF



¬φ ∧


SAT solver

¬φ ∧ ∧

Full

Reduced

CNF

CNF

CNF



¬φ ∧


SAT solver

¬φ ∧ ∧

Full

Reduced

CNF

CNF

CNF



¬φ ∧


SAT solver

¬φ ∧ ∧

Full

Reduced

CNF

CNF

CNF



¬φ ∧


SAT solver

¬φ ∧ ∧

Full

Reduced

CNF

CNF

CNF



¬φ ∧


SAT solver

¬φ ∧ ∧

Full

Reduced

CNF

CNF

CNF



¬φ ∧


SAT solver

¬φ ∧ ∧

Full

Reduced

CNF

CNF

CNF



¬φ ∧


SAT solver

¬φ ∧ ∧

Full

Reduced

CNF

CNF

CNF



¬φ ∧


SAT solver

¬φ ∧ ∧

Full

Reduced

CNF

CNF

CNF



¬φ ∧


SAT solver

¬φ ∧ ∧

Full

Reduced

CNF

CNF

CNF



Generate sets of 1,000 and 5,000 queries

Full: pure solver time

Reduced: selection + composition + solver times

JVM: garbage collection & just-in-time compilationScalaMeter framework

Multiple measurements across different JVM runsWarm-up runsOutlier detection






JVM: garbage collection & just-in-time compilation

ScalaMeter frameworkMultiple measurements across different JVM runsWarm-up runsOutlier detection






JVM: garbage collection & just-in-time compilationScalaMeter framework

Multiple measurements across different JVM runsWarm-up runsOutlier detection


Time to Solve SAT Queries

0226

452

678

904

1,130

1,356

1,582

1,808

2,034

2,260

2,486

2,712

2,938

3,164

3,390

3,616

3,842

4,068

4,294

100

101

102

103

104

105

Time (ms)


Time vs. Number of Selected Submodels

0 2 4 6 8 10 12

1

2

3

4

Selected models

Average

time(m

s)full

reduced


Threats to Validity

Type checking1/6 of available filesOther analyses, product lines?

Reasoning with interfacesSemi-random queriesOnly snapshots of a single modelSingle solver implementationIsolated measurements


Threats to Validity

Type checking1/6 of available filesOther analyses, product lines?

Reasoning with interfacesSemi-random queriesOnly snapshots of a single modelSingle solver implementationIsolated measurements


Documents

Part I Part II Part III Part IV - echtzeitsysteme.github.ioechtzeitsysteme.github.io/fosd2017/themes/yellow-swan/... · An Analysis of the Variability in Forty Preprocessor-Based