Research Questions for Validation and Verification in the Context of Model-Based

Engineering

Catherine DuboisMichalis FamelisMartin GogollaLeonel NobregaIleana OberMartina SeidlMarkus Voelter

ENSIIE, Evry, FranceUniversity of Toronto, University of Bremen, University of Madeira, Funchal, University of Toulouse, Johannes Kepler University Linz, AustriaVoelter Ingenieurburo, Heidenheim, Germany

October 1st , 2013MoDeVVa 2013, Miami, USA

2

Introduction

Abstraction techniques are one of the promising paths for the future advances in the field of verification.

– Clarke, Emerson, Sifakis: Turing Lecture 2008

V&V crucial for MBE– Uncover hidden properties and errors– Verify transformations– Ensure quality, etc.

3

About

• This paper is the result of the working group on V & V at Dagstuhl-Seminar 13182 held in May 2013.

• Dagstuhl: scientific retreatin western Germany

• Seminar topic: Meta-Modeling Model-Based Engineering Tools

• Three break-outs: Informal Modeling, Compositionality, Modeling and V&V

4

Our goal:Identify main areas in the synergy between MBE and V&V where we need to focus research. What is the status, what are the research questions?

• Two-day collaborative brainstorming workshop.

• Culminated in a plenary presentation

…and this report

5

Disclaimer

• No claim of completeness or exhaustiveness

• Represents the informed opinions of the authors

• May have missed existing answers to some questions

• Is this the right level of granularity / level of detail?

6

Thematic Categories

1. Gap between Models and V&V Formalisms2. Need to Refine Existing Methodologies3. Design-time vs. Runtime4. Properties5. Model Transformations6. Informal vs. Formal vs. Incomplete Modeling7. Comparison and Benchmarking8. Domain-Specific Languages

7

Models and V&V Formalisms (context)

Model Property

Model' Property'

Designer Level (can be Domain-Specific)

Verification formalism level

Verification engine

Verification

feed-back

Transformationsand traceability

mechanisms

But behavioral semantics often leads to non-bijective correspondences between design time and runtime artifacts.

V&V tool configuration

Models and V&V Formalisms (questions)

• How to express properties at the level of models in a way understandable to clients?

• How to formulate models and properties in a single language transparent to clients?

• How to report the V&V results and diagnostics in an appropriate form to clients?

• How to bridge the gap between formally expressed and verified properties on one side and client attention on the other side?

• Can modeling language extensions help in making explicit the “needs” of V&V machines?

9

Refining Existing Methodologies (context)

• Integrating V&V in development can support early V&V similar to how debugging is offered by IDEs.

• Generic methodologies identify points where V&V can be used.

• A Model-based development methodology would need to allow variations based on application domain, nature of project, etc.

• Goal: better identify which V&V activities are meaningful at the various phases of design; take full advantage of V&V engines.

10

Refining Existing Methodologies (questions)

• How do we integrate V&V in the overall development and modeling process? – On the technical level of tool exchange? – On the methodological level of using the right technique at

the right time for the right task?

• When are techniques like animation, execution, symbolic evaluation, testing, simulation, proving or test case generation used most efficiently during development?– For which model and model transformation properties can

they be employed?

11

Design-time vs. Runtime (context)

• Models are specified at design time

• During execution these models are instantiated

• The dynamic nature of structure during execution makes it difficult to understand and represent runtime information

• Existing modeling environments offer limited support for precisely specifying instantiation and snapshots

12

Design-time vs. Runtime (questions)

During the V&V phase, how do we obtain an initial model instantiation?

How do we obtain large and meaningful instantiations?

How do we connect design time and runtime artifacts?

How do we deal with the issue of scalability in the context of V&V ?

How do we handle time and space concerns w.r.t. design time and runtime artifacts?

How do we automatically or semi-automatically manage the V&V machine configuration?

13

Properties (context)• Model and model transformation properties relevant in V&V:

– consistency, reachability, dependence, minimality, conformance, safety, liveness, deadlock freeness, termination, confluence, correctness

• Confusion caused by: – Different kinds according to the nature of the model (static/ dynamic),

its level of abstraction, etc.– Many tools and techniques (potentially complementary)

Main challenge: What kind of property to verify on which model at what stage with what kind of technique?

14

Properties (questions)• What are the benefits and trade-offs between expressing properties

on more abstract modeling levels in contrast to expressing them on more concrete levels?

• How do we find the right techniques for uncovering static and dynamic model properties?

• Which techniques are appropriate for uncovering static modeling language inherent properties, which for static model-specific properties?

• Which techniques are appropriate for uncovering dynamic generic properties, which for dynamic model-specific properties?

15

Model Transformations (context)

• Core component of MBE• Many applications:

– Maintaining inter-model consistency– Semantics definition of (domain-specific) modelling languages– …

• Challenge: Verification of model transformations– Proving correctness, termination, confluence– What are the differences to “normal” code?– Is the higher abstraction level beneficial to V&V?

16

Model Transformations (questions)

• What verification techniques are meaningful for verifying model transformations?

• How do we analyse properties like confluence and termination?

• How do we analyse correctness of model transformations w.r.t. a transformation contract?

• How do we infer a transformation contract from a model transformation?

17

Informal vs. Formal vs. Incomplete Modeling (context)

• During V&V: switch on or off particular model elements (in class diagrams, e.g., multiplicities); configure constraints by negating, deactivating or activating them (in class diagrams, e.g., class invariants)

• Different types of granularity (a) all model elements may be relaxed (b) only a manual model element selection can be considered for relaxation (c) a semi-automatic element selection for relaxation may be offered

• Ultimate vision: sliders on the user interface to gradually go from a strict, formal model through various intermediate levels to a totally relaxed and informal model; fewer formal model parts activated means more informality in the model

• Minimal formal frame for test case construction must be preserved: e.g., for class diagrams of central classes and associations and for state charts of central states and transitions

• For a completely informal model no formal scenario (no test cases) can be formulated

18

Informal vs. Formal vs. Incomplete Modeling (questions)

• How do we leverage informal assumptions found in sketches for exploratory V&V?

• Are informal sketches close enough to V&V at all?

• What are appropriate relaxation mechanisms for different degrees of formality?

• How do we handle incomplete or partial models w.r.t. V&V?

• How do we deactivate and activate model units?

• How do we handle the exploration of model properties and alternatives?

19

Comparison and Benchmarking (context)

• Benchmarking can boost research:– Fair comparison of tools– Impartial benchmark selection which covers the spectrum of

interesting test cases– Clear documentation of outcomes; reproducibility– Publicity makes it easy to identify progress, problems

• However, in MBE:– No common standards– No community platform for benchmarks– Not clear what metrics are relevant for measuring “improvement”

20

Comparison and Benchmarking (questions)

• How to compare existing V&V tools w.r.t. functionality, coverage, scalability, expressiveness, executing system (i.e., for models at runtime)?

• Which criteria are appropriate for comparison?

• Can we globally compare fairly at all?– Broad and diverse spectrum of V&V machines:

B, Coq, HOL/Isabelle, SAT, SMT, CSP solvers, Relational logic, enumerative techniques

21

Domain-Specific Languages (Context)

• Most verification tools have hard to use input languages, alien to normal developers.– Hence, verification tools are often not used.

• MBE approaches become more and more mainstream.

• Models can simplify analysis and verification, because of the higher degree of domain semantics they express.

• Potential to exploit the two approaches synergistically– From the high-level models, we can automate the generation of

the input to the verification tools.

22

Domain-Specific Languages (Questions)

• How to define DSLs close to domain concepts but still allow generation of meaningful input for V&V tools?

• V&V tools need the specification of properties. – How to express them at the domain level in a user-friendly way? – Can the property specifications be integrated with the same DSL and/or

model used for describing the to-be-verified system without creating self-fulfilling prophecies?

• How to bring V&V feedback back to the domain level and express it in terms of the DSL-level input?

• Can incremental languages extensions help with making programs expressed in general-purpose languages more checkable? For example, the semantics of a specific extension construct may enable the generation of very rich inputs to the verification tool.

23

Thematic Categories

1. Gap between Models and V&V Formalisms

2. Need to Refine Existing Methodologies3. Design-time vs. Runtime4. Properties5. Model Transformations6. Informal vs. Formal vs. Incomplete

Modeling7. Comparison and Benchmarking8. Domain-Specific Languages

24

Conclusion

• Report-back from working group on MBE and V&V at Dagstuhl in May 2013.

• Main areas in the synergy of MBE and V&V.

• For each: described status and identified research questions.

• Our hope:

Spark discussions and debate, help focus research in MBE and V&V.

25

Questions?