Uncertainty Management With Partial Models

UncertaintyManagementWith Partial

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Uncertainty Management With Partial Models

Michalis Famelis

University of Toronto

October 2nd, 2012,

MODELS’12 Doctoral Symposium

1 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Hi, I’m Michalis Famelis

• 3rd year PhD student.

• ...“a couple years to finish”

• Supervisor: Marsha Chechik

• MSc: Model Management with RelationTypes, CS UofT 2010.

• Undergrad: Model analysis andtransformation. ECE NTUA 2008.

2 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Uncertainty in SE

“The reality of today’s software systems requires us to consider

uncertainty as a first-class concern in the design, implementation,

and deployment of those systems.” [Garlan, 2010]

• Self-adaptive systems[Esfahani et al., 2011, Cheng and Garlan, 2007]

• Probabilistic systems [Hinton et al., 2006]

• Imperfect information in requirements[Noppen et al., 2007]

• Requirements clarifications [Knauss et al., 2012]

• Risk management [Islam and Houmb, 2010]

3 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Design Uncertainty

Uncertainty about design decisions – the contents of a model.

Ultimately, we need to capture and support design uncertainty.

Enable MDSE with design uncertainty:

• Create representations

• Check properties and give feedback

• Do uncertainty-removing refinement

• Enable transformations

• Create tooling support

• Create methodological support

4 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

What is Design UncertaintyExample: a simple class diagram.

What does the modeler know?

5 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

What does the modeler not know?

5 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

5 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

5 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

5 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Dealing with Uncertainty: Options

Classical:

1 Wait until uncertainty is lifted.Inefficient; under-utilization of resources.

2 Plan for all possible solutions.Intractable: they may be too many.

3 Make a decision. Now.Artificial; risk undo or premature commitments.

We propose:

4 “Live” with uncertainty, defer resolution until moreinformation becomes available.

6 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Design Uncertainty

Uncertainty pervasive in MDSE.

But also:Uncertainty about design decisions – the contents of a model.

7 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Modeling Formalisms to RepresentUncertainty

ModalBehavioralSpecification

e.g.:MTSs [Larsen’88]

DMTSs [Larsen’90]

• Restricted to one model type only.

• Only optional elements.

• Over-approximation of set:Information loss

VariabilityModeling

e.g.:FTSs [Classen’10]TVL [Boucher’10]

Clafer [Bak’10]

• Different conceptual domain (SPLs).

• Not rich enough. Eg:– No variable meta-attributes.

(e.g. variable container class)

– No open world option.

• More heavyweight (maintain an SPLfor ephemeral uncertainty?)

8 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Uncertainty: a Set of PossibleRefinements

If we remove all uncertainty, we have a concrete refinement.

9 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Modeling Uncertainty with PartialModels

Explicating uncertainty in a partial model.

• Element is optional.

• Element can be multiplied to many copies.

• Element can be merged with others.

• Model is incomplete.

10 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

• May: Element is optional.

• Element can be multiplied to many copies.

10 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

• Abs: Element can be multiplied to many copies.

10 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

• Var: Element can be merged with others.

10 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

• OW: Model is incomplete.

10 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

• OW: Model is incomplete.

10 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

“Extended“ Partial Models

Additional constraints to correlate points of uncertainty.e.g. May Model: variant presented in [ICSE’12]:

11 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Semantics of Partial Models

12 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

12 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

12 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Compared to Related Formalisms

Partial models vs Modal behavioral specification formalisms

• Language/metamodel independent.

• Many kinds of partiality.

• Exact representation of set.

Partial models vs Variability modeling formalisms

• Specific to uncertainty.(c.f. uncertainty-removing refinement).

• Rich formalism.

• Ultimately disposable.

13 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Status

14 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Completed Work I

Explicating uncertainty

• May models (M-elements + May formula) [ICSE’12]

• MAVO models + FOL semantics [FASE’12]

• Relational-algebraic encoding of MAVO [MoDeVVa’12]

Checking properties

• Verifying May models with SAT [ICSE’12]

• Verifying MAVO models with Alloy [FASE’12]

• Verifying MAV- models with Alloy, CSP, SMT and ASP[MoDeVVa’12]

15 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Completed Work II

Uncertainty-removing refinement

• Semantics and verification [FASE’12]

• Property-driven refinement [ICSE’12]

Tooling

• Python and MathSAT [ICSE’12]

• Alloy [FASE’12]

• Translations from annotated Ecore to RA andfrom RA to SMT, CSP, ASP, Alloy [MoDeVVa’12]

16 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Status

17 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Lifting Transformations [MiSE’12]

May models + Graph transformations

Q1: How do we transform M directly to N?

– Lifted semantics of transformations, using logic.(“transfer predicates”)

Q2: Are the concretizations of N exactly the models n1 . . . nk?

– Check equivalence of encodings using SAT.(“correctness criterion”)

18 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Transformations: Next Steps

• Compositionally test Correctness Criterion.– Reuse Alloy-based verification method from [VOLT’12]?

• Systematically create Transfer Predicates using FOL.– Predicates for atomic operations as building blocks.– May be hard to simplify resulting formulas.

• Handle expanding/contracting model vocabularies.– E.g. adding a new element vs. concretizing.

• Reuse results from Category Theory? [Ehrig et al., 2006]– Generalize to full MAVO.– What are the right morphisms?

19 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Tool support

Full MAVO support for Ecore-based model editors.

• As an EMF Profile? [Langer et al., 2011]

• Via RAMification of metamodels? [Kuhne et al., 2009]– Would solve issues with concrete syntax

(e.g. when the base model is not well-formed)

Model management support

• Integration with the Model Management Tool Framework.[Salay et al., 2007]

• Inherit macro-management tools for partial models– name-based matching– structural merging– etc...

• “Push-button” integration of reasoning.

20 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Status

21 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Guidelines for EncodingUncertainty

“Placement” “Alternatives”

“Bottom-up” patterns of uncertainty.

• Help guide efficient/correct encoding.

• Support for choice of suitable reasoners.

22 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Towards a Methodology for DesignUncertainty

We know how to do reasoning, refinement, transformationsunder uncertainty.

We need to help identify what sorts of reasoning, refinement,etc. are necessary in each SE setting.

“Top-down” patterns of uncertainty:

• How does it appear in SE?

• What are its sources?

• What are its characteristic properties?

• What are refinements/transformations are sensible?

23 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

A Half Baked Example:“Disagreement Pattern”

Potential sources:

• Conflicting stakeholder requirements.

• Merge conflicts after commits.

• View-updates.

Characteristic properties:

• Is the conflict reconciliable?

• Have we reached agreement?

Sensible refinement and transformation checks:

• Does a refactoring expose/conceal the source ofdisagreement?

• Is a series of refinements a negotiation?(“are they making progress removing uncertainty?”)

24 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Outline

“....enable MDSE with design uncertainty:”

How do we go about evaluating all this?

25 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Evaluation of Individual Aspects

Experiments using randomly generated inputs

• To validate feasibility and Scalability of Property Checking[ICSE’12][MoDeVVa’12]

• Experimental driver in MMTF for rapid experiment setups.

Case Studies

• Triangulated results of random experimentation in [ICSE’12].

• Realistic bugfix of real world bug in UMLet

Theorem Proving using Alloy

• Used to verify partial model refinements in [VOLT’12]

• Technique could be adapted for proving transformations

26 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Evaluation of the Entire Approach

Plan for a Power Window Case Study

• Comprehensive case study from the Automotive domain.

• Management of uncertainty throughout the MDE lifecycle.Requirements to implementation via transformations.

However:

• Is such a case study enough?

• Best method for evaluation not clear.

• How to evaluate “green-field” research?

27 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Summary

Not well studied:Uncertainty about design decisions – the contents of a model.

28 / 33

Questions?

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Bibliography I

Bak, K., Czarnecki, K., and Wasowski, A. (2011).

Feature and meta-models in clafer: Mixed, specialized, and coupled.In SLE’10, pages 102–122.

Boucher, Q., Classen, A., Faber, P., and Heymans, P. (2010).

Introducing TVL, a text-based feature modelling language.In VaMoS’10, pages 159–162.

Cheng, S.-W. and Garlan, D. (2007).

Handling uncertainty in autonomic systems.In ASE’07.

Classen, A., Heymans, P., Schobbens, P.-Y., Legay, A., and Raskin, J.-F. (2010).

Model checking lots of systems: efficient verification of temporal properties in software product lines.In ICSE’10, pages 335–344.

Ehrig, H., Ehrig, K., Prange, U., and Taentzer, G. (2006).

Fundamentals of Algebraic Graph Transformation.EATCS. Springer.

Esfahani, N., Kouroshfar, E., and Malek, S. (2011).

Taming uncertainty in self-adaptive software.In ESEC/FSE ’11, pages 234–244.

Famelis, M., Ben-David, S., Chechik, M., and Salay, R. (2011).

“Partial Models: A Position Paper”.In Proceedings of MoDeVVa’11, pages 1–6.

30 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Bibliography II

Famelis, M., Chechik, M., and Salay, R. (2012a).

“Partial Models: Towards Modeling and Reasoning with Uncertainty”.In Proceedings of ICSE’12.

Famelis, M., Salay, R., and Chechik, M. (2012b).

The semantics of partial model transformations.In MISE at ICSE’12, pages 64 –69.

Garlan, D. (2010).

Software engineering in an uncertain world.In FoSER ’10, pages 125–128.

Hinton, A., Kwiatkowska, M., Norman, G., and Parker, D. (2006).

Prism: A tool for automatic verification of probabilistic systems.In Hermanns, H. and Palsberg, J., editors, Tools and Algorithms for the Construction and Analysis ofSystems, volume 3920 of Lecture Notes in Computer Science, pages 441–444.

Islam, S. and Houmb, S. H. (2010).

Integrating risk management activities into requirements engineering.In RCIS’10, pages 299–310.

Knauss, E., Damian, D., Poo-Caamano, G., and Cleland-Huang, J. (2012).

Detecting and classifying patterns of requirements clarifications.In RE’12.

Kuhne, T., Mezei, G., Syriani, E., Vangheluwe, H., and Wimmer, M. (2009).

Explicit transformation modeling.In MODELS’09, pages 240–255.

31 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Bibliography III

Langer, P., Wieland, K., Wimmer, M., and Cabot, J. (2011).

From uml profiles to emf profiles and beyond.In Objects, Models, Components, Patterns, volume 6705, pages 52–67.

Larsen, K. G. and Thomsen, B. (1988).

“A Modal Process Logic”.In Proceedings of LICS’88, pages 203–210.

Larsen, K. G. and Xinxin, L. (1990).

“Equation Solving Using Modal Transition Systems”.In Proc. of LICS’90, pages 108–117.

Noppen, J., van den Broek, P., and Akşit, M. (2007).

Software development with imperfect information.Soft Comput., 12(1):3–28.

Saadatpanah, P., Famelis, M., Gorzny, J., Robinson, N., Chechik, M., and Salay, R. (2012).

Comparing the effectiveness of reasoning formalisms for partial models.In MoDeVVa’12.

Salay, R., Chechik, M., Easterbrook, S., Diskin, Z., McCormick, P., Nejati, S., Sabetzadeh, M., and

Viriyakattiyaporn, P. (2007).An eclipse-based tool framework for software model management.In Proceedings of the 2007 OOPSLA workshop on eclipse technology eXchange, eclipse ’07, pages55–59.

Salay, R., Chechik, M., and Gorzny, J. (2012a).

“Towards a Methodology for Verifying Partial Model Refinements”.In Proceedings of VOLT’12.

32 / 33

Models

M. Famelis

Introduction

ModelingUncertainty

ManagingUncertainty

Completed Work

Current Work

Future Work

Evalutation

Conclusion

Bibliography IV

Salay, R., Famelis, M., and Chechik, M. (2012b).

“Language Independent Refinement using Partial Modeling”.In Proceedings of FASE’12.

33 / 33

Uncertainty Management With Partial Models

Documents

Bayesian Hidden Physics Models: Uncertainty Quantiﬁcation ...Automatic differentiation is used to compute the required partial derivatives from the leaf functions as inputs to the

Mechanistic models supporting uncertainty quantification

Structural design with polymorphic uncertainty models

Quantum Uncertainty Quantification for Physical Models ... · Quantum Uncertainty Quantification for Physical Models using ToQ.jl Author: Daniel O'Malley & Velimir V. Vesselinov Created

Uncertainty and Disagreement in Equilibrium Models

Models of Perceptual Uncertainty and Decision Making

Uncertainty assessment for PA models - Europapublications.jrc.ec.europa.eu/repository/bitstream/JRC47247... · Uncertainty assessment for PA models ... Fax: +31 224 565641 ... of

Reduced-order models for uncertainty quantification and ... · Reduced-order models for uncertainty quantification and parameter estimation in cardiac models Stefano Pagani Alfio

Uncertainty Quantification for Credible Simulation and ... · (probabilistic models) Epistemic / reducible uncertainty from lack of knowledge (non-probabilistic models) vs. Crucial:

DSGE Models with Uncertainty Shocks

Partial Models: Towards Modeling and Reasoning with Uncertainty

Uncertain Covariance Models & Uncertainty-Penalized Portfolio … · Uncertain Covariance Models & Uncertainty-Penalized Portfolio Optimization “BETTER DECISIONS THROUGH ARTICULATED

Sets of Models and Prices of Uncertainty

Uncertainty Linear Regression Models - Conference · Uncertainty Linear Regression Models ... Hence uncertainty WLS regression modeling and distributional theory are developed. Section

Uncertainty, Decisions, Models & People - AusIMM · Industry Performance: ... Portfolio Economics Asset 1 Asset n Prices ... Uncertainty, decisions models & people Steve Begg----?

Using Models in Decision Making Process Under Uncertainty · Using Models in Decision Making Process Under Uncertainty Philosophy of Models in Engineering Design Workshop, KIT ITAS,

Appendix 21 Application of Simulation Models, Uncertainty ... · EIA METHODS - APPLICATION OF SIMULATION MODELS - SENSITIVITY AND UNCERTAINTY ANALYSIS . ... The levels of modelling

Partial versus Full Species Distribution Models

Module 9 Modeling Uncertainty: THEORETICAL PROBABILITY MODELS

Models and uncertainty