It Doesn't Do What You Think It Does

  • Published on

  • View

  • Download

Embed Size (px)



    A Survey of Strategies for Gaining Confidence in Applicationsand Systems

    Justin Stoller@juststoller

  • YOUR TITLE NEEDS A TL;DRway to go...

  • So what do we mean by "Gaining Confidence" and whyshould you care?

  • First that we have a clear idea of what we are trying to gainconfidence in?

    What is the value of our software?

    What are the acceptance criteria?

    InputsOutputsPerformance CharacteristicsVisual Style

  • Does it meet the acceptance criteria?

  • What are the Rewards?What are the Costs?Where are the Risks?

  • A Few of the Topics Out of Scope

    Organization & CommunicationArchitecture & Design

  • The Presentation

    1. We will start with Language Paradigms and how theyaffect our abilities farther up the stack

    2. Using Language constructs to perform both static analysis(type systems) and dynamic analysis (error handling)

    3. Working our way up the stack to explicit low level tests4. And farther up to higher level integration tests5. We will look at combining what we've seen above into a

    simulation testing and the emergent behaviors that arefound with these

    6. And finally continuing the analysis from simulation intoproduction via monitoring

  • You forgot to say why should care...

  • Language Paradigms

    Imperative vs DeclarativeLow Level vs High Level

  • Procedural

    Type Systems that Model the Components of a ComputerTests assert the state of the worldError Handling by way of return codesGoto like statements

  • Object Oriented

    Type Systems that Model the Application DomainTests assert the state of an objectTests assert the passing of messagesMore reliance on type systemsLimit exceptional behaviors to the boundaries of thesystem

  • Functional

    Type Systems that Model the Application DomainTests assert the state of a returned data structureTests assert the distribution returned data structuresMore reliance on type systems

  • Logical/Relational

    Type Systems as Theorem ProversTests assert the end state of the application

  • Testing Layers

  • A single unit test of logic: conditional, etc

    Simple units of logic require simple tests

    However low level tests tend to be very tied toimplementations

    Good design facility but rarely good at ensuring quality

    Often written in the language of the software and containsthe pros and cons of the software's language

  • The behaviors of Integrated Units

    Tests are tied to behaviors of the systems

    Increasing complexity of tests and prerequisites

    Requires decent design, good at gaining confidence

  • The System as a Whole

    What properties does it display

    Analysis of emergent behaviors

  • A wide ranging spectrum

    Higher level languages start farther up the level of strategies

    The performance characteristics of the tests determine howoften they are ran

    Higher level tests with simple ways to instantiate themprovide more value as ways to gain confidence in the


  • Novelty in combining concepts

    Instrumented or "monitored" development environments