Standard Regression Testing Does Not Work

  • View
    630

  • Download
    1

Embed Size (px)

Transcript

  • Standard Regression Testing Does Not WorkDaniel Hansson, CEO Verifyter

  • Standard Regression TestingCheck out the latest from the version control systemKick off a script that runs a lot of tests and present the test results Email the engineer(s) who have committed new changes to the revision control system since the last time the test(s) passed

    *Daniel Hansson, Verifyter AB*

    Daniel Hansson, Verifyter AB

  • Blame 3 Standard OptionsBlame all commits since last pass

    Blame the first commit that caused the test to fail

    Dont blame anyone. Manually debug the test failure

    *Daniel Hansson, Verifyter AB*Large Test Suite (testing every x revision) Small Test Suite (testing each revision)

    Daniel Hansson, Verifyter AB

  • The Standard Approach*Daniel Hansson, Verifyter AB*

    If we just test often enough then we will know who caused each regression failureHey, its this guy!Looks simple, but is it true?

    Daniel Hansson, Verifyter AB

  • We Checked*Daniel Hansson, Verifyter AB*Is that the reason the test fails on the latest?If so, undoing the change should make the test pass

    Daniel Hansson, Verifyter AB

  • The Result*Daniel Hansson, Verifyter AB*We checked 916 bug reports in ASIC projectsWhy is this simple approach so unreliable?In 41% of the cases the bug report was wrong!

    Daniel Hansson, Verifyter AB

  • Accurate/Inaccurate Blame*Daniel Hansson, Verifyter AB*This commit was the reason this test started to failA lot of things happen after the first failureThe test must still fail for the same reason hereAccurate BlameInaccurate Blame

    Daniel Hansson, Verifyter AB

    *

  • *

    Inaccurate Blame*Daniel Hansson, Verifyter ABMany Scenarios

    Daniel Hansson, Verifyter AB

  • Two SolutionsPrevent ComplexityTest one commit at a timeStrict Continuous Integration

    Handle ComplexityYou must debug correctly all complex scenariosAutomatic debugger of regression failures*Daniel Hansson, Verifyter AB*

    Daniel Hansson, Verifyter AB

  • Strict Continuous Integration*Daniel Hansson, Verifyter AB*Test each commit. Only let it in if the tests passPerfect if complete test suite takes minutes and does not contain random tests. Very popular in the software industry.

    Daniel Hansson, Verifyter AB

  • Automatic Debugger*Daniel Hansson, Verifyter AB*

    Daniel Hansson, Verifyter AB

  • Rerun on Old Revisions*Daniel Hansson, Verifyter AB*Rerunning tests on older revisions to find bad commitOnly takes 1 4 reruns of the failing test to find culpritCombines revisions in order to analyze complex scenariosThis is only done for one test per bug

    Daniel Hansson, Verifyter AB

  • Farm Usage*Daniel Hansson, Verifyter AB*Typical cost for re-running tests: 10 jobs on the farmGood investment to avoid next test run to show same state

    Daniel Hansson, Verifyter AB

  • Benefits*Daniel Hansson, Verifyter AB*400% Faster Fixes, 5x Less Discussion, to 11% Shorter Projects

    Daniel Hansson, Verifyter AB

  • Summary*Daniel Hansson, Verifyter AB*

    Only works for very small test suites. Random not supportedCan debug complete test suite, incl random testsWith continuous integration all large test suites still needs to be debugged either manually or by PinDown

    Daniel Hansson, Verifyter AB

    *