©2014 Google -- aleax@google.com

There WILL be bugs! !

http://www.aleax.it/bayp14_twbb.pdf!

1

"Will there be bugs"?YES!!!

with probability above 0.9999... Murphy's Law fully applies (& then some:-)

22

"When will there be bugs"?when you're pushing the envelope

new or new-to-you algorithms, fields, libraries, frameworks, ... ...there are new traps waiting for you

when you're doing humdrum, routine development or maintenance...

...what you've done 1,000 times before...

...your attention level's likely not 100%! ...and, in all cases in-between:-)

33

The 1 exception in my lifemy very first program: 1 run, bug-free 1974: Fortran, conditional probabilities of suits in bridge, mainframe 3 HW majors, bridge enthusiasts (key!), no programming courses (NP), humble (!) punched cards,"big brain" mystique (helped!) "invented" code-reviewing, pair-prog x 1.5 had PCs been easily around (a bit later)...

...bugs would have abounded (later did!-) ...never ever happened again in my life!

44

Fast Forward 40 years...downside: we only got 1 run so couldn't possibly have run tests

not that we'd ever heard of testing!-) nowadays, tests must be at the heart of bug avoidance, discovery, and fixing ...as they should have been in the '70s!-)

Knuth, 1977: "Beware of bugs in the above code; I have only proved it correct, not tested it"...!-)

but code reviews & pairing still help!

55

Where do bugs like to hide?Well, anywhere, actually!-)

66

Common bug-hiding places"advanced" stuff, or where you're "clever"

key fix: *simplify* (+, unit-test!) where you did not fully understand the problem, architecture, platform

ditto +: acceptance tests, code reviews boilerplate/duplicate code

Don't Repeat Yourself (DRY) rarely executed code (error handling, ...)

key fix: unit-test w/mocking for errors previously-buggy code ("regressions")

a special case of TDD

77

Simplify, simplify, simplifyKernighan: "Debugging is twice as hard as writing a program in the first place. If you're as clever as you can be when you write it, how will you ever debug it?" break up long, complicated expressions (intermediate results -> local variables) beware complex decision chains

and nested loops / recursion, esp. with many conditional break/continue stmts

regular expressions MUST be tested a LOT ( also see: regex101.com )

88

Don't Repeat Yourself (DRY)Hunt and Thomas (1999): "Every piece of knowledge must have a single, unambiguous, authoritative representation in a system". AKA Once and Only Once (focus on code)

copy-and-paste coding is a main anti-pattern breaking DRY / OaOO duplicate code is the worst code smell

everything's changing all the time: with duplicates, you'll miss some needed change abstract what varies, merge what doesn't

99

Before DRY, it's WET...:if foo(bar, baz):! return bar!else:! return baz!! ...!!if foo(zip, zap):! return zip!else:! return zap + 1

1010

...let's DRY it up!def picker(a, b, c=None):! if foo(a, b):! return a! else:! return b if c is None else b + c! ...!return picker(foo, bar)! ...!return picker(zip, zap, 1)!

1111

Rarely executed codehandling weird errors, corner cases unittest.mock (3.3+; backports aplenty) with mock.patch(...) as x:!

to locally replace ... with a Mock obj x set return_value, side_effect, ...

after, may assert about x's calls &c BEWARE:

all attributes and methods auto-appear -> high danger of typoes!

use auto_spec to reduce the danger

1212

A mock.patch exampledef foo():! try: never.fails()! except OopsItDid as e:! logging.warm('Failed: %s', e)! raise! ...!with mock.patch('never.fails'‡) as nf:! nf.side_effect = OopsItDid! with self.assertRaises(OopsItDid):! module_i_am_testing.foo()!

‡: add autospec=True &c

1313

Why do bugs happen?Your brain tricks you!

Perception: See what you expect to see Attention lapses Overconfidence Confirmation bias

You've written it: it's YOUR code, it's YOUR baby, you're really proud of it

bugs love to hide there, as `you won't see them` -- you're "too close" to see clearly

1414

Remedies for your “tricksy brain”egoless programming more eyeballs

open source pair programming code reviews

testing `lint` and similar static-analysis tools

1515

Egoless ProgrammingJerry Weinberg, "The Psychology of Computer Programming", 1971 1.Understand and accept that you will

make mistakes; find them early! 2. You are not your code; reviews are to

find problems, and find them they will -- don't take it personally!

... 10. Critique code instead of people – be

kind to the coder, not to the code

1616

Eyeballs: open sourceEric Raymond's "Linus Law" (1997): "given enough eyeballs, all bugs are shallow"

"Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix will be obvious to someone."

Glass "Facts and Fallacies about Software Engineering" (2003): "it's a fallacy"

"the rate at which additional bugs are uncovered does not scale linearly with the number of reviewers"

`goto fail`, `heartbleed`; see Mike Bland's http://martinfowler.com/articles/testing-culture.html

1717

Eyeballs: pair programmingthe "driver" codes, focuses on "tactical" aspects of completing the current task the "observer" reviews-as-it-goes, ponders "strategic" issues, plays "safety net"

frequent role switches; talking through it many empirical studies and meta-research confirm: higher quality, faster time, but larger effort, compared with `solo` progr.

effort estimation here is biased-high e.g: P.P produces a higher `bus number` w/o extra study/analysis effort!

1818

Eyeballs: code reviewstraditional "high ceremony" `Fagan Inspections` -- multiple meetings, printed code, thorough line-by-line critique, ...

high cost, very slow, high efficacy may "fail to see the forest for the trees", "bikeshedding" also a risk

lightweight, informal walkthroughs and critiques - much faster, also effective

especially with the right tool! complementary, not alternative, to pair programming and testing

1919

TestingJacob Kaplan-Moss: "code without tests is broken by design":-) informal "just run it and see if it breaks"

quite ineffective, esp. by original author automated light-weight "unit tests"

must run fast so you can and will keep re-testing even on minor changes great at avoiding unit-specific issues

automated middle-weight "integration tests" complementary, not alternative, to u-t's focus on interaction between subsystems

formal heavy-weight "acceptance tests"/QA2020

Testing vs EyeballsTesting: objectively shows bugs' presence

can never conclusively show absence:-) automated -> re-run all the time

Eyeballs: complementary, not alternative pick up issues testing can never reveal

lack of clarity, bad naming, complexity not automated -> consume human time

A third leg for the anti-bug "stool": lint automated, fast uniform style -> more productive eyeballs

2121

Anti-bug toolsmost important is source code control (svn, hg, git, ...) -- trace what changed and when

beware subtle merges, cherrypicks, ... next, a testing framework (unittest &c) a bug tracker (ideally integrating w/SCC) a code review tool (ditto)

2222

“What about debuggers?”only mild importance for fighting bugs

if the code's so complicated that you have to follow step by step, simplify it

can help as "learning device" unfamiliar code, library, framework

2323

Beware tool-itis!a geek's natural attitude: "if I have the right tools, all bugs will flee in terror"

they won't flee: bugs are courageous!-) tools, at best, help: they don't solve bugs tools can in fact hurt, by distracting you

hours spent in front of a cool debugger ...yak-shaving (tracing just-fine code)...

MUCH more important than tools' details: attitude, skill, care, focus on team&user and esp: good practice (light process)

2424

And yet, tools are cool:-)...but, excessive power can hurt the unwise

debugger to interactively examine values: log them instead! (w/logging.debug)

then code a script to sanity-checks logs big difference: it's automated!

SCC allows fancy integrates, merges, cherrypicks, and generally funky graphs

very high risk of introducing bugs! keep SCC graphs clean and simple!

key idea: good enough IS good enough

2525

Testing frameworksstdlib unittest: a good starting point

+ many extensions: nose & plugins, &c coverage's important (figleaf's ok too:-)

use doctest only for examples in docs! it's designed for that & does it well

automated test runners / CI nosy, nosier, nosyd, PyZen; buildbot

mocks, fakes, stubs, spies, dummies, ... unittest.mock, but use with care!-)

specialized: web, fuzzing, GUI, acceptance...

2626

Web testingeither: simulate a browser

simple, fast -- but, no Javascript, CSS &c! best for unit-tests

or: automate a real browser most powerful, realistic, but, slower best for integration & acceptance tests

specific web frameworks may further help from google.appengine.ext import testbed!from django import test!...

2727

Good enough...pick one from each group

the standard-er, the better

2828

Example: web testingeither: simulate a browser

simple, fast -- but, no Javascript, CSS &c! best for unit-tests

or: automate a real browser most powerful, realistic, but, slower best for integration & acceptance tests

specific web frameworks may further help from google.appengine.ext import testbed!from django import test!...

2929

Linting &cLogilab's pylint

very powerful, configurable unused/unassigned variables too-long modules/functions style violations in naming, wspace, ...

pyflakes, pep8: limited, but fast Clone Digger

finds some "clones" (duplicate code) ...

3030

When to use a bug trackerAlways!-)

"bug tracker" (BT) is a misnomer...: track both bugs AND features

Integration w/SCC: any changeset must identify which BT entry it regards

ideally only 1: keep CSs small! Integration w/code review tool: code reviewer can follow the BT entry to find out WHY the CS under review exists BT entry points back to CSs fixing it

3131

The worst kind of bugrace conditions (& their evil cousins: deadlock, starvation, ...) alas, test don't help much (!); code reviews may if super-duper-hyper careful/thorough prevention is by far the best cure here

avoid shared-RW-memory concurrency message passing, shared-RO-mem OK

if you really can't, rigorously sequence lock acquisitions, a la Dijkstra

maybe a dedicated global-memory-changing thread w/Queue of changes

3232

Test-Driven DevelopmentAgain: Jerry Weinberg, "The Psychology of Computer Programming", 1971 `In program testing, the programmer who gets early "success" with his program is likely to stop testing too soon. One way to guard against this mistake is to prepare the tests in advance of testing and, if possible in advance of coding.` write the tests; see them fail; fix code to pass each test; check they succeed; refactor and check they still succeed.

3333

TDD: worth using? (1)(personal opinion here, but, based on substantial experience...) for adding features...: hmmm...

what exactly do you unit-test? proper test targets for most features, "user stories", are more suitable for acceptance/integration than unit tests

Behavior driven development (BDD): TDD variant based exactly on user stories

ideally written by users/PMs/&c great if you get such users/PMs/&c:-)

3434

TDD: worth using? (2)for fixing bugs...: absolutely YES!

you know exactly what to unit-test for: the very bug you're fixing!

first, you write the new tests ...and make sure all the new tests fail "reproduces the bug" in a known state

then, you fix the bug ...and make sure all the tests pass

the new tests stay in the test-suite insurance against future regressions!

3535

TDD: worth using? (3)for refactoring...: just *NOT APPLICABLE*!

you can NEVER safely refactor code not already well covered by good tests ensuring such coverage is not "the start of the refactoring",

it's a *PRE-REQUISITE* of it! Michael Feathers, "Working Effectively with Legacy Code", 2002 -- http://www.objectmentor.com/resources/articles/WorkingEffectivelyWithLegacyCode.pdf

3636

Q & Ahttp://www.aleax.it/bayp14_twbb.pdf!

37

? !37