Bug best practice

BUG BEST PRACTICE

LIANG GAO ([email protected])

AGENDA

Bug filing best practice

You should learn….

Value added for bug management

Defect management

Bug follow up best practice

BUG IS IMPORTANT

Your performance is partially judged by the quality and amount of the bug you filed.

As an engineer, reputation is the most important thing. A tester’s reputation is as good as the quality of your bug can get.

Most of the time, it is the only way you communicate with the developer.

Who will read your bug report? – developer, your managers, developer’s manager, product manager, marketing folks, customers……

WHERE ARE BUGS COME FROM?Laziness (does not perform error handling)

Inexperience (does not know the impact of code changes)

Carelessness (blind cut-n-paste)

Bad design/infrastructure (bad module design un-maintainable code)

Incomplete documentation (misunderstood customer requirements, missing information in func spec)

Complexity of software (hard to maintain/enhance/test)

Bad tools (compiler, CM tool, merge tool, etc)

Lack unit-level testing (misses lots of basic bugs)

Time pressure (fatigue) Laziness and carelessness

Code scales linearly, the complexity scales exponentially

THE QUALITY OF A TESTER Script

Quality

Test Case Quality

Bug Quality

Case Execution Quality

HOW TO FIND A BUG? You think it is a bug because?

• Understand the product/function spec correctly• Understand the use case scenarios correctly• Put yourself in a user’s perspective.• Your experience counts a lot, 纸上谈兵 is harmful

Bad bugs cause you trouble

• Junk bug is counter productivity, and ruin your reputation

• Bad documented bugs cause you a lot of time and energy, like…. living in a nightmare.

HOW TO FIND A BUG? Debugging is like watching Magic Tricks

(Michael Amma17:40 - 19:00)

James Brown 5:10

WHY I CAN FIND HIGH QUALITY BUGS BUT YOU CAN’T?

High quality bug

• Affect the major functionality QoS still work after delete the configuration

• Not Cosmetic Bug 3744 - WebUI的导航栏没有滚动条 ,不便于用户浏览操作

Because I understand the functionality deeper than you

Because I use more testing types than you.

Because I design more use case than you

Because I carefully execute the cases than you

Because I pay more attention to the details than you.

Because once I found a crack, I’ll keep on going, but you didn’t

NONREPRODUCIBLE BUGS – HEADACHE?

Random is not equal to irreproducible

NONREPRODUCIBLE BUGS – HEADACHE?

Always report non-reproducible bugs, they might be time bombs.

Non-reproducible bugs are always reproducible.

• Timing and delay?• Only happen during installation• Depend on some data like corrupted database?• Sequence of the execution?• Interaction with other programs in the background?

NONREPRODUCIBLE BUGS – ELABORATE

a. I saw this one, but was never able to make it happen again.

b. I see this when I run with this setup, but sometimes I don't.

c. This is the same anomaly I see under several setups, but not all the time.

d. Under X setup, this happens sometimes and not other times. There may be something different from one X to the other, but I' not seeing it.

What is the impact if this happens all the time?....

NONREPRODUCIBLE BUGS – DEBUGGING STORY

²»¿É¸´ÏÖBugÊµÕ½

×¨¼ÒÖ®Ì¸

×¨¼ÒÖ®Ì¸ - 2

AVOID CRITICAL BUG IN THE LATE DEVELOPMENT

Execute test cases based on priorities

Discover critical bug in the early development stage

• It gets fixed early, less resource waste• It gives developer more lead time – critical bugs are hard to fix

MOST IMPORTANT THINGS ON A BUG

Subject, Subject, Subject

• One line• Exactly pinpoint the problem.• Accurately describe the problem

Bug 3747 - WebUI界面上的字体、图标等在 Linux系统下的显示不正常

Bug 3774 - 客户端能够通过 Admin 端口与 DUT 建立 PPTP 连接 , 不符合设计规范

Bug 3768 - admin route: 不能为同一个服务创建多条 admin 路由

Bug 3788 - 在配置 DUT 的时候， DUT 死机

Which one is better?

WHAT IS THE BUG QUALITY

Reproducible

• Testers filed bug need to be reproducible in developer’s environment.

• Customer found bugs sometime is hard to reproduce, but that not tester’s concern.

• Test case is not the step to reproduce.• Need to find the main cause by ruling our the unrelated

factors. Need to pinpoint the root cause (Strong Analytical skills).

• 蛛丝马迹只是现象，需要进一步追踪，找到问题的本质。 ISIC crash the box, can you find out which packet, and why?

HOW TO WRITE A REPRODUCIBLE BUG REPORT

A short paragraph for detail description.

A topology file with detail annotation. (IP address, DUT type etc)

Step to reproduce

• Step 1. execution step with detail output.• Step 2… execution step with detail output• Seep 3: where the problem is. What is the expected output,

what is the actual output.

Detail DUT configuration dump,

Anything else (document) you think can claim the validation of your bug?

Review each others bug reports

AGENDA



Defect management

Integration of bug tracking system with other engineering tools

UNDERSTAND BUG STATE FIRST!N:

A:

I: Information needed: Take action ASAP

R: Need Tester’s verify. Take action ASAP

V:

Monitor your bug daily: usually you will get an email reminder whenever a bug state is changed. Pay special attention to “I” and “R”

Best practice: I and R state bug need to be updated within 24 hours.

Ask your team lead for help, monitoring bug every day is hard.

INTERACT WITH DEVELOPERS -1

Understand the developer’s response.

It is like writing an email, need to be polite. Need to have start paragraph and end paragraph.

It needs to be thorough and accurate (provide as many useful info as possible).

A quick and good response help you gain the respect from developer.

Ask your team lead for help is a good idea.

INTERACT WITH DEVELOPERS -2

Meet the programmers who will read your report

The best approach may be to demonstrate your bugs to the programmers.

When the programmer say it is fixed, make sure it isn’t still broken

Verify bug fixed promptly

When fixes fail, talk with the programmer.

Bug report should be closed by testers..

AN EXAMPLE OF QUERY

State-Changed-From-To: open->infoState-Changed-By: murphywState-Changed-When: Sun, 24 Jun 2007 23:14:08 -0700

State-Changed-Why:

1. ‘set policy id 1 attack "CS:ftp" action close’, does this CLI mean that we should drop FTP traffic?

2. Please help to provide debug information on successful case.

3. Please turn on snoop and snoop detail during debugging.

Thanks!

How Would you response?

AGENDA



Defect management


DEFECT MANAGEMENTHow to know how many bugs filed per person, per day – bug search is your best friend.

How to know how many bugs filed for certain type of testing

• Search for testing type – regression; performance; functionality;…

• Add key word in front of the bug subject field then search the keywordREG: extended access list can not block traffic. Pre-Test:-Pascal-when the packets can't pass through VPN tunnel with deep inspect

• Be careful when add the interest field: those are the person will be notified every time this bug state is changed.

• If there is a field called attribute, use it. It helps to categorize and manage bugs we filed.

FILING BUG REPORTSFile every bug found (except duplicate bugs), annotate if a bug:

• Blocker: High priority bug preventing further testing• Regression: Bugs that introduce in new build• Unreproducable: Bugs that cannot be reproduced• Customer found: Bugs found by customers (Beta/FCS)

Pick right Priority (usage) & Severity (impact)• P3/S1 : Login 200 times with 200 characters user name will

crash the device with exception core dump• P1/S3 : GUI splash screen misspelled company name

Bug report is a statistics tool, use to predict readiness of software and future releases

DEFECTS QUALITY CONTROL PROCESS

BUG TREND CONVERGENCE

Bug Filed

Release Date


Bug Fixed

Release Date

Bug Filed


发现 Bug

正常测试分散测试

发布日期

Bug 修复

Bug 严重程度日期

Bug 数目


发现 Bug


发布日期

Bug 修复

Bug 严重程度日期

Bug 数目


发现 Bug


发布日期

Bug 修复 Bug 严重程度日期

Bug 数目

USE BUG TREND TO FIND QUALITY PROBLEM

USE BUG TREND TO FIND QUALITY PROBLEM

BUG FIELDS FOR METRICS

Status

Date Reported

Who Found

Assigned To

Program Area

Severity

Priority

Resolution

INTERESTING BUG METRICS

Priority 1 Find Rate− This is the find rate of bugs that at any point in time become priority 1. It is a retrospective analysis.

Open Bug Aging Report − A histogram of how long bugs have been open. This is most useful early in long projects.

Deferred Bug Aging Report − A histogram of how long deferred bugs have been waiting on the list. It helps spot chronically deferred bugs.

Promotions/Demotions/Deferrals Chart − Three lines that help us see the triage process at work.

AVOID CONTROL METRICS;EMBRACE INQUIRY METRICS

A control metric is any metric that drives decisions. − Any metric you use to control a self-aware system will be used by that system to control YOU.

An inquiry metric is any metric that helps you ask the right questions at the right time. − An inquiry metric might look like a control metric. The difference is how you use it and what you infer from it.

EXAMPLE CONTROL METRICS

“The developer who creates the fewest bugs will receive a bonus.”

“Testers must average at least three bugs found per day.”

“The product may not be released unless we’ve gone at least one week without finding a bug.”

“The product may not be released with more than 10 high severity bugs.”

EXAMPLE INQUIRY METRICS

Why do some developers have more bugs reported on their code than others? How might it be good to have more bugs reported? How might it be bad?”

“How do find rates differ among testers? What makes them differ?”

“What does the bug find rate tell us about the readiness of the product? Could the find rate be falling because testing is slack, and not because the product is good?”

“What are the high severity bugs? Should we fix them?”

INQUIRY METRICS INVITE LEARNING

Observe: You try to see what's happening.

Inquire: You conjecture about the meaning and significance of the observations. You collect additional observations to corroborate or refute our conjectures.

Model: You form and improve theories about why the system behaves as it does, how you know that it behaves that way, and what you can do about it.

QUESTIONS BUG METRICS MIGHT HELP WITH TEST PROCESS

Productivity

• − Are the testers working at full speed?• − Are they speeding up or slowing down?• − Is anything blocking their work?• − Are the testers burning out or staying sharp?• − Are testers and developers getting along together?

Status

• − How close is the test effort to reaching a point of diminishing returns?

• − What kinds of problems are being found?• − Is the bug tracking system being used properly?

Focus

• − Are the testers focused on the areas of greatest risk or need?• − Could the testers use some outside help?

QUESTIONS BUG METRICS MIGHT HELP WITHQUALITY IMPROVEMENT

Productivity

• − Are the developers working at full speed?• − Are they speeding up or slowing down?• − Is anything blocking their work?

Status

• − Is the product improving?• − How close is the product to being good enough?• − What must happen in order to meet the schedule?

Focus

• − Is the work distributed effectively among the developers?• − Do any areas of the product need more attention to improve

more quickly?• − Is the triage process working well?

DAILY METRICS!

Consider producing a bug metric summary:

• − Daily• − Automatic• − Archived• − Web Based

This becomes a resource for retrospective analysis.

Additionally, daily graphs can help you notice fast breaking dynamics.

Include weekends in the graphs.

EXAMPLE: WHY WAS CONVERGENCE HARDER DURING THE SECOND CYCLE?

CYCLE 1 CRITICAL BUGS

CYCLE 2 CRITICAL BUGS

DIFFERENT SLOPS

WHY SLOW CONVERGENCE?

OPEN BUGS BEFORE ENTER CYCLE 2

LOW PRIORITY BUG WAIT FOR THE FIX

AGENDA



Defect management


VALUE ADDED DEFECT/CODE INTEGRATION

Bug system can be integrated with CVS, Subversion etc code control tool

When check in code, developer can specify the bug id, the bug will be automatically updated with code diff.

When check in code, developer can specify the bug id, and bug’s state will be automatically changed to “R”, and trigger tester to verify it.

Other scenarios you can think of?

This is the area that we can provide the value added for our customer

THINGS YOU SHOULD REMEMBER AFTER THIS

Bug quality means it can be reproduced by developers in their environment.

Most important thing on a bug is its subject

Junk bug is counter productive, and ruin your reputation.

I, R state bug need to be updated ASAP.

Use Sigma Bug Template to file bugs.

BUG REVIEW CHECK LIST - 1Is the summary short (about 50-70 characters) and descriptive?

Can you understand the report?

• As you read the description, do you understand what the reporter did?

• Can you envision what the program did in response?• Do you understand what the failure was?

Is it obvious where to start (what state to bring the program to) to replicate the bug?

Is it obvious what files to use (if any)? Is it obvious what you would type?

Is the replication sequence provided as a numbered set of steps, which tell you exactly what to do and, when useful, what you will see?

BUG REVIEW CHECK LIST - 2

Does the report include unnecessary information, personal opinions or anecdotes that seem out of place?

Is the tone of the report insulting? Are any words in the report potentially insulting?

Does the report seem too long? Too short? Does it seem to have a lot of unnecessary steps? (This is your first impression—you might be mistaken. After all, you haven’t replicated it yet. But does it LOOK like there’s a lot of excess in the report?)

Does the report seem overly general (“Insert a file and you will see” – what file? What kind of file? Is there an example, like “Insert a file like blah.foo or blah2.fee”?)

BUG REVIEW CHECK LIST – 3Can you replicate the bug?

Did you need additional information or steps?

Did you get lost or wonder whether you had done a step correctly? Would additional feedback (like, “the program will respond like this...”) have helped?

Did you have to guess about what to do next?

Did you have to change your configuration or environment in any way that wasn’t specified in the report?

Did some steps appear unnecessary? Were they unnecessary?

Did the description accurately describe the failure?

Did the summary accurate describe the failure?

Does the description include non-factual information (such as the tester’s guesses about the underlying fault) and if so, does this information seem credible and useful or not?

BUG REVIEW CHECK LIST – 4Does the description include non-factual information (such as

the tester’s guesses about the underlying fault) and if so, does this information seem credible and useful or not?

Does the description include statements about why this bug would be important to the customer or to someone else?

VERSION CONTROL

1st draft, Dec 7th, 2007, Liang Gao

2nd draft, Dec 27th, 2007, Liang Gao

Technology

Bug best practice