View
33
Download
0
Category
Tags:
Preview:
DESCRIPTION
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis. Yungbum Jung, Jaehwang Kim, Jaeho Shin , Kwangkeun Yi Programming Research Lab. Seoul National University. Motivation : an Industry’s Challenge. - PowerPoint PPT Presentation
Citation preview
Taming False Alarms from a Domain-Unaware
C Analyzerby a Statistical Post
Analysis
Yungbum Jung, Jaehwang Kim,Jaeho Shin, Kwangkeun Yi
Programming Research Lab.Seoul National University
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 2
Motivation: an Industry’s Challenge
In 2004, a company’s SQA dept. asked us for a C buffer-overrun static analyzer that must be sound must have a reasonable cost must be domain-unaware
Our path Sound analyzer: drive cost-accuracy balance to
a limit Statistical filter: sift out inevitable false alarms
and rank alarms by their true probabilities
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 3
Outline
Airac, Our Analyzer Internals Performance
Statistical Analysis Symptoms Models
Bayesian Analysis Linear Logistic Regression
Sifting out, Ranking
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 4
Airac
Array Index Range Analyzer for C Our static analyzer
Is an abstract interpreter Does numerical interval analysis Is sound
in sense of detecting all possible buffer overruns
Covers full ANSI C + some GNU extensions
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 5
Abstraction
Usual abstraction for stateful programs
Set of concrete machine transition
traces
Map from program points to abstract statesPgmPt State
α
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 6
Abstract Domains
Machine = State x PgmPt State = Stk x Mem x Dmp Mem = Addr Val Val = Interval x 2Addr x 2Array
Addr = PgmVar + AllocSite + AllocSite x Field Array = AllocSite x Base x Size AllocSite = PgmPt [a, b] ∈ Interval = Base = Size
...
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 7
Techniques Used
Accuracy improvement by narrowing after widening flow-sensitivity context pruning (limited to linear expressions) static inlining (parameterized) static loop unrolling (parameterized)
Cost reduction by careful worklist order: lazy at join points selective join/compare stack obviation
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 8
Stack Obviation
Size of Stk proportional to program size Most of the analysis time = join + compare OK to skip join/compare for Stk
if changes of Stk always reflected on Mem By simple syntactic transformation
e1 ? e2 : e3 { if (e1) t = e2 else t = e3; t } e[f()] t = f(); e[t]
3~5 times speed up
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 9
Optimistic Assumption:
i [0, 9] j [0, 18]
Error Recovery During Analysis
1: int a[10], i, j;
2: for (i=0;i<10;i++) {
3: a[i] =2 * i;
4: }
5: j = a[i];
6: a[i] = …
…
buffer overrunsince i [10, 10]
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 10
Warnings about Performance
Assume typeful C programs arrays must be used as the same type declared
Artificial semantics after errors e.g. overrun, null dereference
No side-effect for library functions No main() then
analyze procedures in their defined order No alarms about buffers whose size is top Top value for free variables
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 11
Performance 1/2
Linux kernel 2.6.4 Alarms Real Errors LOC Time (sec)
vmax302.c (79)
1 1 246 3
xfrm_user.c (235) 2 1 1,201 109
usb-midi.c (332) 10 4 2,206 3617
atkbd.c (332) 5 2 811 285
keyboard.c (411) 2 1 1,256 9
af_inet.c (48) 1 1 1,273 79
eata_pio.c (183) 3 1 984 8
cdc_acm.c (468) 5 3 849 119
ip6_output.c (198)
0 0 1,110 45
mptbase.c (777) 2 1 6,158 8251
aty128fb.c (98)
2 1 2,466 3671Performed on a Linux 2.6 box with Pentium4 3.2GHz, 4GB RAM
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 12
Performance 2/2
GNU SoftwareAlarm
sReal Errors LOC Time (sec)
tar-1.13 (2,630)
66 1 20,258 577
bison-1.875 (5,164)
50 0 15,907 809
sed-4.0.8 (461) 29 0 6,053 1154
gzip-1.2.4a (799) 17 0 7,327 794
grep-2.5.1 (187) 2 0 9,297 604Commercial SoftwareAlarm
sReal Errors LOC Time (min)
A 18 9 280,379 8
B 196 563,584,66
4789
C 78 15 119,211 82
D 435 7 806,829 112
E 197 112 517,314 8
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 13
Statistical Post Analysis
1. We collect Samples of true and false alarm Symptoms of each alarm
2. From them, compute trueness of alarms i.e. probability being true given its symptoms
3. With trueness we can Sift out false alarms Report truer alarms first
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 14
Symptoms Syntactic symptoms
- AfterLoop, AfterBranch, AfterReturn, InNestedLoopBody, InNestedBranchBody
+ InLoopCond, InBranchCond, InFunParam, InNestedFunParam, InRightOfAnd
Semantic symptoms- JoinN, NotNarrowed, ComplexData, InCyclicCallChain+ Prunning, PassedValue, ConstantVariable, ConstantIndex, Consta
ntArrayConstantIndex Result symptoms
- TopIndex, HalfInfiniteIndex+ FiniteOffsetFiniteArray, FiniteIndex
Common-sense + shallow inside info
f
g
h
[9, 10][9, 10][9, 10][9, 10]
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 15
Bayesian Analysis For each alarm, we compute its conditional
probability being true given its symptoms
Numbers from “learning samples” Estimated using Monte-Carlo method
We assume symptoms occur independently (naïve Bayesian filtering)
We assume symptoms occur independently (naïve Bayesian filtering)
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 16
Sifting Out Threshold
User’s knob: his/her risk ratio (Rs/Rr)
Minimize risk expectation Risk expectation of an alarm with probability p when
Silencing = Rs x p Reporting = Rr x (1 – p)
We silence if Rs x p < Rr x (1 – p) Hence, sift out when p < Rr / (Rr + Rs)
Risk oftrue
errorsfalse
alarms
silencing
Rs 0
reporting
0 Rr
= 1 / (1 + Rs/Rr)
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 17
Experiments
With alarms from Parts of the Linux kernel Programs in algorithm text-books
Learning and testing 50%/50% randomly chosen
15 times repeated
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 18
Sifting Out Alarms Rs = 3 x Rr threshold = 0.25 74.84% of false alarms filtered out :-) 31.40% of true alarms were also swept out :-(
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 19
Ranking Alarms Show user “truer” alarms first 15.17% of false alarms are mixed up
until the user sees 50% of the true alarms
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 20
Binary Logistic Regression
Trueness of an alarm given its binary symptom vector
Generalized linear model Coefficients from learning set For example,
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 21
Bayesian vs. Logistic Regression 1/2
With threshold 0.25, Bayesian: 74.84% of false, 31.40% of true Logistic Regression: 90.05% of false, 20.85% of true
alarms can be sifted out
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 22
Bayesian vs. Logistic Regression 2/2
Until user sees 50% of true alarms Bayesian: 15.17% Logistic Regression: 4.10%
of false alarms were mixed upConjecture:Logistic regression model respects symptom dependency?
Conjecture:Logistic regression model respects symptom dependency?
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 23
Related Work
Buffer overrun detection ARCHER [Xie, Chou & Engler 2003] SPLINT [Zitser, Lippmann & Leek 2004] CSSV [Dor, Rodeh & Sagiv 2003] ASTRÉE [Cousot et al. 2005, 2003]
Statistical approach Z-ranking [Kremenek & Engler 2003] Error Correlation [Kremenek et al. 2004]
unsound
require
annotation
domain-aware
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 24
Conclusion
Our “sound” static analyzer,Airac is realistic
False alarms are inevitablein domain-unaware situation
Statistical approaches helped viable approach to handle false alarms natural symptoms seem to work orthogonal to other static analysis
techniques generic, depends on learning set
Taming False Alarms from a Domain-Unaware C Analyzer by a Statistical Post Analysis SAS 2005
Jaeho Shin 25
Thank you
Questions?
Demo available at http://ropas.snu.ac.kr/airac
Recommended