Bayesian Methods in Probability of Detection …computationaltools.com/Papers/Aldrin Knopp - 2012 - QNDE -Bayes POD...Bayesian Methods in Probability of Detection Estimation and Model

1 July 19, 2012

Bayesian Methods in Probability of Detection Estimation and Model-assisted Probability of

Detection (MAPOD) Evaluation John C. Aldrin*, Jeremy S. Knopp, Harold A. Sabbagh†

Nondestructive Evaluation Branch (AFRL/RXLP)

Materials and Manufacturing Directorate Air Force Research Laboratory

Wright-Patterson AFB, Ohio, USA

*Computational Tools, Gurnee, Illinois, USA † Victor Technologies LLC, Bloomington, Indiana, USA

Review of Progress in Quantitative NDE

Denver, CO, USA July 19, 2012

2 July 19, 2012

Outline

• Background

• Demonstrations:

– Multiparameter Regression Models

– Integration of Physics-based Models

– Hierarchical Models in POD (MAPOD) Evaluation

• Discussion:

– Software Tools

– Challenges / Future Work

3 July 19, 2012

What is Probability of Detection (POD) and Model Assisted POD (MAPOD)?

• Probability of detection (POD) of a certain discontinuity as a function of some size metric given a defined inspection technique and target population.

• We define “MAPOD” as the collection of approaches that use models of inspections as some portion of the inputs that are processed to yield an estimate of POD.

4 5 6 7 8 9 2 3 4 5 6 7 8 9

4

5

6

7

8

9

2

3

4

5

6

7

8

9

m h 1 8 2 3

E X A M P L E 1 â v s a . x l s

0 . 6

0 . 8

resp

onse

(arb

itrar

y un

its)

discontinuity size (arbitrary units)

noise in the absence of

discontinuity

data

decision threshold

FALSE CALLSβ0

β1

ε

εββ ++= aa lnln 10

^

11

0 and )ln(

whereβδσ

ββ

µ =−

= thy

−

Φ−=σ

µaaPOD ln1)(

:Φ cumulative normal distribution function

Evaluating Reliability Using Simulated and Empirical Data: • To mitigate cost of validation study, one must better assess the

critical sources of error and variation on reliability performance • Hoppe [2009] presented historical case highlighting benefit of improving

the measurement model through including crack length and depth in fit •

•

• Physics-based models provide opportunity for reducing experimental samples and cost

Mitigating Cost of POD Study through Improved Model Accuracy

Objective: Explore Case Studies to Assess Impact of Measurement Model Quality on POD Estimation and Sample Number

εββ ++= 110ˆ aa

εβββ +++= 22110ˆ aaa

Increase Model Accuracy

Reduces

Residuals in Model Fit

Improves Bounds on Parameter Estimates (POD)

Impacts Experimental

Sampling Requirements ( ) εββ ++= 2110 ,ˆ aafa

Model-Assisted POD Model Building Process [MIL-HNBK 1823A, Appendix H (2009)]

Uncertainty Propagation

Model Error

Input Parameter Variability

(Distributions)

Stochastic Models

Model ‘Calibration‘

Revise Model Estimates Using Bayesian Methods

Confidence Bounds (Limited Samples)

Objective: Leverage Bayesian Method in MAPOD Evaluation

Assess Key Factors (Joint PDFs) using Bayesian Methods

Approach: Integrate Modeling and Simulations with Empirical Studies

• Bayesian methods are necessary to incorporate empirical data with NDE models (prior information)

• Application of Bayes’ rule: • : prior probability of θ • : conditional probability (likelihood)

of new evidence (data), x , given θ • : posterior probability of θ given

new evidence x • Posterior distribution can be evaluated, providing

a refinement to the original prior distribution through numerical methods such as Markov Chain Monte Carlo (MCMC) simulation

)()()(

)(x

xx

PPP

Pθθ

θ =

)(θP)( θxP

)( xθP

Models and Simulation

Experiments

posterior

prior

θ

likelihood

pdf

Integration (Bayesian Approach)

50 100 150 200

50

100

150

-200

0

200

400

600

-0.35 -0.3 -0.25 -0.2 -0.15 -0.1 -0.05 0 0.05-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

dR (Ω)

dX ( Ω

)

exp: l=1.0 mmexp: l=1.5 mmexp: l=2.0 mmexp: l=2.5 mmVIC: l=1.0 mmVIC: l=1.5 mmVIC: l=2.0 mmVIC: l=2.5 mm

Bayesian Methods for in POD / MAPOD Evaluation

General References: 1. Gelman, A., Carlin, J.B., Stern, H.S., Rubin, D.B., Bayesian Data Analysis, 2003. 2. Lunn, D.; Spiegelhalter, D.; Thomas, A.; Best, N. (2009). "The BUGS project:

Evolution, critique and future directions". Statistics in Medicine 28: 3049–3067 3. Christensen, R., W. Johnson, and A. Branscum, Bayesian Ideas and Data Analysis:

An Introduction for Scientists and Statisticians,” CRC Press, 2010. NDE References: 1. Meeker, W.Q. and L.A. Escobar, "Introduction to the Use of Bayesian Methods for

Reliability Data," Statistical Methods for Reliability Data, Wiley, 1998, pp. 343-368. 2. Leemans, D.V, and Forsyth, D., “Bayesian Approaches to Using Field Test Data in

Determining the Probability of Detection,” Materials Evaluation, 2004. – early demonstration of Bayesian methods in hit-miss POD evaluation

3. Annis, C., http://www.statisticalengineering.com/ – discusses Bayesian updating, confidence vs. credible bounds,

4. Wang, Y., “Advanced statistical methods for analysis of NDE data ,” Dissertation, 2006. (Advisor: W.Q. Meeker.)

5. Thompson, R.B., A Bayesian Approach to the Inversion of NDE and SHM Data”, Rev. Prog. Quant. Nondestr. Eval, Vol 29, 2010, pp. 679-686.

Bayesian Methods in MAPOD/POD Evaluation – Prior Work

Bayesian Methods in MAPOD/POD Evaluation - Recent Work

NDE References (cont.): 6. Li, Meeker and Hovey, “Joint Estimation of NDE Inspection Capability and Flaw-

size Distribution for In-service Aircraft Inspections,” RNDE, 2012. – Evaluate noise interference model POD and crack distribution

7. Kanzler, D., Muller, C., Pitkanen, J., Ewert, U.,“Bayesian Approach for the Evaluation of the Reliability of Non-Destructive Testing Methods,” WCNDT 2012.

Related Recent Work for AFRL: 1. Statistical Analysis of Hit/Miss Data using Bayes Factors (Model Selection)

[Knopp and Zeng, 2012, submitted for publication] 2. Application of Gaussian Process Models for Quantifying the Accuracy and

Capability of Nondestructive Sensing Methods for Damage Characterization – Victor Technologies Phase I SBIR [Aldrin et al., 2012; ASNT Fall conference ]

Objectives of Presentation: • Present Bayesian methods for POD evaluation with NDE measurement

models of increasing complexity: – Multivariate models – Physics-based models

• Explore Bayesian methods for stochastic model parameter estimation

Demo 1: Eddy Current Inspection of Surface-breaking Cracks in Ti-6Al-4V

Identify Controlling Factors: • Crack Characteristics

– Length and Depth (aspect ratio) – Width (cracks, EDM notches) – Stress state across crack face (closure) – Crack morphology

• Material Properties – Conductivity – Material noise (anisotropy, grain structure) – Surface condition (roughness, residual stress, coldwork)

• Part Geometry (assume locally flat) • Probe (frequency fixed at 2.0 MHz)

– liftoff – tilt – dimensions, windings (probe to probe variability)

• Scan resolution (fixed) • DAQ hardware (Agilent Impedance Analyzer, Nortec 19eII) • Calibration process - isolate liftoff direction in response

- set full screen height for known notch (0.10“) • Human data interpretation - use quantitative metrics in evaluation

x y z

50 100 150 200

50

100

150

-200

0

200

400

600

1.0 mm crack

response in Ti-6Al-4V

-0.35 -0.3 -0.25 -0.2 -0.15 -0.1 -0.05 0 0.05-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

dR (Ω)

dX ( Ω

)


Input Parameters in Study: • a1 = Crack length

– primary variable for POD • a2 = Crack depth (width)

– dependent variable on a1 crack length

– relationship define by function a2(a1) = a4 * a1

– a4 is the aspect ratio and defined as an random variable

• a3 = Liftoff – uncontrolled parameter

during study – estimation of liftoff

could improve POD performance (to verify)

0.06 0.08 0.1 0.120

200

400

600

liftoff2 2.5 3

0

200

400

600

aspect ratio

0 1 2 30

200

400

600

length0 0.5 1 1.5

0

200

400

600

800

width

Simulated POD Studies: EC Inspection of Cracks in Ti-6Al-4V

EC Response wrt Crack Length (variation for a2 and a3 included)

• Example: Eddy Current Inspection of Surface-breaking Cracks in Ti-6Al-4V

• Case Study for Multivariate POD Model Evaluation (ahat-vs-a1-and-a2):

• POD Analysis Implemented in POD Toolkit [TRI/Austin]: • R link to WinBUGS used for Bayesian Analysis

• Compare different model fits and confidence bounds approaches – Analysis 0: Neglect a2 – Analysis 1: Regression (MLE) fit,

Delta method for confidence bds. – Analysis 2: Regression (MLE) fit,

Monte Carlo for confidence bds. – Analysis 2: Bayesian (MCMC) fit

for model and confidence bds. – Good agreement between three

multivariate POD model fits

Analysis 0 Analysis 1 Analysis 2 Analysis 3 survreg() survreg() glm()/MC BayesMCMC

neglect a2 use a1, a2 use a1, a2 use a1, a2 B0 -0.05780 -0.05986 -0.05986 -0.05983 B1 5.39532 2.77503 2.77503 2.77668 B2 0.00000 6.65178 6.65178 6.64630

Delta 0.02538 0.02001 0.02001 0.02061 Threshold 0.10000 0.10000 0.10000 0.10000

var11 0.00003 0.00002 0.00002 0.00002 var22 0.00912 0.11971 0.12346 0.12687 var33 0.00000 0.73494 0.75790 0.78247 a50 0.02925 0.03204 0.03204 0.03203 a90 0.03529 0.03719 0.03721 0.03732

a90/95 0.03616 0.03720 0.03837 0.03733

εβββ +++= 22110ˆ aaa

Bayesian Methods for POD / MAPOD Evaluation (1)

-0.35 -0.3 -0.25 -0.2 -0.15 -0.1 -0.05 0 0.05-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

dR (Ω)

dX ( Ω

)


xyz

),0(~ 2εσε N

Bayesian Methods - Software

Bayesian Analysis Software Options: • OpenBUGS (WinBUGS): Comprehensive tool for MCMC simulation

• Define ‘model’ as separate file

Bayesian Methods - Software

Bayesian Analysis Software Options: • OpenBUGS (WinBUGS) http://www.openbugs.info

• Comprehensive tool for MCMC simulation • Define ‘model’ as separate file • Guide to Running WinBugs and OpenBugs from R

• http://www.stat.columbia.edu/~gelman/bugsR/ • Interface with R / Matlab for Bayesian POD Evaluation

• R: function (x1, y1, a.hat.decision, model.file, winbugs.path) • Very difficult to embed numerical model results in ‘model’

• Matlab - DRAM - Delayed Rejection Adaptive Metropolis [http://www.helsinki.fi/~mjlaine/dram/] • Provides means to embed Matlab function calls in MCMC • Facilitates integration of physics-based (surrogate) models • Not as general and robust as OpenBUGS

• Matlab - Statistics Toolbox • Pymc: Python Toolkit for Markov Chain Monte Carlo sampling

Demo 2: EC Inspection of Fastener Sites for Fatigue Cracks

• C-5 Wing Splice Fatigue Crack Specimens: – Two layer specimens are 14" long and 2" wide, – 0.156" top layer, 0.100" bottom layer – 90% fasteners were titanium, 10% fasteners were steel – Fatigue cracks position at 6 and 12 o’clock positions – Crack length ranged from 0.027" – 0.169“ (2nd layer) – vary: location of cracks – at both 1st and 2nd layer

• AFRL/UDRI Acquired Data (Hughes, Dukate, Martin)

A1-16C0

100 200 300 400 500 600 700

20406080

100120 0

2

4

A1-16C0

100 200 300 400 500 600 700

20406080

100120

-2

0

2

0.110" 0.107"0.110" 0.107"

x

z

ba

2nd layer – corner crack

zb

a

1st layer – corner crack

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.05

0

0.05

0.1

0.15

a1 (in.)

a hat

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.1

-0.05

0

0.05

0.1

a1 (in.)

resi

dual

s

• Example: Eddy Current Inspection of Cracks at Fastener Sites • Case Study for Physics-based Model Evaluation:

• • where f () is a function call for a physics-based model (i.e. VIC-3D)

• Bayesian POD Analysis Performed in Matlab + R: • MCMC library in Matlab used for Bayesian Analysis • Matlab Provides Option for Integration of Model Function in Bayesian Fit

• Compare Ahat-vs-a fit (MLE, Wald bounds) and a Physics-based Model Fit


( ) εββ ++= 2110 ,ˆ aafa

zb

a

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.05

0

0.05

0.1

0.15

a1 (in.)

a hat

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.04

-0.02

0

0.02

0.04

a1 (in.)

resi

dual

s

Ahat-vs-a fit (MLE, Wald bounds) Physics-based model fit (Bayes/MCMC)

Physics-based Model Fit Provides

Better Match and

Residuals Are Reduced

),0(~ 2εσε N

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.180

0.5

1

a1 (in.)

PO

D

POD: physics based model, Bayes/MCMC

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.05

0

0.05

0.1

0.15

a1 (in.)

a hat

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.05

0

0.05

0.1

0.15

a1 (in.)

a hat

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.04

-0.02

0

0.02

0.04

a1 (in.)

resi

dual

s


• • where f () is a function call for a physics-based model (i.e. VIC-3D)

• Bayesian POD Analysis Performed in Matlab + R: • MCMC library in Matlab used for Bayesian Analysis • Matlab Provides Option for Integration of Model Function in Bayesian Fit

• Compare Ahat-vs-a fit (MLE, Wald bounds) and a Physics-based Model Fit


( ) εββ ++= 2110 ,ˆ aafa

zb

a

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.180

0.5

1

a1 (in.)

PO

D

Ahat-vs-a fit (MLE, Wald bounds) Physics-based model fit (Bayes/MCMC)

Result is More Accurate Representation of the Data in

the POD Model Fit

),0(~ 2εσε N

0.8 0.85 0.9 0.95 1 1.05 1.1 1.15 1.2 1.250

2000

4000

6000

1/theta3 (aspect ratio: a/b)

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.05

0

0.05

0.1

0.15

a1 (in.)a ha

t


• • where f () is a function call for a physics-based model • β0 , β1 = model calibration parameters • β2 = random variable associated with crack aspect ratio (b/a) • β3 = random variable associated with liftoff variation

• Results: Fit POD Model and Estimate of Variation in Aspect Ratio [use non-informative priors]

• Issues with ‘Naïve’ Approach: • Need true estimate of variance for

crack aspect ratio random variable → Use hierarchical models

• Address correlated / confounded parameters in estimation problem → Use informative priors and

constraints from expert opinion


( ) εββββ ++= 32110 ,;ˆ afa

zb

a),0(~ 2εσε N

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.160

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

a1 (in.)

a hat

layer 1layer 2

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.05

0

0.05

0.1

0.15

a1 (in.)

a hat

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18-0.1

-0.05

0

0.05

0.1

a1 (in.)

resi

dual

s

• Example: Eddy Current Inspection of Cracks at Fastener Sites • Challenge: Address non-constant variance wrt flaw size • Case Study for Physics-based Model Evaluation:

• • where f () is a function call for a physics-based model • β0 , β1 = model calibration parameters • β2 = random variable associated with crack aspect ratio (b/a)

Hierarchical Models for Estimating Variance of a Random Variable

( ) εβββ ++= 2110 ;ˆ afa

zb

a

),0(~ 2εσε N

experimental results simulated examples

β0 = 0.0 β1 = 1.0

µ_β2 = 0.75 σ_β2 = 0.12 σ_ε = 0.0

1st

2nd

0 0.05 0.1 0.15 0.2-0.1

0

0.1

0.2

0.3

a1 (in.)

a hat

• Example: Eddy Current Inspection of Cracks at Fastener Sites • Challenge: Address non-constant variance wrt flaw size • Hierarchical NDE Measurement Models:

•

• where f () is a function call for a physics-based model • β0 , β1 = model calibration parameters • η = random variable (varying-slope model) • σ2

η = variance in slope parameter • β2 = random variable associated

with crack aspect ratio (b/a) • σ2

η = variance in slope parameter

• Simple Test Case: Fit data from model with varying slope >> noise.


( ) aaa ˆ110ˆ εηββ +++=

zb

a

),0(~ 2ˆˆ aa N σε

1st 2nd

( ) aafa ˆ2110 ;ˆ εβββ ++=

ηεη = ),0(~ 2ηη σε N ),(~ 2

2 22 ββ σµβ N);,0(~ 2ˆˆ aa N σε

physics-based model statistical model

• A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin, Bayesian Data Analysis, 2003. • A. Gelman and J. Hill, Data Analysis Using Regression and Multilevel/Hierarchical Models, 2007.

• Simple Test Case: Fit data from model with varying slope >> noise. • Hierarchical NDE Measurement Models: • Results: Ns = 100


( ) aaa ˆ110ˆ εηββ +++= ),0(~ 2ˆˆ aa N σε

ηεη = ),0(~ 2ηη σε N

Ns = 100 β0 β1 ση σε True Value 0.0000 1.0000 0.3000 0.00100 WinBUGS

Mean -0.00024 1.0312 0.2580 0.00139 95% Credible Bds (-0.00119,0.00364) (0.9848,1.0810) (0.2286,0.2923) (0.00103,0.00210)

Matlab Mean -0.00027 1.0361 0.2768 0.01166

95% Credible Bds (-0.00418,0.00071) (0.9862,1.0863) (0.2593,0.2974) (0.01012,0.01346)

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.20

0.05

0.1

0.15

0.2

0.25

a1 (in.)

a hat

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2-0.1

0

0.1

0.2

0.3

a1 (in.)

a hat

• Simple Test Case: Fit data from model with varying slope >> noise. • Hierarchical NDE Measurement Models: • Results: Ns = 1000





Mean -0.00011 1.0145 0.2966 0.00113 95% Credible Bds (-0.00052,0.00027) (0.9952,1.0330) (0.2853,0.3084) (0.00101,0.00139)

• Simple Test Case: Fit data from model with varying slope > noise. • Hierarchical NDE Measurement Models: • Results: Ns = 100

• Future Work: Perform Bayesian Evaluations with Physics-based Models Addressing Random Variable Parameter Estimation (in Matlab)

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.20

0.05

0.1

0.15

0.2

a1 (in.)

a hat





Mean 0.00193 0.9716 0.1014 0.00545 95% Credible Bds (0.00007,0.00382) (0.9371,1.007) (0.0795,0.1248) (0.00432,0.00677)

Bayesian Methods - Challenges Uncertainty Quantification (UQ) community developing Bayesian framework

for the use of computational models with observational data • SAMSI program on UQ (2012): • SIAM UQ conference 2012: http://www.siam.org/meetings/uq12/

Key Insight / Research Directions: • 1) Must include model discrepancy and not treat it as random error.

• Calibrating (inverting, tuning) a wrong model gives parameter estimates that are wrong (not equal to their true physical values) [O’Hagan, 2012]

• Gaussian Process (GP) models typically used to fit model discrepancy [Kennedy/O’Hagan 2002].

• 2) Use of prior information in Bayesian framework can greatly help. • To learn about model parameters in the presence of discrepancy, better

prior information is needed [Bayarri, 2012] • Elicitation of expert opinion is an active research topic [O’Hagan, 2012]

• 3) Should leverage model form uncertainty (assessment) approaches. • To identify best models and address limitations cited by UQ community

[Grandhi et al, Wright State University]

24 July 19, 2012

Acknowledgements

• This work was partially supported by the U.S. Air Force Research Laboratory under UTC Prime Contract, FA8650-10-D-5210

• Charles Annis, Statistical Engineering

• David Forsyth, TRI/Austin

• Eric Lindgren, AFRL

Bayesian POD Evaluation Examples and Code Coming Soon www.computationaltools.com

Documents

Bayesian Methods in Probability of Detection …computationaltools.com/Papers/Aldrin Knopp - 2012 - QNDE -Bayes POD...Bayesian Methods in Probability of Detection Estimation and Model