Confidence Level

8/13/2019 Confidence Level

1/31

CONFIDENCE LEVELS

We warn the reader that there is no universal convention for theterm confidence level

(The Review of Particles Properties, 1986)

1


2/31

Confidence levels

Part of descriptive statistics

Goal of an experiment: measure a theoretical parameter a

Quoting the result usually involves giving some interval [a,b]:

! Expresses probability that the true value is in this interval

! Allows information consumer to draw conclusions from the

result

! Set upper / lower limit on the true value of a parameter

2


3/31

Confidence level definition

Let some measured quantity bedistributed according to some p.d.f.P(x), we can determine the probability

that x lies within some interval, withsome confidence C

Prob(x x x+) =

x+

x

P(x)dx= C

We say:x lies in the interval [x- , x+] with confidenceC

Note: C is a probability according to the frequency limit3


4/31

Gaussian confidence intervals

If P(x) = Gaussian distribution with mean "and variance #2:some examples of confidence intervals:

x = 1 C= 68%x = 2 C = 95.4%

x = 1.64 C= 90%

x = 1.96 C= 95%

4


5/31

Types of confidence intervals

3 conventional ways to choose an interval around the center:

1. Symmetric interval: x-and x+equidistant from the mean2. Shortest interval: minimizes (x+- x-)

3. Central interval:

x

P(x) dx=

+x+

P(x) dx=1C

2

Prob(x x x+) =

x+

x

P(x)dx= C

For Gaussian (and any symmetric distribution):3 definitions are equivalent

5


6/31

One-tailed

confidence intervals

So far, we considered two-tailed intervals.

Useful as well: one-tailed limits

! Upper limit: x lies below x+at confidence level C:

! Lower Limit: x lies above x-at confidence level C:

x+

P(x) dx= C

+x

P(x) dx= C

6


7/31

Confidence intervals

in estimationIn a measurement two things involved:

! Physical parameter(s) X: mass, lifetime, ...

! Measurement of this parameter x

Given X,there is a p.d.f. for measuring x(resolution, QM,...)

But what you want to know:

! Given measurement x!x, what can I say about X ?

7


8/31

Can I say that

X lies within [x-!x, x+!x] with 68% probability?

Not in the sense of a frequency:X is not a random variable!!!

8


9/31


10/31

Possible experimental valuesx

parameter

! x2!!"# !

2!x"

x1!!"# !

1!x"

x1!!

0" x2!!0"

D(")

!0

Confidence belt Construction

10

Neyman Construction:

1. For each $find D($) with

probability C

2. Confidence interval includes

all $with observation at x0

NOTE: this is not a statementabout the probability of "butabout the interval!


11/31

Lower / Upper limits using the

confidence beltGiven measurement x0find X-and X+from confidence belt:

X+ upper limit at C.L. 1-%:

i.e. if X &X+: Probability to measure x 'x0is less than %

X-lower limit at C.L. 1-%:

i.e. if X ' X-: Probability to measure x &x0is less than %

11

+x0

P(x|X+) dx= 1

x0

P(x|X) dx= 1


12/31

12


13/31

Gaussian confidence levels

P(x|X): Gaussian with standard deviation #

Apply method to determine a 90% C.L. interval for X given ameasurement x0:

Equation for X-: requires that x0lies some number of standard deviations

above X-, which is the same as saying that X-lies the same number of #

below x0

(k: depends on the desired C.L.)

Confidence belt limited by two straight lines

13

x0

12

e (xX+)

2

22 dx = 0.05 =

+x0

12

e (xX)

2

22 dx

X = x0 k


14/31

Confidence levels near

a physical boundaryAssume a mass measurement with resolution 20 MeV

The true mass is 10 MeV

Use a 2#(95.4%) C.I. to quote the result: x 40 MeV

Consider cases:

(2.3 % probability that measurement > 50 MeV

(Measurement in range 40-50 MeV: limits will be true

(x = 0.2 40 MeV: correct lower limit to 0 and OK

(BUT what if x = - 50 MeV 40 MeV : X < -10 MeV @ 95% C.L. !!!???It is strictly speaking correct but ridiculous!

Only means of escape: BAYES TO THE RESCUE!

14


15/31

Bayesian Confidence Intervals

Bayes theorem:

P(theory): assume all positive masses equally likely

Now apply Bayes theorem:

For x = - 50 MeV 20 MeV: Denominator is one-sided 2.5 #Gaussian tail: 0.0062

Look for 90 % C.L. upper limit: Integral of numerator must be ~0.0006: 3.24 #

Results: mass < -50 MeV + 3.24 * 20 MeV = 15 MeV @ 90 % C.L.

15

P(theory|data) =P(data|theory)P(theory)

P(data)

P(true mass) = 0, m


16/31

Bayesian Confidence Interval:

ExampleMass measurementMeasurement x = - 50 MeV 20 MeV

Prior : assume all positive masses equally likely:

Denominator is one-sided 2.5 #Gaussian tail: 0.0062

Integral of numerator must be ~0.0006: 3.24 #

Result: mass < -50 MeV + 3.24 * 20 MeV = 15 MeV @ 90 % C.L.

For comparison: Frequentist 90 % Upper Limit: - 10 MeV

16

P(true mass) = 0, m


17/31

Confidence Intervals for

Discrete distributions Physical parameter: real

Measured variable discrete:

number of counts (Poisson)

Number of successes (Binomial)

May be unable to select region

with exact confidence C (e.g.90%):

Play safe: gives overcoverage

17


18/31

Binomial Confidence Intervals

Observed value: Number of successes (out of N trials) - DISCRETE

True value: single trial probability R - CONTINUOUS

If m successes found in N trials:

! Limits on the individual probability p: Find p+and p-such that (Using 95%central limit):

(CLOPPER-PEARSON COEFFICIENTS)

! In words: Were p&p+: Probability to get m counts or less is '2.5%

Were p'p-: Probability to get m counts or more is '2.5%

18

r=m+1

P(r;p+, N) = 0.975m1

r=0

P(r;p, N) = 0.975


19/31

19

From the archives:


20/31

Poisson

confidence intervalsn events observed from Poisson process of unknown mean "

The 90 % Poisson upper limit is the value "+such that:

i.e.: if true value of "is really "+probability for getting a number ofcounts n or smaller is 10%

Similarly, the 90 % Poisson lower limit is the value "-such that:

20

r=n+1

P(r; +) = 0.90, or equivalently

n

r=0

P(r; +) = 0.10

n1

r=0

P(r; ) = 0.90, or equivalently

r=n

P(r; ) = 0.10


21/31

Some Poisson Limits

21


22/31

Confidence intervals using

likelihood functionFor non-Gaussian estimators, still possible to determine confidenceinterval with a simple approximate technique using the likelihoodfunction, or equivalently the !2function

For a ML estimator for a parameter aIn the large sampleapproximation:

! The p.d.f. g(,a) becomes Gaussian:

! The likelihood function itself becomes Gaussian with the same #

Can extract #from likelihood scan!

22

g(a; a) = 1

2aexp

(a a)2

22a

L(a) = Lmaxexp

(a a)2

22a


23/31

Prescription for setting confidence

intervals using the likelihood1. Extract #from log-likelihood scan using:

2. Use fact that g(,a) is Gaussian to set confidence intervals:! e.g.: [c, d] = [ - #, + #] : 68% central confidence interval

Can be shown that this procedure can be used even if thelikelihood function is not Gaussian

! Exact only in the large sample limit

! May need to use asymmetric intervals around

23

logL(aNa) =logLmax N2

2


24/31

Example: lifetime fit

24

0 0.5 1 1.5 2 2.5 30

0.2

0.4

0.6

0.8

1h_tau

Entries 5

Mean 1.201

RMS 0.5881

0.5 1 1.5 2 2.5 3 3.5

-8

-7.5

-7

-6.5

-6

-5.5

-5

!"

-!#"

-!"

+!#"

+!"

0 0.5 1 1.5 2 2.5 30

0.5

1

1.5

2

2.5

3

3.5

4

h_tau

Entries 50

Mean 0.815

RMS 0.6943

0.8 0.9 1 1.1 1.2 1.3 1.4

-54

-53.5

-53

-52.5

-52

-51.5 !"

-!#"

-!"

+!#"

+!"

0 0.5 1 1.5 2 2.5 30

2

4

6

8

10

12

14

16

18

20

22h_tau

Entries 500

Mean 0.8455

RMS 0.7131

0.94 0.96 0.98 1 1.02 1.04 1.06 1.08 1.1 1.12 1.14

-519

18.5

-518

17.5

-517

16.5!"

-!#"

-!"

+!#"

+!"

Using estimator : = 1n

n

i=1

ti

logLmax

logLmax- 1/2

True )=1


25/31

Several variables:

Confidence regions1-D case: look for single parameter "

constructed interval [a,b] which contains true value with someprobability C

N-D case: Look for parameters "=("1, ..., "n)

In general, cannot find ai,biso that ai


26/31

Multidimensional Confidence

regionsUse the properties of the likelihood function in the large sample limit:

1. The ML estimator p.d.f. is Gaussian:

2. The Likelihood function is Gaussian with the same V:

3. If p.d.f. described by n-dim. Gaussian: Q(, a) distributed according to !2distribution with n d.o.f. :

26

g(a;a) =(2)n|V|1/2

exp2Q(a;a)

, Q(a;a) = (a a)TV1(a a)

V1 : inverse covariance matrix

L(a) =Lmaxexp

1

2Q(a, a)

,

Prob(Q(a, a) Q) =

Q0

f2(z, n) 2 distr. f o r n d.o.f.

dz


27/31

Constructing n-dimensional

Confidence regionPrescription for findingconfidence region at @ C.L. 1-#:

1. Determine value of Q#(tablesor numerically) so that:

2. Find contour at which

1. *logL = - Q#/ 2

27

Q0

f2(z, n)dz= 1


28/31

Q#values for some values of coverage and numbers of

parameters:

28


29/31

Upper limit on the mean of

Poisson variable with backgroundObserved number of events is sum of signal + background:n = ns+ nb(expectation values +s, +b)

Goal: construct upper limit for +s

n is Poisson distributed with mean +s+ +b:

ML estimate for +s: n - +b

Lower limit:

Upper limit:

Upper and lower limits are related to the limits without background

Problem if number of counts smaller than expected background, upper limit is < 0!

BAYES TO THE RESCUE!

29

= P(s obss ;

los ) =

nnobs

(los + b)n

n! e

(los +b)

= P(s obss ;

ups ) =

nnobs

(ups + b)n

n! e

(ups +b)

lo

s = lo

s (no background)

b

ups =

ups (no background) b


30/31

Upper limit with background:

Bayesian approach Bayesian approach (flat prior)

Solution

Reduces to classical solution for no background

30


31/31

Upper limit with background

31

Bayesian solution

Documents

Confidence Level