Goodness of fit, confidence intervals and limits Jorge Andre Swieca School Campos do Jordo, January,2003 fourth lecture Slide 2 References Statistical Data Analysis, G. Cowan, Oxford, 1998 Statistics, A guide to the Use of Statistical Methods in the Physical Sciences, R. Barlow, J. Wiley & Sons, 1989; Particle Data Group (PDG) Review of Particle Physics, 2002 electronic edition. Data Analysis, Statistical and Computational Methods for Scientists and Engineers, S. Brandt, Third Edition, Springer, 1999 Slide 3 Limits Tens, como Hamlet, o pavor do desconhecido? Mas o que conhecido? O que que tu conheces, Para que chames desconhecido a qualquer coisa em especial? lvaro de Campos (Fernando Pessoa) Se tm a verdade, guardem-na! Lisbon Revisited, lvaro de Campos Slide 4 Statistical tests How well the data stand in agreement with given predicted probabilities hypothesis. null hypothesis H 0 alternative function of measured variables: test statistics error first kind significance level power = error second kind power to discriminate against H 1 Slide 5 Neyman-Pearson lemma Where to place t cut ? H 0 signal H 1 background 1-D: efficiency (and purity) m-D: def. of acceptance region is not obvious Neyman-Pearson lemma: highest power (highest signal purity) for a given significance level region of t-space such that determined by the desired efficiency Slide 6 Goodness of fit how well a given null hypothesis H 0 is compatible with the observed data (no reference to other alternative hypothesis) coins: N tosses, n h, n t = N - n h coin fair? H and T equal? test statistic: n h binomial distribution, p=0.5 N=20, n h =17 E[n h ]=Np=10 0 1 2 3 17 18 19 20 10 Slide 7 Goodness of fit P=0.0026 P-value: probability P, under H 0, obtain a result as compatible of less with H 0 than the one actually observed. P-value is a random variable, is a constant specified before carrying out the test Bayesian statistics: use the Bayes theorem to assign a probability to H 0 (specify the prior probability) P value is often interpreted incorrectly as a prob. to H 0 P-value: fraction of times on would obtain data as compatible with H 0 or less so if the experiment (20 coin tosses) were repeated under similar circunstances Slide 8 Goodness of fit Easy to identify the region of values of t with equal or less degree of compatibility with the hypothesis than the observed value (alternate hypothesis: p 0.5) optional stopping problem Slide 9 Significance of an observed signal Whether a discrepancy between data and expectation is sufficiently significant to merit a claim for a new discovery signal event n s, Poisson variable S background event n b, Poisson variable b prob. to observe n events: experiment: n obs events, quantify our degree of confidence in the discovery of a new effect ( S 0) How likely is to find n obs events or more from background alone? Slide 10 Significance of an observed signal Ex: expect b =0.5, n obs = 5 P(n>n obs )=1.7x10 -4 this is not the prob. of the hypothesis S =0 ! this is the prob., under the hypothesis S =0, of obtaining as many events as observed or more. Slide 11 Significance of an observed signal How to report the measurement? estimate of : misleading: only two std. deviations from zero impression that S is not very incompatible with zero yes:prob. that a Poisson variable of mean b will fluctuate up to n obs or higher no:prob. that a variable with mean n obs will fluctuate down to b or lower Slide 12 Pearsons test histogram of x with N bins niinii construct a statistic which reflects the level of agreement between observed and expected histograms data aprox. gaussian, Poisson distributed with follow a distribution for N degrees of freedom regardless of the distribution of x distribution free larger larger discrepancy between data and the hypothesis Slide 13 Pearsons test (rule of thumb for a good fit) Slide 14 Pearsons test Slide 15 Before Poisson variable with Set n tot = fixed n i dist. as multinomial with prob. Not testing the total number of expected and observed Events, but only the distribution of x. large number on entries in each bin p i known Follows a distribution for N-1 degrees of freedom In general, if m parameters estimated from data, n d = N - m Slide 16 ML: estimator for Standard deviation as stat. error n observations of x, hypothesis p.d.f f(x;) analytic method RCF bound Monte Carlo graphical standard deviation measurement repeated estimates each based on n obs.: estimator dist. centered around true value and with true estimated by and Most practical estimators: becomes approx. Gaussian in the large sample limit. Slide 17 Classical confidence intervals n obs. of x, evaluate an estimator for a param. obtained and its p.d.f. (for a given unknown) prob. prob. Slide 18 Classical confidence intervals prob. for estimator to be inside the belt regardless of monotonic incresing functions of Slide 19 Classical confidence intervals Usually: central confidence interval a: hypothetical value of for which a fraction of the repeated estimt. would be higher than the obtain. Slide 20 Classical confidence intervals Relationship between a conf. interval and a test of goodness of fit: test the hypothesys using having equal or less agreement than the result obtained P-value = (random variable) and = a is specified Confidence interval: is specified first, a is a random quantity depending on the data Slide 21 Classical confidence intervals Many experiments: the interval would include the true value in It does not mean that the probability that the true value of is in the fixed interval is Frequency interpretation: is not a random variable, but the interval fluctuates since it is constructed from data. Slide 22 Gaussian distributed Simple and very important application Central limit theorem: any estimator linear function of sum of random variables becomes Gaussian in the large sample limit. known, experiment resulted in Slide 23 Gaussian distributed Slide 24 Choose quantile 10.682710.8413 20.954420.9772 30.997330.9987 Choose confidence level 0.901.6450.901.282 0.951.9600.951.645 0.992.5760.992.326.