Click here to load reader

G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics Meeting CERN, 15 October, 2009 Glen Cowan Physics Department

  • View
    214

  • Download
    0

Embed Size (px)

Text of G. Cowan RHUL Physics page 1 Status of search procedures for ATLAS ATLAS-CMS Joint Statistics...

  • G. CowanRHUL Physics

    page *Status of search procedures for ATLASATLAS-CMS Joint Statistics MeetingCERN, 15 October, 2009Glen CowanPhysics DepartmentRoyal Holloway, University of [email protected]/~cowan

    Comment on use of LR for limits

  • G. Cowan / RHULpage *IntroductionThis talk describes (part of) the view from ATLAS with emphasis on searches using profile likelihood-based techniques;using as an example the combination of Higgs channels described in Expected Performance of the ATLAS Experiment: Detector, Trigger and Physics, arXiv:0901.0512, CERN-OPEN-2008-20.Other methods are also being explored and the view presentedhere should not be taken as a final word on methodology.Some issues are:Basic formalism, definition of significanceMethod used for setting limitsLook-elsewhere effect

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Search formalism Define a test variable whose distribution is sensitive to whetherhypothesis is background-only or signal + background.E.g. count n events in signal region:events foundexpected signalexpected backgroundstrength parameter m = s s/ ss,nominal

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Search formalism with multiple bins (channels)Bin i of a given channel has ni events, expectation value isExpected signal and background are:m is global strength parameter, common to all channels.m = 0 means background only, m = 1 is nominal signal hypothesis.btot, qs, qb arenuisance parameters

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Subsidiary measurements for backgroundOne may have a subsidiary measurement to constrain the background based on a control region where one expects no signal.In bin i of control histogram find mi events; expectation value iswhere the ui can be found from MC and q includes parametersrelated to the background (mainly rate, sometimes also shape).In some measurements there may be no explicit subsidiarymeasurement but the sidebands around a signal peak effectivelyplay the same role in constraining the background.

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Likelihood functionFor an individual search channel, ni ~ Poisson(msi+bi), mi ~ Poisson(ui). The likelihood is:Parameter of interestHere q represents allnuisance parametersFor multiple independent channels there is a likelihood Li(m,qi) for each. The full likelihood function is

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Systematics "built in" as long as some point in q-space = "truth". Presence of nuisance parameters leads to broadening of theprofile likelihood, reflecting the loss of information, and givesappropriately reduced discovery significance, weaker limits.

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Modified test statistic for exclusion limitsFor e.g. data generated with m = 0.8, -2 ln l(m) can come out large forIf , then data more compatible with a higher value of m. sodo not include this in critical region. For upper limit, test hypothesis that strength parameter is m.Upper limit is smallest value of m where this hypothesis can be rejected at significance level less than 1-CL.Critical region of test is region with less compatibility withthe hypothesis than the observed

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Test statistic for exclusion limitsTherefore for exclusion limits, define the test statistic to becritical regionThus distribution of modified qm corresponds to lower branchonly of U-shaped plot above.For low m, this distribution falls off more quickly than theasymptotic chi-square form and thus gives conservative limit.

    Comment on use of LR for limits

  • G. Cowan / RHULpage *p-value / significance of hypothesized mTest hypothesized by givingp-value, probability to see data with compatibility with compared to data observed:Equivalently use significance,Z, defined as equivalent numberof sigmas for a Gaussian fluctuation in one direction:

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Distribution of qmSo to find the p-value we need f(qm|m) .Method 1: generate toy MC experiments with hypothesis m, obtain at distribution of qm.OK for e.g. ~103 or 104 experiments, 95% CL limits.But for discovery usually want 5s, p-value = 2.8 10-7, so needto generate ~108 toy experiments (for every point in param. space).Method 2: Wilk's theorem says that for large enough sample,f(qm|m) ~ chi-square(1 dof)This is the approach used in the ATLAS Higgs Combination exercise; not yet validated to 5s level.If/when we are fortunate enough to see a signal, then focus MC resources on that point in parameter space.

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Example from validation exercise: ZZ(*) 4l5 level(One minus)cumulativedistributions.Band gives 68%CL limits.2222Distributions of q0 for 2, 10 fb from MC compared to 2

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Significance from qmIf we take f(qm|m) ~ c2 for 1dof, then the significance is simply:For n ~ Poisson (ms+b) with b known, testing m = 0 givesTo quantify sensitivity give e.g. expected Z under s+b hypothesis

    Comment on use of LR for limits

  • G. Cowan / RHULpage *SensitivityDiscovery:Generate data under s+b (= 1) hypothesis;Test hypothesis = 0 p-value Z.Exclusion:Generate data under background-only (= 0) hypothesis;Test hypothesis .If = 1 has p-value < 0.05 exclude mH at 95% CL.Estimate median significance (sensitivity) either from MC or by using a single data set with observed numbers set equal tothe expectation values ("Asimov" data set).

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Example of ATLAS Higgs searchCombination of Higgs search channels (ATLAS) Expected Performance of the ATLAS Experiment: Detector, Trigger and Physics, arXiv:0901.0512, CERN-OPEN-2008-20.Standard Model Higgs channels considered: H gg H WW (*) enmn H ZZ(*) 4l (l = e, m) H t+t- ll, lhNot all channels included for now; final sensitivity will improve.Used profile likelihood method for systematic uncertainties:nuisance parameers for: background rates, signal & background shapes.

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Combined discovery significanceDiscovery signficance (in colour) vs. L, mH:Approximations used here not always accurate for L < 2 fb-1 but in most cases conservative.

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Combined 95% CL exclusion limits1 - p-value of mH (in colour) vs. L, mH:

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Comment on combination softwareCurrent ATLAS Higgs combination shows median significancesObtained using median significances from each channelWhat we will need is the significance one would have from asingle (e.g. real) data sample.Requires full likelihood function, global fit software.Since summer 2008 ATLAS/CMS decision to focus joint statisticssoftware effort in RooStats (based on RooFit, ROOT). Provides facility to construct global likelihood forcombination of channels/experimentsEmphasis on retaining modularity for validation byswapping in/out different components.

    Comment on use of LR for limits

  • G. Cowan / RHULpage *The "look-elsewhere effect"Look for Higgs at many mH values -- probability of seeing a large fluctuation for some mH increased.Combined significance shown here relates to fixed mH.False discovery prob enhanced by ~ mass region explored / smFor Hgg and HWW, studied by allowing mH to float in fit:H gg

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Summary / conclusionsCurrent philosophy (ATLAS/CMS) is to encourage a variety of methods, e.g., for limits: classical (PL ratio), CLs, Bayesian,...If the results agree, it's an important check of robustness.If the results disagree, we learn something (~ Cousins)Discussion on look-elsewhere effect still ongoingDerive trials factor from e.g. MC or other considerationsFloating mass searchD0, CDF, CMS, ATLAS need to compare like with like.Need discussions on e.g. formalism for discovery,limits, combination, treatment of common systematics,

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Extra slides

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Fast Fourier Transform method to find distribution; derivesn-event distribution from that of single event with FFT.Hu and Nielson, physics/9906010Solves "5-sigma problem".Used at LEP -- systematics treated by averaging the likelihoodsby sampling new values of nuisance parameters for each simulated experiment (integrated rather than profile likelihood).An alternative (in simple cases equivalent) test variable isComment on "LEP"-style methods

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Setting limits: CLsAlternative method (from Alex Read at LEP); exclude m = 1 if where This cures the problematic case where the one excludes parameterpoint where one has no sensitivity (e.g. large mass scale)because of a downwards fluctuation of the background.But there are perhaps other ways to get around this problem,e.g., only exclude if both observed and expected p-value < a.

    Comment on use of LR for limits

  • G. Cowan / RHULpage *

    Comment on use of LR for limits

  • G. Cowan / RHULpage *Some issuesThe profile likelihood method "includes" systematics to theextent that for some point in the model's parameter space, thedifference from the "truth" is negligible.Q: What if the model is not good enough?A: Improve the model, i.e., include additional flexibility (nuisance parameters). Increased flexibility decrease in sensitivity.How to achieve optimal balance in a general way is not obvious.

    Corresponding exercise in Bayesian approach:Include nuisance parameters in model with prior probabilities -- also not obvious in many important cases, e.g., uncertainties in correlations.

    Comment on use of LR for limits

    ********