G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 1

Report from the Statistics Forum

ATLAS Analysis Strategies Workshop

CERN, 6 September, 2007

Glen CowanPhysics DepartmentRoyal Holloway, University of Londong.cowan@rhul.ac.ukwww.pp.rhul.ac.uk/~cowan

G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 2

The ATLAS Statistics Forum

Initiated at StatisticsWorkshop in January 07

Statistics Forum startedin March (Karl Jakobs)

Since then, meetings~monthly

(plus ~3 joint meetings with CMS)

G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 3

Current MembershipConveners: Eilam Gross, Glen Cowan

“Statistics Experts”: GDC, Kyle Cranmer, Tom Junk,Bill Murray, Alex Read

Tool developers: Bill Quayle, Krzysztof Ciba

Representatives from ATLAS groups:Higgs: Eilam Gross & Guillaume UnalSUSY: Dan ToveySM: Maarten BoonekampExotics: Samir FerragB-Physics: Eduard BureloTop: Yoram RozenHeavy Ions, ...

Ex officio: Ian Hinchliffe & Karl Jakobs

G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 4

Mandate

Define ATLAS Statistics Standards in contact with Physics Coordination and ATLAS groups.

Develop, validate and approve standard statistical toolsin close contact with the software group.

Represent ATLAS in discussions on statistical issues with CMS.

Discuss and make recommendations on statistical issues such as blind analyses, model independent searches, etc.

G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 5

Web pagehttps://twiki.cern.ch/twiki/bin/view/Atlas/StatisticsTools

G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 6

Hypernewshttps://hypernews.cern.ch/HyperNews/Atlas/get/physics-statistics.html

G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 7

cf. BaBar Statistics Group Hypernews

Detaileddiscussions

Heavy traffic

G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 8

Some ongoing activities

ATLAS Statistics Book

Higgs combination (different channels & ATLAS + CMS)

Software development (RooStat)

Sequential analysis

G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 9

Twiki, i.e., a Living Document (thanks, Krzysztof) Core authors assigned, detailed outline discussed+ worked examples, particularly related to systematics+ FAQ, recommendationsGoal: first draft by October (need to work hard...)

G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 10

Higgs CombinationHiggs suggested as a first major exercise for the group, but we would like to undertake similar efforts with other Physics Groups.

Many people involved; see next talk by Eilam.In particular, frequentist combination based on profilelikelihood very advanced (Kyle, Eilam, Ofer, ...)

But a brief comment on the Bayesian methods:Bayesian limits already widely used e.g. in CDF (→Tom J.).We are also investigating use of Bayes factors as a means of quantifying significance of a discovery:

Higgs

no Higgs Bayes factor B10prior odds

posterior odds

G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 11

Higgs combination based on event countsN independent channels, countni events in search regions:

Expected number of signal events:( is global parameter, = 1 for SM).

Consider a fixed Higgs mass and assume SM branching ratios Bi.

Suggested method: constrain with limit up; consider mH excluded if upper limit up < 1.0.

For discovery, compute Bayes factor for H0 : = 0 vs. H1 : = 1

Constrain expected backgroundbi with sideband measurements:

G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 12

Bayes factor ( = ratio of integrated likelihoods)

Here B10 and profile likelihood ratio close; comparisons ongoing.

B1

0

B10 at = 1 gives changein odds of SM Higgs vs. no Higgs in light of data,

i.e., measures of weight of evidence provided by datain support of one hypothesisover another.

G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 13

Statistical softwareBill Quayle, Kyle Cranmer (from Statistics Forum) + developers of

TMVA (Andreas Hoecker et al.),

RooFit (Wouter Verkerke et al.),

MadTools/StatTools (Wisconsin group), ...

(see e.g. Bill Quayle’s 15 March 07 talk)

Some issues:

ATLAS specific (private) or HEP-wide (public)?

Use FrameworkXXX or, integrate ComponentXXX into anexisting framework?

Distribution (phystat.org, bundle with ROOT?)

G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 14

RooFit/RooStats

Wouter Verkerke, Kyle Cranmer

Developments reported in ATLAS/CMS Forum, i.e., HEP-wide,not ATLAS-specific software.

E.g. Combination Toy Model (W. Verkerke -- 17 July 2007 ATLAS/CMS meeting)

‘Atlas’

Combined

Prof

ile li

kelih

ood

‘CMS’→ talk by Wouter

15

Looking at data as they are recorded and compute a new p-value or likelihood ratio test every 10 pb-1.

What is probability that p-value (or LR test) < 0.01 versus number of data looks ?

• looking 1 time at data : p(p-value < 0.01|H0) = 0.01• looking sequentially 100 times at data : p(at least 1 p-value < 0.01|H0) ~ 0.08 !

p-value

Likelihood ratio test

b = 200 fb

p-Value result versus L when H0 is true

P(at least 1 p-Value < 0.01) versus L when H0 is true

Sequential analysis (Renaud Bruneliere, 7 June and 3 July 07)

→ ‘Look-elsewhere effect in time’. Need to understand how to best quantify / deal with this!

G. CowanRHUL Physics Analysis Strategies Workshop -- Statistics Forum Report page 16

Conclusions, thoughtsStatistics Forum working since March, several projects ongoing:

Statistics BookHiggs combinationSoftware toolsSequential analysis

Want to increase interaction with Physics Groups,

If your group has a statistics issue/problem, is developing a statistical tool, etc. come tell us.(Also we will come to you...)