46
The Power of Tests for Signal Detection in High Dimensional Data Marc Ditzhaus joint work with Arnold Janssen Mathematical institute Heinrich-Heine-University Düsseldorf Wellington, December 2017 M. Ditzhaus Tests for Signal Detection Wellington, December 2017

The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

The Power of Tests for Signal Detection in High

Dimensional Data

Marc Ditzhaus

joint work with Arnold Janssen

Mathematical instituteHeinrich-Heine-University Duumlsseldorf

Wellington December 2017

M Ditzhaus Tests for Signal Detection Wellington December 2017

Motivation

genome analysis early detection of common deceases

high dimensional observation vector Xn = (Xn1 Xnn) for

each patient

healthy patient Xn behaves some noisy background

ill patient Xn contains rare and weak signals

Our interest The asymptotic power of tests which decide whether

there are any signals

Other applications astronomy cosmology disease

surveillance

M Ditzhaus Tests for Signal Detection Wellington December 2017

Motivation

genome analysis early detection of common deceases

high dimensional observation vector Xn = (Xn1 Xnn) for

each patient

healthy patient Xn behaves some noisy background

ill patient Xn contains rare and weak signals

Our interest The asymptotic power of tests which decide whether

there are any signals

Other applications astronomy cosmology disease

surveillance

M Ditzhaus Tests for Signal Detection Wellington December 2017

Astrophysics

Fig 2mdash Example artificial images Left Artificial sources only Right Artificial sources in the presenceof noise as used in the simulations emphasising that real sources close to or below the noise level becomedifficult or impossible to detect even visually

11

Source Hopkins et al (2002)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heterogeneous normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n

εn = nminusβ

(signal probability)

ϑn =radic

2r log n

(signal strength)

see Ingster (1997)

Donoho amp Jin (2004)

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heteroscedastic normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ

2) 1 le i le n

solid (mdash)

Gaussian limit

dashed (- - -)

non-Gaussian but

infinitely divisible

Cai Jeng Jin (2011)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our aims

Detectable and undetectable areas of LLRT and HC (in the

literature various results)

LLRT on the boundary (few results only normal distributions)

HC on the boundary (no results until now)

M Ditzhaus Tests for Signal Detection Wellington December 2017

General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0

probability measures microni Pni (Without loss of generality)

General testing problem

H0n P(n) =notimes

i=1

Pni against

H1n Q(n) =notimes

i=1

Qni with Qni = (1minus εni)Pni + εnimicroni

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)

dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 2: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Motivation

genome analysis early detection of common deceases

high dimensional observation vector Xn = (Xn1 Xnn) for

each patient

healthy patient Xn behaves some noisy background

ill patient Xn contains rare and weak signals

Our interest The asymptotic power of tests which decide whether

there are any signals

Other applications astronomy cosmology disease

surveillance

M Ditzhaus Tests for Signal Detection Wellington December 2017

Motivation

genome analysis early detection of common deceases

high dimensional observation vector Xn = (Xn1 Xnn) for

each patient

healthy patient Xn behaves some noisy background

ill patient Xn contains rare and weak signals

Our interest The asymptotic power of tests which decide whether

there are any signals

Other applications astronomy cosmology disease

surveillance

M Ditzhaus Tests for Signal Detection Wellington December 2017

Astrophysics

Fig 2mdash Example artificial images Left Artificial sources only Right Artificial sources in the presenceof noise as used in the simulations emphasising that real sources close to or below the noise level becomedifficult or impossible to detect even visually

11

Source Hopkins et al (2002)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heterogeneous normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n

εn = nminusβ

(signal probability)

ϑn =radic

2r log n

(signal strength)

see Ingster (1997)

Donoho amp Jin (2004)

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heteroscedastic normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ

2) 1 le i le n

solid (mdash)

Gaussian limit

dashed (- - -)

non-Gaussian but

infinitely divisible

Cai Jeng Jin (2011)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our aims

Detectable and undetectable areas of LLRT and HC (in the

literature various results)

LLRT on the boundary (few results only normal distributions)

HC on the boundary (no results until now)

M Ditzhaus Tests for Signal Detection Wellington December 2017

General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0

probability measures microni Pni (Without loss of generality)

General testing problem

H0n P(n) =notimes

i=1

Pni against

H1n Q(n) =notimes

i=1

Qni with Qni = (1minus εni)Pni + εnimicroni

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)

dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 3: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Motivation

genome analysis early detection of common deceases

high dimensional observation vector Xn = (Xn1 Xnn) for

each patient

healthy patient Xn behaves some noisy background

ill patient Xn contains rare and weak signals

Our interest The asymptotic power of tests which decide whether

there are any signals

Other applications astronomy cosmology disease

surveillance

M Ditzhaus Tests for Signal Detection Wellington December 2017

Astrophysics

Fig 2mdash Example artificial images Left Artificial sources only Right Artificial sources in the presenceof noise as used in the simulations emphasising that real sources close to or below the noise level becomedifficult or impossible to detect even visually

11

Source Hopkins et al (2002)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heterogeneous normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n

εn = nminusβ

(signal probability)

ϑn =radic

2r log n

(signal strength)

see Ingster (1997)

Donoho amp Jin (2004)

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heteroscedastic normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ

2) 1 le i le n

solid (mdash)

Gaussian limit

dashed (- - -)

non-Gaussian but

infinitely divisible

Cai Jeng Jin (2011)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our aims

Detectable and undetectable areas of LLRT and HC (in the

literature various results)

LLRT on the boundary (few results only normal distributions)

HC on the boundary (no results until now)

M Ditzhaus Tests for Signal Detection Wellington December 2017

General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0

probability measures microni Pni (Without loss of generality)

General testing problem

H0n P(n) =notimes

i=1

Pni against

H1n Q(n) =notimes

i=1

Qni with Qni = (1minus εni)Pni + εnimicroni

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)

dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 4: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Astrophysics

Fig 2mdash Example artificial images Left Artificial sources only Right Artificial sources in the presenceof noise as used in the simulations emphasising that real sources close to or below the noise level becomedifficult or impossible to detect even visually

11

Source Hopkins et al (2002)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heterogeneous normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n

εn = nminusβ

(signal probability)

ϑn =radic

2r log n

(signal strength)

see Ingster (1997)

Donoho amp Jin (2004)

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heteroscedastic normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ

2) 1 le i le n

solid (mdash)

Gaussian limit

dashed (- - -)

non-Gaussian but

infinitely divisible

Cai Jeng Jin (2011)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our aims

Detectable and undetectable areas of LLRT and HC (in the

literature various results)

LLRT on the boundary (few results only normal distributions)

HC on the boundary (no results until now)

M Ditzhaus Tests for Signal Detection Wellington December 2017

General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0

probability measures microni Pni (Without loss of generality)

General testing problem

H0n P(n) =notimes

i=1

Pni against

H1n Q(n) =notimes

i=1

Qni with Qni = (1minus εni)Pni + εnimicroni

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)

dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 5: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heterogeneous normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n

εn = nminusβ

(signal probability)

ϑn =radic

2r log n

(signal strength)

see Ingster (1997)

Donoho amp Jin (2004)

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heteroscedastic normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ

2) 1 le i le n

solid (mdash)

Gaussian limit

dashed (- - -)

non-Gaussian but

infinitely divisible

Cai Jeng Jin (2011)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our aims

Detectable and undetectable areas of LLRT and HC (in the

literature various results)

LLRT on the boundary (few results only normal distributions)

HC on the boundary (no results until now)

M Ditzhaus Tests for Signal Detection Wellington December 2017

General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0

probability measures microni Pni (Without loss of generality)

General testing problem

H0n P(n) =notimes

i=1

Pni against

H1n Q(n) =notimes

i=1

Qni with Qni = (1minus εni)Pni + εnimicroni

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)

dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 6: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heterogeneous normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n

εn = nminusβ

(signal probability)

ϑn =radic

2r log n

(signal strength)

see Ingster (1997)

Donoho amp Jin (2004)

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heteroscedastic normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ

2) 1 le i le n

solid (mdash)

Gaussian limit

dashed (- - -)

non-Gaussian but

infinitely divisible

Cai Jeng Jin (2011)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our aims

Detectable and undetectable areas of LLRT and HC (in the

literature various results)

LLRT on the boundary (few results only normal distributions)

HC on the boundary (no results until now)

M Ditzhaus Tests for Signal Detection Wellington December 2017

General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0

probability measures microni Pni (Without loss of generality)

General testing problem

H0n P(n) =notimes

i=1

Pni against

H1n Q(n) =notimes

i=1

Qni with Qni = (1minus εni)Pni + εnimicroni

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)

dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 7: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Model

Null we observe

Xni = Zni sim Pni i = 1 n (known noisy background)

Alternative we observe

Xni =

Yni sim microni if Bni = 1 (unknown signal)

Zni if Bni = 0 (noisy background)

where εni = P(Bni = 1) is unknown

In short Xni sim (1minus εni)Pni + εnimicroni = Qni

Independence assumption

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heterogeneous normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n

εn = nminusβ

(signal probability)

ϑn =radic

2r log n

(signal strength)

see Ingster (1997)

Donoho amp Jin (2004)

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heteroscedastic normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ

2) 1 le i le n

solid (mdash)

Gaussian limit

dashed (- - -)

non-Gaussian but

infinitely divisible

Cai Jeng Jin (2011)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our aims

Detectable and undetectable areas of LLRT and HC (in the

literature various results)

LLRT on the boundary (few results only normal distributions)

HC on the boundary (no results until now)

M Ditzhaus Tests for Signal Detection Wellington December 2017

General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0

probability measures microni Pni (Without loss of generality)

General testing problem

H0n P(n) =notimes

i=1

Pni against

H1n Q(n) =notimes

i=1

Qni with Qni = (1minus εni)Pni + εnimicroni

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)

dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 8: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

The detection boundary heterogeneous normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n

εn = nminusβ

(signal probability)

ϑn =radic

2r log n

(signal strength)

see Ingster (1997)

Donoho amp Jin (2004)

M Ditzhaus Tests for Signal Detection Wellington December 2017

The detection boundary heteroscedastic normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ

2) 1 le i le n

solid (mdash)

Gaussian limit

dashed (- - -)

non-Gaussian but

infinitely divisible

Cai Jeng Jin (2011)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our aims

Detectable and undetectable areas of LLRT and HC (in the

literature various results)

LLRT on the boundary (few results only normal distributions)

HC on the boundary (no results until now)

M Ditzhaus Tests for Signal Detection Wellington December 2017

General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0

probability measures microni Pni (Without loss of generality)

General testing problem

H0n P(n) =notimes

i=1

Pni against

H1n Q(n) =notimes

i=1

Qni with Qni = (1minus εni)Pni + εnimicroni

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)

dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 9: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

The detection boundary heteroscedastic normal

H0n Xniiid sim N (01) 1 le i le n

H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ

2) 1 le i le n

solid (mdash)

Gaussian limit

dashed (- - -)

non-Gaussian but

infinitely divisible

Cai Jeng Jin (2011)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our aims

Detectable and undetectable areas of LLRT and HC (in the

literature various results)

LLRT on the boundary (few results only normal distributions)

HC on the boundary (no results until now)

M Ditzhaus Tests for Signal Detection Wellington December 2017

General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0

probability measures microni Pni (Without loss of generality)

General testing problem

H0n P(n) =notimes

i=1

Pni against

H1n Q(n) =notimes

i=1

Qni with Qni = (1minus εni)Pni + εnimicroni

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)

dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 10: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our aims

Detectable and undetectable areas of LLRT and HC (in the

literature various results)

LLRT on the boundary (few results only normal distributions)

HC on the boundary (no results until now)

M Ditzhaus Tests for Signal Detection Wellington December 2017

General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0

probability measures microni Pni (Without loss of generality)

General testing problem

H0n P(n) =notimes

i=1

Pni against

H1n Q(n) =notimes

i=1

Qni with Qni = (1minus εni)Pni + εnimicroni

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)

dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 11: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Problem in practice the signal probability εni and the signal

distribution microni are unknown

rArr LLR cannot be used

But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area

I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our aims

Detectable and undetectable areas of LLRT and HC (in the

literature various results)

LLRT on the boundary (few results only normal distributions)

HC on the boundary (no results until now)

M Ditzhaus Tests for Signal Detection Wellington December 2017

General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0

probability measures microni Pni (Without loss of generality)

General testing problem

H0n P(n) =notimes

i=1

Pni against

H1n Q(n) =notimes

i=1

Qni with Qni = (1minus εni)Pni + εnimicroni

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)

dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 12: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Our aims

Detectable and undetectable areas of LLRT and HC (in the

literature various results)

LLRT on the boundary (few results only normal distributions)

HC on the boundary (no results until now)

M Ditzhaus Tests for Signal Detection Wellington December 2017

General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0

probability measures microni Pni (Without loss of generality)

General testing problem

H0n P(n) =notimes

i=1

Pni against

H1n Q(n) =notimes

i=1

Qni with Qni = (1minus εni)Pni + εnimicroni

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)

dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 13: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0

probability measures microni Pni (Without loss of generality)

General testing problem

H0n P(n) =notimes

i=1

Pni against

H1n Q(n) =notimes

i=1

Qni with Qni = (1minus εni)Pni + εnimicroni

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)

dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 14: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)

dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 15: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 16: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

LLRn = log(

dQ(n)

dP(n)(Xn1 Xnn)

)dminusrarr

ξ1 isin R cup minusinfin under H0

ξ2 isin R cup infin under H1

Undetectable ξ1 equiv 0 equiv ξ2

Detectable ξ1 equiv minusinfin ξ2 equiv infin

On the boundary (until now) ξ1 ξ2 infinitely divisible on R

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 17: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εnimicroni

(εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 18: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Our tool

All three cases are uniquely determined by the two sums

I1n(x) =nsum

i=1

εniEPni

( dmicroni

dPni1εni

dmicroni

dPnigt x

)

I2n(x) =nsum

i=1

ε2ni EPni

((dmicroni

dPni

)2

1εni

dmicroni

dPnile x

minus 1

)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 19: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one

(b) Suppose ξ1 isin R with probability one Then we have

(i) ξ1 sim ν1 is infinitely divisible

(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie

ξ2 sim (1minus a) εinfin + a ν2 a isin (01]

where ν2 is an infinitely divisible probability measure on (RB)

and a = P(ξ2 isin R)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 20: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Spike chimeric alternatives

Let h (01)rarr [0infin) be measurable withint 1

0h dλλ = 1 and

int 1

0h2 dλλ isin (0infin)

Let Pni = λλ|(01) and microni be defined by

dmicroni

dPni(u) =

0 if u isin [τni 1)

1τni

h(

uτni

)if u isin (0 τni)

whereτni isin (01) and max

1leilenτni rarr 0 as nrarrinfin

Literature Khmaladze (1998)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 21: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Illustration of x 7rarr dmicronidPni (x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 22: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 23: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 24: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Nonparametric detection boundary

Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]

non-trivial power

mdash ξj is normal (only depend onint 10 h2 dλλ)

bull ξj has non-trivial Leacutevy measure

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 25: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Extended detection boundary

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 26: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Extended detection boundary

Figure heteroscedastic normal Figure chimeric alternatives

ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin

ξ1 sim εminus 12

and ξ2 sim eminus12 εminus 1

2+ (1minus eminus

12 )εinfin

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 27: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt

1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 28: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Pitmanrsquos asymptotic relative efficiency

EQ(n)true(ϕnchoice)rarr Φ

(uα +

radiclt ht ht gt ARE

)

where ARE =lt ht hc gt

2

lt ht ht gtlt ht hc gt1βt = βc

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 29: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Tukeyrsquos higher criticsm

suggested by Tukey (asymp 1976)

modified by Donoho and Jin 2004

Multiple tests H0i Pni against H1i Qni for all i = 1 n

Use p-values pni = Tni(Xni) with PTnini = λλ|(01)

HCn = sup0ltαlt1

radicn

Fnp(α)minus αradicα(1minus α)

α isin (01)

where Fnp denotes the empirical distribution function of the p-values

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 30: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 31: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Tukeyrsquos higher criticsm

Connection to stochastic processes

Convergence under the null (see Jaeschke and Eicker resp)

an HCn minus bndminusrarr Z (Gumbel distributed)

an =

radic2 log2(n) and bn = 2 log2(n) +

12

log3(n)minus 12

log(π)

Critical value asympradic

2 log2(n)

Modifications of HCn absolute value ( 1n α0) ( 1

n 1minus1n )

Advantage HCn does not depend on the unknown parameters

M Ditzhaus Tests for Signal Detection Wellington December 2017

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 32: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

For simplicity

We consider here only the rowwise identical case

microni = micron Tni = Tn εni = εn

The first result holds also in the more general case

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 33: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 34: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Let v isin (0 12)

Hn(v) =radic

n εn

∣∣∣microTn

n (0 v ]minus v∣∣∣+∣∣∣microTn

n (1minus v 1]minus v∣∣∣

radicv

Theorem (D amp Janssen 2017)Under some more conditions we have

(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn

Q(n)minusminusminusrarrinfin (complete separation)

(b) supvisinIn

anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 35: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Detection areas coincide for different (new) model assumptions in

particular we solved an open problem in Cai and Wu (2014)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 36: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Optimality of HC

Figure heterogeneous normal Figure chimeric alternatives

Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 37: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Summary

Fruitful theory for the detection boundary

Tool to determine the limit distribution for all cases

Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))

Tools for HC for general distributions (in literature mainly normal)

Detection areas coincide

No power of HC on the boundary

M Ditzhaus Tests for Signal Detection Wellington December 2017

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 38: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

References I

ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808

DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND

WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094

M Ditzhaus Tests for Signal Detection Wellington December 2017

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 39: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

References II

INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 40: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 41: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives

DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on

nonparametric detection boundaries Submitted (arXiv 170907264)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 42: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable

(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 43: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 44: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 45: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)

ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)

Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense

subset D of (0infin) such that for all x isin D

I1n(x)rarr M(x infin] and σ2 = limε0

lim suplim infnrarrinfin

I2n(ε)

Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1

= exp σ2j = σ2 and

γj = (minus1)j σ2

2+

int(0infin)

(ex minus 1 +

x1 + x2

)dηj(x)

M Ditzhaus Tests for Signal Detection Wellington December 2017

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017

Page 46: The Power of Tests for Signal Detection in High ...€¦ · Left: Arti cial sources only. Right: Arti cial sources in the presence of noise as used in the simulations, emphasising

h-model in general

Suppose that

nsumi=1

ε2ni

τnirarr K isin [0infin] and

nsumi=1

ε2ni rarr 0 as nrarrinfin

Then

If K = 0 then ξ1 = ξ2 = 0 (Undetectable)

If K =infin and lim supnrarrinfinmax1leilenεniτni

lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)

If K isin (0infin) and max1leilenεniτnirarr 0

then ξj sim N(

(minus1)j σ2

2 σ2)

with σ2 = Kc2 (non-trivial+Gaussian)

M Ditzhaus Tests for Signal Detection Wellington December 2017