Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
The Power of Tests for Signal Detection in High
Dimensional Data
Marc Ditzhaus
joint work with Arnold Janssen
Mathematical instituteHeinrich-Heine-University Duumlsseldorf
Wellington December 2017
M Ditzhaus Tests for Signal Detection Wellington December 2017
Motivation
genome analysis early detection of common deceases
high dimensional observation vector Xn = (Xn1 Xnn) for
each patient
healthy patient Xn behaves some noisy background
ill patient Xn contains rare and weak signals
Our interest The asymptotic power of tests which decide whether
there are any signals
Other applications astronomy cosmology disease
surveillance
M Ditzhaus Tests for Signal Detection Wellington December 2017
Motivation
genome analysis early detection of common deceases
high dimensional observation vector Xn = (Xn1 Xnn) for
each patient
healthy patient Xn behaves some noisy background
ill patient Xn contains rare and weak signals
Our interest The asymptotic power of tests which decide whether
there are any signals
Other applications astronomy cosmology disease
surveillance
M Ditzhaus Tests for Signal Detection Wellington December 2017
Astrophysics
Fig 2mdash Example artificial images Left Artificial sources only Right Artificial sources in the presenceof noise as used in the simulations emphasising that real sources close to or below the noise level becomedifficult or impossible to detect even visually
11
Source Hopkins et al (2002)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heterogeneous normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n
εn = nminusβ
(signal probability)
ϑn =radic
2r log n
(signal strength)
see Ingster (1997)
Donoho amp Jin (2004)
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heteroscedastic normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ
2) 1 le i le n
solid (mdash)
Gaussian limit
dashed (- - -)
non-Gaussian but
infinitely divisible
Cai Jeng Jin (2011)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our aims
Detectable and undetectable areas of LLRT and HC (in the
literature various results)
LLRT on the boundary (few results only normal distributions)
HC on the boundary (no results until now)
M Ditzhaus Tests for Signal Detection Wellington December 2017
General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0
probability measures microni Pni (Without loss of generality)
General testing problem
H0n P(n) =notimes
i=1
Pni against
H1n Q(n) =notimes
i=1
Qni with Qni = (1minus εni)Pni + εnimicroni
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)
dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Motivation
genome analysis early detection of common deceases
high dimensional observation vector Xn = (Xn1 Xnn) for
each patient
healthy patient Xn behaves some noisy background
ill patient Xn contains rare and weak signals
Our interest The asymptotic power of tests which decide whether
there are any signals
Other applications astronomy cosmology disease
surveillance
M Ditzhaus Tests for Signal Detection Wellington December 2017
Motivation
genome analysis early detection of common deceases
high dimensional observation vector Xn = (Xn1 Xnn) for
each patient
healthy patient Xn behaves some noisy background
ill patient Xn contains rare and weak signals
Our interest The asymptotic power of tests which decide whether
there are any signals
Other applications astronomy cosmology disease
surveillance
M Ditzhaus Tests for Signal Detection Wellington December 2017
Astrophysics
Fig 2mdash Example artificial images Left Artificial sources only Right Artificial sources in the presenceof noise as used in the simulations emphasising that real sources close to or below the noise level becomedifficult or impossible to detect even visually
11
Source Hopkins et al (2002)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heterogeneous normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n
εn = nminusβ
(signal probability)
ϑn =radic
2r log n
(signal strength)
see Ingster (1997)
Donoho amp Jin (2004)
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heteroscedastic normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ
2) 1 le i le n
solid (mdash)
Gaussian limit
dashed (- - -)
non-Gaussian but
infinitely divisible
Cai Jeng Jin (2011)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our aims
Detectable and undetectable areas of LLRT and HC (in the
literature various results)
LLRT on the boundary (few results only normal distributions)
HC on the boundary (no results until now)
M Ditzhaus Tests for Signal Detection Wellington December 2017
General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0
probability measures microni Pni (Without loss of generality)
General testing problem
H0n P(n) =notimes
i=1
Pni against
H1n Q(n) =notimes
i=1
Qni with Qni = (1minus εni)Pni + εnimicroni
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)
dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Motivation
genome analysis early detection of common deceases
high dimensional observation vector Xn = (Xn1 Xnn) for
each patient
healthy patient Xn behaves some noisy background
ill patient Xn contains rare and weak signals
Our interest The asymptotic power of tests which decide whether
there are any signals
Other applications astronomy cosmology disease
surveillance
M Ditzhaus Tests for Signal Detection Wellington December 2017
Astrophysics
Fig 2mdash Example artificial images Left Artificial sources only Right Artificial sources in the presenceof noise as used in the simulations emphasising that real sources close to or below the noise level becomedifficult or impossible to detect even visually
11
Source Hopkins et al (2002)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heterogeneous normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n
εn = nminusβ
(signal probability)
ϑn =radic
2r log n
(signal strength)
see Ingster (1997)
Donoho amp Jin (2004)
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heteroscedastic normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ
2) 1 le i le n
solid (mdash)
Gaussian limit
dashed (- - -)
non-Gaussian but
infinitely divisible
Cai Jeng Jin (2011)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our aims
Detectable and undetectable areas of LLRT and HC (in the
literature various results)
LLRT on the boundary (few results only normal distributions)
HC on the boundary (no results until now)
M Ditzhaus Tests for Signal Detection Wellington December 2017
General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0
probability measures microni Pni (Without loss of generality)
General testing problem
H0n P(n) =notimes
i=1
Pni against
H1n Q(n) =notimes
i=1
Qni with Qni = (1minus εni)Pni + εnimicroni
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)
dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Astrophysics
Fig 2mdash Example artificial images Left Artificial sources only Right Artificial sources in the presenceof noise as used in the simulations emphasising that real sources close to or below the noise level becomedifficult or impossible to detect even visually
11
Source Hopkins et al (2002)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heterogeneous normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n
εn = nminusβ
(signal probability)
ϑn =radic
2r log n
(signal strength)
see Ingster (1997)
Donoho amp Jin (2004)
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heteroscedastic normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ
2) 1 le i le n
solid (mdash)
Gaussian limit
dashed (- - -)
non-Gaussian but
infinitely divisible
Cai Jeng Jin (2011)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our aims
Detectable and undetectable areas of LLRT and HC (in the
literature various results)
LLRT on the boundary (few results only normal distributions)
HC on the boundary (no results until now)
M Ditzhaus Tests for Signal Detection Wellington December 2017
General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0
probability measures microni Pni (Without loss of generality)
General testing problem
H0n P(n) =notimes
i=1
Pni against
H1n Q(n) =notimes
i=1
Qni with Qni = (1minus εni)Pni + εnimicroni
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)
dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heterogeneous normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n
εn = nminusβ
(signal probability)
ϑn =radic
2r log n
(signal strength)
see Ingster (1997)
Donoho amp Jin (2004)
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heteroscedastic normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ
2) 1 le i le n
solid (mdash)
Gaussian limit
dashed (- - -)
non-Gaussian but
infinitely divisible
Cai Jeng Jin (2011)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our aims
Detectable and undetectable areas of LLRT and HC (in the
literature various results)
LLRT on the boundary (few results only normal distributions)
HC on the boundary (no results until now)
M Ditzhaus Tests for Signal Detection Wellington December 2017
General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0
probability measures microni Pni (Without loss of generality)
General testing problem
H0n P(n) =notimes
i=1
Pni against
H1n Q(n) =notimes
i=1
Qni with Qni = (1minus εni)Pni + εnimicroni
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)
dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heterogeneous normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n
εn = nminusβ
(signal probability)
ϑn =radic
2r log n
(signal strength)
see Ingster (1997)
Donoho amp Jin (2004)
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heteroscedastic normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ
2) 1 le i le n
solid (mdash)
Gaussian limit
dashed (- - -)
non-Gaussian but
infinitely divisible
Cai Jeng Jin (2011)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our aims
Detectable and undetectable areas of LLRT and HC (in the
literature various results)
LLRT on the boundary (few results only normal distributions)
HC on the boundary (no results until now)
M Ditzhaus Tests for Signal Detection Wellington December 2017
General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0
probability measures microni Pni (Without loss of generality)
General testing problem
H0n P(n) =notimes
i=1
Pni against
H1n Q(n) =notimes
i=1
Qni with Qni = (1minus εni)Pni + εnimicroni
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)
dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Model
Null we observe
Xni = Zni sim Pni i = 1 n (known noisy background)
Alternative we observe
Xni =
Yni sim microni if Bni = 1 (unknown signal)
Zni if Bni = 0 (noisy background)
where εni = P(Bni = 1) is unknown
In short Xni sim (1minus εni)Pni + εnimicroni = Qni
Independence assumption
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heterogeneous normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n
εn = nminusβ
(signal probability)
ϑn =radic
2r log n
(signal strength)
see Ingster (1997)
Donoho amp Jin (2004)
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heteroscedastic normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ
2) 1 le i le n
solid (mdash)
Gaussian limit
dashed (- - -)
non-Gaussian but
infinitely divisible
Cai Jeng Jin (2011)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our aims
Detectable and undetectable areas of LLRT and HC (in the
literature various results)
LLRT on the boundary (few results only normal distributions)
HC on the boundary (no results until now)
M Ditzhaus Tests for Signal Detection Wellington December 2017
General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0
probability measures microni Pni (Without loss of generality)
General testing problem
H0n P(n) =notimes
i=1
Pni against
H1n Q(n) =notimes
i=1
Qni with Qni = (1minus εni)Pni + εnimicroni
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)
dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heterogeneous normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn1) 1 le i le n
εn = nminusβ
(signal probability)
ϑn =radic
2r log n
(signal strength)
see Ingster (1997)
Donoho amp Jin (2004)
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heteroscedastic normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ
2) 1 le i le n
solid (mdash)
Gaussian limit
dashed (- - -)
non-Gaussian but
infinitely divisible
Cai Jeng Jin (2011)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our aims
Detectable and undetectable areas of LLRT and HC (in the
literature various results)
LLRT on the boundary (few results only normal distributions)
HC on the boundary (no results until now)
M Ditzhaus Tests for Signal Detection Wellington December 2017
General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0
probability measures microni Pni (Without loss of generality)
General testing problem
H0n P(n) =notimes
i=1
Pni against
H1n Q(n) =notimes
i=1
Qni with Qni = (1minus εni)Pni + εnimicroni
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)
dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
The detection boundary heteroscedastic normal
H0n Xniiid sim N (01) 1 le i le n
H1n Xniiid sim (1minus εn)N (01) + εnN (ϑn τ
2) 1 le i le n
solid (mdash)
Gaussian limit
dashed (- - -)
non-Gaussian but
infinitely divisible
Cai Jeng Jin (2011)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our aims
Detectable and undetectable areas of LLRT and HC (in the
literature various results)
LLRT on the boundary (few results only normal distributions)
HC on the boundary (no results until now)
M Ditzhaus Tests for Signal Detection Wellington December 2017
General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0
probability measures microni Pni (Without loss of generality)
General testing problem
H0n P(n) =notimes
i=1
Pni against
H1n Q(n) =notimes
i=1
Qni with Qni = (1minus εni)Pni + εnimicroni
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)
dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our aims
Detectable and undetectable areas of LLRT and HC (in the
literature various results)
LLRT on the boundary (few results only normal distributions)
HC on the boundary (no results until now)
M Ditzhaus Tests for Signal Detection Wellington December 2017
General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0
probability measures microni Pni (Without loss of generality)
General testing problem
H0n P(n) =notimes
i=1
Pni against
H1n Q(n) =notimes
i=1
Qni with Qni = (1minus εni)Pni + εnimicroni
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)
dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Problem in practice the signal probability εni and the signal
distribution microni are unknown
rArr LLR cannot be used
But Tukeyrsquos higher criticism (HC) is applicable and has the samedetection area
I normal mixtures Donoho and Jin (2004) Cai Jeng Jin (2011)I Poisson mixtures Arias-Castro and Wang M (2015)I general class of exponential families Cai and Wu (2014)I
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our aims
Detectable and undetectable areas of LLRT and HC (in the
literature various results)
LLRT on the boundary (few results only normal distributions)
HC on the boundary (no results until now)
M Ditzhaus Tests for Signal Detection Wellington December 2017
General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0
probability measures microni Pni (Without loss of generality)
General testing problem
H0n P(n) =notimes
i=1
Pni against
H1n Q(n) =notimes
i=1
Qni with Qni = (1minus εni)Pni + εnimicroni
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)
dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our aims
Detectable and undetectable areas of LLRT and HC (in the
literature various results)
LLRT on the boundary (few results only normal distributions)
HC on the boundary (no results until now)
M Ditzhaus Tests for Signal Detection Wellington December 2017
General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0
probability measures microni Pni (Without loss of generality)
General testing problem
H0n P(n) =notimes
i=1
Pni against
H1n Q(n) =notimes
i=1
Qni with Qni = (1minus εni)Pni + εnimicroni
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)
dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
General conditionstriangular scheme (εni)nisinN in [01] with max1leilen εni = εnn rarr 0
probability measures microni Pni (Without loss of generality)
General testing problem
H0n P(n) =notimes
i=1
Pni against
H1n Q(n) =notimes
i=1
Qni with Qni = (1minus εni)Pni + εnimicroni
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)
dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)
dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
LLRn = log(
dQ(n)
dP(n)(Xn1 Xnn)
)dminusrarr
ξ1 isin R cup minusinfin under H0
ξ2 isin R cup infin under H1
Undetectable ξ1 equiv 0 equiv ξ2
Detectable ξ1 equiv minusinfin ξ2 equiv infin
On the boundary (until now) ξ1 ξ2 infinitely divisible on R
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εnimicroni
(εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Our tool
All three cases are uniquely determined by the two sums
I1n(x) =nsum
i=1
εniEPni
( dmicroni
dPni1εni
dmicroni
dPnigt x
)
I2n(x) =nsum
i=1
ε2ni EPni
((dmicroni
dPni
)2
1εni
dmicroni
dPnile x
minus 1
)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(a) Either ξ1 isin R or ξ1 equiv minusinfin with probability one
(b) Suppose ξ1 isin R with probability one Then we have
(i) ξ1 sim ν1 is infinitely divisible
(ii) ξ2 is infinitely divisible on ((minusinfininfin]+) ie
ξ2 sim (1minus a) εinfin + a ν2 a isin (01]
where ν2 is an infinitely divisible probability measure on (RB)
and a = P(ξ2 isin R)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Spike chimeric alternatives
Let h (01)rarr [0infin) be measurable withint 1
0h dλλ = 1 and
int 1
0h2 dλλ isin (0infin)
Let Pni = λλ|(01) and microni be defined by
dmicroni
dPni(u) =
0 if u isin [τni 1)
1τni
h(
uτni
)if u isin (0 τni)
whereτni isin (01) and max
1leilenτni rarr 0 as nrarrinfin
Literature Khmaladze (1998)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Illustration of x 7rarr dmicronidPni (x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Nonparametric detection boundary
Let εni = nminusβ and τni = nminusr for β isin (12 1] and r isin (01]
non-trivial power
mdash ξj is normal (only depend onint 10 h2 dλλ)
bull ξj has non-trivial Leacutevy measure
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Extended detection boundary
Figure heteroscedastic normal Figure chimeric alternatives
ξ1 sim εminus1 and ξ2 sim eminus1εminus1 + (1minus eminus1)εinfin
ξ1 sim εminus 12
and ξ2 sim eminus12 εminus 1
2+ (1minus eminus
12 )εinfin
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt
1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Pitmanrsquos asymptotic relative efficiency
EQ(n)true(ϕnchoice)rarr Φ
(uα +
radiclt ht ht gt ARE
)
where ARE =lt ht hc gt
2
lt ht ht gtlt ht hc gt1βt = βc
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
suggested by Tukey (asymp 1976)
modified by Donoho and Jin 2004
Multiple tests H0i Pni against H1i Qni for all i = 1 n
Use p-values pni = Tni(Xni) with PTnini = λλ|(01)
HCn = sup0ltαlt1
radicn
Fnp(α)minus αradicα(1minus α)
α isin (01)
where Fnp denotes the empirical distribution function of the p-values
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Tukeyrsquos higher criticsm
Connection to stochastic processes
Convergence under the null (see Jaeschke and Eicker resp)
an HCn minus bndminusrarr Z (Gumbel distributed)
an =
radic2 log2(n) and bn = 2 log2(n) +
12
log3(n)minus 12
log(π)
Critical value asympradic
2 log2(n)
Modifications of HCn absolute value ( 1n α0) ( 1
n 1minus1n )
Advantage HCn does not depend on the unknown parameters
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
For simplicity
We consider here only the rowwise identical case
microni = micron Tni = Tn εni = εn
The first result holds also in the more general case
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Let v isin (0 12)
Hn(v) =radic
n εn
∣∣∣microTn
n (0 v ]minus v∣∣∣+∣∣∣microTn
n (1minus v 1]minus v∣∣∣
radicv
Theorem (D amp Janssen 2017)Under some more conditions we have
(a) aminus1n Hn(vn)rarrinfin rArr an HCn minus bn
Q(n)minusminusminusrarrinfin (complete separation)
(b) supvisinIn
anHn(v)rarr 0 rArr an HCn minus bndminusrarr Z under Q(n) (no separation)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Detection areas coincide for different (new) model assumptions in
particular we solved an open problem in Cai and Wu (2014)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Optimality of HC
Figure heterogeneous normal Figure chimeric alternatives
Theorem (D amp Janssen 2017)HC has no power on the detection boundary (for all our examples)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Summary
Fruitful theory for the detection boundary
Tool to determine the limit distribution for all cases
Extension of the detection boundaryI New limit distributions (P(ξ2 isin R) isin (01))
Tools for HC for general distributions (in literature mainly normal)
Detection areas coincide
No power of HC on the boundary
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
References I
ARIAS-CASTRO E AND WANG M (2015) The sparse Poisson means model Electron JStat 9 no 2 2170ndash2201CAI T JENG J AND JIN J (2011) Optimal detection of heterogeneous andheteroscedastic mixtures J R Stat Soc Ser B Stat Methodol 73 no 5 629ndash662CAI T AND WU Y (2014) Optimal Detection of Sparse Mixtures Against a Given NullDistribution IEEE Trans Inform Theory 60 no 4 2217-2232DITZHAUS M (2017) The power of tests for signal detection in high-dimensional dataDissertation Heinrich Heine University Duumlsseldorf httpsdocservuni-duesseldorfdeservletsDocumentServletid=42808
DONOHO D AND JIN J (2004) Higher criticism for detecting sparse heterogeneousmixtures Ann Statist 32 no 3 962ndash994DONOHO D AND JIN J (2015) Higher Criticism for Large-Scale Inference Especially forRare and Weak Effects Statist Sci 30 no 1 1ndash25HOPKINS A M MILLER C J CONNOLLY A J GENOVESE C NICHOL R C AND
WASSERMAN L (2002) A new source detection algorithm using the false-discovery rateAstron J 123 1086ndash1094
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
References II
INGSTER Y (1997) Some problems of hypothesis testing leading to infinitely divisibledistributions Math Methods Statist 6 no 1 47ndash69JANSSEN A MILBRODT H AND STRASSER H (1985) Infinitely divisible statisticalexperiments Lecture notes in Statistic 27 Springer-Verlag BerlinKHMALADZE EV (1998) Goodness of fit tests for Chimeric alternativesStatistNeerlandica 52 no 1 90ndash111
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Thank you for listeningFigure heteroscedastic normal Figure chimeric alternatives
DITZHAUS M AND JANSSEN A (2017) The power of big data sparse signal detection tests on
nonparametric detection boundaries Submitted (arXiv 170907264)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Theorem (D amp Janssen 2017)(i) Ijn(x)rarr 0 for some x gt 0hArr Undetectable
(ii) I1n(x)rarrinfin or I2n(x)rarrinfin for some x hArr Detectable
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
Recall ξ1 sim ν1 where ν1 has L-K-triplet (γ1 σ21 η1)
ξ2 sim (1minus a) εinfin + a ν2 where ν2 has L-K-triplet (γ2 σ22 η2)
Theorem (D amp Janssen 2017)(iii) Suppose that there is a finite measure M on (0infin] and a dense
subset D of (0infin) such that for all x isin D
I1n(x)rarr M(x infin] and σ2 = limε0
lim suplim infnrarrinfin
I2n(ε)
Then a = exp(minusM(infin)) η2minusη1 = M|(0infin)dη2dη1
= exp σ2j = σ2 and
γj = (minus1)j σ2
2+
int(0infin)
(ex minus 1 +
x1 + x2
)dηj(x)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017
h-model in general
Suppose that
nsumi=1
ε2ni
τnirarr K isin [0infin] and
nsumi=1
ε2ni rarr 0 as nrarrinfin
Then
If K = 0 then ξ1 = ξ2 = 0 (Undetectable)
If K =infin and lim supnrarrinfinmax1leilenεniτni
lt infinthen ξ1 = minusinfin and ξ2 =infin (Detectable)
If K isin (0infin) and max1leilenεniτnirarr 0
then ξj sim N(
(minus1)j σ2
2 σ2)
with σ2 = Kc2 (non-trivial+Gaussian)
M Ditzhaus Tests for Signal Detection Wellington December 2017