FDR Controlling Step-up-down Tests Related to the Asymptotically

Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC

FDR Controlling Step-up-down Tests Relatedto the Asymptotically Optimal Rejection Curve

By Helmut Finner

Institute of Biometrics & EpidemiologyGerman Diabetes Center

Leibniz-Institute at the Heinrich-Heine-University, Duesseldorf, Germany

µTOSS 2010 Berlin - Multiple comparisons fromtheory to practice

February 15-16, 2010


Co-Workers

Main Co-Workers:

Thorsten Dickhaus (Berlin Institute of Technology)Veronika Gontscharuk (DDZ)Markus Roters (Omnicare Clinical Research)

Further:

Marsel Scheer (DDZ)Klaus Strassburger (DDZ)


References

• Finner, H., Dickhaus, T. & Roters, M. (2009). On the falsediscovery rate and an asymptotically optimal rejectioncurve. Ann. Statist. 37, 596-618.

• Finner, H. & Gontscharuk, V. (2009). FDR controllingstep-up-down tests related to the asymptotically optimalrejection curve. Submitted for publication.


Personal view on FDR

• 1995: Very sceptical, FDR allows cheating; Proof in B&H‘incomplete’; no reference to Eklund (1961-1963), Eklund& Seeger (1965)

• 1996: First talk on FDR, Title: Controlling the falsediscovery rate: a doubtful concept in multiple comparisonsBut statement: There will be hundreds of papers on FDRin the near future

• 1997-1999: Draft with the same title• 2001: Publication (Biom. J.) with a more relaxed title• 2002: Theoretical paper (Ann. Statist.) on expected errors

(FWER and FDR), Asymptotics• 1998-2003: Work on the Partitioning Principle (FWER)• 2005-2007: FDR, dependency, asymptotics, optimal

rejection curve


Acknowledgment

We should all be very grateful to

Prof. Yoav Benjamini and Prof. Yosef Hochberg

for their 1995 paper which can be viewed as the initial point fora new era in multiple testing !

Benjamini, Y. and Hochberg, Y. (1995). Controlling the falsediscovery rate: A practical and powerful approach to multipletesting. J. R. Stat. Soc., Ser. B, 57, 289-300.


OverviewIntroduction

Warm-up: Linear step-up test (LSU)

Critical value functions and rejection curves

FDR bounds for step-up procedures

Asymptotically optimal rejection curve

Finite FDR control

Recursion for the computation of critical values: Exact solving

Simple adjustments for the AORC

Iterative method

A general results on FDR bounding curves and rejection curves


Notation & Definition of FDRΘ parameter space

H1, . . . , Hn null-hypotheses; p1, . . . , pn p-values

ϕ = (ϕ1, . . . , ϕn) multiple test procedure

Vn = |{i : ϕi = 1 and Hi true }| = number of true hypotheses rejected

Rn = |{i : ϕi = 1}| = number of hypotheses rejected

FDRϑ(ϕ) = Eϑ

[Vn

Rn ∨ 1

]actual false discovery rate given ϑ ∈ Θ

Definition 1. Let α ∈ (0, 1) be fixed.

ϕ controls the false discovery rate (FDR) at level α if

FDR(ϕ) = supϑ∈Θ

FDRϑ(ϕ) ≤ α.


False Discovery Proportion (FDP)

Define FDP = 0 if Rn = 0 and for Rn > 0

FDP =Vn

Rn

=number of true hypotheses rejected

number of hypotheses rejected

=number of false significances

number of significances

FDR = expectation of FDP = E[FDP]


Eklund 1961-1963, Seeger 1965, 1968


The FDR-Theorem: Benjamini & Hochberg (1995)Hi true for i ∈ In,0, Hi false for i ∈ In,1

In,0 + In,1 = In = {1, . . . , n}, n0 = |In,0|

Basic independence assumptions (BIA):

• pi ∼ U([0, 1]), i ∈ In,0, independent

• (pi : i ∈ In,0), (pi : i ∈ In,1) independent

p1:n ≤ · · · ≤ pn:n ordered p-values

Linear step-up procedure (LSU) ϕLSU based on Simes’ crit. val.αi:n = iα/n:

Reject all Hi with pi ≤ αm:n, where m = max{i : pi:n ≤ αi:n}.

Then

FDRϑ(ϕLSU) = Eϑ

[Vn

Rn ∨ 1

]=

n0

nα


Linear step-up in terms of ecdf

Reformulate the LSU test in terms of the rejection curver(t) = t/α. We call r(t) = t/α Simes’ line.

Let Fn denote the empirical cdf (ecdf) of the p-values, that is,

Fn(t) =1n

n∑i=1

I[0,t](pi).

Definet∗ = sup{t ∈ [0, α] : Fn(t) ≥ t/α}

and reject all Hi with pi ≤ t∗.

Call t∗ the largest crossing point (LCP).

Note: t∗ ∈ {0, α1:n, . . . , αn:n} with αi:n = iα/n = r−1(i/n).


RemarkIt is sometimes convenient to write

Vn(t) = {i ∈ In,0 : pi ≤ t},Rn(t) = {i ∈ In : pi ≤ t},

where t may be replaced by a random threshold τ .

Many multiple testing procedures can be defined in terms of asingle (random) threshold τ resulting in the decision rule

Reject Hi if and only if pi ≤ τ.

Here we give a proof for the LSU-FDR-Theorem based onOptional Stopping.

(Idea from Storey, Taylor, Siegmund, (2004). J. R. Statist. Soc.B 66, 1, 187-205. Sketch of proof faulty.)


Re-Definition of LSU Threshold

t∗ in the definition of LSU can be replaced by

τ = max{t : Fn(t) ∨1n

=tα}

= max{t : Rn(t) ∨ 1 =nα

t}

= max{αi:n : Rn(αi:n) ∨ 1 = i, i ∈ In},

hence,

FDRϑ(ϕLSU) = Eϑ

[Vn(τ)

Rn(τ) ∨ 1

]=

α

nEϑ

[Vn(τ)

τ

].

It remains to show that Eϑ

[Vn(τ)

τ

]= n0.


Martingale and Stopping Time

Fact 1: τ is a stopping time with respect to the backwardfiltration Ft = σ(I{pi≤s}, t ≤ s ≤ 1, i ∈ In), 0 < t ≤ 1.

Proof. It suffices to show that {τ ≤ αi:n} ∈ Fαi:n . Obviously,

{τ ≤ αi:n} =n⋂

j=i+1

{pj:n > αj:n} ∈ Fαi+1:n ⊆ Fαi:n for i = 1, . . . , n−1.

For i = n we have αn:n = α and {τ ≤ α} ∈ Ft for all t ∈ (0, 1].


Martingale and Stopping TimeFact 2: V(t)/t for 0 < t ≤ 1 is a martingale with time runningbackwards with respect to the filtration Ft, 0 < t ≤ 1.

Proof. Easy to check:

(1)V(t)

tis Ft −measurable for 0 < t ≤ 1,

(2) Eϑ

[V(s)

s|Ft

]=

V(t)t

for 0 < s ≤ t ≤ 1,

(3) Eϑ

[V(t)

t

]= n0 < ∞.

Finally, the Optional Stopping Theorem yields

Eϑ

[Vn(τ)

Rn(τ) ∨ 1

]=

α

nEϑ

[Vn(τ)

τ

]=

α

nEϑ

[Vn(1)

1

]=

n0

nα.


Can we improve LSU ?

Since

FDRϑ(ϕLSU) =n0

nα < α for n0 < n,

an obvious question is whether LSU can be improved.


Do there exit better rejection curves ?

What happens with the FDR if we use

non-linear critical values / a non-linear rejection curve

for an SU- or more generally a step-up-down procedure ?

Does there exist a rejection curve such that

Eϑ

[Vn(τ)

Rn(τ) ∨ 1

]= α

for some ϑ’s with n0 < n ?


Critical value functions

Let ρ : [0, 1] → [0, 1] be non-decreasing and continuous with

ρ(0) = 0 and positive values on (0, 1].

Define critical values

αi:n = ρ(i/n) ∈ (0, 1], i = 1, . . . , n.

We call ρ a critical value function.

Define r by

r(x) = inf{u ∈ [0, 1] : ρ(u) = x} for x ∈ [0, 1].

We call r a rejection curve.

Often: r(x) = ρ−1(x).


Least favorable configurations for SU based on ρ

From Benjamini and Yekutieli (2001) we get the following.

Assume BIA and consider an SU-procedure based on ρ.

If ρ(x)/x is non-decreasing in x ∈ (0, 1] then the FDR is largest

if pi = 0 a.s. for all alternative p-values pi, i ∈ In,1,

i.e., Dirac-uniform configurations are least favorable (LFC) forthe FDR.

Note: ρ(x)/x non-decreasing in x iff αi:n/i non-decreasing in i

We refer to this monotonicity property as (M).


FDR bound for SULet ϑ ∈ Θ such that n0 = |In,0| = number of true nulls,

let ρ satisfy (M) and let I′n,0 = In,0 \ {i0} for some i0 ∈ In,0.

Then the FDR of the SU-test ϕSU,ρ(n) based on ρ is bounded by

FDRϑ(ϕSU,ρ(n) ) ≤ n0

nEI′n,0

[ρ(Rn/n)

Rn/n

],

with ” = ” for the Dirac-uniform configuration DU(n, n0).

Moreover, if

limn→∞

n0

n= ζ and lim

n→∞

Rn

n= γζ a.s. (w.r.t. DU(n, n0)),

then the FDR is asymptotically bounded by

ζρ(γζ)γζ

.


Idea: Optimize ρ for an SU procedure

Try to find a ρ such that

ζρ(γζ)γζ

≡ min{ζ, α}

for as many ζ ’s as possible !

In other words, try to find a critical value function (or rejectioncurve) such that the FDR-level α is (asymptotically) exhaustedunder DU-configurations.

Solution: ρ(t) =αt

1− (1− α)t

r(t) = fα(t) =t

α + (1− α)t

fα: Asymptotical Optimal Rejection Curve (AORC)


Simes’ line and AORC for α = 0.1


A further heuristic for the AORCConsider DU(n, n0)-models with limn→∞ n0/n = ζ.

Reject all Hi with pi ≤ t (t ∈ (0, 1)). Then the asymptotic FDR(depending on ζ and t) is given by

FDRζ(t) =tζ

(1− ζ) + tζ.

Aim:FDR ≡ α for possibly all ζ ∈ (0, 1).

We have:

FDRζ(tζ) = α ⇐⇒ tζ =α(1− ζ)ζ(1− α)

(=⇒ ζ ∈ (α, 1))

⇐⇒ ζ =α

tζ(1− α) + α.


A further heuristic for the AORCAnsatz: tζ crossing point between rejection curve r and thelimiting cdf Gζ(t) = 1− ζ + ζt,

i.e., r(tζ) = Gζ(tζ).

Plugging in ζ = αtζ(1−α)+α yields

r(tζ) = Gζ(tζ)= (1− ζ) + ζtζ

=tζ

tζ(1− α) + α.

Hence, r = fα defined by

fα(x) =x

x(1− α) + α, x ∈ [0, 1],

is the curve we are looking for !


Problem: SU does not work with fα

Critical values induced by AORC:

αi:n = f−1α (i/n) =

inα

1− in(1− α)

=iα

n− i(1− α), i = 1, . . . , n,

are not valid for SU because of αn:n = 1.

Ways out of this dilemma:

• Adjust the AORC slightly• Step-up-down procedures


Step-down (SD), step-up (SU), step-up-down (SUD)

(1) SUD(λ) test ϕλ with parameter λ ∈ {1, . . . , n}:

pλ:n ≤ αλ:n ⇒ SD-part:m = max{j ∈ {λ, . . . , n} : pi:n ≤ αi:n for all i ∈ {λ, . . . , j}} ,

pλ:n > αλ:n ⇒ SU-part:m = max{j ∈ {1, . . . , λ− 1} : pj:n ≤ αj:n} , (max ∅ = −∞).

Reject all Hi with pi ≤ αm:n.

λ = 1 ⇒ step-down test

λ = n ⇒ step-up test.

(2) SUD(λ) based on β-adjusted AORC:

fα,βn = (1 + βn/n)fα for a suitable βn > 0.


SUD(λ) test based on fα with λ1 = 20 and λ2 = 40(n = 50, α = 0.1)


Upper FDR bounds for SUD(λn) tests

Theorem: (Finner et al (2009))

Let ϑ ∈ Θ such that n0 hypotheses are true.

Under (BIA) and (M) we have for an SUD(λ) test ϕλ based on ρ

FDRϑ(ϕλ) ≤ n0

nEn,n0−1

[αRn:n

Rn/n

]

with ”‘ = ”’ for a step-up test ϕSU under DU(n, n0).

Note: αRn:n = n0n ρ(Rn/n).

Important: b(n, n0|λn) := En,n0−1

[αRn:nRn/n

]is computable!


Upper FDR bounds for SUD(λn) tests: Asymptotics(A) For SUD(λn) tests with λn/n → κ ∈ (0, 1] andn0/n → ζ ∈ [0, 1]:

limn→∞

b(n, n0|λn) = limn→∞

FDRn,n0 . (1)

(B) For κ = 0 (e.g. SD tests) and ζ ∈ [0, 1), (1) holds too.

(C) But: For κ = 0, ζ = 1 it is possible that

limn→∞

b(n, n0|λn) > limn→∞

FDRn,n0 .

Example: Let n0 = n. Then the FDR (=FWER) of an SD testbased on fα is equal to 1− (1− α1:n)n. Moreover,

limn→∞

FDRn,n = 1− exp(−α) < α = limn→∞

b(n, n|1).


Upper FDR bounds for SUD(λ) tests

Given (BIA) and (M), the upper bounds for the FDR of SUD(λn)tests do not depend on the specific ϑ ∈ Θ. Set

b(n, n0|λ) = n0

n0∑j=1

αn−n0+j:n

n− n0 + jPn,n0−1(Vn = j− 1)

and b∗n = max1≤n0≤n

b(n, n0|λ).

Then supϑ∈Θ

FDRϑ(ϕ) ≤ b∗n.

Computing time for the upper bounds b(n, n0|λ) of a SUD(λ)test can be much longer than for an SU test.

This is due to much more complicated recursive formulas forPn,n0−1(Vn = j− 1) for SUD(λ) tests.


SU FDR control implies SUD FDR control

Theorem: Given (BIA) and (M). Consider an SU test ϕn and anSUD test ϕλ with λ ∈ {1, . . . , n− 1} and critical values (αi:n)n

i=1.Then

FDRϑ(ϕλ) ≤ FDRϑ(ϕn) for all ϑ ∈ Θ

andb(n, n0|λ) is non-decreasing in λ.


Violation of the FDR level for finite n

SD tests based on fα with α = 0.05 and n = 100, 150, 200

Aim: Tests closely related to AORC with FDR controlled atlevel α.


Stepwise search for suitable critical valuesIt always holds b(n, n0|λ) ≤ n0/n. Therefore we reformulate ouraim.

Aim: Set g∗(ζ) = min(ζ, α) and try to find critical values suchthat

b(n, n0|λ) = g∗(n0/n) for n0 ∈ {1, . . . , n}.

Recursion:

αn:n = ng∗(1/n) and αn−n0+1:n = hn0(αn−n0+2:n, . . . , αn:n)

withhn0(αn−n0+2:n, . . . , αn:n) =

n− n0 + 1n0Pn,n0−1(Vn = 0)

g∗(n0/n)− n0

n0∑j=2

αn−n0+j:n

n− n0 + jPn,n0−1(Vn = j− 1)

.

Does not work even for small n !!!


Recursion for SUTheorem. For an SU test, that is λ = n, b(n, n0|n) canalternatively be calculated by

b(n, n0|n) =n0∑

j=1

jn− n0 + j

Pn,n0(Vn = j) (2)

and it even holds

b(n, n0|n) = FDRn,n0(ϕn)

and

Pn,n0(Vn = j) =n0

jαn−n0+j:nPn,n0−1(Vn = j− 1) for j = 1, . . . , n0.

(3)Equation (3) makes computation much faster for SU than forSUD.


Why does it not work ?


Recursion with new FDR bounding curves

Question: Does there exist a g : [0, 1] → [0, 1] with g ≤ g∗ suchthat

b(n, n0|λ) = g(n0/n) for n0 ∈ {1, . . . , n}

results in admissible critical values (αi:n)ni=1 satisfying (M) ?

The only known solution is g(ζ) = αζ which results in the LSUtest with

FDRn,n0 = g(n0/n) =n0

nα.


FDR bounding curves: Example

Define g(ζ|γ, η) = α(1− (1− ζ/γ)η)I[0,γ] + αI(γ,1], 0 ≤ ζ < 1with 1 ≤ η ≤ γ/α and α ≤ γ ≤ 1.

α = 0.05, γ = 0.5 and η = 6, 8, 10


FDR bounding curves: Transformation

Idea: Apply the linear transformation with (1, 0) → (1, 0) and(0, 1) → (1, 1).

g(ζ|γ, η) and transformed g(ζ|γ, η) with α = 0.1, γ = 1, η = 50


FDR bounding curves

Examples:

• Choose cdf of a β-distribution:

Gη(x) = α(1− (1− x/γ)η)I[0,γ)(x) + αI[γ,1](x),

• Choose cdf of an exponential distribution:

Gη(x) = α(1− exp(−ηx))I[0,1](x).

and transform Gη to g(·|η) ≤ g∗ as described before.


FDR bounding curves

The recursion with a suitable FDR bounding curve often

yields admissible critical values with (M).

Advantage: The FDR (or the upper bound) is exactly known.

Disadvantage: The recursion may fail.

General problem: For larger values of n the computation takesa long time.


Adjusted AORC

Adjusted AORC: fα,βn(t) = (1 + βn/n) fα(t) with βn > 0 or a’snapped off’ version.

α = 0.05, n = 10, 30, 100, β10 = 1.23, β30 = 1.41, β100 = 1.76and n = 100 with β∗100 = 1.41, β∗100 = 1.30


FDR curves for adjusted versions of AORC

α = 0.05, n = 100, βn = 1.76, β∗n = 1.41, k = 95, β∗n = 1.3, k = 90


Behaviour of βn and β∗n

• βn ≥ 1 =⇒ FDRn,n0 ≤ α for SD tests (Gavrilov et al (2009)).• βn ≥ 2 =⇒ b(n, n0|1) ≤ α for SD tests.

Red: SU; Green:SD; Yellow: Snapped-off SU with β∗n


Behaviour of βn

Summary:For the best possible choice of βn in fα,βn it holds:• βn non-decreasing in λ

• βn bounded for SD tests (βn ≤ 1)• limn→∞ βn/n = 0 for all types of SUD Tests• limn→∞ βn = ∞, for SU tests (λ = n)• βn ≤ (1− α)n (not very helpful) .

Conjecture:

βn is bounded for SUD(λn) tests with λn/n → κ ∈ [0, 1).


β-adjustment

β-adjusted methods always yield admissible critical values.

Corresponding FDR curves (resp. upper FDR bounds)

converge to α or a well defined FDR bounding curve.


Critical values with most influence on FDR under DU


Fix Point Ansatz with respect to critical values

Suppose (αi:n)ni=1 satisfies (M).

Idea: Re-write critical values (αi:n)ni=1 as critical values

of an AORC fα, that is, write

αi:n =ici

n− i(1− ci)= f−1

ci(i/n), i ∈ In.

c = (c1, . . . , cn): Vector of ”‘local FDR levels”’.

Mapping: ci → αci

FDRn,n0(i)(c), i = 1, . . . , i∗,

ci → ci, i = i∗ + 1, . . . , n.

n0(i) integer closest to n− i(1− α),i∗ = i∗(n, k, α) = b(n− k)/(1− α)c,only FDRn0,n-values for n0 = k, . . . , n are iterated.


Iterative method with β-adjustment: FDR curves

SU test with n = 100, α = 0.05Start with β100 = 1.76-adjusted critical values.Choose k = 15, i∗ = 89 and J = 50 iterations.


Iterative Method

Iterative Method yields FDRs close to α.

Advantage:

Critical values always admissible.

Disadvantage:

FDRs sometimes slightly larger than α.

General problem:

Convergence properties of the algorithm not clear.


Summary of methods

(M1) Critical values based on alternative FDR curves and exactsolving;

(M2) Adjusted critical values based on fα with βn or β∗n ;

(M3) Critical values based on iterative method.

Recommendation for large n:For example, for α = 0.05 and n ≥ 2000:

SUD tests with λn = 0.7n and β ∈ [β2000, 2] = [1.58, 2]or SU tests with β∗n ∈ [β2000, 2] = [1.45, 2] for k ≈ n(1− 2α).


FDR bounding curves and rejection curves

Remember the heuristic for AORC.

Let g denote an FDR bounding curve and consider

DU-models with n0/n → ζ. Try to find a rejection curve such that

FDRζ(t) = g(ζ).

This leads to an implicit definition of r and ρ = r−1:

r(

g(ζ)(1− ζ)ζ(1− g(ζ))

)=

1− ζ

1− g(ζ), ζ ∈ (0, 1) (4)

ρ

(1− ζ

1− g(ζ)

)=

g(ζ)(1− ζ)ζ(1− g(ζ))

, ζ ∈ (0, 1). (5)



The following lemma shows that r and ρ are well defined forsuitable FDR bounding curves g.

Lemma 1. Let g be a continuous FDR bounding curve suchthat g(ζ)/ζ is non-increasing in ζ ∈ (0, 1] and letb = limζ→0 g(ζ)/ζ. Then r : [0, b] → [0, 1] and ρ : [0, 1] → [0, b]are well defined via (4) and (5), respectively, and by settingρ(0) = r(0) = 0 and r(b) = 1, ρ(1) = b. Moreover, ρ fulfills (M).



Theorem. Let g be an FDR bounding curve with the sameproperties as in Lemma 1. Consider SUD(λn) tests ϕn based onr defined in (4) with λn/n → κ. Then we obtain for the limitingFDR in DU(n, n0) models with n0/n → ζ that

limn→∞

FDRn,n0 = g(ζ)

for (i) κ ∈ (0, 1] and ζ ∈ [0, 1] if b < 1, (ii) κ ∈ (0, 1) and ζ ∈ [0, 1]if b = 1 and (iii) κ = 0 and ζ ∈ [0, 1).


Open Problems

• Adjusted AORC: βn bounded for SUD ?• We know that DU is LFC for SU.• Is DU LFC for SUD(λ) for λ < n ?• No counterexample is known to the speaker.• Bounds are not sharp for SUD.• Do there exist FDR bounding curves working for all n

(others than g(ζ) = αζ) ?• FDR bounds for plug-in methods (e.g. Storey’s procedure)

are not sharp.• Is DU LFC for plug-in methods ?• Do there exist better formulas for the computations of SUD

tests ?


References• Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery

rate: A practical and powerful approach to multiple testing.J. R. Stat. Soc., Ser. B, 57, 289-300.

• Benjamini, Y. and Yekutieli, D. (2001). The control of the false discoveryrate in multiple testing under dependency. Ann. Stat., 29, 4, 1165-1188.

• Finner, H. , Dickhaus, T. , and Roters, M. (2009). On the false discoveryrate and an asymptotically optimal rejection curve. Ann. Stat., 37, 2,596-618.

• Finner, H. and Gontscharuk, V. (2009). FDR controlling step-up-downtests related to the asymptotically optimal rejection curve. Submitted.

• Finner, H. and Gontscharuk, V. (2009). Controlling the familywise errorrate with plug-in estimator for the proportion of true null hypotheses.J. R. Stat. Soc. B, 71, 1031-1048.

• Gavrilov, Y., Benjamini, Y. and Sarkar, S. K. (2009). An adaptivestep-down procedure with proven FDR control. Ann. Stat., 37, 2,619-629.

´


Thanks for your patience !

Helau !

Altweiber 2010: Jecke feiern auf dem Rathausplatzin Dusseldorf

Documents

FDR Controlling Step-up-down Tests Related to the Asymptotically