Springer Series in Statistics

Advisors: P. Bickel, P. Diggle, s. Fienberg, K. Krickeberg, 1. Olkin, N. Wennuth, s. Zeger

Springer Science+Business Media, LLC

Springer Series in Statistics

AndersenIBorganiGilllKeiding: Statistical Models Based on Counting Processes. Andrews/Herzberg: Data: A Collection of Problems from Many Fields for the Student

and Research Worker. Anscombe: Computing in Statistical Science through APL. Berger: Statistical Decision Theory and Bayesian Analysis, 2nd edition. Bolfarine!Zacks: Prediction Theory for Finite Populations. Borg/Groenen: Modem Multidimensional Scaling: Theory and Applications Bremaud: Point Processes and Queues: Martingale Dynamics. BrockwelllDavis: Time Series: Theory and Methods, 2nd edition. DaleyIVere-Jones: An Introduction to the Theory of Point Processes. Dzhaparidze: Parameter Estimation and Hypothesis Testing in Spectral Analysis of

Stationary Time Series. Efromovich: Nonparametric Curve Estimation: Methods, Theory, and Applications. Fahrmeirlfutz: Multivariate Statistical Modelling Based on Generalized Linear

Models. F arebrother: Fitting Linear Relationships: A History of the Calculus of Observations

1750 -1900. Farrell: Multivariate Calculation. Federer: Statistical Design and Analysis for Intercropping Experiments, Volume I:

Two Crops. Federer: Statistical Design and Analysis for Intercropping Experiments, Volume II:

Three or More Crops. FienberglHoagliniKruskal/Fanur (Eds.): A Statistical Model: Frederick Mosteller's

Contributions to Statistics, Science and Public Policy. Fisher/Sen: The Collected Works ofWassily Hoeffding. Good: Permutation Tests: A Practical Guide to Resampling Methods for Testing

Hypotheses. GoodmaniKruskal: Measures of Association for Cross Classifications. Gourieroux: ARCH Models and Financial Applications. Grandell.· Aspects of Risk Theory. Haberman: Advanced Statistics, Volume I: Description of Populations. Hall: The Bootstrap and Edgeworth Expansion. Hardie: Smoothing Techniques: With Implementation in S. Hart: Nonparametric Smoothing and Lack-of-Fit Tests. Hartigan: Bayes Theory. HedayatiSloaneiStufken: Orthogonal Arrays: Theory and Applications. Heyde: Quasi-Likelihood and its Application: A General Approach to Optimal

Parameter Estimation. Heyer: Theory of Statistical Experiments. HuetiBouvier/GruetiJolivet: Statistical Tools for Nonlinear Regression: A Practical

Guide with S-PLUS Examples. Jolliffe: Principal Component Analysis. KolenIBrennan: Test Equating: Methods and Practices. Kotz/Johnson (Eds.): Breakthroughs in Statistics Volume I.

(continued after index)

Dimitris N. Politis Joseph P. Romano Michael Wolf

Subsampling

i Springer

Dimitris N. Politis Department of Mathematics University of California, San Diego 9500 Gilman Drive La JoIla, CA 92093-0112 USA politis@math.ucsd.edu

Michael Wolf Departamento de Estadistica y Econometria Universidad Carlos III de Madrid Calle Madrid 126 28903 Getafe Spain mwolf@est-econ.uc3m.es

Joseph P. Romano Department of Statistics Stanford University Stanford, CA 94305-4065 USA romano@stat.stanford.edu

Library of Congress Cataloging-in-Publication Data Politis, Dimitris N.

Subsampling / Dimitris N. Politis, Joseph P. Romano, Michael Wolf. p. cm. - (Springer series in statistics)

lncludes bibiiographicai references and index. ISBN 978-1-4612-7190-1 ISBN 978-1-4612-1554-7 (eBook) DOI 10.1007/978-1-4612-1554-7 1. Bootstrap (Statistics). 1. Romano, Joseph P., 1960-

II. Wolf, Michael, 1960- . III. Title. IV. Series. QA276.7.P646 1999 519.5'44-dc21 99-15017

Printed on acid-free paper.

© 1999 Springer Science+Business Media New York OriginaIly published by Springer-Verlag New York, Inc. in 1999 Softcover reprint of the hardcover 1 st edition 1999 AII rights reserved. This work may not be translated or copied in whole or in part without the written permission ofthe pubIisher (Springer Science+Business Media, LLC), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dis similar methodology now known or hereafter developed is forbidden. The use of general descriptive names, trade names, trademarks, etc., in this publication, even ifthe former are not especiaIly identified, is not to be taken as a sign that such names, as understood by the Trade Marks and Merchandise Marks Act, may accordingly be used fteely by anyone.

Production managed by Robert Bruni; manufacturing supervised by Thomas King. Photocomposed pages prepared from the authors' U-'IEJX fiIes.

9 8 7 6 5 4 3 2 1

ISBN 978-1-4612-7190-1

To our families and friends

Preface

Since Efron's (1979) profound paper on the bootstrap, an enormous amount of effort has been spent on the development of bootstrap, jackknife, and other resampling methods. The primary goal of these computer-intensive methods has been been to provide statistical tools that work in complex sit­uations without imposing unrealistic or unverifiable assumptions about the data-generating mechanism. Bootstrap methods often can achieve this goal, but like any statistical method, they cannot be applied without thought. While the bootstrap will undoubtedly serve as a widely used tool well into the 21st century, it sometimes can fail. Moreover, the asymptotic justification of the bootstrap is often peculiar to the problem at hand.

It was realized in Politis and Romano (1992c, 1994b) that a very general approach to constructing asymptotically valid inference procedures exists by appropriate use of subsampling. That is, the statistic of interest, such as an estimator or test statistic, is evaluated at subsamples of the data, and these subsampled values are used to build up an estimated sampling distribution. Historically, the roots of subsampling can be traced to Que­nouille's (1949) and Tukey's (1958) jackknife. In fact, Mahalanobis (1946) suggested the use of subsamples to estimate standard errors in studying crop yields, though he used the term interpenetrating samples. Hartigan (1969, 1975) exploited the use of subsamples to construct confidence inter­vals and to approximate standard errors; he obtained some finite sample results in the symmetric location problem, as well as some asymptotic re­sults for asymptotically normal statistics. Hartigan's constructions involve recalculating a statistic over all subsamples and all subsample sizes, but he also considers using only randomly chosen subsamples, or what he calls

viii Preface

balanced subsamples. Of course, the use of subsamples is quite related to the delete-d jackknife, which was developed mainly in the context of vari­ance estimationj see Wu (1986) and Shao and Wu (1989). In the time series context, Carlstein (1986) considered the use of subsamples, or what he calls subseries, to approximate the variance of a statistic. Later, Wu (1990) real­ized how jackknife pseudo-values can be used for distribution estimation as well, but only in the context of independent, identically distributed (i.i.d.) observations when the statistic is approximately linear so that asymptotic normality holdsj his arguments were specific to this context.

The main contribution presented here can be summarized as follows: the consistency properties of estimated sampling distributions based on sub­sampling hold under extremely weak assumptions, and even in situations where the bootstrap fails. In contrast to the aforementioned results, we can obtain quite general results by considering subsamples of fixed size (but depending on the sample size and possibly data-dependent). In fact, it is hard to conceive of a theory for the construction of confidence inter­vals that works (in a first order sense) under weaker conditions. Moreover, the mathematical arguments to justify such a claim are based on fairly simple ideas, thus allowing for ease of generalization. Therefore, we will consider subsampling not just in the usual context of independent and iden­tically distributed observations, but also in the context of dependent data situations, such as time series, random fields, or marked point processes.

The idea of using subsamples as a diagnostic tool to describe the sampling distribution of an estimator was also considered in Sherman and Carlstein (1996). In the Li.d. context, subsampling is closely related to the usual bootstrap, the main difference being resampling the data without replace­ment versus with replacement. It follows that the bootstrap also enjoys general first order consistency properties, but only if one is willing to re­sample at sizes much smaller than the original sample size, as noted in Politis and Romano (1992c, 1993c)j the benefits of using a smaller resam­pIe size when bootstrapping are further exploited in Bickel, Gotze, and van Zwet (1997). Of course, the usual bootstrap can work perfectly well, and it enjoys good higher order properties in some nice problems. Subsampling, on the other hand, is more generally applicable. Ultimately, it may be used as a robust starting point toward even more refined procedures (some of which we present).

The primary goal of this book is to lay some of the foundation for subsam­pling methodology and related methods. The book is laid out in two parts. Chapters 1-7 provide the basic theory of the bootstrap and subsampling. Chapter 1 is devoted to developing some of the basic consistency properties of the bootstrap. Obviously, one can write volumes about the bootstrap, and there are now several books whose main subject is the theoretical de­velopment of the bootstrap. Our purpose is to provide some mathematical tools needed in studying consistency properties of the bootstrap, thereby setting the stage for studying subsampling. Thus, this chapter serves as

Preface ix

a tutorial on the bootstrap that can be included in a graduate course on asymptotic methods. The chapter is largely based on Beran (1984) and one of the author's lecture notes in a class on the bootstrap taught by Beran at U.C. Berkeley in the fall of 1983. Chapter 2 considers the use of subsam­pling in the case of independent and identically distributed observations. Some comparisons are made with the bootstrap. Chapter 3 considers sub­sampling stationary time series, and generalizations to the nonstationary or heteroskedastic case are presented in Chapter 4; the moving blocks boot­strap is also considered. Generalizations to random fields are developed in Chapter 5, and marked point processes are discussed in Chapter 6. In Chapter 7, a general theorem on subsampling is developed that allows the parameter space of interest to be quite abstract. This concludes the first of two parts of the book. The second part of the book is concerned with extensions of the basic theory, as well as practical issues in implementa­tion and applications. Chapter 8 addresses complex situations where the convergence rate of the estimator is unknown since it depends on param­eters that are not assumed known; subsampling is shown to be generally useful here as well. The issue of choice of block size is discussed in Chap­ter 9. Chapter 10 is devoted to higher-order accuracy and extrapolation. Two important cases where the convergence rate is unknown are detailed in Chapters 11 and 12. Chapter 11 considers the case of the mean with heavy tails (infinite variance), while Chapter 12 considers inference for an autoregressive parameter in the possibly integrated case. In Chapter 13, we consider a financial application by using subsampling to discuss whether stock returns can be predicted from dividend yields. The appendices contain some results on mixing sequences that are used throughout the book.

This book is intended for graduate students and researchers; it can be used for an advanced graduate course in statistics that assumes a basic knowledge of theoretical statistics usually taught at the first-year Ph.D. level. Even if one does not want to devote an entire course to subsampling, Chapters 1-3 are designed so that they can be included in any course on asymptotic methods in statistics.

We have benefitted from the support of many friends and colleagues. Special thanks are due to Patrice Bertail for his instrumental help toward the original development of the ideas in Chapters 8 and 10. Some of the book was presented in graduate courses taught at Stanford University and U.C. San Diego in 1997 and 1998, and we thank our students for providing helpful comments. We also would like to thank the National Science Foun­dation for supporting much of this research and writing over the past few years.

Contents

Preface

I Basic Theory

1 Bootstrap Sampling Distributions 1.1 Introduction ........ .

1.1.1 Pivotal Method .. . 1.1.2 Asymptotic Pivotal Method 1.1.3 Asymptotic Approximation. 1.1.4 Bootstrap Approximation

1. 2 Consistency.............. 1.3 Case of the Nonparametric Mean 1.4 Generalizations to Mean-like Statistics 1.5 Bootstrapping the Empirical Process 1.6 Differentiability and the Bootstrap 1. 7 Further Examples 1.8 Hypothesis Testing. 1.9 Conclusions .....

2 Subsampling in the I.I.D. Case 2.1 Introduction....... 2.2 The Basic Theorem ... 2.3 Comparison with the Bootstrap 2.4 Stochastic Approximation ....

vii

1

3 3 4 6 6 7 9

11 16 19 23 30 33 38

39 39 41 47 51

xii Contents

2.5

2.6 2.7 2.8 2.9

General Parameters and Other Choices of Root 2.5.1 Studentized Roots . . . . . 2.5.2 General Parameter Space .... Hypothesis Testing. . . . . . . . . . . . Data-Dependent Choice of Block Size. Variance Estimation: The Delete-d Jackknife. Conclusions ............ .

52 52 53 54 59 62 64

3 Subsampling for Stationary Time Series 65 3.1 Introduction..................... 65 3.2 Univariate Parameter Case . . . . . . . . . . . . . 67

3.2.1 Some Motivation: The Simplest Example. 67 3.2.2 Theory and Methods for the General Univariate

Parameter Case . . . 68 3.2.3 Studentized Roots. . 74

3.3 Multivariate Parameter Case 74 3.4 Examples........... 77 3.5 Hypothesis Testing. . . . . . 90 3.6 Data-Dependent Choice of Block Size 92 3.7 Bias Reduction. . . . . . . . . 93 3.8 Variance Estimation . . . . . . . . 95

3.8.1 General Statistic Case. . . 95 3.8.2 Case of the Sample Mean. 97

3.9 Comparison with the Moving Blocks Bootstrap 98 3.10 Conclusions..................... 100

4 Subsampling for Nonstationary Time Series 101 4.1 Introduction......... 101 4.2 Univariate Parameter Case . 102 4.3 Multivariate Parameter Case 106 4.4 Examples........... 108 4.5 Hypothesis Testing and Data-Dependent Choice of Block

Size ......... 117 4.6 Variance Estimation 117 4.7 Conclusions..... 119

5 Subsampling for Random Fields 120 5.1 Introduction and Definitions . . . . . . . . . . . . . . . .. 120 5.2 Some Useful Notions of Strong Mixing for Random Fields. 121 5.3 Consistency of Subsampling for Random Fields 123

5.3.1 Univariate Parameter Case. . . . 123 5.3.2 Multivariate Parameter Case. . . . . . . 127

5.4 Variance Estimation and Bias Reduction . . . . 129 5.5 Maximum Overlap Subsampling in Continuous Time 132 5.6 Some Illustrative Examples . . . . . . . . . . . . . . . 134

Contents xiii

5.7 Conclusions . . . . . . . . . . . .

6 Subsampling Marked Point Processes 6.1 Introduction .......... .

137

138 138

6.2 Definitions and Some Different Notions on Mixing 140 6.3 Subsampling Stationary Marked Point Processes 144

6.3.1 Sampling Setup and Assumptions. 144 6.3.2 Main Consistency Result . 146 6.3.3 Nonstandard Asymptotics . . . 149

6.4 Stochastic Approximation. . . . . . . . 151 6.5 Variance Estimation via Subsampling . 154 6.6 Examples. 155 6.7 Conclusions............ .158

7 Confidence Sets for General Parameters 7.1 Introduction .................. . 7.2 A Basic Theorem for the Empirical Measure. 7.3 A General Theorem on Subsampling 7.4 Subsampling the Empirical Process 7.5 Subsampling the Spectral Measure. 7.6 Conclusions ............. .

159 159 160 164 166 168 170

II Extensions, Practical Issues, and Applications 171

8 Subsampling with Unknown Convergence Rate 8.1 Introduction .................... . 8.2 Estimation of the Convergence Rate. . . . . . . .

8.2.1 Convergence Rate Estimation: Univariate Parameter Case . . . . . . . . . . . . . . .

8.2.2 Convergence Rate Estimation: Multivariate

173 173 177

177

Parameter Case . . . . . . . . . . . . . . 182 8.3 Subsampling with Estimated Convergence Rate 184 8.4 Conclusions..................... 187

9 Choice of the Block Size 9.1 Introduction ........... . 9.2 Variance Estimation ....... .

9.2.1 Case of the Sample Mean. 9.2.2 General Case . . . . . . . .

9.3 Estimation of a Distribution Function. 9.3.1 Calibration Method ..... 9.3.2 Minimum Volatility Method

9.4 Hypothesis Testing ..... 9.4.1 Calibration Method .....

188 188 189 189 192 193 194 197 200 200

xiv Contents

9.4.2 Minimum Volatility Method 9.5 Two Simulation Studies .

9.5.1 Univariate Mean. 9.5.2 Linear Regression

9.6 Conclusions 9.7 Tables .......... .

201 202 203 204 206 208

10 Extrapolation, Interpolation, and Higher-Order Accuracy 213 10.1 Introduction .......... 213 10.2 Background............... 216 10.3 Ll.D. Data: The Sample Mean . . . . 218

10.3.1 Finite Population Correction. 218 10.3.2 The Studentized Sample Mean. 219 10.3.3 Estimation of a Two-Sided Distribution 220 10.3.4 Extrapolation ..... 221 10.3.5 Robust Interpolation . 224

10.4 Ll.D. Data: General Statistics 225 10.4.1 Extrapolation ..... 225 10.4.2 Case of Unknown Convergence Rate to the

Asymptotic Approximation. . . 226 10.5 Strong Mixing Data . . . . . . . . . . . . . . . . 228

10.5.1 The Studentized Sample Mean. . . . . . 228 10.5.2 Estimation of a Two-Sided Distribution 231 10.5.3 The Unstudentized Sample Mean and the General

Extrapolation Result . . . . . . . . . . . . . . .. 232 10.5.4 Finite Population Correction in the Mixing Case 235 10.5.5 Bias-Corrected Variance Estimation for Strong

Mixing Data . . . . . . . . . . . . . . . . . . 236 10.6 Moderate Deviations in Subsampling Distribution

Estimation 244 10.7 Conclusions....................... 251

11 Subsampling the Mean with Heavy Tails 11.1 Introduction ......... . 11.2 Stable Distributions ......... .

253 253 254

11.3 Extension of Previous Theory .... 257 11.4 Subsampling Inference for the Mean with Heavy Tails 260

11.4.1 Appealing to a Limiting Stable Law .... 260 11.4.2 Using Self-Normalizing Sums. 263

11.5 Choice of the Block Size. . 265 11.6 Simulation Study . 266 11. 7 Conclusions 267 11.8 Tables . . . . . . . 269

12 Subsampling the Autoregressive Parameter 12.1 Introduction ......... . 12.2 Extension of Previous Theory ..... .

12.2.1 The Basic Method ........ .

Contents xv

270 270 272 272

12.2.2 Subsampling Studentized Statistics . . . . . 274 12.3 Subsampling Inference for the Autoregressive Root 278 12.4 Choice of the Block Size. . . . . . . . . . . . . .. 284 12.5 Simulation Study. . . . . . . . . . . . . . . . . . . . . . .. 286 12.6 Conclusions................ . . . . . . . .. 287 12.7 Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 289

13 Subsampling Stock Returns 291 13.1 Introduction ........... 291 13.2 Background and Definitions .. 292

13.2.1 The GMM Approach . . 293 13.2.2 The VAR Approach. . 294 13.2.3 A Bootstrap Approach . 296

13.3 The Subsampling Approach. . . . . . . . 297 13.4 Two Simulation Studies . . . . . . . . . . 298

13.4.1 Simulating VAR Data. . . . . . . 299 13.4.2 Simulating Bootstrap Data. . .. ..... 301

13.5 A New Look at Return Regressions . . . . . . . . . . . .. 302 13.6 Additional Looks at Return Regressions .......... 303

13.6.1 A Reorganization of Long-Horizon Regressions .. 303 13.6.2 A Joint Test for Multiple Return Horizons . 305

13.7 Conclusions...... . . . . . . 307 13.8 Tables . . . . . . . . . . . . . . . . . . . . . . . . . . 308

Appendices 315

A Some Results on Mixing 315

B A General Central Limit Theorem 321

References 327

Bibliography 327

Index of Names 341

Index of Subjects 345