Proceedings of the 1989 Winter Simulation Conference E.A. MacNair, K.J. Musselman, P. Heidelberger (eds.)

* '

A GRAPHICAL TEST BED (GMTBED) FOR ANALYZING AND REPORTING THE RESULTS OF A STATISTICAL EXPERIMENT

Peter A. W. Lewis Department of Operations Research

Naval Postgraduate School Monterey, CA 93940.

ABSTRACT The main innovation in the version in Lewis, Orav and Uribe (1989) -referred to as SMTBPC12 in micro-

It is unfortunately true that statisticians seldom ap- computer versions and SIMTBED12 in the mainframe ply the tools of their own trade to the analysis of simula- version-is the ability to restart the simulation after tion experiments. This refers to the experimental design a certain number of super-replications have been per- for the input, and the data presentation and summary formed. This would be the case if not enough precision

which the output of a simulation experiment can be re- @judge since the program has a built in facility for assess- and analysis for the output. t e s n e d in had been attained in the simulation, and this is easy to

U

ported and analyzed is described. From the graphics and the associated numerics the experimenter can sum- marize and see simultaneously relative properties&" as bias, normality and sp read lo f several estimators of a characteristic of the population for up to sixteen sam- ple sizes. The evolution of these properties with sample size is also displayed and analyzed using regression and asymptotic expansions. The graphics is supported on a line printer to make i t and the program portable. The use of the program is illustrated by a study of estimates of serial correlation in normal time series.

4 .!4

The latest version of SIMTBED incorporates a restart facility which makes the program ideal for ef- ficient utilization of microcomputers. A microcomputer version which uses color in the output to differentiate factors in the simulation is described, as well as im- provements which are being incorporated into the latest version. This latest version will run on micros, minis or m a i n f r a m e s 1

1. INTRODUCTION

The SIMTBED program is a graphical test bed for analyzing the results of a statistical simulation experi- ment. An earlier version of the program was reported in Lewis, et a1 (1985) and the latest version, which was used extensively for i l l idrative purposes in Lewis and Orav (1989), is described quite extensively in Lewis, Orav and Uribe (1989).

The purpose of this report is to summarize further enhancements to the package, some of which have been implemented, and some of which are still experimental.

ing the variability of estimates of characteristics of the distribution of the statistic or statistics which are under investigation. This facility is particularly useful for mi- crocomputers since i t allows the experimenter to use his microcomputer for simulations when he is not using i t , for example, overnight.

2. FURTHER ENHANCEMENTS

The original intent in designing SIMTBED was to make i t as portable as possible, and we therefore used a line-printer plotter package for graphical output. It is clear now, however, that better graphics are neces- sary for better visual discriniination of the output. For example, using cheap color dot-matrix printers we have been able to coalesce three of the basic output figures, such as those in figures 1, 2 and 3 so that the effects of both sample size and different methods of bootstrap- ping can be seen simultaneously. While i t is usable, this enhancement still suffers from the lack of resolution of line printer graphics.

As another example, on the mainframe we have used the DISSPLA package to obtain high resolution graphics-these plots are shown in Figures 1 to 6 and should be compared to the identical line plotter figures in Lewis and Orav (1989, Figures 9.6.1 to 9.6.6). The improvement is quite clear and the problem right now is to find a suitable package on the microcomputer to refine the graphics.

Joint properties of estimators are also of interest and the issue here is the usual one of compactification. In Figure 7 we show an attempt to incorporate the factors

334

of sample size and different statistics into a single plot. Each cluster of scatter plots is a draftsman's plot of the scatter plots shsowing the relationship between the statistics a t a given sample size. The effect of sample size is seen, either singly or jointly, by looking along the diagonal.

3. THE BOOTSTRAP SIMULATION

Bootstrapping ixj an important new idea for assessing variability in statistical situations and in simulations. Figures 1 to 6 are equivalent to those in Lewis and Orav (1989), as mentioned above, and that book should be referenced for detaills. The simulation experiment is an extension, with graphics, of results given in Efron and Gong (1983).

Briefly one can see (Figure 2) that the Jackknife es- timate of the standard deviation of an estimate of the correlation coefficient in a bivariate normal sample is more variable-and has bigger outliers-than the nor- mal theory estimate (Figure 1). The bootstrap estima- tor (Figure 3) is better than the jackknife estimator and the bootstrap estimator improves as B, the number of replications in the bootstrap simulation, increases from 128 (Figure 3) to 512 (Figure 4 ) , although the improve- ment is slight. Again, the Normal-smoothed bootstrap gives even more improvement (Figures 5 and 6) .

4. SUMMARY AND CONCLUSIONS

Other possible (enhancements and extensions of the simulation test bed will be discussed in the talk.

335

t P . P i

0.3M3 0.21Z 0.1771 0.1431 0.1411 0.1311 0.1215 0.1110 0.1115 O.MIS3E-01 0.391Q-01 0.2woE-0I 0.23Z-01 O.WJX-01 0.2299C-01 O.!SSEQI nnu si0 570 XR* 0.8427E-02 0.6053E-02 0.46wE-02 0.40945-02 0.354UC-02 O.XLUC-02 0.4130E-Ol 0.Z33Z-m

X W S S -0.63s -O.X?S -0.5838 -0.8l8Ot-OI -0.6368 -0.2597 -0.6997 -1.m KL~TCSIS -0.5ml -0.9571 0 . ~ 7 9 -0.w 0.~218 -0.01s O.SWC-OI 0 . m

Figure 1: SIMTBED runs to investigate the sample properties of the Normal theory estimate of the standard deviation of the correlation estimate in a bivariate Normal sample. The estimate is (1 - > 2 ) / ( m - 3)''2. The true value of p

is 0.5.

336

LWa)

,

3 I 3 . 1

I

1 6 0

I 1

I

4

I 4 W 16 ?a 5(1 44 y1 5LDsRN SIX

0.w o.mi 0 . m a.is:s o . ! c s 0 . 1 ~ 4 o.im o.im 0 . l d 0.84OEEQI 0.637E-01 0.64>25-01 0.33PE-OI 0.2B51E-01 0.23WQl 0.16%Z-O1 m SI0 ST0 M 0.12111:Q1 O.BtC5E-02 t . 7 6 1 X a 0.89622-02 0.517C5-02 0.4768E-02 0.53JtT-02 0.3130E-CQ

S K W S S 0.9923 0.6315 0.3525 W T O J I J 1.- 0.5818 -0.101s

1.421 0.1258 0 . 4 W 0.7734 4 . 2 3 7 4 1.W -0.1138 0.134 2.079 -0.lmS

Figure 2: SIMTBED runs to investigate the properties of the jackknife estimate of the standard deviation of an estimate of the correlation coefficient in a bivariate Normal sample ( p = 0.5). The estimate whose standard deviation is being investigated is actually the estimate, j, after complete jackknifing, as a t Equation 9.5.1 in Lewis and Orav (1989), to remove the bias term of order l /m.

337

I I 1 I

I

t

a.= 2 . 1 m 0.1746 0 . 1 1 ~ 6 o . 1 ~ o.iw 0.1160 0 . 1 0 ~ O.!C53 O.6253E-01 0.51765-01 0.173OE-01 0.2S78E-01 0.BIIC-01 0.2708T-Ol 0.157SC-Ol FA STC 5m nEM 0.795EdI 0.653:-W 0.642EE-07 0.6407E-02 0.154.E-02 0.416-X-01 0.41164L-01 O.ae5S-a

SKWESS -0.1153 0.2901E-01 0.1166 0.6375 0.703s:-01 0.2- 0.6836C-01 0.2121 C U V O S l f O.lOSZ-01 -0.4385 -0.8128 0.2160 -0.33SBE-Ol - 3 . 7 W - 0 1 0.3010 1.W

Figure 3: SIMTBED run to investigate the properties of the standard bootstrap estimate of the standard deviation of the correlation ,b in a bivariate Normal sample ( p = 0.5) for B = 128 (number of replications in the bootstrap simulation)

3.38

I 3 1

P 6 1

0.2915 0.1958 o.1m 0 . 1 m 0.1367 0.1253 o.lln 0 .1m 0.1% 0.52792-01 O.SlS2Z-01 0.4692-0l 0.25lSE-01 O.2399E-01 0.2516E-01 O.IH2Z-Ol m

S i 0 SD F?RN 0.7577!-@2 0.52iSh-02 0.6153E-LU 0.614754 0.4293E-02 0.3BIE-02 O.4E99E-02 0.2913E-02 SKCUNCSS -0 .75w-01 0.179tL-01 0.1806 0.7522 -0.5131E-02 0.X26E-01 0.3110 -0.19lB K I T m l S -0.icEi -0.1782 -0.3751 0 . W -0.2110 -0.EBD O.lI1S 0.714IC-01

Figure 4: SIMTBED run to investigate the properties of the standard bootstrap estimate of the standard deviation of the correlation /j in a bivariate Normal sample ( p = 0.5) for B = 512 (number of replications in the bootstrap simulation)

339

6 1

I I 4 zo 26 32 34 1 Y)

O.ZI9 0.20?9 0.1721 O.I4z(1 0.13BE 0.1233 0.111 0.lm 0.845lC-31 0.4979:-01 0 .3660-01 0.3027E-01 0.2514E-01 C.2116E-01 0.2297E-01 O.IiBBE-0l KRH 5m

SCNICSS -3.8325 -C.1108 -0.6800 -0.IClI -0.4202 -0.2393 -0.5322 - 1 . 3 3 KU9TOSIS 0.316zE-07 - 0 . 7 5 9 0.7228 -1.093 -0.3701 Q.!I44 -O.@X?C-Ol 1.383

sm ERN o.s=-cu o . ~ ~ ~ o c - u o . w ~ - 0 7 o.+isec-o2 O . I E ~ ~ E - U o.msc-02 0 . 1 1 2 ~ ~ 2 0 . 2 ~ i z f - m

Figure 5 : SIMTBED run to investigate the properties of the Normal-smoothed bootstrap estimate of the standard deviation of the correlation (Here the number of replications in the bootstrap simulation is b = 128.)

in a bivariate Normal sample ( p = 0.5).

340

SIBS= SIZE I 4 2c zb ?a 44 so

0.~111 0 . 2 ~ 9 o.im O . I ~ o . 1 4 ~ o.im 0.1206 o.iim

KUiOSIS 0 . l l l S -0.9028 0 . m Q.Wl 0.2(155 -0.37Q 0.1074 1 . a

0.g494E-01 0.5203’-01 0.1492-01 0.28lZE-Ol 0.7167E-01 O.lW!C-0I 0.1147E-01 0.l42SC-01 ICm

Sm WEE 0.0421542 0.SX)E-02 0.4174E-07 0.386P-01 0.33ME-(U 0.3168E-(U 0.385SE-02 0.---02 Y W E S S -0.578 -0.3156 -0.6647 -0.1W -0.7363 -0.72725-01 -0.6739 -1.310

sm

Figure 6: SIMTBED run to investigate the properties of the Normal-smoothed bootstrap estimate of the standard deviation of the cosrrelation p in a bivariate Normal sample ( p = 0.5) . (Here the number of replications in the bootstrap simulation is b = 512.)

34 1

. . . .

._ C A

-.. j .

. -.. .. .. . . .. 2 1 ,

Figure 7 : Experimental graphics to show the correlation and dependence between various statistics, and to show the change in the dependence as a function of sample size.

342

ACKNOWLEDGMENTS

The research of l’.A.W.Lewis was supported a t the Naval Postgraduate School under an Office of Naval Re- search Grant.

REFERENCES

Efron, B. and Gong, G. (1983). A leisurely look a t the Bootstrap, the Jackknife and Cross-Validation. Ameri- can Statistician 37, 36 - 48.

Lewis, P. A. W., Orav, E. J., Drueg, H. W., Linnebur, D. G. and Uribe, L. (1985). An implementation of Graph- ical Analysis in Statistical Simulations. International Statistical Review 53, 69-90.

Lewis, P. A. W. and Orav, E. J. (1989). Simulation Methodology for Statisticians, Operations Analysts and Engineers. Wadsworth & Brooks/Cole, Pacific Grove, California.

Lewis, P. A. W., Ora.v, E. J. and Uribe, L. (1989). En- hanced Simulation and Statistics Package. Wadsworth & Brooks/Cole, Pacific Grove, California.

AUTHOR’S BIOGRAPHY

PETER A. W. LEWIS is Distinguished Professor of Statistics and Operations Research in the Department of Operations Research a t the Naval Postgraduate School in Monterey, California. He holds an A.B. degree (1954) from Columbia University, B.S. and M.S. degrees (1955 and 1957) from Columbia School of Engineering, and a Ph. D. in Statistics from the University of London in 1964. He is co-author, with D.R. Cox, of the book ‘The Statistical Anal,ysis of Series of Events’ (1966), ed- itor of the Volume entitled ‘Stochastic Point Processes: Statistical Analysis, Theory and Applications’ (1972) and co-author, with E.J.Orav, of the book ‘Simulation Methodology for Statisticians, Operations Analysts and Engineers’ (1989). He has also published a software package ‘Enhanced Simulation and Statistics Package’ (1989) with E.J.Orav and L.Uribe. He is the author of over eighty refereed papers in technical journals, and is a Fellow of the American Statistical Association and the Institute of Mathemartical Statistics.

Peter A.W. Lewis Code 55LW Naval Postgraduate School Monterey, CA 93943 (408) 646-2283

343