One Possible Frame for Thinking about Experiential Learning

One Possible Frame for Thinking about Experiential LearningAuthor(s): George W. CobbSource: International Statistical Review / Revue Internationale de Statistique, Vol. 75, No. 3(December 2007), pp. 336-347Published by: International Statistical Institute (ISI)Stable URL: http://www.jstor.org/stable/41509874 .

Accessed: 16/06/2014 10:12

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact [email protected].

.

International Statistical Institute (ISI) is collaborating with JSTOR to digitize, preserve and extend access toInternational Statistical Review / Revue Internationale de Statistique.

http://www.jstor.org

This content downloaded from 185.44.77.62 on Mon, 16 Jun 2014 10:12:04 AMAll use subject to JSTOR Terms and Conditions

http://www.jstor.org/action/showPublisher?publisherCode=isi

http://www.jstor.org/stable/41509874?origin=JSTOR-pdf

http://www.jstor.org/page/info/about/policies/terms.jsp


International Statistical Review (2007), 75, 3, 336-347 doi. lO.l 1 1 l/j.l751-5823.2007.00034.x

One Possible Frame for Thinking about

Experiential Learning

George W. Cobb

Mount Holyoke College, South Hadley, MA 01075 USA. E-mail: [email protected]

Summary I argue that teaching statistical thinking is harder than teaching mathematics, that experimental

design is particularly well suited to teaching statistical thinking and that in teaching statistics, variation is good. We need a mix of archival data, simulations and activities, of varying degrees of complexity. Within this context, I applaud the important contributions to our profession represented by Darius et al. (2007), and Nolan & Temple Lang (2007), the first for showing us how to make simulation-based learning simultaneously more flexible and more realistic than ever before, and the second for showing us a path-breaking technology that can make archival data the basis for active learning at an impressively high level of sophistication, embedding statistical thinking within real scientific and practical investigations.

Key words : Education; experimental design; experiential learning; transfer; abstraction; simulation; activities.

1 Introduction

The economist Paul Krugman (1994, preface) recounts a parable of karma and reincarnation in economics: the careful, rigorous, intellectually virtuous economist is rewarded in his next life by reincarnation as a physicist; the careless, loose-thinking, intellectually slovenly economist is punished by reincarnation as a . . . (gasp!) sociologist.

Plato would not have approved of this parable. In the curriculum he prescribed for philosopher kings, Plato had candidates start with a decade (age 20-30 years) devoted to the study of mathematics, a subject that would outrank even physics in Krugman 's parable, but which Plato regarded as the easiest subject, good mainly for preparing young minds for the harder challenge of subjects like sociology and political science, subjects better reserved for the mature intellect.

As statisticians, we should resist physics envy and embrace Plato. Our subject, because it deals with real problems, is harder in Plato's sense than mathematics, albeit perhaps easier than sociology. In particular, those of us who teach statistics can recognize the truth of Plato's thinking. Teaching statistics as mathematics is comparatively easy, but teaching true statistical thinking is hard. It is in this context that I applaud the authors of the two papers - Paul Darius, Kenneth Portier and Eddie Schrevens (from now on DPS) and Deborah Nolan and Duncan Temple Lang (NL) - for addressing head-on the challenge of how we can help our students to grapple with the things that make statistics much harder than mere recipes and theorems. In what follows I group my comments under seven headings: After a first section that presents two general goals for learning in statistics courses, I argue in a short section that teaching design of experiments is

© 2007 The Authors. Journal compilation © 2007 International Statistical Institute. Published by Blackwell Publishing Ltd, 9600 Garsington Road, Oxford ОХ4 2DQ, UK and 350 Main Street, Maiden, MA 02148, USA.



One Possible Frame for Thinking 337

particularly suited to advancing those goals. Section 4 offers a way to classify data sets by their sources; Section 5 amplifies on this classification, summarizes with a two-way table, locates DPS and NL within it and uses the table for a brief fly-over. The two penultimate sections, 6 and 7, give some concrete ways to make effective use of two possibly undervalued cells in the table. Finally, Section 8 offers a summing up.

2 Goals for Learning: Transfer and Abstraction

It has become a commonplace that both the range of things to learn and the range of ways to learn them are expanding rapidly, perhaps more rapidly than at any time in history. Amidst all this change, I suggest, we can find comfort in recognizing that, Krugman's parable notwithstanding, our sense of what it means to be well educated seems essentially the same now as it was at the time of Plato's Republic. The content of our thinking may be vastly different, but on the most basic level, the way we think about that new content has not changed much. Such stability suggests that one usefiil approach to the question of what we want our students to learn in our statistics courses is to embed statistical thinking in the context of broader goals for learning based on an enduring sense of what matters most. Accordingly, I shall not try to define statistical thinking, or to characterize it in any but the most general ways, by asserting that in thinking about what we want students to learn in our courses, we would do well to focus on two main goals, transfer and abstraction.

Accordingly, one theme of my comments is the challenge of transfer - how can we enable our students to transfer statistical thinking from the small handful of particular applied contexts they see in class to the myriad other, different contexts they may encounter later on? For concreteness, consider two extreme examples from a course in design of experiments:

Example la. Computing SSwilhin far a one-way balanced design. Typically, this is taught by formula, e.g., J2¿=i J2j=i (Уи ~ Ü'-)2- Here transfer is easy. Students need only learn to plug numbers into placeholders. The size of the groups may change from one context to another, the number of groups may change, but neither change is subtle, and a single numerical example is typically all students need to master the formula.

Example lb. Recognizing blocks and units in a balanced complete block design. Here transfer is hard. Students need to learn about blocking by subdividing, as in Fisher's varieties of wheat, where plots of land are partitioned into sub-plots that serve as units. Students also need to learn about blocking by grouping, as in a matched pairs design, and about blocks created by reusing, as in a repeated measures design. Many instances are ambiguous: if I am a block, with my two eyes as units, am I being subdivided, or are my eyes being paired?

Because the concepts of block and unit are too hard to capture in simple mathematical definitions, it takes many, many examples to reach understanding. (To this day I remember how shaken I was after sitting for a midterm in William Cochran's introductory design course. I'd already breezed through a graduate course in design at the Medical College of Virginia, and I went into Cochran's exam confident I knew what I was doing. The resulting observed data ruthlessly overrode my puny prior.)

The concept of a block nicely illustrates the challenge of transfer. We want our students in a design course to recognize blocks wherever they might arise, regardless of context, regardless of whether created by matching, subdividing or reusing. (Although this notion of transfer comes

International Statistical Review (2007), 75, 3, 336-347 © 2007 The Authors. Journal compilation © 2007 International Statistical Institute



338 G. W. Cobb

from 20th century theories of learning, note that it is not very different from what Plato might have meant had he said that we want students to understand the block as an ideal form.)

Transfer is particularly hard in statistics because of the essential tension between pattern and context, a tension that is largely absent in pure mathematics. (I heartily agree with DPS: "one of the vital skills of the designer is being able to select the relevant parts from the context.") As I have written elsewhere before, although mathematicians rely on applied context as a source of research problems, and as a way to build intuition, ultimately, pure mathematicians are studying patterns. Context is part of the irrelevant detail that must be boiled off over the flame of abstraction to reveal the resulting crystal of pure structure: In mathematics, context obscures structure. Plato's forms are at their starkest in this "easiest" of subjects. By contrast, context is an essential part of statistical thinking, and some of the worst teaching of statistics occurs when the teacher or textbook tries to treat context as irrelevant. Although statisticians seek patterns in numbers, and rely also on abstract mathematical logic, ultimately statisticians are studying data, for which context is essential. What patterns mean, and indeed, whether they mean anything at all, depends on context. It follows that in data analysis there can be no such thing as an isomorphism. Every data set has its own unique context, and its own special features. No two are alike. Transfer is hard.

If we accept that we want students to learn in a way that allows them to transfer thinking learned in one applied context to other contexts, then, I suggest, we should make it a major goal to teach abstraction-as-process. "Abstraction" by itself is not precise enough. The two examples from before serve to illustrate why. Example la, computing the sum of squares, is an example that might be called "abstraction-as-template" or "abstraction as pattern-matching". The essential feature is that the template serves as a way to avoid thinking. When well-used, a formula can be a compact fossil record of the once-living thought process that brought the formula into being, but for most of our students, most of the time, it is all too easy to treat a formula as a vehicle for intellectual hitch hiking: you climb aboard, cast off any residual thought that anchors the formula to meaning, and drift about in the aimless hope of running aground on "the answer". Successful use of a formula amounts to pattern matching: treat all problems as isomorphs, and plug the proper numbered pegs into the corresponding holes. Traditional "top down" teaching in mathematics is often just a less-extreme form of abstraction as pattern-matching. You start with definitions and theorems, then look at one or two or three essentially isomorphic examples. The main point of the examples is to teach you to strip away the husk of context in order to get to the identical little golden kernels of abstract value inside.

By contrast, Example lb, recognizing blocks of units, involves abstraction-as-process. Al- though we teachers or our textbooks may give a definition of a block (e.g., "a set of similar units"), no definition can possibly prepare students to recognize blocks except in the simplest situations. Examples are essential. Many, many examples. The concept of a block emerges from extended practice, much as a sculpture emerges from a lump of clay, at first only in roughest outline, but refined by time, mind and eye, with increasing clarity and precision.

Although the concept of a block illustrates one instance of abstraction-as-process in statistics, no single instance can show how potentially universal this goal can be, if only we choose to make it so. Once again, a contrast with mathematics is useful, and for concreteness, I will use beginning courses for my examples, but I find the same differences to be present at all levels. Imagine first, a beginning calculus course. Many exercises involve computing derivatives using short-cut rules, or perhaps finding derivatives by taking limits. In these exercises, the thinking remains at one fixed level of abstraction, and involves working with symbols unburdened by any pretense in the direction of applied context. Proofs may be more challenging, but they, too, float at a high level of abstraction serenely remote from any earthbound applications. Beginning calculus students tend to get their biggest intellectual challenge from the dreaded "word problems" in





those comparatively short parts of the course that treat max-min and related rates. These are the two areas in a first-year calculus course that offer experience with abstraction-as-process, but even here, it is possible to identify a handful of archetypes and use them as templates, an approach that squanders any chance of stretching the mind.

In a good first statistics course it is impossible to succeed by mere pattern-matching. We may be able to offer students some useful abstractions as guides, along the lines of David Moore's thoughtful and pithy "plot, shape, center, spread" for univariate data sets, but ... no two plots are alike, there's no short catalog of shapes to refer to. In sum, there are no isomorphisms. An abstract understanding of what to look for and how to make sense of the patterns is something the student must build, painstakingly, from examples. Similar opportunities to learn via abstraction-as- process occur throughout the first course in statistics, and indeed throughout the entire statistics curriculum. In all of applied statistics, it is essential to work with the very concrete, because that is where you find the context that is your source of meaning. It is equally essential to work with the much more abstract, because, to borrow from Willy Sutton, that is where the patterns are, and also where the transfer occurs. Data sets are concrete, statistical models are abstract, and every instance of modeling is an instance of abstraction-as-process.

At the risk of oversimplifying, I suggest a two-dimensional model for transfer and abstraction- as-process: My goal in every course is for students to see each concrete example as an intersection of complementary sets of threads in a tapestry of meaning, with the structural, load-bearing threads provided by the abstract concepts - blocks, units and the like - and the complementary threads of context providing the meaning, richness, color and story line. While granting that every metaphor should come packaged with a warning label about the dangers of reification, I want to suggest a consequence of this two-dimensional view of transfer: To help students see each example as part of an integrated whole, we need to encourage their imaginations to reach out in both directions from each example, following the structural threads, as they cut across contexts while holding the concepts fixed and also following the complementary contextual threads, to see how a variety of concepts interweave through the same applied context. Sec- tions 6 and 7 give explicit examples.

As promised, I have not attempted to define statistical thinking, but I am confident that any instructors who work hard at transfer and teaching abstraction-as-process will do well by their students, and I find these two goals useful for evaluating contributions to statistics education. Transfer is what makes learning useful; without it, our reach will ne'er exceed the grasp of whatever few examples our teachers may have chosen to hand us. Abstraction-as-process is our intellectual callisthenic - the way we develop our capacity for transfer, the way we extend our reach - and so, abstraction-as-process should be a principal focus of all education.

Judged by these criteria, NL and DPS are important contributions that create new opportunities to help our students learn the things that matter most.

3 The Importance of Design

As DPS rightly emphasize, the ability to design experiments is "critical to modern science". If you work with data, the quality of your conclusions depends on the quality of your data, and the quality of your data depends on the quality of your planning. In particular, the entire scientific edifice rests on a layer of randomized data production, the mortar that binds observed data to the foundation in probability that supports any formal inferences. This logic alone ought to be enough to ensure a bigger place for design in the statistics curriculum than it currently enjoys. In support of this claim, consider a zero-sum tradeoff between design and analysis from the two viewpoints of user and producer. First users: Most students who take the introductory statistics




340 G. W. Cobb

course will be users of data, in the sense that their roles as voters and spenders will be shaped by their understanding of data-based conclusions presented by the media. Which is more important for such users: to know how to do a Mest on a properly randomized study, along with the rest of the first-course technical arsenal, or to know how to evaluate studies based on their designs? (For a one-syllable answer, consult Homer Simpson.) Now producers: Many graduate students who take a service course in statistics because they will be doing empirical work for their dissertation find themselves in a course that emphasizes analysis (e.g., if they are fortunate, a course based on a book I admire, Ramsey and Shafer's Statistical Sleuth , 2002). But consider: which kind of course (despite my admiration for the Sleuth) would serve these students better: a course on how to analyze data from an experiment designed by someone else, or a course on how to design experiments, even if the emphasis on design means shortchanging analysis?

I regard my last pair of questions as largely rhetorical, their answers a foregone conclusion. Why, then, in the typical post-"reform" introductory statistics course, are issues of design still such a small part of the curriculum? Why also, at the level of graduate service courses, does a design course often come late, only after at least one course on analysis? Although I can offer nothing more than a strongly held conviction, here it is: Like mathematics and dying, teaching analysis is easy; like sociology and comedy, teaching design is hard.

The reason design is so hard is precisely the reason why we ought to be teaching a lot more of it. Just as few, if any, other academic subjects put the tension between pattern and context front and center the way statistics does, few if any other areas of statistics put that tension front and center the way design does. Statistics generally, and the design course in particular, provides an excellent environment for learning abstraction as process. The concept of a block is a particularly good example for seeing this, but others abound. Abstractly, the difference between crossing and nesting might seem clear, but if, like I, you teach design as a first course to students who have not taken statistics before, it can be surprising how hard they find it to distinguish between the two. (My experience suggests that because of where design ordinarily appears in our curriculum, we tend to underestimate how hard the distinction can be.) Another instance: The difference between fixed and random effects is truly hard, I think. Even for graduate students who have taken some statistics already, that difference can be confusing at first. Yet more: Interpretation of mixed designs, or diagramming factors for multilevel designs, presents even harder challenges. With these and many other concepts, mastering the Platonic ideals is hard, requiring multiple examples and careful thought. Transfer is hard. Abstraction-as-process is essential. Had Plato lived long enough to meet Fisher, he might have modified his curriculum by inserting experimental design between mathematics and the social sciences.

Even though it is implicit in my comments and might go without saying, I will say it anyway: DPS made an important decision when they chose design as the area for their computer simulations.

4 Data sources: Variability is Good

I find it useful to think in terms of three main sources of data: archives, simulation and hands- on activities. Each source has characteristic advantages and disadvantages, and I will argue that it is good pedagogy to rely on data from all three sources.

a. Archival data, by which I mean data already collected in advance by someone other than the students, offer the advantage that the time-consuming work of planning the study and collecting the data have already been done by someone else. At the same time, archival data offer the major disadvantage that the critically important work of planning the study and collecting the data have already been done by someone else. Students are not able to participate in deciding what data to collect; at best, they can only read about the design and discuss its strengths and





weaknesses. A secondary disadvantage is that students may tend to regard archival data like hand-me-down clothes, and feel less interested in a data set they did not help to choose or create. This means that the teacher must devote extra thought, effort and class time to creating a sense of student ownership. A tertiary disadvantage is that real data, like preservative-free baked goods, has a limited shelf life. The overriding advantage of archival data, potentially more than enough to outweigh the disadvantages, is that students can work with data analysis and design issues of great intrinsic interest or unmistakable import, without having to worry about getting informed consent from human subjects, writing grants to pay for equipment, or waiting years for a longitudinal study to run its course.

The example of spam detection in NL illustrates some of these features of archival data, but also illustrates how thoughtful planning can moderate some of the inherent disadvantages. Import : Anyone who uses e-mail will have no trouble recognizing the relevance of the spam study. Design : Although design is not meant to be prominent as an issue in this example the way it is in the DPS examples, all the same the spam archive has already been chosen, which means that, as with other archival data sets, students do not have the option of designing their own study. However, thanks to the thoughtful way NL structure things, students do have the opportunity to create new variables and add them to the data base. Ownership : I would expect the spam example to do better than many archival data sets when it comes to student involvement, first, because the topic is well chosen to be relevant to students' lives, but also because NL have been thoughtful about building in ways for students to make the data their own. Shelf-life: It is with deepest regret that, in attempting to assess future relevance, I predict a long shelf life for this example.

b. Simulated data, whether created by computer as in DPS or by physical simulation, can never be fully authentic, and that lack of realism is the main shortcoming of simulated data. (DPS: "virtual experiments lack the reality of real project experiments.") There is a corresponding burden on the teacher and students to employ their imaginations to bridge the gap between the artificial data and the real situation it represents. However, as DPS have shown, a thoughtfully conceived interface such as the drag-and-drop feature of the tomato plants in the greenhouse experiment can do a lot to aid the imagination. Simulation offers the two main advantages of quick data production and easy repeatability. Quick production means that once a student has chosen a design, getting the data is essentially instantaneous. This reduces the time it takes to do assignments based on simulated data, and can serve also as a form of quick positive reinforcement. Repeatability can be valuable, as DPS suggest, by allowing students to try out different designs using the same mechanism for generating the data (more in Section 7) in order to see which ones work better. Repeatability also allows students to experience first-hand the conceptually difficult connections that link single observed values, their probability distribution and the underlying chance mechanism. For example, many textbooks now use a picture of 100 simulated confidence intervals to illustrate coverage probability.

c. Activity-based data have neither the import of well-chosen archival data, nor the quick and easy repeatability of computer simulation. All the same, the widespread adoption of activities from Richard Scheaffer's influential, NSF-supported project "Activity-Based Statistics" (Gnanadesikan et al., 1997) speaks to the effectiveness of hands-on participation as a way to generate and maintain student interest. Evidently, the main advantage of activity-based data is the sense of ownership that active participation gives our students. A second important advantage is that some activities offer opportunities to think about issues of design. (See Section 7.) DPS mention George Box's paper helicopters (Box, 1992) as a pioneering instance in the spirit of their own response surface example.

Because the three sources of data enjoy complementary advantages, and each source has inherent shortcomings, I suggest that as a general principle , we should plan our courses to offer students a mix of all three sources (Table 1). Students need to see authentic data sets that




342 G. W. Cobb

Table 1 Three sources of data, with natural advantages and disadvantages.

Source Import Repeatability Ownership Planning / Production Archives -f+ - - - Simulation - /+ ++ +/- +/- Activities - +/- ++ +/-

Table 2 A two-way table for experiential learning.

Complexity Simple Intermediate Elaborate

Textbook examples and NL spam lab Archival exercises Stat Labs Large rich data sets in a

Blocks (Section 6) Kuiper Stat2Labs consulting-based course Case studies (Sleuth)

Source Simulated Confidence intervals DPS labs - Martin v. Westvaco

Activity-Based Statistics Box Helicopters Activities A Bag of Tricks Gunter Funnel Term projects

Sit/Stand (Section 7) Confectionary Ballistics

really matter, like the NL spam example, even if that means they cannot participate actively in designing the study or gathering the data. Students also need to get their hands dirty, producing their own data and choosing their own designs, even if that means working with "toy" scenarios. Well-planned simulations, like the two examples in DPS, can provide a compromise of sorts, offering far more participation in design and data production than archival data sets offer, while coming closer to virtual import than many hands-on activities can.

For a parallel, imagine learning a language. It is important to listen to native speakers, even if that can only be done by watching a movie or playing a CD. It is also important to practice routine repetitive drills of grammar and vocabulary, even though you will never be expected to conjugate verbs out loud on your next trip abroad. Finally, it is important to participate at a language table and try to converse, even if that means halting sentences and garbled grammar at first. Over time, with persistence, you become fluent.

5 Complexity: Variability is Good

Classifying data by source helps to identify some characteristic strengths and weaknesses, but within each of the three classes, there is a wide range of complexity, from the very simple to the very elaborate, with a corresponding range of potential uses for teaching. (Table 2 presents a summary.) I don't propose to define simple or elaborate, except by example. Suffice it to say that I intend the words to mean pretty much what you would expect. Simple instances have comparatively few details, are quick to present, quick to complete, and, if grading is called for, comparatively quick to grade (though grading can never be quick enough!). If examples are simple, it should be possible to present or assign several per class meeting. At the other extreme, I think of term projects as elaborate, with a size and scope that would make it almost prohibitively ambitious to ask students to do more than one per course. As I understand NL and DPS, the spam lab and the two design labs are intermediate between simple and elaborate: students would do more than one per semester, but probably no more than a small handful.





For archival data, textbook examples and exercises sit at the simple end of the range. Extended examples, sometimes called case studies, as in The Statistical Sleuth (Ramsey & Shafer, 2002), are more elaborate (only two per chapter in the Sleuth ), and the case studies in Nolan & Speed's Stat Labs (2000) and Shonda Kuiper's Stat2Labs (2005) are even more elaborate, though not as elaborate as the spam lab in NL. At the far extreme, a data set can be so large and rich that it serves as the basis for an entire semester-long course, as can happen in some consulting-based courses.

For simulated data, the simplest instances have a narrow focus on a single concept, and can be presented or experienced quickly. A set of 100 simulated confidence intervals, for example, has a focus on the narrow albeit important concept of coverage probability, and can be demonstrated in a matter of minutes. In the same spirit, a simple simulation lasting only a few minutes can be used to illustrate the /7-value from a permutation test, as in the example of Martin v. Westvaco (Watkins et al ., 2007, pp. 1 1-20.) The DPS industrial and greenhouse examples illustrate a much more elaborate use of simulated data. All the same, despite their richness, I regard them as intermediate, in that it would not be at all ambitious to do two or more of these in course. I'm not aware of any simulation-based data sets or scenarios so elaborate as to take up an entire course, and I find it hard to imagine why one might want such an example or such a course.

Activities for producing data can take anywhere from a few minutes to an entire semester to complete. At the simple end are activities of the sort found in Activity-Based Statistics (Scheaffer et al ., 1 997) and Teaching Statistics : A Bag of Tricks (Gelman & Nolan, 2002). Term projects (see the references in DPS) are at the other end of the scale. In between, there are activities like George Box's paper helicopters (1992), that would take a couple of hours or so to complete. (Two other examples of roughly the same size are Berton Gunter's funnel experiment (Gunter, 1993), and an investigation of confectionary ballistics - launching soft candy bears from a simple catapult (Cobb & Miao, 1998).

Independent of data source, there are characteristic advantages and disadvantages associated with each level of complexity. Simple examples have the advantages that they can be done quickly, and can be sharply focused, but the sharp focus comes at a cost. A set of 100 simulated confidence intervals can have a sharp focus on coverage, but the level of abstraction is fixed, and at a comparatively high level. By contrast, well-conceived examples of a more elaborate sort, as in NL and DPS, can ensure that students relate a concrete applied context to more abstract methods and models, and so such more elaborate examples are more effective in teaching abstraction as process.

It would be a mistake, however, to conclude that more elaborate examples are necessarily better. I like to rely on such examples, but I don't feel that when I rely on shorter, more traditional examples, I should be apologetic for relying on inferior pedagogy. On the contrary, I think a strong case for such simpler examples can be based on the ideas about transfer that are captured in the metaphor of a tapestry. Because simple examples are quick, you can afford to do several; because they can be narrow in focus, they can offer the instructor more control and flexibility. By way of analogy, think of elaborate examples as modular housing, and simple examples as building blocks. Careful choice of simple examples lets you follow a single concept through several very different contexts, or to revisit a single recurring context as the course progress through a succession of concepts.

The next two sections are intended to show one way this can be done. Section 6 uses the example of a block to illustrate a set of generic exercises that can accommodate a large variety of contexts. Then Section 7 uses a simple, hands-on experiment to illustrate many concepts in a single context.




344 G. W. Cobb

6 Blocks of Units: One Concept, Many Contexts

DPS, in their introductory remarks, write about exercises for which "obtaining the data is the endpoint" rather than the starting point, saying "the classical book format is not well suited for such exercises". If they mean only that books are not a good vehicle for generating pseudo- random numbers from a model, something for which the computer is ideal, I completely agree. If they also mean that "many books with 'design' in the title don't really teach design; they teach ways to analyze experiments designed by someone else", I couldn't agree more if I had written the words myself, as in fact I did. However, it is also important to recognize that just because many textbooks focus on analysis at the expense of design, this unfortunate and misplaced emphasis is not a requirement for writing a book. Indeed, valuable as computer simulations can be, the right kinds of paper-and-pencil exercises from a book can offer important advantages over simulations for certain kinds of learning about experimental design. In particular, textbook examples and exercises tend to be shorter and quicker than are computer simulations, which means students can do more of them at a sitting, which in turn means that textbook exercises can promote transfer and abstraction-as-process by leading students to think about a single concept in a variety of contexts. Once more, although I rely on design for examples, I think of those examples as illustrating a more general use of similar examples to promote transfer across the statistics curriculum.

Here are some sample exercises of the sort I have in mind, taken from a textbook for a design course (Cobb, 2002). None involve any computations or data analysis. For each, there is a simple abstract structure (the "stem") that can easily accommodate an arbitrarily large number of authentic studies as concrete instances.

Example 2a. "Each of the examples listed below has blocks as a nuisance factor, (a) For each, name the nuisance factor, (b) Tell what the unit is. (c) Tell how the units and blocks are related: by . . . grouping . . . subdividing . . . reusing?" (p. 250.)

(After this generic stem, one can list or describe a variety of instances of block designs, without using the words "block" or "unit" in the description. For example: In a study to compare the effect of city living on human lungs, investigators found seven remarkable pairs of twins: (1) the twins were identical, (2) with one twin in each pair living in the country, the other in the city, and (3) all 14 individuals were willing to inhale an aerosol of radioactive Teflon particles. This last allowed the investigators to measure the percentage of radioactivity remaining an hour later, as a way to compare lung function for the two kinds of environments.)

Example 2b. "None of the examples below uses blocks as a nuisance factor in its design. For each I have listed a nuisance influence. Give a good reason why the influence was not made a factor of the design." (p. 250.)

(After this stem, one can list a number of instances of designs without blocks, e.g., completely randomized designs, together with a potential confounder or source of variability that might on the surface seem suitable for creating blocks, but for some important reason was not. For example, in a learning experiment, subjects are a source of variability, but cannot serve as blocks because they cannot be reused.)

Example 2c. "Tell how you could run the experiment in the last problem [a split plot repeated measures study of hypnosis and learning, with conditions of interest as a





between-subjects factor] with the same response, the same set of subjects, and the same set of conditions to compare, but using a randomized complete block design: tell what the units and blocks are, and how many units per block you would have." (p. 300.)

I also include examples and related exercises that appear on the surface to be based on block designs, but in fact are not. For example, a study of feeding behavior in frogs measured the amount eaten by each of 12 frogs under three conditions: dining solo, dining tête-à-tête with a companion frog, and competing at a communal trough along with all 1 1 of the other frog subjects. A standard rectangular format for the data makes the experiment look like a complete block design, with one row of data for each frog, one column for each of the three conditions, but of course the actual design is not so simple, (p. 363 ff.) Note that although this can be a useful example for learning, it would not be worth the effort to turn it into a computer simulation.

Here, as elsewhere, although I've used blocking as an example, one can do the same sort of thing with other concepts. Here is a generic stem for exercises in the same spirit, this time with a focus on factorial crossing:

"Each situation below involves a response and a factor of interest. For each, find a second factor that might be of interest. There is no single right answer, but try to find factors that are likely to be important, given the context of the experiment." (p. 374.)

To conclude this section, I want to be clear: None of what I have said here is meant to suggest that I am anything other than enthusiastic about the computer simulations in DPS. At the same time, I do think we serve our students best when we allow them to experience the comparative strengths of archival data, of simulated data, and of activity-based data, using examples that range from simple to elaborate.

In this section I have illustrated the use of simple archival data to present a single concept in multiple contexts. In the next section I illustrate the use of simple activities to present a variety of experimental designs in the context of the same recurring applied scenario.

7 Sit or Stand: One Context, Many Concepts

Certain short activities lend themselves to multiple uses. Just as it is important for students to see hard ideas like blocking applied in a large number of contexts, and it is sometimes important for students to see multiple analyses of the same data set, it can be useful for them to see a variety of designs in the same recurring applied context. A well-planned computer simulation of the sort described in DPS can sometimes make this a natural and easy thing for students to do, especially if the instructions explicitly ask for multiple designs.

Yet for other purposes, elaborate computer simulations may not be practical - they may take too long or require more computers than are available if done in class, or they may entail too slow a turnaround time if assigned to be done outside of class. Sometimes activities work better. For teaching basic concepts of design, I rely on a set of short activities (among other things, of course), all based on the same simple applied scenario. The response is the number of heartbeats in 20 seconds. I ask students to find a pulse; then, I say "go" and wait 20 seconds, then say "stop". We use a variety of experimental designs to compare pulse rates under two conditions: sitting, eyes closed, thinking relaxing thoughts, and standing, eyes open, jaw clenched, recalling an angry moment. These two conditions work well in the sense that there is a real difference, but not a large one. Thus the difference can often be detected by a good design, but not by a simpler, less thoughtful design. For some activities, we also consider a second, observational




346 G. W. Cobb

factor: exercise level. For a small class, I divide students into two roughly equal-sized groups based on how many hours per week they say they exercise. With a larger class, I create three equal-sized groups. Sometimes we find an interaction between the two factors, with standing having a smaller effect for those who exercise a lot; sometimes there is no detectable interaction.

I use this basic scenario to illustrate all of the following:

Randomization and the one-way completely randomized design.

Observational (exercise) versus experimental (sit/stand) conditions.

Two ways to create blocks: by pairing using a pre-test and by repeated measures (within- subjects design).

Factorial crossing, the two-way design, and interaction.

Split-plot/repeated measures design (what psychologists call a mixed design).

In a typical offering of my one-semester course on design, I may do four or five of these related activities during the first few weeks of the semester. Each activity takes less than five minutes to complete, although a good discussion may run to half an hour.

8 Final Comments

Almost four decades ago, as a graduate student of Hans Carter and Ray Myers, I faced a simulation-based challenge abstractly parallel to the DPS industrial example. Back then the computer's hard drive was bigger than a dishwasher; its 8 К memory capacity was smaller than 1/10 the capacity of the most primitive floppy of two decades into the fixture; it took 24 hours, real time, to get back observed values for each combination of the control variables; and the visual interface consisted of a page of 16" wide, tractor-fed, green-and-white-striped paper with IBM print-ball characters: any connection between the numbers and a real- world story was by default purely a matter of imagination. DPS have brought us up to date with their carefully planned flexibility and effective visual interface. Surely they have opened one door to the future of statistics education, and invited us to look ahead to new possibilities.

With DPS, I am confident in expecting the important path they show us to become well travelled. With NL, I am less certain, but even more excited at the prospects. Their path is riskier, but theirs is a profound, radical and potentially generation-changing article. My main reaction is one of immense enthusiasm and admiration. While DPS have challenged us to think about what we want students to learn, and how best to engage them in those areas, NL challenge us at the even more fundamental level of the sociology of our profession, to re-evaluate what kinds of contributions are most potent in energizing, sustaining, and expanding our mutual enterprise.

Statistics, at its best, is a collaborative undertaking. At a micro-level, all who consult collaborate. At a macro-level, the R-project shows what our community can accomplish when we work together. NL have done the hard work of finding a way to harness technology so we can unite in a grand effort to create the Stat Ed analog of the R-project (http://www.R-project.org), or even more radically, the Greensboro lunch counter sit-ins.

In conclusion, I ask you to consider how best to integrate three approaches to the future, one from DPS, one from NL, and one from Plato:

1. DPS show us how to take ideas from time past and use technology to make them far more compelling and far more flexible than ever before.





2. NL spur us to strike out in a totally new direction, with an awe-inspiring potential for root-deep changes.

3. As we look ahead, we should seek to shake the earth, but remember that to be most effective we should work to find the right synthesis of radical change and enduring truths. When it comes to blocks, Plato may have been behind by two millennia. When it comes to education, he may have been ahead by two millennia. We should strive to learn as much mathematics as we can, all the while aspiring to become sociologists in our next life.

References

Box, G.E.P. (1992). Teaching Engineers Experimental Design with a Paper Helicopter. Quality Engineering , 4(1), 453-459.

Cobb, G. W. (2002). Introduction to Design and Analysis of Experiments. New York: Springer- Verlag. Cobb, G.W. & Miao, W. (1998). Bears in Space: Activities to Introduce Basic Ideas of Design. Statistical Education -

Expanding the Network : Proceedings of the Fifth International Conference on Teaching of Statistics , Eds. L. Pereira- Mendoza, L.S. Kea, T.W. Kee, and W.-K. Wong, http://www.stat.auckland.ac. nz/~iase/publications/2/Topic2n.pdf, pp. 14-20.

Darius, P.L., Portier, K.M. & Schrevens, E. (2007). Virtual experiments and their use in teaching experimental design. Int. Statist. Rev. ,75, 281-294.

Gelman, A. & Deborah, N. (2002). Teaching Statistics: A Bag of Tricks. New York: Oxford University Press. Gnanadesikan, M., Scheafřer, R.L., Watkins, A.E. & Witmer, J. A. (1997). An Activity-Based Statistics Course. Journal

of Statistics Education 5(2). Gunter, В. (1 993). Through a Funnel Slowly with Ball Bearing and Insight to Teach Experimental Design. The American

Statistician 47(4), 265-269. Kuiper, S., Moore, T. & Collins, L. (2005). Collaborative Research: Integrating Science and Active Learn-

ing into Data-Oriented Post-Calculus Probability and Statistics Courses, NSF CCLI DUE # 0510392 http://web.grinnell.edu/individuals/kuipers/stat21abs/.

Krugman, P. (1994). Peddling Prosperity -Economic Sense and Nonsense in the Age of Diminished Expectations. New York: Norton.

Nolan, D. & Speed, T.P. (2000). Stat Labs : Mathematical Statistics Through Applications. New- York: Springer. Nolan, D. & Temple Lang, D. (2007). Dynamic, interactive documents for teaching statistical practice. Int. Statist.

Rev. ,75, 295-321. Ramsey, F. & Schafer, D. (2002). The Statistical Sleuth: A Course in Methods of Data Analysis. Belmont, CA: Duxbury. Scheaffer, R.L., Watkins, A., Witmer, J. & Gnanadesikan, M. (1997). Activity-Based Statistics. Berkeley, CA: Key

College Publishing. Watkins, A., Scheaffer, R. & Cobb, G. (2007). Statistics in Action, (2nd edition). Berkeley, CA: Key College Publishing.

[Received September 2007, accepted September 2007]




Documents

One Possible Frame for Thinking about Experiential Learning