Metacognitive mediation in learning with computer-based simulations

Compurers in Human Behavior, Vol. 10, pp. 93-106.1994 Printed in the U.S.A. All rights reserved.

0747-5632/94 $6.00 + .tXl Copyright 0 1993 Pergamon Press Ltd.

Metacognitive Mediation in Learning With Computer-Based Simulations

Marcel V. J. Veenman, Jan J. E/shout, and Vittorio K Busato

University of Amsterdam

Abstract - The objective of this study was to determine whether providing students with metacognitive instructions during experimentation in u computer simulation environment results in better learning outcomes thun unguided discovery learning. High and low intelligent students worked in either a metacognitive-mediated (MM) or unguided discovery (UD) environment for learning principles of electricity. Analyses of thinking-aloud protocols showed that MM subjects exhibited a better working method than subjects in the UD condition. MM subjects performed better than UD subjects on u posttest tupping qualitative knowledge, but on a posttest of quantitative problems only low intelligent MM subjects showed enhanced performance. No learning effects of metacognitive instruction were detected in the unalyses of the retention tests. These results are discussed in relation to the nature of metucognitive activities.

The objective of this study is to determine whether metacognitive mediation in a computer simulation environment can improve both the quality of working method and the learning performance of students. Because of its exploratory nature, learning in simulation environments puts high cognitive and metacognitive demands on students, especially on novice students. However, simulations can be powerful instructional tools if they incorporate instructional guidance that successfully sup- ports discovery learning. One of the characteristics of novice problem solving behavior is a lack of metacognitive skill, as reflected by a poor working method or problem approach. Several studies substantiated the notion that the quality of working method is positively related to learning outcomes. Hence, metacognitive mediation directed at raising the level of working method may be instrumental to novice learning in simulation environments.

Simulations, used as an instructional tool, allow for learning by doing and discovery under restricted, realistic conditions. The benefits of using simulations in an

Requests for reprints should be addressed to Marcel V. J. Veenman, Psychological Laboratory, Roetersstraat 15, 1018 WB Amsterdam, The Netherlands.

Y3

94 Veewnan, Elshout, ntzd Bwzto

educational setting are summarized by De Jong (1991): They enable students to bridge the conceptual gap between reality and abstract knowledge, and they improve motivation and enhance learning by an active student interaction. Essential for the use of simulations as instructional tools is the continuous presence of learner activities. Students need to engage in exploring a domain thoroughly by generating hypotheses and testing these hypotheses by actively designing and performing experiments in the simulated environment (Goodyear, Njoo, Hijne, & van Berkum, 1991; Reimann, 1989). Because of its exploratory nature, learning with simulations involves complex problem solving and inductive reasoning, which puts a high cognitive demand on the student (De Jong, 1991; Goodyear et al., 1991). A large number of alternative actions might overwhelm students and result in random behavior (Van Berkum & De Jong, 1991). On the other hand, students might also adopt a passive attitude, not using the opportunities for learning that the simulation environment offers (Njoo & De Jong, 1989, 1991) or not being able to use them properly due to limited intellectual resources or a poor working style (Veenman, Elshout, & Bierman, 1991).

A completely learner-controlled situation is regarded as problematic, especially for the weaker students. Learning in simulation environments should be accompa- nied with at least some instructional guidance (De Jong, 1991). However, little empirical research on this topic is available, and those studies do not invariably show positive learning outcomes due to instructional guidance (Njoo & De Jong, 1991; Shute, 1990; Van Berkum & De Jong, 1991). A series of experiments by Veenman and Elshout (Elshout & Veenman, 1990; Veenman & Elshout, 1990, 1991a, 1991b) with computerized simulations in a variety of domains showed that extensive structuring of the learning environment (e.g., by offering students instructions of how to perform a number of “telling” experiments, by presentation of the subject matter in a structured sequence, and by providing students with extensive feedback) did not improve learning in novice students, compared to unguided discovery learning. Even when guided by instruction, the learning activities of especially low intelligent subjects invariably needed correction. By not read- ing an instruction thoroughly (and as a consequence doing the wrong things), by not completing a sequence of tasks, and by skipping the feedback, some students reduced the structured environment to an unstructured one (Veenman & Elshout, 1991a). This poor working method, reflecting a lack of metacognitive skill that is typical for novices (Chi, Glaser, & Farr, 1988), may have prevented them from tak- ing advantage of the structured learning environments.

There is ample evidence that metacognitive skills are relevant to problem solving and learning processes (Campione, Brown, & Ferrara, 1982; Elshout, 1987; Glaser, 1990; Glaser & Bassok, 1989; Swanson, 1990; Veenman et al., 1991). An important distinction between novices and experts, related to differences in the organization of the knowledge base, also reflects characteristics of a metacognitive nature. Before actually acting, experts pass through an extensive qualitative analysis of the problem through which schemata of forward problem solving strategies are activated (Anderson, 1985; Larkin, McDermott, Simon, & Simon, 1980). Novices, on the other hand, analyze a given problem in terms of superficial fea- tures (Chi, Feltovich, & Glaser, 1981; Chi, Glaser, & Rees, 1982) and are inclined to act immediately and unsystematically (Elshout, 1987; Jansweijer, Elshout, & Wielinga, 1990). This lack of metacognitive control results in a poor problem representation to which only weak problem solving strategies like means-ends analysis and working backwards can be applied (Anderson, 1985; Glaser, 1990; Larkin et al., 1980). These strategies, drawing heavily on working memory, often result in

Metacognitive mediation 95

a cognitive overload. Novice learning is restricted further by the disorderly pattern of memory traces of the problem solving process due to their haphazard way of acting (Elshout, 1987).

However, not all novices are limited to equally poor problem solving behavior (Schoenfeld & Herrmann, 1982). An “expert novice” is a novice who is rapidly accumulating expertise in a domain. Contrary to genuine novices, expert novices tend to act more expertlike by orientating on a problem, working more systematically and more accurately, and by evaluating their problem solving activities more during task performance (Elshout, 1985). Furthermore, research by Veenman and Elshout (1990, 1991b) showed that the quality of elaboration activities is highly related to learning outcomes in novices. Chi, Bassok, Lewis, Reimann, and Glaser (1989) demonstrated that better students generate more self-explanations during the learning process. 1 Deep orientation, systematical orderliness, evaluation, and elaboration are the characteristics of a proficient working method, reflecting the metacognitive skill that is brought in by the student in order to structure the leam- ing process (Veenman, Elshout, & Bier-man, 1989). By analyzing a problem thoroughly, a student is likely to focus on relevant information given in the problem statement, on which a more detailed action plan can be built. Such an action plan, containing goals and directions for activities, entails the possibility of more process control during problem solving activities. Working systematically according to that plan enables the student to keep track of progress being made. Evaluation activities, directed at detecting faulty procedures and mistakes, are more fruitful within the framework of an action plan. Finally, elaboration activities make more sense if they build on clear-cut results from experiments testing explicit hypotheses. These metacognitive activities that constitute an effective working method not only help to solve the problem given, they also provide a better context for learning to solve this type of problems in general (Elshout, 1987; Mettes, Pilot, & Roossink, 1981). Especially when sufficient domain knowledge is lacking, metacognitive control is instrumental to learning (as is typically the case with expert novices).

Research also addressed the relation between working method and intellectual ability as predictors of learning. Elshout and Veenman (1990) concluded tentative- ly from the results of an experiment with a computer-simulated heat lab that working method could be just another manifestation of general intelligence. Their high intelligent subjects showed both a more proficient working method and better learning outcomes than low intelligent ones. Working method correlated significantly with learning measures, but when intelligence was partialed out these correlations approximated zero. However, other experiments with a simulated electricity lab (Veenman & Elshout, 1990) and a statistics lab (Veenman & Elshout, 1991a, 1991b) revealed that working method clearly has some unique predictive value to learning. Although high intelligent subjects exhibited a better working method than low intelligent subjects, most of the correlations between working method and learning measures remained high as intelligence was partialed out. Research by Swanson (1990) also indicated that metacognitive skill can compensate for intellectual ability. Working method or the underlying metacognitive skill at least partly has its own virtue in learning by discovery.

The impact of an effective working method on novice learning raises the question of the malleability of novice problem solving behavior by instruction. Mettes et al. (1981) designed a programme of actions and methods (PAM) that provided students with instructions of metacognitive strategies integrated into an existing thermodynamics course. Working with PAM during the lOO-hr course resulted in a substantial increase of learning outcomes. Another experiment by De Jong and

96 Veenman, Elshout, md Bustrto

Ferguson-Hessler (1984) showed that a 6-hr training of a systematical problem approach following principles of PAM did not result in an improvement of problem solving behavior during a course in electricity and magnetism. However, this metacognitive training was given apart from the physics course, thereby requiring farther transfer of applying the systematical problem approach. Apparently, inte- gration of metacognitive directions into the actual instruction of the subject matter is a prerequisite for being effective.

The objective of this study is to determine whether metacognitive mediation by providing students with prompts directed at raising the level of working method during experimentation in a simulation environment might improve novice learning. If metacognitive skill is less a matter of acquired technique related to intelligence than, for instance, of good habit or invested effort, it might be worthwhile to prompt students to do so. The predictions derived from our hypothesis are twofold. It is predicted in the first place that metacognitive prompts will lead to problem solving behavior reflecting a more effective working method and, secondly, that this advanced working method will result in enhanced performances. Both predictions are being tested explicitly.

METHOD

Subjects

Some months prior to the experiment intellectual ability of first-year psychology students was assessed by a series of abilities tests, representing several components of the Structured-of-Intellect model (Guilford, 1967). Those students whose com- posite score deviated at least one standard deviation from the mean (M = 19.50, SD = 4.27, N = 492) were denominated as either high or low intelligent. Furthermore, only students who had received less than 4 years of physics education at high school were selected. Twenty-nine students (15 high and 14 low intelligent novices in the domain of physics) participated in the experiment.

Learning Conditions

A computer-simulated Electricity Lab that was used by Veenman and Elshout (1990) in previous research was adapted for our purposes. In this electricity lab several devices like a bulb, a switch, and different resistors could be placed within a circuit frame, while voltage and current could be measured on different loca- tions. The source voltage and the value of resistors could be modified easily. Actually, the electricity lab had two forms corresponding with two subsequent parts of the learning program, a single device circuit with the switch in one slot and another open slot available for placement of either the bulb or a resistor, and a serial circuit with two open slots, By experimenting with the electricity lab containing the single device circuit, students had to discover basic principles of electricity theory (Ohm’s law, Kirchoff’s law, and Power). Subsequently, the students had to find out how to apply these basic principles to the more complex situation of a serial circuit (e.g., by determining that Ohm’s law still is valid if one calcu- lates the substitutional resistance and that voltage is distributed over the resistors proportionally to their magnitude).

Students in the unguided discovery (UD) condition had to design and conduct their own experiments without instructional guidance. In both the single device cir-


cuit and the serial circuit they were simply asked to find out how voltage, current, resistance, and power are interrelated. Both parts of the learning program terminat- ed with the presentation of a short list of relevant formulas.

The metacognitive-mediated (MM) version of the electricity lab provided students with instructions for a number of “telling” experiments. MM students were asked to find out how voltage and current are distributed within a single device circuit, how current is related to voltage and to resistance (by altering the values of these variables systematically one at the time), and how effects on current are bal- anced by increasing or decreasing voltage and resistance simultaneously. Furthermore, they had to find out how voltage and current are distributed in a serial device circuit with two equivalent and two unequal resistors, and how Ohm’s law is applicable to a serial circuit on both the local level (a specific resistor with its specific voltage) and the overall level (the substitutional resistance with the source voltage).

Previous research with the same electricity lab (Elshout & Veenman, in press; Veenman & Elshout, 1990) showed that these experimental instructions by them- selves did not result in an improvement of working method, nor did they enhance learning performance compared to unguided discovery learning. High and low intelligent subjects worked for several hours in either an unguided or a guided discovery version of the electricity lab. The unguided discovery version was identical to the UD condition of this experiment, and the guided discovery version presented students with similar experimental instructions, but without metacognitive mediation. Guided discovery affected neither the quality of working method nor the learning performance of subjects. An aptitude-treatment-interaction hypothesis suggesting that low intelligent subjects would profit relatively more from a guided discovery environment was not confirmed as well. Similar results were established for simulation environments representing other domains (Elshout & Veenman, 1990; Veenman & Elshout, 1991 a, 1991 b). In the current experiment the experimental instructions were the required vehicles for metacognitive mediation. Training of metacognitive skills and strategies can be effective only if this training is embedded in the instruction of the subject matter (Derry & Murphy, 1986; Glaser, 1984; Mettes, Pilot, & Roossink, 1981; Vanderlocht & Van Damme, 1989; Volet, 1991).

Metacognitive mediation consisted of prompts directed at raising the level of working method. Before actually performing each experiment, students in the MM condition were prompted to paraphrase the question, to generate a hypothesis, to think out a detailed action plan, and to make notes of it. Furthermore, after execution of an action plan they were requested to evaluate their experimental outcomes (to check whether the experimental procedure was implemented correctly and whether the question was fully addressed to), to draw a conclusion elaborating on the subject matter, and to make notes. It must be emphasized, however, that except for presentation of metacognitive prompts, the learning program did not monitor or interfere with the student’s activities during experimentation, as, for instance, the computer coach of Smithtown (Shute, Glaser, & Raghavan, 1989) did by interven- ing after a sequence of ineffective, buggy behaviors of the student. After each experiment the student was provided with a summary of the subject matter.

Procedure

Subjects were assigned randomly to either the MM or UD condition. The UD condition contained 7 low and 8 high intelligent subjects, whereas 7 low and 7 high

intelligent subjects were engaged in the MM condition. The experimental groups did not differ in sex or age. On entering the simulation environment all subjects were trained in handling the electricity lab by a computerized instruction. After this training all subjects were allowed to experiment freely with the single device circuit for a short period of time (with a IO-min time limit). Next, the subjects entered either the UD or the MM environment. During their work in the electricity lab thinking-aloud protocols of all subjects were tape-recorded. Time on task was not controlled, but was included in the study for postexperimental inspection. Student notes, if any, were preserved in order to be analyzed in relation to the thinking- aloud protocols. The transcribed protocols were analyzed on quality of working method (the five scales being Orientation Activities, Systematical Orderliness, Accuracy, Evaluation, and Elaboration Activities) by three “blind” judges who received no prior information about the student’s intelligence test scores. They performed the analyses together, arguing until agreement was reached. Truly, this method of protocol analysis lacks the possibility of assessing an interjudge reliability, but we believe that enabling the judges to scrutinize their judgments mutually enhances the reliability.

Aspects of working method were scored separately from two protocol segments corresponding to the subdivisions in the learning program that relate to working in a single device circuit and a serial circuit, respectively. These protocol segments appeared to have the substantial lengths that are required for protocol analysis. Quality of orientation activities was judged on indications of analyzing a problem, activities of building a mental model of the task, generating predictions, and designing an action plan. Orientation Activities might be regarded as all metacognitive activities that are preparatory to actually performing the task. Judgments of systematical orderliness were based on the quality of systematical execution of an action plan, completing an orderly sequence of actions, and the avoidance of unsystematical events (such as varying two variables at the same time). Criteria for accuracy were precision of calculation, correct usage of quantities and unities, and the avoidance of negligent mistakes. Evaluation activities were judged on monitoring and checking, while judgments of elaboration were based on indications of generating explanations, relating the subject matter, and recapitulating. As was argued by Veenman and Elshout (1991a), these aspects of working method were judged on the quality of performing metacognitive activities, not on the quality of information these activities produced. For instance, a subject drawing an elaborated but incorrect conclusion scored high on Elaboration while accumulating incorrect knowledge. All aspects of working method were rated on a 7-point scale, and for each subject average scores on the five subscales were computed over the two subdivisions.

Measures of Learning

A multiple choice questionnaire tapping qualitative knowledge about electricity theory was administered in a pretest-posttest design (some months) prior to and shortly after the experiment. For instance, an item was: “What is the nature of the relation between current and resistance in an electrical circuit? a) proportional, b) inverse proportional, c) logarithmic, or d) unrelated.” After a 3-week delay a reshuffled version of this questionnaire was presented as a retention test. The questionnaires consisted of 24 items with an adequate internal consistency (alpha was .71 for the questionnaire administered to a large group of first year psychology students, N = 455, including our subjects).

Metncogrzitive nteriicrtiotz 99

After completion of the posttest questionnaire subjects were asked to solve a series of 16 problems while thinking aloud. A subset of 11 problems was denominated as Quantitative Problems (requiring simple to complex calculations with Ohm’s law and the Power formula), and the remaining subset of 5 problems was denominated as Comprehension Problems (requiring more transfer and a deeper understanding of electricity principles). A comprehension problem was, for instance:

A bulb is connected by two copper wires to an accumulator, one wire being 1 square mil- limeter thick and the other one 2 square millimeters. Copper is a good, but not perfect, con- ductor of electricity. What is the proportion of, respectively, the current in and the power of the two wires? Explain your answer.

Solving these problems required a fair amount of quantitative reasoning based upon understanding of electricity theory. Another series of equivalent problems (again 11 Quantitative and 5 Comprehension Problems) was presented with the retention test after the 3-week delay. The quantitative problems of the retention test were similar to those of the posttest, but with different quantities. The comprehension problems of the retention test presented the same sort of comprehension problems in a somewhat different context. The analyses of the problem protocols were performed by the judges prior to the protocols from the electricity lab (in order to avoid interference with judgments of working method). Each problem was rated as a school mark on a 7-point scale (ranging from 0 to 6).

RESULTS

Working Method

Alpha reliabilities for the five subscales of working method were .91, .89, .84, .SS, and .95 for, respectively, Orientation Activities, Systematical Orderliness, Accuracy, Evaluation Activities, and Elaboration. An analysis of variance (ANOVA) on Orientation Activities revealed significant effects of both intellectual ability (F(l, 25) = 10.61, p c .004) and learning condition (F(1, 25) = 10.73, p -c .004), with high intelligent subjects as well as subjects in the MM condition showing better Orientation Activities. A similar pattern emerged from the analyses of Systematical Orderliness. High intelligent subjects worked more orderly than the low intelligent ones (F(1, 25) = 10.53, p < .004), while MM subjects worked more systematically compared to UD subjects (F(1, 25) = 14.45, p < .OOl). For Accuracy, only the effect of intellectual ability was marginally significant, with F(1, 25) = 4.06 (p < .06). The insignificance of the learning condition effect (F(l, 25) = 0.71) was not surprising, since the MM condition did not include prompts directed at improving accuracy. An ANOVA on Evaluation Activities showed effects of intellectual ability (F( 1, 25) = 12.50, p c .002) and learning condition (F(1, 25) = 5.63, p c .03) in line with Orientation Activities and Systematical Orderliness. Finally, Elaboration showed a significant effect of intellectual ability, with F(1, 25) = 8.52 (p < .Ol), whereas the learning condition effect (F(1, 25) = 2.63) was not significant. None of these ANOVAs showed significant interaction effects.

A general Working Method score with a high internal consistency (alpha was .95) was composed of all scores on the distinct measures. An ANOVA on these Working Method scores showed significant effects of both intellectual ability

100 Veenrnan, Elshout, and Busato

Table 1. Results of the Correlational Analyses

Learning Measures IA

UD Subjects

WM WM Part. Corr. IA

MM Subjects

WM WM Part. Corr.

Quesfionn~ife Posttest Retention test

Quantitative problems Posttest Retention test

Comprehension problems Posttest Retention test

.32* .34 .26 .21 -.04 -.18

.45* .64” .59’ .39’ .29 .12

.73”” .67” .67** .40’ .35* .19 50” .78” .72” .31 .44” .34

.66”” .61** .56” .47** .48* .32

.46”’ .47’ .38 .49” .49** .32

Note. All correlations are corrected for selection of extreme groups of intellectual ability. IA = intellectual ability, WM = working method, Part. Corr. = partial correlation with intellectual ability partialed out. +p < .05. **p c .Ol

(Ffl, 25) = 12.78, p c .OOZ) and learning condition (F(1) 25) = 8.16, p < .009). Again, the interaction effect was not s~gni~cant (F(l,25) = 0.39).

Additionally, for both the MM and UD conditions correlations between intellectual ability, working method, and learning measures were calculated and subsequently corrected for selection of extreme groups intellectual ability, folIowing the procedures of Gulliksen (1961). The corrected correlation between intellectual ability and working method was .33 (p < .05) for the UD group and .50 (I? < .Ol) for the MM group. Next, intellectual ability was partialed from the correlations between working method and learning measures (see Table 1). For the UD group most of these partial correlations were significant, whereas for the MM group none of them reached significance. Metacognitive mediation may have had a reductive effect on differences in learning outcomes between subjects, while it may have strengthened the relation between intellectual ability and working method simultaneously.

Questionnaires

The pretest scores revealed no significant effects of intellectual ability (F(1, 25) = 0.80) or learning condition (F(1, 25) = 0.33). In spite of the random assignment of subjects to the learning conditions, a significant interaction effect showed up in the pretest scores with F(1, 25) = 6.05 (p < .05). Low intelligent subjects in the MM condition and high intelligent subjects in the UD condition appeared to have more prior knowledge to begin with (see Table 2).

Table 2. Means (and Standard Deviations) of the Questionnaire

Condition

Pretest Posttest Retention Test

Low IA High IA Low IA High IA Low IA High IA

UD 6.57 9.62 9.71 12.50 9.14 13.62 (2.22) (2.45) (2.43) (2.56) (2.04) (3.82)

MM 8.29 6.66 12.71 14.86 10.71 14.14 (1.98) (3.02) (2.14) (3.53) (2.87) (3.13)

Note. IA = intellectual ability, UD = unguided discovery, MM = metacognitive- mediated.

Metacognitive mediation I01

Because of the differences in prior knowledge, an analysis of covariance (ANCOVA) was performed on the posttest scores, with the pretest scores as covariate. The requirements of homogeneity of within-class regressions were met (F(3, 21) = 0.36), and r2 within was .16. This ANCOVA resulted in significant effects of both intellectual ability (F(l, 24) = 4.87, p < .04) and learning condition (F(1, 24) = 8.09, p < .Ol). No significant interaction effect was obtained (F(1, 24) = 0.10). The adjusted means were, respectively, 10.26 and 12.55 for low intelligent UD and MM subjects, and 11.78 and 15.29 for high intelligent UD and MM subjects. High intelligent subjects performed better than low intelligent subjects, as subjects in the MM condition did compared to UD subjects.

In order to allow for an ANCOVA on the retention test scores, with the pretest scores as covariate, the homogeneity of within-class regressions was inspected (F(3, 21) = 1.34). The r2 within appeared to be low (only .OS). This ANCOVA showed a strong effect of intellectual ability, with F(1, 24) = 10.70 (p c .004). However, significant effects of learning condition or interaction failed to appear (respectively, F( 1, 24) = 0.92 and F(l, 24) = 0.03). The adjusted means were, respectively, 9.62 and 10.58 for low intelligent UD and MM subjects, and 13.01 and 14.51 for high intelligent UD and MM subjects. Compared to posttest scores, the mean score of low intelligent MM subjects dropped two points, whereas the mean score of high intelligent MM subjects remained on nearly the same level (see Table 2).

Quantitative and Comprehension Problems

The internal consistencies of the posttest problems were adequate (alpha was, respectively, .89 and .65 for the quantitative and comprehension problems). An ANOVA on the quantitative problems revealed a strong effect of intellectual ability, with F(1, 25) = 22.53 (p c .OOOl), indicating that high intelligent subjects performed much better than low intelligent subjects (see Table 3). Effects for learning condition (F(l) 25) = 2.02) and interaction (F(l) 25) = 1.70) were not significant. However, post hoc analyses showed a significant effect of learning condition for low intelligent subjects only (t = 2.20, df = 12, p < .025). Accordingly, an ANOVA on the comprehension problems showed a strong effect of intellectual ability (F( 1, 25) = 22.80, p c .OOOl) and no significant effects of learning condition (F(1, 25) = 1.19) or interaction (F(1, 25) = 1.31). Post hoc analyses only showed a marginal significant effect of learning condition for low intelligent subjects taken apart (t = 1.61, df= 12, p < .07).

Alpha reliabilities of the retention test problems were adequate for the quantitative problems (.89) but lower for the comprehension problems (.53). An ANOVA

Table 3. Means (and Standard Deviations) of the Quantitative Problems

Posttest Retention Test

Condition Low IA High IA Low IA High IA --

UD 6.00 36.12 14.29 35.12 (4.40) 0;:;; 1 (14.72) (11.92)

MM 19.57 16.00 31.71 (15.70) (15.96) (16.45) (21.86)

/Vole. IA = intellectual ability, UD = unguided discovery, metacognitive-mediated.

102 Veenmatl, Elshout, and Busnto

on the quantitative problems clearly showed an effect of intellectual ability (F(1, 25) = 8.89, p < .Ol). Effects due to learning condition or interaction were insignifi- cant (respectively, F( 1, 25) = 0.02 and F( 1, 25) = 0.18). An ANOVA on the comprehension problems yielded similar results. The effect of intellectual ability was significant (F(1, 25) = 13.02, p < .002), whereas effects of learning condition and interaction were negligible (respectively, F(1, 25) = 0.10 and F(1, 25) = 0.02). When comparing retention test scores with posttest scores (see Tables 3 and 4), it must be kept in mind that the retention test problems were similar, but not identical to the posttest problems.

Time on Task

The time on task (including the free experimentation period) of both learning conditions differed considerably (F(l,25) = 120.88, p < .OOOl). It took MM subjects 3 hr 17 min on the average to work through the program, whereas UD subjects only spent about 1 hr on the average. However, time on task correlated rather low with the learning measures, with correlations ranging from -.13 to .34 (N = 29). Separate analyses for both learning conditions only showed negative correlations between time on task and measures of learning (ranging from -.37 to -.09 for UD subjects and from -.38 to -.08 for MM subjects).

DISCUSSION

The results clearly confirmed the first prediction that metacognitive mediation improves the working method of students. More specifically, metacognitive mediation encouraged the occurrence and quality of orientation activities, systematical orderliness, and evaluation activities. Furthermore, the analyses of general Working Method scores showed that the effects of intellectual ability and learning condition on the quality of working method are additive. It suggests that we are actually dealing with two distinct aspects of working method, one aspect being the metacognitive skill that is brought in by the student (a mixture of acquired technique and good habit) and the other being the behavioral consequence of MM instructions. This conception is supported by the results of the correlational analyses of intellectual ability, working method, and learning measures. The significant partial correlations in the UD group, indicating that working method accounted for residual variance in learning measures, clearly replicated the outcomes of earlier experiments in the domains of electricity and statistics (Elshout & Veenman, in

Table 4. Means (and Standard Deviations) of the Comprehension Problems

Posttest Retention Test

Condition Low IA High IA Low IA High IA

UD 3.67 13.66 6.00 12.12 (2.08) (5.69) (2.31) (5.38)

MM 7.45 13.57 6.71 12.43 (5.83) (3.08) (5.02) (4.08)

Note. IA = intellectual ability, UD = unguided discovery, MM = metacognitive-mediated.


press; Veenman & Elshout, 1990, 1991a, 1991b). Without metacognitive mediation, working method reflects the metacognitive skill of the student that is deter- mined partly by intellectual ability and partly has a unique source of variance. Metacognitive mediation may have decreased the effect of this unique source of variance in working method by stimulating all MM subjects to invest effort in performing metacognitive activities. Consequently, the significance of the acquired techniques related to intellectual ability may have been increased, as was indicated by the higher correlation between intellectual ability and working method for MM subjects. We will return to this issue after discussion of the learning outcomes.

In order to evaluate the second prediction of enhanced performance due to metacognitive mediation, the results on the questionnaires tapping qualitative knowledge should be distinguished from the results on the quantitative and comprehension problems. Analyses of the posttest questionnaire clearly showed a main effect of metacognitive mediation that was in line with the second prediction. In the analyses of the retention test questionnaire this effect of metacognitive mediation diminished. However, results on the retention test revealed that the relative improvement of high intelligent MM subjects was sustained on nearly the same level, whereas the mean score of low intelligent MM subjects decreased.

The analyses of the posttest problem scores showed significant effects of metacognitive mediation on the quantitative problems and marginally significant effects on the comprehension problems for low intelligent subjects only. High intelligent subjects scored significantly higher than low intelligent subjects, regard- less of what learning condition they belonged to. For high intelligent subjects the effects of intellectual ability perhaps overpowered those of metacognitive mediation, whereas for low intelligent subjects the limitations of their intellectual resources were partly compensated for by the effects of metacognitive instruction. Again, the analyses of retention test problems showed that metacognitive mediation did not have a long-lasting effect on learning outcomes of low intelligent subjects. Quite surprisingly, low intelligent UD subjects spontaneously recovered at these retention tests.

The learning conditions differed considerably in time on task. MM subjects defi- nitely needed more time to work through the program due to the metacognitive prompts that were presented within a framework of preset experiments. UD subjects, on the other hand, were free to leave the electricity lab at any time. Yet, learning measures correlated low with time on task. Similar results were obtained for time on task in experiments with simulation environments representing different domains (Elshout & Veenman, 1990, in press; Veenman & Elshout, 1990, 1991a, 1991b). Apparently, the time spent in the laboratory is not as relevant as the quality of (metacognitive) activities performed. This may be the case particularly in simulation environments where students have to explore a domain actively (in contrast with performing a study task).

Summarizing the results concerning the second prediction, low intelligent subjects clearly showed enhanced performances due to metacognitive mediation on the posttests, but these effects decayed over time, whereas for high intelligent subjects metacognitive mediation improved learning outcomes related to qualitative knowledge over an extended period of time. No such effect of learning conditions could be expected if working method was exclusively a personal quality (trait) of a student. However, when engaging in metacognitive activities is evoked by instructions, the effects of this invested effort appear to be more enduring if the student can draw on intellectual resources and the acquired techniques that are related to intelligence.

There is growing evidence that the content of a discipline and the relevant metacognitive strategies should be taught simultaneously (Schoenfeld, 1985; Vanderlocht & Van Damme, 1989; Volet, 1991). In the MM condition this integra- tion of content and strategy instruction was established by presenting metacognitive prompts embedded within instructions of “telling” experiments. It has been pointed out before that these experimental instructions alone did not improve the quality of working method or learning performance in previous experiments (Elshout & Veenman, in press; Veenman & Elshout, 1990). Consequently, the positive results of the MM condition in this experiment should be ascribed to metacognitive mediation presented additionally to experimental instructions. The experimental design, however, does not allow us to draw a conclusion about the necessity of these experimental instructions for the effectiveness of metacognitive mediation. A replication study with an extended design, including a condition with metacognitive mediation but without experimental instructions, might settle this matter.

What do the results of this experiment tell us about the development of simulation environments or discovery learning environments in general? Clearly, both the quality of working method and learning outcomes can be affected positively by metacognitive mediation (at least within a framework of subject matter-oriented instructions). However, metacognitive mediation only had short-term effects on the learning performance of low intelligent subjects. Although low intelligent MM subjects exhibited a better working method than low intelligent UD subjects, their action plan still contained faulty procedures or disregarded relevant activities. From the thinking-aloud protocols it appeared, for instance, that very often low intelligent subjects did not come up with the idea to place the voltage meter over just one of the devices in a serial circuit, which was necessary for acquiring a notion of voltage distribution. A computer coach that both monitors the student’s activities on buggy behaviors (e.g., see Shute et al., 1989) and presents suggestions of relevant actions that fail to appear might boost the working method of low intelligent students on the content level.

NOTE

1. Though self-explanations are somewhat different from elaborations, the first term referring to generating new knowledge and the latter one referring to embedding a piece of information into existing knowledge (Chi & VanLehn, 1991), both activities deal with processing information that evolves from the problem solving process and might therefore be regarded as elaboration activities in a broad- er sense.

REFERENCES

Anderson, J. R. (1985). Cognitive psycl~ology md its implications. New York: Freeman. Campione, J. C., Brown, A. L., & Ferrara, R. A. (1982). Mental retardation and intelligence. In R. J.

Stemberg (Ed.), Handbook ofh14nznr~ intelligence (pp. 392-490). Cambridge, England: Cambridge University Press.

Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13, 145-182.

Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121-l 52.

Chi, M. T. H., Glaser, R., & Fan; M. J. (1988). The nature of expertise. Hillsdale, NJ: Erlbaum.


Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J. Stemberg (Ed.), Advances of the psychology of human intelligence (Vol. 1, pp. 7-75). Hillsdale, NJ: Erlbaum.

Chi, M. T. H., & VanL.ehn, K. A. (1991). The content of physics self-explanations. The Journal of the Learning Sciences, 1,69-105.

de Jong, T. (1991). Learning and instruction with computer simulations. Education & Computing, 6, 215-227.

de Jong, T., & Ferguson-Hessler, M. G. M. (1984). Strategiegebmik bij het oplossen van problemen in een semantisch rijk domein: Electriciteit en magnetisme [The use of strategy in solving problems in a semantically rich domain: Electricity and magnetism]. Ttj&chrifr voor Ona’erwijsresearch, 9, 3-15.

Deny, S. J., & Murphy, D. A. (1986). Designing systems that train learning ability: From theory to practice. Review of Educational Research, 56, l-39.

Elshout, J. J. (1985). Een goed begin is het halve werk: Over determinanten van effectief beginners- gedrag [Well begun is half done: About determinants of effective novice problem solving behavior]. In F. J. Miinks & P. Span (Eds.), Hoogbegaafden in de samenleving (pp. 201-210). Nijmegen, The Netherlands: Dekker & v.d. Veght.

Elshout, J. J. (1987). Problem solving and education. In E. de Corte, H. Lodewijlcs, R. Parmentier, & P. Span (Eds.), Learning and instruction (pp. 259-273). Oxford, England: Pergamon.

Elshout, J. J., & Veenman, M. V. J. (1990). Wat doet intelligentie voor studenten? [What does intellectual ability do to students?]. In M. J. Ippel & J. J. Elshout (Eds.), Training van hogere-orde denkprocessen (pp. 89-99). Lisse, The Netherlands: Swets & Zeitlinger.

Elshout, J. J., & Veenman, M. V. J. (in press). Intellectual ability and working method as predictors of learning. Journal of Educational Research.

Glaser, R. (1984). Education and thinking. American Psychologist, 39, 93-104. Glaser, R. (1990). The reemergence of learning theory within instructional research. American

Psychologist, 45,29-39. Glaser, R., & Bassok, M (1989). Learning theory and the study of instruction. Annual Review of

Psychology, 40,63 l-666. Goodyear, P., Njoo, M., Hijne, H., & van Berkum, J. J. A. (1991). Learning processes, learner

attributes and simulations. Education & Computing, 6,263-304. Guilford, J. P. (1967). The nature ofhuman intelligence. New York; McGraw-Hill. Gulliksen, H. (1961). Theory of mental tests. New York: Wiley. Jansweijer, W., Elshout, J. J., & Wielinga, B. J. (1990). On the multiplicity of learning to solve prob-

lems. In H. Mandl, E. de Corte, N. Bennett, & H. F. Friedrich (Eds.), Learning and instruction (Vol. 2.1, pp. 127-145). Oxford, England: Pergamon.

Larkin, J. H., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Models of competence in solving physics problems. Cognitive Science, 4, 317-345.

Mettes, C. T. C. W., Pilot, A., & Roossink, H. J. (1981). Linking factual and procedural knowledge in solving science problems: A case study in a thermodynamics course. Instructional Science, 10, 333-361.

Njoo, M., & De Jong, T. (1989, September). Learning processes of students working with a computer simulation in mechanical engineering. Paper presented at the EARL1 conference, Madrid.

Njoo, M., & De Jong, T. (1991, April). Support for learning with computer simulations: Giving hints, supporting learning processes, and providing hypotheses. Paper presented at the annual meeting of the AERA, Chicago.

Reimann, P. (1989). Modeling scientific discovery learning processes with adaptive production systems. In D. J. Bierman, J. Breuker, & J. Sandberg (Eds.), Artificial intelligence and education: Synthesis and rejlection (pp. 218-227). Amsterdam: 10s.

Schoenfeld, A. H. (1985). Mathematical problem solving. Orlando, FL: Academic Press. Schoenfeld, A. H., & Herrmann, D. J. (1982). Problem perception and knowledge structure in expert

and novice mathematical problem solvers. Journal of Experimental Psychology: Learning, Memory & Cognition, 8,484-494.

Shute, V. J. (1990, April). A comparison of inductive and deductive learning environments: Which is betterfor whom and why?. Paper presented at the annual meeting of the AERA, Boston.

Shute, V. J., Glaser, R., & Raghavan, K. (1989). Inference and discovery in an exploratory laboratory. In P. L. Ackerman, R. J. Stemberg, & R. Glaser (Eds.), Learning and individual differences (pp. 279-326). New York: Freeman.

Swanson, H. L. (1990). Influence of metacognitive knowledge and aptitude on problem solving. Journal of Educational Psychology, 82, 306-314.

van Be&urn, J. J. A., & de Jong, T. (1991). Instructional environments for simulations. Education & Computing, 6, 305-358.

106 Veentnan, Elshour, and Busaro

Vanderlocht, M., & Van Damme, J. (1989). Probleemaanpak en het oplossen van natuurkundige problemen: Een literatuuroverzicht [Problem approach and solving physics problems: A review]. Pea&o&he Studi&, 66.97-106.

Veenman, M. V. J., & Elshout, J. J. (1990). De meerwaarde van een goedc probleemaanpak [The sur- plus value of an effective working method]. Tija5chrift voor Ondetwijsreseurch, 15.337-346.

Veenman, M. V. J., & Elshout, J. J. (1991a). Intellectual ability and working method as predictors of novice learning. Learning and Itlsrruction, 1, 303-3 17.

Veenman, M. V. J., & Elshout, J. J. (1991b). Intelligentie en probleemaanpak als voorspellers van beginnend leren [Intelligence and working method as predictors of novice learning]. In B. H. A. M. van Hout-Walters & L. F. W. De KIerk (Eds.), Onrlerwijsleerprocessen: Cognirie en ttzotivatie (pp. 173-182). Amsterdam: SCO.

Veenman, M. V. J., Elshout, J. J., & Bierman, D. J. (1989). Structured and unstructured approaches to simulation: What do the students think?. In Proceedings of the Sixrh Canadian Symposium on Instructional Technology (pp. 457-461). Halifax, Canada: NRCC.

Veenman, M. V. J., Elshout, J. J., & Bierman, D. J. (1991). Structuredness of learning environment versus individual differences as predictors of Icaming. In I? A. M. Kommers, D. H. Jonassen, & T. Mayes (Eds.), Mindtools: CogtriGve rechtlologies for modeling ktrowledge (pp. 227-239). Heidelberg, Germany: Springer-Verlag.

Volet, S. E. (1991). Modelling and coaching of relevant metacognitive strategies for enhancing university students’ learning. Learning and fttstruction, 1, 3 19-336.

Documents

Metacognitive mediation in learning with computer-based simulations