Enhancing learning outcomes in computer-based training via self-generated elaboration

Enhancing learning outcomes in computer-based trainingvia self-generated elaboration

Haydee M. Cuevas • Stephen M. Fiore

Received: 27 December 2012 / Accepted: 14 March 2014� Springer Science+Business Media Dordrecht 2014

Abstract The present study investigated the utility of an instructional strategy known as

the query method for enhancing learning outcomes in computer-based training. The query

method involves an embedded guided, sentence generation task requiring elaboration of

key concepts in the training material that encourages learners to ‘stop and think’ about the

information already presented before proceeding to new concepts. This study also inves-

tigated the effect of varying the level of elaboration (low or high) prompted by the queries.

Fifty-one undergraduate students from the general psychology department subject pool at a

major university in the southeastern United States received instruction on the basic prin-

ciples of flight via one of three versions of a computer-based tutorial (no query, low-level

elaboration query, or high-level elaboration query). Participants had no prior knowledge or

previous experience with the aviation domain. A one-way between-groups design was

employed, with the query method serving as the independent variable and a sample size of

17 per condition. Dependent variables included knowledge organization, knowledge

acquisition, and instructional efficiency. Overall, results showed that incorporating low-

level elaboration queries into the training resulted in improved organization, integration,

and application of task-relevant knowledge and higher instructional efficiency. High-level

elaboration queries consistently failed to produce significantly better post-training out-

comes, possibly due to the increased cognitive load imposed on learners during training.

The discussion centers on theoretical and practical implications for promoting and

assessing learning outcomes in computer-based training.

H. M. Cuevas (&)Department of Doctoral Studies, College of Aviation – Room 137E, Embry-Riddle AeronauticalUniversity, 600 South Clyde Morris Blvd, Daytona Beach, FL 32114, USAe-mail: [email protected]

S. M. FioreCognitive Sciences – Department of Philosophy, College of Arts and Humanities, University of CentralFlorida, 3100 Technology Parkway, Suite 140, Orlando, FL 32826, USAe-mail: [email protected]

123

Instr SciDOI 10.1007/s11251-014-9315-8

Keywords Cognitive load � Instructional efficiency � Knowledge acquisition �Knowledge organization � Self-generated elaboration

Introduction

Differences in learning outcomes in computer-based training environments can be clas-

sified as being due to factors internal and external to the learner (Fiore and Salas 2007).

Internal factors increasingly important due to the nature of training are those associated

with characteristics of the learner (e.g., individual aptitudes, knowledge structure devel-

opment). External factors correspond to the instructional techniques incorporated into such

environments (e.g., diagrams, animations). In this paper, we describe a study devised to

assess how the interaction of such factors influences learning in these environments. First,

we briefly describe the learning outcomes of interest to this study. We then describe how

embedded prompting within computer-based training may be a viable instructional strategy

to produce greater learning.

Knowledge acquisition and knowledge organization

Important learning outcomes include not only learners’ mastery of basic factual infor-

mation, but also the organization, integration, and application of task-relevant knowledge

to novel situations (Chipman et al. 2013). Thus, training should be targeted at facilitating

learners’ ability to effectively integrate differing knowledge components and apply this

newly acquired knowledge in a variety of dynamic task-relevant scenarios (Cuevas et al.

2002; Fiore et al. 2002, 2003). Such post-training outcomes would demonstrate if learners

have acquired a more flexible, higher level of understanding of the material. In addition,

training should also promote learners’ organization of their newly acquired knowledge,

that is, the degree to which elements of knowledge are interconnected and integrated

within meaningful networks in long-term memory, which has been shown to be critical to

the development of task expertise. In particular, acquiring task expertise requires one to

understand, not just the meaning and function of concepts associated with their domain, but

also how these fit together (Salas and Rosen 2010).

Cognitive load and instructional efficiency

Sweller’s (1994) Cognitive Load Theory proposes that training materials and activities

required of learners during training should be structured to minimize any avoidable load on

learners’ cognitive resources (e.g., working memory capacity) and maximize knowledge

structure development (knowledge organization). To achieve this objective, training design

needs to consider the cognitive load generated from the complex interaction between the

demands of the learning material, learning process, and presentation mode (Brunken et al.

2004, p. 131; for a discussion, see Sweller et al. 2011). Specifically, cognitive load may be

influenced by both the intrinsic cognitive load associated with the content of the training

material as well as extraneous cognitive load resulting from how the training material is

presented (Sweller et al. 1998, 2011). Intrinsic cognitive load arises from the inherent

complexity and level of integration of the concepts to be learned. Intrinsic cognitive load

(and the corresponding strain on learners’ limited cognitive resources) is highest when

H. M. Cuevas, S. M. Fiore

123

many information-rich elements need to integrated for knowledge structure development to

occur (Sweller 2010). Extraneous cognitive load is artificially produced due to inadequate

presentation modes that impose split-attention and redundancy of information (Kalyuga

et al. 1999, 2004; Mayer et al. 2001). Training programs that simultaneously impose high

levels of both intrinsic and extraneous cognitive load would be expected to lead to sub-

optimal post-training outcomes (Sweller et al. 2011).

Research has also investigated germane cognitive load, which stems from cognitive

effort purposely generated by instructional strategies designed to maximize elaboration of

the targeted material (Sweller 2010; van Merrienboer et al. 2002). In this case, training

design should promote germane cognitive load since deeper processing of the targeted

material may encourage learners to organize this knowledge into a coherent cognitive

representation, integrated with other representations and prior knowledge within mean-

ingful networks in long-term memory (Mayer et al. 2005). As such, germane cognitive load

is required for knowledge organization and knowledge integration of the to-be-learned

material in long-term memory (Kirschner 2002). Accordingly, one of the issues this study

explored was how instructional strategies might influence germane cognitive load. Theo-

retically, germane cognitive load produces optimal levels of processing of training content,

which, in turn, facilitates comprehension and retention of the to-be-learned material

(Kirschner 2002; Sweller et al. 2011). Training designers should, therefore, attempt to

identify what are the optimal levels of germane cognitive load in a given learning situation.

In order to understand the influence of cognitive load on the learner, research has

assessed the instructional efficiency of the learning experience. This is described as the

relative efficiency of the instructional program in terms of the demands imposed by the

training on learners’ cognitive resources. This measure of training effectiveness involves

examining the observed relation between subjective cognitive effort and task performance

in a particular learning condition (see Paas and van Merrienboer 1993). Cognitive (or

mental) effort, as indicated by subjective ratings, is the amount of resources allocated by

the learner to meet the cognitive load demands imposed by the task (Paas et al. 1994).

Training programs that reduce learners’ cognitive load while facilitating knowledge

acquisition have been shown to yield higher instructional efficiency (for a review, see

Clark et al. 2006).

Self-generated elaboration—query method

Explicitly prompting self-generated elaboration of concepts (via self-explanation) has been

shown to facilitate knowledge acquisition and result in a greater understanding of the

domain (e.g., Catrambone and Yuasa 2006; de Bruin et al. 2007; Hermanson et al. 1997;

O’Reilly et al. 1998; Wong et al. 2002). Our past research has also highlighted the ben-

eficial effects of self-generated elaboration on learning outcomes (e.g., Cuevas et al.

2004b; Fiorella et al. 2012). In particular, teaching students the cognitive strategy of

generating questions for new material has been shown to lead to significant gains in

comprehension, as measured by standardized and experimenter-generated tests following

the interventions (Rosenshine et al. 1996). For example, in a series of studies that

examined several learning and study strategies, King (1992) demonstrated how a guided

learner-generated questioning strategy (i.e., open-ended question stems; e.g., ‘How are

____ and ____ related?’), designed to prompt high-level elaboration of new material,

facilitated knowledge acquisition and led to superior performance on objective and essay

tests on the material, compared to learners not presented with such strategies.

Enhancing learning outcomes in computer-based training

123

The present study explored the effectiveness of the query method, a guided sentence-

generation task that prompts learners to elaborate on key concepts presented in the training

material. The query method can be described as an embedded content-dependent strategy

(Osman and Hannafin 1992). By embedding queries within the lessons, learners are

prompted to attend to and interact with the critical concepts in the presented material,

increasing processing of the information and facilitating knowledge acquisition. By

making the queries content-dependent, emphasis can be placed on the lesson’s concepts

and their unique interrelations, promoting learning of the target domain. Thus, the query

method may provide learners with active control of instruction (i.e., their knowledge

acquisition process) by enabling elaboration of the material (Fonseca and Chi 2011).

Relation to past research

Our past research efforts have explored the factors underlying knowledge acquisition and

integration within two distinct contexts: 1) the effect of diagrammatic presentations on the

acquisition of domain knowledge in aviation (Cuevas et al. 2002; Fiore et al. 2003), and 2)

knowledge integration and cognitively diagnostic assessment within distributed team

training environments (Cuevas et al. 2004b; Fiore et al. 2002; Scielzo et al. 2004, 2005).

Over the past successive iterations of our investigation, we have examined not only

knowledge acquisition and integration, but also the cognitive precursors to the develop-

ment of this knowledge, including constructs such as instructional efficiency (e.g., Paas and

van Merrienboer 1993) as well as the role of individual differences in learner aptitudes

(e.g., Gully and Chen 2010), to better converge on an understanding of technology-med-

iated learning. The overall goal of the present study was to further increase our under-

standing of the cognitive processes involved in learning within computer-based training

environments and to investigate how embedded content-dependent strategies can support

this learning process.

Hypotheses

The elaboration prompted by the query method was hypothesized to assist learners in

building internal associations between multiple concepts in the to-be-learned material and

as such, potentially foster a deeper, more integrative understanding of the information (as

indicated by greater similarity to an expert model), which, in turn, would facilitate suc-

cessful performance on questions requiring integration and application of knowledge

(Fonseca and Chi 2011). This effect would not be expected to be as strong for basic factual

knowledge assessment (cf. Cuevas et al. 2002, 2004b; Fiore et al. 2003). Furthermore, by

better enabling learners to build internal associations among the concepts presented, the

query method may reduce the demands imposed by the training on learners’ cognitive

resources (i.e., cognitive load), leading to higher performance on a knowledge assessment

task with less cognitive effort exerted. Thus, this study also evaluated the beneficial effect

of the query method on the training program’s instructional efficiency. Finally, by varying

the level of elaboration prompted by the queries, this study could determine what level of

processing of the to-be-learned material was necessary for this instructional strategy to be

most effective (cf. King 1992). Varying the level of elaboration would help us to under-

stand how embedded content-dependent strategies can be effectively used to promote

optimal levels of germane cognitive load, leading to productive learning outcomes. The

specific hypotheses investigated in this study were as follows:


123

Hypothesis 1 Participants presented with the query method, particularly those in the

high-level elaboration training condition, were hypothesized to exhibit significantly greater

accuracy in their knowledge organization of the presented concepts (as measured via a card

sort task) than participants not presented with the query method.

Hypothesis 2 Participants presented with the query method, when compared to partici-

pants not presented with this instructional strategy, were hypothesized to exhibit signifi-

cantly greater performance on integrative knowledge questions.

Hypothesis 3 No significant difference in performance was hypothesized for declarative

knowledge or perceptual knowledge (concept recognition) questions.

Hypothesis 4 Incorporating the query method into its instructional design was hypoth-

esized to significantly improve the training program’s instructional efficiency in relation to

performance on the knowledge assessment task (declarative, perceptual, and integrative

questions).

Method

Participants

Fifty-four undergraduate students (14 males and 40 females, mean age = 21.39) from the

general psychology department subject pool at a major university in the southeastern

United States participated in this experiment for course credit. Participation in the

experiment was open to all students, regardless of age, race, gender, or nation of origin. A

demographic form was used to screen participants to ensure that data analyzed were only

from participants with no prior knowledge or previous experience in the aviation domain;

no participants indicated prior aviation knowledge or experience. Data from three partic-

ipants were excluded from the analysis due to procedural problems, resulting in a final N of

51 (13 males and 38 females, mean age = 21.39 years). Treatment of all participants was

in accordance with the ethical standards of the American Psychological Association.

Design

This study employed a one-way between-groups design, with the query method serving as

the independent variable, manipulated at three levels: no query (NQ), low-level elaboration

query (LLEQ), and high-level elaboration query (HLEQ). Participants were randomly

assigned to conditions to ensure that each participant had an equal chance of being

assigned to any one of the three experimental groups. Dependent variables included

knowledge organization, knowledge acquisition, and instructional efficiency.

Materials and apparatus

Aviation training tutorial

Three versions (NQ, LLEQ, HLEQ) of an interactive computer-based instructional tutorial

on the principles of flight were developed for this study (see example lesson in Fig. 1).

Material for the tutorial was adapted from the Jeppesen Sanderson Private Pilot Manual

(Jeppesen Sanderson Training Systems 1996c) and the Jeppesen Sanderson Private Pilot


123

Maneuvers Manual (Jeppesen Sanderson Training Systems 1996b), both standard training

products for the instruction of pilots in the private and public sector. The tutorial consisted

of the following three modules.

Module 1 (Airplane Parts) described a number of airplane parts critical for standard

flight operations. Participants were presented with an overview slide and two main slides

(wings, tail), with hyperlinks to four additional slides that provided more detailed expla-

nation of the concepts (e.g., ailerons, elevator).

Module 2 (Flight Movements) discussed the aerodynamics of flight, including infor-

mation about the axes around which an airplane moves and the movements possible in

standard airplane flight. Participants were presented with an overview slide and two main

slides (axes, movements), with hyperlinks to six additional slides that defined the various

axes and movements (e.g., vertical axis, yaw movement).

Module 3 (Flight Instruments) introduced the six primary flight instruments traditionally

used by pilots to navigate the airplane. Participants were presented with an overview slide

and two main slides (pitot-static instruments, gyroscopic instruments), with hyperlinks to

12 additional slides that described how to read the instruments and explained how changes

in the airplane’s movements affected the information displayed on the instruments (e.g.,

airspeed indicator, turn coordinator). Participants were also presented with a hyperlink to

an animated demonstration depicting each of the six flight instruments in motion.

Participants proceeded through the tutorial at their own pace, navigating the hyperlinks

embedded in the tutorial using a standard point-and-click mouse. Participants were free to

move backward and forward through the tutorial. After all the lessons in the respective

module had been viewed, participants in the query method conditions were presented with

the guided sentence-generation task, prompting them to engage in either low or high level

elaboration of the concepts (further described next). Participants in the no-query condition

were presented with the relevant information in the lessons only. At the end of each

module, all participants were given the opportunity to go back and review the lessons

before proceeding to the next module. Though no time limit was imposed, participants

took, on average, approximately 24 min to complete the tutorial.

Fig. 1 Illustrative content of Aviation Training Tutorial (Module 1—Airplane Parts)


123

Query method

Upon completion of the lessons in each module, participants in the query conditions were

presented with a ‘Stop and Think’ exercise in an open-ended question format that asked

them to generate one sentence from a list of key concepts presented in the training.

Participants in the LLEQ condition were prompted to generate a sentence using only one of

the terms from this list (simple sentence). Participants in the HLEQ condition were

prompted to generate a sentence that connected three or more concepts from the list that

best described the relation among those concepts (complex sentence). For example, the list

for Module 1 included the terms: Wings, Tail, Ailerons, Flaps, Vertical Stabilizer, Hori-

zontal Stabilizer, Rudder, Elevator. Choosing among these concepts, a participant in the

LLEQ condition could generate the following simple sentence: ‘‘Pilots use the rudder to

move the airplane’s nose left and right.’’ Conversely, a participant in the HLEQ condition

could generate a complex sentence linking three or more of these concepts in the following

manner: ‘‘During flight, pilots initiate a turn by using the ailerons on the wings in com-

bination with the rudder, which is attached to the back of the vertical stabilizer.’’ Once

submitted, participants could not change their responses to the sentence generation task.

Tutorial survey (instructional efficiency)

Following the tutorial, all participants were presented with a brief Tutorial Survey, which

included an item designed to assess their perceived cognitive load associated with learning

the training material. Specifically, participants were asked to report how easy or difficult

they found it to understand the concepts presented in the training, with responses recorded

on a 7-point anchored scale, ranging from 1 (very easy) to 7 (very difficult). These

responses were used to calculate the training’s instructional efficiency (described later in

the analysis of the results). The use of a single-item measure to assess learner’s perceived

cognitive load has been successfully demonstrated in our previous work (e.g., Cuevas et al.

2002; Scielzo et al. 2005) as well as in other studies (e.g., Kirwan et al. 1997).

Card sort task (knowledge organization)

To evaluate participants’ knowledge organization of task-relevant concepts, the present study

employed the TPL-KATS-card sort software (Harper et al. 2003), a computer-based knowledge

structure assessment product developed by the Team Performance Laboratory at the University

of Central Florida (Copyright 2001). Participants were instructed to group 26 key concepts

extracted from the Aviation Training Tutorial into as many categories as they desired and then

name or describe the categories they created. Though no time limit was imposed, participants

took, on average, approximately 20 min to complete this task, including receiving instructions

on the software program and performing the card sort task itself.

Knowledge assessment task (knowledge acquisition)

Participants completed a self-paced computer-based knowledge assessment task (overall

a = .859) consisting of 48 multiple-choice questions, divided into three sections (see

example question in Fig. 2). Only one question was presented at a time on the screen and

participants proceeded from one question to the next using a standard point-and-click

mouse. Participants were not able to go back and review or change their responses once

they had proceeded to the next question and no feedback was provided as to the accuracy


123

of their responses. Participants, on average, completed this task in about 20 min. The three

sets of questions were presented in the following order:

Factual knowledge assessment (declarative knowledge questions) Twenty questions

(a = .741), adopted from a standard introductory flight manual (Jeppesen Sanderson

Private Pilot Exercise Book; Jeppesen Sanderson Training Systems 1996a), assessed

participants’ mastery of basic factual information associated with the training (e.g., defi-

nitions of the various parts of the plane). Participants were presented with text-based

definitions taken from the Aviation Training Tutorial and were asked to identify the

concept being described.

Concept recognition assessment (perceptual knowledge questions) Eighteen questions

(a = .675) tested participants’ perceptual knowledge with regard to their ability to rec-

ognize key concepts from the Aviation Training Tutorial. Participants were presented with

static illustrations of the principle concepts (airplane parts, axes, movements, and instru-

ments) from the tutorial and were asked to identify the concept depicted.

Airplane function assessment (integrative knowledge questions) Ten questions

(a = .634) assessed participants’ ability to integrate and apply task knowledge. Partici-

pants were presented with a dynamic animated scenario (using audio–video interleaved file

format) illustrating an airplane performing a maneuver and were asked to determine, for

example, which airplane parts and flight instruments were being utilized in this maneuver.

Verbal comprehension ability (covariate)

Since the nature of the material presented in the Aviation Training Tutorial required the

understanding and integration of complex concepts and relations, Part 1 (Verbal

Fig. 2 Illustrative content of knowledge assessment task (integrative knowledge question)


123

Comprehension) of the Guilford-Zimmerman Aptitude Survey (Guilford and Zimmerman

1981) was administered to covary out any effects due to individual differences in this

learner aptitude. Part 1 has a computed odd–even estimate of reliability of .96 and factorial

validity is demonstrated from the results of three factor analyses, with factor loadings

ranging from .70 to .86 (Guilford and Zimmerman 1981). For this paper-and-pencil task,

participants were given 10 min to respond to 72 multiple-choice questions assessing

knowledge of semantic meanings. For each item, participants were presented with a word

(e.g., earth) and were asked to select from among five other words (e.g., sugar, farm, sun,

soil, horse) which one had a meaning like the first word (e.g., soil).

Apparatus

The software program for the Aviation Training Tutorial, knowledge assessment task, and

card sort program were hosted on an IBM compatible Pentium 586 computer with a 15-inch

color monitor, run on Windows XP� operating system. The Aviation Training Tutorial and

knowledge assessment task were presented utilizing Microsoft PowerPoint XP�. Participants

navigated through the tutorial and test using a standard point-and-click mouse. Both keyboard

inputs and the use of a standard point-and-click mouse were required to perform the card sort

task. Multimedia presentation using audio–video-interleaved files were incorporated into the

Airplane Function Assessment (integrative knowledge questions). A paper-and-pencil for-

mat was used to record participants’ responses to the queries embedded in the training, the

Tutorial Survey, and the knowledge assessment task.

Procedure

Upon arrival, participants were randomly assigned to one of the three experimental groups.

Participants first completed an informed consent form, a biographical data form (age,

gender, prior knowledge of aviation), and the measure of verbal comprehension ability.

Participants then proceeded with self-paced computer-based instruction on the principles of

flight using the Aviation Training Tutorial. For the query conditions, the sentence task was

embedded immediately following the last lesson presented in each module. Following the

tutorial, participants were asked to complete the Tutorial Survey. Participants were then

presented with the card sort task, followed by the knowledge assessment task. Finally,

participants were debriefed and extra credit was assigned. On average, the total length of

the experiment, including training and assessment, was approximately 95 min.

Results

Analyses

The experimental results were analyzed separately in terms of post-training learning

outcomes (knowledge organization, knowledge acquisition) and instructional efficiency.

Multivariate analysis statistics are reported using Roy’s largest root. Pairwise comparison

statistics are reported using Fisher’s Least Significant Difference (LSD). Correlations were

calculated, as appropriate. An alpha level of .05 was used for all statistical analyses. Verbal

comprehension ability was significantly correlated with several of the measures and thus

was treated as a covariate in the analysis of these variables. Table 1 lists the


123

intercorrelations, means, and standard deviations for the dependent measures and verbal

comprehension ability.

Check of random assignment

To check the effectiveness of the random assignment procedure used in this study, a

univariate one-way ANOVA was performed on participants’ scores on the verbal com-

prehension ability measure, as this variable should not have been influenced by the

manipulation. Analysis revealed no significant differences among the three experimental

groups on this variable, F (2, 48) \ 1.

Effect of query method on knowledge organization and knowledge acquisition

The cognitive measures were analyzed using a one-way between-groups MANCOVA, with

query method serving as the independent variable and verbal comprehension ability treated

as a covariate. The dependent measures included knowledge organization, as measured via

the card sort task, and knowledge acquisition, as measured using the knowledge assessment

task. Multivariate analysis revealed a significant effect for the query method on these

cognitive measures, F (4, 45) = 3.202, p = .021, gp2 = .222. Verbal comprehension ability

was a significant covariate, F (4, 44) = 2.733, p = .041, gp2 = .199. Univariate analysis

for each of the measures will be presented next. Adjusted means and standard errors are

reported in Table 2.

Knowledge organization (card sort task) (Hypothesis 1)

The card sort task was used to assess the degree to which presentation of the query method

affected similarity to an expert model. A quantitative measure was derived from the card

sort data to determine the connectedness among concepts. First, a list of all possible

pairings of the 26 concepts was generated (N = 325). A value of 1 was assigned to pairings

of concepts falling within the same group (i.e., if the participant grouped the pair of

concepts together in the same category) and a value of 0 was assigned for the remaining

concept pairs (i.e., for pairings where the participants did not group the two concepts

together in the same category). Each participant’s card sort data (i.e., the generated list of

Table 1 Intercorrelations, means, and standard deviations of verbal comprehension ability and cognitivemeasures a

Dependent variable 1 2 3 4 5 6 M SD

(1) Verbal comprehension – .05 .38** .32* .41** .23 .359 .123

(2) Knowledgeorganization

– .30* .27 .39** .06 .519 .219

Knowledge acquisition

(3) Total – .89** .90** .77** .571 .173

(4) Declarative – .68** .52** .582 .195

(5) Perceptual – .61** .611 .185

(6) Integrative – .478 .235

a N = 51

* p \ .05 (two-tailed). ** p \ .01 (two-tailed)


123

the participant’s pairings of all the concepts) was compared to the card sort data generated

by the subject matter expert. This expert had approximately 7,000 h as a pilot and

approximately 2,700 h as an instructor and participated in the creation and evaluation of

the tutorial. By calculating the Pearson r correlation coefficient between the participant’s

and subject matter expert’s card sort data, a participant’s sensitivity to identifying the

critical relations among the concepts can be evaluated. Hence, the similarity of their

pairings to the expert model would indicate the accuracy of the participant’s knowledge

organization of the task.

Univariate tests revealed a significant effect of the query method on participants’

knowledge organization, as indicated by the mean correlation between the participants’ and

the expert model card sort pairings (see Table 2). However, pairwise comparisons showed

an unexpected pattern of results. Specifically, the LLEQ participants exhibited a signifi-

cantly greater mean correlation with the expert model than the HLEQ participants,

p = .016. Although the LLEQ participants’ card sort mean correlation to the expert model

was also greater than the NQ participants, this difference was not significant, p = .098. No

significant difference was found between the NQ and the HLEQ participants, p = .413.

Knowledge acquisition (Hypotheses 2 and 3)

Univariate tests revealed a significant effect of the query method on participants’ overall

performance on the knowledge assessment task (see Table 2). Pairwise comparisons

showed that the LLEQ participants significantly outperformed the NQ participants on the

knowledge assessment task overall, p = .006. Although the LLEQ participants’ perfor-

mance was also greater than the HLEQ participants, this difference was not significant,

Table 2 Adjusted means and standard errors for knowledge organization and knowledge acquisitionmeasures by training condition

Dependent variable Training conditiona ANOVA results

NQ LLEQ HLEQ F (2, 47) p, gp2obs. power

Knowledge organization .499 (.051)[.395, .602]

.621 (.051)a

[.518, .725].438 (.052)b

[.334, .542]3.269 .047, .122, .594

Knowledge acquisition

Total .500 (.037)c

[.426, .574].650 (.037)d

[.576, .725].564 (.037)[.489, .638]

4.177 .021, .151, .708

Declarative .526 (.045)[.436, .616]

.646 (.045)[.556, .736]

.575 (0.45)[.485, .666]

1.840 .170, .073, .365

Perceptual .551 (.040)a

[.471, .631].691 (.040)b

[.612, .771].591 (.040)[.511, .671]

3.343 .044, .125, .604

Integrative .358 (.052)c

[.253, .464].585 (.052)d

[.480, .690]492 (.052)[.387, .598]

4.742 .013, .168, .765

Knowledge Organization values represent mean correlation with expert model. Values for KnowledgeAcquisition represent mean percent correct. Numbers in brackets represent 95 % confidence interval.gp2 = partial eta squared; obs. power = observed power computed using alpha = .05. Means in the samerow with different superscripts ‘a’ and ‘b’ differ significantly at p \ .05, two-tailed, by the Fisher LeastSignificant Difference (LSD) comparison; different superscripts ‘c’ and ‘d’ indicate significant difference atp \ .01, two-taileda n = 17 for each condition


123

p = .106. No significant difference was found between the NQ and the HLEQ participants,

p = .230.

Univariate tests revealed no significant difference in performance on the declarative

knowledge (basic factual knowledge) questions (see Table 2). Although the LLEQ par-

ticipants did perform better than the NQ participants and HLEQ participants, these dif-

ferences were not significant, p = .062 and p = .270, respectively.

Univariate tests revealed a significant difference in performance on the perceptual

knowledge (concept recognition) questions (see Table 2). Pairwise comparisons showed

that the LLEQ participants significantly outperformed the NQ participants, p = .016.

Although the LLEQ participants’ performance was also greater than the HLEQ partici-

pants, this difference was not significant, p = .081. No significant difference was found

between the NQ and the HLEQ participants, p = .476.

Univariate tests revealed a significant difference in performance on the integrative

knowledge (airplane function) questions (see Table 2). Pairwise comparisons showed that

the LLEQ participants significantly outperformed the NQ participants, p = .004. Although

the LLEQ participants’ performance was also greater than the HLEQ participants, this

difference was not significant, p = .219. No significant difference was found between the

NQ and the HLEQ participants, p = .078.

Effect of query method on instructional efficiency (Hypothesis 4)

The means and standard errors for perceived cognitive load and the instructional efficiency

scores associated with the different training conditions are reported in Table 3. A one-way

between-groups ANOVA was conducted, comparing participants’ self-reported cognitive

load in the three training conditions. Univariate tests revealed no significant differences in

perceived cognitive load during training.

Next, the instructional efficiency (E) of the training program was calculated by plotting

the standardized scores on measures of cognitive effort (R) (i.e., self-report of task diffi-

culty as indicated by responses on the Tutorial Survey cognitive load item) against the

standardized scores on measures of performance (P) (declarative, perceptual, and inte-

grative questions), displayed as a cross of axes (for a detailed description of this procedure,

see Paas and van Merrienboer 1993). Instructional efficiency was calculated using the

following equation (adapted from Kalyuga et al. 1999): E = (P – R)/SQRT (2). The values

of P and R determine the sign of E. If P [ R, then E will be positive, indicating higher

efficiency (i.e., cognitive effort exerted is less, relative to the standard effort required to

achieve that level of performance). If P \ R, then E will be negative, indicating lower

efficiency (i.e., cognitive effort exerted is greater, relative to the standard effort required to

achieve that level of performance). Baseline (or standard level of efficiency) is represented

by E = 0.

Instructional efficiency (E) scores were analyzed using a one-way between-groups

MANOVA, with query method serving as the independent variable. Verbal comprehension

ability was not a significant covariate, and thus was not included in the analysis. The

dependent measures reflected the instructional efficiency of the training program in relation

to performance on the declarative, perceptual, and integrative questions. Multivariate

analysis revealed a significant effect for the query method on the training program’s

instructional efficiency, F (3, 47) = 3.648, p = .019, gp2 = .189. Univariate tests revealed

a significant effect of the query method on the training program’s instructional efficiency in

relation to performance on all three sets of knowledge questions (see Table 3).


123

The LLEQ training condition consistently yielded positive instructional efficiency

scores (i.e., greater performance was achieved with less perceived cognitive effort),

whereas the NQ training condition consistently yielded negative instructional efficiency

scores (i.e., poorer performance with greater perceived cognitive effort) (see Fig. 3).

Instructional efficiency for the HLEQ training condition was typically at baseline (near

zero) (i.e., standard level of performance was achieved relative to perceived cognitive

effort). Pairwise comparisons showed that instructional efficiency scores between the

LLEQ and NQ training conditions significantly differed for the declarative (p = .005),

perceptual (p = .003), and integrative (p = .002) questions. Significant differences

between the LLEQ and HLEQ training conditions were found only for the perceptual

questions, p = .043. No significant differences were found between the NQ and HLEQ

training conditions.

Table 3 Means and standard errors for cognitive load and instructional efficiency scores by trainingcondition

Dependent variable Training conditionsa ANOVA results

NQ LLEQ HLEQ F (2, 48) p, gp2obs. power

Cognitive load 3.412 (0.283)[2.843, 3.981]

2.471 (0.283)[1.902, 3.039]

2.941 (0.283)[2.372, 3.510]

2.768 .073, .103, .520

Instructional efficiency

Declarative -0.457 (0.236)c

[-0.932, 0.017]0.520 (0.236)d

[0.046, 0.994]-0.065 (0.236)[-0.539, 0.410]

4.346 .018, .153, .727

Perceptual -0.476 (0.247)c

[-0.972, 0.200]0.599 (0.247)ad

[0.103, 1.096]-0.125 (0.247)b

[-0.622, 0.371]4.937 .011, .171, .784

Integrative -0.618 (0.266)c

[-1.154, -0.83]0.605 (0.266)d

[0.069, 1.141]0.011 (0.266)[-0.525, 0.547]

5.272 .009, .180, .811

Instructional Efficiency values represent mean E score. Numbers in brackets represent 95 % confidenceinterval. gp2 = partial eta squared; obs. power = observed power computed using alpha = .05. Means inthe same row with different superscripts ‘a’ and ‘b’ differ significantly at p \ .05, two-tailed, by the FisherLeast Significant Difference (LSD) comparison; different superscripts ‘c’ and ‘d’ indicate significant dif-ference at p \ .01, two-taileda n = 17 for each condition

Fig. 3 Effect of query method on training program’s instructional efficiency


123

Time-on-task

To evaluate any potential differences among the query conditions in the time to complete the

training and performance tasks, a one-way between-groups MANOVA was conducted, with

query method serving as the independent variable. Dependent variables included time-on-

task (recorded in minutes) for completing the Aviation Training Tutorial (training) and the

card sort and knowledge assessment tasks (performance). Multivariate analysis revealed a

significant effect for the query method on time-on-task, F (3,47) = 14.899, p \ .0005,

gp2 = .487. However, univariate tests revealed a significant difference in time-on-task only

for completion of the Aviation Training Tutorial (refer to Table 4). As would be expected,

pairwise comparisons showed that both the LLEQ and the HLEQ participants invested sig-

nificantly greater time-on-task for their training than the NQ participants, p \ .0005. No

significant difference for training time-on-task was found between the LLEQ and HLEQ

participants, p = .190. Univariate tests revealed no significant difference for time-on-task in

performing either the card sort task or the knowledge assessment task. Overall, these findings

indicate that participants in the query method condition invested more time in their training;

however, this was likely due to the extra time required to complete the sentence generation

task. Yet, the LLEQ and HLEQ participants did not necessarily need to take any longer than

the NQ participants in performing the post-training tasks.

Content analysis of query responses

The sentences generated by the participants in the LLEQ and HLEQ training conditions

were evaluated with regard to the number of concepts used in each sentence as well as the

accuracy of the sentences. Recall that the HLEQ training condition prompted participants

to generate a sentence using three or more concepts from the list presented whereas the

LLEQ training condition prompted participants to generate a sentence using only one of

these concepts. The number of concepts used in the sentence generated for each of the

three modules was summed to calculate the total number of concepts used during training.

The expected minimum number of concepts possible would be three for the LLEQ training

condition (one concept per module) and nine for the HLEQ training condition (at least

three concepts per module). Accuracy of the sentence generated for each module was rated

by a subject matter expert as either ‘0’ for inaccurate or ‘1’ for accurate, resulting in a

range from 0 (none of the sentences were accurate) to 3 (all three sentences were accurate).

Table 4 Means and standard errors for time-on-task by training condition

Dependent variable Training conditionsa ANOVA results

NQ LLEQ HLEQ F (2, 48) p, gp2

Time on task

Tutorial 17.824 (1.378)a 25.529 (1.378)b 28.118 (1.378)b 15.111 \.005, .386

Card sort 18.765 (1.225) 20.529 (1.225) 18.588 (1.225) \ 1 .470, .031

Knowledge assessment 18.706 (1.231) 20.235 (1.231) 21.000 (1.231) \ 1 .413, .036

Values for Time on task represent mean time in minutes. gp2 = partial eta squared. Means in the same rowwith different superscripts ‘a’ and ‘b’ differ significantly at p \ .01, two-tailed, by the Fisher Least Sig-nificant Difference (LSD) comparisona n = 17 for each condition


123

An independent samples t test was conducted to analyze the total number of concepts

used and sentence accuracy, with training condition (LLEQ or HLEQ) serving as the

between-groups factor. As would be expected, analysis showed that participants in the

HLEQ training condition (M = 10.940; SD = 2.384) used a significantly greater total

number of concepts in generating their sentences than participants in the LLEQ training

condition (M = 5.060; SD = 2.164), t (32) = 7.532, p \ .0005, two-tailed. These results

suggest that participants were able to complete the sentence generation task as instructed.

With regard to the accuracy of the sentences, participants in the LLEQ training condition

(M = 2.820; SD = 0.529) generated significantly more accurate sentences than partici-

pants in the HLEQ training condition (M = 2.350; SD = 0.786), t (32) = 2.049, p = .049,

two-tailed.

Discussion

Training effectiveness can be evaluated in terms of how well instruction enhances learning

outcomes while minimizing the cognitive load imposed on learners during training. The

present study explored the effectiveness of a guided, learner-generated instructional

strategy (query method) in achieving this objective within computer-based training. This

study also examined the effect of varying the level of elaboration prompted by the queries,

asking participants to generate either simple (low-level elaboration) or complex (high-level

elaboration) sentences. Overall, results consistently highlighted the beneficial effect of

presenting participants with low-level elaboration queries, as compared to no queries or

high-level elaboration queries. Further, incorporating the high-level elaboration queries

into the training consistently failed to produce significantly better post-training outcomes

than the no-query training condition. These findings are next discussed in the context of

instructional strategies.

Learning as a constructive cognitive activity

The results of the present study support the utility of embedded content-dependent strat-

egies within computer-based training. Embedding the low-level elaboration queries within

the training may have prompted participants to attend to and interact with the critical

concepts in the presented material, increasing the efficiency of their information processing

and facilitating their knowledge acquisition. The content-dependent nature of the queries

may have promoted learning of the target domain by emphasizing the key concepts and

their unique interrelations, resulting in more accurate knowledge organization and better

integration of task-relevant concepts.

Query method—low versus high level elaboration

Incorporating the low-level elaboration queries into the training program significantly

enhanced post-training learning outcomes. Participants presented with the low-level

elaboration queries exhibited significantly more accurate knowledge organization (as

indicated by greater similarity to an expert model), better concept recognition (perceptual

knowledge questions), and superior performance on integrative knowledge questions

involving the integration and application of task-relevant concepts. Consistent with our

previous studies (Cuevas et al. 2002, 2004b; Fiore et al. 2003), no significant differences in

performance were found on basic factual knowledge assessment. Presentation of the low-


123

level elaboration queries also significantly improved the training program’s instructional

efficiency; that is, greater performance was achieved with less perceived cognitive effort.

Elaboration, learner control, and cognitive load

Prompting a high level of elaboration did not result in a significant difference in post-

training outcomes, when compared to no elaboration. Two potential explanations may

account for this somewhat counterintuitive finding. First, participants proceeded through

the aviation training tutorial at their own pace and could freely review the material,

allowing for a high level of learner control during initial training. However, one could

argue that learner control for reinforcing their newly acquired knowledge may have varied

across the two query method conditions. Specifically, arbitrarily forcing learners to gen-

erate complex sentences may have limited the level of control they could exercise as they

attempted to complete tasks designed to further develop their understanding of the training

concepts. In contrast, giving learners the option of generating simple sentences may have

allowed them to freely choose the appropriate level of complexity required in their elab-

oration of the material (Hasler et al. 2007). Learner control is essential if learners are to be

encouraged to take an active approach to learning, that is, make a deliberate and systematic

cognitive effort to engage the material during the learning process (Fonseca and Chi 2011).

A second explanation for this lack of effect for the high-level elaboration query focuses

on the cognitive load imposed on learners during their training. The findings in the present

study suggest that there may exist an optimal level of elaboration necessary to achieve the

desired learning gains. Indeed, the most optimal performance outcomes were yielded by

prompting participants to engage in a low level of elaboration of the training material. No

elaboration of the training material, as associated with the no-query training condition,

yielded significantly lower levels of post-training performance. Yet, requiring participants

to generate a high level of elaboration of the training material did not yield a corresponding

gain in post-training outcomes. Moreover, performance was not significantly greater than

the no-query training condition.

This lack of significant effect for the high-level elaboration query training condition

may have been due to the increased cognitive load associated with generating complex

sentences. According to Marcus et al. (1996, p. 50), ‘‘If multiple elements must be con-

sidered simultaneously because of high element interactivity, cognitive load may be high

and understanding difficult.’’ Similarly, forcing learners to attend to multiple sources of

information simultaneously may overburden their working memory capacity, reducing the

cognitive resources available for successful learning to occur (Sweller et al. 2011).

Interestingly, no significant differences were found on ratings of perceived cognitive load

during training between the two query method conditions. One possible explanation for

this result may be because participants did not consider the sentence generation task in

their ratings and focused on rating the training material itself. Further research is warranted

to investigate this unexpected finding.

By prompting participants to build internal associations between three or more concepts

in the to-be-learned material, the high-level elaboration queries were expected to promote a

deeper, more integrated understanding of the information (Fonseca and Chi 2011), leading

to more accurate knowledge organization. Yet, participants presented with the low-level

elaboration queries exhibited significantly more accurate knowledge organization and

generated more accurate sentences than those presented with the high-level elaboration

queries. Additionally, no significant differences were found between the no-query and

high-level elaboration query training conditions on knowledge organization, knowledge


123

acquisition, or instructional efficiency. These findings suggest that generating complex

sentences was no more effective than not generating any sentences at all.

The beneficial effects of the query method on learners’ cognitive processes may have

been diminished by the increased cognitive load associated with the complexity of com-

pleting the high-level elaboration sentence generation task. Although this method was

designed to purposely enhance processing of the training material, the high-level elabo-

ration may have increased cognitive processing over a theoretical optimal level, thus

possibly imposing too great a germane cognitive load on participants during their training

or inadvertently introducing extraneous cognitive load. Specifically, when coupled with the

high intrinsic cognitive load of the training material, the increased germane or extraneous

cognitive load associated with the high-level elaboration may have minimized the cog-

nitive resources available for enhanced learning, as compared to the low-level elaboration

training condition. However, these findings are tentative pending further research to more

precisely investigate the effect of the query method on different types of cognitive load

(intrinsic, germane, and/or extraneous) and evaluate the resulting impact on learning

outcomes.

Instructional efficiency

The present study also demonstrated that evaluating a training program’s instructional

efficiency may serve as a potentially diagnostic measure of training effectiveness. Com-

bining subjective ratings of cognitive effort with performance scores may reveal useful

information about the effectiveness of training programs in terms of the cognitive costs

associated with complex task training over and above what would be found by using

measures of cognitive effort or performance alone (Paas and van Merrienboer 1993). As

discussed, results indicated that requiring participants to generate complex sentences (high-

level elaboration query) may have inadvertently increased the cognitive load associated

with the training, limiting the cognitive resources available for successful learning to

occur. Although, overall, this training condition yielded standard levels of performance

relative to the perceived cognitive effort exerted during training, such baseline (near zero)

instructional efficiency scores are not as ideal as the significantly higher instructional

efficiency scores yielded for the low-level elaboration query training condition.

Furthermore, an important distinction should be made between time and effort. On the

one hand, efficiency can be discerned from performance and amount of time spent on

learning. On the other hand, efficiency can be discerned from performance and amount of

effort invested in learning. Although time and effort are associated, these constructs are not

always equal. In this study, we focused on the effort/performance relationship for inves-

tigating the instructional efficiency of training programs.

Limitations and implications for future research

Key limitations to the present study should be noted. First, the training material used in this

study was based on introductory concepts related to the principles of flight. Although this

domain is more complex relative to the training material explored in prior studies (e.g.,

how a bicycle pump works, how lightning forms; Mayer, 2001; Moreno and Mayer, 2002),

to increase the external validity of this study’s findings, the query method needs to be

investigated with concepts of increasing complexity that are more relevant to advanced

training programs. Also, because participants used in this study were undergraduate stu-

dents enrolled in psychology courses, the generalizability of these findings to training in


123

complex operational environments is limited. Thus, it is necessary to examine how the

query method might affect learners from different populations, such as students in tech-

nical training courses.

In addition, the utility of the query method could be evaluated with learners with greater

prior knowledge in the target domain. In this way, future research can investigate the

degree to which the query method may have a differential effect on more experienced

populations (cf. de Bruin et al. 2007). Researchers in the study of expertise have argued

that instructional strategies need to be better tailored to where the learner falls on the

continuum of expertise (Fiore et al. 2008). As noted earlier, participants used in this study

had no prior knowledge or previous experience in the aviation domain. Yet, it is possible

that the high-level elaboration queries may be more beneficial for learners who have

already acquired some prior knowledge of this domain (e.g., student pilots in basic aviation

courses).

Second, although the reliability of the overall knowledge assessment task was accept-

able, the internal consistency of the individual sections was not as high as would be desired

to draw definitive conclusions from these results. This decrease may be due to the limited

number of questions in each section. This issue can be addressed with further testing using

a larger set of items. Finally, although the findings of this study are promising, conclusions

drawn from these results are limited by the small sample size (n = 17 per condition). Thus,

further research is required to validate the effectiveness of the query method.

Theoretical and practical implications

The results reported here build upon the findings of our past research investigating the use

of interactive computer-based training technology to facilitate knowledge acquisition and

integration for complex task training ( Cuevas et al. 2002, 2004b; Fiore et al. 2002, 2003;

Scielzo et al. 2004, 2005; for a review, see Cuevas et al. 2004a). This final section attempts

to integrate the findings in the present study with our prior work in order to highlight both

theoretical and practical implications for enhancing learning outcomes in computer-based

training.

Theoretical implications

Our earlier work (Cuevas et al. 2002; Fiore et al. 2003) demonstrated how diagrammatic

presentation can be effectively used in computer-based training to facilitate knowledge

acquisition of complex concepts. Illustrative diagrams may provide learners with a scaf-

folding framework for knowledge construction, enabling them to integrate the concepts

presented in the training more effectively. In particular, diagrams supported the devel-

opment of mental models more similar to an expert, leading to increased performance on

tests requiring the integration of concepts from across the training. The present study

continued with this line of research by demonstrating how learner-generated elaboration

may also support knowledge integration of complex concepts.

In addition, the findings in the present study suggest that when attempting to master an

already inherently complex domain (high intrinsic cognitive load), instructional strategies

that force the integration of these concepts (e.g., requiring generation of complex sen-

tences) may create too great a germane cognitive load or inadvertently introduce extra-

neous cognitive load, negatively interfering with learners’ knowledge construction by

overloading their already limited cognitive resources. As such, instructional strategies


123

should be designed to lead learners to levels of cognitive effort appropriate to the training

content and not be overly burdensome if the content itself is already complex (i.e., has a

high intrinsic load).

Practical implications

Our programmatic research efforts have consistently demonstrated the value of adopting a

multi-faceted approach for evaluating post-training outcomes within computer-based

training environments (Cuevas et al. 2002; Fiore et al. 2002, 2003; Scielzo et al. 2004). As

in our previous work, this study employed a computer-based training and performance

assessment system that incorporated instructional strategies (query method) with tests

designed with different types of knowledge assessment questions (declarative, perceptual,

integrative). These current findings provide further validation for utilizing several distinct

yet related approaches designed to evaluate the impact of instructional strategies, such as

the query method, on learning outcomes.

In addition to mastery of both basic factual knowledge and concept recognition, suc-

cessful training programs must also prepare learners to effectively apply their newly

acquired knowledge to more complex situations than were experienced during their

training (Chipman et al. 2013). As such, post-training assessment needs to include tasks,

utilizing dynamic task-relevant scenarios, that provide opportunities for evaluating how

well learners can integrate and apply different knowledge concepts (Fiore et al. 2002).

Post-training assessment of learning outcomes should also evaluate the level of accuracy of

learners’ knowledge organization, that is, the degree to which learners’ knowledge

structures exhibit similarity to an expert model. The beneficial effect of the low-level

elaboration queries on learners’ cognitive processes was better diagnosed via such a multi-

faceted approach to knowledge assessment. Finally, the present study also highlighted the

importance of evaluating a training program’s instructional efficiency, as this measure may

be more diagnostic in determining why seemingly useful instructional design features may

not lead to the most optimal post-training outcomes.

Conclusion

In sum, the results of this study suggest important implications with regard to the design

and evaluation of computer-based training programs. Prompting learners to generate low

level elaborations of the training material led to improved post-training learning outcomes,

including more accurate knowledge organization, better concept recognition, superior

performance on tasks involving integration and application of concepts, and higher

instructional efficiency. These findings also implicate the importance of utilizing a multi-

faceted approach to assessing training effectiveness. Equally important is gauging the

training program’s instructional efficiency; that is, the evaluation of a training program’s

design must also consider the cognitive costs associated with the training relative to the

performance achieved. Advances in technology and instructional design will continue to

drive organizations to rely more on technology-mediated learning approaches. As such, it

is essential that training program designers incorporate useful instructional strategies,

guided by theory and research, to prompt learners to actively engage the material during

the learning process (Fonseca and Chi 2011).


123

Acknowledgments The views herein are those of the authors and do not necessarily reflect those of theorganizations with which the authors are affiliated. The research reported in this paper is based upon thedoctoral dissertation of Haydee M. Cuevas, University of Central Florida. Portions of this paper werepresented at the Human Factors and Ergonomics Society 50th Annual Meeting. This research was partiallysupported by funding through Grant Number F49620-01-1-0214 from the Air Force Office of ScientificResearch to Eduardo Salas, Stephen M. Fiore, and Clint A. Bowers.

References

Brunken, R., Plass, J. L., & Leutner, D. (2004). Assessment of cognitive load in multimedia learning withdual-task methodology: Auditory load and modality effects. Instructional Science, 32, 115–132.

Catrambone, R., & Yuasa, M. (2006). Acquisition of procedures: The effects of example elaborations andactive learning exercises. Learning and Instruction, 16(2), 139–153.

Chipman, S. F., Segal, J. W., & Glaser, R. (Eds.) (2013). Thinking and learning skills: Vol. 2: Research andopen questions. New York, NY: Routledge–Taylor and Francis Group.

Clark, R. C., Nguyen, F., & Sweller, J. (2006). Efficiency in learning: Evidence-based guidelines to managecognitive load. San Francisco: Jossey-Bass.

Cuevas, H. M., Fiore, S. M., Bowers, C. A., & Salas, E. (2004a). Fostering constructive cognitive andmetacognitive activity in computer-based complex task training environments. Computers in HumanBehavior, 20, 225–241.

Cuevas, H. M., Fiore, S. M., Bowers, C. A., & Salas, E. (2004b). Using guided learner-generatedinstructional strategies to transform learning into a constructive cognitive and metacognitive activity.Proceedings of the 48th Annual Meeting of the Human Factors and Ergonomics Society, (pp.1049–1053). Santa Monica, CA: Human Factors and Ergonomics Society.

Cuevas, H. M., Fiore, S. M., & Oser, R. L. (2002). Scaffolding cognitive and metacognitive processes in lowverbal ability learners: Use of diagrams in computer-based training environments. Instructional Sci-ence, 30, 433–464.

de Bruin, A., Rikers, R., & Schmidt, H. (2007). The effect of self-explanation and prediction on thedevelopment of principled understanding of chess in novices. Contemporary Educational Psychology,32(2), 188–205.

Fiore, S. M., Cuevas, H. M., & Oser, R. L. (2003). A picture is worth a thousand connections: Thefacilitative effects of diagrams on task performance and mental model development. Computers inHuman Behavior, 19, 185–199.

Fiore, S. M., Cuevas, H. M., Scielzo, S., & Salas, E. (2002). Training individuals for distributed teams:Problem solving assessment for distributed mission research. Computers in Human Behavior, 18,729–744.

Fiore, S. M., Hoffman, R. R., & Salas, E. (2008). Learning and performance across disciplines: An epiloguefor moving multidisciplinary research towards an interdisciplinary science of expertise. MilitaryPsychology, 20(S1), S155–S170.

Fiore, S. M., & Salas, E. (Eds.). (2007). Toward a science of distributed learning. Washington, DC:American Psychological Association.

Fiorella, L., Vogel-Walcutt, J. J., & Fiore, S. (2012). Differential impact of two types of metacognitiveprompting provided during simulation-based training. Computers in Human Behavior, 28(2), 696–702.

Fonseca, B., & Chi, M. T. H. (2011). The self-explanation effect: A constructive learning activity. In R.E. Mayer & P. A. Alexander (Eds.), The handbook of research on learning and instruction (pp.296–321). New York: Routledge—Taylor and Frances Group.

Guilford, J. P., & Zimmerman, W. S. (1981). Manual of instructions and interpretations for the Guilford-Zimmerman Aptitude Survey (revised ed.). Palo Alto, CA: Consulting Psychological Press.

Gully, S., & Chen, G. (2010). Individual differences, attribute-treatment interactions, and training outcomes.In S. W. J. Kozlowski & E. Salas (Eds.), Learning, training, and development in organizations (pp.3–64). New York: Routledge–Taylor and Francis Group.

Harper, M. E., Jentsch, F., Berry, D., Lau, H. C., Bowers, C., & Salas, E. (2003). TPL-KATS—Card Sort: Atool for assessing structural knowledge. Behavior Research Methods, Instruments, and Computers,35(4), 577–584.

Hasler, B. S., Kersten, B., & Sweller, J. (2007). Learner control, cognitive load and instructional animation.Applied Cognitive Psychology, 21, 713–729.

Hermanson, D. R., Hermanson, H. M., & Tompkins, J. G, I. V. (1997). The impact of self-generatedelaboration on students’ recall of finance concepts. Journal of Financial Education (Fall), 23, 27–34.


123

Jeppesen Sanderson Training Systems. (1996a). Jeppesen Sanderson Private Pilot Exercises Book. Engle-wood, CO: Jeppesen Sanderson Inc.

Jeppesen Sanderson Training Systems. (1996b). Jeppesen Sanderson Private Pilot Maneuvers Manual (6thed.). Englewood, CO: Jeppesen Sanderson Inc.

Jeppesen Sanderson Training Systems. (1996c). Jeppesen Sanderson Private Pilot Manual (15th ed.).Englewood, CO: Jeppesen Sanderson Inc.

Kalyuga, S., Chandler, P., & Sweller, P. (1999). Managing split-attention and redundancy in multimediainstruction. Applied Cognitive Psychology, 13, 351–371.

Kalyuga, S., Chandler, P., & Sweller, J. (2004). When redundant on-screen text in multimedia technicalinstruction can interfere with learning. Human Factors, 46, 567–581.

King, A. (1992). Facilitating elaborative learning through guided student-generated questioning. Educa-tional Psychologist, 27, 111–126.

Kirschner, P. A. (2002). Cognitive load theory: Implications of cognitive load theory on the design oflearning. Learning and Instruction, 12, 1–10.

Kirwan, B., Evans, A., Donohoe, L., Kilner, A., Lamoureux, Atkinson, T. & MacKendrick, H. (1997).Human factors in the ATM system design life cycle. In Paper presented at the FAA/Eurocontrol ATMR&D Seminar, Paris, France, 16–20 June 1997. Retrieved from http://www.atmseminar.org/seminarContent/seminar1/papers/p_007_CDR.pdf. Accessed 19 Jan 2012.

Marcus, N., Cooper, M., & Sweller, J. (1996). Understanding instructions. Journal of Educational Psy-chology, 88, 49–63.

Mayer, R. E. (2001). Multimedia learning. Cambridge, England: Cambridge University Press.Mayer, R. E., Hegarty, M., Mayer, S., & Campbell, J. (2005). When static media promote active learning:

Annotated illustrations versus narrated animations in multimedia instruction. Journal of ExperimentalPsychology: Applied, 11(4), 256–265.

Mayer, R. E., Heiser, J., & Lonn, S. (2001). Cognitive constraints on multimedia learning: When presentingmore material results in less understanding. Journal of Educational Psychology, 93, 187–198.

Moreno, R., & Mayer, R. E. (2002). Verbal redundancy in multimedia learning: When reading helpslistening. Journal of Educational Psychology, 94(1), 156–163.

O’Reilly, T., Symons, S., & MacLatchy-Gaudet, H. (1998). A comparison of self-explanation and elabo-rative interrogation. Contemporary Educational Psychology, 23(4), 434–445.

Osman, M. E., & Hannafin, M. J. (1992). Metacognition research and theory: Analysis and implications forinstructional design. Educational Technology Research and Development, 40, 83–99.

Paas, F. G. W. C., & van Merrienboer, J. J. G. (1993). The efficiency of instructional conditions: Anapproach to combine mental effort and performance measures. Human Factors, 35, 737–743.

Paas, F. G. W. C., van Merrienboer, J. J. G., & Adam, J. J. (1994). Measurement of cognitive load ininstructional research. Perceptual and Motor Skills, 79, 419–430.

Rosenshine, B., Meister, C., & Chapman, S. (1996). Teaching students to generate questions: A review ofthe intervention studies. Review of Educational Research, 66, 181–221.

Salas, E., & Rosen, M. A. (2010). Experts at work: Principles for developing expertise in organizations. In S.W. J. Kozlowski & E. Salas (Eds.), Learning, training, and development in organizations (pp. 99–134).New York: Routledge–Taylor and Francis Group.

Scielzo, S., Cuevas, H. M, & Fiore, S. M. (2005). Investigating individual differences and instructional efficiency incomputer-based training environments. Proceedings of the Human Factors and Ergonomics Society 49thAnnual Meeting (pp. 1251–1255). Santa Monica, CA: Human Factors and Ergonomics Society.

Scielzo, S., Fiore, S. M., Cuevas, H. M., & Salas, E. (2004). Diagnosticity of mental models in cognitive andmetacognitive processes: Implications for synthetic task environment training. In S. G. Schiflett, L.R. Elliott, E. Salas, & M. D. Coovert (Eds.), Scaled worlds: Development, validation, and applications(pp. 181–199). Aldershot, UK: Ashgate.

Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning andInstruction, 4, 295–312.

Sweller, J. (2010). Element interactivity and intrinsic, extraneous, and germane cognitive load. EducationalPsychology Review, 22, 123–138.

Sweller, J., Ayers, P., & Kalyuga, S. (2011). Cognitive load theory. New York: Springer.Sweller, J., van Merrienboer, J. J. G., & Paas, F. (1998). Cognitive architecture and instructional design.

Educational Psychology Review, 10, 251–296.van Merrienboer, J. J. G., Schuurman, J. G., de Croock, M. B. M., & Paas, F. G. W. C. (2002). Redirecting

learners’ attention during training: Effects on cognitive load, transfer test performance and trainingefficiency. Learning and Instruction, 12, 11–37.

Wong, R. M. F., Lawson, M. J., & Keeves, J. (2002). The effects of self-explanation training on students’problem solving in high-school mathematics. Learning and Instruction, 12, 233–262.


123

http://www.atmseminar.org/seminarContent/seminar1/papers/p_007_CDR.pdf

http://www.atmseminar.org/seminarContent/seminar1/papers/p_007_CDR.pdf

Documents

Enhancing learning outcomes in computer-based training via self-generated elaboration