Learning Performance and Computer Software:

An Exploration of Transfer

Dr. Robin Kay

University of Ontario Institute of Technology

Presented at SITE 2004, Atlanta.

faculty.uoit.ca/kay/site2004/site2004kay.htm

Overview

Context of Research Literature Review Purpose Method Results Conclusions Suggestions for Educators Future Research

Context

Software From To VersionsAverage(Months)

Internet Explorer 1995 2001 7 10

Excel 1993 2003 8 15

Windows 1990 2001 7 19

Visual Basic 1991 2002 7 19

Word 1995 2003 5 19

Adobe Photoshop 1990 2004 7 24

Average 7 18

Intel Processors 1993 2003 12 10

Context

How educators keep up with the Rapid pace of change Range of computer software available

One approach: Investigate knowledge transfer

Literature ReviewOverview

Few studies have looked at transfer and computer software (Stine & Wildemuth, 1990)

Little is known about what constitutes proper transfer design in training (Yamhill & McLean, 2002)

Literature ReviewDetails

General Knowledge Overwhelming evidence suggests that general skills do not

readily transfer to new contexts (e.g., Anderson et. al., 1996; Carroll, 1990)

Specific Knowledge Need to focus on what tasks are similar enough to transfer (e.g.,

Bransford, et. al, 1999)

Terminology Examining terminology might be a promising route (e.g.,

Norman, 1986; Caroll & Mack, 1984)

Literature ReviewMore Details

Learning Context Learning context is often formal, yet many people

learn in a more informal, exploratory environment (Rieman, 1996)

Ability More transfer is seen with users who have better

mental models (Shih & Alessi, 1993)

Purpose

1.Describe the kind of transfer knowledge that is related to learning performance

2.Assess the relative effectiveness of specific types of transfer knowledge

3.Examine the effect of ability on transfer

4.Discuss theoretical and practical implications

MethodSample n = 36

Software Learned Spreadsheet Tasks

Measures Think-Aloud Protocols (TAP)

Computer Ability Scale (CAS)

Predictor Variables Transfer Categories

Sample Transfer Behaviours (TAP)

Response Variables Influence on Learning

Total Effect Score

Learning Performance

Results

Frequency

Total Effect Scores

Learning Performance

Task Type

Conclusions

General knowledge had little effect on learning performance(consistent with literature)

Specific knowledge had prominent effect (consistent with literature)

Considerable negative and positive transfer in certain areas – actions, concepts, terminology (previously not examined)

Subjects are able to learn in an exploratory format(consistent with literature)

Transfer independent of ability, except for software concepts and terminology(partially consistent with literature)

Suggestions for Educators

General problem solving skills not particularly useful Basic understanding of the keyboard and common

movement keystrokes Terminology – discuss synonyms Focusing on software concepts and actions/procedures Identifying common tasks among software packages

could be an important first step to facilitate transfer Using previous software as a reference point

Future Research

Explore a wider variety of software Examine common task structures among

software packages Look closely at misconceptions

(terminology, concepts, actions) Explore conceptual understanding of

advanced users

For a copy of this presentation and the revised paper go to

faculty.uoit.ca/kay/site2004/site2004kay.htm

Sample

18 males, 18 females Age ranging from 23 to 49 (M=33) 12 beginners, 12 intermediate, and 12

advanced Highly educated with university degrees

or higher

Think-Aloud Protocols

Subjects verbalize what comes to their mind while they are doing a task

Presents a view of diversity and process that is unattainable using survey or interview methods

Essentially, the think-aloud procedure offers a “partial” window into the internal talk of a subject while he/she is learning.

Computer Ability Scale

Beginner, Intermediate, and Advanced Levels Criteria used to determine levels:

years of experience previous collaboration previous learning software experience number of application software packages used number of operating systems known application software and programming languages known

A multivariate analysis showed that beginners, intermediates, and advanced users had significantly different scores on all seven measures (p<.005)

Transfer Categories

General KnowledgeMetacognitive

Personal experience

Miscellaneous

SoftwareConcepts

Actions

Keystrokes

Other packages

HardwareKeyboard

Miscellaneous

Terminology

Transfer BehavioursTransfer Category Sample

General Knowledge "Uhm, I'm just thinking is it really necessary for me to do this, at this point. But, ah, I think ah I guess what maybe I'm feeling is that I'm a real novice. I've never used this before and I may as well start at the beginning"

Hardware Knowledge Subject reasons that maybe a spreadsheet can have a finite number of rows and columns because there are memory limitations

Software "There's probably a way to highlight the entire thing and do it all but … Well, I was thinking there must be a way to kind of do the entire column“

"I'm assuming that pressing [p] escape would get you out of the program entirely"

Terminology "Move to the command retrieve. That's a file thing so … Uhm I'll … and there's is retrieve. I'm at it"

Interrater reliability ranged from .84 to .97 (6 external raters)

Total Effect Score

For each transfer sub category Frequency Mean Influence Percent of subjects

Total effect Frequency x Mean Influence x Percent

Keyboard Total Effect 155 observations x 0.66 x 97% = 99.2

Frequency of Transfer

Percent of Total Transfer Behaviour

0%

10%

20%

30%

40%

50%

60%

Software Terminology Hardware General

Total Effect ScoresCategory Sub Category N % Mean Infl.

(S.D.) TotalEffect

Percent

Software Keystrokes255

100% 0.66 (1.0) 168.3

38 %

Hardware Keyboard155

97% 0.66 (1.0) 99.2

22 %

Software Actions229

92% 0.36 (1.3) 75.8

17 %

Terminology Terminology332

97% 0.15 (1.2) 48.3

11 %

Software Concepts138

94% 0.17 (1.3) 22.1

5 %

General Miscellaneous49

75% 0.53 (1.0) 19.5

5 %

Software Other Packages102

92% 0.03 (1.3) 2.8

1 %

Hardware Miscellaneous 8 19% 0.00 (0.8) 0.0 0 %

General Metacognitive45

53% -0.02 (1.0) -0.5

0 %

General Personal Exp.11

19% -0.55 (1.4) -1.1

0 %

Total1324

435.1

Total Effect – Pos. vs. Neg.Category Sub Category Total Effect Negative

EffectPositiveEffect

Combined

Terminology (4) Terminology 48.3 -69.5 64.5 134.0

Software (1) Keystrokes 168.3 -13.0 80.8 93.8

Software (3) Actions 75.8 -39.8 47.8 87.6

Hardware (2) Keyboard 99.2 -10.4 48.1 58.5

Software (5) Concepts 22.1 -29.9 20.1 50.0

Software (7) Other Pack 2.8 -23.2 14.6 37.8

General (6) Miscellaneous 19.5 -1.9 7.3 9.2

General (10) Personal Exp. - 1.1 -1.1 0.1 1.2

General (9) Metacognitive -0.5 -1.1 2.0 3.1

Hardware (8) Miscellaneous 0.0 -0.2 0.1 0.3

Total 435.5 - 189.0 285.3 474.3

Learning PerformanceCategory Sub Category N Correlation Controlling

for Ability

Terminology Terminology 34 .59 (p < .001) p < .10

Software Actions 33 .83 (p < .001) p < .0001

Concepts 34 .66 (p < .001) n.s.

Other packages 34 .58 (p < .001) p < .001

Keystrokes 36 .37 (p < .05) p < .10

Hardware Miscellaneous 7 .44 (n.s.) sample too small

Keyboard 35 .25 (n.s) n.s.

General Personal exp. 7 .77 (p < .05) sample too small

Metacognitive 18 .05 (n.s.) sample too small

Miscellaneous 26 -.02 (n.s.) n.s.

Task Type

52%

39%

24%

33%28%

58%

15%

33%

18%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Move Menu Data

Task Type

Per

cen

t U

sed

Keyboard

Action & Concepts

Terminology