A TEST OF STRATEGIES FOR ENHANCED LEARING

OF DESCRIPTIVE CHEMISTRY

by

Suhasini Kotcherlakota

A DISSERTATION

Presented to the Faculty of

The Graduate College at the University of Nebraska

In Partial Fulfillment of Requirements

For the Degree of Doctor of Philosophy

Major: Educational Studies

Under the Supervision of Professor David W. Brooks

Lincoln, Nebraska

January, 2007

A TEST OF STRATEGIES FOR ENHANCED LEARING

OF DESCRIPTIVE CHEMISTRY

Suhasini Kotcherlakota, M.S.

University of Nebraska, 2007

Adviser: David W. Brooks

The Advanced Placement (AP) Descriptive Chemistry Website allows students to

repeatedly practice chemistry problems. The current study involves the redesign of the

AP Descriptive Chemistry Website using worked examples to enhance learner

performance. The population sample for the study includes students interested in learning

descriptive chemistry materials. Students’ usage patterns with the Website were analyzed

to assess learner performance using the Independent Samples t-test. No important

differences were found in learning between the worked example and more conventional

strategies.

i

Dedication

To my parents and grand parents

ii

Acknowledgements

I wish to express my gratitude and sincere thanks to each one of you for your support,

encouragement and guidance throughout my doctoral studies.

Dr. David Brooks (mentor)

Faculty, Staff & Colleagues, UNL

Family members and friends

iii

TABLE OF CONTENTS

I. Introduction .................................................................................................................. 1

Statement of the Problem........................................................................................ 2 Research study in 1999 ............................................................................... 2 Research study in 2000 ............................................................................... 4 Research study in 2005 ............................................................................... 5

Purpose of the Study ............................................................................................... 6 Research Questions................................................................................................. 6 Significance of the Study ........................................................................................ 7 Limitations and Advantages ................................................................................... 7 Definition of Terms................................................................................................. 9

II. Review of Literature.................................................................................................... 10 Introduction........................................................................................................ ...10 Cognitive Load Theory ......................................................................................... 10 View of Memory................................................................................................... 11 Human Cognitive Architecture ............................................................................. 12 Categories of Cognitive Load ............................................................................... 12

Intrinsic Cognitive Load (ICL) ................................................................. 12 Extrinsic Cognitive Load (ECL)............................................................... 13 Germane Cognitive Load (GCL) .............................................................. 13

Reducing Cognitive Load and Enhancing Learning............................................. 14 Feedback ............................................................................................................... 14 Worked examples.................................................................................................. 16

III. Methodology............................................................................................................... 19 Procedural Steps.................................................................................................... 20 Population and Sample ......................................................................................... 20 Chemistry Study Items.......................................................................................... 25 Eight-item Sample Tests ....................................................................................... 27 Feedback ............................................................................................................... 28 Review .................................................................................................................. 29 Surprise Quizzes ................................................................................................... 30 Design of Website................................................................................................. 32 Data File Structure ................................................................................................ 33 The Use Pattern..................................................................................................... 33 Use Analysis ......................................................................................................... 34

IV.Results.......................................................................................................................... 36 Usage Pattern ........................................................................................................ 37 Date of First Access .............................................................................................. 37 Serious Users ........................................................................................................ 38 Total Time of Practice .......................................................................................... 39 The Number of Interactions .................................................................................. 41

iv

Chemistry Quiz Items ........................................................................................... 43 Tutor Access ......................................................................................................... 45 Worked Examples................................................................................................. 47 Eight Item Practice Quizzes.................................................................................. 48 Review .................................................................................................................. 49 Surprise Tests........................................................................................................ 50

V. Discussion ................................................................................................................... 53 Usage Pattern ........................................................................................................ 53 Chemistry Quiz Items ........................................................................................... 55 Tutor Access ......................................................................................................... 55 Eight Item Practice Quizzes.................................................................................. 56 Worked Examples................................................................................................. 56 Surprise Tests........................................................................................................ 56

VI.Summary and Recommendations ................................................................................ 59 References......................................................................................................................... 61

Appendix A....................................................................................................................... 64

Informed Consent Form........................................................................................ 65

Appendix B ....................................................................................................................... 68 Surprise test 1........................................................................................................ 69 Surprise test 2........................................................................................................ 70 Surprise test 3........................................................................................................ 71 Surprise test 4........................................................................................................ 72 Surprise test 5........................................................................................................ 73

v

LIST OF TABLES

TABLE PAGE

31 Chemistry study items and number of quiz categories 25

32 Access to correct answer 35

41 Users from the experimental and control group 36

42 Users Website transaction 37

43 Group Statistics for user transaction time elapsed rates 41

44 Independent Sample t-test for user transaction time

elapsed rates

41

45 Independent Sample t-test for number of user interactions

with the Website

43

46 Independent Sample t-test for number of user interactions

with the Website

43

47 Group Statistics for correctly answered practice items 44

48 Independent Sample t-test for correctly answered

practice items

44

49 Group Statistics for Incorrectly answered practice items 45

410 Independent Sample t-test for incorrectly answered

practice items

45

411 Group Statistics for tutor access 46

412 Independent Sample t-test for tutor access 46

413 Descriptive Statistics for worked example-1 and worked

example-2

48

vi

414 Group Statistics for eight item practice quizzes 48

415 Independent Sample t-test for eight item practice quizzes 49

416 Group Statistics for surprise quizzes 51

417 Independent Sample t-test for surprise quizzes 52

vii

LIST OF FIGURES

FIGURE PAGE

31 Descriptive Chemistry Website log-in page 21

32 Informed consent 23

33 Descriptive chemistry study site showing three

entry areas

24

34 A chemistry quiz question item 26

35 Worked example 28

36 A Chemistry quiz question item feedback answer 29

37 A chemistry quiz question item feedback answer

with reported errors and accepted answers

information

29

38 Student review of transaction record 30

39 Sample Surprise Quiz 31

310 Student transaction record XML file 32

41 Month-wise user first logins 38

42 Month-wise serious user registrations 39

43 Box plot of user time elapsed rates 40

44 Users vs Number of interactions with the

chemistry Website

42

45 Number of worked examples accessed by users 47

46 Number of times users reviewed their records 50

51 Surprise test items 58

1

I. INTRODUCTION

The College Board, a not-for-profit organization, has offered the Advanced

Placement Program® (AP) to millions of students in United States and other countries

since 1955 allowing them to take college-level courses and exams for earning college

credit or placement while still in high school (College Board, 2006). According to the

College Board (2006), 68 percent of United States public schools now participate in AP.

Since 2000, students from all 50 states and the District of Columbia have succeeded on

the AP exam. In 2005, The College Board reported a national total of 57,102 exam takers

who were qualified for receiving chemistry college credit or advanced placement.

The AP chemistry course is broken down into five major topic areas indicated by

percentage of approximate proportion of multiple choice questions that pertain to each

topic: structure of matter (20%), states of matter (20%), reactions (35-40%), descriptive

chemistry (10-15%), and laboratory (5-10%) (College Board, 2006).

Crippen (2000) states: “Anecdotal evidence suggests that the descriptive

chemistry section of the AP chemistry exam is traditionally difficult for high school

students. The exam's difficulty can be attributed to the nature of the material and the

current structure of the AP curriculum. Descriptive chemistry is difficult to teach because

it requires either a large amount of memorization or experience; it tends to be disjointed

within the traditional curriculum.” The College Board outlines descriptive chemistry as

follows (College Board, 2006):

“Knowledge of specific facts of chemistry is essential for an

understanding of principles and concepts. These descriptive facts, including the

2

chemistry involved in environmental and societal issues, should not be isolated

from the principles being studied but should be taught throughout the course to

illustrate and illuminate the principles. The following areas should be covered:

1. Chemical reactivity and products of chemical reactions

2. Relationships in the periodic table: horizontal, vertical, and diagonal

with examples from alkali metals, alkaline earth metals, halogens, and

the first series of transition elements

3. Introduction to organic chemistry: hydrocarbons and functional groups

(structure, nomenclature, chemical properties). Physical and chemical

properties of simple organic compounds should also be included as

exemplary material for the study of other areas such as bonding,

equilibria involving weak acids, kinetics, colligative properties, and

stoichiometric determinations of empirical and molecular formulas.”

Statement of the Problem

This study was based on the review of three research studies carried out in 1999

and 2000, and 2005 on the Descriptive Chemistry Website.

Research study in 1999

The chemistry Website was developed in 1997 and made accessible since then as

a learning aid to serve users all over the world. The Website was dedicated solely to the

descriptive portion of the advanced placement chemistry exam. About 200 descriptive

chemistry questions in the form of quiz items were devised. These were intended to help

3

high school students prepare for the AP exam, and for chemistry teachers to use as a

learning or teaching resource. The questions were stored in a database. They were served

to the user upon request. Students were allowed to practice chemistry quizzes repeatedly.

Feedback was provided for student’s errors in responding to the quiz questions. The

following is an example of a descriptive chemistry quiz question:

A solution of tin (II) chloride is added to a solution of iron (III) chloride.

A correct answer response to the above question would be:

Sn2+ + Fe3+ ---> Sn4+ + Fe2+ (Not balanced)

The 1999 study is reported by Crippen (2000) who notes:

“An Analysis of the Web server log for 1999 suggests the following conclusions:

• The maximum number of hits in a single day (n = 1,336) occurred on Sunday,

May 16, 1999, the day before the 1999 AP exam.

• While the site shows consistent use, its use tended to be cyclic around the AP

exams. For the 1999 term, site use reached a peak within the week before the

AP exam. Use culminated the day before the exam, and dropped off

significantly after the exam.

• 77.7 percent of the users requested to have hints sent with their quizzes. An

almost equal number of hits were for exams as for grading responses, including

answer keys; 94.2% of the exams were graded.

• The site had consistent use between the hours of 10:00 a.m. and 11:00 p.m.;

Sunday and Wednesday were peak days, though use is consistent throughout

the week. These two statistics suggest strongly that the site was used in

classrooms at schools.

4

• Self-reported AP chemistry students make up the largest proportion of the

users. AP students are using the site between 10:00 a.m. and 11:00 p.m. with

heaviest use closer to 10:00 a.m. High school teachers are using the site at the

same time as the AP students, yet their use is heavier towards the later hours of

the school day.”

Research study in 2000

One of the major conclusions from the 1999 data suggests that AP chemistry

students and/or teachers used the chemistry Website extensively as a learning tool. In the

2000 study, Crippen questions that the 1999 study did not provide documentation about

how the Website was used as a learning tool and what learning occured. Another

conclusion from the 1999 study suggests that the Website provided tutoring components

and feedback to the users. Crippen (2000) states that not only the effectiveness of the

tutorial components but also the implications for the use of tutoring components by users

for chemistry learning were not documented in the 1999 study.

Hence, in the 2000 study, user patterns including user access to tutorials and

feedback were tracked and stored in the database. This was carried out in order to

understand the users’ perspective in learning descriptive chemistry. The Website operated

as follows. Once a user logged in and requested a quiz, a randomly generated set of eight-

items was drawn from 14 arbitrary categories defined by the researcher. Each quiz item

came from a different category, and quiz items were not duplicated. The user might or

might not opt for tutoring when requesting quizzes. The tutoring included detailed text

5

instruction, sometimes with images, and the category from which the quiz item was

posed to the user.

Further, user perceptions were analyzed by conducting a post-AP-examination

survey. Comparisons were made between individual users’ perception about the

effectiveness of the site and site usage. The study demonstrated that effective teaching

and learning could be conducted over the Web through repetitive practice and feedback.

However, in this study survey, respondents indicated problems with formula typing and

bias in the scoring due to formula entry problems. Further, the quiz design lacked

flexibility in providing tutoring and item-specific feedback.

Crippen (2000) recommended enhancements in three areas for the AP

Descriptive Chemistry Website for maximizing student learning. First was the redesign of

the user interface to provide appropriate options for users to access chemistry Website

learning materials. The options include menus that help in constructing formulas or

formula typing for quizzes and eliminating bias in graded quizzes. Second, enhanced

feedback and tutorial components were developed for each quiz item. This included

providing appropriate tutoring material related to each quiz item and feedback given to

students based on their responses to the quizzes. Third, users were not permitted to

resubmit graded quiz items.

Research study in 2005

A more recent study at the modified Website, Crippen and Brooks (2005)

analyzed over 250,000 incorrect responses made by students. Student errors were

recorded and categorized. It was found that the highest percentages of errors were

6

misconceptions (28%) and typographical errors (28%). Next highest percentages of errors

included inappropriate "chemical" dissociations (27%) and/or writing incorrect formulas.

The rest of the errors included not recognizing weak electrolytes and including spectator

ions.

Purpose of the Study

The current study is based on the conclusions and recommendations from the past

studies. The rationale for conducting this study is to minimize student errors and

misconceptions in order to improve learning and overall performance of students

preparing for the AP exam. The redesigned Website focuses on providing tutoring

components for quiz categories and specific performance-related feedback for each quiz

item. The Descriptive Chemistry Website was redesigned by adopting effective

instructional design methods based on the cognitive design principles for fostering

learning. While providing worked examples and feedback was the main target of this

study, tutoring and a redesign also were included.

Research Questions

1. Does a student’s performance in descriptive chemistry improve with descriptive

practicing chemistry problems?

2. Does the use of worked examples decrease the rate of errors when solving

descriptive chemistry problems?

7

Significance of the Study

This study is important for two primary reasons. One is the notion of using a

Web-based tool and the application of technology to enhance teaching and learning. The

other is the development of teaching and learning strategies that help in improving the

performance of students solving chemistry problems.

In this study, three major arguments are hypothesized.

First, it is hypothesized that the repetitive practicing with worked examples will

enhance learning in users relative to practicing quiz items alone.

Second, it is hypothesized that tutoring components in the study will help students

in learning the chemistry material.

Third, it is hypothesized that specific feedback provided in response to the

submitted answers will improve learners’ performance.

The definition of learning used in the study by Crippen (2000) will be used here:

“Learning is defined as an increase in the quiz score, or item score as a function of time

or the number of graded quizzes returned.” Data analysis is performed from the

automatically recorded transaction records of the users.

Limitations and Advantages

The current design of the study has limitations and advantages. The redesigned

Descriptive Chemistry Website attempts to retain the advantages but to minimize the

limitations raised in the past studies.

The Website allows anonymous user log-ins with valid or invalid email address

and password, and the researcher is unable to verify such user information. As mentioned

8

by Crippen (2000), “the advantage of covertly tracking a self-motivated user's actions in

an environment designed to produce learning makes this study appealing.”

“Hardware and software have the potential to limit any study where they are

significant components. Computers crash, hard drives fail, and software programs do not

function as advertised. All of these hardware/software issues have the potential to limit

the study” (Crippen, 2000). The current Website is served to the World Wide Web via a

computer server for increasing the speed of information transfer through the Internet.

This was made possible by utilizing greater bandwidth and transmission speed modem

lines. Unlike in the previous studies, where the Websites were developed on

HyperCard™-based programming techniques, the current study was redesigned

completely by utilizing the scripting programming methods of Runtime Revolution™.

Similar to the 2000 study, this design strategy provided a powerful environment for

automatically tracking user information, generating and grading quizzes, and capturing

users’ actions while interacting with the site. It allowed the researcher to remain removed

from the users and the data until the completion of the study.

Other similar aspects retained from the previous studies were:

• the unique characteristics of the users that include accessing the Internet and

having basic computing skills for learning online material.

• the researcher remains un-associated with the Website.

The current study did not collect user’s perspectives of the Website. That is, no follow-up

surveys were undertaken.

9

Definition of Terms

The following defined terms are required the purpose of this study:

Feedback: “Any message generated in response to a learner’s action. The

outcomes of feedback include helping learners to identify errors and become aware of

misconceptions” (Mason & Bruning, 2000).

Redesigned user interface: Enhancement in the appearance of the Website

features to make a more user-friendly Website as well as to provide more user options.

Runtime Revolution™: A Scripting language and developmental tool used to

develop the current AP Descriptive Chemistry Website.

Server Request: Commonly referred to as hit on the Web server. A server request

involves a user asking the Web server to do something. This includes sending/grading a

quiz or tutoring. Al user-initiated interactions with the Web server are defined as server

requests.

Tutoring: Tutoring is a detailed instruction of the category from which the

chemistry question was posed to the user.

Worked Example: “A worked example is a step-by-step demonstration of how to

solve a problem or perform a task” (Clark & Mayer, 2003).

10

II. REVIEW OF LITERATURE

Introduction

Research studies in 1999, 2000 and 2005 provided extensive literature reviews

supporting the effectiveness of Web instruction. The studies conclude that the Web is a

powerful teaching and learning tool for delivering instruction through repetitive testing

with feedback. This review does not re-emphasize the supporting views of effective Web

instruction and repetitive testing with feedback mentioned in the previous studies, but

instead provides grounded literature that focuses on adopting a design theory approach

that helps in reducing cognitive load and enhancing students’ learning performance.

The following literature review provides an overview of the cognitive load theory,

view of memory, cognitive architecture, and instructional design techniques to reduce

cognitive load, feedback and worked-examples.

Cognitive Load Theory

There has been an increased focus on the role of education and training, and

effectiveness of various instructional design strategies among many researchers. Some of

the most important breakthroughs in this regard have come from cognitive science, which

deals with the mental processes of learning, memory and problem solving (Cooper,

1998). Cognitive load theory (CLT) is a major theory that has undergone substantial

development over the past three decades and has gained importance in both traditional

and technology-based instruction. The human brain is considered to be the center for

human thought process and memory. Cognitive psychologists have aimed at a deeper

11

understanding exploring the most potent concepts of how humans perceive, store, and

retrieve information.

View of Memory

Researchers divided memory processes into stages of acquisition, storage, and

retrieval of information. Sweller’s (1999) ‘modal model’ of memory consists of three

types of memory: sensory memory, working memory, and long-term memory. Sensory

memory refers to the perception of things by the incoming stimuli from our senses.

Information is initially processed in sensory memory and then passed to working

memory. Working memory refers to the conscious cognitive processing of information

that occurs and has a very limited capacity in terms of the amount or complexity of

information that can be retained or processed at once. According to Miller (1956),

humans can process only seven plus or minus two elements at a time. If the limit

exceeded, learners experience what is called "cognitive overload" and no learning takes

place. However, in the recent literature, Sweller points out that working memory can

handle only a very limited number (possibly no more than two or three) novel interacting

elements (Paas et al., 2003). They further note that working memory can process simple

information very easily when carrying out cognitive activities. Long term memory stores

general world knowledge. The information that is present can be very sophisticated and

enable learners to perceive, think, and problem-solve. It is more than a mere

memorization of learned facts. The mental structures formed in memory are called

schema or schemata. They compose the knowledge base which plays a crucial role in the

human thinking process.

12

Human Cognitive Architecture

According to Sweller (1999), cognitive load theory assumes that some learning

environments impose greater demand than others and, as a consequence, impose a higher

information processing load on limited cognitive resources in working memory.

Cognitive load depends upon the degree to which one efficiently rehearses tasks and

skills over various domains and gains knowledge over a period of time. Human cognitive

architecture interacts with instructional material in various ways. Information present in

human memory varies on a continuum of low-high interactivity. "Each element of low

interactivity material can be understood and learned individually without consideration of

any other elements. The elements of high interactivity material can be learned

individually, but they cannot be understood until all of the elements and their interactions

are processed simultaneously. As a consequence, high interactivity materials are difficult

to understand" (Paas et al., 2003).

Categories of Cognitive Load

Based on different sources of cognitive load, Sweller (1999) distinguished three

types of load: Intrinsic, extraneous, and germane load.

Intrinsic cognitive load (ICL)

Element interactivity is the driver of this category because demands on working

memory capacity imposed by element interactivity are intrinsic to the material being

learned. Intrinsic cognitive load cannot be altered by instructional manipulations. A

simpler learning task that omits some interacting elements can be chosen to reduce ICL.

13

Extrinsic Cognitive Load (ECL)

The manner in which information is presented to the learners and the learning

activities required of learners can also impose cognitive load. When that load is

unnecessary (ECL), it may interfere with schema acquisition and automation.

Instructional procedures developed without any consideration or knowledge of the

structure of information or cognitive architecture impose heavy ECL because working

memory resources must be used for activities that are irrelevant to schema acquisition

and automation. Instructional designs intended to reduce cognitive load are most effective

when the ICL is high. When ICL is low, instructional designs intended to reduce

cognitive load have little or no effect.

Germane or Effective Cognitive Load (GCL)

The manner in which information is presented to the learners and the learning

activities required of learners can also impose cognitive load. While ECL interferes with

learning, GCL may enhance learning. Instead of working memory resources being used

to engage in searching, for example when dealing with ECL, GCL results in those being

devoted to schema acquisition and automation. Increases in effort or motivation can

increase the cognitive resources devoted to a task. In a sense, GCL amounts to what

generally is called self-regulation in the educational psychology literature.

In summary, Swellers’ view of learning is as follows:

Learning can occur when [(Intrinsic Cognitive load + Extraneous Cognitive load +

Germane Cognitive load) = Total Cognitive Load] < Working Memory capacity.

14

With respect to multimedia learning, Mayer and Moreno (2003) explain that,

when processing demands evoked by the learning task exceed the processing capacity of

the cognitive system or working memory capacity per-se, the result is cognitive overload.

Reducing Cognitive Load and Enhancing Learning

Cooper (1998) illustrates how extraneous load occurs in instruction: "When

intrinsic cognitive load is low (simple content) sufficient mental resources may remain to

enable a learner to learn from ‘any’ type of instructional material, even that which

imposes a high level of extraneous cognitive load. If the intrinsic cognitive load is high

(difficult content) and the extraneous cognitive load is also high, then total cognitive load

will exceed mental resources and learning may fail to occur. Modifying the instructional

materials to engineer a lower level of extraneous cognitive load will facilitate learning if

the resulting total cognitive load falls to a level that is within the bounds of mental

resources."

Feedback

Mason and Bruning (2000) explain that, “Feedback is any message generated in

response to a learner’s action. The outcomes of feedback include helping learners to

identify errors and become aware of misconceptions.” Research shows that, in everyday

classrooms and online learning environments, feedback is provided in one or several

methods of instruction to improve student learning and achievement. Brooks et al.,

(2005) cited methods that provide feedback including repetitive testing with immediate

feedback, encouraging in-class pair discussions where students evaluate and provide

15

feedback to each other, promoting cooperative learning in team-led groups, just-in-time

dynamic expert feedback provided in computer assisted instructional settings,

scaffolding, and self-explanation.

Further, Brooks et al. (2005) strongly suggest that “performance-related feedback

is the hallmark of efficient instruction” and teachers can utilize one or more forms of

feedback in their instruction to improve learning. Immediate and quick feedback without

delays is effective in increasing student comprehension on the learning materials.

(Bangert-Drowns, Kulik, Kulik, & Morgan, 1991).

Student learning increases when feedback is corrective in nature (Walberg, 1999).

Asking students to work repeatedly on a task until they are successful enhances student

achievement (Marzano, Pickering, & Pollock, 2001). Crippen et al. (2000) assert that

“Feedback from practice need not be confined to the correctness of an answer. Since

feedback points out missing or misunderstood knowledge, it provides an ideal teaching

moment. Thus, any feedback that further explains what students do not know adds to the

value of feedback.”

Instructional strategies and course design play an important role in achieving

higher learner outcomes for content, especially that taught over the Web (Clark & Mayer,

2003; Dick et al., 2001; Gagne et al., 1992; Sweller, 1999). A good instructional design

accommodates complex information, reduces cognitive load on working memory, and

enables learners to form effective schema and automate their learning processes. Recent

developments in instructional design have investigated ways to reduce cognitive load

(Paas et al., 2003; Mayer & Moreno, 2003). Cooper (1998) stresses that “The quality of

16

instructional design will be raised if greater consideration is given to the role and

limitations of working memory.”

Worked Examples

Worked examples are among the earliest and probably the best-known cognitive

load reducing techniques (Paas et al. 2003). Traditional ways of learning by solving lots

of problems load working memory. Worked examples help learners to reduce this load by

freeing working memory resources and building new knowledge (Clark & Mayer, 2003).

The effectiveness of worked examples depends on the previous domain

knowledge of students. When students cannot form schemas within a disciplinary area of

study, it is difficult for teachers to find suitable aspects of the area for them to explore

(Sweller & Tuovinen, 1999). Further research on worked examples is based on the

level(s) of cognitive skill acquisition. Cognitive skills are acquired by learners in three

phases. In the early phase, learners attempt to gain a basic understanding of the domain

by studying principles in instructional materials. Worked examples are most effective in

this early phase. During the intermediate phase, learners begin solving problems with the

knowledge acquired previously from the early phase. As the learners practice and solve

problems, they self-explain concepts (VanLehn, 1996). In the late phase, learners practice

striving for speed and accuracy. In this phase, skills are becoming automated. It is

uncertain from an instructional point of view of how best learners should structure the

transition from an example-based initial phase learning to problem solving in the

intermediate phase (Renkl & Atkinson, 2003). For learners who are already familiar with

the skills of problem-solving, interpreting a worked example may be redundant and

17

impose a greater cognitive load than simply providing a solution to the problem (Kalyuga

et al., 2003). One way to circumvent this problem is to design worked examples as

“completion” problems. van Merrienboer et al. (2003) suggest that the intrinsic cognitive

load can be decreased by practicing the simplest version of the whole task encountered by

experts in the real world and progress towards increasingly more complex versions.

The second approach is used to decrease extraneous load by scaffolding worked

examples followed by completion problems and then full problems. Sweller (1999)

suggests that worked examples are effective only under conditions where students do not

have to mentally integrate disparate sources of mutually referring information as well as

to eliminate redundant information during problem-solving. As learners acquire cognitive

skills and gain experience, using worked examples will cause redundancy. Devoting

working memory to redundant information leads to allocating limited cognitive capacity

to the redundancy and results in little or no learning. Redundant information may even

interfere with the schemas constructed by experienced learners and may also have

negative consequences. This phenomenon is known as the expertise reversal effect

(Kalyuga et al., 2003).

Renkl and Atkinson (2003) suggest that, in the earlier stages of learning when

intrinsic cognitive load is high because of problem-solving demands and fewer schemas,

learners should study instructions. During intermediate stages, when schema information

has freed some working memory capacity, they should study worked examples and

increase germane load by using self-explanations. In the final phase, there should be

sufficient working memory capacity to permit more problem-solving. This is called the

“fading procedure” of using worked examples for cognitive skill acquisition in solving

18

problems and learning. The fading technique facilitates transition from initial through

intermediate to final phases where complete worked examples are faded by successively

eliminating sections of the worked example until eventually only a full problem remains.

Research studies on the use of full or partial worked examples along with the

emphasis on other cognitive load reducing techniques and strategies are on going and

experimental. Based on what is known by research so far and to test the criterion lays the

foundation for this study.

In summary, this study conducts a test of instructional design strategies and

evaluates the results for enhancing learning.

19

III. METHODOLOGY

The AP Descriptive Chemistry Website was initially designed in 1997 and was

redesigned in 2000. This Website was visited by AP chemistry users extensively over the

past years for AP chemistry exam preparation (Crippen et al., 2000).

Crippen (2000) in his literature states that “The data set for 1997-99 is limited due

to the design of the site. Those data are limited because it does not allow for tracking any

one individual's use of the site. In addition to quizzing, the site offered a tutoring

component. The tutoring component is used but is not correlated to users and their

scores.” In order to overcome the problems in 1999 study, the 2000 study redesigned the

HyperCard Website and for tracking of an individual's use and surveying user

perceptions.

The current Website was once again redesigned for understanding the

effectiveness of the site as a learning tool. The Website was reconstructed by adopting

instructional design methods based on the cognitive design principles for fostering

learning. Worked examples with feedback and feedback only are the two main

approaches administered in the current chemistry Website. The rationale for identifying

these approaches in this study was to minimize student errors and misconceptions in

order to improve learning and overall performance.

20

Procedural Steps

1. Obtain Institutional Review Board (IRB) approval.

2. Redesign the HyperCard database using Runtime Revolution.

3. Announce the site to the AP chemistry community.

4. Gather user access data.

5. Perform data analysis.

Population and Sample

The research sample for this study represents individuals interested in descriptive

chemistry, especially as it applies to the AP chemistry examination. A total number of

1373 subjects were recruited for this study. Subjects presumably were AP chemistry

students and teachers.

The sample was recruited by word of mouth, conference presentations, email, and

newsgroup postings (misc. education. science, K-12.education.science). The URL for the

chemistry Website http://dwb2.unl.edu/apchem/main.html was made available for access

for the users worldwide. The homepage of the Descriptive Chemistry Website indicates

to the users that the site is a teaching site for chemistry and that research is conducted on

learning.

As shown in Figure 3.1, users logged-in to the chemistry Website using their

email address and password. If the user was logging in for the first time, they were

requested to provide an email and password that they wished to use for future access to

the Website. Consent was obtained from the site registrant after their first log-in attempt

to the Website.

21

Figure 3.1. Descriptive Chemistry Website log-in page.

Once informed consent was obtained (see Figure 3.2), each new user had access

to the chemistry quizzes and all the content in the Website. Each time a user logged in, all

of his/her transactions were recorded together with the e-mail login. The access time,

access address (computer IP number), specific identity of the chemistry items accessed in

the Website, and responses made all were recorded automatically.

22

23

Figure 3.2. Informed consent.

In this study, users were randomly assigned to one of two groups. One group had

access to worked examples and the other group did not. The group that received the

worked examples was the experimental group and the group that did not was the control

group. Presenting worked examples was the independent variable while the learning

performance of the students was the dependent variable in this study. The current

chemistry Website allowed students to practice chemistry problems repetitively. Based

upon the students’ performance (correct or incorrect responses), immediate feedback was

given. Immediate feedback was provided to students about their errors in answering the

24

quiz questions. Students in the control group were provided with correct answers and

asked to enter correct responses until they demonstrated proficiency. Students in the

experimental group were provided with completed worked examples in addition to the

explanation components for the first few problem sets in each chemistry category. After

three pairs of such worked examples (one pair for each of the three initial quiz questions),

the students were treated identically in both the experimental and control groups.

Tutoring (access to related, appropriate text material) was accessible to both groups.

As shown in Figure 3.3, the main Webpage contained three entry areas.

Figure 3.3. Descriptive chemistry study site showing three entry areas.

25

The first area is the Study section where users can access Chemistry Study Items.

The second area provides access to Eight-item sample tests (a model AP question), and

the third area provides access to Review their transactions with the Website.

Chemistry Study Items

The Chemistry study items were drawn from 14 arbitrary categories defined by

David Brooks and based upon his review of the AP questions over nearly four decades.

They are displayed in Table 3.1 along with the number of items available.

Chemistry Study Category Number of quiz items

Acid-Base General 30

Acid-Base anhydrides 20

Acid-Base Lewis 5

Acid –Base Hydrolysis 16

Redox Aqueous 35

Redox Metal Aqueous 5

Metals/Other 5

Redox/Other 10

Organic 12

Combustions 10

Precipitations 26

Complex Ion 21

Thermal decomposition 8

Mixed 5

Table 3.1. Chemistry study items and number of quiz categories.

26

A user may select any one of the categories or select the random option from the

study item area drop-down list. Once the user makes a selection of a category, he/she is

presented with a chemistry quiz item along with a tutoring component (see Figure 3.4).

Figure 3.4. A chemistry quiz question item

Each chemistry category contained a varied number of quiz items (see Table 3.1).

A chemistry quiz item consisted of the question number, question, three text field boxes

each for reactants and products, and a submission link for submitting answers. Students

entered their answers in the text field boxes and submitted them for evaluation.

Instructions for entering the answer formulas and a graphic of the periodic table were

provided to students in the same Webpage for reference. Students could choose to study

the material in the tutoring before answering the quiz question. Specific tutoring was

27

provided to users based on the quiz item category. Users could access tutoring any

number of times and return back for answering the same quiz question.

When users submitted the answers correctly, both the experimental and control

group received new quiz items based on their selection from the Chemistry Study Items

area.

However, when users from the worked examples group submitted an answer

incorrectly, they were provided with a completed worked example for the first two

attempts and a correct answer thereafter (see Figure 3.5). Control group users were

provided with a correct answer if they failed to answer a quiz item (See Table 3.1).

Eight-Item Sample Tests

Eight-item Sample Tests is the second entry area in the main chemistry Webpage.

This item format has been used up until 2006 for the AP descriptive chemistry question.

(The AP is changing this format beginning in 2007.) When the user selects this option,

s/he is provided with a set of eight quiz items. These items are randomly generated from

the database when the user requests them. Hence, each user may get a different set of

items each time they access. Users answer all eight-items and submit their answers for

evaluation. Unlike the Chemistry study items, users do not have access to the tutoring

component.

28

Figure 3.5. Worked example

Feedback

Feedback was provided for submitted answers for both the study item quiz

questions and the eight-item sample tests. Feedback included indicating the number of

wrong responses submitted and the number of expected correct responses (see Figures

3.6 & 3.7).

29

Figure 3.6. A Chemistry quiz question item feedback answer

Figure 3.7. A chemistry quiz question item feedback answer with reported errors and

accepted answers information.

Review

Review is another entry area in the main chemistry Webpage. Users can view all

transactions they made in the Website by selecting this option (see Figure 3.8). Review

contains the most recent transaction down to the first transaction with the Website.

30

Figure 3.8 . Student review of transaction record

Surprise Quizzes

Besides the eigh-item quizzes, both the experimental and control group users

received the same 3-item “surprise” quizzes.

Five such quizzes were developed, and these are presented in sequence after the

20th, 40th, 60th, 80th, and 100th interactions. Unlike other items such as the chemistry study

quizzes or eight-item tests whose selection is based upon either user choice or random

choice, all users saw the same “surprise quizzes” in the same order (see Figure 3.9).

31

Figure 3.9. Sample Surprise Quiz

32

Design of Website

The AP Descriptive Chemistry Website was re-designed using Runtime

Revolution™, JavaScript, cgi and XML technologies. Runtime Revolution™ and

Javascript were the main two scripting languages used for rewriting the code to redevelop

the old HyperCard-based Website. cgi is the engine used for interfacing connections with

the Website program and the Web server. XML was the file format used for storing the

user transactions with the Website. Each user has an XML-like record, which contains

transaction information (see Figure 3.10).

Figure 3.10 . Student transaction record XML file

33

MySQL was used during data analysis after the collection of the XML files. That

is, data from the XML files were aggregated into a MySQL database.

Data File Structure

The transaction data were stored in a folder named apchem in the Webserver

folder under the Documents folder. The apchem folder contained two folders named

Adata and Bdata. Adata folder contained transaction record files of all the users assigned

to the worked-example, and the Bdata folder contained transaction record files of all the

users assigned to the no-worked-example group. Each transaction record is an XML-like

file and contained information about the time of access, date of access, IP address, and

information about the user transaction within the three entry areas along with the quiz

items accessed and answers submitted (see Figure 3.10).

The Use Pattern

“A user’s use pattern is defined by a compilation of statistics from the database”

(Crippen, 2000). Similar to the 2000 study, descriptive statistics for each user were

compiled based upon the use patterns. The average usage patterns were then used to

understand the site’s usage and the effectiveness of the site as a learning tool.

Statistics for creating a use pattern for each user of the Descriptive Chemistry Website.

1. Usage pattern

a. Use time, dates and number of log-in’s

2. Usage pattern of chemistry quiz items

a. Average number of chemistry quiz items taken

34

b. Average number of chemistry quiz items graded

c. Average quiz item scores

d. Average tutoring request by subtopic

e. Average number of worked examples accessed

3. Usage pattern of eight-item sample tests

a. Average number of correct answers

b. Average number of eight-item sample tests taken

c. Average scores of eight-item sample tests.

4. Usage pattern of surprise quizzes

a. Average number of correct answers

b. Average number of surprise quiz items taken

c. Average scores of surprise quiz items

Use Analysis

The variables under analysis from the transaction records will be usage patterns of

practice items, usage patterns of the eight-item sample tests, and usage patterns of

surprise quizzes. Each usage pattern involves descriptive statistics of the subject such as

the times and dates the Website was accessed. In particular, statistics for practice items

including average number of practice items taken, average number of practice items

graded, average practice scores and average request of practice items by topic were

collected. For the experimental group, the average number of worked examples accessed

for each practice item was measured. Similarly, statistics for an eight-item sample test

included the number of correct answers, the average number of eight-item sample tests

35

taken, and the average scores of the eight-item sample tests. Further, statistics for surprise

quizzes were the average number of surprise quizzes taken and average scores on the

surprise quizzes.

The results obtained from the experimental variables (transaction record data)

were gathered in a database and analyzed using SPSS software. The final results are

discussed using ANOVA and regression analysis methods on the data.

Failed to answer quiz question correctly in the first attempt.

Failed to answer quiz question correctly in the second attempt and/or need more help or tutoring.

Failed to answer the quiz question correctly in the third attempt.

Worked Example Group

Provide first worked example

Provide second worked example

Provide correct answer

No Worked example group

Provide correct answer

Provide correct answer Provide correct answer

Table 3.2. Access to correct answer

36

IV. RESULTS

The AP Descriptive Chemistry Website was designed based on the

recommendations from the previous studies conducted in 1999, 2000 and 2005. The

current redesign is consistent with the literature on worked examples, feedback and

repetitive practice. The Website recorded everything users did while accessing the AP

descriptive chemistry Website during the 2006 academic term. Users were randomly

assigned to one of two groups. The worked-example group (WE) is the experimental

group and no-Worked-example group (NWE) is the control group. Analyses were

performed based on the user transactions and comparisons were made between the

experimental and the control groups.

A total of 1,373 unique users have registered and used the Website for practicing

chemistry problems. Each unique user has been identified with a user e-mail address that

they have chosen to provide during the initial registration with the Website. All the

Website interactions of the users were tracked and saved in the database for data analysis.

The selection of users for this study has been based on the completion of five “surprise

tests.”

Users who did not complete all 'five'

surprise tests

Users who completed all five surprise tests

Total number of users

Worked- example 637 50 687 No-Worked-example 620 66 686 Table 4.1. Users from the experimental and control group.

37

Table 4.1 illustrates that a total 50 out of 687 users from worked-example group

and a total of 66 out of 686 from the no-worked-example group completed all five

surprise tests. Approximately 93% of the students in the experimental group and 90% in

the control group did not complete all five surprise tests.

Usage Pattern

The usage pattern includes the dates users accessed the Website, the total amount

of time spent using the Website for practicing chemistry problems, and the number of

user logins to the descriptive chemistry Website.

Date of First Access

User transactions were collected from January until August, 2006. Students used

the Website increasingly up through the month of May. Very minimal usage occurred

thereafter during the months of June, July and August (Table 4.2 and Figure 4.1).

20-Jan-06 3-Aug-06 Jan Feb Mar Apr May Jun Jul Aug

Worked- example 94 124 119 166 148 17 15 4 687

No-Worked-example 95 124 117 166 147 17 16 4 686

Total 189 248 236 332 295 34 31 8 1373 Table 4.2. Users Website transaction.

38

0

20

40

60

80

100

120

140

160

180

jan Feb mar apr may jun jul aug

Month

Num

ber o

f use

rs re

gist

ered

with

AP

desc

riptiv

e ch

emis

try w

ebsi

te

Worked-ExampleNo-Worked-Example

Figure 4.1. Month-wise user first logins.

Serious users

The Website for this research study was designed to provide chemistry materials

to users interested in learning descriptive chemistry. The selection of (experimental [WE]

and control group [NWE]) users in the study was based on users who consistently used

the Website and reached the point of attempting all the five sets of surprise tests. This

group of 116 users is called the serious users. Although data analysis shows that a

number of users interested in learning chemistry materials had registered to the AP

descriptive chemistry Website, the final results indicate that only a small fraction of the

total number of users met the selection criterion of "serious user.” That is, of the 1373

users for whom some data were available, only 116 were serious enough to persist in

using the Website. Usage patterns and user associations such as the dates accessed the

39

total amount of time spent, and number of each user logins were considered to understand

the users’ improvement in practicing chemistry problems and thus the performance.

0

5

10

15

20

25

30

Jan Feb Mar Apr May May+

Month

Num

ber o

f ser

ious

use

rs re

gist

ered

with

AP

desc

riptiv

e ch

emis

try w

ebsi

te

Worked-example No-workedexample

Figure 4.2. Month-wise serious user registrations.

Figure 4.2 shows the number of serious users registered with the AP descriptive

chemistry website from the beginning of the study in the month of January until the day

of the actual AP exam in the month of May.

Total Time of Practice

The box plot (Figure 4.3) shows that the elapsed time variability between the

worked example and the no-worked-example group. The worked example group has

slightly greater variability than the no-worked example group. The figure also indicates

that there are some outliers and extreme variables. The median time elapsed rates for

worked example and no-worked example group are 5.72 and 4.52 hours respectively. The

40

maximum (177.37 and 193.85) and minimum elapsed rates (0.71 and 0.94) respectively

indicate the variability in the time spent for practicing chemistry problems in the Website.

Both the groups are skewed to the right (i.e., toward shorter time intervals)

2.001.00

(1.00) Worked Example and (2.00) No-Worked Example Group

200.00

150.00

100.00

50.00

0.00

Elap

sed

Tim

e ra

tes

Figure 4.3. Box plot of user time elapsed rates.

Overall, no statistical significant difference was found between the groups for

elapsed time (mean values MWE = 12.12; MNWE =13.05).

41

Time Elapsed - Group Statistics

50 12.1202 27.78812 3.9298366 13.0539 24.91376 3.06667

Group1.002.00

ValueN Mean

Std.Deviation

Std. ErrorMean

Table 4.3. Group Statistics for user transaction time elapsed rates

Time Elapsed - Independent Samples Test

.342 .560 -.190 114 .850 -.93374 4.90990 -10.66022 8.79274

-.187 99.135 .852 -.93374 4.98478 -10.82447 8.95699

Equal variancesassumedEqual variancesnot assumed

ValueF Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)Mean

DifferenceStd. ErrorDifference Lower Upper

95% ConfidenceInterval of the

Difference

t-test for Equality of Means

Table 4.4. Independent Sample t-test for user transaction time elapsed rates.

The group statistics indicate the sample sizes (N), means, and standard deviations

of both the control and experimental groups. Independent Samples t-test Table 4.4 shows

the Levene’s Test for the Equality of Variances in the two groups for elapsed time rates.

The significance column shows that assumption is not violated (p = 0.56) is not

significant. Because homogeneity can be assumed, the two-tailed significance, p = 0.85

indicates that the observed difference in the means between the WE and NWE is not

significant. The output also indicates that the observed difference in the means is not

significant, t(114) = -0.19, p = 0.85.

The Number of Interactions

The total number of transactions each user made with the descriptive chemistry

Website was measured. Users practiced on chemistry practice quizzes, eight-item

quizzes, surprise tests, and viewed their usage transaction record. Each such interaction

42

was counted towards the total number of transactions (Figure 4.4). The group statistics

indicate that the mean value of interactions for the WE group is 259.48 and NWE is

294.05.

Independent Samples t-test Table 4.5 shows the Levene’s Test for the Equality of

Variances in the two groups for number of interactions with the website. The significance

column shows that assumption is not violated because the significance (p = 0.07) is not

significant.

64615855524946434037343128252219161310741

User

1400.00

1300.00

1200.00

1100.00

1000.00

900.00

800.00

700.00

600.00

500.00

400.00

300.00

200.00

100.00

0.00

Num

ber o

f int

erac

tions

with

the

chem

istr

y w

ebsi

te

No Worked Example GroupWorked Example group

Figure 4.4. Users vs. Number of interactions with the chemistry Website.

43

User Interactions - Group Statistics

50 259.4800 156.88736 22.1872266 294.0455 218.22913 26.86215

Group1.002.00

InteractionsN Mean

Std.Deviation

Std. ErrorMean

Table 4.5. Independent Sample t-test for number of user interactions with the Website

Because homogeneity can be assumed, the two-tailed significance, p =0.34

indicates that the observed difference in the means between the WE and NWE is not

significant. The output also indicates that the observed difference in the means is not

significant, t(114) = -0.95, p = 0.35.

Independent Samples Test

3.386 .068 -.949 114 .345 -34.56545 36.41966 -106.713 37.58161

-.992 113.726 .323 -34.56545 34.84032 -103.586 34.45473

Equal variancesassumedEqual variancesnot assumed

InteractionsF Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)Mean

DifferenceStd. ErrorDifference Lower Upper

95% ConfidenceInterval of the

Difference

t-test for Equality of Means

Table 4.6. Independent Sample t-test for number of user interactions with the Website

Chemistry Quiz Items

Users practiced chemistry quiz items of their choice from a set of the descriptive

chemistry materials and then submitted the quiz item for grading. Each quiz item carried

a maximum of three possible reactant answers and three possible product answers. A

final “score” for each practice quiz item was calculated based on correctly answering the

chemistry quiz item, i.e. submitting all the correct reactant and product answers. Credit

was not given to the user if one or more incorrect answers were submitted. The answer

was then considered incorrect.

44

The chemistry practice quiz items do not include the eight-item quiz questions

practiced or the surprise quiz items. The group statistics of the chemistry practice quiz

items indicate that the mean value of correctly answered practice quiz items for the WE

group is 26.72 and NWE is 35.17 (Table 4.7).

Practice Items Correct Answered - Group Statistics

50 26.7200 23.97111 3.3900366 35.1667 41.15135 5.06538

Group1.002.00

PracticeItemsN Mean

Std.Deviation

Std. ErrorMean

Table 4.7. Group Statistics for correctly answered practice items

The Independent Samples t-test Table 4.8 shows the Levene’s Test for the

Equality of Variances in the two groups for correctly answered practice items. The

significance column shows that the assumption is violated (p= 0.00 being less than 0.05)

was significant. Because homogeneity cannot be assumed, the two-tailed significance,

p =0.17 indicates that the observed difference in the means between the WE and NWE is

not significant. The output also indicates that the observed difference in the means is not

significant, t(107.63) = -1.39, p = 0.17.

Practice Items Correct Answered - Independent Samples Test

11.241 .001 -1.294 114 .198 -8.44667 6.52859 -21.37976 4.48643

-1.386 107.626 .169 -8.44667 6.09511 -20.52871 3.63537

Equal variancesassumedEqual variancesnot assumed

PracticeItemsF Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)Mean

DifferenceStd. ErrorDifference Lower Upper

95% ConfidenceInterval of the

Difference

t-test for Equality of Means

Table 4.8. Independent Sample t-test for correctly answered practice items.

The group statistics indicate that the mean value of incorrectly answered practice

items for the WE group is 40.44 and NWE is 56.83 (Table 4.9).

45

Group Statistics

50 40.4400 28.88836 4.0854366 56.8333 48.80466 6.00744

Group1.002.00

PiIncorrectN Mean

Std.Deviation

Std. ErrorMean

Table 4.9. Group Statistics for Incorrectly answered practice items

Independent Samples t-test Table 4.10 shows the Levene’s Test for the Equality

of Variances in the two groups for incorrectly answered practice items. The significance

column shows that assumption was violated, (p = 0.03) was significant. Because the

assumption of homogeneity cannot be assumed, the two-tailed significance, p = 0.03

indicates that the observed difference in the means between the WE and NWE was

significant. The output also indicates that the observed difference in the means was

significant, t(108.30) = -2.26, p = 0.03.

Independent Samples Test

4.788 .031 -2.110 114 .037 -16.39333 7.76841 -31.78250 -1.00417

-2.256 108.298 .026 -16.39333 7.26499 -30.79335 -1.99331

Equal variancesassumedEqual variancesnot assumed

PiIncorrectF Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)Mean

DifferenceStd. ErrorDifference Lower Upper

95% ConfidenceInterval of the

Difference

t-test for Equality of Means

Table 4.10. Independent Sample t-test for incorrectly answered practice items.

Tutor Access

Tutoring was provided for each descriptive chemistry quiz item category.

Students in both the WE and NWE groups can access tutors when practicing the

chemistry quiz items.

The group statistics indicate that the mean value of tutors accessed for the WE

group is 4.26 and NWE is 3.73 (Table 4.11).

46

Tutor Access Group Statistics

38 4.2632 3.65901 .59357

52 3.7308 3.83030 .53117

groupWorked Example TutorNo Worked ExampleTutor

valueN Mean

Std.Deviation

Std. ErrorMean

Table 4.11. Group Statistics for tutor access

The Independent Samples t-test Table 4.12 shows the Levene’s Test for the

Equality of Variances in the two groups for tutor access. The significance column shows

that assumption was not violated, (p = 0.92) was not significant. Because homogeneity

can be assumed, the two-tailed significance, p =0.51 indicates that the observed

difference in the means between the WE and NWE was not significant. The output also

indicates that the observed difference in the means was not significant, t(88) = 0.66,

p = 0.51.

Tutor Access Independent Samples Test

.009 .924 .664 88 .509 .53239 .80228 -1.06198 2.12675

.668 81.888 .506 .53239 .79653 -1.05220 2.11698

Equal variancesassumedEqual variancesnot assumed

valueF Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)Mean

DifferenceStd. ErrorDifference Lower Upper

95% ConfidenceInterval of the

Difference

t-test for Equality of Means

Table 4.12. Independent Sample t-test for tutor access

47

Worked Examples

Two worked examples (worked example-1, WE1 and worked example-2, WE2)

were provided for first few sets of chemistry quiz items. The worked example group

alone had access to these examples when practicing chemistry quiz items. The mean

value for worked-example 1 is 16.28 and for worked-example 2 is 6.74 (Figure 4.5 and

Table 4.13).

494745434139373533312927252321191715131197531

Users

60.00

50.00

40.00

30.00

20.00

10.00

0.00

Num

ber o

f wor

ked

exam

ples

acc

esse

d

Example2Example1

Figure 4.5. Number of worked examples accessed by users.

48

Descriptive Statistics

50 2.00 56.00 16.2800 11.4785350 .00 28.00 6.7400 5.9618250

Example1Example2Valid N (listwise)

N Minimum Maximum MeanStd.

Deviation

Table 4.13. Descriptive Statistics for worked example-1 and worked example-2

Eight-item Practice Quizzes

The eight practice questions are a set of eight practice quiz items. The final score

for the practice quiz item is calculated based on the total number of reactant and product

answers submitted correctly out of the maximum possible correct answers for each quiz

item. Total score of the each set of practice item answered is considered for the analysis.

The group statistics indicate that the mean value of eight-item practice quiz items for the

WE group is 3.37 and NWE is 3.78 (Table 4.14).

8 Item Practice Quiz - Group Statistics

25 3.3692 1.54697 .3093932 3.7756 1.40723 .24877

value1.002.00

groupsN Mean

Std.Deviation

Std. ErrorMean

Table 4.14. Group Statistics for eight-item practice quizzes

The Independent Samples t-test Table 4.15 shows the Levene’s Test for the

Equality of Variances in the two groups for eight-item practice quizzes. The significance

column shows that assumption was not violated, (p= 0.70) was not significant. Because

homogeneity can be assumed, the two-tailed significance, p = 0.31 indicates that the

observed difference in the means between the WE and NWE was not significant. The

49

output also indicates that the observed difference in the means was not significant

t(55) = -1.04, p = 0.30.

8 Item Practice Quiz - Independent Samples Test

.152 .698 -1.036 55 .305 -.40643 .39234 -1.19269 .37984

-1.024 49.156 .311 -.40643 .39700 -1.20416 .39131

Equal variancesassumedEqual variancesnot assumed

groupsF Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)Mean

DifferenceStd. ErrorDifference Lower Upper

95% ConfidenceInterval of the

Difference

t-test for Equality of Means

Table 4.15. Independent Sample t-test for eight-item practice quizzes

Review

A total of 24 out of 50 users from the worked-example group and 33 out of 66

users from the no-worked-example group reviewed their transactions with the descriptive

chemistry website. Figure 4.6 shows the total number of times each user reviewed their

record.

The mean values for number of times record reviewed for worked-example group

and no-worked-example group are 1.74 and 2.36 respectively. These mean values are

extremely low compared to the mean values of total interactions (worked-example group,

259 and no-worked example group, 294) with the website indicating that the review area

in the website was not utilized often by the users.

50

0

5

10

15

20

25

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33

Users

Nu

mb

er

of

tim

es

reco

rd r

evie

wed

Worked-Example No-Worked-Example

Figure 4.6. Number of times users reviewed their records.

Surprise Tests

A total of 15 surprise test items were analyzed. The users had no control over the

appearance of surprise tests, and the surprise test items appeared only once (in the context

of a surprise test). Thus the surprise test items were the measurable features that all users

of the Website saw in common and at the same relative time during the site usage. The

group statistics of the surprise tests were calculated as shown in the Table 4.16.

The Independent Samples t-test Table 4.17 shows the Levene’s Test for the

Equality of Variances in the two groups for surprise quiz items. The significance column

is considered to determine whether the assumption is violated or not violated. Based on

the homogeneity of the assumption and the two-tailed significance value, the significance

51

of the difference between the groups is determined. All the 15 Independent sample t-tests

indicated that there was no significant difference between the WE and NWE groups.

Table 4.16. Group Statistics for surprise tests

Surprise Tests Group Statistics

50 .4550 .34140 .04828 66 .4811 .32907 .04051 50 .4450 .42043 .05946 66 .4545 .36927 .04545 50 .3730 .45504 .06435 66 .3936 .44893 .05526 50 .2500 .23690 .03350 66 .2500 .23205 .02856 50 .4650 .41958 .05934 66 .4773 .41074 .05056 50 .2900 .33640 .04757 66 .3295 .34851 .04290 50 .4600 .28067 .03969 66 .4606 .34368 .04230 50 .4700 .35942 .05083 66 .4356 .40266 .04956 50 .6050 .41061 .05807 66 .4962 .44179 .05438 50 .3500 .29881 .04226 66 .2992 .30134 .03709 50 .6500 .37457 .05297 66 .5606 .41385 .05094 50 .2730 .35470 .05016 66 .2521 .33139 .04079 50 .0200 .06061 .00857 66 .0273 .06916 .00851 50 .2920 .21077 .02981 66 .2424 .23407 .02881 50 .4398 .45912 .06493 66 .4041 .47697 .05871

group Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example

T1

T2

T3

T4

T5

T6

T7

T8

T9

T10

T11

T12

T13

T14

T15

N MeanStd.

DeviationStd. Error

Mean

52

Surprise tests Independent Samples Test

.134 .715 -.416 114 .678 -.02606 .06270 -.15027 .09815

-.414 103.572 .680 -.02606 .06302 -.15104 .09892

3.838 .053 -.130 114 .897 -.00955 .07351 -.15517 .13608

-.128 97.823 .899 -.00955 .07484 -.15807 .13898

.004 .948 -.244 114 .808 -.02064 .08466 -.18835 .14708

-.243 104.906 .808 -.02064 .08482 -.18882 .14755

.234 .629 .000 114 1.000 .00000 .04390 -.08696 .08696

.000 104.500 1.000 .00000 .04403 -.08730 .08730

.009 .924 -.158 114 .875 -.01227 .07773 -.16625 .14170

-.157 104.466 .875 -.01227 .07796 -.16685 .14231

.103 .748 -.614 114 .540 -.03955 .06438 -.16707 .08798

-.617 107.502 .538 -.03955 .06406 -.16653 .08744

5.287 .023 -.010 114 .992 -.00061 .05965 -.11876 .11755

-.010 113.320 .992 -.00061 .05801 -.11553 .11432

2.363 .127 .477 114 .634 .03439 .07212 -.10848 .17727

.484 110.899 .629 .03439 .07099 -.10629 .17508

2.803 .097 1.354 114 .179 .10879 .08037 -.05042 .26800

1.367 109.275 .174 .10879 .07956 -.04889 .26646

.280 .598 .902 114 .369 .05076 .05629 -.06076 .16228

.903 106.110 .369 .05076 .05623 -.06072 .16223

3.749 .055 1.200 114 .233 .08939 .07452 -.05822 .23701

1.216 110.375 .226 .08939 .07349 -.05625 .23503

.321 .572 .326 114 .745 .02088 .06405 -.10600 .14775

.323 101.705 .747 .02088 .06465 -.10737 .14913

1.431 .234 -.591 114 .556 -.00727 .01230 -.03165 .01710

-.602 111.535 .548 -.00727 .01208 -.03121 .01666

.969 .327 1.179 114 .241 .04958 .04206 -.03375 .13290

1.196 110.573 .234 .04958 .04146 -.03258 .13173

1.140 .288 .406 114 .686 .03571 .08800 -.13862 .21004

.408 107.639 .684 .03571 .08754 -.13781 .20923

Equal variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumed

T1

T2

T3

T4

T5

T6

T7

T8

T9

T10

T11

T12

T13

T14

T15

F Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)Mean

DifferenceStd. ErrorDifference Lower Upper

95% ConfidenceInterval of the

Difference

t-test for Equality of Means

Table 4.17. Independent Sample t-test for surprise tests

53

V. DISCUSSION

This chapter discusses and addresses the research questions based on the results

obtained from the data analysis of this study.

The Website for this research study was designed to provide chemistry materials

to users interested in learning descriptive chemistry. The selection of (experimental [WE]

and control group [NWE]) users in the study was based on users who consistently used

the Website and reached the point of attempting all the five sets of surprise tests. This

group of 116 users is called the serious users. Although data analysis shows that a

number of users interested in learning chemistry materials had registered to the AP

descriptive chemistry Website, the final results indicate that only a small fraction of the

total number of users met the selection criteria of "serious user" and utilized the Website.

That is, of the 1373 users for whom some data are available, only 116 were serious

enough to persist in using the Website. Usage patterns and user associations such as the

dates accessed, the total amount of time spent, and number of each user logins were

considered to understand the users’ improvement in practicing chemistry problems and

thus the performance.

Usage Pattern

During the 2006 academic term, the dates of access to the Website show that

many users registered increasingly at the Website each month until the date of the AP

chemistry exam (May 2006). After the AP examination, new registrations declined

precipitously. This pattern of increase and decrease in the number of registrations was

similar in both the experimental and control groups. That is, even for serious users, the

54

date of the first Website visit was skewed toward the time of the AP examination with

more users making their first visit at a time nearer that of the examination. More users

were prone to practice chemistry problems just before the approaching AP chemistry

examination rather than practicing them at an earlier period of time.

The results of the total amount of time spent, which are indicated through the

elapsed time rates, suggest that a majority of all users spent little time practicing

descriptive chemistry problems. Also, it was found that both the WE and NWE group

followed an overall similar pattern of spending brief Website transaction times. That is,

there were no significant differences between the groups suggesting that the use of

worked examples did not make a difference in having users spend more or less time when

practicing chemistry problems (MWE = 12.12 hours, MNWE = 13.05 hours, p = 0.85, Tables

4.3 and 4.4). The extreme variables and outliers in the analyzed results data indicate that

there were a few cases where users appeared to spend either very long or very short times

using the Website. This is a case where having a laboratory situation rather than an

anonymous Website access approach would have given insights as to the reasons for

outliers. For example, for very long times, the users were likely to be off task due to

interruptions.

The total number of user interactions with the Website was analyzed. No

statistically significant difference between the experimental and control group means was

found with respect to average number of interactions (MWE = 259.48 interactions,

MNWE = 294.05 interactions, p = 0.35, Tables 4.5 and 4.6). Serious users in both groups

accessed the chemistry quiz items, eight-item quizzes, tutoring, and feedback. In addition,

users in the worked-example group received worked examples for practice.

55

Chemistry Quiz Items

The final scores of correctly and incorrectly answered chemistry quiz items were

analyzed to distinguish the differences between the experimental and control groups.

Quiz items were chosen by the user from one of the descriptive chemistry categories, and

the submitted quiz item was considered to be answered correctly when all the product and

reactant answers matched with the answers in the database for the particular quiz item.

Both the experimental and control groups accessed various quiz items and submitted their

quizzes for analysis. The total number of quiz items accessed varied within the groups,

however. Statistically significant differences were not found for correctly answered

practice items between the two groups (MWE = 26.72 correct, MNWE = 35.17 correct,

p = 0.17, Tables 4.7 and 4.8). A significant difference was found for incorrectly

answered quiz items. The worked-example group reported fewer incorrect answers than

did the no-worked-example group (MWE = 40.44 incorrect, MNWE = 56.83 correct,

p = 0.04, Tables 4.9 and 4.10).

Tutor Access

Users accessed tutoring resource components for each chemistry category of

interest. Some tutoring resources were accessed repetitively suggesting that the users

reviewed the tutoring more than one time. During the data analysis, the repetitive tutoring

was eliminated from calculations and only the unique tutoring accessed by users was

considered. However, significant differences were not found between the groups; both

groups accessed a similar number of tutoring resources (MWE = 4.26 tutors, MNWE = 3.73

tutors, p = 0.51, Tables 4.11 and 4.12).

56

Eight-item Practice Quizzes

Any user could access an eight-item quiz reminiscent of an actual AP descriptive

chemistry examination question. Significant differences were not found between the two

groups for accessing eight-item quizzes. Users from each group would receive a set of

eight quiz items. Each set was analyzed based on the number of correctly answered

products and reactants. In analyzing the eight-item quizzes, therefore, users were given

credit for partially correctly answered questions (MWE = 3.37 eight-item correct, MNWE =

3.78 eight-item correct, p = 0.31, Tables 4.14 and 4.15).

Worked Examples

Only users in the WE group had access to worked examples. Two worked

examples were possible for each of the first three practice items for each of the 15

practice group categories. The mean values for both the worked-example-1 and worked-

example-2 were considered when analyzing the worked example group user pattern in

utilizing the examples when practicing chemistry problems. It was found that worked-

example-1 was accessed more than twice as often as worked-example-2 (MWE-1 = 16.28

accesses; MWE-2 = 6.74 accesses; Table 4.13).

Surprise Tests

The principal part of this study was based upon the use of surprise tests. In the

many previous studies of this Website, there were no differential treatments nor were all

users assessed in some consistent way. The surprise tests were considered to be the

principal indicators for analyzing the performance of users when practicing chemistry

57

quiz items. Both groups received the same set of questions after the same number of

interactions with the Website. As closely as possible, the twentieth, fortieth, sixtieth,

eightieth, and one hundredth accesses to the site confronted the user with a surprise 3-

item quiz. The results are displayed in Table 4.16.

Based upon previous studies, it was thought that using worked examples would

lead to improved learning. Statistically significant differences were not found in any of

the five sets of surprise tests, however (Table 4.17). Each surprise test included 3 items,

and each of these items (15 total) was studied separately. There was also no discernable

pattern of improvement of scores from the first surprise test to the fifth surprise test

(Figure 5.1). Because the intrinsic difficulty of items cannot be controlled in the context

of realistic practice for the AP examination, variations when only three items are included

can be large. The first item of surprise-test-5 ("excess aqueous 1.0 M sodium iodide is

added to acidified 1.0 M sodium iodate") was perceived of by users as importantly more

difficult than item 2 of surprise-test-4 ("a few drops of methanol are burned in an excess

of air"). Most experienced high school or college chemistry teachers likely would have

guessed this outcome. However, item 3 of surprise-test-1 ("aqueous sodium sulfate is

added to aqueous strontium chloride") is really quite similar to item 3 of surprise-test-5

("aqueous barium chloride is mixed with aqueous potassium sulfate") but the average

scores (MWE = 0.37, p= 0.44 and MNWE = 0.39, p = 0.40) are not dramatically improved.

Eighty interactions took place with the Website between the first and last of these

“surprise test” items.

58

Surprise test items

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Test item number

Scor

e

Worked-example No-worked-example

Figure 5.1. Surprise test items

59

VI. SUMMARY AND RECOMMENDATIONS

This study has had several interesting, if not surprising, outcomes. Unlike the

previous studies which were conducted in 1999, 2000 and 2005, this study has included a

manipulation with experimental and control groups to better understand whether

students’ performance improved in the course of their practice and interaction with the

Website and whether the worked examples helped in reducing the errors that occur in

answering the quiz items.

Performance is the success rate at which students in both the worked-example

group and the no-worked-example group answered the practice quiz items, eight-item

quizzes and surprise tests. The results of the study indicate that there no significant

differences found between the groups in answering the practice quiz items, eight-item

quizzes, and surprise tests.

Although students in the worked-example group were provided with worked

examples during their practice with the descriptive chemistry Website, there were no

significant differences found between the groups and the rate of errors did not decrease or

increase. In other words, both the groups had very similar learning patterns in all the

occasions in using the descriptive chemistry Website.

The researcher finds this outcome to be interesting and further infers the two

reasons that might have caused such a result. User interaction with the Website was

found to be very limited. A larger percentage of users interacted with the Website only

just before the actual AP examination. Also, a majority of users did not continuously

utilize the Website. There were a number of registrants who visited the Website just once

60

or twice and did not further revisit the Website. Early registration (for example, at least

more than two months use and interaction with the Website) might have improved users’

performance.

Extensive use of worked examples and tutoring components might have had

added value to improve the performance. A majority of the students did not access both

of the worked examples provided to them but instead accessed only one or none.

Likewise, users might have referred to the tutoring components more frequently for

answering the chemistry quiz items for understanding the chemistry concepts before

practicing the quiz items.

61

REFERENCES

Bangert-Downs, R. L., Kulik, C. C., Kulik, J. A., & Morgan, M. (1991). The instructional

effects of feedback in test-like events. Review of Educational Research, 61(2),

213-238.

Brooks, D. W., Schraw, G.,Crippen, K, I. (2005). Performance-related feedback: The

Hallmark of Efficient Instruction. Journal of Chemical Education, 82(4), 641-644.

Clark, R. C., & Mayer, R. E. (2003). E-learning and the science of instruction. San

Francisco, CA: John Wiley and Sons, Inc.

College Board, (2006). Bulletin for AP students and parents 2005-2006. Retrieved

August 15, 2006, from

http://www.collegeboard.com/prod_downloads/student/testing/ap/AP-bulletin.pdf

Cooper, G. (1998). Research into cognitive load theory and instructional design at

UNSW. Retrieved September 25, 2005, from

http://education.arts.unsw.edu.au/CLT_NET_Aug_97.HTML.

Crippen, K. J., & Brooks, D. W. (2005). The AP descriptive chemistry question: Student

errors. Journal of Computers in Mathematics and Science Teaching, 24(4), 357-

366.

Crippen, K.J. (2000). Analysis of learning at an advanced placement descriptive

Chemistry Website. Unpublished doctoral dissertation, University of Nebraska at

Lincoln.

Crippen, K. J., Brooks, D. W., & Abuloum, A. (2000). A Website supporting the AP

62

descriptive chemistry question. Journal of Chemical Education, 77(8), 1087-1088.

Dick, W., Carey, L., & Carey, J. O. (2001). The systematic design of instruction (5th ed.).

New York, NY: Addison Wesley, Longman.

Gagné, R. M., Briggs, L. J., & Wager, W. W. (2001). Principles of instructional design

(5th ed.). Fort Worth: Harcourt Brace Jovanovich College Publishers.

Kalyuga, S., Ayres, P., Chandler, P., & Sweller. J., (2003). The expertise reversal effect.

Educational Psychologist, 38, 23-31.

Marzano, R.J., Pickering, D.J., & Pollock, J.E. (2001). Classroom instruction that works:

Research-based strategies for increasing student achievement. VA: ASCD.

Mason, B. J., & Bruning, R. (2000). Providing feedback in computer-based instruction:

What the research tells us. Retrieved September 25, 2005, from

http://dwb.unl.edu/edit/MB/MasonBruning.html.

Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia

learning. Educational Psychologist, 38(1), 43-52.

Miller, George A., (1956). The magical number seven, plus or minus two: some limits on

our capacity for processing information. Retrieved August 5, 2006 from

http://www.well.com/user/smalin/miller.html.

Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design:

recent developments. Educational Psychologist, 38(1), 1-4.

Renkl, A., & Atkinson, R. K., (2003). Structuring the transition from example study to

problem solving in cognitive skills acquisition: A cognitive load perspective.

Educational Psychologist, 38(1), 15-22.

Sweller, J. (1999). Instructional design. Camberwell, Vic.: ACER Press.

63

Sweller, J., Tuovinen, J. E. (1999). A comparison of cognitive load associated with

discovery learning and worked examples. Journal of Educational Psychology,

91(2): 334-341.

van Merrienboer, J. J. G., Kirschner, P. A.,Kester, L. (2003). Taking the Load Off a

Learner's Mind: Instructional Design for Complex Learning. Educational

Psychologist, 38(1): 5-13.

VanLehn, D. A. (1996). Cognitive skill acquisition. Annual Review of Psychology, 47,

513-39.

Walberg, H. J. (1999). Productive teaching. In H. C. Waxman & H. J. Walberg (Eds.),

New directions for teaching practice and research, 75-104. Berkeley, CA:

McCutchen.

64

APPENDIX A

INFORMED CONSENT FORM

65

Contd.

66

Contd.

67

68

APPENDIX B

SURPRISE TESTS

69

Surprise test 1

70

Surprise test 2

71

Surprise test 3

72

Surprise test 4

73

Surprise test 5