Phillips Johnson: Online Homework versus Intelligent Tutoring Systems: Pedagogical Support for Transaction Analysis and Recording

8/14/2019 Phillips Johnson: Online Homework versus Intelligent Tutoring Systems: Pedagogical Support for Transaction Analys

1/24

Online Homework versus Intelligent Tutoring Systems:Pedagogical Support for Transaction Analysis and Recording

Fred Phillips*

Edwards School of Business25 Campus Drive

Saskatoon, SK, Canada S7N 5A7

University of [email protected]

Benny G. Johnson

Quantum Simulations, Inc.5275 Sardis Road

Murrysville, PA [email protected]

November 13, 2009

Key Words: Artificial intelligence, tutoring, online homework, accounting cycle

* Corresponding author. We thank Linda Chase, Sara Harris, and David Johnson for assistance in

developing the transaction analysis tutor described in this paper, and Sandy Hilton, Christine

Kloezeman, Barbara Phillips, Regan Schmidt, Ganesh Vaidyanathan, two anonymous reviewers,an anonymous associate editor, and participants at the 2009 American Accounting Association

and Canadian Academic Accounting Association annual conferences for comments on a prior

version of the paper. This research was facilitated by financial support from the George C. Baxter

Scholarship at the University of Saskatchewan. Data are available upon request from the first

author. In the interests of full disclosure, Fred Phillips reports an indirect financial interest in the

online homework system examined in this paper. Fred Phillips is a coauthor of textbookssupported by digital products that are based on the online homework system studied in this paper.

Benny Johnson reports a direct financial interest in the intelligent tutoring system examined in

this paper. Benny Johnson founded and operates the organization that developed and markets the

intelligent tutoring system. These potential conflicts of interest work in competing directions;

nonetheless, both authors assert that they have been objective in the research reported in this

paper.
mailto:[email protected]:[email protected]:[email protected]:[email protected]


2/24

Online Homework versus Intelligent Tutoring Systems:

Pedagogical Support for Transaction Analysis and Recording

Abstract

Prior research demonstrates that students learn more from homework practice when using

online homework or intelligent tutoring systems than a paper-and-pencil format.

However, no accounting education research directly compares the learning effects of

online homework systems with the learning effects of intelligent tutoring systems. This

paper presents a quasi-experiment that compares the two systems and finds that students

transaction analysis performance increased at a significantly faster rate when they used an

intelligent tutoring system rather than an online homework system. Implications for

accounting instructors and researchers are discussed.

Key Words: Artificial intelligence, tutoring, online homework, accounting cycle


3/24

Online Homework versus Intelligent Tutoring Systems:

Pedagogical Support for Transaction Analysis and Recording

INTRODUCTION

Example-based instruction and homework practice have been key pedagogies in

accounting for decades. Indeed, Ijiri (1983, 170) remarked in the inaugural volume of

Issues in Accounting Education that (t)he practicality of accountants can perhaps be

attributed largely to this mode of teaching and learning that seems to have existed since

the dawn of accounting education. Over the years, educators have attempted to enhance

this mode of teaching and learning (Schmidt and Bjork 1992; Bonner and Walker 1994;

Wynder and Luckett 1999; Halabi, et al. 2005; Lindquist and Olsen 2007), most recently

by developing new technologies such as online homework systems (Gaffney et al. 2009)

and intelligent tutoring systems (Johnson et al. 2009). These systems provide greater

practice and more timely feedback than ever before. As these systems become

increasingly available, instructors will be expected to choose which system to

recommend or require for students. The purpose of this paper is to inform such a choice

by assessing the relative impact of an online homework system (OHS) and an intelligent

tutoring system (ITS) on students learning.

On the surface, both an OHS and an ITS offer benefits that make them appear

equally desirable. For example, both systems offer the opportunity to practice and gain

instant feedback on a seemingly unlimited number of algorithmically generated

problems. Both systems also provide structures (e.g., response fields, drop-down menus)

that may initially help students in breaking-down analyses and organizing their responses.

An ITS claims to offer the added benefit of allowing students to ask questions of the

tutoring system or to request that it demonstrate the steps needed to solve a particular

1


4/24

problem. While intended as a benefit, this feature could impede learning if students use it

merely as a shortcut to a solution rather than as an enrichment of the learning process. As

this discussion demonstrates, it is not possible to identify the most effective system by

merely comparing the features of the two systems. Rather, empirical analysis is required.

Prior accounting education research empirically compares paper-and-pencil

homework with an OHS (Gaffney et al. 2009) and with an ITS (Johnson et al. 2009),

finding that both systems enhance student performance. However, prior research does not

directly compare the two systems with one another. In this study, we assess the relative

effects of an OHS and ITS on student learning. Participants included 139 business

students enrolled in four sections of an introductory financial accounting course. The first

and third class sections completed a homework assignment using an OHS while the

second and fourth class sections completed the same homework assignment using an ITS.

All students were tested and then proceeded to complete a second homework assignment

using the other system. Analyses of in-class tests held immediately before and after each

homework assignment showed that, on average, students gained knowledge by

completing homework practice. However, the gains were greater immediately after

having used the ITS than after having used the OHS.

This research makes an important contribution to both academic research and

teaching. Until now, no research has directly compared an ITS with an OHS, despite their

increasing prevalence in accounting education. We provide evidence that suggests

students learn more when using an ITS than when using an OHS. We also discuss some

of the system limitations that instructors may wish to contemplate before choosing

between the two systems.

2


5/24

The following sections review prior research, describe the research method used

to assess the relative impact of an OHS and ITS, present empirical results, and conclude

with general observations, limitations, and directions for future research.

BACKGROUND AND PRIOR RESEARCH

Online Homework Systems

Most textbook publishers provide an online homework system (OHS) for

introductory and intermediate accounting courses.1 An OHS offers students the

opportunity to gain immediate feedback on their answers to a seemingly unlimited

number of algorithmically generated homework problems. It also offers instructors

benefits, such as generating consistent and timely outcome feedback on large numbers of

homework problems and automatically entering student grades into a gradebook.

Although an OHS can have offsetting drawbacks (e.g., limited tolerance for minor

departures from problem solutions), research suggests that an OHS contributes to student

learning and that the majority of students prefer an OHS to paper-and-pencil homework

(Bonham et al. 2001; Dillard-Eggars et al. 2008). This enhanced learning and preference

for an OHS is believed to arise from the opportunity for frequent practice and the

immediacy of outcome feedback, both of which have been shown in basic research to

enhance learning (Balzer et al. 1989; Kulik and Kulik 1988).

In accounting, two prior studies assess the impact of an OHS on student

performance. Dillard-Eggars et al. (2008) studied whether accounting principles students

course grades were associated with homework scores obtained when using an OHS.

Results indicated a strong, positive relationship, suggesting that the completion of online

1 Although user interfaces and brand names vary from one publisher to the next (e.g., CengageNow,

WileyPlus), online homework systems share the same basic features discussed in this paper. The OHS used

in this study was McGraw-Hills Homework Manager.

3


6/24

homework may have contributed to student learning. However, this research was not

designed to determine whether the OHS generated incremental learning gains beyond

other forms of homework. Thus, Gaffney et al. (2009) conducted a study in which they

compared student performance across two sections of a financial accounting course; one

section completed homework throughout the course using an OHS and the other section

completed identical homework in a paper-and-pencil format. After statistically

controlling for differences in student profiles between the two sections, analyses

indicated that students in the OHS section outperformed those in the paper-and-pencil

section on course components that resembled the homework.

The results from the accounting studies are generally consistent with research in

other disciplines. Studies in calculus, chemistry, and physics generally report that

students score higher on tests when they have prepared for those tests using an OHS

rather than a paper-and-pencil method (Arasasingham et al. 2005; Cheng et al. 2004;

Dufresne et al. 2002; Hirsch and Weibel 2003; Zerr 2007).

A few studies have found that an OHS does not outperform paper-and-pencil

homework, but these studies have been plagued with confounding factors. In one study,

paper-and-pencil homework scored better than an OHS but the paper-and-pencil

homework was accompanied by extensive instructor debriefing and feedback on both

outcomes and problem-solving processes whereas the OHS provided only outcome

feedback (Bonham et al. 2003). In another study, students who had been assigned to a

paper-and-pencil condition circumvented the experimental treatment by accessing the

OHS via friends in other sections of the course (Cole and Todd 2003). In general, studies

with strong research designs find that OHS contribute to better exam performance.

4


7/24

Despite offering benefits over paper-and-pencil homework, an OHS is not perfect.

Feedback is often limited to the accuracy of students answers, remaining silent about the

methods students use to reach their answers. Further, an OHS provides little help to

students who do not know where to start because the system can assess only the

outcome of the problem-solving process. To address these limitations, a system must be

capable of reading and comprehending each students problem-solving analyses and offer

direct instruction when requested by the student. The artificial intelligence used in an ITS

provides a means of creating this additional support in an online platform. To our

knowledge, only one such system currently exists in accounting. This system is discussed

briefly below and is more fully described by Johnson et al. (2009).2

Intelligent Tutoring Systems

Prototypes of an ITS originated in chemistry, physics, and mathematics. Like an

OHS, an ITS allows students to work on a seemingly unlimited number of homework

problems that are either read into or algorithmically generated by the system. Unlike an

OHS, however, an ITS provides feedback on not only the accuracy of an answer but also

the process used to reach an answer (Johnson et al. 2009). Through this process

orientation, an ITS provides explicit instruction on the process needed to reach a solution,

thereby providing step-by-step assistance to students who have no idea where to start on a

problem. Another advantage of an ITS is that it provides context-specific answers to

students questions about concepts or applications of concepts. This context specificity

allows students to obtain help on the particular problem they are working, rather than a

generic problem that students may perceive as unrelated to their particular problem.

2 A demo of the system, developed by Quantum Simulations, is available at quantumsimulations.com.

5


8/24

The ITS examined in this study focuses on transaction analysis. The tutor includes

four main components: (1) comprehend problem information (i.e., lists of transactions),

(2) identify the accounting equation effects of each transaction, (3) prepare journal entries

to record these effects, and (4) post these journal entries to general ledger accounts

(represented by T-accounts).

The transaction analysis tutor allows a range of interactions similar to what could

occur with a human tutor. For example, if a student does not know how to start a

problem, he can ask the tutor to analyze each step of the problem for him using

conversational-style natural language. On the other hand, if a student wishes, he can

analyze the transaction without seeking help or feedback from the tutor. If errors arise in

his analyses, the tutor will provide corrective feedback when he proceeds from one

subgoal (e.g., analyzing accounting equation effects) to the next (preparing a journal

entry). The student can ask the transaction analysis tutor to check his work, explain how

the tutor would think through a particular part of the problem, or provide instruction on

specific topics (e.g., why is contributed capital categorized as equity?). An important

feature of the tutor is that it dynamically generates its explanations and instructional

points for each individual student, based on the specific part of the particular problem on

which each student is working and the particular responses the student has previously

given. After each explanation or instruction, the tutor allows the student to ask as many

follow-up questions as the student feels is necessary. This back-and-forth exchange

focused on the problem-solving process is the primary reason for expecting that an ITS

will be more effective than an OHS.

6


9/24

An initial study examining the effectiveness of the transaction analysis tutor was

reported by Johnson et al. (2009). In that study, students were initially given a pretest and

then one class section worked on homework using the ITS and another class section

worked on the same homework in a paper-and-pencil format. Performance on a

subsequent test differed significantly between groups; those who used the ITS enjoyed an

improvement of 27 percentage points whereas those who completed paper-and-pencil

homework improved only 8 percentage points. While these differences in performance

are consistent with the ITS providing significant learning benefits over paper-and-pencil

homework, they are subject to limitations. The ITS is an electronic media whereas paper-

and-pencil homework is paper-based. Most students, including those who participated in

the study, were accustomed to completing their homework using an electronic system not

paper-and-pencil. In addition, the research design (i.e., a nonequivalent control group)

used in the prior study did not fully control for alternative explanations of the research

results. For example, differential maturation between groups could account for the

differences in performance (Cook and Campbell 1979). To address these limitations, we

designed the follow-up study described in the following section.

METHOD

Participants and Procedure

This study was conducted with undergraduate business students registered in four

sections of one instructors sophomore-level Financial Accounting course. The students

had an average age of 19, had an average GPA of 2.94, and were split approximately

equally between sexes (56% female). The students completed all homework assignments

and tests, including those analyzed in this study, as part of their regular course

7


10/24

requirements. At the time they completed these course requirements, the students did not

know that their homework and test responses might later be analyzed as part of a research

study. After final course grades had been assigned, all students (n=192) were contacted

by email and asked to become research participants, by giving their informed consent.

Participation in this research did not require any additional work on the part of the

students. After one follow-up request, 139 students (72%) consented to participate.

Participants did not differ from non-participants in regard to their GPA (p=0.299) or

proportion of men and women (p=0.669), suggesting that participants were representative

of the larger student group.

3

To assess the effects of the different homework systems on student learning, we

analyzed students grades from three tests that were administered in class around the time

students were completing homework assignments on the topic of transaction analysis and

recording (Chapters 2 and 3 of the textbook). The first test, administered on the third day

of the course, assessed students ability to analyze and prepare journal entries for 10

transactions that affected only balance sheet accounts. This 15-minute paper-and-pencil

test was given immediately after formal in-class instruction on these topics, prior to any

homework or tutorial practice on these topics. Before the next (fourth) class meeting,

students completed a related set of homework problems requiring analysis and recording

of 19 transactions. Depending on the condition to which students had been assigned,

these homework problems were presented using either an OHS or ITS. At the beginning

of the fourth class meeting, students completed a second (20-minute) paper-and-pencil

test, comprising 14 transactions. Like the first test, this test assessed students ability to

3 The approved research ethics protocol for this study prevents us from analyzing and reporting non-

participants performance data relating to specific elements of the course.

8


11/24

analyze and prepare journal entries for transactions that affected only balance sheet

accounts. The remainder of the class meeting was used to instruct students on analyzing

and preparing journal entries for transactions that affect both balance sheet and income

statement accounts. Before the fifth class meeting, students completed a related set of

homework problems requiring analysis and recording of 28 transactions, using either the

OHS or the ITS. At the beginning of the fifth class meeting, students completed a third

(24-minute) paper-and-pencil test comprising 17 transactions that affected both balance

sheet and income statement accounts.4

To ensure consistency in instruction, the four sections were taught by the same

instructor in sequential 80-minute classes.5 The required class materials in the four

sections were identical, consisting of a textbook and a set of student class notes that were

synchronized with the content of the instructors PowerPoint slides. As in Gaffney et al.

(2009), the OHS allowed students to access homework assignments between specified

dates, receive preliminary feedback on the accuracy of their homework answers, change

their answers, and submit them for grading at any time prior to the deadlines.

Research Design

To familiarize students with the OHS that would be used later in the course, we

required all students to use the OHS to complete an initial homework assignment on

topics in Chapter 1 of the text (e.g., the financial statements and their main users). This

homework assignment did not involve transaction analysis or journal entries, which are

4 Presenting transactions as balance sheet-only effects or combined balance sheet and income statement

effects was consistent with the way they were presented in the related textbook chapters. The test and

homework questions are available from the first author on request.5 The instructor is an author of this paper. One potential concern with this relationship is that students might

feel pressured to participate unwillingly in the research. To minimize this risk, our REB-approved research

protocol required that students be invited to participate in the study only after final grades were assigned

and approved by the department head.

9


12/24

the focus of this study, but it did provide data that we used as a covariate in our analyses.

Following the completion of Chapter 1 homework, the four sections of the course were

assigned randomly to one of two conditions, in a removed treatment nonequivalent

control group with pretest and posttest design (Cook and Campbell 1979). In this design,

the ITS treatment is given to one group of students and withheld from the other group,

which instead completes identical homework problems using the OHS. These conditions

are then reversed for the subsequent homework assignment. Using notation similar to

Cook and Campbell, but designating test scores as T and the experimental treatments as

ITS or OHS, this design can be diagrammed as follows.

GroupCh. 1

Homework Class 3Ch.2 Homework

(Treatment 1) Class 4Ch. 3 Homework

(Treatment 2) Class 5

ITS Earlier OHS T1 ITS T2 OHS T3ITS Later OHS T1 OHS T2 ITS T3

In anticipation that students characteristics might differ between sections, we

alternated the assignment of sections to experimental conditions such that the first and

third class sections were assigned to the ITS Earlier group and the second and fourth class

sections were assigned to the ITS Later group.6 An alternative design would have been to

randomly assign individual students rather than entire course sections to the experimental

conditions, but this design was not chosen for two reasons. First, to ensure equity in final

grade assignment, our universitys REB recommended that grades be standardized across

class sections based on each sections midterm and final exam performance. By assigning

class sections rather than individual students to experimental treatments, we were able to

implement this recommended method of grade determination and adjust for differences in

6 This design choice was intended to evenly distribute varying student characteristics across the two

conditions. It also reduces the effects of information leakage (about test content) from early class sections

to late sections, should any exist.

10


13/24

homework scores that arose between class sections. Second, assigning experimental

treatments to class sections rather than to individual students reduced the risk that students

would question their use of different homework systems for the same homework

assignments. We wanted students to perceive the homework as homogeneous within each

class section; anecdotal observations suggested that we were successful in this because no

students commented on the cross-sectional variation in homework completion method.

RESULTS

Analyses of Homework Practice

For the OHS and ITS to be effective, students must actively engage with them

when attempting the homework practice. Data limitations prevented us from comparing

the time spent by students using the OHS and ITS, but we were able to ensure they had

fully completed the homework assignments. Students in the ITS Earlier group completed

100 percent of the problems in the Chapter 2 homework assignment using the ITS and

earned an average grade of 95 percent when completing the Chapter 3 homework

assignment using the OHS. Students in the ITS Later group earned an average grade of

94 percent when completing the Chapter 2 homework assignment using the OHS and

completed all the problems in the Chapter 3 homework assignment using the ITS.

Analyses of Test Scores

A multivariate analysis of variance (MANOVA) was conducted to determine

whether the two experimental conditions differed in overall test performance. Results

indicate that student performance differed between the experimental conditions (Wilks

=0.949, F=2.415, p=0.035). Average scores on each of the three tests are reported by

condition in Table 1, along with the average Chapter 1 homework score and GPA.

11


14/24

Insert Table 1 about here

Despite alternating the assignment of course sections to condition, we noted that

the average Test 1 score of students in the ITS Later group (80.7 percent) was greater

than that in the ITS Earlier group (72.9 percent) at a statistically significant level (t=2.35,

p=0.020). A similar pattern exists in average Chapter 1 homework scores and GPAs, with

the ITS Later group exceeding that of the ITS Earlier group (t=3.28,p=0.001 and t=2.44,

p=0.016, respectively). These differences are not caused by the experimental treatments

because the treatments were administered later, after Chapter 1 homework and Test 1 had

been completed. Instead, these differences suggest that the ITS Later group comprised

higher performing students than the ITS Earlier group.

To control for these differences, we calculated gain scores for each student by

subtracting from each students subsequent test scores his or her immediately preceding

test score. These gain scores control for differences in each individuals initial level of

performance by examining incremental gains, similar to within-subject analyses of

variance (Rosenthal and Rosnow 1985). Gain scores offer the added benefit of ease of

interpretation. That is, gain scores directly measure performance improvements over

time. For example, by subtracting Test 1 scores from Test 2, we obtain EarlyGain, which

indicates the extent to which each students performance improved between Tests 1 and

2. If students learn more when completing homework with the ITS than the OHS,

EarlyGain will be greater in the ITS Earlier group than in the ITS Later group.7

EarlyGain scores, reported in the middle columns of Table 2 Panel A, indicate

that the performance of the ITS Earlier group increased by 11.8 percent between Tests 1

7 We also attempted to control for differences by reanalyzing the data using the Chapter 1 homework score

and GPA as covariates. These measures were positively correlated with Test 1 scores (p


15/24

and 2. A one-sample t-test indicates that this difference was greater than zero (t=5.06,

p


16/24

al. (2009), which found that students (who performed relatively poorly on an initial test)

improved more after completing homework with an ITS than peers who completed the

same homework using paper-and-pencil. To improve on the design of Johnson et al.

(2009) and rule-out the differential maturation explanation, we incorporated a removed

treatment design in the current study. By removing the ITS treatment from the ITS

Earlier group and giving it to the ITS Later group near the midpoint of our study, we are

able to place the effects of differential maturation in direct opposition to the effects of the

experimental treatments. That is, if differential maturation accounted for the ITS Earlier

groups greater improvement between Tests 1 and 2, the ITS Earlier group would

continue to enjoy greater (or, at worst, equal) gain scores between Tests 2 and 3. If, on

the other hand, the ITS treatment caused the different levels of improvement between

Tests 1 and 2, then providing the ITS treatment to the ITS Later group would reverse the

pattern of differences; the ITS Later group would improve more between Tests 2 and 3

than the ITS Earlier group.

To determine the extent to which performance improved between Tests 2 and 3,

we calculated LateGain by subtracting Test 2 scores from Test 3. The middle columns of

Table 2 Panel B report descriptive statistics for LateGain, which indicate that the

performance of the ITS Earlier group increased by 2.0 percent between Tests 2 and 3. A

one-sample t-test indicates that this increase was not statistically different from zero

(t=1.37, p=0.177). In contrast, the ITS Later group increased its average performance

between Tests 2 and 3 by 5.2 percent. This improvement was statistically greater than

zero (t=3.38, p=0.001). The right-hand column of Table 2 confirms these results by

reporting t-test comparisons of the LateGain scores for the ITS Earlier and ITS Later

14


17/24

groups. As shown, the ITS Earlier groups improvement of 2.0 percent was less than the

ITS Later groups improvement of 5.2 (t=1.49,p=0.069).8These results suggest that the

students who used the ITS between Tests 2 and 3 learned more than the students who

used the OHS during that time. Importantly, these results differ from the pattern that

would be expected if groups matured at different rates as a result of their different

starting points.

DISCUSSION AND CONCLUSION

This study examined the performance of students who had practiced transaction

analysis and recording using an online homework system (OHS) and an online intelligent

tutoring system (ITS). Results indicated that students ability to account for transactions

increased over the period of the study, but at a significantly faster rate when students used

the ITS rather than the OHS. These effects are noteworthy because they do not merely

reflect an active learning effect; students were actively engaged in homework practice

whether it was presented via the OHS or the ITS. Also, the results were not driven by

mere differences in modality (i.e., online versus paper-and-pencil), which is a potential

weakness of prior studies (e.g., Johnson et al. 2009); in the current study, both systems

were accessed online.

If the performance differences were not attributable to differences in active

learning or modality, to what were they attributable? We believe two key attributes of

these systems were responsible for causing the performance differences. First, the nature

of feedback provided by each system differed markedly. Although feedback from the

OHS was immediate, it was limited solely to the accuracy of each student response. After

8 The likelihood of detecting increased scores between Tests 2 and 3 is lower than that between Tests 1 and

2; at Test 2, the average student score in both groups was within one standard deviation of 97 percent.

15


18/24

receiving this outcome feedback, students were left to infer for themselves which element

of their thinking process went astray. Perhaps they recorded the wrong account type (e.g.,

an asset rather than a revenue), wrong account name (e.g., inventory rather than supplies),

or wrong amount (e.g., a partial payment rather than the entire purchase cost); a stumble

with any one of these elements would lead to an incorrect answer. In contrast, the ITS

provided detailed feedback on these elements. Students using the ITS learned not only

whether the final outcome was accurate, but also whether the thinking process that led to

that final outcome was sound.

A second key difference between the OHS and the ITS related to the amount of

instructional support they provided students. The ITS was designed to support students

throughout the problem-solving process. Consequently, students could ask the system

questions tailored specifically to their particular point of uncertainty or confusion. A

student who was unsure why a particular event would be considered an accounting

transaction was able to ask the system to explain that point. Another student who

understood this previous point but did not see the distinction between assets and revenues

was able to ask the ITS to explain that later stage of the problem-solving process. A

student who did not know where to start could ask the ITS to work through and

explicitly model each step of the thinking process, just as a human tutor would. In

contrast, an OHS is primarily concerned with assessing outcomes, so it provides

relatively less instructional support. This point is not a criticism of an OHS, but rather

just an acknowledgement of a different focus. Many participants in our study anecdotally

commented that they appreciated the assessment function that the OHS provided and

some criticized the ITS for its emphasis on helping rather than assessing. These latter

16


19/24

students apparently preferred to use homework practice as an early signal of exam

performance rather than an opportunity to build knowledge and skills.9

Although we are able to describe key differences in the feedback and instructional

support that the two systems provide, our research did not directly test the impact of these

differences on student learning. This limitation provides a worthwhile direction for future

research. Also, being one of the first studies in this area, our research did not manipulate

the many features available in an OHS or ITS to determine their impact on student

learning. Consequently, we cannot conclude that the observed differences in student

learning generalize beyond the specific ways in which the two systems were implemented

in this study. A final limitation is that the current study examines only a subset of the

potential costs and benefits of an OHS and ITS. Beyond their impact on student exam

performance, these systems are likely to impact student perceptions, instructor efficiency,

and possibly instructor effectiveness. As Ijiri (1983, 173) noted more than twenty-five

years ago, computers can make accounting education more efficient (and) leave us

more time in the classroom to spend on the other important dimension of accounting,

namely the human dimension. We hope that the research reported in this paper will

encourage accounting educators to take advantage of these opportunities.

9 Another limitation of the ITS examined in this study is that it covers only a subset of financial accounting

topics (i.e., transaction analysis, adjustments, and financial statement preparation). An ITS has not yet been

developed for topics such as inventory costing, depreciation, and so on.

17


20/24

References

Arasasingham, R. D., M. Taagepera, F. Potter, I. Martorell, and S. Lonjers. 2005.

Assessing the effect of web-based learning tools on student understanding of

stoichiometry using knowledge space theory. Journal of Chemical Education 82

(August): 1251-1262.

Balzer, W. K., M. E. Doherty, and R. OConnor. 1989. Effects of cognitive feedback on

performance.Psychological Bulletin 106: 410-433.

Bonham, S. W., D. L. Deardorff, and R. J. Beichner. 2003. Comparison of student

performance using web and paper-based homework in college-level physics,

Journal of Research in Science Teaching40 (December): 1050-1071.

Bonham, S. W., R. J. Beichner, and D. L. Deardorff. 2001. Online homework: Does it

make a difference? The Physics Teacher39 (May): 293-296.

Bonner, S. E., and P. L. Walker. 1994. The effects of instruction and experience in the

acquisition of auditing knowledge. The Accounting Review 69 (1): 157-178.

Borthick, A. F., M. B. Curtis, R. S. Sriram. 2006. Accelerating the acquisition of

knowledge structure to improve performance in internal control reviews.

Accounting, Organizations and Society 31(4-5): 323-342.

Cheng, K. K., B. A. Thacker, R. L. Cardenas, and C. Crouch. 2004. Using an online

homework system enhances students learning of physics in an introductory

physics course.American Journal of Physics 72: 1447-1453.

Cole, R. S., and J. B. Todd. 2003. Effects of web-based multimedia homework with

immediate rich feedback on student learning in general chemistry. Journal of

Chemical Education 80 (November): 1338-1343.

18


21/24

Cook, T. D., and D. T. Campbell. 1979. Quasi-Experimentation: Design & Analysis

Issues for Field Settings. Boston, MA: Houghton Mifflin Company.

Dillard-Eggars, J., T. Wooten, B. Childs, and J. Coker. 2008. Evidence on the

effectiveness of on-line homework. College Teaching Methods & Styles Journal4

(May): 9-15.

Dufresne, R. J., J. P. Mestre, D. M. Hart, and K. A. Rath. 2002. The effect of web-based

homework on test performance in large enrollment introductory physics courses,

Journal of Computers in Mathematics and Science Teaching21: 229-251.

Gaffney, M. A., D. Ryan, and C. Wurst. 2009. Do on-line homework systems improve

performance? Working paper, Temple University.

Halabi, A. K., J. E. Tuovinen, and Farley, A. A. 2005. The cognitive load of computer

based learning materials for introductory accounting. Issues in Accounting

Education 20: 21-32.

Hirsch, L., and C. Weibel. 2003. Statistical evidence that web-based homework helps.

Focus 23 (February): 14.

Ijiri, Y. 1983. New dimensions in accounting education: Computers and algorithms.

Issues in Accounting Education 1: 168-173.

Johnson, B. G., F. Phillips, and L. G. Chase. 2009. An intelligent tutoring system for the

accounting cycle: Enhancing textbook homework with artificial intelligence.

Journal of Accounting Education 27 (March): 30-39.

Kulik, J. A., and C. C. Kulik. 1998. Timing of feedback and verbal learning. Review of

Educational Research 58 (Spring): 79-97.

19


22/24

Lindquist, T. M., and L. M. Olsen. 2007. How much help, is too much help? An

experimental investigation of the use of check figures and completed solutions in

teaching intermediate accounting.Journal of Accounting Education 25(1-2): 103-

117.

McNamara, D. S., E. Kintsch, N. B. Songer, and W. Kintsch. 1996. Are good texts

always better? Interactions of text coherence, background knowledge and levels

of understanding in learning from text. Cognition and Instruction 14: 1143.

Rosenthal, R., and R. Rosnow. 1985. Contrast Analysis: Focused Comparisons in the

Analysis of Variance. New York: Cambridge University Press.

Schmidt, R. A., and R. A. Bjork. 1992. New conceptualizations of practice: Common

principles in three paradigms suggest new concepts for training. Psychological

Science 3(4): 207217.

Wynder, M. B., and P. F. Luckett. 1999. The effects of understanding rules and a worked

example on the acquisition of procedural knowledge and task performance.

Accounting and Finance 39 (2): 177-203.

Zerr, R. 2007. A quantitative and qualitative analysis of the effectiveness of online

homework in first-semester calculus. Journal of Computers in Mathematics and

Science Teaching26(1): 55-73.

20


23/24

Table 1

Analysis of Mean Performance Scores

(Standard Deviation in Parentheses)

Performance ScoreITS Earlier

(n = 71)ITS Later

(n = 68) Test Statistics

Test 1 72.9 (21.9) 80.7 (16.5) t=2.35, p = 0.020

Test 2 84.7 (12.7) 86.8 (10.0) t=1.05, p = 0.294

Test 3 86.7 (14.6) 92.0 (13.4) t=2.19,p = 0.030

Chapter 1 Homework 91.8 (0.05) 94.6 (0.05) t=3.28,p = 0.001

GPA 2.88 (0.29) 3.00 (0.27) t=2.44,p = 0.016

21


24/24

Table 2

Analysis of Mean Gain Scores

(Standard Deviation in Parentheses)

ITS Earlier

(n = 71)ITS Later

(n = 68) Test Statistics

Panel A:

EarlyGain* 11.8 (19.5) 6.1 (15.9) t=1.89,p=0.030

Difference from zero t=5.06,p

Documents

Phillips Johnson: Online Homework versus Intelligent Tutoring Systems: Pedagogical Support for Transaction Analysis and Recording