Embed Size (px)
Citation preview
This article was downloaded by: [University of California Santa Cruz]On: 26 November 2014, At: 02:14Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH,UK
Studies in Higher EducationPublication details, including instructions for authorsand subscription information:http://www.tandfonline.com/loi/cshe20
Enriching the LearningExperience: A CALM approachCarol Johnston & Nilss OlekalnsPublished online: 25 Aug 2010.
To cite this article: Carol Johnston & Nilss Olekalns (2002) Enriching the LearningExperience: A CALM approach, Studies in Higher Education, 27:1, 103-119, DOI:10.1080/03075070120099403
To link to this article: http://dx.doi.org/10.1080/03075070120099403
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all theinformation (the “Content”) contained in the publications on our platform.However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness, orsuitability for any purpose of the Content. Any opinions and views expressedin this publication are the opinions and views of the authors, and are not theviews of or endorsed by Taylor & Francis. The accuracy of the Content shouldnot be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions,claims, proceedings, demands, costs, expenses, damages, and other liabilitieswhatsoever or howsoever caused arising directly or indirectly in connectionwith, in relation to or arising out of the use of the Content.
This article may be used for research, teaching, and private study purposes.Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly
forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
Studies in Higher Education Volume 27, No. 1, 2002
Enriching the Learning Experience:a CALM approachCAROL JOHNSTON & NILSS OLEKALNSUniversity of Melbourne, Australia
ABSTRACT The article outlines and evaluates a new learning strategy implemented in the Facultyof Economics and Commerce at the University of Melbourne. The strategy is an internet-basedassignment delivery and assessment system designed to (i) equip students to make the link betweenmacro economic theory and important real-world issues, (ii) develop positive attitudes to the subject,(iii) develop deep approaches to learning, (iv) develop a facility for critical analysis and problem-solving, and (v) develop effective study habits. Using a multidimensional evaluation strategy, theindications are that the new approach has succeeded in its aims.
Introduction
In this article, we describe an Internet-based subject delivery and assessment system, Criticaland Analytical Learning in Macroeconomics (CALM), and report the results of an evaluationof its impact on learning outcomes. The system was introduced into a compulsory � rst-yearmacroeconomics subject in the Faculty of Economics and Commerce at the University ofMelbourne in 1999, and then subsequently re� ned, improved, and used again in 2000.
CALM aims to encourage positive attitudes towards macroeconomics, deeper ap-proaches to learning and con� dence in critical thinking skills. The project’s design is basedon the assumption that the achievement of these aims is more likely when students re� ect ontheir own learning, where the stimulus for learning is real-world problems and issues, andwhere assessment rewards the ability to analyse, synthesise and critically evaluate complexmaterial. CALM places particular emphasis on encouraging appropriate study skills and thecapacity to apply knowledge in a variety of contexts, as well as on the development ofcon� dence in using the new electronic teaching and learning technologies. The principal aimhas been to improve the quality of both the level and the type of learning outcomes.
The article proceeds as follows. First, we discuss the background to the CALM project.We then describe the operation of CALM, and follow this with the results of our evaluationof its effect. The article concludes with a discussion of the wider implications of CALM forcourse delivery and content.
Background
The University of Melbourne is a research-based university that attracts very able studentsand places a strong emphasis on providing a high-quality, effective learning environment. Inthis regard, over the past 4 years, the University has invested heavily in assisting academicsto implement the appropriate use of electronic learning technologies. The Faculty of Eco-nomics and Commerce is one of the largest in the University, having approximately 5000
ISSN 0307-5079 print; ISSN 1470-174X online/02/010103-17 Ó 2002 Society for Research into Higher EducationDOI: 10.1080/0307507012009940 3
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
104 C. Johnston & N. Olekalns
students and over 130 academic staff. The cultural background of students is diverse, andover a third of the Faculty’s students are drawn from surrounding South-East Asiancountries. This diversity requires an approach to teaching and learning that is both inclusiveand � exible. Some of the elements that have informed the development of CALM in the lightof this background are cognisance of student approaches to learning, the development ofimportant thinking skills, issues related to transition from school to university, and theopportunities now available through electronic learning technologies.
Approaches to Learning
It is now well understood that students’ learning outcomes are enhanced when there is arecognition of the inter-relatedness of the teaching methods employed with students’ learningstyles and approaches to learning (Entwistle & Ramsden 1983; Bartlett, 1996). Qualitativedifferences in student learning can result from students’ personal attitudes towards a learningtask and the context in which the learning occurs.
This is most clearly articulated in the distinction between deep and surface learning(Marton & Saljo, 1976). Students who use a deep approach are personally involved in thelearning task and seek to obtain some underlying meaning. Such students are likely to readextensively around a given topic, are motivated to discuss the topic with others, andultimately achieve higher grades on assessment tasks (assuming the assessment instrument isdesigned to assess more than a simple recall of facts) than students who use a surfaceapproach (Biggs, 1987, 1999). Surface learners, on the other hand, see learning as a meansto achieve a short-term goal, which may be simply to do enough work to pass an assessmenthurdle.
The CALM project aims to foster a deep approach to the learning task. While there area number of strategies that can be used to link theory to practice, including the use ofsimulations (Grimes & Ray 1993; Chapman & Sorge, 1999), we chose to tie the achievementof this aim into an issues-based assessment strategy. Students are presented with backgroundinformation relating to a particular event drawn from contemporary experience. Wherepossible, the event is something that (i) has occurred in the very recent past (sometimes inthe same week that the assignment is made available to students), and (ii) is amenable toanalysis using the theoretical tools developed in the lecture and tutorial programme and inthe set text. The aim is to encourage students to see the practical usefulness of their subjectmaterial, and to equip them to make more sense of current economic events. This wasthought to have an advantage over simulations, as students were able to respond quickly tocontemporaneous events. Such an immediacy of response is not always possible withpre-existing simulation packages.
CALM also builds on studies which show that an individual student’s achievement isconsistently and positively related to the level of help that the student gives to others(Palinscar & Brown, 1984; Slavin, 1990). The CALM system provides the opportunity tointeract in a structured way with peers, through reading other students’ work and comment-ing upon it. This process compels students to externalise their thoughts and make their ideasexplicit. Enhanced understanding results because students must think about the material anddevelop and structure explanations. Other bene� ts of students interacting directly with theirpeers about their learning include improved communication skills, increased individualself-con� dence and new levels of openness to ideas. Strategies for active learning of this kindhave been widely documented (see, for example, Meyer & Jones, 1993).
Assessment is a powerful tool to stimulate a deeper approach to learning (Ramsden,1992; see also Marton et al., 1984). The amount and type of assessment that students are
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
Enriching the Learning Experience 105
asked to undertake will in� uence their approach to learning. If they are assessed too much,and if the thinking skills that are assessed are of a lower order, students will respond byadopting a surface approach to learning in preparing for such assessment tasks. CALM usesassessment as a vehicle for the encouragement of deeper approaches to learning. The formof assessment is designed to encourage students to re� ect on their learning, and to encouragethe application of theory to real-world issues. Surface-level learning is not rewarded orencouraged.
Critical Thinking
Critical thinking skills are an important goal of higher education. Recognition in recent years,especially since the Dearing report (1997), of the need for students to ‘learn how to learn’,which incorporates critical thinking, has provided an added imperative to more actively seekto attain this goal (Whitson, 1998).
De� nitions of the concept of critical thinking abound in the literature (see, for example,Feeley, 1976; Johnson, 1992), each of which emphasises different aspects. While criticalthinking is a key pedagogical goal of the CALM project, explaining to students what we meanwhen we use the concept is problematic. The CALM project has broadly adopted anoperational approach and the following criteria are used in assessing student achievement inthis respect. Students should be able to: distinguish between positive and normative state-ments, determine the reliability of a source, determine the factual accuracy of a statement,distinguish the relevant from the irrelevant, identify unstated assumptions, identify inappro-priate use of assumptions and ambiguous arguments, recognise logical inconsistencies in theline of reasoning, and determine the strengths of an argument.
The development of critical thinking skills in students is in part a function of the teachingstrategies that are incorporated into curriculum design. Meyer (1986) outlines severalfeatures of teaching critical thinking that are common to most discipline areas. These are thatcritical thinking is a learnable skill, with teachers and peers serving as resources; thatproblems, questions and issues serve as the source of motivation for the learner; that subjectsare assignment-centred rather than text- or lecture-oriented; that goals, methods and evalu-ation emphasise using content rather than simply acquiring it; that students need to formulateand justify their ideas in writing; and that students collaborate to learn and enhance theirthinking. The CALM project has sought to build these elements into the learning pro-gramme.
Transition Issues
The losses incurred when students do not make the transition from school to universitysuccessfully are great. McInnis & James (1995) report, in their study of Australian � rst-yearstudents, that managing the workload was a key factor in contributing to a satisfactorytransition, and that feedback on progress was of crucial importance to all � rst-year students.
These issues were addressed in the design of the CALM project, � rst, by the adoptionof an assessment strategy that seeks to in� uence the nature of student learning. The methodsused in assessing students are a major in� uence on learning (Boud, 1991; Ramsden 1992).The project assessment design assumes that students learn more effectively and at a deeperlevel if they work consistently throughout the semester, rather than cramming for anexamination at the end. Yet, incoming students often � nd it dif� cult to develop theseconsistent work habits. In order to address this, CALM operates for 10 of the 12 semester
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
106 C. Johnston & N. Olekalns
weeks, leaving the � rst week free for demonstrations and the last week available for re� ectionon the subject material.
Second, CALM provides students with earlier, more detailed feedback on their progress.Students are required to re� ect on their learning through a critical analysis of their peers’responses to an issue. Students are thought to learn more from assessment by their peers, orby re� ective assessment of their own work, than by an academic staff member’s mark(Ramsden, 1992). The form of feedback in CALM gives students a sense of responsibilityand control, and can reveal the extent of the student understanding more clearly than otherassessment methods. Finally, CALM recognises the need to develop � rst-year students’con� dence in their computer-related and Internet-related skills, that they use in later years ofstudy and in the workplace.
Internet-based Teaching Technologies
Face-to-face teaching is only one of the means by which learning is brought about, and isdecreasing in importance with the rise of more � exible, electronically distributed, open,learner-controlled forms of learning. While evaluation of the effect of online learning is stillin its early stages, there is evidence that technology, when an integral part of a comprehensiveplan for instructional change, improves student mastery of basic skills, test scores, writing andengagement (Dwyer, 1996). Importantly, in relation to the approach taken in this project,Levin & Thurston (1996) report that there are positive ‘audience effects’ of publishing on theWeb for others. Students make a greater effort to produce polished essays and assignmentsif these are to be published on the Web. The asynchronous nature of online interactions alsoallows students time to re� ect on a topic before completing an online task. More generally,the advantages of online delivery of elements of subjects are; the increased interactionbetween students and other students and staff in terms of both the quantity and intensity ofthe interaction; better access to group knowledge and support, a more democratic environ-ment where students respond to content rather than to personalities; convenience of access;and, for many, increased motivation (Laurillard, 1993; Harasim et al., 1997).
The CALM Project
Introductory Macroeconomics at the University of Melbourne is taught in two 1-hourlectures per week and one tutorial, typically of 18 students, over a 12-week semester. Thereare approximately 1200 students enrolled each year. The lectures are taken by the samelecturer, are repeated three times on the same day, and each lecture commonly has 400students attending at a time. The tutorials are organised on a Collaborative Problem Solving(CPS) basis, whereby students are further divided into groups of four to solve previouslyunseen problems based on pre-reading exercises (Johnston et al., 2000). Each student is amember of a ‘live’ tutorial as well as an online tutorial composed of the same students.Assessment comprises an examination (70%), tutorial participation and attendance (10%)and CALM responses and critical comments (20%).
Components
The CALM system comprises several inter-related components: the issues page, the onlinetutorial group’s responses and comments pages, the online tutor, and the individual student’spage for private messages from the ‘online’ and ‘live’ tutors and assessment results andcomments. A demonstration module, which allows users to experience CALM from the
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
Enriching the Learning Experience 107
perspective of a student, can be found at k http://www.tlu.ecom.unimelb.edu.au/calmDemo/studentl . Users of the demonstration module should follow the audio links, in numericalorder, in order to navigate the system.
Process
In the � rst week of the semester, all students are assigned to their ‘live’ CPS tutorial and theCALM online tutorial. Three macroeconomic issues are posted to the ‘Issues’ page over thecourse of the semester; one in week 2, one around week 5 and one in week 8. In 2000, theseissues involved (i) a discussion and application of the criteria used to rank countries’macroeconomic performance; (ii) a rationalisation of an apparent contradiction in twonewspaper articles published on the same day, one reporting on Australia’s high rate ofeconomic growth, the other reporting on a major slump in Australia’s housing constructionindustry; and (iii) an explanation of why the value of the Australian dollar against the USdollar fell immediately after the release of national accounts � gures con� rming Australia’shigh rate of economic growth. Students are asked to respond to each issue. This requires thestudent to apply the economic theory developed in lectures and ‘live’ tutorials to the issue.All responses are electronically submitted to the CALM ‘Responses’ page. Students can editand change their own responses up to the submission date. After this date, all students intheir respective tutorials can view all of the responses derived from that tutorial. Studentsidenti� cation is removed when responses appear on the responses page, to allow for privacyconcerns. The display of the tutorial members’ responses provides students with usefulfeedback on the standard of their own work in relation to others in the tutorial, and alsoallows them to see the range of responses to the same issue that are possible.
The next stage in the process is designed to develop re� ective and critical thinking skillsin students. Students read the responses posted by their tutorial group and select and reserveone response on which to critically comment. Students are provided with a list of criteria onwhich to base their comment and a sample generic critical comment on which to model theirwork. Following � nal submission, all responses and comments are available for all tutorialmembers to view. This process is repeated for each of the three issues, meaning that studentssubmit a total of six pieces of work during the semester (three responses and three comments)at intervals of around 2 weeks.
At the end of each issue, responses and comments are assessed and results posted to thestudent’s individual page, so that students have a record of how well they are performing.The � rst issue carries relatively less weight than the other two issues in the overall assessmentof the subject, to allow students to become familiar with the system.
Feedback to students is provided in a number of forms. Students receive a mark on theirresponse and comment; are able to view other students’ work in their own tutorial; can viewmodel responses and comments provided by the lecturer; are provided with the online tutor’soverview report at the end of each issue; and can obtain individual personal comment fromtheir tutor who has assessed their work. The learner is expected to take an active role in thefeedback process, through self-re� ection and comparison with others’ work and the modelresponses.
The Online Tutor
An online tutor, known as the Webmaster, is available to all students. Students can askquestions of the Webmaster at any time and, typically, these will be answered within a24-hour period. The questions and answers are posted to a bulletin board for all students to
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
108 C. Johnston & N. Olekalns
see. This allows all students to have access to the information, not just the student who posedthe question.
Evaluation
CALM has been evaluated in a variety of ways: (1) students completed a survey at the startand end of the semester; (2) they also completed a university-wide Quality of Teachingsurvey; and (3) tutors were interviewed. We report brie� y on the results of each of theseforms of evaluation based on the operation of CALM in Semester 2, 2000.
1. Online Survey
Students completed the � rst survey, online, as a prelude to beginning work on their � rstassignment in the second week of the semester and the second survey, also online, in thetenth week of the semester, prior to submitting their comment for their third assignment. The� rst survey was completed by 1077 students and the second by 1046 students.
The survey was divided into four sections. In Section A, the students were questionedabout their language background, whether they had completed Introductory Microeconom-ics, a prerequisite subject, in the previous semester, and their use and perception of theMicroeconomics subject home page on the Internet. Section A of the second survey differedonly in so far as students were asked about their use and perceptions of the IntroductoryMacroeconomics home page. Section B of the survey asked students to rate their con� dencein relation to key skills. In Section C, students were asked to report on their attitude tomacroeconomics and aspects of their learning approach in the subject. Statements in thissection that relate to student attitudes to macroeconomics were drawn from the Attitudes toMacroeconomics dimension of the Survey on Economic Attitudes (SEA) instrument devel-oped for the US Joint Council of Economic Education (Soper & Walstad, 1983). Statementsthat relate to deep learning were drawn from the Approaches to Study Inventory (Entwistle& Ramsden, 1983). Other statements that relate to collaboration, consistency of work,feedback and relevance were designed speci� cally to evaluate performance in relation to otherobjectives of the CALM project. The � nal section asked students to provide an open-endedcomment on any aspect of the project.
Section A: student characteristics. Results of the � rst section of the pre- and post-surveys arereported in Table I. The table shows that the cohort is almost evenly split between men andwomen, with students from non-English–speaking backgrounds comprising about 40% of therespondents. Nearly 90% of respondents had completed the Microeconomics prerequisitesubject in the preceding semester. There is an interesting difference in the students’ percep-tions of the usefulness of the home pages in the two subjects. Introductory Microeconomicsdid not use Internet-based assessment systems such as CALM, preferring to use its homepages as a repository for lecture materials (including copies of the overhead slides used inlectures). Students from non-English-speaking backgrounds were more likely than those fromEnglish-speaking backgrounds to associate this with an improvement in subject delivery. TheIntroductory Macroeconomics home page was the gateway for students into CALM. Lecturenotes or slides were not made available to students. In contrast to the Introductory Microe-conomics course, there is very little difference between English and non-English speakingstudents as to their perceptions of whether the Introductory Macroeconomics home pageimproved subject delivery. Both English- and non-English-speaking students in Macroeco-nomics used the Internet subject pages more in the second semester and, relative to the � rst
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
Enriching the Learning Experience 109
TABLE I. Student characteristics (percentages in brackets)
Pre Post
Total NESB Total NESB
Total respondents 1077 415 (39) 1046 401 (39)Male 562 (52) 201 (48) 547 (52) 159 (47)Female 515 (48) 214 (52) 497 (48) 212 (53)English-speaking background 662 (61) — 643 (62) —(NESB) Non-English-speaking background 415 (39) — 401 (38) —
Respondent did Introductory Macroeconomics 960 (89) 361 (87) 923 (88) 345 (86)insemester 1 2000
In semester 1 respondent used IntroductoryMicroeconomics home page (pre). In semester 2respondent used Introductory Microeconomicshome page (post).
Never 134 (12) 41 (10) 44 (4) 17 (4)Sometimes 665 (62) 263 (63) 433 (42) 168 (42)Regularly 278 (26) 111 (27) 567 (54) 216 (54)
If respondent used the IntroductoryMicroeconomics (Macroeconomics)home page, did it improve perceptionof subject delivery?
Yes 281 (26) 252 (61) 773 (74) 306 (76)No 796 (74) 163 (39) 271 (26) 95 (24)
Pre: beginning of semester survey.Post: end of semester survey.
semester, students were more likely to report that use of the subject home page improvedtheir perception of subject delivery.
Section B: skills con� dence. The second section of the survey asked students to rate theircon� dence in relation to key skills, using a scale ranging from ‘always con� dent’ (with anumerical value of 4) through to ‘never con� dent’ (with a numerical value of 0). Results arereported in Table II.
Looking � rst at the results for the entire class (i.e. English- and non-English-speakingbackgrounds combined), the Table II shows that students reported high levels of con� dencein their computer-related and Internet-related skills, and these appear to have improved overthe course of the semester. Mann–Whitney tests for the equality of the pre- and post-meanseasily rejected the null hypothesis for both computer- and Internet-related skills in favour ofthe one-sided alternative that the post-mean score was higher. For the computer-relatedskills, the test statistic was 6.08, while for Internet con� dence it was 6.55. These results alsoextended to the English-speaking and non-English-speaking background students when thegroups were analysed separately. For the English-speaking background students, the Mann–Whitney test statistics are, respectively, 3.90 and 4.96, while the corresponding test statisticsfor the non-English-speaking background students are 5.00 and 4.43. All of these teststatistics are signi� cant at the 5% level.
The students’ perception of their con� dence in their critical thinking abilities alsoincreased over the course of the semester. The Mann–Whitney test statistics were, respect-
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
110 C. Johnston & N. Olekalns
ively, 2.89 (entire class), 2.14 (English-speaking background) and 2.14 (non-English-speak-ing background). These statistics, which are all signi� cant at the 5% level, are consistent withCALM having achieved one of its key aims, namely, to produce an improvement in students’con� dence in applying critical analysis to complex problems.
This improvement was not as apparent in the students’ perceptions of their con� dencein their problem-solving skills. The Mann–Whitney test statistics for this question were,respectively, 1.67 (entire class), 1.35 (English-speaking background) and 1.10 (non-English-speaking background). Of these, the � rst two are signi� cant, albeit at only the 10% level.
Section Ci: attitudes, approaches and experiences. Table III shows the results from the thirdsection of the respective surveys, where students were asked to report on aspects of theirattitudes and learning approach. The students were given a series of statements and thenasked to indicate whether they ‘strongly disagree’, ‘disagree’, are ‘uncertain’, ‘agree’ or‘strongly agree’. Their answers were scored using a scale ranging from 0 (for stronglydisagree) through to 4 (for strongly agree).
Considering � rst the entire class, the results indicated an improvement in attitudestowards macroeconomics over the course of the semester. There were signi� cant improve-ments in the mean scores for statements related to the practical application of macroeconom-ics, and students were more likely to report that macroeconomics is one of their favouritesubjects. Students’ enthusiasm for macroeconomics seems to have increased, in particularwith regard to their willingness to read unassigned macroeconomics material and with theirlevel of enjoyment when reading about macroeconomics topics. Students were more likely toreport that they found macroeconomics less boring and less of a waste of time by the end ofthe semester. Students appeared to be more aware of the relevance of macroeconomics, asthere were signi� cant increases in the mean scores for statements that related to students’ useof macroeconomics in analysis and in their recognition of the relevance of macroeconomicsto real-world situations. Students indicated that they worked more collaboratively, in thatthey were signi� cantly more likely to report that they discussed macroeconomics and thatthey valued what other students had to say. Finally, students were more likely to report thatthey received suf� cient feedback on their progress in macroeconomics than in � rst semestermicroeconomics.
Deep approaches to learning as measured by student responses to the statements in thissection of the survey showed no change. Similarly, there was no change in student responsesto the consistent study statements in this section of the survey, although a positive responsein this regard is apparent in other elements of the evaluation.
For the most part, there was a common pattern of responses across English-andnon-English-speaking background students. However, there were some differences; English-speaking background students were more likely to rate macroeconomics as one of theirfavourite subjects and value what other students said in class in the post-survey results.Non-English-speaking background students were less likely to see macroeconomics as a wasteof time and were more likely to see macroeconomics as practical.
Section Cii: direct experiences with CALM. In the second survey, students were also asked toevaluate their direct experiences with CALM. These results are presented in Table IV. Onceagain, the students were given a series of statements and then asked to indicate whether they‘strongly disagree’, ‘disagree’, are ‘uncertain’, ‘agree’ or ‘strongly agree’. The results wereconsistent with students having found CALM to be an easy system to use. Three-quarters ofthe sample either agreed or strongly agreed with the statement that CALM encouraged aconsistent work pattern through the semester. Around 60% of the sample agreed or strongly
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
Enriching the Learning Experience 111
TA
BL
EII
.S
kills
con
�d
ence
(%of
sam
ple
and
mea
ns)
Rat
eyo
ur
con
�d
ence
inre
lati
onto
the
follo
win
gsk
ills
Alw
ays
Gen
eral
lyS
omet
imes
Gen
eral
lyN
ever
Mea
nco
n�
den
tco
n�
den
tco
n�
den
tn
otco
n�
den
tco
n�
den
tS
cale
43
21
0
Su
rvey
pre
pos
tp
repo
stp
rep
ost
pre
post
pre
post
pre
pos
t
%%
%%
%%
%%
%%
All
Stu
den
tsC
omp
ute
r-re
late
d19
.326
.54
5.4
48.9
24.9
20.2
8.3
3.9
2.1
0.6
2.71
2.97
*In
tern
et-r
elat
ed19
.628
.24
5.2
48.6
27.4
18.8
6.2
3.7
1.5
0.7
2.75
3.00
*C
riti
cal
thin
kin
g10
.912
.84
1.6
44.7
36.9
35.4
8.6
6.3
1.9
0.8
2.51
2.63
*P
rob
lem
-sol
vin
g11
.612
.34
0.1
42.4
37.3
36.8
8.7
7.4
2.2
1.1
2.50
2.57
En
glis
h-s
pea
kin
g-b
ackg
rou
nd
stu
den
tsC
omp
ute
r-re
late
d21
.529
.15
0.6
49.8
20.4
17.2
5.7
3.6
1.8
0.3
2.84
3.04
*In
tern
et-r
elat
ed21
.631
.04
8.0
49.1
24.8
16.3
4.4
3.3
1.2
0.5
2.84
3.07
*C
riti
cal
thin
kin
g12
.816
.14
8.5
49.4
32.5
30.7
5.1
3.7
1.1
0.2
2.67
2.78
*P
rob
lem
-sol
vin
g14
.015
.34
5.8
47.5
32.9
31.9
5.9
5.1
1.4
0.2
2.66
2.73
Non
-En
glis
h-sp
eaki
ng
bac
kgro
un
dst
ud
ents
Com
pu
ter-
rela
ted
15.9
22.5
37.
147
.332
.525
.012
.34.
52.
71.
02.
512.
85*
Inte
rnet
-rel
ated
16.6
23.8
40.
747
.831
.623
.09.
24.
51.
91.
92.
612.
89*
Cri
tica
lth
inki
ng
7.7
7.5
30.
637
.344
.143
.014
.210
.53.
43.
42.
252.
38*
Pro
ble
m-s
olvi
ng
7.7
7.5
31.
134
.044
.344
.813
.311
.03.
63.
62.
262.
33
*in
dic
ates
that
the
post
-mea
nsc
ore
issi
gni�
can
tly
hig
her
,at
the
5%le
velo
fsi
gni�
can
ce,t
han
the
pre-
mea
nsc
ore
bas
edon
aM
ann
–Wh
itn
eyte
stfo
rth
en
ull
ofeq
ual
ity
ofm
ean
s,ve
rsu
sth
eon
e-si
ded
alte
rnat
ive
that
the
pos
t-m
ean
ishi
gher
than
the
pre
-mea
nsc
ore.
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
112 C. Johnston & N. OlekalnsT
AB
LE
III.
Att
itu
des
,ap
pro
ach
esan
dex
per
ien
ces
(mea
nsc
ores
)
All
Stu
den
tsE
SB
NE
SB
pre
post
pre
pos
tpr
ep
ost
Att
itud
eto
mac
roec
onom
ics
Ien
joy
read
ing
arti
cles
abou
tec
onom
icto
pics
.2
.24
2.51
*2
.26
2.57
*2.
202.
40*
On
occa
sion
Ire
adan
un
assi
gned
book
/art
icle
onm
acro
econ
omic
s.1
.84
2.04
*1
.79
2.06
*1.
932.
02M
acro
econ
omic
sis
bor
ing.
†1
.51
1.36
*1
.45
1.30
*1.
601.
47*
Iha
tem
acro
econ
omic
s.†
1.2
11.
151
.13
1.07
1.34
1.29
Mac
roec
onom
ics
isa
dif
�cu
ltsu
bje
ctfo
rm
e.†
2.1
42.
142
.10
2.06
2.20
2.26
Mac
roec
onom
ics
isea
syfo
rm
eto
un
der
stan
d.
2.0
11.
942
.02
1.99
1.98
1.86
Mac
roec
onom
ics
ison
eof
my
favo
uri
tesu
bje
cts.
2.0
42.
18*
2.0
42.
19*
2.05
2.15
Stu
dyi
ng
mac
roec
onom
ics
isa
was
teof
tim
e.†
1.1
11.
01*
1.0
10.
921.
271.
15*
Mac
roec
onom
ics
isp
ract
ical
.2
.74
2.83
*2
.85
2.29
2.58
2.69
*
Rel
evan
ceI
can
see
the
rele
van
ceof
mac
roec
onom
ics
tore
alw
orld
issu
es.
3.0
93.
20*
3.2
13.
34*
2.90
2.99
*I
use
mac
roec
onom
ics
toan
alys
esi
tuat
ion
s.2
.16
2.44
*2
.08
2.36
*2.
282.
56*
Dee
pap
proa
chI
�n
dit
help
ful
to‘m
apou
t’a
new
topi
cin
mac
roec
onom
ics
for
mys
elf
by
seei
ng
how
2.2
62.
292
.23
2.23
2.32
2.40
the
idea
s�
tto
geth
er.
Ige
ner
ally
put
effo
rtin
totr
yin
gto
un
der
stan
dth
ings
wh
ich
init
ially
seem
dif
�cu
lt.
2.6
92.
632
.68
2.60
2.71
2.69
Itr
yto
rela
tem
acro
econ
omic
sth
eory
tore
alsi
tuat
ion
s.2
.50
2.73
*2
.50
2.71
*2.
512.
75*
Ien
joy
the
inte
llect
ual
chal
len
geof
mac
roec
onom
ics.
2.4
82.
512
.49
2.51
2.48
2.52
Con
sist
ent
wor
kI
usu
ally
atte
mp
tal
lth
ew
ork
set
for
mac
roec
onom
ics
tuto
rial
s.2
.36
2.37
2.3
42.
302.
412.
48I
revi
sem
ym
acro
econ
omic
sw
ork
regu
larl
y.2
.04
1.95
1.9
21.
822.
232.
16
Col
labo
rati
onI
regu
larl
yd
iscu
ssm
acro
econ
omic
sw
ith
oth
erst
ud
ents
.1
.74
1.93
*1
.64
1.85
*1.
892.
07*
Iva
lue
wh
atot
her
stu
den
tsh
ave
tosa
yin
clas
s.2
.77
2.83
*2
.76
2.83
*2.
772.
84
Fee
dbac
kI
rece
ived
suf�
cien
tfe
edb
ack
onm
ypr
ogre
ssin
Mic
roec
onom
ics
(pre
)/1
.70
2.21
*1
.61
2.22
*1.
862.
21*
Mac
roec
onom
ics
(pos
t)la
st(p
re)/
this
(pos
t)se
mes
ter.
Th
em
ean
sar
eba
sed
ona
scal
era
ngi
ng
from
0(f
orst
ron
gly
dis
agre
e)th
rou
ghto
4(f
orst
ron
gly
agre
e).
*in
dic
ates
that
the
post
-mea
nsc
ore
issi
gni�
can
tly
hig
her,
atth
e5%
leve
lof
sign
i�ca
nce
,th
anth
ep
re-m
ean
scor
e,ba
sed
ona
Man
n–W
hit
ney
test
for
the
nu
llof
equ
alit
yof
mea
ns,
vers
us
the
one-
sid
edal
tern
ativ
eth
atth
ep
ost-
mea
nis
high
erth
anth
ep
re-m
ean
scor
e.†
mea
ns
that
aM
ann
-Whi
tney
test
for
the
nu
llof
equ
alit
yof
mea
ns
vers
us
the
one-
sid
edal
tern
ativ
eth
atth
epo
st-m
ean
issm
alle
rth
anth
ep
re-m
ean
scor
ew
asu
sed
(*in
dic
ates
sign
i�ca
nce
atth
e5
per
cen
tle
vel)
.
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
Enriching the Learning Experience 113
agreed that their critical and problem-solving skills improved during the semester, and thatthey learned from reading the responses of their peers and valued their feedback.
Section D: open-ended comments. In the � rst survey, 188 comments were received, and in thesecond, 306. The � rst survey comments largely related to student perceptions developed inthe � rst semester. There was evidence of concern about feedback, relevance, dif� culty,computers and the need to work more consistently. Typical comments were:
Feedback:I was told very little about how I was doing in Microeconomics last semester. Itseemed as though you were by yourself completely and although it’s good to pushyourself, it would be nice to have had someone there to at least let me know if I’mon the wrong track altogether!
Relevance:I believe that the � rst year economics courses would bene� t from increased empha-sis on topical issues and problems perhaps through the use of newspaper articles intutes [tutorials] or assignments. I believe this would help students to further relateeconomic study and theory to the ‘real world’ and provide increased motivation andinterest in the topics and subject as a whole.
Dif� culty:Macroeconomics is a paradox for me—I see the huge importance it has on the worldyet its ambiguities and unknowns are particularly frustrating. It claims it is a scienceyet has none of the empirical solid grounds of real sciences.
Computers:I hate computers and � nd the idea that it is compulsory dif� cult especially ifeverything is on the ‘homepage’. I have no access to the Internet at home. Thatmeans I have to do it at uni during the day which I am not pleased about.
Study habits:Last semester the teaching staff were fantastic in their willingness to help students.Unfortunately I failed to fully utilise the help on offer throughout the semester andfound myself ‘cramming’ when it came to the exam and consequently got a resultwhich didn’t re� ect my true capabilities. This semester I plan to keep up with thework and seek help when having dif� culties.
Comments derived from the second survey indicated a positive response to the subject, aconsistent approach to study, an appreciation of the real-world application of the subject andthe convenience of online assessment, an acknowledgement of the higher level of feedbackand greater con� dence in the use of the technology. Typical comments were:
Feedback:I think that the idea of CALM is good because it allows you to get feedback overa longer period and it’s more spread out. You can get a clearer idea of what isexpected of you.
Relevance:
As a student who failed Macro a few years ago, and was very apprehensive aboutrepeating it again, the CALM project has allowed me to critically think about
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
114 C. Johnston & N. Olekalns
TA
BL
EIV
.D
irec
tex
per
ien
ces
wit
hC
AL
M(%
ofsa
mpl
ean
dm
ean
s)
Sca
le
Rat
eyo
ur
agre
emen
tw
ith
(4)
(3)
(2)
(1)
(0)
Mea
nM
ean
Mea
nth
efo
llow
ing
stat
emen
ts%
%%
%%
All
ES
BN
ES
B
CA
LM
Ifo
un
dC
AL
Mea
syto
use
.2
1.9
59.0
10.9
6.8
1.4
2.9
32.
962.
88H
elp
wit
hC
AL
Mw
asea
sily
avai
labl
e.1
4.7
49.3
26.2
7.8
1.9
2.6
72.
632.
75
Con
sist
ent
wor
kC
AL
Mh
elp
edm
ew
ork
con
sist
entl
yth
rou
ghth
ese
mes
ter.
20.
454
.617
.16.
01
.92
.85
2.87
2.83
Rel
evan
ceT
heec
onom
icis
sues
rais
edin
CA
LM
help
edm
eto
un
der
stan
dth
eap
plic
atio
nof
econ
omic
theo
ryto
real
28.
156
.511
.92.
11
.43
.08
3.08
3.08
wor
ldec
onom
icis
sues
.
Ski
lls Iha
veim
pro
ved
my
crit
ical
anal
ysis
skill
sas
are
sult
ofco
mm
enti
ng
onth
ere
spon
ses
ofm
yp
eers
.1
0.8
47.5
28.4
10.
03
.32
.53
2.49
2.59
Iha
veim
pro
ved
my
pro
blem
-sol
vin
gsk
ills
du
rin
gth
ese
mes
ter
8.3
49.6
33.3
7.4
1.4
2.5
62.
542.
60
Col
labo
rati
onI
lear
ned
from
read
ing
the
resp
onse
sof
my
pee
rs.
10.
449
.123
.71
2.3
4.4
2.4
92.
412.
62I
valu
edth
efe
edba
ckI
obta
ined
from
the
com
men
tsof
my
pee
rs8.
251
.623
.21
1.8
5.2
2.5
02.
382.
60
Sca
le:
(4) 5
De�
nit
ely
agre
e;(3
) 5ag
ree;
(2) 5
un
cert
ain
;(1
) 5d
isag
ree;
(0) 5
de�
nit
ely
dis
agre
e.
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
Enriching the Learning Experience 115
Macroeconomics and, more importantly, see Macroeconomics in a real-worldsense. In this way, I have gained unexpected satisfaction from this subject, havebeen able to apply theory to my part-time job in funds management, and generallygain con� dence in my economic thinking.
Enjoyment:After struggling to understand, enjoy and pass Micro in semester 1, I was notlooking forward to Macro. However, Macro has been my favourite subject thissemester. I have enjoyed the structure of the subject. The way tutes are run and theCALM assignments helped me to understand the subject and forced me to workthroughout the semester.
Computers:After initially being apprehensive about using the computer to hand in assignments,I have found it easy and relatively stress free.
Study habits:I think that CALM is a very interesting and extremely innovative scheme to reallykeep us slacker students on our toes and to actually keep abreast with whatever workthat is being taught throughout the semester. It has been interesting especially whenwe had to deal with real world issues and try to � gure it out by applying whatevermeagre knowledge and common sense that we have.
Online tutor:Online tutor questions were very interesting to read, both the questions raised andthe answers given. The online questions helped me think of other issues, i.e. theissues raised by the other students. CALM also improved my interest in readingeconomic articles in the papers.
Convenience:CALM is good because you can get help and other students’ ideas to help throughthe online tutor. And also because you don’t have to go to Uni to hand your workin.
These comments were largely consistent with the data from the � rst three sections of thesurveys.
Negative comments primarily related to the fact that some tutors had not commentedspeci� cally on student work. While students recognised the value of other student’s com-ments, they still wanted the tutor to comment individually on their work. A typical commentin this regard was:
My biggest criticism of CALM would be that although we get a mark and criteriasheet, we don’t receive any comments from our Tutor explaining why we got themark that we did, or how we could have improved our response. Although a studentreceives feedback from another student, the respondent may often disagree or feelthat they did not go into enough depth. I am sure the Tutor can still give you thebest feedback, especially on areas where marks were lost.
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
116 C. Johnston & N. Olekalns
TABLE V. Quality of teaching survey results (mean scores)
Department� rst and
second yearSubject mean*mean (n [1st] 5 1925,
Rate your agreement with the following statements (n 5 713) n [2nd] 5 1225)
I had a clear idea of what was expected of me in this subject. 3.7 3.5This subject was well taught. 4.1 3.5This subject was intellectually stimulating. 4.0 3.5I received helpful feedback on how I was going in this subject. 3.5 3.0In this subject teaching staff showed an interest in the academic
needs of students. 3.8 3.5The multimedia-based technology helped me to learn effectively. 4.1 3.7Using the Web helped me to learn effectively. 4.0 3.7I found it useful to access information and subject materials using
the subject web site. 4.0 3.4This subject has improved signi� cantly my understanding of the
discipline of macroeconomics. 4.1 3.6
The University’s Quality of Teaching surveys ask the students to rank their agreement with the statementsaccording to the following scale: (5) 5 Strongly agree; (4) 5 Agree, (3) 5 Neither agree nor disagree;(2) 5 Disagree; (1) 5 Strongly disagree.* The Department of Economics offers seven subjects in the second semester of � rst year and second year.
In this regard, there is still a way to go in encouraging some students to take advantage of thewide variety in forms of feedback that the CALM system allows. Responses to statements inTable III and Table IV indicate, however, that the perception of feedback had improved overthe semester for most students. There were a few other isolated negative comments aboutCALM, but these were idiosyncratic, relating to very particular student grievances.
2. University-wide Quality of Teaching Survey Results
All students are requested to complete a survey in each subject for which they are enrolled,at the end of each semester. The results for Introductory Macroeonomics for the semester inwhich CALM operated are reported in Table V. Introductory Macroeonomics was perceivedmore positively by students than any other � rst or second year subject in the department.While it is dif� cult to separate the factors that have contributed to this result, it appears thatthe online materials, of which CALM was a signi� cant part, were perceived by students toassist them in their learning.
3. Interviews with Tutors
Two tutors were formally interviewed at the end of the semester to obtain their views of howthe CALM process operated in their tutorial groups. Tutor 1 was both a ‘live’ tutor as wellas the online tutor. Broadly, their observations of students correlated well with students’ ownperceptions of how they worked with the CALM system. Students were seen to be enjoyingmacroeconomics and to have a greater appreciation of its relevance:
The strength of CALM is the practical nature of the macroeconomics issues thathave been set. Students like learning about stuff that is relevant to them in their
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
Enriching the Learning Experience 117
daily lives. Let’s face it, if you can’t stimulate students in � rst year and make themwant to learn then we have lost a � ne opportunity. (Tutor 1)
In addition, the tutors noted that the opportunity for cheating was sharply reduced using thisprocess. This reduction was seen as result of the fact that all student work is displayed in eachonline tutorial, and other students are required to read it very carefully and comment on it.
The tutors also noted an improvement over the semester in student ability to criticallyanalyse another’s work:
Students’ approach to the critical comment improved over the semester. Thisdemonstrated that students were re� ecting on what they had written in their originalresponse more than they had done in the past. Previously, students would hand ina piece of work and then promptly forget about it. They would wait passively for thetutor comment but were not actively engaged in the process. (Tutor 2)
Tutor 1 thought that students from non-English-speaking backgrounds were more likely touse the online tutor than those from English-speaking backgrounds. The reasons for this, hebelieved, related to cultural differences. In some cultures, asking questions could be seen toshow disrespect, and students who were new to the country may not have developednetworks of friends in the subject, and were, therefore, more likely to make use of the 24-houronline tutor facility.
Conclusions and Wider Implications
Given its multidimensional nature, it is dif� cult to pinpoint with any certainty exactly whatit is about CALM that has produced the favourable outcomes documented above. Webelieve, however, that some valuable general lessons have been learnt about student ap-proaches to learning and about pedagogical techniques that are capable of producingimproved learning outcomes.
First, there is the importance of tying together abstract coursework with real-worldissues, and of using assessment as a motivational device to encourage students to develop forthemselves the links between the two. There are many pay-offs to this. Students’ commandof the material increases as they are forced to re� ect on their lectures and their reading inorder to apply the material to real-world issues. Students also gain an appreciation of thepractical usefulness of the discipline, which helps to foster a more positive attitude towardstheir studies.
The emphasis on peer feedback has had a signi� cant impact on our students. For many,this will have been the � rst time that they have been given open access to the work of theirpeers. The bene� t of doing this is that it has enabled students to make an evaluation of howtheir work stands in relation to the work of their peers. In order to make sense of thisinformation, students have to re� ect on their own learning, and, in doing so, adopt acognitive approach that is allowing of self-criticism. Students cannot rely on a tutor orlecturer to simply tell them where they have gone wrong, an extremely passive form offeedback, but are required to take on board some measure of responsibility for the feedbackthat they receive. Students also learn important critical skills as they re� ect on the work ofothers, and develop a capacity to identify strengths and weaknesses in arguments, and to offerconstructive suggestions as to how the work of their peers might be improved.
Similarly, the Webmaster’s bulletin board has provided a forum in which students cangain access to the concerns and questions of their peers, and see how those concerns andquestions have been addressed. Tutors often report that, despite their best efforts, many
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
118 C. Johnston & N. Olekalns
students � nd tutorials intimidating and are reluctant to ask or answer questions, presumablyfor fear of being revealed as a poor student. The anonymity provided by the bulletin board,and the opportunity to see that a particular question may have been troubling many otherstudents, can, we believe, be quite empowering for students.
The development of consistent work patterns is another aspect of CALM that we believehas made an important contribution to student learning. The development of a consistentwork effort is part of the process by which students gain a deep knowledge of the coursematerial. However, student responses to statements in the survey, which were designed todetect a deepening of their knowledge, showed there was little change as a result of CALM.Nevertheless, by using an assessment instrument such as CALM, which requires a consist-ency of work effort throughout the semester, our students maintain a study schedule thatallows them the opportunity to develop a deeper level of knowledge than would otherwise bethe case. Ensuring that deeper levels of learning eventuate will be an important part of futurere� nements to CALM.
Students’ satisfaction with the course improved, because they valued the � exibility thatan online assessment tool provides. It is clear from the open-ended comments on the secondsurvey that students saw this � exibility as being an extremely attractive feature. Likewise, the24-hour availability of the Webmaster’s bulletin board, and the ability to ask questions andreceive answers away from campus.
Finally, we did not � nd wide differences between English-speaking and non-English-speaking background students. Non-English-speaking background students comprise over athird of the student body, and so the provision of a learning programme suited to theirparticular needs is important. CALM appears to cater equally well for both languagebackgrounds.
These positive outcomes have only been possible with the use of the Internet technology.With a class size approaching 1200, it would not be possible to give students access to thepeer feedback provided by CALM if a paper-based system was adopted. Nor would itpossible to provide students with assessable course assignments based on contemporaryreal-world events (printing lags alone mean that with 1200 students, paper-based materialsoften have to be prepared well in advance of their distribution). The Webmaster’s bulletinboard, of course, would also not exist were it not for the Internet.
Much has been gained from using the CALM system. Students’ con� dence in theirabilities, both computer- and Internet-related, and in relation to their facility for critical andanalytical thinking, showed signi� cant improvement, as did their approach to their studiesand their perception of the practical usefulness of macroeconomics. Student attitudes tomacroeconomics were more positive at the end of the semester and, given that the subject iscompulsory, this result is particularly satisfying. The evidence we have is that the CALMsystem is an effective way in which to assist students to learn, and in which to foster the skillsthat we believe are required of graduates.
Acknowledgement
A preliminary version of this paper was presented at the European Association for Institu-tional Research Forum, Freie Universitatat, Berlin, September 2000. We would like to thankparticipants at the Forum for their comments, as well as two anonymous referees for thisjournal. Any errors are our responsibility. The project was funded by a grant from theUniversity of Melbourne’s Teaching and Learning (Multimedia and Electronic LearningTechnologies) Committee. Mr Mark McCrohon (Teaching and Learning Unit, Faculty ofEconomics and Commerce, University of Melbourne) undertook the programming for theproject.
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
Enriching the Learning Experience 119
Correspondence: Dr Carol Johnston, Teaching and Learning Unit, Faculty of Economics andCommerce, University of Melbourne, Victoria, 3010, Australia; e-mail: [email protected]
REFERENCES
BARLETT, R. (1996) Discovering diversity in introductory economics, Journal of Economic Perspectives, 10, pp.141–153.
BIGGS, J. (1987) Student Approaches to Learning and Studying, Hawthorn, Victoria, Australian Council forEducational Research.
BIGGS, J. (1999) Teaching for Quality Learning at University: what the student does (Buckingham, Society forResearch into Higher Education and Open University Press).
BOUD, D. (1991) Three principles for good assessment practice, The New Academic, 1, pp. 4–5.CHAPMAN, K. & SORGE, C. (1999) Can simulations help achieve course objectives? An exploratory study
investigating differences among instructional tools, Journal of Education for Business, 74, pp. 225–230.DEARING, R. (1997) Higher Education in the Learning Society. Report of the National Committee of Inquiry into
Higher Education (London, HMSO).DWYER, D. (1996) We’re in this together, Educational Leadership, 54(3), pp. 24–26.ENTWISTLE, N. & RAMSDEN, P. (1983) Understanding Student Learning (London, Croom Helm).FEELEY, T. (1976) Critical thinking: towards a de� nition, paradigm and research agenda, Theory and Research
in Social Education, 4, pp. 1–19.GRIMES, P. & RAY, M. (1993) Economics: microcomputers in the college classroom—a review of the academic
literature, Social Science Computer Review, 11, pp. 453–483.HARASIM, L., HILTZ, S., TELES, L. & TUROFF, M. (1997) Learning Networks: a � eld guide to teaching and learning
online (Cambridge, MA, MIT Press).JOHNSON, R.H. (1992) The problem of de� ning critical thinking, in S.P. Norris (Ed.) The Generalizability of
Critical Thinking: multiple perspectives on an educational ideal (New York, Teachers College Press).JOHNSTON, C., JAMES, R., LYE, J. & MCDONALD, J. (2000) An evaluation of the introduction of collaborative
problem-solving for learning macroeconomics, Journal of Economic Education, 31, pp. 13–29.LAURILLARD, D. (1993) Rethinking University Teaching; a framework for the effective use of educational technology
(London, Routledge).LEVIN, J. & THURSTON, C. (1996) Research summary: educational electronic networks, Educational Leadership,
54(3), pp. 46–50.MARTON, F. & SALJO, R. (1976) On qualitative differences in learning: 1—Outcomes and process, British
Journal of Educational Psychology, 46, pp. 4–11.MARTON, E., HOUNSELL, D.J. & ENTWISTLE, N.J. (1984) The Experience of Learning (Edinburgh, Scottish
Academic Press).MCINNIS, C. & JAMES, R. (1995) First Year on Campus Study (Canberra, Australian Government Printing
Service).MEYER, C. (1986) Teaching Students to Think Critically. (San Francisco, CA, Jossey–Bass).MEYER, C. & JONES, T. (1993) Promoting Active Learning: strategies for the college classroom (San Francisco, CA,
Jossey–Bass).PALINSCAR, A. & BROWN, A. (1984) Reciprocal teaching of comprehension-fostering and comprehension-
monitoring activities, Cognition and Instruction, 2, pp. 117–175.RAMSDEN, P. (1992) Learning to Teach in Higher Education (London, Routledge).SLAVIN, R. (1990) Research on co-operative learning: consensus and controversy, Educational Leadership,
47(4), pp. 52–54.SOPER, J.C. & WALSTAD, W.B. (1983) On measuring economic attitudes, Journal of Economic Education, 14(4),
pp. 4–17.WHITSON, K. (1998) Key skills and curriculum reform, Studies in Higher Education, 23, pp. 307–319.
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 02:
14 2
6 N
ovem
ber
2014
