1
Version14 23 November 2012
Reading and learning from screens versus print: a study in changing habits
Judith Stoop(1), Paulien Kreutzer (2), and Joost Kircz (3)
[email protected] [email protected] [email protected]
CREATE-IT Applied Research
University of Applied Sciences Amsterdam
Rhijnspoorplein 1, 1091 GC Amsterdam, The Netherlands.
(1) Also: Scherp & Zinnig. http://www.scherpenzinnig.nl
(2) Also: OneTwentyone. http://www.onetwentyone.nl/
(3) Also KRA Publishing research. www.kra.nl
And Corresponding Author
Abstract
In this paper, we report in-depth tests on reading and working with screens. In the present
transition period between information and knowledge representations on paper to
representations on screens, a series of practical as well as intrinsic issues emerged. Both are
crucial aspects for the creation of the written form and for the production of electronic
teaching material. From the practical point of view, ergonomic aspects are the most obvious,
though these aspects are in a state of turbulent flux, as about every 6 months, new and better
devices hit the market. This fact makes tests on those aspects tentative. On the intrinsic side,
we deal with issues such as the materiality of the reading substrate, the capability of making
notes, dog-ears and one’s own summaries. Furthermore, the very act of reading from different
sources, from paper, e-ink screen and LCD screen, creates a difference in the digestion of the
content. In the project Amsterdam Boekenstad (Amsterdam E-book City), we performed two
large tests comparing reading long texts from paper and from an e-ink reader. One test was
with City Council Members and one with students reading a textbook. In both cases, reading
from paper was considered superior. In two large subsequent tests, we used shorter text
materials and asked the students questions on the content of the material. In one test, we
compared print and reading from screens with the same text, which was structured differently
on paper than electronically. In the other test, we used consecutive text from print and a
mindmap representation of the teaching material. Our conclusions indicate that a one-to-one
transition from paper to screen does not work for teaching material. This in contradistinction
to the transition to e-novels. Learning with texts is not a linear activity. This means that if we
make the transition to electronic learning, not only must a series of software improvements,
such as the capability of annotation, be improved, but also the very structure of the material
needs reconsideration and modification.
1 Introduction In our research programme Amsterdam E-Boekenstad (Amsterdam E-book City), we
conducted a range of extensive tests on how professionals and students in higher education
read e-texts. In this project, we investigated the use of e-reading devices in order to
understand how authors and publishers must change their practices, traditionally based on a
paper world, towards the flexible text-representations electronic rendering of text allows.
Contrary to many informative large-scale surveys, such as Rowlands, Nicholas, Jamali &
2
Huntington (2007), Nicholas, Rowlands, Clark, Huntington, Jamali & Olló (2008), Jamali,
Nicolas & Rowlands (2009), which are based on online questionnaires and more inclined
towards measuring and forecasting the potential market, we tried to grasp the possible
differences in understanding a textbook in its various presentation forms by direct interaction
with the users. A methodical interesting study by Ackerman & Goldsmith (2011) shows that
paper is still best suited for learning. A serious issue is the rapid pace of technical
development and acceptance of electronic information and therefore an increasing
acquaintance with electronic reading by students, which makes results of many studies
(Hernon, Hopper, Leach, Saunders & Zhang, 2007; Letchumanan & Tranzi, 2011), including
our own, only tentative. This rapidly changing technological field is also the reason why we
cite mainly papers published over the last years. For a good overview for the implications of
digital text for reading in academe, we refer to Cull (2011). In a recent paper Daniel & Wood
(2013) report an elaborate study in comparing the two reading modes of the same text (print v.
screen) with about 300 students. Thought they don’t report different performance in
understanding, the act of reading was distinctly different.
The usage per se of e-books is also a serious concern for libraries, and various studies deal
with the actual use of e-books available in the library. These studies are mostly carried out by
means of questionnaires and show an increasing interest, due to, among other factors, the ease
of access (Christianson & Aucoin, 2005; Shelburne 2009; Berg, Hoffmann &, Dawson, 2010).
But those e-books are mostly just electronic copies of the paper version, often enhanced with
clickable references etc., but which leave the structure of the book as it is. So, they only
address a different way of accessing traditionally structured information and don’t yet deal
with the aspect of changing learning patterns through use of electronic materials. For a more
general discussion on the tension between technology and communication using e-readers see
Kircz (2012).
The Amsterdam E-Boekenstad project was funded by the SIA-RAAK foundation whose goal
is to foster the relationship between knowledge-seeking companies and knowledge-
generating schools. Hence, the tests were conducted in close collaboration with educational
publishers and in the first phase of the project also with an e-reader manufacturer.
The importance for publishers is not only the transition of carrier (paper to screen) but also
the ways in which content is consumed. In particular, in the case of distance learning, all
document exchange tends now to go via electronic communications; however, but does this
mean that printing at the end-user site is obsolete? This question is highly dependent on the
structure and quality of the texts dedicated to reading from screens. As our first ambition was
to understand the different reading experiences between reading from screen and reading from
paper, we were fortunate to be able to integrate a large project with city councils who also
wanted to test the pros and cons of a changeover from paper to screen.
Interestingly, when the two-year project started on January 1st 2010, the interest was still
rather lukewarm as the e-ink devices were just making inroads into the market. Despite the
harsh winds of marketing, sales, and computer aficionados, the educational publishers’ world
did not advance very fast in developing electronic teaching materials. The Amazon Kindle
was launched in autumn 2007 and became a great success for reading novels. At the other end
of the spectrum, the big scientific publishers already had their huge electronic repositories
with scientific papers in HTML and PDF. For the educational market, it was not yet clear
what the opportunities for e-readers were and to what extent this development would not only
change the business model but also, and more importantly, the way students consume
educational materials. After all, common knowledge, and also the research reported below,
that centuries of refining paper publishing created a most versatile and uniquely usable
product. In that sense, we can use the words of Hillesund (2010) that we can speak of reading
in a period of transition in the field of reading research.
3
Only after the introduction of Apple’s iPad in April 2010, did a shockwave ripple through the
publishing world. The tablet computer proved to be a portable multi-media device,
outflanking the still relatively slow read-only e-ink readers, though e-ink readers are still
superior for reading in an outdoor environment compared to backlit LCD displays. Hence, our
objectives were therefore extended from reading studies only, to wider-scope efforts to
understand educational knowledge transfer, and the relationship between pure reading as a
vehicle and the introduction of multi-media aids, such as instruction films.
Unfortunately, not much research has yet been carried out and published on the actual use of
e-reading devices under controlled conditions. Most of the published work is based on (large-
scale) questionnaires or relatively small local tests; e.g., the most recent large-scale American
report on e-reading by Rainie, Zickuhr, Purcell, Madden, & Brenner (2012) provides an in-
depth overview of the ethnicity, the age, etc., of the users as well as the usage compared to
print books, the purchasing patterns, etc., but does not break down the usages of e-books into
genres. A main extra problem is that e-reading devices are developing so fast that many
negative experiences of only a couple of years ago are now often no longer an issue at all.
Think about the loading time of a page, the speed of browsing through a text, etc. Many
ergonomical aspects are now solved.
In our research programme, we tried to avoid the obvious and concentrate on a few
fundamental aspects in the usage of e-reading devices. Our ultimate aim is to assist publishers
of educational material on how to design, write, model and shape e-books for students. In
order to do so, we executed four major tests. The first test was an attempt to measure the ease
of reading huge amounts of text from an e-ink reader. In this case, we did not use educational
texts. For this test we used the IREX1000D e-ink reader
(http://en.wikipedia.org/wiki/Digital_Reader_1000 ) which had a large screen of 26 cm (10.2
inches) in diagonal, which made it an excellent device for reading PDF files in A4 format (the
hegemonic paper size in Europe). In order to guarantee a heavy reading load as well as an
enthusiastic and disciplined corpus of readers, we collaborated with nine city councils in and
around Amsterdam. In these tests with 45 people over a period of two months, all the
documents council members received for their meetings were uploaded on the Irex1000
readers.
In conjunction with this test and apart from two device usability tests, not reported here (1),
with students of the University of Applied Sciences in Amsterdam, whose qualitative results
are interesting as a check list for designers, we performed three major tests comparing reading
from a screen and reading from paper, in each test working with a different educational
publisher. In the first test with students, we could use a book that suited the study programme
perfectly. In the two subsequent tests, the emphasis was more on learning from e-text per se,
without prior knowledge of the subject. The choice for the specific study material (stimuli)
was also dependent on what the publisher wanted to test and had available.
We started with 80 students, who, for an examination, had to read a book of Van Duuren
Media Publishers ( http://www.vanduurenmedia.nl ), from the Irex1000D, a laptop and on
paper, respectively.
In the next test together with the educational publisher SDU (http://www.sdu.nl ), with 196
students, we compared the scores made in tests of comprehension of a text, which was the
same in content, on paper, and on an LCD screen, but differently arranged.
In the final test with the educational publisher Noordhoff ( http://www.noordhoffuitgevers.nl )
with 173 students, we compared the results between learning from a paper source, from
1 All full reports of the entire project are available in Dutch online at http://eboekenstad.nl
4
learning with the aid of a website with complementary materials, and from learning with the
use of a mind map, with all the add-ons integrated in a digital file on a computer.
We analysed the results of these last two studies statistically. In addition, we had extensive
group discussions with the participating students in order to interpret the results.
We first briefly discuss below the issues dealing with on-screen reading, and subsequently we
report on all four tests in detail.
1.1 Measuring usage Research on reading from electronic devices is not straightforward. We must consider the
following aspects: reading as such, navigation between documents, navigation within
documents and technical aspects.
1- Reading as such has related ergonomical aspects such as: zooming, layout, sharpness, font
design, etc., and physical aspects such as: wearing of the eyes, and the feel and smell of a
book. These aspects have everything to do with the materiality of the device. On this issue,
we see currently important neurological research, comprehensively presented by, e.g., Wolf
(2007) and Dehaene (2010). This research is closely related to the way we write and how texts
are made and its materiality as discussed by Haas (1996). Mangen (2008, 2013) presents
research about nonlinear reading. For educational texts, the issue becomes even more
important though here we are dealing with an ever-stronger interaction between plain text,
graphs, drawings and photos, captions, videos and increasingly interactive elements. The
reader not only consumes the material, but must be able to internalize the content and also be
able to reproduce it. Hillesund (2010) concludes that online immersion is very different from
the hermeneutics of traditional reading. This is exactly what we try to understand, as making
new - digitally born - learning materials must start from this observation. Noyes & Garland
(2006) concluded from an elaborate questionnaire filled out by 274 participants that books are
viewed more favourably than computers, primarily due to the physical and practical aspects of
the two media. In their final conclusion, they state that as books and computers are different
tools and both are equally useful, we need to build upon this in the learning situation. In other
words, what content can be provided through computers and when is a print medium
superior? This is what we tried to find out in our studies.
2- Navigational aspects linking between pages such as: flipping through the pages, skipping
pages, book marking and dog-ears, searching for words, paragraphs, sections or chapters, etc.,
is manifestly evident. It is important to stress that reading an educational book, or as it is in
our first test, political documents, is different than reading a novel. In the latter case, we
“thumb” through the pages, eager to reach the conclusions. But in our tests, consecutive page
turning is not an issue at all.
As said above, the fast development of e-reading devices makes it almost impossible for some
features to use data older than a year. The rapid development of – entertainment-driven –
devices makes it difficult to make a comparative study of the pros and cons of the various
devices. For a recent comparative overview of devices, see Gibson & Forbes Gibbs (2012).
Our choice for the Irex1000D ( http://support.irexnet.com ) was based on the fact that it had
the largest screen available and hence full pages could be displayed equal in size to the paper
book page. The Irex Technologies Company was also a partner in the project, but had to file
bankruptcy in June 2010 due to the slow take-off of it sales in the US. As an unfortunate
result we did not have the opportunity to test the - much faster - successor (IREX DR800SG).
This means that some negative reactions on the use of e-ink, in particular issues dealing with
the speed of page loading and browsing, could not be re-evaluated.
5
3- Navigational aspects on the sentence level, such as: references in the text to and from other
text elements, illustrations, endnotes and footnotes, etc. This aspect deals with hyperlinks
within a text, which is an essential feature of electronic publications. Nevertheless, given the
materials we tested, this aspect did not become a research object in the present study. For this
discussion, see Kircz (1998), Kircz & Harmsze (2000) and Kircz & Den Boef (2013).
4- Ergonomical aspects such as: portability of the device, battery lifetime, the need for
ambient light in the case of e-ink readers versus backlit screens in the case of LCD screens.
Recent usability evaluations are given by Kang, Wang & Lin (2009), with an overview by
Cull (2011).
This aspect is also a standard issue in many studies on the actual use and acceptance of e-
reading devices, such as the large-scale surveys by the London CIBER group: Rowlands,
Nicholas, Jamali & Huntington (2007), Nicholas, Rowlands, Clark, Huntington, Jamali &
Olló (2008), Jamali, Nicolas & Rowlands (2009), already mentioned.
The above list is just a beginning, as presently most e-books are representations of paper
versions. The basic format of a book is still the page, which is a strictly paper-oriented
remnant of the past. The size of a paper page is typically a historical result of the human
reading habit. The eye does not like too long sentences, and also too short lines are usually
judged as unpleasant to read. In newspaper typography, we have clear rules for the column
width and as it turns out, left and right justification of the line of type reads more easily than
ragged right line-endings. Moen (2000) names the following items: legibility is determined by
at least 8 factors: (1) type design, (2) type size, (3) line width, (4) word spacing and letter
spacing, (5) leading, or line spacing, (6) form, (7) contrast, and (8) reproduction quality.
In the case of electronic devices, the page as a unit for information as well as a graphical
container of content becomes obsolete: page-based standards such as PDF are like polaroids
of oil paintings; they try to mimic the old, but do not share their intrinsic features. There is a
good reason for doing so, as almost half a millennium of reading experiences with print
resulted in many proven concepts of typography and layout.
The biggest change until now is ‘reflowable’ texts as we have in most e-book formats. Here,
the page is often, but not always, kept as it was on paper, but enabling various physical
formats of screens to represent the text. The big design challenge is not how to manipulate the
sentences, but how to keep the unity of text, illustrations, tables, white lines, etc., which are
essential for comprehension of most art, educational and scientific books as well as many
poetic works.
The truism of this observation can be seen daily in the huge increase of sales of fiction and
non-fiction books in an e-format like Epub, Mobipocket, etc. reported by all internet
bookshops, which does not demand a fixed relationship between running text and
illustrations.
Electronic schoolbooks are different and similarly with scientific books: the option of screen
rotation, or switch between portrait and landscape modes, plays havoc with the page layout in
cases where lay-out is essential, as indeed it is in many fields.
Below we will discuss our four research studies. Firstly we report two qualitative studies
dealing with the issue of sustained reading from screens. Subsequently we report two studies
under controlled conditions in which we also tried to address the differences in text structure
between paper and electronic texts. The reports are followed by a conclusion and
recommendations for e-study books. The first test was conducted with professional readers,
6
the others with students from our school, and institutions of higher education for
professionals.
2 First test: Sustained reading from a screen by city council members 2.1 Introduction
Reading novels on an e-reader is becoming a normal practice, but what about other types of
documents, where layout and structure are important? In this test, we investigated the ease of
use and the constraints of professional reading of large piles of documents on e-readers. Our
partners in this sub-project were the companies Notubiz (http://notubiz.nl/) and Docwolves
(http://docwolves.nl/ ). These collaborating companies in document management systems
work for a great variety of, mainly public, organisations such as city councils. Originating
from a minutes-taking company, they developed an elaborate document management system
including the electronic distribution of all documents, covering the order of business of
meetings, the archiving, as well as of now, the video registration and indexing of city council
meetings. Their research in the distribution of structured documents for reading from the
screen dovetails with the needs and questions of educational publishers.
2.2 Research objective
The general research quest was to what extent the central distribution of all kinds of
documents covering meetings from the Council Information System to individual council
members could be organised using a paper-free e-reader environment? An important aspect
for our partners, the registrars and commercial partners, is the routing of versions and the
dynamics of mailing lists, as in the process of policy-making, these lists change when
confidentiality aspects change. We do not discuss this further in the present paper because it
pertains more to information management systems than to reading content. But even if all
logistics were solved, the pertinent question of document consumption by reading remains the
main issue. The preparations started in autumn 2009, whilst the actual tests were executed
between August 2010 and March 2011.
Our central quest was: how do council members and members of the registry use and
appreciate the use of an e-reader in their work, and what points for improvement could be
identified? Note that we allowed participants to download e-books of any kind onto their
reader in order to get acquainted with e-readers, but restricted our questions and evaluation
discussions to reading from council meeting minutes.
2.3 Method
With the active participation of the registrars and their clerks, we approached various city
councils in the Amsterdam area to interest council members in tests in which they would
receive all documents on an e-reader during a period of about two months. In most cases, they
also received the traditional pile of paper documents; a few enthusiastic participants insisted
on working electronically only. In most cases, we used the Irex1000D, as described above.
The unique, crucial features of the device are the great legibility of text and the screen size,
which allow a good reading of the standard A4 format.
During council meetings, constant reference is made to particular sentences, clauses and other
items in documents. And after deliberations, the various texts must be voted on. For that
reason, the fixed page layout is essential.
Before the test, a questionnaire was sent to all 241 council members of the participating
councils to find out how the actual document handling was performed. After the tests, we held
evaluation discussions to find out what the experiences were. We opted not for use of a
second questionnaire, as it was not a quantitative test but a qualitative one to try and find out
7
real usage and experiences. In this report, we focus on the qualitative conclusions as they
inform us about the possibilities of using e-readers in professional reading. As, during the
tests, many council members became accustomed to reading from screens and now use
notebooks, tablets, e-readers, etc., our interest was focussed on what can be improved if we
change from paper to screens. As mentioned in the introduction, the field is moving rapidly
and many ergonomical and logistic aspects improve by the day. So, we omit those comments
that are irrelevant for the appreciation of present-day e-readers.
2.4 Sample
We performed tests in 9 cities and boroughs in the wider Amsterdam area. The participating
council members and registrars received their documents via Notubiz’ infrastructure by
logging into a dedicated server. Depending on the enthusiasm, 5 to 10 people per council
worked with e-readers within a minimum of 9 weeks. This period was chosen because it
allowed for a minimum of 2 commission meetings and 2 full council meetings.
In 2 cities, Amstelveen and Zaanstad, next to the Irex1000D, we also used the iPad 1, which
had just hit the market. In the case of Amstelveen, the test, which was more a first
exploration, was not fully the same as with the others councils, but we incorporated the results
as expressed in the evaluation discussions in our conclusions. In the case of Zandvoort,
unfortunately no clear reportable results emerged and they are left out in the results. In table
1, an overview of the cities and participants who completed the test is given.
Table 1: Overview of the cities and participants of this study
City Inhabitants
rounded off
Council
members
Participants Percentage
female
Youngest
age (years)
Oldest
age
(years)
Almere 190,700 39 4 50% 33 71
Amstelveen 81,800 37 2 No data
Amsterdam-
Borough
Centre
82,700 29 4 0% 25 61
Amsterdam
Borough –
West
13,000 29 6 16,6% 26 50
Edam-
Volendam
28,600 21 5 60% 19 57
Waterland 17,100 17 4 50% 38 61
Zaanstad 146,900 39 11 45% 31 64
Zandvoort 16,600 17
Zeevang 6,350 13 4 75% 51 68
2.5 Results:
2.5.1 The document deluge
At the start of the test, a questionnaire about the actual use of paper documents was sent to all
council members of the participating cities, totalling 241 people, of which 90 returned it on
paper and 22 digitally, (total of n=112 or 46%). On average, council members receive more
than 100 pages of text per meeting. In meetings where complicated dossiers are discussed,
such as budgets and zoning plans, the amount of paper can go up to several hundred pages per
meeting. Often, (parts of) the documents – in an earlier version – are being distributed to
council commission meetings before final versions reach the full council meeting. For council
8
members, which in all cases are only part-time politicians, the information overload is large
and storing and handling is complicated.
In 80% of the cases, council members received the documents on paper as well as
electronically, per e-mail or as a file to be downloaded from the council web page; 13%
received a paper version only, and 7% an electronic version only. Given the number of pages
involved, this means that the logistics and paper consumption are considerable. This was one
of the reasons for the registrars to join this project.
In the case of updates - e.g., new versions - we saw an increase in electronic dissemination to
only 17%. Interestingly 12% of the respondents did not compare updates of the documents
with the originals, whilst 15% said that they compare new documents, line by line, with
previous versions.
On average, council members spend 17 hours per week on this work, most of the time in
meetings. When a member is the spokesperson for a subject, 98% of them will carefully read
the documents. However, if a member is not the spokesperson, only 10% of the members
fully read the documents. As said above, all documentation for this, as well as documentation
for the following tests, are available in Dutch on our website.
2.5.2 Paper document use As with educational reading material, the documents are information-carrying tools and
writing on the documents is part of the process of digestion of the content. More than 30% of
the respondents underlined sentences, scribbled in the margin, used markers etc. About 7%
used post-it stickers or look-alike aids to mark text. 6% makes dog-ears to retrieve the
relevant pages and 16% used separate paper sheets to note all comments, often on the first or
last page of the document in question.
It goes without saying that in a digital-only environment, this memorizing and commenting
behaviour has to be transcended into electronic aids. Only 5% of our respondents throw all
paper documents away immediately after the meeting, independently of whether or not the
issue at stake will return on the agenda. 12% retain those documents that deal with still-
pending subjects. 60% retain documents they consider important as such. A solid 16% retain
all paper documents. In the case of electronic versions, which remain on the online archive of
the council, 37% of the members indicated to retain everything on their own computer.
2.5.3 Electronic document usage As asked in the evaluation discussions, the participants rated the importance of the various
functionalities typical for the Irex e-reader, targeted to their use as council member (and not,
e.g., by using the device for reading downloaded books). We discuss the answers below
starting with the most important feature.
1) Readability. Participants were most pleased with the legibility of the text and the ease of
reading. Unfortunately, if the letter size was enlarged (e.g., in order to read without
spectacles), the overview of the page was lost. This is typically the dilemma between the
order of the pages that must be the same for all users, and reflowable text as in e-novels where
font size can be changed without consequences, but where the coordination with other users
gets lost. In that case, two or more readers cannot refer to the same page anymore, since fixed
pages are no longer to be found.
2) Search function. Searching in the documents hosted in the e-reader is possible. Though a
strong demand came to the fore that online searching for related information, not only in the
council document archives, but also on internet pages or services, was imperative, here we see
that the step from paper to an electronic substrate immediately induces demands that belong
to electronic storage as such. In the paper world, a simple index was considered sufficient, but
9
as soon as the step to electronic is taken, all available electronic features are considered
relevant and are demanded by all.
So, going over to an electronic document, all electronic capabilities have to be incorporated.
Whilst our prime concern was the ease and quality of reading, these aspects became essential.
Furthermore, the need for systematic indexing of documents by subject came to the fore, as
free-text searching is not sufficient. At present and also as result of these conclusions, Notubiz
has all public information of the cities they work with in an online searchable archive
(http://www.politiekarchief.nl/ ).
3) Text editing. Text editing and making notes on the text were perfectly possible with a
special stylus. Unfortunately, the speed of the stylus was too slow. This is a typical hardware
and software problem on how overlays or changes in documents by the reader can be
implemented and stored. This technical issue is still in full development in all e-reading
device developments.
4, 5, 6) Browsing speed, battery lifetime, and memory. These are typical issues that
continuously improve performance. These comments made are largely obsolete now, as the
newest e-readers and tablets perform much better within every 6 months.
7) Screen size. Here we touch on the important issue of full-page presentations. Present- day
tablets have about the same size as the Irex1000D and are well suited for full-page viewing.
That it was not rated as the most important feature might be induced by the fact that the pages
were actually readable in full, and no tests with smaller screens were performed.
8) Weight. This was considered no problem as our test people were used to stacks of paper.
9) Private use. Interestingly, those testers using the ink-reader did not see a serious usage for
private activities. We have to take into account that the explosion of e-novels was still to
come. However, the few users, in Zaandam and Amstelveen, who used the iPad 1, did
indicate that it would be good for usages other than only for reading official documents.
2.5.4 Conclusions of test one One of the interesting aspects of this test was the logistics and coherence of the various
documents streams. After all, a set of documents pertaining to various subjects is discussed
and often voted on at meetings. On the one hand, we see the issues of version control,
including auxiliary documents such as motions and amendments to policy papers that are
often tied to a particular version of a document. Here, we have a subject-related collection. On
the other hand, we also have collections that overarch many subjects, such as all documents in
preparation for budget and control meetings. On top of that, many policy papers are deeply
related to others, e.g., the budget for school building renovations with plans for an auto-free
street, the role and place of the public library and many aspects of social welfare. In that
sense, we see an environment with a rich multi-dimensional structure, fit for structured
hypertext. In the test, the documents for one meeting were integrated as one large file with
internal hyperlinks. Nevertheless, people preferred to receive the documents piecemeal, in
particular because most council members want to read documents and prepare interventions
and possible amendments and motions stretched out over the week. Reading from an e-reader
was certainly an advance as they could take, so they say: “the whole pile of documents”
wherever they went. The e-reader also mimics the private ‘archive’ and enables one to look
back at earlier related discussions, versions and drafts.
For all those reasons, we concluded that in order to go over to a digital-only environment, we
have to start with a proper document management system and clear logistics that enable
reading from multiple reading devices, including print on paper, as well as clear indexing of
related documents. This last point is particularly tedious, as, after new elections, the division
of fields in commissions changes. A public space might be one with housing and squares and
in a next phase in the same cluster as traffic and the environment. Thus, the labels to
10
documents in a parliamentary period, and even the way they are phrased (Greens and Parks,
can be renamed in Environmental Planning) do not always guarantee a consistent wholeness
with the contents discussed.
For the registrars involved, it became crystal clear that the way documents are written and
prepared must change. Traditionally, the documents are built up in a scheme that starts with a
whole review of how this issue came into the discussion and what already has been said and
done in the particular case. On paper people can simply skip those pages, in order to go to
those parts that demand a vote. Browsing on a reader or scrolling on a screen is a different
thing. The materiality of paper allows for quick flipping to and from pages with the traditional
aids as dog-ears and sticky notes. In an electronic environment, it turned out to be much more
cumbersome. This induces a discussion on the changing practice of the writing order and
structure of a document.
It is important to note that these problems are obviously similar with current electronic
educational environments, where courses can change, and often are renamed, whilst keeping
large chunks of information unchanged. E-reading is not only a matter of reading, but is
intrinsically deeply tied with the structure of the document and subdocument management.
The pile of documents with a green marker for all documents dealing in some way with parks,
greens and trees stowed away in the hall closet, cannot be transferred to an e-reader without a
total redesign of the documents at issue.
3 Second test: paper versus laptop versus e-ink e-reader
3.1 Introduction
In our second study, 81 students from our ‘Marketing, Media, Publishing’ programme
participated. We used the marketing textbook ‘Digitale Marketing & Communicatie’ (Digital
Marketing & Communication’) by Schuurmans (2008), which is part of the curriculum and
required reading for the subject ‘Online Marketing’. This study took place in the period from
December 2009 to February 2010.30 students studied from the printed book. Another 28
students received the material as a PDF file and studied the material on their own laptops. The
last group of 23 students received the same PDF version, but used the E-ink Irex1000D e-
reader, mentioned above. The important issue here is that we deal with the same page layout
in all cases. Interesting studies of Chong, Lim & Ling (2008, 2009) report the student
preferences for the pure navigation and layout aspects of PDF versions of books. These
recommendations are important in order to convert a page to a web environment, but do not
deal with the structure of the content.
3.2 Research objective
Our central research question was defined as: “How do students, who enrol on a course on the
subject ‘Online Marketing’, experience and appreciate studying from their prescribed book by
using a PDF-file on an Irex1000D, the same PDF-file on a laptop, and the print book version,
respectively?”
3.3 Method
Since - especially at that time - not much was known about the use and appreciation of e-
books in higher education, this research project was exploratory and had a qualitative set-up.
The primary goal was to gain insight and understanding. Before, during and after the test
group discussions with the students were held. In the first round of discussions, before the
experiment took place, 30 students participated, while 66 students participated in the
discussions during and after the tests. Students were asked to keep a logbook. The discussions
11
during and after the experiment were analysed together. The total duration of the experiment
was 12 weeks.
Just as like in our first study, reading the material provided was needed in order to function in
a real-life setting, rather than laboratory tests as the subsequent tests are. Participants were
motivated to give it their best try, since it affected their own functioning in school. The broad
discussion meetings on the teaching material and the reading experience showed sufficient
insight to abstain from statistical analysis, which would not provide deeper knowledge about
the issues at stake.
3.4 Results 3.4.1 Test group Irex users
The students placed in the ‘Irex group’ were very enthusiastic about it at the beginning, but
when the actual experiment started and time passed, their enthusiasm diminished quickly.
The logbooks they kept told us that initially they used their Irex quite often, not so much to
study from, but to show other interested people, who had never seen an e-reader before at that
time. They tried and tested all functions and quickly built a negative image of the e-reader.
All in all, these students concluded that the PDF-file on an Irex offered little to no added
value compared to a paper book. This conclusion concurs with the conclusions of the first test.
During the test period, the students were asked to study only from the version they received at
the beginning of the test. The most striking result was that almost none of the ‘e-reader group’
students ended up studying from the Irex; they put the Irex aside.
They felt using the Irex for the e-book version was an obstacle to their learning process. The
biggest obstacle was that since the Irex was relatively slow, both in starting up as well in
processing and ‘turning pages’, they couldn’t study properly with it. Studying people do not
read a book from cover to cover, but jump from page to page, and chapter to chapter. They go
to and fro through texts and compare pages, pictures and tables. The e-reader was too slow for
this type of reading. Another problem was that the e-version only shows one page at the time.
It does not permit fingers or pieces of paper between pages, though bookmarks are available.
This is a general problem of e-books. It looks more a scroll than a book. Furthermore, in the
e-version, students could not make notes easily. The Irex does allow making notes with a
special stylus, but writing legibly demanded serious training and patience. A remarkable
insight was that the ‘e-reader group’ postponed studying for their exam a longer time than
seen in any other groups. The expectations they held beforehand about studying with an e-
reader couldn’t be met. Apart from technological disadvantages, the design of the e-reader
wasn’t appealing either. They could not believe that it was in black and white only, and found
it too large to take with them in their bags, which is remarkable as the paper book was not
much smaller but heavier and less vulnerable. The e-reader appeared not to be as cool a
gadget as they thought it would be. E-ink readers are well suited for continuous reading, but
lack the functions of a laptop. The perception of the students was that a novel device must
incorporate all functions, a thing nobody expects from a book. Hence, the change from paper
to e-ink was considered old-fashioned and not useful. As it turned out, the students from the
‘e-reader group’ either bought the paper book, used the e-book on a computer or laptop, or
found ways to print the e-book.
3.4.2 Test group laptop users
Furthermore, the students using a PDF-file on their own laptops experimented with the
possibilities of the e-book. They soon found out that there was not much they could do with
the file. In their logbooks and in the group discussions, they complained that they could not
mark, save changes, or copy and paste. Since a large part of this group were used to making
their own summaries on the computer, it would have been very handy if they could have
12
manipulate the document. Apart from technological constraints, another big disadvantage
concerns the lack of easy reading. Since this text was a book of about 200 pages, many people
complained about weary eyes. The e-book was protected, so they could not ‘copy and paste’
parts in order to make a personal summary, nor make notes in the document. Most
importantly, the laptop they needed for reading their book, turned out to be a very distracting
medium. Pop-ups from e-mail, Facebook and other social media often spoiled their
concentration. It was a disappointment that this ‘e-book’ was nothing more than a scanned
version of the paper book, whereas technology - in principle - would make it possible to add
more dynamic functions such as audio or video.
The biggest advantage was that now their book was ‘mobile’. They could consult it from any
location, since it was downloadable from the web. In practice they always had it with them as
they carried their laptops wherever they went. Another advantage was the search function.
The clickable index was also appreciated.
Interestingly, half of the ‘laptop group’ students cracked the code of the protected e-book and
printed it on paper. Learning such a large text from screen did not work for them.
The people that did learn from the screen either could not crack the code or considered buying
the paper book as too expensive. As with the Irex group, the laptop group did not consider
learning from screen an improvement, it didn’t provide any added value for the students who
used them. Indeed it hindered them in their studies.
3.4.3 Test group paper book users Finally the paper group: at first they thought they would miss out on an interesting
experience, being in the ‘boring’ group, but afterwards, they had nothing to complain about
when they learned about the negative experiences of their fellow students. From their
logbooks, it becomes clear that of all students, this group of students spent relatively the
longest time studying. But since many students from groups 1 and 2 switched back to paper
during the experiment, data are hard to compare. Students studying the print book told us they
studied as they usually did, they didn’t meet obstacles, they read great parts of the prescribed
texts and they made summaries. The ease of use of the print book is great, since they are used
to it. Their concentration during their studies was high, since they were not easily distracted
by other functionalities of the ‘device’. The only disadvantage they could think of would be
the weight of the book. But since they were also used to carrying books, this was not
considered a real disadvantage.
3.4.4 Conclusions of test two
One conclusion we can draw from this experiment is that our students were willing to try
something new. Functionality turned out to be the key criterion for continued use or not. The
functionality of both the e-book itself (an ‘ordinary’ PDF-file that was overprotected and
therefore couldn’t be manipulated) and the e-ink reader were very disappointing for these
students and hindered them in their studies. Obviously, reading for study is something quite
different from reading a novel. While studying, one wants to be active with a text, as was the
case with our test with council members: that means marking, making notes, copying and
pasting. Furthermore one doesn’t start on page 1 to end at the last page, but one needs to go
to and fro through a text. This is in line with the conclusion of Ackerman and Goldsmith
(2011, p29) that the decisions to print digitally presented material before study might be
viewed as a meta-metacognitive control decision that transfers the study materials to the more
subjectively reliable context of paper learning. So the navigation technology must be easy and
speed must be high. Students of groups 1 and 2 experienced little to no added value; on the
contrary; they struggled with the immature technology. The ‘e-book’ offered nothing more
13
than the print book in a scanned version. Students from groups 1 and 2 experienced a lack of
overview, due to a lot of necessary scrolling in the PDF-file on the laptop and due to technical
imperfections, in particular page-turning speed of the Irex. This is also the conclusion of
Woody et al. (2010, p945), in a test with 91 students half of whom used an e-book version.
They also conclude, as we do in this paper, that the design of an e-book may need to differ
from that of a textbook to make a more constructive user experience. Furthermore, Brunet,
Bates, Gallo III & Strother (2011) conclude in a 9- month study with dental students that the
electronic bookshelf leads to mixed opinions.
So, at the time of this research project, neither e-reader technology nor e-books were
sufficiently advanced to add significant value when reading for study purposes.
Apart from the required faster and more advanced technology, these students also expected
the e-book content and design to be innovative. An e-book should be more than a simple
digital version of a print book. It should also make use of opportunities like adding sound and
video and also linking it to internet. The Irex was not perceived as a ‘cool gadget’ (whereas
the iPhone is). The design reminded them of the first black-and-white televisions, which
interestingly they do not know from their own experience. They talked about it as ‘that thing’,
which shows that they did not find it especially appealing.
4 Third test: learning from paper versus computer screen
4.1 Introduction
After our more qualitative tests on sustained reading from screens, we turned to more
controlled tests in which we confronted students with new material, not part of the
curriculum, but still within their intellectual reach. The main, practical, reason for this was
the availability of teaching material from our publisher partners.
A group of 196 students from the University of Applied Sciences of Amsterdam took part in
our third experiment. Here, we used a text on internal communications within companies
(Van Riel, 2010). This text was not part of the curriculum, but had relevance within the
broader reach of the study programme. An important aspect here and in the subsequent test is
that we dealt with a single aspect of a larger treatise. As it became clear from all our
discussions, students have the tendency to read only those parts of a book the teacher
prescribes, therewith limiting their self-education to the formal minimum of examination
requirements. Hence, in breaking up learning lines into comprehensible chunks introduces the
danger of a limitation of the context of the teaching material.
We created two research groups. Half of the students worked with a paper version, whilst the
other half used the computer. The paper version comprised a couple of paragraphs from the
book, a separate dictionary and a separate list of rehearsal questions derived from the material
at the end, all put together in a paper reader.
The other group studied exactly the same text, though presented on a series of seven
consecutive web pages. In this version, the text was restructured to fit onto a computer screen.
A mouse-over enabled the dictionary (so, a small pop-up window containing the lemma
becomes visible if the mouse hovers over a word), while test questions were situated together
with the relevant parts of the text.
4.2 Research objective
In this test, our purpose was to test the following hypothesis: “When study material is offered
in an interactive form, the learning results of the user are higher than when the same material
is offered in print.” A secondary goal was to understand why one method would score better
in learning results than the other method. Furthermore, we wanted to discuss the future of
study material with these students in order to be able to make recommendations to our
14
collaborating publishers. In this test, we worked together with SDU Publishers, who provided
us with both the print version as well as the electronic version of the text. This experiment
took place in June 2011.
4.3 Method
The first part of this study entails a quantitative experiment to detect possible differences in
‘learning results’ between the two groups. The groups read the texts and could check their
understanding via rehearsal questions. These results were measured in a separate knowledge
test, containing 24 questions that all participating students had to complete immediately
following 25 minutes of study. The experiment took place in a controlled situation. The
students were randomly assigned to the two experimental conditions:
1. Experimental group (n = 100) – using the interactive version on a PC.
2. Control group (n = 96) – using the print version.
In order to gain further understanding on why one method would score better than the other,
we held group discussions with 31 students. These were mixed discussions that took place
immediately following completion of the knowledge test was, and in a different room of that
where the tests were carried out. Apart from gaining insight into how the students had
experienced the way of learning, we also took this opportunity to brainstorm with them about
the future of ‘study material’.
4.4 Sample
196 students from the School of Design and Communication participated. Participation was
voluntary, but participants received €10 as a teaser. Table 2 shows how they were randomly
assigned to each one of the groups. Please note that RMC (Advertising, Marketing and
Communication), MMP (Media Marketing Publishing), IM (information and media), RMP
(Editing Media Production), NM (News and Media) and IAM (Interactive Media) are all
specific discipline branches of study directions within the school.
Table2: Study compared to experimental condition
What is your discipline?
Total How did you study?
From print From pc
% % %
First year Communications 24.0 24.0 24.0
RMC 18.9 18.8 19.0
MMP 7.1 5.2 9.0
IM 6.6 7.3 6.0
RMP 6.1 8.3 4.0
NM 7.1 6.3 8.
IAM 18.9 18.8 19.0
other study 11.2 11.5 11.0
N 196 96 100
4.5 Results 4.5.1 Quantitative results
In 6 of the 24 test questions, the ‘print group’ scored better, though statistically speaking, this
was not significant. On the other hand, in the remaining 18 cases, the ‘PC group’ scored better
of which 6 had a statistical significance of 90% or higher. Chi-square analysis was done in
order to determine whether or not there were significant relationships between learning result
15
per test question and experimental condition. The result of these 6 questions that showed a
significant relationship are presented in Table 3.
Table 3: Overview of test questions showing significant relationships
Question % correct in print
group (control
group)
% correct in computer
group (experimental
group)
χ2 % of significance
1 87.5% 97% 6.25 98%
4 90.6% 97% 3.46 90%
5 78.1% 89% 4.24 95%
8 85.4% 98% 10.34 99%
15 88.5% 95% 2.73 90%
20 61.5% 73% 2.97 90%
When one looks at the structure of these 6 questions, no striking similarities can be found. In
other words, it is not a specific type of question that made the computer group score
significantly better. This hypothesis can be confirmed for 6 out of the 24 test questions, with a
level of significance of at least 90%.
4.5.2 Qualitative: why the ‘pc group’ scored better
Group discussions were held with 31 students. In these mixed discussions, students explained
how they had experienced studying in their respectively different ways. This helped us to
better interpret the results of the quantitative part. The conclusions can be summarized as
follows.
Students have problems with long consecutive texts and prefer reading comprehensive
chunks, immediately followed by exercises and questions, as was the case with the ‘computer
group’. Contrary to the second study discussed above, where students had to read a whole
book, this study material consisted of ‘only’ 7 web pages with auxiliary texts. These pages
fitted the screen, so no scrolling was needed. None of the students complained about weary
eyes or a lack of overview.
Students study actively, which means that they make notes and summarize read material in
their own words. This turns out to be very difficult with present-day electronic equipment.
Scrap paper was available for all participating students. Although it was not measured how
many of them used scrap paper exactly, it was clear from the amount of used paper that most
of them did.
The rehearsal questions by themselves were appreciated by both groups, but used differently.
The ‘computer group’ used them intensively, while the ‘print group’ hardly used them at all.
Rehearsal questions help to determine what the main issues of the text are. The fact that in the
electronic version, the questions are related to the text, contrary to the print version where the
questions were placed at the end of the text, acted as stimulation to the groups to answer all
the questions and to do so immediately. Students of the ‘computer group’ first read the
question, and then the text, in order to get a better understanding of the text. Students of the
‘print group’ did not attempt to search out the questions while reading (although the
introduction indicated that there were questions offered at the very end of the text), and only
consulted them after reading, and only when they had time left. If they had more time, they
would have liked to have a better look at them. So, these students read and used the material
in the order that it was presented to them.
For the ‘computer group’ the dictionary was enabled by a mouse fly-over. This acted as a
stimulation for them to actively use the dictionary while reading the text, whereas the ‘print
group’ only consulted the dictionary after reading, at the end of the text where it was situated,
and only if they had time left.
16
We, of course, cannot prove a correlation between actively using the rehearsal questions and
the dictionary on the one hand, and having better scores on the test on the other hand (because
there might have been other influences), but this more likely explains the higher scores of the
‘computer group’. Students were better able to digest the knowledge if the primary text, the
dictionary and the rehearsal questions, were intertwined. Of course, this tells us something
about the modes of learning of the students in our applied educational environment. This
might be different in a research university setting and hence suggests the need for
comparative tests.
4.5.3 Qualitative: the future of study material
Our group discussions with the students ended with a brainstorm session about the future of
study material. So this part of our study might help to promote an understanding of what the
actual future will look like as seen from the present. It indeed helps us to understand what
these students think, here and now, and what, is realistic and/or useful to them.
An interesting result is that these students would like to get rid of the term ‘book’. Why
should study material be presented (mainly) as books? With new technologies and interactive
possibilities, one might wonder whether a book is still the most effective way to transfer
knowledge. Almost all of the students, more-or-less, considered a kind of website or computer
program that would replace books in the future. It would be something that would consist of
multiple layers. The upper layer would be a sort of summary, and if one wants to know more
about a topic, one can click to enter deeper levels of understanding.
Written text, as a way to transfer knowledge, will always remain the basis, according to these
students. This is mainly due to the external pacing of written text. The texts should be short
and manageable. Scrolling should become something of the past. Having a good overview is
very important. Texts should be accompanied by video, audio, interactive tests and games. In
this way, studying could become more of a ‘total experience’.
All students we spoke to during our different studies told us they like to study in a more active
fashion. That means ‘doing’ something with the material. In the easiest way, that can be
explained as marking, making notes and making summaries, but with new technologies
‘doing something with the material’ could mean much more, such as: doing tests, interactive
games, rehearsal questions, etc.
Furthermore, social media could be integrated within the material, so people can work
together on topics and/or ask questions to each other, and have discussions. And one last
advantage of ‘e-study material’ is that the author can update the content.
One of the most interesting results of our brainstorms sessions, however, is that almost all of
these students got really excited and enthusiastic when thinking about future possibilities.
They thought study could become a really pleasant and exciting activity this way.
4.5.4 Conclusion of test three
Our hypothesis that the general learning results would be higher in the group that studied in
the interactive form can be confirmed for 6 out of the 24 test questions. Probable reasons for
this result are that students who used the interactive form were more likely to use the
dictionary and use the rehearsal questions as part of their learning process, whereas the
students in the ‘print group’ only consulted these two ‘extras’ at the end and only if they had
time left over. For this last group, these features were not perceived as an integrated part of
the learning material. Therefore, we can tentatively conclude that the order and way in which
learning material is presented influence the degree in which it is actually used and consulted.
Extra features, such as a dictionary or rehearsal questions, should be firmly tied in to the
relevant parts of the text.
17
5 Fourth test: interactive mind map versus printed text with additional
website
5.1 Introduction
A one-to-one translation of a traditional printed study book into an electronic book, e.g., in
PDF form, does not provide any significant added value, as our second study illustrated. Only
if the electronic book has some additional interactive features, such as a dictionary that pops
up via a mouse-over, and where rehearsal questions are related to the relevant text parts, does
the learning result appear to become affected positively, as our third study indicates. Now
what happens if we take this electronic form one step further? That is what we did in our
fourth study. Here we collaborated with the educational publisher Noordhoff, who provided
us with the test material. For this test, we used ten pages from the book ‘Geowijzer’ by Peters
& Westerveen (2011). These are texts about Dutch landscapes. This subject has no direct
relation to the curriculum of these students, but provides a nice mix of text and video and is
intellectually on a par with the level of our students.
Again we tested differences in learning results. In this case, roughly half of the participants
studied from an interactive mind map (see: http://nyjmolen.home.xs4all.nl/mind map.htm ),
and the other half studied from a print reader supported by a website (see:
http://nyjmolen.home.xs4all.nl/geowijzer.htm) offering video material, summaries, rehearsal
questions and a dictionary. We chose for this combination, as currently, more and more study
books offer a supporting website with additional features students can consult. It is important
to mention, that in this test, both experimental conditions offered literally the same text and
the same additional features. But both forms of presentation differed completely.
5.2 Research objective
The hypothesis we wanted to test was: “When study material is offered in an interactive mind
map, the learning result is greater than when the same material is offered in print supported by
an additional website”. Even more than with our previous study (test 3), the relevant material
was tied together in the interactive mind map (i.e., video’s, dictionary, etc. were located right
at the relevant part of text), whereas the students in the control group (print with website) had
to look up the additional features on a computer that was in front of them and preset on the
right web page. A secondary objective was to gain insight into why one method would score
better than the other.
5.3 Method
The first part of this study entailed a quantitative experiment to determine possible differences
in ‘learning results’ between the two groups. This was measured by a knowledge test of 24
questions that all participating students (n = 173) had to answer directly after 30 minutes of
study. The experiment took place in a controlled situation. The students participated
voluntarily, but received a €10 reward, and were randomly assigned to the two experimental
conditions.
The first experimental group of 82 people used the interactive mind map; the control group of
91 people used the print version with supporting website.
In order to gain a further understanding as to why one method would score better than the
other, we held group discussions with 25 students. These were mixed discussions that took
place immediately following the knowledge test in a separate room.
5.4 Sample
18
172 of all 173 participating students studied at our School of Design and Communications of
the University of Applied Sciences in Amsterdam.
5.5 Results 5.5.1 Quantitative
The group that studied from print with supporting website scored better on 8 out of the 24 test
questions. The mind map group on the other hand, scored better on the remaining 16 test
questions. The first impression is therefore that the mind map group had a higher learning
result. However, when we restrict ourselves to statistically significant differences only, it
appears that the print group scored better. On the 8 test questions that this group scored more
highly on, in 3 cases it was significant, hence meaningful. Conversely, the mind map group
scored significantly more highly on only one question.
Table 4 provides an overview of the questions where significant differences between the 2
groups were found. The calculated χ2 should be greater than 2,706 for a significant difference
with a reliability of at least 90%. The higher the calculated χ2 value, the more meaningful the
observed difference and the higher the reliability rate are.
Table 4: Overview of test questions with significant differences between the groups
Question % correct in print
group (control
group)
% correct in mind
map group
(experimental
group)
χ2 Percentage of
significance
1 83.5 93.9 4.55 95%
5 65.9 48.8 5.20 95%
16 92.3 84.1 2.81 90%
20 93.4 82.7 4.77 95%
It appears that the non-significant differences were often not even "almost significant", and
where they are, they equal out for both groups. We must conclude that the results are far from
unequivocal. Nor can we conclude that one study method works better than the other.
Increasing the sample size would probably not lead to clearer results, since there is no
evidence that this would be in favour of one of the test groups. For the learning results, it did
not matter much whether participants were studying from print with supporting website, or
from the interactive mind map. Both forms had advantages and disadvantages. This is further
discussed in the group discussions. So, we have to reject our hypothesis that the interactive
mind map gives better study results in this context.
5.5.2 Other interesting quantitative findings
Apart from answering the 24 test knowledge questions, the participants also answered some
other questions about their behaviour. These responses are analysed below.
Firstly, we wanted to know how many video clips the students had viewed. In the interactive
mind map, 4 video clips (about the texts) could be found. The clips were related to the
relevant text part (and therefore set apart from each other). For the other group, the 4 video
clips were presented side-by-side on the supporting website. The % of students that watched
all four fragments is significantly higher in the print group (42.9%), compared to the mind
map group (22%). But interestingly, the % of students that did not watch any video at all is
also slightly higher in the paper group (30.8%), compared to the mind map group (26.8%).
The students in the print group were more inclined to view all videos, once they had decided
to watch one, clearly because the 4 videos were presented side by side. In the mind map
group, we see more variation in the number of videos viewed. This is presumably due to the
fact that in the mind map, the video clips were spread over the map. Students studying from
19
the mind map watched 2 or 3 clips more often compared to the students from the other group.
Thus, the study method has an influence on the number of videos watched, but it is not such
that, with one method, more videos were viewed in absolute terms, than with the other
method. χ2 analysis confirmed a significant association between the study form and the
number of videos viewed. χ2 is 19,4, indicating a reliability of at least 99%. If videos are
offered together, the student is more inclined to watch all the videos. However, if videos are
presented separately, but related to the relevant text parts, students are more inclined to watch
the video immediately after or before reading the relevant text parts. Table 5 shows the
percentages.
Table 5: Amount of watched videos compared to study method
Did you watch the available
video clips?
Total 1. How did you study?
Print, with additional
website.
Interactive mind
map.
% % %
yes, I watched 4 videos. 32.9 42.9 22.0
yes, I watched 3 videos. 15.0 6.6 24.4
yes, I watched 2 videos. 12.7 7.7 18.3
yes, I watched 1 video. 10.4 12.1 8.5
no, I didn’t watch any
videos. 28.9 30.8 26.8
N 173 91 82
The participants were allowed, if they wished, to use scrap paper during the study. They had
to hand their notes in before the knowledge test started. In our previous study (study 3), we
observed that many students used scrap paper. Therefore, in this test, we decided to measure
this use and see whether study method had an influence on the use of scrap paper. However,
there appeared to be no significant relation between these variables (χ2 is 0.82). The results
are shown in Table 6. We also analysed qualitatively the way the different groups used scrap
paper, but there were no remarkable differences in how they used it, see also point (5) below.
Table 6: Use of scrap paper compared to study method
Did you use scrap paper while
studying?
Total How did you just study?
Paper, with additional
website.
Interactive mind
map.
% % %
no, I
didn’t. 22.5 25.3 19.5
yes, I did. 77.5 74.7 80.5
N 173 91 82
Finally, the students were asked which group they themselves thought would score better on
this test. This led to an interesting result. The students believe that students from their own
group will score better. In the print group 79.1% shared this opinion. The students of the mind
map group were less outspoken. It should be noted that the students answered that question
without actually knowing how the other study method worked. Hence, their belief is based on
a prejudice. The relationship between study method and the expectation of which of the
groups would score better, is highly significant (χ2 is 27.33, so the level of significance is over
99%). See Table 7.
20
Table 7: Expectation of test result compared to study method
With this experiment, half of the students
studied from print with additional website,
the other half studied from an interactive
mind map. Which group scores better on the
knowledge test (that you also just made)?
Total How did you just study?
Print, with extras on
the computer.
From the
Mind map
on the
computer.
% % %
The print group' 60.7 79.1 40.2
The 'mind map group' 39.3 20.9 59.8
N 173 91 82
5.5.3 Qualitative: the use of the interactive mind map
Students first had to get accustomed to the mind map, as for them it was a new way of
representing teaching materials. In the discussions, the following items emerged. The biggest,
perceived, advantage of the mind map is that by itself, it represents a kind of folded
‘summary’. In particular, since these students had only 30 minutes to study the material, such
a summary was appreciated. For larger amounts of learning material, and if they would be
allowed to study for longer than 30 minutes, which is the case in regular learning situations,
the mind map is considered questionable. A folded ‘summary’ felt too constrained. They
suggested a nervousness of missing important things.
On the other hand, a disadvantage of the interactive mind map is the perceived lack of
overview. The computer screen becomes an obstacle between the students and the mind map.
The unfolded mind map is very large and a large fraction does not fit the screen. This means
that the screen window has to be manipulated over the map. In particular, this becomes
problematic as at the start it’s unclear how large the unfolded mind map actually is. Thus, the
students could not easily estimate how much material they had to study and accordingly how
to spend their available time. Although the mind map in the test was perceived as a summary,
many students still felt the need to make their own summaries on scrap paper. Writing things
down is part of the learning process. This brings us to an interesting conclusion that clearly
flipping through book pages does provide an easy understanding of the total volume of the
material, while moving a screen window over a large two-dimensional map does not.
5.5.4 Qualitative: the use of the print reader with additional website
Almost everybody in our evaluation ‘print’ group also consulted the additional website. This
mimics the situation that is now common practice, that study books offer an additional
website with extra material. There was no threshold in our setup to do so, since the computer
was placed right in front of the students and pre-set on the specific webpage. Interestingly, the
students told us they hardly ever consult the extra sites of their study books. They consider it
too much trouble to swap from their paper book to the computer, find the specific website and
use necessary logins codes. This hurdle was removed in our test.
Again (as in our previous studies), we can conclude that students use/read/consult study
material when it is presented to them in a convenient and integrated way. Packing together
different technologies does not improve the learning experience. For the educational
publishers, our students suggested that the additional material on the websites must be
advertised at the relevant place in the text of the book. They also should make it very clear
what can be found on the site, and why it should be accessed. An additional website can be
very helpful, these students think. It could help them fulfil the need to study actively, by
offering extra explanations, summaries, definition trainers, video material or even educational
21
games. The conclusion of these discussions is again that various media have to become
integrated and mutually referenced in a coherent and systematic way. It must be clear from
the learning process why, when, and how a swap between screen and paper is a favourable
activity in the learning process.
5.5.5 Qualitative: the use of scrap paper
In all previous experiments, we encountered the pressing issue of reader annotations. In this
study, we followed Haas (1996, in particular ch.4), by looking at the scrap paper used. White
paper sheets were handed out to all participants and collected afterwards. Most students did
indeed use scrap paper, see Table 6.
Interestingly, apart from scribbles, almost all paper was used to make notes, overviews and
summaries, and in many cases just quotes. It clearly proves the old observation, everybody
also knows from her/his own experience that writing up something is a most important mental
aid for the memorization and internalization of knowledge. This is also stressed by Grafton
(1999) when he discusses the new active way of reading, and a clear break with the scholastic
past, by humanists in the late fifteenth century, when the book began to be used for
annotations and for abstraction and reformulation of the content by the reader.
Summarizing is a filtering process. By summarizing, the reader emphasizes her own priorities
and decides on which topics is no further attention needed.
More research is needed on the question of to what extent making notes on separate sheets of
paper or making a summery by typing on a computer, makes a difference. Though, on a
computer ‘cut & paste’ is an easy way of summarization, it misses the active reformulation of
the content via writing. It goes without saying that on the computer, students are more able to
create order, because that is what they are after. However, the didactic question remains to
what extent rewriting helps in internalizing the content. Scribbles are found frequently; their
role in the learning process also demands further investigations.
5.4.6 Conclusion of test four
There are no clear differences in the learning results of the group that studied from the
interactive mind map, compared to the group that studied from print with the help of an
additional website. We have to reject the hypothesis, for this situation, that the mind map
enhances the results in an unequivocal way.
The mind map was perceived as a summary, which was considered as most useful when one
has only 30 minutes study time. For larger texts and more elaborated topics, an interactive
mind map would probably be less practical. The main negative aspect of the mind map was
that students felt a lack of overview, since they could not predict how large the map would
become; the unfolded mind map didn’t fit the screen size by a long way. The screen actually
stood in their way as a hindrance to a comprehension of the full richness of the mind map.
The students in the print group did visit the additional website and found that useful.
‘Normally’ though, they told us they are not so eager to consult additional websites that come
with study books. It is perceived as too much trouble and offering too little benefit. Therefore,
additional sites should be clearly integrated with the text and each other, with high quality and
relevance as well as being easily accessible. They should really offer added value, and when
they do, these students would very much welcome them.
For the way people study, it matters in what order study material is presented. If videos are
offered together with the other material, then students are more inclined to watch all the
videos. However, if videos are presented separately, connected to the relevant text parts,
students are more inclined to watch the video immediately after or before reading the relevant
text parts.
22
So, in order to let students make optimal use of available material, material should be located
at relevant places and not be ‘hidden away’ at the end or put away on an additional website
with no ‘natural’ accessibility. An important issue reported by Zacharis (2011), is the
relationship between learning style and material used. Further investigations along those lines
demand longitudinal studies, which were unfortunately beyond our reach, though, Emerson &
Mackay (2011) report better results for students who learned the lessons from print rather than
online; no pertinent reasons could be identified.
6 Summary, general conclusions and recommendations
In our programme, Amsterdam E-book City, we have tried to understand how reading from a
digital screen could influence the quality of learning and how the learning material might be
reconstructed in order to increase efficiency and effectively. With this in mind, we started
with the issue of reading per se, using an e-ink reader known for its good jitter free screen
quality. In the first test we collaborated with nine City Councils in the Amsterdam area.
Council members are dedicated readers with a professionally high reading load. The question
was to what extent the document flow and the handling of documents could be improved by
electronic means. During this test period, with third generation e-ink readers, the iPad was
introduced. In a way, this interfered with the original quest, as now more functions, such as
communication and web searching became available. However, it sharpened our research
goals. Reading from an e-ink reader turned out to be appreciated on essentially the same
principles as why people love novels on an e-reader. It is portable, clearly readable and a
reader can host a great many documents. Apart from the more technical aspects of documents
delivery in tune with the meetings scheme of the councils, using e-readers was considered an
advantage compared to print, as long as we deal with the reading experience as such.
The real problem turned out to be the capabilities of browsing, annotating and referencing.
Despite electronic aids, dog ears and sticky notes are superior in re-finding a pertinent
paragraph. Note taking could be done on the device but a piece of paper did better.
Furthermore, the fixed format of an A4 print with hand written notes is easier to comprehend
than electronic comments. We have to take into account that we used PDF files, in where, at
that time, notes could not be integrated.
A more intrinsic aspect that emerged was the very structure of the documents. Parliamentary
documents have a more-or-less fixed structure. People are used to skipping easily parts where
they do not expect to find those particular aspects they are looking for. Electronic documents
can have a much more modular structure, enabling different readers to have different reading
paths. Furthermore, switching to and fro between related pages and having various related
documents “open” is an important demand. In discussions with the Council Registrars, this
was considered an important issue in an attempt to reduce the workload of council-members
while keeping the integrity of the completeness of the documents. These discussions
dovetailed with the ideas developed earlier on the modularisation of academic texts (Kircz
1998, Kircz & Harmsze 2000, Kircz & Den Boef 2012, and references therein).
Interestingly and not unexpectedly, the results of this reading test also merged with the
subsequent studies. Partly simultaneously, together with our partner Van Duuren Media, we
tested 90 students divided into equally large groups. These students read a book that was
preparatory for an examination. One group got the book as a PDF-file on an e-reader, one as
PDF on their laptop and one read the book from print. Here, reading as such, also turned out
not to be the main issue. Again, browsing and annotating turned out to be essential.
Interestingly, opposed to the council members who considered the integration of electronic
23
documents with online searching an important novel asset, our – professional – students
considered the pressure of online communications very distractive. Print won in all categories.
Interestingly, in our subsequent studies, students rated the integration of social media in the
study material of great value in order to exchange findings and problems, although this could
not be substantiated. In a way, one can phrase this as a tension between the discipline of the
book as an independent object and the desire to share knowledge with peers, though within
the context of the learning material.
Subsequently, we oriented our research more on the comparison of the same text in different
presentations, as acquiring knowledge is a complex process surpassing the reading of a text.
In these tests, both with almost 200 students, we asked the students to study a text for about
25 minutes and subsequently fill out a knowledge test. With a selection of the students, round-
table discussions were held. Our test students showed a great interest in a precise definition of
the material, exemplified in the better results of the group who used well-defined chunks of
text on the computer as compared to those who used the full consecutive print version. In the
case of a mind map presentation the lack overview was a serious point.
Students commented that teachers often only prescribe parts of books and therefore inferred
that only part of the material is worthwhile reading for an examination. Here we encounter a
serious issue, as it exemplified the tension between the conscious build-up of the material by
the expert author of a book, on the one hand, and on the other hand the course goals as
defined by the local teacher. This goal is mostly defined by the demands for passing an exam
and not necessarily for full comprehension of a field. This fact together with the conception of
many students that you do not have to read whole books to pass the exam, we are immediately
confronted with the challenge as to how to write and edit study material that stimulates the
students to read and view, in the case of videos rather than the bare minimum. A deep
knowledge of the various interactions between a basic text, auxiliary material, and related
texts is needed and demands further longitudinal research. Note that our school is geared to
educate professionals and not academic researchers, which makes such research difficult to
undertake.
In all our tests, the use of scrap paper turned out to be a universal aid. The writing of
keywords, short sentences or mini-abstracts helps to internalise the content. The same is true
for scribbling in the margins, underlining and highlighting with a marker. In the various
discussion groups, students stressed the tactility of the book. Having a document in your hand,
and keeping it close is also a way for isolating the learning material from the outside world.
Working on a laptop was distracting because of the many other applications “at your
fingertips”.
Two main conclusions can be drawn here.
1) In the transition from paper study materials to electronic materials, a full capability for
marking, scribbling notes, etc., is imperative. This does not mean that people cannot become
accustomed to all kinds of note-making applications in document-handling software.
However, there remains a big difference between writing, typing, and ‘cut & past’. In our
opinion, this conclusion is a hard demand for the development of educational learning
materials.
2) The second conclusion is that authors have to be aware that reflective questions and
exercises have to be designed differently when an e-reader or computer is used. The relation
between questions, exercises and the running text must be integrated. Turning pages is
something completely different from scrolling down screen text. The place and use of all
auxiliary texts via mouse-overs, or hyperlinking, demands intense research in order to allow a
24
new screen-based order in which the integrity and integration of the main text and other texts
is guaranteed and where switching between the main text and auxiliary texts becomes fluent.
As mentioned above, our investigations were carried out in the middle of a stream of
changing technology. The screen technology and software developments are still changing
continuously. In that way, the other bank of the stream is still ill-defined and receding.
Whilst in our first test, the speed of turning pages on an e-ink reader was indicated as an
obstacle, the latest versions of e-readers are many times faster. In addition, the screens of
tablets improve year by year, slowly closing the distance in readability between e-ink screens
and LCD screens.
In this paper, we also report the first tests in providing teaching material in different forms,
using print text versus differently structured electronic text as well as a mind map and an
auxiliary website.
Though we can be sure that the hardware and software in question will considerably improve
over the next years, we can conclude from these early tests that, in the presentation of
educational material, form and content are firmly connected. The more data or facts, the easier
an electronic model will work. The more reasoning, examples and digressions are needed, the
more clearly will written text with integrated multimedia components and test questions be
needed. This all indicates that electronic text-books and other electronic learning materials are
much more than a collection of snippets of text linked to a database of pictures, videos and
audio files. The new way of reading and learning, using electronic devices, allows for fast and
comprehensive delivery of materials but also induces new ways of composing, structuring and
mutually relating the various presentations of the underlying knowledge and instructions.
Further research is needed in longitudinal studies to find out to what degree habituation plays
a role, as well as in-depth tests with authors and students on how the print structure of a text-
book can be transposed to the more flexible electronic future.
7 Acknowledgements
We thank our colleagues Gerlof Donga, Jacob Molenaar, and Jos Vrolijk for their continuing
support during the course of the project. Alexa Brinkman participated as research student in
our first test. Her persistency was of great value. We thank Keith Jones for his many helpful
corrections. We also thank Rens Groeneveld of Notubiz/Docwolves, Bob van Duuren of
VanDuurenMedia, Marieke Gierveld of Noordhoff andTannet Remmelts of SDU for their
kind collaboration and interactions. The project received a grant of the SIA-RAAK foundation
under number 2009-14-3H, which is gratefully acknowledged.
25
8 References
Ackerman, R. & Goldsmith, M. (2011). Metacognitive regulation on text learning: on screen
versus on paper. Journal of Experimental Psychology: Applied. 17(1), 18-32.
Berg, S.A., Hoffmann, K. & Dawson, D. (2010). Not on the same page: Undergraduates’ s
Information Retrieval in Electronic and Print Books. The Journal of Academic Librarianship.
36(6), 518-525.
Brunet, D. P., Bates, M.L., Gallo III, J.R,, & Strother, E. A..(2011). Incoming Dental
Students’expectations and Acceptance of an Electronic Texbook program. Journal of Dental
Education. 75(5), 646-652.
Chong, P. F., Lim, Y-P. & Ling, S.W. (2008). E-book design preferences: A case study.
ITSim 2008. International Symposium on Information Technology, 2008,.1(1-8), 26-28 Aug.
2008
doi: 10.1109/ITSIM.2008.4631538
http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4631538&isnumber=4631524.
Assessed 7 Nov.2012.
Chong, P.F., Lim, Y-P. & Ling, S.W. (2009). On the design preferences for ebooks. IEEE
Technical review. 26( 3), 213-222.
Christianson, M. & Aucoin, M. (2005) Electronic or print books: Which are used. Library
Collections Acquisitions, & technical Services. 29, 71-81.
Cull, B. W. (2011). Reading revolutions: Online digital text and implications for reading in
academe First Monday, 16(6).
http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/rt/printerFriendly/3340/2985.
Assessed 7 Nov. 2012.
Daniel, D., B. & Woody, W., D. (2013). E-textbooks at what costs? Performance and use of
electronic v. print texts. Computers & Education 62(1), 18-23.
Dehaene, S. (2010). Reading in the brain. The new science of how we read. London: Penguin
Books.
Emerson, L. & MacKay, B.. (2011). A comparison between paper-based and online learning
in higher education. British Journal of Educational Technology. 42(5), 727-735.
Gibson, C. & Gibbs, F. (2012). An evaluation of second-generation ebook readers The
Electronic Library, 29(3), 303-319.
Grafton, A. (1999). The Humanist as Reader. In G. Cavallo & R. Chartier: A History of
Reading in the West (pp. 179-212). London: Polity.
Haas, C. (1996). Writing Technology. Studies on the materiality for literacy. Mahwah:
Lawrence Erlbaum Ass. Inc. Publishers. Reprinted by NY: Routledge 2009.
26
Jamali, H.R., Nicolas, D. & Rowlands, I. (2009). Scholarly e-books: the view of 16.000
academics: results from the JISC National E-Book Observatory. Aslib proceedings, 61(1) 33-
47.
Hernon, P., Hopper, R., Leach, M.R., Saunders, L.L. & Zhang, J.. (2007). E-book use by
students: Undergraduates in Economics, Literature, and Nursing. The journal of academic
librarianship. 13(1) 3-13.
Hillesund, T. (2010). Digital reading spaces: How expert readers handle books, the web and
electronic paper. First Monday, 15(4-5).
http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2762/2504
Assessed 19 November 2012.
Kang, Y-Y, Wang, M-J. J. & Lin, R. (2009). Usability evaluation of E-books. Display 30, 49-
52
Kircz, J. G. (1998). Modularity: the next form of scientific information presentation? Journal
of Documentation. 54(2) 210-235.
http://www.kra.nl/Website/Artikelen/Jdoc98.htm
Kircz, J.G. & Harmsze, F.A.P. (2000). Modular scenarios in the electronic age. In: P. van der
Vet en P. de Bra (eds.) CS-Report 00-20. Proceedings Conferentie Informatiewetenschap
2000. De Doelen Utrecht (sic), 5 April 2000.31-43. :
http://www.kra.nl/Website/Artikelen/mod2k.html
Kircz, J. (2012). E-readers are for Reading: An Informal Consideration of Reading's Next
Steps. Scholarly and Research Communication 3(2).
http://www.src-online.ca/index.php/src/article/view/74/84. Accessed 19 November 2012.
Kircz, J. & Den Boef, A. H. (2011). Writing differently in the digital era. Hamlet in:. J.G.
Kircz and A. van der Weel (eds.) Hyperborg. Proceedings The Unbound Book Conference,
Amsterdam 2011.
In production 2013.
A final draft version is available at:
http://www.kra.nl/Website/Artikelen/Schoon-%20KirczDenBoef-29-8-2012.pdf
Letchumanan, M & and Tarmzi, R.A. (2011). E-book utilization among mathematics students
of Universiti Putra Malaysia (UPM). Library Hi Tech 29(1), 109-121
Mangen, A. (2008). Hypertext fiction reading: haptics and immersion. Journal of Research in
reading, 41(4), 433-453.
Mangen, A. (2013). The digitisation of narrative reading: Theoretical considerations and
empirical evidence. In: J.G. Kircz and A. van der Weel (eds.) Proceedings The Unbound
Book Conference, Amsterdam 2011.
In production 2013
27
Moen, D. (2000). Newspaper Layout & Design: A Team Approach by Daryl R. Moen Iowa
State University Press. Fourth edition.
http://web.missouri.edu/~moend/design/books/nld5.htm
Assessed 19 November 2012
Nicholas, D., Rowlands, I., Clark, D., Huntington, P., Jamali, H.R. & Olló C.(2008) UK
scholarly e-book usage: a landmark survey. Aslib Proceedings Vol 60(4) 311-334.
Noyes, J. & Garland, K. (2006). Explaining students’ attitudes toward books and computers.
Computers in Human Behavior 22(3), 351-363
Peters, A. & Westerveen, F. (2010). Geowijzer. Groningen. Noordhoff Uitgevers B.V.
Rainie, L. Zickuhr, K., Purcell, K., Madden, M., Brenner, J. (2012). The rise of E-reading.
Pew Reserach Center’s Internet & American Life Project.
http://libraries.pewinternet.org/2012/04/04/the-rise-of-e-reading/
Riel, C.B. M. van. (2010). Identiteit en Imago. Den Haag, SDU uitgevers.
Rowlands, I., Nicholas, D., Jamali, H.R. & Huntington, P. (2007). What do faculty and
students really think about e-books. Aslib Proceedings. 59 (6), 489-511.
Schuurmans, U. (2008). Handboek Digitale Marketing en Communicatie. Culemborg, Van
Duuren Media.
Shelburne. W. A. (2009) E-book usage in an academic library: User attitudes and behaviors.
Library Collections, Acquisitions, & technical Services. 33(2-3), 59-72
Wolf, M. (2007). Proust and the Squid. The story and science of the reading Brain. NY.
Harper Collins Publishers.
Woody, W. D., Daniel, D.B. & Baker, C. A. (2010). E-books or textbooks: Students prefer
textbooks. Computers & Education 55(3), 945-948
Zacharis, N. Z. (2011). The effect of learning style on preference for web-based courses and
learning outcomes. British Journal of Educational Technology. 42(5) 790-800.
Recommended