Feliciati User Studies Dill 2010

8/8/2019 Feliciati User Studies Dill 2010

1/89

International Masterin Digital Library Learning (Dill)

Parma, 2010 September 23rd

Digital Libraries User Studies:importance, methodologies

and case studies

Pierluigi Feliciati

University of Macerata (IT) - Department of cultural heritage

[email protected]
mailto:[email protected]:[email protected]


2/89

23/09/2010 Pierluigi Feliciati - DILL 2

Quoting Plato...

MENO: And how will you enquire, Socrates, into that which you donot know? What will you put forth as the subject of enquiry? And ifyou find what you want, how will you ever know that this is the thing

which you did not know?

SOCRATES: I know, Meno, what you mean; but just see what a

tiresome dispute you are introducing. You argue that a man cannotenquire either about that which he knows, or about that which he

does not know; for if

he knows, he has no need to enquire; and if not, he cannot; for hedoes not know the very subject about which he is to enquire

Plato, Meno, or Virtue, IV century b.C


3/89


Basic references

This presentation comes from:

my activities inside the MINERVA WG5, in particulartogether with my friend Maria Teresa Natale (OTEBAC);

the collaboration and friendship with Milena Dobreva(University of Glasgow), and our joint (in print) paper for

VAST 2010 Conference, User Studies in Digital Libraries:State of the Art

Cultural information systems and DL's users (especiallymy University of Macerata students) feedback

[email protected]
mailto:[email protected]:[email protected]


4/89


Scheduled program (1, 2, 3)

User Studies in the DL domain: issues, virtualmodels and empirical research

Standards, Models, Methodologies: users in DL

From Models to Practical Guidance in CH sector:the MINERVA Handbook on Cultural WebInteraction

Methods and case studies (AIB-WEB, DiSCmap,The European Library, Europeana, CIBER...)


5/89


Scheduled program (4)

Interactive session

We will share a focus group, where I will playthe moderator and you the panel users. Our

study subject will be Europeana portal.This session would be useful for: knowing this huge portal

experiencing what does it mean a focusgroup and how it works


6/89


user studiesin the DL domain

We notice an increased interest to user studies inthe digital library (DL) domain in the recent years.

The number of publications on such studies isgrowing; and user modelling and evaluation are

standard tasks within most project developing orenhancing DLs.

At the same time, research on DL users, their

expectations, needs and perception has notyet grown enough to offer consistent models andrecommendations.


7/89


user studies in the DL domain

the general knowledge of user behaviour is a mixture ofcommon sense, presumptions and prejudices (Sundqvist07)

In the case of digital library researchers, the focus of

research is often on technical issues (e.g., informationretrieval methods, software architecture, etc.) rather thanon user-centered issues. (Koo et al. 09)

The Institute of Museum and Library Services (IMLS)

reported that:

The most frequently-used needs assessment methods donot directly involve the users.


8/89



If we can not find user-related guidance in thestudies of DL for specific user communities, wecould consult the areas of information behaviourand user experience (UX) studies to find an helpful

starting point. These areas currently do not offerdefinitive guidance:

even now, while UX is well discussed on conferences andsymposia, it only rarely enters the relevant academic

journals. We believe that the lack of empirical research isone of the reasons for this. The absence of empiricalresearch whether qualitative or quantitative impedestheoretical advancement(Hassenzahl and Tractinsky 06)


9/89



What types of research help us to understand theuser point of view in the DL domain?

Most frequently, they address evaluation of existingDLs, i.e. studies on how users react to an existing

service. Normally they give a strong emphasis onusability.

The studies done before a DL is developed or while itis in development are less common and nromally they

are lunched just for big projects (Euroepana, BibliothecaAlexandrina).

There are also publications which present just methods,mostly on webmetrics.


10/89



There are several characteristics of such studies: In most cases they evaluate existing Dls, whileDLs

in development are addressed less frequently. But atthe same time knowing more about the users before a

DL is developed is essential for delivery of content andservices which would match better those for whom theDLs are created.

Many studies are stand-alone; they address a

specific DL or a small group of DLs; it is much rarer tofind studies which contextualise the specific DL withinthe larger picture of the DL domain.


11/89



There are several characteristics of such studies: The studies focus mostly on specific aspects such as

usability; more work needs to be done to contextualisebetter specific DL user studies and information

behaviour as well as user experience studies. In many cases the studies address a limited set of

user communities. This is acceptable in the case of asingle DL although the digital world does not put barriers

to any type of user entering the DL. But now we work inthe time of federated resources, aggregation, buildingvery large DLs. In this domain the question how theaggregation of resources is changing the user base isnot studied in detail.


12/89



We take as a starting point that from delivery ofinformation about cultural object, the culturalsubjectswill move towards facilitation ofcultural experiences.

This requires a new level of integration ofdigital collections and digital objects of varioustypes, including virtual models.

When different types of content areaggregated, they create new problems ofproviding reliable services in conditions wherehomogeneity is impossibleper se.


13/89



As regards the DLs, we can consider it also as asoftware product, so its quality could be definedthe capability to enable specified users toachieve specified goals with effectiveness,

productivity, safety and satisfaction in specifiedcontexts of use. (ISO/IEC 9126-1:2001)

Thus, the quality of a software product lies not in the absence offaults, richness of functions, or technical innovation, but in the extentof which products are usable and accessible according to the needsof the users in their context of use.

Whatever views we have on what a DL is, it alwaysexists to meet the needs of users.


14/89



There are at least 3 core elements in a DL: thecontent it makes available, the services itprovides, and the users. Understanding theneeds of the users, whenever these needs are

explicit or not, is core to the delivery of a service. In many cases DLs are addressing specific

types of content; but one major currenttendency is the aggregation of content fromvarious sources as it is done in Europeana orthe World Digital Library (portals).


15/89



For the users, a portal is surely only useful if itmeets a real need that users have, and in a way

with which they are comfortable. As such, the

portal needs to do more than any of the currentoffers being presented. To facilitate this, there isneed for continued work on ensuringinteroperability of systems (Miller 01)


16/89



Anyway, regardless of the size and the profile ofa DL, it always helps to understand betterusers and their needs.

Even in the case of DLs addressing specificcontent where the user community might seemclearly defined and its needs and preferences

obvious, the very nature of the digital worldmeans unexpected users and surprising uses.


17/89



What are the central research questions in thisdomain?

1.The continuum of user needs > user expectations> user perceptions had not been studied

systematically yet.Knowing more about the connection between the usersatisfaction and the consciously expressed needscombined with expectations is essential to developresources which sustain a stable user community.

The understanding of this continuum would help also to design DLswhich succeed to attract the user attention in the very first secondswhen a novice user decides if this resource is useful or not.


18/89




2.The modes of interaction (browsing and searching simple or advanced, using boolean operators or not,

etc.) can be further studied.Knowing more about interaction characteristics andbehaviours can help to provide the best possible supportaccording to the mode(s) followed by the user.


19/89




3. A framework of Methods for user studies must bedrawn.

There is a wide range of users study methods, someinvolving directly users (e.g. focus groups, media labs,questionnaires), as well as methods based on the use ofdata gathered in the process of DL use, e.g. user logs.

The selection of the most suitable method according to thequestion addressed is not a trivial task and the provision ofmore guidance is essential.


20/89




4. An agreement on Measures (objective andsubjective).

There are different ways to measure user performance(s).There is still no consensus on measures which can beused for DL user studies. Having an agreed set couldfacilitate future comparative studies and benchmarking.


21/89


Standards, Models, Methodologies

Presently different Standards, Models,Methodologies exist, changing their interpretationand study of users. We'll start from the marketing approach because it is

rarely used in the DL context but it is relevant to userstudies

Then, the ISO standard for Human-centred design forinteractive systems

The DELOS Digital Library Reference Model The 5S model (Scenarios and Societies)

the 5M model (Modelling the contexts)

d d d l h d l i


22/89



A common approach to users is the marketingperspective, which aims to classify them on the basisof their possible quality as consumers.

The users are not treated individually, but into consumermarket segments, groups with a similar perception of arequirement. That brings to demonstrate an homogeneousbehaviour in solving the problem represented by therequirement. The criteria for a successful classification are:

homogeneity within the segment and heterogeneity between segments;

measurability; identifiability;

accessibility of information;

enough quantity to be profitable.

S d d M d l M h d l i


23/89



The variables used for marketing targetsegmentation include geographic, demographic,psychographic and behavioural variables. Whenenough information is combined to create a clear

picture of a typical member of a segment, this isreferred to as aprofile (or type).

Many study methods of web users are developed underthe push of industry, in particular for the definition of

techniques and metrics for audience measurement, i.e. themethods used for calculating how many people form part ofan audience, i.e. a group of people reached by a message.

St d d M d l M th d l i


24/89



Audience metrics too is a discipline whichoriginated within the sphere of advertising andmarketing. In the web context, its major role is thatof providing qualitative and quantitative

indicators for the analysis of web applicationeffectiveness.

Audience, expressed into numbers are defined as ratings:although traditional media are by now standardized (for

example, average minutes for TV) the new medium is nolonger characterized by a simple exposure model, but isan interactive space of action, where the user is not justlooking but is encouraged to interact by providing a form ofreply to the medium.



25/89



Another recent method developed in the context ofmarketing studies is the so-called neuromarketing,

which adapts neurological analysis techniques toinvestigate target users/consumers behaviours.

They combine EEG (Electric Encephalography), fMRI(functional Magnetic Resonance Imaging), eye-trackingand non-verbal communication analysis to gather data onusers behaviour performing some pre-defined tasks.



26/89



The ISO 9241-210:2010(E) standard Ergonomics ofhumansystem interaction -Human-centred designfor interactive systems introduces the basicconcepts within the ergonomics of human-system

interactionThis standard is very wide and complex. The 210 sectionprovides requirements and recommendations for human-centred design principles and activities throughout the lifecycle of computer-based interactive systems.

The current DLs are interactive systems so it is usefulthe professional community to be informed on thisstandard.

Standards Models Methodologies


27/89



The standard uses a number of definitions.

User is the person who interacts with theproduct (definition taken from ISO 9241-11:1998).The definition of a user experience is:

Person's perceptions and responses resulting from the useand/or anticipated use of a product, system or service. Itincludes all the users' emotions, beliefs, preferences,perceptions, physical and psychological responses,behaviours and accomplishments that occur before, duringand after use.

So, user experience is not only about being informed butalso about her/his emotional response and involvement



28/89



The standard stretches also the factors whichinfluence user experience:

User experience is a consequence of brand image,presentation, functionality, system performance, interactivebehaviour and assistive capabilities of the interactivesystem, the user's internal and physical state resulting fromprior experiences, attitudes, skills and personality, and thecontext of use.

User studies often concentrate just on the success of

finding information in a DL, but it is necessary to see it as apart of a complex system including previous experience,interface characteristics and user attitudes.



29/89



The standard also present the interdependence ofhuman-centred design activities as a lifecycle

which starts with a design planning, and continueswith several stages which are repeated until the

evaluation against requirements is satisfactory.These repeating stages include: understanding and specifying of the context of use

specifying the user requirements

producing design solutions evaluating the designs against the requirements



30/89



Interdependence of human-centred design activities.(from ISO 9241-1998)

In many cases of

DL design, theuser requirementsare not studiedand reflected insuch a sequence

of steps andbesides really bigprojects, thedesign of the DL isnot an iterative

process but israther done once.



31/89



The well known DELOS Digital LibraryReference Model DLRM (Candela et al. 08)individuates six domainswithin the digital libraryuniverse: Content, Functionality, User,

Architecture, Policy and Quality



32/89



DLRM introduces several concepts from thesedomains, together with the possible relationshipsbetween them.

For example, one could start with the User area

and see how it related to Content orFunctionality, or what Quality parameters areessential from the User point of view.

The work on DLRM currently continues within the DL.org

project (http://www.dlorg.eu/) and maybe it will take intoaccount new developments such as ISO 9241:2010 todevelop better the system-centred and the human-centreduser design issues.

http://www.dlorg.eu/http://www.dlorg.eu/


33/89



According to DLRM every domain in the DL universe is

decomposed into a finer level of granularity and its centralconcept is defined.

In the User domain the central concept is Actor.

The end-user appears as one of the possible roles playedby Actors, alongside with DL designer, DL SystemAdministrator, DL Application Developer, distinguishedfrom the actor profiles and functions.

The end users are further refined to Content creator,

Content consumer and Librarian.

Actor, we have to underline it, is a resource



34/89



The model does not

provide anextensive set ofroles: testing andevaluation are notincluded in the

roles, even if theyare essential in theDL life-cycle.

In addition, besideshuman actors,

there are also bots,intelligent agentsand other machineactors.



35/89



The 5S modelwas proposed (Gonalves et al. 04)

as a formal framework which explores the contentand functionalities of DLs as mathematicalobjects.

The 5S defined are Streams, Structures, Spaces,Scenarios, and Societies.

As regards our theme, this model does not look at asingle user (although it could be modelled as a

Society with a single member); another arearelated to the use of DLs is Scenarios.



36/89



A representation

of user needscould be donecombining twopoints of view:

what Scenarios

are most typicalfor the users,

what properties ofa Society specifya particular

community ofusers (like thedesignedcommunityofOAIS model)



37/89



A further development of the 5S model ideas

focused on DL evaluation was provided in(Gonalves et al. 07) which presents some qualitydimensions:

accessibility, accuracy, completeness, composability,conformance, consistency, effectiveness, efficiency,extensibility, pertinence, preservability, relevance,reliability, reusability, significance, similarity, andtimeliness.

This paper also presents measurable characteristics: responsetime (connected to efficiency), cost of migration (connected topreservability), and number of service failures (connected toreliability).



38/89



Another model, 5M, is suggested by Daranyi etal. 10

Starting from the current complexity of DLs andtheir extensive use in an international

environment, the authors argue that theconcepts which characterise the currenttechnological and international dimensions ofDLs are multiculturality, multilinguality and

multimodality.This model should not be seen as yet another model butas a bridge between DELOS DLRM and 5S, taking intoaccount not only digital objects, but also context.



39/89



The DL community [will be motivated] to add a highly

integrated 5M layer to the 5S formal model of DL ()where 5M stands for DL collections with Multicultural,Multilingual, Multimodal content; indexed, categorized andretrieved byMultivariate algorithms; and harnessing theirevolving semantics by the Modelling of collections and

users.

The practical implementation of this model will continue toevolve, seeingwhat is thecontent of the digital librariesnow;what models can best accommodate the complexity

of the DLs and how the context of the digital objects canbe represented and used. We can say that thissummarise some of the most serious challenges in the DLdomain.



40/89



User Models and Evaluation of DL

Another aspect of the models of DL + users is how they arebeing used in the evaluation of DLs, another extensivedomain (Fuhr et al. 07). The various types of evaluationhave different relevance for user studies in DLs.

Schematising: on one hand users can take part in the evaluation;

knowing more about the users helps to plan evaluationbetter;

evaluation can also help us to learn more not onlyabout the DL itself but about its users.



41/89




A good framework which contextualises the metrics which can beuser in the evaluation process and also addresses the user, is theInformation Triptych Framework(Tsakonas et al. 06).

This schemasummarizes

the relationsbetweenUser, Systemand Contentand makesclear what

exactly canbe measuredif we look atone of thesedichotomicpairs



42/89


, , g


The Information Triptych Frameworkshows a top-down point of view on the DL evaluation. If data hadbeen collected, it can very easily be seen where

they fit in the picture and what else could beevaluated.

This model look at the user studies as somethingwhich happens inside the DL environment, so is

specifically targeted just to the situationwhen auser communicates with an existing DL.

From Models to Practical Guidance


43/89


An interesting example of the work on adoptingguiding principles and providing guidance to thememory institutions in their delivery of culturalcontent online had been developed within the

European MINERVA eC thematic network. Since the beginning of the quality actvities within

MINERVA, it was statued that user interactionand satisfaction is a core issue within the

quality domain of cultural web applications.



44/89


Some key messages in MINERVA 2003-07

documents on cultural web quality were: Quality must be planned into a web application from

the start of the project The user is critical involve him at every stage Relationships with other resources (contents,

agents, events) must be considered, both present(interoperability) and future (long term preservation)

But some questions posed were left without an answer:

What do users want? How do users behave? How can weunderstand their use of our web applications? Do effective methodsexist to ask users about their expectations (before) and their degreeof satisfaction (after)?



45/89


Handbook on cultural web user interaction edited by the MINERVA EC Working Group Quality,

Accessibility and Usability (editing P. Feliciati M.T.Natale)

European Editorial Committee: Monika Hagedorn Saupe(coordinator), Germany, Hanna Arpiainen, Finland, Pierre-Yves Desaive,Belgium, Nathanael Dupr Latour, Czech Republic, Axel Ermert, Germany,Pierluigi Feliciati, Italy, Gabriele Freschista, Austria, Susan Hazan, Israel,Karine Lasaracina, Belgium, Maria Teresa Natale , Italy, Tzanetos Pomonis,Greece, Maria Sliwinska, Poland, Hans Van der Linden, Belgium, FrancZakrajsek, Slovenja

Freely available onwww.minervaeurope.org website

Italian and russian versions available

http://www.minervaeurope.org/http://www.minervaeurope.org/


46/89


MINERVA Handbook contents

1.Users and cultural contents on the web: state ofthe art

2.Finding ones way

3.Practical tools

4.The importance of using metadata

Annexes



47/89

23/09/2010 Pierluigi Feliciati - DILL47

1. Users and cultural contents on the web: stateof the art

1.Users and services in cultural web applications:websites, digital libraries and portals (a synthetic andup-to-date panorama of users and cultural content

providers on the web)

2.Current trends in web services: Web 2.0-3.0 (the roleof cultural institutions in the current and futureinformation society and the changing face of the

institutions as they present, and represent theinstitution online)

With 24 case studies and good practices commented.



48/89


2. Finding one's way

1.Cultural entity types: a cultural entity can be a person,an organisation, an institution or a group of differententities combined to deliver a cultural product, whichmay also deploy web technologies in achieving its aims.

2.Web application types: a schematic, up-to-date,presentation of the main types of web applications thatcultural subjects can promote as tools for achieving theirmission in whole or in part.

3.Web applications life cycle: some phases ofapplication's life-cycle are particularly critical to gatherinformation about user expectations or to focus on theactivities to meet these expectations.



49/89


2.3 Web applications life cycle

(Priority measures from blank to **)

1.Website planning **

2.Website design **

3.Content selection **4.Digitisation process and collection of digital contents *

5.Storage of the digital masters *

6.Metadata creation and capture **

7.Website implementation and test of the prototype **8.Online publication

9.Ongoing maintenance **



50/89



4. Users and usage

This section aims to provide tools of reflection for designingapplications that satisfy citizens expectations and is divided intothree parts:

1. the web user: state of the art and trends in definitions proposes variousapproaches with respect to the users, according to the most commonapproaches taken to the design and creation of web applications

2. the web user: who is he? tries to classify web users based on the role thatthey play (within a cultural institute or as end users of information andservices), or that it seems to the web designers that they want or can have(abstracting them into types, typical behaviour, profiles and scenarios).

3. Systems adapting their behaviour to users. Often it is difficult to find theright information at the right time and at the right level of detail. In order to finda solution to this problem, researchers are developing systems which adapttheir behaviour to the interests, task, and goals of single users or groups ofusers.



51/89


2. Finding one's way4.1 the web user: state of the art and trends in definitions

1. The user for ICT professionals

2. The user in marketing

3. The user according to MINERVA

4. The user according to usability gurus (user centred design)

5. The user in current trends (prosumers, transceivers...)

6. The automatic user (search engines, web services, extraction and

reprocessing of XML feeds, mash-ups between functions,harvesting of metadata and data...)



52/89


2. Finding one's way4.2 The web user who is he?

1. The in-home user

2. The simulated user

1.User types and roles

2.Personas

3. Use simulation

1.Use cases

2.Scenarios

4. The final user point of view: user stories

Users and usage


53/89


Create

Publish

Ask

DiscussLearn

Enjoy

View

Acquire

Identify

and

AcquireDescribe

Preserve

Provide

Share

Enable

Teach

DesignAssess

Validate

Interpret

Affirm

Disseminate

Communicate

USERS IN CURRENT TRENDS



54/89


2. Finding one's way4.3 Systems adapting their behaviour to users

1. Customisation

The ability of the user to modify the interface to meet individualrequirements. The user can configure an interface and create a

profile manually, adding and removing elements in the profile.2. Personalisation

In this case, modifications in access to content and page structureare performed automatically by the system, using information on theuser taken by the user profile, provided explicitly by the user(registration forms, questionnaires and reviewing),or implicitly(recording the surfing behaviour and preferences, through cookiesand web server log files which track user routes through the site).



55/89



5. Interactive and user-side servicesThis section aims to present ICT applications that can be adapted forthe end user, providing interfaces and selections of contents and ofpersonalized services. The single services are organised into somemacro-categories, to help readers to make their choice:

1. Interactive communication services mediated by the informationprovider

2. Interactive learning services

3. Virtual interactive tours

4. Commercial interactive services5. Interactive forms

6. User-side services

7. MUVEs



56/89



2.6 Audience measurement on the InternetThe aim of this section is to review the techniques and metrics usedfor audience measurement in the Web. By audience measurementwe mean the methods used for calculating how many people formpart of an audience, that is a group of people reached by a message.

One of the fundamental issues is that regarding the process used fordata gathering:

1. in census data modality, the measurement is carried out on thetotal reference population (web analytics is to the study of the

behaviour of network users);2. in sample (or user-based) data modality, the measurement is

done on samples of the population (The sample chosen is usuallydescribed as a panel,



57/89


3 Pratical tools

3.1 A self-evaluation questionnaire for planning a user-centred web application

Addressed to those cultural subjects that are about to develop a newweb application (or want to update one already on-line) and whose

objective is to seriously evaluate users expectations, theirsatisfaction and the possibility of foreseeing advanced forms ofinteraction.

3.2 Websites and portals feedback form

Standardized interview model to be distributed to users of web sitesand cultural portals. It can be used as a reference for theconstruction of a personalized questionnaire, that responds to therequirements of ones own web application.



58/89


4 The importance of using metadataThis chapter presupposes a basic ICT knowledge, but isnevertheless proposed to cultural heritage professionals, inconsideration of the increasing importance of metadata issues incultural web world. The practical application of this chapters contentsis devoted to ICT professionals.

4.1 Why use metadata for describing websites?

4.2 Benefits of using metadata

4.3 The Dublin Core standard4.4 Another way to expose resources: Syndication & RSS

4.5 Towards semantic integration

Case studies


59/89


Here are some examples to see how a range of

different user study/modelling methods have beenapplied in practice. To highlight the different studygoals, approaches and outcomes, They arepresented in this subsections:

What is the method in general?

What were the aims of the particular study?

How were the methods applied?

What were the difficulties/lessons learnt? What were the outcomes?

Case studies1


60/89


Questionnaires

The most extensively used method of investigation of mediaaudiences, including websites and portals, is the standardisedquestionnaire.

It is done by asking preliminary structured questions to all

users or to a group of selected individuals. This method entailsthe direct involvement of the subjects to be analysed.

The aim is to investigate their preferences, habits andbehaviour, in order to verify effectiveness in terms of usersatisfaction (with choices made and to study behaviour) duringnetwork navigation.

The choice of those to be interviewed can be casual or notcasual, according to whether or not the choice of those to beinterviewed should be probabilistic or not.

1

Case studies1


61/89


Questionnaires

The casual methods for choosing a panel: entertainment survey or polls unrestricted self-selected survey (invitations to participate in a

survey) volunteer opt-in panel: self-proposed volunteers recruited through

sites and portals

The non-casual methods for choosing a panel are: interviews intercepted among a sites navigators (intercept

survey): questionnaires completed by the visitor and randomly

selected panels based on lists of known names (list-based sample) pre-recruited panels: recruitment of users that are not self-chosen

or volunteers

1

Case studies1


62/89


Questionnaires

Advantages limited costs rapid planning and completion times capacity to reach users regardless of geography and other

variables possibility of using multimedia content (audio and video) control of the processes in real time.

Main issues Truthfulness of the statements (art. regarding user types,

complicated on the Web by the increase of virtual beings) Completeness of answers (it's smart to offer a gadget for

users completing all the questions)

1

Case studies1


63/89


AIB-WEB study

To illustrate the method of questionnaires, it's relevant arecent study of AIB-Associazione Italiana Bibliotechewebsite, aiming to gather feedback which would be usedin the process of the redesign of a professional community

portal.The AIB-Associazione Italiana Biblioteche is the professionalassociation of Italian librarians and libraries, founded in 1930 andpartner of IFLA.

AIB-WEB, AIB web site (http://www.aib.it), born on 1995(on 1997 under this domain) has actually 15.000+ pages,created and maintained by a distributed editorial staff(120+ people).

1

Case studies1
http://www.aib.it/http://www.aib.it/


64/89


AIB-WEB study

AIB-WEB has always been focused on the content, inthe name of accessibility and simplicity, sacrificing someelements such as graphics, presentation, and inclusion ofmultimedia content.

An admirable severity in the management of a universal accesspolicy was not accompanied by a gradual updating of the rich andcomplex web site to the obvious web environment changes (bothtechnical and in users interaction).

When AIB executive board statued the restructuring ofAIB-WEB, the editorial board has decided to launch ausers satisfaction survey, to gather a base for theremodelling of the site and its future enrichment.

Case studies1


65/89


AIB-WEB study

For this survey the method of unrestricted self-selectedsurvey(by questionnaire) was chosen: the sample wasopen and the survey, with the duration of 2 months, waspublicized through calls via Web portals, popular websites,

discussion lists, etc. The questionnaire was administered via theWeb platform SurveyMonkey.

The questionnaire, based mostly on the MINERVA model,consisted of 37 questions, organized in 6 sections and

including a mixture of: open/closed questions, questionswith predefined answers, questions with free textanswers,multiple choice answers and/or votes.

Case studies1
http://www.surveymonkey.com/http://www.surveymonkey.com/


66/89


AIB-WEB study

A high number of users (645) answered, and 74,7% filledin the entire questionnaire.

This excellent response, both in quality and quantity showed that thetarget community of the DL appreciated the initiative and needed to

be involved in website life, providing a huge number of personalopinions and suggestions choosing often to make the most of thefree text option. This tendency meant extra time for analysing freeanswers and to disentangle some contradictions between closed-choice and free answers.

This huge quantity of qualitative data gave the evidencefor some common needs and opinions, even if it was noteasy to extract clear recommendations.

Case studies2


67/89


DiSCmap project

DiSCmap (Digitisation of Special Collections: mapping,assessment, prioritisation) was commissioned by JISC(Joint Information Systems Committee) in 2008 to theCentre for Digital Library Research of the University of

Strathclyde, Glasgow and has been completed betweenSeptember 2008 and May 2009.

The project had as its primary goal to study the user needs indigitised special collections in the higher education institutionsin the UK, traditionallyled by supply rather than demand.

The project, based on a set of inter-connected tasks, aimed atassessing the current landscape of digitisation from the point of viewof the needs of the researcher and teachers within UK highereducation institutions.

Case studies2


68/89


DiSCmap project

It included several components: compiling a list of collections nominated for digitisation by users. constructing a framework of criteria for user-driven prioritisation of

digitisation.

illustrating how the framework can be used in prioritisingcollections from digitisation

To achieve these goals, the project used a combination of methodsto gather data: web questionnaires, focus groups, interviews as wellas social networking. Theweb questionnairewas used for two

basic purposes:1. As a means to gather data on nominated collections for digitisation

2. As a tool for compiling to build the framework of criteria for user-driven digitisation.

Case studies2


69/89


DiSCmap project

The questionnaires included free text explanation why acertain collection is nominated for digitisation; this set ofexplanations was used to produce an initial set of user-driven criteriawhich had been later refined through focus

groups, interviews and contextualised with existingprioritisation studies and frameworks. Social networking(through a blog and a Facebook group) did not proveespecially helpful in this study.

In the case of DiSCmap the web questionnaire was developedspecifically for the projectnot using a ready-made platform. Since thedata collected (free text explanations) covered nominations ofcollections, they had to be revised (deduplicated and checked formisspellings or alternative collection titles).

Case studies3


70/89


Expert studies

The expert evaluation method is used when a group ofexperts studies in detail an existing DL and providesrecommendation on it.

The method of analysis could involve preliminary agreed

structure, or can be based on a selected set ofheuristics.

One good example was that of The European Librarypresented by Klas et al. This study aimed to evaluate

The European Library and was designed around aframework called SDWS (Simulated Domain and Work-Task Scenario).

Case studies3
http://www.theeuropeanlibrary.org/portal/index.htmlhttp://www.theeuropeanlibrary.org/portal/index.html


71/89


Expert studies - The European Library

3 scenarios were identified, all recreating real-lifesituations. Each of them were described on 2 levels (oneaddressing the domain of work and the second the topicof the query and a description of the search task).

The three scenarios addressed history (with two specific tasks -research on the first four crusades and on the pilgrim paths inEurope), and prehistoric religions in Europe (searching for socalled Black Madonnas).

The evaluation was performed by 6 highly qualified evaluators,

whose basic issues identified were related to usability, search andresult functionality, as well as language. This study method wasquite productive, nearly 40 observations on various minor issueshad been made, useful for future redesigning of the DL.

Case studies3


72/89


Expert studies

The expert evaluation puts an emphasis on a thoroughpreparation of the evaluation methodology and requiresqualified experts to take part in it.

As a qualitative method it provides guidance and of

course the points of view of the evaluation experts donotrepresent the feedback which could be received fromthe user community.

But it is especially helpful for Dls under development

because at this stage it could provide precious insightswhat development trends are expected to attract andsatisfy the user.

Case studies4


73/89


Focus Groups and Media Labs

Focus groups are a typical example of study whichinvolves directly users and tries to gather richqualitative feedback involving the participants in a jointdiscussion.

In the digital library domain such studies normally involveeither a demonstration made to the group, or personalhand-on experience (tasks).

Here we will use as an example a recent User and

Functionality Testingwhich was made for Europeana.eu(Dobvreva et al. 10) .

Case studies4


74/89


Europeana users study

Europeana (http://www.europeana.eu/portal/) is a singleaccess point for digitised cultural heritage materialsprovided by various European libraries, museums,archives, galleries, audiovisual collections and other

memory institutions. The DL brings together anunprecedented amount of materials and thus plays the roleof the major specialised aggregator in the cultural heritagedomain for Europe.

Currently it provides access to over 4.6 million objects with the aim ofreaching a target of 10 million objects in 2010; more than 1000institutions are providers of the cultural content in Europeana andtheir number and geographic coverage are steadily growing.

Case studies4
http://www.europeana.eu/portal/http://www.europeana.eu/portal/


75/89



It is more than an aggregator since it also seeks to provideinnovative ways of searching and visualising the richcultural contents. This is being achieved through thegradual developments of the interface with improved data

organisation, search and browsing functionalities.Stakeholders developing Europeana are in regulardiscussion regarding how best to approach and serve itsusers.

User studies for Europeana should involve a combination of methodsincluding gathering expert opinion, organising focus groups,carrying out observations of users and conducting aweb survey(Europeana 09).

Case studies4


76/89



The Europeana User and Functionality Studywas coordinated by theCDLR (University of Strathclyde, Glasgow) jointly with the Universityof Macerata, Italy, and the Emotion Lab of Glasgow CaledonianUniversity.

This study aimed to address two specific usercommunities (young people and members of the generalpublic) across 4 countries through a series of focusgroups and media labs. Its purpose was to establish abetter understanding of users expectations as well as the

difficulties and stumbling blocks encountered while usingthe Europeana prototype.

Case studies4


77/89



Unlike other Europeana studies, this study: addressed participants responses combined with

analysis of the evidence of user actions during thecompletion of a standardised task in all the groups;

included homogenous groups and an equaldistribution of the number of participants in each of the 4countries.

The number of participants (total 89) is not sufficient to

make any statistically significant conclusions but the datagathered are of interest and can be compared with theoutcomes of other user studies.

Case studies4


78/89



One key feature of the study was the design of a protocol to bebeen applied throughout the focus groups. It included a briefintroduction to the study and Europeana, filling in questionnaires,and individual work on an assignment combining 8 searchscenarios which were comparable for the various groups. The

assignment was to prepare a virtual portrait of the participantscity with predefined slides which had to be filled in; the scenarioswere selected so that participants would make a variety of searches for texts, images, audio/video, looking for a very specific piece ofinformation.

The analysis of the feedback provided by the participants did notseem to be influenced by the number of available resources; thestudy made 24 suggestions for changes in Europeana organised in3 groups Content, Functionality/Usabilty and Navigation.

Case studies5


79/89


User Logs analysis

The analysis of user logs provides evidence on thebehaviour of the user without the element of directobservation. They could be used to follow the actions ofthe individual user, or could summarize characteristics of

large groups of users.The interpretation of network traffic makes it possible to extractrelevant indicators, for example, to: the number of accesses the navigation routes

the behaviour models the devices configuration, etc.

This analysis should includewide margins of approximation.

Case studies

U L l i

5


80/89


User Logs analysis

Advantages

special hardware or softwareinstallations are not required

both current and historicalinformation is always available

all events that happen in theserver are recorded

even low traffic sites can be

analysed.

Disadvantages

difficulties in standardizing the metrics

a significant proportion of the visitedresources escapes the logging process

(i.e. dynamic pages)

failure to measure traffic from client-sidememory or cache

difficulty of calculating the time reallyspent on a page

lack of social-demographic information

Case studies

U L l i CIBER t d

5


81/89


User Logs analysis CIBER study

An example of user logs study which provided usefulobservations on the information behaviour of young people(the so called Google Generation) had been done recentlyby CIBER (CIB 2008).

This study looked for information behaviour characteristicsbut the method can be used for research digital librariesand currently CIBER is involved in the logs analysis forEuropeana.

One detail of interest is that the study of user logs allowsseparating the human users from the machine users suchas bots and agents.

Case studies

P

6


82/89


Personae

The termpersona is used to describe archetypalrepresentations of typical users.

In the use of personae the aim is to describe in detailtypical users. To create such detailed descriptions

researchers could use other available personae, outcomesfrom other types of user studies, research on DL orinformation behaviour.

Once the descriptions are available they can be used for

other types of evaluation, e.g. expert evaluation, whenexperts will scrutinise the DL from the point of view of aparticular persona.

Case studies

P E

6


83/89


Personae - Europeana

This method was applied for defining a set of typicalpersonae for Europeana (Petersen 2010).

In addition to the methods for defining personae,Europeana organised a workshop used to brainstorm on

the differences between personae. To assure consistencyin approach, this work aimed to result in personae whichwould differ in the preferred search strategy (navigation orexploration) and in the level of digital literacy, ITknowledge, competences and preferred language.

Europeana currently has 4 primary and 3 auxiliarypersonae which will be used in future evaluation.

Case studies

Personae

6


84/89


Personae

At the first glance this method is not to study the users butto describe them.

However the personae can be used in evaluation of DLsand also the process of detailed description of users and

building an archetypical representation of a group of usersis requiring profound study of the user.

Possibly this method would be most useful for big DLprojects because it requires extensive expert involvement

and meticulous study of information behaviour resource.

Case studies

Additional methods: Eye tracking7


85/89


Additional methods: Eye tracking

Although this method is popular in the web search studiesit still has not been widely applied in the evaluation of DLuser interfaces. The study of Europeana included eyetracking media labs. It can be used to gather quantitativedata on the fixation and trajectories of the gaze.

The outcomescan bevisualised asheat maps (topresent areas

which attractmost of thesubjects gaze)

or by areas ofinterest (torepresent the timeof fixation) and

typicaltrajectories.

Case studies

Some conclusion on methods


86/89


Some conclusion on methods

The methods described above are quite diverse in terms oftime/money investment, number of experts needed,and methods for users engagement. It might seem quitedifficult to select the method which would be beneficial in a

specific situation and will best answer the particular goalsof the study.

In reality many studies use a combination of methods. Thedifferent methods contribute in a different way to creating amore holistic picture of the user.

But when a DL wants to make a user study which methodwould be most suitable? Next slide summarizes sometypical questions and recommended methods.

User methods summary

Question Method

H t fi d t A b ti i b d t dd t ti ll l it f f t


87/89


How to find out moreabout the users of a DL

which is currentlyplanned?

A web questionnaire can be used to address potentially large community of futureusers. Their responses could be helpful to define personae.

How to evaluate theusability of an existingDL?

This could be achieved through an expert group study. Direct user involvementcould be accomplisehed through focus groups.

How to conduct a studywhich will help to

identify measures forextending the usercommunity?

Web questionnaire and user logs could help to build a picture of the current usercommunity. For fine-tuning what measures could be used to expand the user base,

expert evaluation could be helpful. This might be connected to finding possiblereuse scenarios.

How to understandbetter what are thestumbling blocks for theusers of an existing DL?

Bet suited are methods with direct user involvement (focus groups or media labs);the logs analysis could also be of help.

How to compare my DLwith others?

Comparison in this domain is still difficult; expert evaluation seems the best optionhere.

How to make thenavigation on the website easier?

Use media labs to see what the individual users are doing and how and use logs todefine typical patterns of actions within the DL.

References (1)

Candela, L. et al.: The DELOS Digital Library Reference Model - Foundations forDigital Libraries Version 0 98 (2008)


88/89


Digital Libraries. Version 0.98. (2008),http://www.delos.info/files/pdf/ReferenceModel/DELOS_DLReferenceModel_0.98.pdf

CIB 08 - Information Behaviour of the Researcher of the Future. CIBER, (2008),http://www.jisc.ac.uk/media/documents/programmes/reppres/gg_final_keynote_11012008.pdf

Darnyi, S., Wittek, P., Dobreva, M.Adding a 5M Layer to the 5S Model of DigitalLibraries, To appear in: Digital Libraries for International Development workshop,Brisbane, Australia, 25 June 2010.

Dobreva, M., Birrell, D. et al. The DiSCmap project: Digitisation of Special

Collections: mapping, assessment, prioritisation. World Library and InformationCongress: 75th IFLA General Conference and Council, 23-27 August 2009, Milan,Italy, http://www.ifla.org/files/hq/papers/ifla75/92-dobreva-en.pdf

Dobreva M., Feliciati P. et al.. User and Functional Testing. Final report. (2010),http://version1.europeana.eu/web/europeana-project/documents

Dobreva M., Feliciati P., User Studies in Digital Libraries: State of the Art, to appear

in The 11th International Symposium on Virtual Reality, Archaeology and CulturalHeritage VAST (2010)

Europeana Online Visitor Survey. Research report, IRN research. 2009,http://www.edlfoundation.eu/c/document_library/get_file?uuid=e165f7f8-981a-436b-8179-d27ec952b8aa&groupId=10602

Feliciati P., Natale M.T.(ed.), Handbook on Cultural Web Interaction, MINERVAproject, 2008, http://www.minervaeurope.org/publications/handbookwebusers.htm

References (2)

Fuhr N., Tsakonas, G., Aalberg T., Agosti M., et al. Evaluation of digitallib i I t ti l J l Di it l Lib i 8 (1) 21 38 (2007)
http://www.delos.info/files/pdf/ReferenceModel/DELOS_DLReferenceModel_0.98.pdfhttp://www.jisc.ac.uk/media/documents/programmes/reppres/gg_final_keynote_11012008.pdfhttp://www.ifla.org/files/hq/papers/ifla75/92-dobreva-en.pdfhttp://version1.europeana.eu/web/europeana-project/documentshttp://www.edlfoundation.eu/c/document_library/get_file?uuid=e165f7f8-981a-436b-8179-d27ec952b8aa&groupId=10602http://www.minervaeurope.org/publications/handbookwebusers.htmhttp://www.minervaeurope.org/publications/handbookwebusers.htmhttp://www.edlfoundation.eu/c/document_library/get_file?uuid=e165f7f8-981a-436b-8179-d27ec952b8aa&groupId=10602http://version1.europeana.eu/web/europeana-project/documentshttp://www.ifla.org/files/hq/papers/ifla75/92-dobreva-en.pdfhttp://www.jisc.ac.uk/media/documents/programmes/reppres/gg_final_keynote_11012008.pdfhttp://www.delos.info/files/pdf/ReferenceModel/DELOS_DLReferenceModel_0.98.pdf


89/89


libraries, International Journal on Digital Libraries 8 (1) , pp. 2138. (2007)

Goncalves, M., Fox, E., Watson, L., and Kipp, N. 2004. Streams, structures,spaces, scenarios, societies (5s): A formal model for digital libraries. ACMTransactions on Information Systems 22(2) (2004) pp. 270-312.

Gonalves, M. A., Moreira, B. L., Fox, E. A., and Watson, L.T. What is a gooddigital library? - A quality model for digital libraries. Inf. Process. Manage. 43, 5(Sep. 2007), 1416-1437

Hassenzahl M, Tractinsky N., User experience a research agenda. Behaviour &Information Technology, Vol. 25, No. 2, March-April 2006, pp. 91 97.

hoo, M., Donahue, R.A. Evaluating Digital Libraries With Webmetrics, Bulletin ofIEEE Technical Committee on Digital Libraries, Volume 4 Issue 1, Spring 2008,http://www.ieee-tcdl.org/Bulletin/v4n1/khoo/khoo.html

Miller P., The concept of the portal, Ariadne, issue 30 (2001),

http://www.ariadne.ac.uk/issue30/portal/

Petersen G., The Europeana Personas (presentation). May 2010,http://www.europeanaconnect.eu/documents/Personas_Gitte_Petersen_20100511.pdf

Sundqvist, A. The use of records a literature review. Archives & Social Studies:A Journal of Interdisciplinary Research, 1(1), 623-653. (2007)
http://www.ieee-tcdl.org/Bulletin/v4n1/khoo/khoo.htmlhttp://www.ariadne.ac.uk/issue30/portal/http://www.europeanaconnect.eu/documents/Personas_Gitte_Petersen_20100511.pdfhttp://www.europeanaconnect.eu/documents/Personas_Gitte_Petersen_20100511.pdfhttp://www.ariadne.ac.uk/issue30/portal/http://www.ieee-tcdl.org/Bulletin/v4n1/khoo/khoo.html

Documents

Feliciati User Studies Dill 2010