Trained vs. untrained searchers’ interaction with searchfeatures in digital libraries: A case study

Yuelin LiSCILS, Rutgers University, 4 Huntington Street, New Brunswick, NJ 08901lynnlee@scils.rutgers.edu

Xiangmin ZhangSCILS, Rutgers University, 4 Huntington Street, New Brunswick, NJ 08901xzhang@scils.rutgers.edu

Jingjing LiuSCILS, Rutgers University, 4 Huntington Street, New Brunswick, NJ 08901jingjing@scils.rutgers.edu

Ying ZhangSCILS, Rutgers University, 4 Huntington Street, New Brunswick, NJ 08901yzhang@scils.rutgers.edu

This case study explores how trained and untrained searchers interact with searchfeatures in a digital library. Thirty five trained and untrained searchers completedthe experiment. The results indicate that the usage of the search modes issignificantly associated with the participants’ search training background. Thestudy also suggests that the untrained participants prefer to accept the system’sdefault search field and search results display format. The suggestions forimproving the interaction design in IEEE Xplore digital library are proposed andfuture studies are addressed.

Introduction

The purpose of this study was to investigate how trained and untrained searchers wouldinteract with search features in digital libraries. Trained searchers are those who havereceived professional training in information search. They are assumed to have bettersearch skills and know more search tactics, and thus can search more effectively in digitallibraries than untrained searchers. By comparing different types of users’ search

interactions with a digital library, the study intended to inform the design for searchfeatures in digital libraries to support untrained users.

The study was conducted with the IEEE Xplore digital library (v.2.0). Search features in this digital library include:

search modes (Basic search, Advanced search, Author search, CrossRef search)search fields provided in “Advanced search” (All fields, Full text & All fields,Document Title, Author, Publication Title, Abstract, Index Terms, and Affiliation.)query submission facility (query box and search button)result presentation (citation, citation & abstract, organize result, select date range)source (publications, other sources)

This study concentrated on how trained and untrained searchers would interact with these elements, and then addressed the issue of how to improve interaction design of search feature in this digital library.

Related studies

Many studies investigated and compared experienced and novice users’ search behaviorand performance. Both Penniman (1981) and Tolle (1983) found that the frequentsearchers used more search commands, conducted more extensive searches, and madefewer errors when searching in MEDLINE and CATLINE respectively. Similarly, Fenichel(1981) examined the searching behavior of novice and expert searchers, and found thatthe novices searched more slowly, made more errors, and used less thesaurus terms thanthe experienced participants. Vigil (1988) claimed that a more experienced searchershould have greater capacity to deal with complex search statements and know betterwhat need to do next. Hsieh-Yee’s (1993) study indicated that search experience affectedthe use of many search tactics. Yuan (1997) discovered that search experience impingedon command and features used during the search, search speeds, learning approaches,and so on. Lazonder (2000) and Lazonder et al. (2000) investigated the differencesbetween novice and experienced users in the Web search. The results demonstrated thatexperienced searchers spent less time in strategy selection when they were engaging inthe low and medium search task than novices. This suggests that a user with moreknowledge in domain and searching may take more advanced search strategies, whichpossibly lead to better search performance. Hembrooke et al. (2005) also found that thesetwo groups of users were divergent in strategies of selecting search terms.

Some researchers attempted to develop new system functions for helping novices improvetheir search performance. Bhavnani (2001) observed how domain (healthcare) experts didsearch for locating comprehensive information. He and his colleagues (Bhavnani et al.,

2006) incorporated these experts’ search procedures into the system design. They foundthat these search procedures were helpful for average users. Providing automatedassistance during the search process is another approach to helping average users.Jansen’s (2004) study indicated that though only 50% subjects preferred to use theautomated assistance embedded in the system, over 80% of those subjects implementedthat assistance. This indicated that to some extent the automated assistance was helpfulfor the users.

Aforementioned studies suggested that it was valuable to identify how expert searchersinteract with the interface of IR systems in order to improve interaction design. However,few studies specifically focus on trained and untrained searchers’ interaction with differentelements of search feature in digital libraries, especially from interaction designperspective. The articulation of this issue could benefit search feature design in digitallibraries, and thus enhance novice users’ search experience.

Method

Participant: Totally thirty-six students were recruited from undergraduate engineering students (UE), engineering graduate students (GE), and library and information science graduate students (LIS) at Rutgers University, each group with 12 students. The LIS grouphad completed a search course from the MLIS program. They were trained searchers. Theother two groups (GE and UE) were untrained searchers. Since graduate students usually search digital libraries more frequently than undergraduate students, the GE group could be viewed as a more experienced group than the UE group in information search.

Task: One topic search task, which asked the participants to locate relevant documents onprotecting the on-line repository from fraudulent activity by watermarking, was assigned to the participants. Participants were asked to search and to identify the relevant items, with unlimited search time.

Procedures: An experiment was conducted in a usability lab. The participant was asked to sign consent form first, then the experimenter showed a quick demo about the IEEE Xplore digital library. Before the experiment, the participant was asked to fill out an entry questionnaire and a pre-search questionnaire. After the search, she or he was required to finish a post-search questionnaire and an exit interview. During the experiment, the participant was asked to think-aloud. The software Morae was used to record the whole session of the experiment.

Data analysis: One of the researchers went through the recordings of the experiments,and observed how the participants interacted with the elements. The number of

participants who utilized different search modes, changed search fields when using“Advanced search”, and changed search results display format in different groups wascounted respectively. The average query length and the mean number of queries issued inthe three groups were also calculated. Univariate statistic analyses were conducted andthe software SPSS was used to do the analyses.

Results

This paper focuses on those most frequently used elements and reports on how trained and untrained searchers interacted with them. One participant (in LIS group) dropped off from the study, so in the end we analyzed the data collected from 35 participants.

Search modes: It was observed that the three groups were very different in using the search modes. Figure 1 shows how these search modes (Basic search, advance search, or combination) were used by different groups:

Figure 1. Distribution of search mode used

It is obvious that the LIS group prefers advanced search (82%). Moreover, no participants in this group used basic search, and two out of 11 participants (18%) used both search modes (i.e. combination) for completing the search task. In the GE group, half of the participants (50%) used the combination and 33% of the participants used advanced search. The UE group exhibited a different pattern: half of them (50%) employed basic search, and only 17% used advanced search. A Chi-square test detected that the usage of

different search modes was significantly associated with the different groups (?2 (4, N=35)= 14.35, p< .01). This indicated that the trained searchers preferred the advanced search

feature more than the untrained searchers, who seemed to be more comfortable with the basic search feature.

Query submission: This study examined the mean number of queries issued and the average query length (i.e., the mean number of words in a query). Table1 shows that the participants from the LIS and GE group submitted more queries than those in the UE group, and the LIS group also submitted the longest query on average. However, ANOVA tests did not discover any significant difference between the three groups whether in termsof the number of queries issued or the average query length.

Table 1. Queries issued and query length

LIS GE UE

Mean # of queries issued 6.36 6.83 5.25

Average query length 2.83 2.73 2.61

Search field: Advanced search usually provides different search fields for users to select. The participants might change the search field (Changed) for better search results or not change them and accept the default search field of the system. Figure 2 shows that 55% of the participants in the LIS group changed search fields. However, only 33% of the participants in the GE group and 8% of participants in the UE group changed search field. 83% of the participants in the UE group did not change the search field at all. It seemedthat the trained users (the LIS group) could take advantage of this element, though no significant difference in working with the search fields among the three groups was discovered by Chi-square tests. Also, there was a trend that if a participant had more expertise in information search, she or he more preferred to change the search fields.

Figure 2. Distribution of working with the search field

Display format: Two display formats, i.e., “citation” and “citation & abstract”, are providedin the IEEE Xplore digital library. The difference between these two formats is that theformer one does not supply the abstracts of retrieved documents, but the latter one does.The default format is “citation.” Some participants preferred to change the format from“citation” to “citation & abstract” in order to read the summary for making relevancejudgment. Table 2 shows the number of participants who changed the display format indifferent groups and the number of times they made this change.

Table 2. Distribution of display format changed

LIS GE UE

# of the participants changed 6 (55%) 4 (33%) 2 (17%)

# of times the participants changed 10 7 4

We can see that 55% of the participants in the LIS group changed the display format. Only33% of the participants in the GE group and 17% in the UE group took this action. This result indicated that there was a trend that the trained searchers might know better way to get as much information as possible for relevance judgment, though no significant difference was detected by Chi-square tests.

To summarize, the participants in the trained group overall made more uses of the searchfeatures in the IEEE Xplore digital library. They were able to manipulate advanced searchand search fields better, and they realized that they could get more information about theretrieved items if changing result display format to “citation & abstract”. On the other hand,

the untrained searchers, especially the UE group, seemed not to know how to use theseinteraction features.

Discussion and conclusion

The aim of this study was to investigate how the trained and untrained searchersinteracted with the search features of the IEEE Xplore system. The results confirmed theresearch findings in the previous studies (e.g. Hembrooke et al. (2005); Lazonder et al.2000). That is, the trained searchers might use different elements more wisely than theuntrained searchers. Therefore, it is necessary to take into account the characteristics ofuntrained searchers when considering design issues. Based on the results of this study,the following issues should be considered when designing search interactions for digitallibraries: (1) Since the untrained searchers did not know or prefer not to change searchfields, a more effective default search field may be critical for them to obtain desiredsearch results. For example, full-text search as a default would be better than fieldedsearch for them (in IEEE Xplore). (2) Advanced search is usually hidden behind basicsearch. Even though sometimes the untrained searchers clicked the link of “Advancedsearch” in the IEEE Xplore, they would give up finally since the complicated interface(compared to basic search) might make them unconfident to use it. Therefore, it may bedesirable to provide an easy-to-understand advanced search interface for the untrainedsearchers. This could be adding some instructions, for example, in a pop-up window, onthe interface. (3) Not many untrained searchers, particularly in the UE group, knew thatthey could change the display format. Based upon the findings in this study, it is suggestedproviding a more helpful default display format which should at least includes both thecitation and abstract.

The results reported here are preliminary since part of the data has not been analyzed yet.Several issues need to be further addressed in the next step of the study. For example, why the trained and untrained searchers interacted with these elements differently, and how the differences impact their search performance. After exploring these issues, a set ofmore specific suggestions for designing the search features may be proposed.

Acknowledgement

Our thanks to IEEE, Inc. for sponsoring this project.

References

Bhavnani, S. K. (2001)Important cognitive components of domain-specific search knowledge In E. M. Vorhees

& D. K. Harman (Eds.) NIST Special Publication 500-250: The Tenth Text RetrievalConference (TREC ’01) pp. 571-578. Washington, DC: NIST

Bhavnani, S. K., Bichakjian, C. K., Johnson, T. M., Little, R. J., Peck, F. A., Schwartz, J. L., & Strecher, V. J. (2006)Strategy hubs: Domain portals to help find comprehensive information Journal of the

American Society for Information Science and Technology 57(1), 4-24

Fenichel. C. H. (1981)Online searching: Measures that discriminate among users with different types of

experiences Journal of the American Society for Information Science 32, 23-32

Hembrooke, H. A., Granka, L. A., Gay, G. K., & Liddy, E. D. (2005) The effects of expertise and feedback on search term selection and subsequent learning Journal of the American Society for Information Science and Technology 56(8), 861-871

Hsieh-Yee, I. (1993)Effects of search experience and subject knowledge on the search tactics of novice and

experienced searchers Journal of American Society for Information Science 44(3), 161-174

Jansen, B. J. (2004)Seeking and implementing automated assistance during the search processInformation Processing and Management 41, 909-928

Lazonder, A. W. (2000)Exploring novice users’ training needs in searching information on the WWW Journal

of Computer Assisted Learning 16, 326-335

Lazonder, A. W., Biemans, H. J. A., & Wopereis, I. G. J. H. (2000) Differences between novice and experienced users in searching information on the World Wide Web Journal of the American Society for Information Science 51(6), 576-581

Penniman, W. D. (1981) Modeling and evaluation of online user behavior Final report to the National Library of Medicine under extramural program grant no. NLM/EMP (1 R01 LM03444-01). Dublin, OH: OCLC

Tolle, J. E. (1983)Understanding patrons use of online catalogs: Transaction log analysis of the search

method In R. F. Vondran (Ed.) Proceedings of the 46th annual ASIS meeting pp.167-171. White Plains, NY: Knowledge Industry, 20

Vigil, P. J. (1988) Online retrieval: analysis and strategy New York: John Wiley & Sons

Yuan, W. (1997)End-user searching behavior in information retrieval: A longitudinal study Journal of the

American Society for Information Science 48(3), 218-234