Upload
astin
View
32
Download
2
Embed Size (px)
DESCRIPTION
Designing and Evaluating Search Interfaces. Prof. Marti Hearst School of Information UC Berkeley. Outline. Why is Supporting Search Difficult? What Works? How to Evaluate?. Why is Supporting Search Difficult?. Everything is fair game Abstractions are difficult to represent - PowerPoint PPT Presentation
Citation preview
1
Designing and Evaluating Search Interfaces
Prof. Marti HearstSchool of InformationUC Berkeley
2
Outline
⢠Why is Supporting Search Difficult?⢠What Works?⢠How to Evaluate?
3
Why is Supporting Search Difficult?
⢠Everything is fair game⢠Abstractions are difficult to represent⢠The vocabulary disconnect⢠Usersâ lack of understanding of the technology⢠Clutter vs. Information
4
Everything is Fair Game
⢠The scope of what people search for is all of human knowledge and experience.
â Other interfaces are more constrained(word processing, formulas, etc)
⢠Interfaces must accommodate human differences in:
â Knowledge / life experienceâ Cultural background and expectationsâ Reading / scanning ability and styleâ Methods of looking for things (pilers vs. filers)
5
Abstractions Are Hard to Represent
⢠Text describes abstract conceptsâ Difficult to show the contents of text in a visual or compact
manner
⢠Exercise:â How would you show the preamble of the US Constitution
visually?â How would you show the contents of Joyceâs Ulysses
visually? How would you distinguish it from Homerâs The Odyssey or McCourtâs Angelaâs Ashes?
⢠The point: it is difficult to show text without using text
6
Vocabulary Disconnect
â If you ask a set of people to describe a set of things there is little overlap in the results.
7
The Vocabulary ProblemData sets examined (and # of participants)
â Main verbs used by typists to describe the kinds of edits that they do (48)
â Commands for a hypothetical âmessage decoderâ computer program (100)
â First word used to describe 50 common objects (337)â Categories for 64 classified ads (30)â First keywords for a each of a set of recipes (24)
Furnas, Landauer, Gomez, Dumais: The Vocabulary Problem in Human-System Communication. Commun. ACM 30(11): 964-971 (1987)
8
The Vocabulary ProblemThese are really bad results
â If one person assigns the name, the probability of it NOT matching with another personâs is about 80%
â What if we pick the most commonly chosen words as the standard? Still not good:
Furnas, Landauer, Gomez, Dumais: The Vocabulary Problem in Human-System Communication. Commun. ACM 30(11): 964-971 (1987)
9
Lack of Technical Understanding
⢠Most people donât understand the underlying methods by which search engines work.
10
People Donât Understand Search Technology
A study of 100 randomly-chosen people found:â 14% never type a url directly into the address bar
⢠Several tried to use the address bar, but did it wrongâ Put spaces between wordsâ Combinations of dots and spacesâ ânursing spectrum.comâ âconsumer reports.comâ
â Several use search form with no spaces⢠âplumberâslocal9â âcapitalhealthsystemâ
â People do not understand the use of quotes⢠Only 16% use quotes⢠Of these, some use them incorrectly
â Around all of the words, making results too restrictiveâ âlactose intolerance ârecipiesâ
Âť Here the â excludes the recipesâ People donât make use of âadvancedâ features
⢠Only 1 used âfind in pageâ⢠Only 2 used Google cache
Hargattai, Classifying and Coding Online Actions, Social Science ComputerReview 22(2), 2004 210-227.
11
People Donât Understand Search Technology
Without appropriate explanations, most of 14 people had strong misconceptions about:
⢠ANDing vs ORing of search termsâ Some assumed ANDing search engine indexed a smaller collection;
most had no explanation at all
⢠For empty results for query âto be or not to beââ 9 of 14 could not explain in a method that remotely resembled
stop word removal
⢠For term order variation âboat fireâ vs. âfire boatââ Only 5 out of 14 expected different resultsâ Understanding was vague, e.g.:
Âť âLycos separates the two words and searches for the meaning, instead of whatâre your looking for. Google understands the meaning of the phrase.â
Muramatsu & Pratt, âTransparent Queries: Investigating UsersâMental Models of Search Engines, SIGIR 2001.
12
What Works?
13
Cool Doesnât Cut It
⢠Itâs very difficult to design a search interface that users prefer over the standard
⢠Some ideas have a strong WOW factorâ Examples:
⢠Kartoo⢠Groxis⢠Hyperbolic tree
â But they donât pass the âwill you use itâ test
⢠Even some simpler ideas fall by the waysideâ Example:
⢠Visual ranking indicators for results set listings
14
Early Visual Rank Indicators
15
16
17
Metadata Matters
⢠When used correctly, text to describe text, images, video, etc. works well
⢠âSearchersâ often turn into âbrowsersâ with appropriate links
⢠However, metadata has many perilsâ The Kosher Recipe Incident
18
Small Details Matter
⢠UIs for search especially require great care in small detailsâ In part due to the text-heavy nature of searchâ A tension between more information and
introducing clutter
⢠How and where to place things importantâ People tend to scan or skimâ Only a small percentage reads instructions
19
Small Details Matter⢠UIs for search especially require endless tiny adjustments
â In part due to the text-heavy nature of search
⢠Example:â In an earlier version of the Google Spellchecker, people
didnât always see the suggested correction⢠Used a long sentence at the top of the page:
âIf you didnât find what you were looking for âŚâ
⢠People complained they got results, but not the right results.⢠In reality, the spellchecker had suggested an appropriate
correction.
Interview with Marissa Mayer by Mark Hurst: http://www.goodexperience.com/columns/02/1015google.html
20
Small Details Matter
⢠The fix: â Analyzed logs, saw people didnât see the correction:
⢠clicked on first search result, ⢠didnât find what they were looking for (came right back to the
search page⢠scrolled to the bottom of the page, did not find anything⢠and then complained directly to Google
â Solution was to repeat the spelling suggestion at the bottom of the page.
⢠More adjustments:â The message is shorter, and different on the top vs. the
bottom
Interview with Marissa Mayer by Mark Hurst: http://www.goodexperience.com/columns/02/1015google.html
21
22
Small Details Matter
⢠Layout, font, and whitespace for information-centric interfaces requires very careful design
⢠Example:â Photo thumbnailsâ Search results summaries
23
What Works for Search Interfaces?⢠Query term highlighting
â in results listingsâ in retrieved documents
⢠Term Suggestions (if done right)
⢠Sorting of search results according to important criteria (date, author)
⢠Grouping of results according to well-organized category labels (see Flamenco)
⢠DWIM only if highly accurate:â Spelling correction/suggestionsâ Simple relevance feedback (more-like-this)â Certain types of term expansion
⢠So far: not really visualization
Hearst et al: Finding the Flow in Web Site Search, CACM 45(9), 2002.
24
Highlighting Query Terms
⢠Boldface or color⢠Adjacency of terms with relevant context is a
useful cue.
25
26
27
Highlighted query term hits using Google toolbar
US
Blackout
PGA
Microsoft
found!
found!
donât know
donât know
Microso
28
How to Introduce New Features?
⢠Example: Yahoo âshortcutsââ Search engines now provide groups of enriched
content⢠Automatically infer related information, such as sports
statistics
â Accessed via keywords⢠User can quickly specify very specific information
â united 570 (flight arrival time)â map âsan franciscoâ
⢠Weâre heading back to command languages!
29
30
31
32
33
Introducing New Features⢠A general technique: scaffolding
⢠Scaffolding:â Facilitate a studentâs ability to build on prior
knowledge and internalize new information. â The activities provided in scaffolding instruction are
just beyond the level of what the learner can do already.
â Learning the new concept moves the learner up one âstepâ on the conceptual âladderâ
34
Scaffolding Example⢠The problem: how do people learn about these
fantastic but unknown options?
⢠Example: scaffolding the definition functionâ Where to put a suggestion for a definition?â Google used to simply hyperlink it next to the
statistics for the word.â Now a hint appears to alert people to the feature.
35
Unlikely to notice the function here
36
Scaffolding to teach what is available
37
Query Term Suggestions
38
Query Reformulation
⢠Query reformulation:â After receiving unsuccessful results, users modify
their initial queries and submit new ones intended to more accurately reflect their information needs.
⢠Web search logs show that searchers often reformulate their queriesâ A study of 985 Web user search sessions found
⢠33% went beyond the first query⢠Of these, ~35% retained the same number of terms
while 19% had 1 more term and 16% had 1 fewer
Use of query reformulation and relevance feedback by Excite users,Spink, Janson & Ozmultu, Internet Research 10(4), 2001
39
Query Reformulation
⢠Many studies show that if users engage in relevance feedback, the results are much better.â In one study, participants did 17-34% better with RFâ They also did better if they could see the RF terms
than if the system did it automatically (DWIM)
⢠But the effort required for doing so is usually a roadblock.â Before the web and in most research, searches have
to select MANY relevant documents or MANY terms.
Koenemann & Belkin, A Case for Interaction: A Study of Interactive Information Retrieval Behavior and Effectiveness, CHIâ96
40
Query Reformulation
⢠What happens when the web search engines suggests new terms?
⢠Web log analysis study using the Prisma term suggestion system:
Anick, Using Terminological Feedback for Web Search Refinement âA Log-based Study, SIGIRâ03.
41
Query Reformulation Study
⢠Feedback terms were displayed to 15,133 user sessions. â Of these, 14% used at least one feedback termâ For all sessions, 56% involved some degree of query refinement
⢠Within this subset, use of the feedback terms was 25%â By user id, ~16% of users applied feedback terms at least once
on any given day
⢠Looking at a 2-week session of feedback users:â Of the 2,318 users who used it once, 47% used it again in the
same 2-week window.
⢠Comparison was also done to a baseline group that was not offered feedback terms.â Both groups ended up making a page-selection click at the same
rate.
Anick, Using Terminological Feedback for Web Search Refinement âA Log-based Study, SIGIRâ03.
42
Query Reformulation Study
Anick, Using Terminological Feedback for Web Search Refinement âA Log-based Study, SIGIRâ03.
43
Query Reformulation Study⢠Other observations
â Users prefer refinements that contain the initial query terms
â Presentation order does have an influence on term uptake
Anick, Using Terminological Feedback for Web Search Refinement âA Log-based Study, SIGIRâ03.
45
Prognosis: Query Reformulation
⢠Researchers have always known it can be helpful, but the methods proposed for user interaction were too cumbersomeâ Had to select many documents and then do feedbackâ Had to select many termsâ Was based on statistical ranking methods which are hard
for people to understand
⢠RF is promising for web-based searchingâ The dominance of AND-based searching makes it easier to
understand the effects of RFâ Automated systems built on the assumption that the user
will only add one term now work reasonably well â This kind of interface is simple
46
Supporting the Search Process
⢠We should differentiate among searching:â The Webâ Personal informationâ Large collections of like information
⢠Different cues useful for each⢠Different interfaces needed⢠Examples
â The âStuff Iâve Seenâ Projectâ The Flamenco Project
47
The âStuff Iâve Seenâ project
⢠Did intense studies of how people work⢠Used the results to design an integrated search
framework⢠Did extensive evaluations of alternative designs
â The following slides are modifications of ones supplied by Sue Dumais, reproduced with permission.
Dumais, Cutrell, Cadiz, Jancke, Sarin and Robbins, Stuff I've Seen: A system for personal information retrieval and re-use. SIGIR 2003.
48
Searching Over Personal Information
⢠Many locations, interfaces for finding things (e.g., web, mail, local files, help, history, notes)
Slide adapted from Sue Dumais.
49
The âStuff Iâve Seenâ project
⢠Unified index of items touched recently by userâ All types of information, e.g., files of all types, email, calendar, contacts, web pages, etc.â Full-text index of content plus metadata attributes (e.g., creation time, author, title, size)â Automatic and immediate update of indexâ Rich UI possibilities, since itâs your content
Search only over things already seen Re-use vs. initial discovery
Slide adapted from Sue Dumais.
50
SIS Interface
Slide adapted from Sue Dumais
51
Search With SIS
Slide adapted from Sue Dumais
52
Evaluating SIS
⢠Internal deploymentâ ~1500 downloadsâ Users include: program management, test, sales,
development, administrative, executives, etc.
⢠Research techniquesâ Free-form feedbackâ Questionnaires; Structured interviewsâ Usage patterns from log dataâ UI experiments (randomly deploy different versions)â Lab studies for richer UI (e.g., timeline, trends)
⢠But even here must work with usersâ own content
Slide adapted from Sue Dumais
53
SIS Usage DataDetailed analysis for 234 people, 6 weeks usage
⢠Personal store characteristicsâ 5k â 100k items; index <150 meg
⢠Query characteristicsâ Short queries (1.59 words)â Few advanced operators or fielded search in query box
(7.5%)â Frequent use of query iteration (48%)
⢠50% refined queries involve filters â type, date most common⢠35% refined queries involve changes to query⢠13% refined queries involve re-sort
⢠Query contentâ Importance of people
⢠29% of the queries involve peopleâs names
Slide adapted from Sue Dumais
54
SIS Usage Data, contâdCharacteristics of items
opened⢠File types opened
â 76% Email â 14% Web pagesâ 10% Files
⢠Age of items openedâ 7% todayâ 22% within the last weekâ 46% within the last month
⢠Ease of finding informationâ Easier after SIS for web, email,
filesâ Non-SIS search decreases for web,
email, files Files Email Web Pages0
1
2
3
4
5
6
Pre-usage
Post-usage
0
20
40
60
80
100
120
0 500 1000 1500 2000 2500
Freq
uenc
y
Days Since Item First Seen
Log(Freq) = -0.68 * log(DaysSinceSeen) + 2.02
Slide adapted from Sue Dumais
55
SIS Usage, contâd
UI Usage⢠Small effects of Top/Side,
Previews⢠Sort order
â Date by far the most common sort field, even for people who had Okapi Rank as default
â Importance of timeâ Few searches for âbestâ
match; many other criteria ⌠Num
ber
of Q
uerie
s Is
sued
Date Rank0
500
1000
1500
2000
2500
3000
3500
Starting Default Sort Column
Date
Rank
Slide adapted from Sue Dumais
56
Web Sites and Collections
A report by Forrester research in 2001 showed that while 76% of firms rated search as âextremely importantâ only 24% consider their Web siteâs search to be âextremely usefulâ.
Johnson, K., Manning, H., Hagen, P.R., and Dorsey, M. Specialize Your Site's Search. Forrester Research, (Dec. 2001), Cambridge, MA; www.forrester.com/ER/Research/Report/Summary/0,1338,13322,00
57
There are many ways to do it wrong
⢠Examples:â Melvyl online catalog:
⢠no way to browse enormous category listings
â Audible.com, BooksOnTape.com, and BrillianceAudio:⢠no way to browse a given category and simultaneosly
select unabridged versions
â Amazon.com: ⢠has finally gotten browsing over multiple kinds of features
working; this is a recent development⢠but still restricted on what can be added into the query
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
The Flamenco Project
⢠Incorporating Faceted Hierarchical Metadata into Interfaces for Large Collections
⢠Key Goals:â Support integrated browsing and keyword search
⢠Provide an experience of âbrowsing the shelvesâ
â Add power and flexibility without introducing confusion or a feeling of âclutterâ
â Allow users to take the path most natural to them
⢠Method:â User-centered design, including needs assessment
and many iterations of design and testing
Yee, Swearingen, Li, Hearst, Faceted Metadata for Image Search and Browsing, Proceedings of CHI 2003.
73
Some Challenges
⢠Users donât like new search interfaces.⢠How to show lots more information without
overwhelming or confusing?⢠Our approach:
â Integrate the search seamlessly into the information architecture.
â Use proper HCI methodologies.â Use faceted metadata
74
The Flamenco Interface
⢠Hierarchical facets⢠Chess metaphor
â Openingâ Middle gameâ End game
⢠Tightly Integrated Search⢠Expand as well as Refine⢠Intermediate pages for large categories⢠For this design, small details *really* matter
75
76
77
78
79
80
81
82
83
84
What is Tricky About This?
⢠It is easy to do it poorlyâ Yahoo directory structure
⢠It is hard to be not overwhelmingâ Most users prefer simplicity unless complexity really
makes a difference
⢠It is hard to âmake it flowââ Can it feel like âbrowsing the shelvesâ?
85
Using HCI Methodology⢠Identify Target Population
â Architects, city planners
⢠Needs assessment. â Interviewed architects and conducted contextual inquiries.
⢠Lo-fi prototyping. â Showed paper prototype to 3 professional architects.
⢠Design / Study Round 1. â Simple interactive version. Users liked metadata idea.
⢠Design / Study Round 2: â Developed 4 different detailed versions; evaluated with 11
architects; results somewhat positive but many problems identified. Matrix emerged as a good idea.
⢠Metadata revision. â Compressed and simplified the metadata hierarchies
86
Using HCI Methodology
⢠Design / Study Round 3. â New version based on results of Round 2â Highly positive user response
⢠Identified new user population/collectionâ Students and scholars of art historyâ Fine arts images
⢠Study Round 4â Compare the metadata system to a strong,
representative baseline
87
Most Recent Usability Study
⢠Participants & Collectionâ 32 Art History Studentsâ ~35,000 images from SF Fine Arts Museum
⢠Study Designâ Within-subjects
⢠Each participant sees both interfaces⢠Balanced in terms of order and tasks
â Participants assess each interface after useâ Afterwards they compare them directly
⢠Data recorded in behavior logs, server logs, paper-surveys; one or two experienced testers at each trial.
⢠Used 9 point Likert scales.⢠Session took about 1.5 hours; pay was $15/hour
88
The Baseline System
⢠Floogle⢠Take the best of the existing keyword-based
image search systems
89
Comparison of Common Image Search Systems
System Collection # Results /page
Categories?
# Familiar
Google Web 20 No 27
AltaVista Web 15 No 8
Corbis Photos 9-36 No 8
Getty Photos, Art 12-90 Yes 6
MS Office Photos, Clip art
6-100 Yes N/A
Thinker Fine arts images
10 Yes 4
BASELINE Fine arts images
40 Yes N/A
90
swordsword
91
92
93
94
Evaluation Quandary
⢠How to assess the success of browsing?â Timing is usually not a good indicatorâ People often spend longer when browsing is going
well.⢠Not the case for directed search
â Can look for comprehensiveness and correctness (precision and recall) âŚ
â ⌠But subjective measures seem to be most important here.
95
Hypotheses
⢠We attempted to design tasks to test the following hypotheses:â Participants will experience greater search
satisfaction, feel greater confidence in the results, produce higher recall, and encounter fewer dead ends using FC over Baseline
â FC will perceived to be more useful and flexible than Baseline
â Participants will feel more familiar with the contents of the collection after using FC
â Participants will use FC to create multi-faceted queries
96
Four Types of Tasksâ Unstructured (3): Search for images of interest â Structured Task (11-14): Gather materials for an art
history essay on a given topic, e.g.⢠Find all woodcuts created in the US⢠Choose the decade with the most⢠Select one of the artists in this periods and show all of
their woodcuts⢠Choose a subject depicted in these works and find
another artist who treated the same subject in a different way.
â Structured Task (10): compare related images⢠Find images by artists from 2 different countries that
depict conflict between groups.
â Unstructured (5): search for images of interest
97
Other Points⢠Participants were NOT walked through the
interfaces.⢠The wording of Task 2 reflected the metadata;
not the case for Task 3⢠Within tasks, queries were not different in
difficulty (tâs<1.7, p >0.05 according to post-task questions)
⢠Flamenco is and order of magnitude slower than Floogle on average.â In task 2 users were allowed 3 more minutes in FC
than in Baseline.â Time spent in tasks 2 and 3 were significantly longer in
FC (about 2 min more).
98
Results
⢠Participants felt significantly more confident they had found all relevant images using FC (Task 2: t(62)=2.18, p<.05; Task 3: t(62)=2.03, p<.05)
⢠Participants felt significantly more satisfied with the results (Task 2: t(62)=3.78, p<.001; Task 3: t(62)=2.03, p<.05)
⢠Recall scores:â Task2a: In Baseline 57% of participants found all
relevant results, in FC 81% found all.â Task 2b: In Baseline 21% found all relevant, in FC
77% found all.
99
Post-Interface Assessments
All significant at p<.05 except simple and overwhelming
100
Perceived Uses of Interfaces
What is interface useful for?
6.44
5.475.91
4.91
7.97 7.91
6.646.16
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
9.00
Useful for mycoursework
Useful forexploring anunfamiliarcollection
Useful for findinga particular image
Useful for seeingrelationships b/w
images
SHASTA
DENALI
Baseline
FC
101
Post-Test Comparison
15 16
2 30
1 29
4 28
8 23
6 24
28 3
1 31
2 29
FCBaseline
Overall Assessment:
More useful for your tasksEasiest to useMost flexible
More likely to result in dead ends
Helped you learn moreOverall preference
Find images of rosesFind all works from a given period
Find pictures by 2 artists in same media
Which Interface Preferable For:
102
Facet Usage
⢠Facets driven largely by task contentâ Multiple facets 45% of time in structured tasks
⢠For unstructured tasks, â Artists (17%)â Date (15%)â Location (15%)â Others ranged from 5-12%â Multiple facets 19% of time
⢠From end game, expansion fromâ Artists (39%)â Media (29%)â Shapes (19%)
103
Qualitative Observations
⢠Baseline:â Simplicity, similarity to Google a plusâ Also noted the usefulness of the category links
⢠FC:â Starting page âwell-organizedâ, gave âideas for what to
search forââ Query previews were commented on explicitly by 9
participantsâ Commented on matrix prompting where to go next
⢠3 were confused about what the matrix showsâ Generally liked the grouping and organizingâ End game links seemed useful; 9 explicitly remarked
positively on the guidance provided there.â Often get requests to use the system in future
104
Study Results Summary
⢠Overwhelmingly positive results for the faceted metadata interface.
⢠Somewhat heavy use of multiple facets.⢠Strong preference over the current state of the
art.⢠This result not seen in similarity-based image
search interfaces.⢠Hypotheses are supported.
105
Summary
⢠Usability studies done on 3 collections:â Recipes: 13,000 itemsâ Architecture Images: 40,000 itemsâ Fine Arts Images: 35,000 items
⢠Conclusions:â Users like and are successful with the dynamic
faceted hierarchical metadata, especially for browsing tasks
â Very positive results, in contrast with studies on earlier iterations
â Note: it seems you have to care about the contents of the collection to like the interface
106
Using DWIM
⢠DWIM â Do What I Meanâ Refers to systems that try to be âsmartâ by guessing usersâ
unstated intentions or desires⢠Examples:
â Automatically augment my query with related termsâ Automatically suggest spelling correctionsâ Automatically load web pages that might be relevant to
the one Iâm looking atâ Automatically file my incoming email into foldersâ Pop up a paperclip that tells me what kind of help I need.
⢠THE CRITICAL POINT:â Users love DWIM when it really worksâ Users DESPISE it when it doesnât
⢠unless not very intrusive
107
DWIM that Works⢠Amazonâs âcustomers who bought X also bought Yâ
â And many other recommendation-related features
108
DWIM Example: Spelling Correction/Suggestion
⢠Googleâs spelling suggestions are highly accurate
⢠But this wasnât always the case. â Google introduced a version that wasnât very
accurate. People hated it. They pulled it. (According to a talk by Marissa Mayer of Google.)
â Later they introduced a version that worked well. People love it.
⢠But donât get too pushy.â For a while if the user got very few results, the page was
automatically replaced with the results of the spelling correction
â This was removed, presumably due to negative responses
Information from a talk by Marissa Mayer of Google
109
What Weâve Covered
⢠Introductionâ Why is designing for search difficult?
⢠How to Design for Searchâ HCI and iterative designâ What works?â Small details matterâ Scaffoldingâ The Role of DWIM
⢠Core Problemsâ Query specification and refinementâ Browsing and searching collections
110
Final Words
⢠User interfaces for search remains a fascinating and challenging field
⢠Search has taken a primary role in the web and internet business
⢠Thus, we can continue to expect fascinating developments, and maybe some breakthroughs, in the next few years!
111
Thank you!
Marti Hearsthttp://www.ischool.berkeley.edu/~hearst
112
References
Anick, Using Terminological Feedback for Web Search Refinement âA Log-based Study, SIGIRâ03.
Bates, The Berry-Picking Search: UI Design, in âUser Interface Designâ, Thimbley (ED), Addison-Wesley 1990
Chen, Houston, Sewell, and Schatz, JASIS 49(7)Chen and Yu, Empirical studies of information visualization: a meta-analysis,
IJHCS 53(5),2000Dumais, Cutrell, Cadiz, Jancke, Sarin and Robbins, Stuff I've Seen: A system
for personal information retrieval and re-use. SIGIR 2003.Furnas, Landauer, Gomez, Dumais: The Vocabulary Problem in Human-
System Communication. Commun. ACM 30(11): 964-971 (1987) Hargattai, Classifying and Coding Online Actions, Social Science Computer
Review 22(2), 2004 210-227.Hearst, English, Sinha, Swearingen, Yee. Finding the Flow in Web Site
Search, CACM 45(9), 2002.Hearst, User Interfaces and Visualization, Chapter 10 of Modern Information
Retrieval, Baeza-Yates and Rebeiro-Nato (Eds), Addison-Wesley 1999.Johnson, Manning, Hagen, and Dorsey. Specialize Your Site's Search.
Forrester Research, (Dec. 2001), Cambridge, MA
113
References
Koenemann & Belkin, A Case for Interaction: A Study of Interactive Information Retrieval Behavior and Effectiveness, CHIâ96
Marissa Mayer Interview by Mark Hurst: http://www.goodexperience.com/columns/02/1015google.html
Muramatsu & Pratt, âTransparent Queries: Investigating Usersâ Mental Models of Search Engines, SIGIR 2001.
OâDay & Jeffries, Orienteering in an information landscape: how information seekers get from here to there, Proceedings of InterCHI â93.
Rose & Levinson, Understanding User Goals in Web Search, Proceedings of WWWâ04
Russell, Stefik, Pirolli, Card, The Cost Structure of Sensemaking , Proceedings of InterCHI â93.
Sebrechts, Cugini, Laskowski, Vasilakis and Miller, Visualization of search results: a comparative evaluation of text, 2D, and 3D interfaces, SIGIR â99.
Swan and Allan, Aspect windows, 3-D visualizations, and indirect comparisons of information retrieval systems, SIGIR 1998.
Spink, Janson & Ozmultu, Use of query reformulation and relevance feedback by Excite users, Internet Research 10(4), 2001
Yee, Swearingen, Li, Hearst, Faceted Metadata for Image Search and Browsing, Proceedings of CHI 2003