www.lib.umich.edu/imls-grant-topic-clustering

USERS IN THE WILD Unmoderated Assessment of Topic Modeling in the Digital Library

University of MichiganYale UniversityUniversity of California, IrvineUniversity of MichiganUniversity of Michigan

Kat HagedornYoun Noh

Dave NewmanMike Kargela

Suzanne Chapman

Using Morae software, participants were presented with a series of tasks & were asked to complete each task in the live test interface. There was no moderator present. The software recorded their interactions & provided the team with tools for analyzing & downloading the results. For this study, there was 1 dedicated machine placed in a library public area for several continuous weeks.

OK, but what's the goal?

The goal of the test itself was to find out:

• Did users use topics?• Did they like them?• How did they use them?

How did the participants perform the test? Test = 3 tasks & follow-up questions.

Build your test scripts... carefully...

• Revised & revised until it was down to the bare minimum to match our goals

• Tested the test with library staff who were very familiar with both usability testing & web interfaces

• Testing the test is super important but won't find all potential problems! Always try to pilot the test in the testing environment

Results! (preliminary)

Please go up the stairs to the

• Yale & Michigan each ran 2 studies, 1 image and 1 text based• 306 participants, mostly undergraduate & graduate students• Test took 5 - 15 minutes to take• Each participant was given a $15 Amazon gift certificate

Funded by the Institute of Museum and Library Services (IMLS) and is being conducted by Yale University, the University of Michigan, and the University of California, Irvine.

What's topic modeling?

Topic modeling is a fairly new computer algorithm (in the text and data mining category of algorithms) that can extract semantically similar topics from LARGE collections of textual items. It can be run on heterogenous and homogenous collections, and tends to work very well on items with very little text (e.g., metadata).

We created our script and then...

Very important! We tested 3 times!

It's very hard to design questions when you know

you won't be there if the participant goes off track!

• Face-to-face testing: moderated testing asking similar questions, also comparing topics to LCSH

• Topic assignment: more robust comparison of topics with LCSH & with more students

• Topic segmentation: both students & algorithm identify topics for sections of books

What's next...DON'T FORGET...

• Decide if you're going with out of the box or homegrown tools.

• Contact your IT department NOW.• Plan how you will gather data and

send incentives.• Budget time to do the analysis.

Keep returning to your goals so that you are not out of scope and off topic

• 41% claimed satisfaction with their results, but... • When they used the facets, this climbed to 78%• 42% claimed satisfaction with the TOPIC facet, but... • They also overwhelmingly liked the topic facet• Further testing implies users prefer topic facet & subject

headings facet TOGETHER rather than separately

!"#$

%&$'%$

#$'#$(#$)#$"#$

!##$!'#$!(#$!)#$!"#$'##$

*+,$ -./$012+$ -.$!"#$%&#"'()"#"%

*+",%-%./$0#1%&#"/23"%4/#"5()%67%89%1(/%:#$#%3(%/"#%3;<"%:#="<3#%+>+<)?%:(/2@%1(/%/"#%3;#%A*('<B"A%<)%3;#%

2#C%B(2/D)%3(%;#2'%1(/%$#E)#%1(/$%"#+$B;F%

But! Less than 1/2 the participants said the facet column was

useful...

Satisfaction peaks at a maximum of 2 facets used

Participant Comments

If NOT asked to look at topics, happened 32% of the time. If asked to look at all facets, chose topic facet 77% of the time.

What's unmoderated assessment?

Morae did the job but wasn't the perfect tool for this project

Topic modeling should... make search and discovery easier, more accessible, more powerful, and greatly improve how content is organized and accessed in digital libraries.

At the end of the semester, your professor asks for a final overview-- the architecture of religious buildings.

Use the "Topic" section of the "Narrow Search" column on the left to do this. Click the title for a book that seems like the best fit.

Did you find the topics column useful in refining your results?

Incentives make a huge difference!

Study specifics

topics as facets

topics as browse links

Users interacted with home-built interfaces.

“”

I could not appropriately target results that we[re] both specific enough and yet general overviews.

It was great to browse by topic. Lots of unexpected and stimulating results.“

”

# of Facets Selected0 1 2 3

Completely Satisfied

Satisfaction Rating : # of Facets Selected

1

2

3

4

5

Not At All Satisfied

Total FacetsTopic Facets

# of Facets Selected

0 1 2 3 4

Task 2 Task 3