Sentiment Visualization Widgetsfor Exploratory Search

Eduardo Graells-GarridoUniversitat Pompeu FabraBarcelona, Spain

Mounia LalmasYahoo LabsLondon, UK

Ricardo Baeza-YatesYahoo LabsBarcelona, Spain

1st Social Personalization WorkshopSeptember 1st, 2014Santiago, Chile

Introduction: Context

ExploratorySearch

Sentiment Analysis

HCI

Our work!We use sentiment

visualization widgets for exploratory search

no concrete taskno expertise

big and complex information spaces

faceted search needs structure

positivity, negativity, ambivalence of textcan be visualizedunderstood by userscan be estimated from data

user interfacesindividual differences

information visualization

Research Questions

do visual approaches foster exploration in a sentiment-based exploratory search setting?

we propose design guidelines for visualization widgetsand we evaluate them in a pilot study

who benefits from visual approaches?not all users are equal!

we propose a simple way to identify user archetypes

Background

Sentiment Analysis

> 50% positive reviews depicts a “fresh” movie, “rotten” otherwise.Is sentiment unary(0 to 1), binary (pos/neg) or trinary (pos/neu/neg)?

Sentiment Analysis: Ambivalence

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank [Socher et al., 2005]

Sentiment Analysis: Ambivalence

Visualizing Ambivalence, showing what mixed feelings look like [Panger et al., 2013]

Faceted Search

We can have facets to enable exploratory search. Key points:1) facets need structure. 2) sentiment is not displayed with ambivalence.

We Feel Fine: Faceted Search with Sentiment

We Feel Fine [Kamvar and Harris, 2011] is mostly an art project. It’s cool! But we want to see the effects of visual interfaces in more general contexts.

Visualization Widgets for Exploratory Search

Visgets: Coordinated visualizations for web-based information exploration and discovery [Dörk et al., 2008]

Visualization WidgetsDesign Guidelines

Sentiment Visualization Widgets - Guidelines

Depict ambivalenceAmbivalence is present in many categories and genres of textual content, including public discourse, fiction and news articles. Sentiment is not a simplistic dimension!

Show sentiment distributionHow did the U.S. feel when Obama was elected?

Allow sentiment filteringHighlight/discard sentiment values of (no) interest (show me the bad reviews only*)

*I actually do that ;)

Widgets

Scatterplot Parallel Coordinates

Widget I: Scatterplot

Widget: Parallel Coordinates

Dataset

Dataset: Wikipedia and DBPedia

WikipediaOpen Encyclopedia with a “Neutral Point of View”, but not emotionless content. Positive and negative things do happen and are documented.

DBPedia OntologyShallow hierarchy of 405 classes assigned to each Wikipedia article.

Dataset737863 articles annotated with sentiment.Source: [Mejova et. al, 2013].

DBPedia: Agent > Person > Artist

Aggregated Sentiment in Biographies

Sentiment Distribution

There is sentiment in Wikipedia.We can use it as a facet for exploration!

Evaluation

Pilot User Study

HypothesisIn exploration on sentiment-based scenarios, participants perform more queries and spend more time when using visualisation widgets

Treatments: Scatterplot, Parallel Coordinates, Text Baseline

13 Users recruited from an open call in social networks (within-subjects design).(5 male, 8 female, 3.46 [of 5] average score of knowledge in web search)

Three standard exploratory search tasks with an introduced sentimentality component. In different contexts, users had to find sets of items with some specific requirements (both sentiment and not sentiment related).

Variables

System LogsNumber of queries issuedTask completion time (seconds)

User FeedbackOpen-text questions about the user interfacePerceived time of task completion (minutes)Aesthetic value of the user interface (Likert 1 to 5)

Cognitive EngagementDifference between real task completion time and perceived time of task completion. A positive value indicates a positive experience, as in “time flies!”

Treatments

Scatterplot Parallel Coordinates Text-based

Results

Results are not significant, except on task time.Post-hoc testing: when using the Scatterplot people spends more time.

Is that positive or negative? How to find the answer?

Individual Differences in BehaviorWe split users into explorers and achievers, inspired by MUD Games [Bartle, 1996]:

Explorers those who “interact” with the

worldAchievers

those who “act” in the world

We classified users considering the geometric mean of queries and time. Lower 50% -> achievers.

Explorers have positive engagement when using those approaches (and much more when using SC than BA).

Open FeedbackBaseline

boring [P8], easiest for me to find results [P4]. filters were really easy to use [P3]

I think the most useful one is the buttons one because it has more precise information reflected on it. [P10]

The one with the numbers was misleading for me [P6].

Scatterplotattractive [P8]

[BA] and [SC] are easy to use. They are helpful and easy to understand [P3]. But... needs more concentration [P6].

this is the task that I enjoy the most! I liked pretty much the graphics [P8]

this is the approach I liked the most, it was easier to filter the results [P9]

Parallel Coordinatesinteresting [P8], cooler than the other ones [P1]

helps me to know if it is positive or negative faster. I really like how [PC] worked [P6].

[PC] was not appealing nor easy to understand or use [P3]. It's confusing [P11]

Visual approaches seem to be more emotional.

Baseline seems to be more utilitarian.

Conclusions

Discussion

Do visual approaches foster exploration? Partially.

Scatterplot showed significant differences in task time (users spent more time using the system).

For whom? Explorers.

Longer task time can be explained by positive user engagement... when users are explorers.

Qualitative feedback is favorable to visual approaches. It is more emotional (joy), while the baseline is perceived as more utilitarian (easier to use, it’s familiar).

Discussion

Individual differences can be identified from user models.

Since those users can be identified (i.e., query logs and user modeling), it is possible to personalize user interfaces for explorers, i.e.,

If users are explorers, show a scatterplot as widget. Otherwise, show the text-buttons widget.

Our taxonomy was simple enough to find a significant difference, however, there are more complex taxonomies that are worth to explore in future work (fast surfers, broad scanners and deep divers [Heinström, 2002]).

Conclusions

Visualization techniques in user interfaces have potential to engage users in different ways than current text-based interfaces.

No two users are equal! This should be accounted when doing:

Experimental design which variables are likely to help discriminate between user categories?

System design which user interface is more likely to engage and enhance the user activity?

Simple rules to identify individual differences are enough to find which visualization widget is better in some cases.

Questions and Comments are Welcome!

Thanks for attending! :)

Special ThanksWorkshop Organizers

Anonymous ReviewersParticipants in the User Study

Ilaria Bordino and Yelena MejovaLuca Chiarandini

https://www.flickr.com/photos/blackham/97529032