New Approaches to Interactive Multimedia Content Retrieval from different Sources

New Approaches to Interactive Multimedia Content Retrieval from

different SourcesJulián Moreno Schneider

LaBDA Group, Computer Science DepartmentUniversidad Carlos III de Madrid, Spain

[email protected]

Content

Motivation Background Objectives Proposal

Sports-domain Scenario and Validation Adaptation techniques and Validation Health-domain Scenario and Evaluation

Future directions Publications

Motivation (I) Multimedia content is increasing at staggering

rates

Devices and formats are very diverse and move away from traditional modes.

New Approaches to Interactive Multimedia Content Retrieval from different Sources

4

Motivation (II)


5

Motivation (III) Problem description

Current Limitation: multimedia elements retrieved by textual metadata

Users need access in a transparent, faster and easier way to many independent sources containing information in different formats (such as video, text, audio, images, graphics, etc.).


6

Motivation (IV) Clarifying the problem

Seeking the album of a song having the audio file and the artist’s name

+ ‘I want you back’ The Jackson 5


7

Content


Formal model Sports-domain Scenario and Validation Adaptation techniques and validation Health-domain Scenario and Evaluation



8

Organization by the components Multimodal Information (Collections) Query Information Retrieval Approaches Retrieval Selection Fusion Interactions

Background (I)


9

Multimodal InformationBackground (II)

Image and Text

Image and Audio

Image and Video

TextandVideo

Multimodal

Federated Web Search Track

Jou et al. [2013]


10

Background (III) Query Modalities

Text (Monomodal)

Image (Monomodal)

Text and Image

Video and Image

Text and Audio

Multimodal

Yang et al. [2002]

de Vries [1998]

Marchand-Maillet et al. [2011]


11

Background (IV) Retrieval Approaches

Text Retrieval Low-level features Combined Indexes

Low-level featuresText-based(metadata)retrieval

Full text retrieval

Salton et al. [1975]

Romberg et al. [2012]

Lana-Serrano et al. [2011]


12

Background (V) Retrieval Engine Selection Strategy

Unknown StrategyBy Elements

By Query Terms

Probabilistic

Renaud and Azzopardi [2012]Demner-Fushman et al. [2012]Romberg et al. [2012].

Chernov et al. [2006]

Balog et al. [2012]


13

Background (VI) Result Fusion or Aggregation

Pre-RE fusion: Joint indexes (prior fusion)

Post-RE fusion Randomness Source or type Scores (unification)

Aggregated search

Arampatzis et al. [2011]Balog et al. [2012]Romberg et al. [2012]


14

Background (VII) Semantic Knowledge

Annotation-based Retrieval Multimedia Ontology Retrieval Combination of multimedia ontologies

Worring et al. [2007]Medina-Ramírez [2007] Castells et al. [2007]


15

Background (VIII) User Interactions

• Relevance Judgmentso Directo Indirect

Document browsing Clicks logging and analysisQuery history

• Log Analysis

• Surveys

• Dwell time• Eye tracking• Gestures, lip motion, speech and facial expression


16

Discussion Limitations

Handler strategy (specially adapted to the user experience)

Multimodality in query and results Multimodal semantically related collection Spanish

Out of the scope of this thesis Retrieval approaches Fusion algorithms Innovation in Interaction Logging


17

Content


Formal model Sports-domain Scenario and Validation Adaptation techniques and validation Health-domain Scenario and Evaluation



18

Objectives (I)

1 • Propose a formal model to define

multimodal information retrieval (IMR) systems.

2• Develop two multimodal prototypes

based on the proposed model and evaluate them

3• Design and define techniques to

adapt MIR System based on user experience.


19

Objectives (II) Methodology

Formal Model

Interactions

Sports Domain Scenario

Adaptation techniques

Evaluation

Evaluation

1

2 3

4 5

Health Domain Scenario

6


20

Content


Formal Model Sports-domain Scenario and Validation Adaptation techniques and validation Health-domain Scenario and Evaluation



21

Formal Model (I) Architecture is composed by the most

common components used in IR models.


22

Formal Model (II) Multimodal Information

{text, audio, video, image} SemanticRelations

Multimedia: isFrameOf(image17, video004)

Semantic: shows(image23,FC_Barcelona)

mentions(video12,FC_Barcelona)


23

Formal Model (III) Multimodal Query

RetrievalEngines (RE)

Example of RE:


24

Formal Model (IV) Handler

: set of rules

Example:


25

Formal Model (V) Results Fusion

Interactions

useridentifier, sessionidentifier, timestamp and additionalinformation

Visualizationusingexistingtechniques (clouds, lists, grouping, …)


26

Content





27

Proposal: Sports-Domain Prototype (XII) Architecture


28

Proposal: Sports-Domain Prototype (VI) Buscamedia Collection

Developed in the framework of the Buscamedia Project

Sports Domain Multimodal documents

10000 Texts 350 Images 15 Videos

Recruited in October 2010 Semantically Related


29

Proposal: Sports-Domain Prototype (VII) Multimodal Query

Text, Audio and Text + Image

Información sobre el accidente de la foto +


30

Proposal: Sports-Domain Prototype (VIII) Retrieval Engines

Question Answering (QA), Full Text Search (FT), Ontology-based Search (ONT), Object Detection in Image (ODI), OCR in Image (OCRI), Audio Transcription (AT)

RE selection (Handler) Simple Approach Expert-defined rule-based approach

Question {QA,FT} Txt(short)+img {ONT,FT,{ODI,OCRI}}


31

Proposal: Sports-Domain Prototype (X) Fusion Strategy: Round-Robin Approach


32

Proposal: Sports-Domain Prototype (XI) User Interactions

Searches Documents Browsing Relevance Judgments Visualizations


33

Validation and Results (I) Objective

User Preferences Requested sources? Preferred modes? Preferred visualizations? More used query modes?

Expert-defined Rules Validation Comparison with Baseline (Full Text Search Engine)

Web Interface to test with users 2 months 235 users


34

Validation and Results (II) What query types are used?

981 queries: 239 predefined and 742 user-generated.

Short, long and question queries more often than concepts.

Sources ‘usage’ by query type.

Visualizations Answer List, Answer / Concept Cloud, Concept

Groups, Individual Document


35

Validation and Results (III) Baseline: logs from users IR

performance Mean

Average Precision (MAP)

Mean Reciprocal Rank (MRR)

R-Precision


36

Adapting IR Functionality (I)





37

Adapting IR Functionality (II) Rule-Based MIR

(qmode=t, qtype=long) ont , qa , f t(qmode=t, qtype=question, qlength=14) qa , f t , ont(qmode=t, qtype=short, qlength=2, qentities=alonso) ont, qa, ft


38

Adapting IR Functionality (III)

Adaptation architecture


39

Adapting IR Functionality (IV) Classification Algorithms

Decision trees, multilayer perpectron and simple K-means

Query features Mode, type, length, number of entities, entities,

number of verbs, topic Ranking Scores Interaction-based

Lowest-position Average-position Iteration Mathematical


40

Validating IR Functionality Adaptation (I) Definition of SilverStandard

Example with 4 entity features:qmode=‘t’; qtype=‘short’; qlength=‘1’; qentities=‘Barcelona’ ft, ont, qa

Query: Barcelona


41

Validating IR Functionality Adaptation (II) The best combination is:

Query features: mtle Classification algorithms: J4.8 Ranking scores: Average Position Score


42

Monitoring health social media (I)





43

Monitoring health social media (I) Online: http://

trendminer.daedalus.es/views/dashboard.php

http://trendminer.daedalus.es/views/dashboard.php

http://trendminer.daedalus.es/views/dashboard.php


44

Monitoring health social media (III) Annotation Pipeline

Documents Index

Twitter Saluspot

Relations Manager

Disambiguation

Medical Events Filter

Topics Analyzer

Morpho-syntactic Parser

Language Identification

Resources• DrugsGaz• DrugsATC• AdrsMedDRA• DiseasesUMLS• SpanishDrugEffectDB

Anot

atio

n Pi

pelin

e


45

Monitoring health social media (IV) IMIR System


46

Monitoring health social media (V) Results’ Combination


47

Health-domain Prototype Evaluation No User Evaluation NER & Relation Extraction Performance

NER

Relations extraction

Drugs R P F-mStrict 0,68 0,75 0,76Lenient

0,68 0,75 0,76

Effects R P F-mStrict 0,43 0,75 0,54Lenient 0,47 0,83 0,6

SpanishDrugEffectDB

Coocurrences

Wind. R P F-m R P F-m30 Strict 0,08 0,57 0,14 0,63 0,44 0,5230 Lenient 0,13 0,96 0,24 0,88 0,61 0,72


48

Conclusions (I) Formal model for IMIR systems

Two prototypes based on the formal model in two different scenarios: Sports domain Health social media

Scenario 1: Adaptation of multimodal IR Best result: NDCG=81,54% (2,81% gain)

Good RE performance Small improvements


49

Future Lines (I) Multimodal Query


50

Future Lines (II) Second Screen


51

Publications: Journals Bedmar, I. S., Martínez, P., Arenaz, R. R., and

Schneider, J. M. (2015). Exploring spanish health social media for detecting drug effects. BMC Medical Informatics and Decision Making, 15. 183, 216

Martínez, P., Fernández, J. L. M., Bedmar, I. S., Schneider, J. M., Luna, A., and Arenaz, R. R. (2015). Turning user generated health-related content into actionable knowledge through text analytics services. Computers in Industry.


52

Publications: Conferences SEPLN

Julián Moreno-Schneider, José Luis Martínez Fernández, Paloma Martínez, and Thierry Declerck. Prueba de Concepto de Expansión de Consultas basada en Ontologías de Dominio Financiero.

AMR Julián Moreno-Schneider, José Luis Martínez Fernández, and

Paloma Martínez. A Proof-of-Concept for Orthographic Named Entity Correction in Spanish Voice Queries.

González, M., Moreno Schneider, J., Martínez, J. L., and Martínez, P. (2013). An illustrated methodology for evaluating asr systems.

Schneider, J. M., Salazar, M. G., Martínez, P., and Fernández, J. L. M. (2011). Some experiments in evaluating asr systems applied to multimedia retrieval.


53

Publications: Conferences CLEF Conference

Vicente-Díez, M. T., Moreno-Schneider, J., and Martínez, P. (2010a). Temporal information needs in respubliqa: an attempt to improve accuracy. the uc3m participation at clef 2010.

Vicente-Díez, M. T., De Pablo-Sanchez, C., Martínez, P., Moreno-Schneider, J., and Salazar, M. G. (2009). Are passages enough? the miracle team participation in qaclef2009.

SemEval Vicente-Díez, M. T., Moreno-Schneider, J., and Martínez,

P. (2010b). Uc3m system: Determining the extent, type and value of time expressions in tempeval-2.


54

Research and Development (R&D) projects Trendminer (FP7-ICT 287863)

Buscamedia (CEN-20091026)

Bravo (Búsqueda de Respuestas Avanzada Multimodal y Multilingüe) (TIN2007-67407-C03-01)

MAVIR (S-0505/TIC-0267) and MAVIR2 (S-2009/TIC-1542)


55

‘‘New Approaches to Interactive Multimedia Content Retrieval from different Sources’’

Julián Moreno [email protected]

Thank you for your attention

Technology

New Approaches to Interactive Multimedia Content Retrieval from different Sources