Upload
xavier-giro
View
256
Download
0
Tags:
Embed Size (px)
Citation preview
Visual Summary of Egocentric Photostreams by Representative
Keyframes
Author: Ricard MestreSupervisor: Xavier GiróDate: Tuesday, 17th of February 2015
1
Contents● Collaboration● Motivation and goals● State of the art● Methodology● Evaluation● Conclusions and future work
2
Contents● Collaboration● Motivation and goals● State of the art● Methodology● Evaluation● Conclusions and future work
4
Motivation and goals
● Lifelogging with Narrative Clip
● Up to 2000 images/day
● A visual summary can help the memory of Alzheimer affected people
5
Motivation and goals● Extract a visual summary
of a day
○ Clustering strategy for event detection
○ Automatic selection of representative frames
6
Contents● Collaboration● Motivation and goals● State of the art● Methodology● Evaluation● Conclusions and future work
7
State of the art
Chandrasekar et al, “Efficient retrieval from large-scale egocentric visual data using a sparse graph representation” (CVPR Workshop 2014)
8
State of the art
Lu and Grauman, ”Story-driven summarization for egocentric video” (CVPR 2013)
9
Contents● Collaboration● Motivation and goals● State of the art● Methodology● Evaluation● Conclusions and future work
10
Feature extraction
● Convolutional Neural Networks (CNN) trained with ImageNet.
12
Jia et al, “Caffe: Convolutional Architecture for Fast Feature Embedding” (ACM MM 2014)
Clustering● Obtain separated events● Agglomerative clustering
14
cutoff parameter
Talavera, E., Dimiccoli, M., Bolaños, M., Aghaei, M., & Radeva, P. (2015). “R-Clustering for Egocentric Video Segmentation”. In 7th Iberian Conference on Pattern Recognition and Image Analysis (ACCEPTED).
Fusion● Short clusters (less than 5 images) are not
representative
● Join the short events into larger ones
19
?
?
Keyframe extraction● Criterion: visual similarity-based keyframe● Graph-based approach:
25Similarity GraphAdjacency Matrix
Contents● Collaboration● Motivation and goals● State of the art● Methodology● Evaluation
○ Database○ Clustering○ Keyframe extraction
● Conclusions and future work30
Evaluation: Database● 5 days● 3 users● 4005 images● Groundtruth available
31
Talavera, E., Dimiccoli, M., Bolaños, M., Aghaei, M., & Radeva, P. (2015). “R-Clustering for Egocentric Video Segmentation”. In 7th Iberian Conference on Pattern Recognition and Image Analysis (ACCEPTED).
Contents● Collaboration● Motivation and goals● State of the art● Methodology● Evaluation
○ Database○ Clustering
■ Jaccard index■ Linkage effect■ Relabelling effect
○ Keyframe extraction● Conclusions and future work 32
Contents● Collaboration● Motivation and goals● State of the art● Methodology● Evaluation
○ Database○ Clustering○ Keyframe extraction
■ Blind taste test■ Representative quality of keyframe■ Summary validations
● Conclusions and future work 36
Evaluation: keyframe extraction● User Surveys:
○ Representative quality of keyframe
○ Quality of summary
37
● Methodology: Blind taste test
38Lu and Grauman, ”Story-driven summarization for egocentric video” (CVPR 2013)
Figure: brandchannel.com
Blind taste test: quality of keyframe
Representative quality of keyframe
41
Do you think that the image of the left/center/right can represent the event?
Representative quality of keyframe
43
What image is more representative of the event, in your opinion?
Contents● Collaboration● Motivation and goals● State of the art● Methodology● Evaluation● Conclusions and future work
47
Conclusions and future work● New methodology taking into account visual and
temporal information
● Keyframe extraction through graph-based approaches
48
Conclusions and future work● 0.53 Jaccard index of segmentation
● 88-86% user acceptance with our summaries
● 58% users choose our summaries as best option
49
Conclusions and future work● Temporal information causes important improvements
● First method of summary extraction for high temporal resolution sets
50
Conclusions and future work
● Apply object detection
● Different criteria of representativity
● Clinical application of this work
51