Embed Size (px)
Citation preview
An Approachto Personalized Video SummarizationBased on User Preferences Analysis
Maria Miniakhmetova, Mikhail [email protected], [email protected]
South Ural State University, Chelyabinsk, Russia
9th International Conference on Application of Informationand Communication Technologies
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 1 / 21
Video Summarization
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 2 / 21
Personalized Video Summary
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 3 / 21
Using Extra Devices
Eye movements tracking Facial activity recognition
Analyzing physiological responses Using data from personal cameras
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 4 / 21
Without Extra Devices
Analyzing text description Specifying preferences manually
Manual key frames selection
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 5 / 21
Approach Proposed in This Work
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 6 / 21
User’s Evaluations
Formal definition
E = {e+, e0, e−}
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 7 / 21
Video Structuring
Formal definition
V = {si}ni=1; 0 < n ≤ F
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 8 / 21
Video Summary
Formal definition
r(V ) = {sj}tj=1; sj ⊆ V
d(r(V )) =
t∑j=1
d(sj) ≤ dmax
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 9 / 21
Object Detection
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 10 / 21
Object Indexing
Formal definition
O = {oi}Mi=1
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 11 / 21
Importance of Objects
Formal definition
P (Ai) = P (Ai|e+) · P (e+) + P (Ai|e0) · P (e0) + P (Ai|e−) · P (e−)
P (Ai) = P (Ai ∩ e+) + P (Ai ∩ e0) + P (Ai ∩ e−)
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 12 / 21
Importance of Objects
Formal definition
P (Ai|e+) =Li
L; P (Ai|e0) =
Ni
N; P (Ai|e−) =
Di
D
P (e+) =L
W; P (e0) =
N
W; P (e−) =
D
W
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 13 / 21
Importance of Objects
Formal definition
Imp(oi) = func(P (Ai), Li, Di, Ni)
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 14 / 21
Impact of shots
Importance of object Imp(oi)
Imp(oi) = P (Ai) ·Li −Di
max(1, Ni)
Impact of shot
Imp(sj) = sgn(argmax
oi∈sj
(|Imp(oi)|
))· maxoi∈sj
|Imp(oi)| ·∑oi∈sj
|Imp(oi)|
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 15 / 21
Personalized Video Summary
Formal definition
pr(V ) =
t⋃i=1
si|si ⊆ V ;
t∑i=1
d(si) ≤ dmax;
∀si ⊆ pr(V ), sj 6⊆ pr(V ) : |Imp(si)| ≥ |Imp(sj)|
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 16 / 21
Algorithm of Summary Construction
1 Detect shots on video files stored in database.
2 Detect objects O on shots of the video files stored in database.
3 Calculate importance Imp(oi) of each object for the particular user.
4 Calculate the impact Imp(sj) of each shot of the video file that hasn’t beenwatched by the user yet.
5 Rank shots of the video file on the basis of the absolute values of their’simpact.
6 Select top t shots, the total duration of which doesn’t exceed the predefinedthreshold value dmax.
7 Join all of the selected shots to construct personalized video summary pr(V ).
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 17 / 21
Summary Construction
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 18 / 21
Evaluation of the Personalized Video Summary
Forecast evaluation of the original video file
Imp(V ) = func(Imp(sj))
Variants of the function
1
Imp(V ) = sgn(∑sj∈V
Imp(sj))
2
Imp(V ) = sgn(∑sj∈V
Imp(sj) ·d(sj)
d(V ))
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 19 / 21
Evaluation of the Personalized Video Summary
Actual evaluation of the original video file
E = {e+, e0, e−}
Adequacy of the constructed video summary
Ad(pr(V )) =
{1, (E = Imp(V ))
−1, (E 6= Imp(V ))
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 20 / 21
Future Work
M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 21 / 21
