MediaMixer: facilitating media fragments mixing and its rights management using semantic...

MediaMixer !facilitating media fragments mixing and its rights

management using semantic technologies!

!"#$%&"'()%*+),'-./0$%1/&)&'2$'34$/2),'56)/.'!"#$%!$&"'!"#$%&''()**+,-./0*12-3*-41501+6!

789!:+,1!;<=>!?@AB!C1,1D3!

06/06/13 Slide 2 of 28

MediaMixer"Community set-up and networking for the !reMIXing of online MEDIA fragments !•! FP7 Coordination Action!

–!Started November 2012 !–!Create an European market for media

fragments re-purposing and re-selling!E!#-F.)5/!)G!CH!D-)I1,(1!"J!K1+.15F6!!#$%&''FI-21F#3510,1.'.5),(/'38F1*3,L(8*+IL*12-38M1N8(513.183,,).3.18%51F1,.83,28F#3518/)+58*12-3!!

06/06/13 Slide 3 of 28

Introduction!•! The MediaMixer project and community promote

the use of semantic technologies for media mixing!•! Real use cases and demos that showcase these

technologies!–! Fragmenting media assets!–! Annotating them using semantic descriptions!–! Exposing these descriptions for fragment level search

and selection!–! Representing rights information using a copyright

ontology that integrates licenses, policies and rights expressions languages!

06/06/13 Slide 4 of 28

Coordination. Semantic Web technologies R&D

IT solutions for broadcasters (WebTV, EPG,…)

Asset Management solutions

videolectures.net academic content platform

Semantic multimedia research

Semantic copyright management research

Partners!

06/06/13 Slide 5 of 28

!"#$%$&'!"#$%$&'!"#$%$&'

!"#$%&'" (%)*+,-#.&'"

(&)$*+!,")-%.$"/

!"#&/0'"

!"#$%$&'!"#$%$&'01,&&2&/.'

!,")-%&)+(&)$*

(&)$*+3,*12&/.' 4"/!$%.'

Producer checks reused media licenses vs. internal policies and agreements (with providers and rights holders)!

Policy example: “…avoid media fragments showing violence when producing content for children”. !

Problems:!1.) Semantic annotation!

a) For instance, automatically tag"violent content!

2.) Rights Integration!a) Policies and agreements (unstructured documents) !b) Media fragments licenses "(rights expression languages)!

3.) Intelligently detect conflicts!

Example Use Case!

06/06/13 Slide 6 of 28

Current Situation!

!"!#"$%&'()*

+",$-./""0"#,

122"/$3/4.0"#,

-./""0"#,

%&'()*

!"4'

5"6(4$3/4.0"#,

%/&67)"/8'".4'9

:(.;,<$=&'6"/

%/&>(6"/

+"4/);$3/4.0"#,

?;")@A"/0<

:"<&'>"$?&#"(),

%/&67)"/8)&#,"#,$04#4."/9

Issues:!•! scalability "

(mostly a manual process)!•! conflicts detected too late

(added costs)!•! missed opportunities "

(media already available)!•! …!!

06/06/13 Slide 7 of 28

Proposals:"Assisted Policy and Agreement Modelling!•! Model policies and

agreements!•! Formal model that allows

automatic integration and intelligence support!

•! Semantic model based on Copyright Ontology!

•! Editor with assistance guided by the ontology!

06/06/13 Slide 9 of 28

Media Fragments Ingestion!Provider offers media fragment:!!

•! Content: "automatic annotation!

•! Detect topics: Violence? Outdoors? Cars?...!

•! Rights:"automatic translation of rights language to Copyright Ontology!

•! DDEX, ODRL, MPEG-21, Creative Commons…!

06/06/13 Slide 10 of 28

Semantic Annotation!

10!

•! Transform whole videos to sets of meaningful, indexable and re-usable video fragments!

O15F),B!P,)MB!Q511FB!!@+-I2-,RB!PS-B!T!

@3F1N3IIB!Q#5)M-,RB!P%)5.FB!!OI3,.B!K+,,-,RB!T!

U-.(#1,B!V,2))5B!W3S1B!T!

T!

06/06/13 Slide 11 of 28

Semantic Annotation"Key Technologies!

11!

•! Temporal video fragmentation to!–! Shots (elementary video segments)!–! Scenes (groups of consecutive shots)!

•! Video annotation (to make video fragments indexable / searchable)!–! Concept detection!–! Event detection!–! Event recounting!

•! Video object re-detection (to create spatio-temporal video fragments)!

V,%+.&!),1!*12-3!-.1*!X+.%+.&!F1.!)G!*12-3!G53R*1,.F!

V,%+.&!),1!*12-3!-.1*!Y!143*%I1!)G!F)+R#.!)NZ1(.!X+.%+.&!F1.!)G!*12-3!G53R*1,.F!

V,%+.&!),1!.1*%)53I!*12-3!G53R*1,.!X+.%+.&!F1.!)G!I3N1IF!"(),(1%.B!1D1,.6!

06/06/13 Slide 12 of 28

Semantic Annotation"Metadata!•! Metadata for semantic annotation:!

–!Descriptive metadata for media characteristics!–!Provenance metadata to credit source and

specify rights!–!Conceptual metadata to reflect what media is

perceived to represent!

06/06/13 Slide 13 of 28

Conceptual Metadata!• Globally unambiguous

identifiers for any concept needed–! Ideally more information

about the concept itself is available via the identifier!

–! Use of Web URIs Linked Data concept space!

–! Would allow for inference of concept type and relationship to other concepts!

06/06/13 Slide 14 of 28

Producer Proposed Workflow!

!"#$%&"'()*+$,-

."&(/#'""0"1&

)'*23,"'4+"#5+6

."5',7(8'5#0"1&

97",:;%"

)'*23,"'4,*1&"1&(0515#"'6

/#'""0"1&<=

)*+$,-<=

9*>-'$#7&!"5%*1"'

)*+$,-(512(/#'""0"1&(?*2"++$1#(/%%$%&51&

."051&$,(?"2$5(!">*%$&*'-

/#'""0"1&(<= @"5+(?"2$5(8'5#0"1&(<=

A1&"12"2(;%"

)*+$,-(

<=

?"2$5(8'5#0"1&

<=."051&$,(?"&525&5

/++*B"2(3%"%

@$%++*B"2(3%"%

06/06/13 Slide 15 of 28

Implementation!•! Rights, License and Policies "

implemented as Classes!–! Reproduction Right, Copy, "

Copy Pattern for fragment #1,...!•! Uses implemented as Instances!

–! u: Peter copies media fragment #1 "in the US on 2013-06-03 using…!

•! if u Copy Pattern then"pattern authorises u !

Reproduction Right

Copy

Copyright

Copy Pattern

u

?

06/06/13 Slide 16 of 28

Pattern Building"Actions (“verbs”)!

!"#$%"&'

(')%$'*+

,$'"-+)%$%"&'

.&/0

1+/-&/2$'*+

2$'"-+)%!"#$%&' ()*%$''$'

3&224'"*$%"&'

+"'&),%&-*.'

%/$')-&/2

"25/&6")+

7")%/"84%+

*&59

*&59

-"#

5+/-&/2

*&224'"*$%+

/+%/$')2"%

!"#$%&'()*%+,''-.,'/",0&123.,'

Copyright

EconomicRights

RelatedRights

MoralRights

DistributionRight

ReproductionRight

PublicPerformanceRight

FixationRight

CommunicationRight

AttributionRight

TransformationRight

IntegrityRight

DisclosureRight

WithdrawalRight

PermorfersRights

ProducersRights

BroadcastersRights

RentalRight

ImportationRight

SoundRecordRight

MotionPictureRight

BroadcastingRight

MakingAvailableRight

AdaptationRight

TranslationRight

06/06/13 Slide 17 of 28

Pattern Building"Case Roles (from linguistics)!•! Connect verbs to the action participants!

–!Example: “The dog ate food”!

initiator resource goal essence Action agent, instrument result, patient,

effector recipient theme Process agent, matter result, patient,

origin recipient theme Transfer agent, instrument, experiencer, theme

origin medium recipient Spatial origin path destination location Temporal start duration completion pointInTime Ambient reason manner aim, condition

consequence

Example: “The dogP+NZ1(.B!5)I1![3R1,.\!

food”XNZ1(.B!5)I1![.#1*1\!

06/06/13 Slide 18 of 28

DDEX Sample-08.04.xml!<Deal>!

!<DealTerms>!! !<ValidityPeriod>!! ! !<StartDate>2013-01-01</StartDate>!! !</ValidityPeriod>!! !<Usage>!! ! !<UseType>!! ! ! !OnDemandStream"! ! !</UseType>!! ! !<DistributionChannelType>!! ! ! !Internet!! ! !</DistributionChannelType>!! !</Usage>!! !<TerritoryCode>ES</TerritoryCode>!! !<TerritoryCode>US</TerritoryCode>!!</DealTerms>!

</Deal> !

Copyright Ontology Model!<http://media.com/agreement#1> owl:Class;!

!co:start "2013-01-01" ;!!owl:intersectionOf (!! !ddex:OnDemandStream !! ![ a owl:Restriction ;!

! ! !owl:onProperty co:medium ;"! ! !owl:someValuesFrom ddex:Internet ]!! ![ a owl:Restriction ;!! ! !owl:onProperty co:location ;!! ! !owl:someValuesFrom "! ! ![ a !owl:Class ;!! ! ! !owl:oneOf (territory:ES territory:US) ]!! ![ a owl:Restriction ;!! ! !owl:onProperty co:theme ;!! ! !owl:hasValue!! ! ! !<http://my.tv/video.ogv#t=60,100> ] !

!! !]!!) . !

DDEX to RDF

<http://co:startowl:intersectionOf

1</StartDate> ddex:[ a [ a

[ a owl:Restriction

[ a

[ a m

DistributionChannelType>

owl:onProperty

TerritoryCode>TerritoryCode>

owl:someValuesFrom[ a

[ a owl:Restrictionowl:onPropertyowl:hasValue

[ a

[ a owl:Restriction

[ a

06/06/13 Slide 19 of 28

])R-(3I!-,.15%51.3L),!)G!3!I-(1,F1!*)21I!

Reasoning with DDEX

06/06/13 Slide 20 of 28

Reasoning with DDEX

Policy #1 Agreement #1

06/06/13 Slide 21 of 28

!"#$%&"!'('&")'!"#$%"*&")'&'()*+!,-./0*++'!"#$%&"!,-%#./&0%-%12"$*+'

3'4%#.".5'2$%6788#9,%)-'*0"!!20%!2"$+:'..";,%)-:'<%=>!21?0%&0%,%)-'!"#$%&"!,-%#.@%-2<>*AB%C0!"#D2&1E&FC,%)-A+'!"#$%&"!,-%#.42<"&$"*AGGHIJC#D=-"KLM,LN,%)-A+'O$"7'('&")'1#"*O$"7FPE:'AO$"!L7A:'A/&G"D#&.C0!"#DA:'AE&0"!&"0A:''

Ahttp://my.tv/video.ogv#t=60,100A:'AHCA:'A9L76KLNK7QA+'!"#$%&"!,-%#.*O$"7+'!"$=%&$"'('!"#$%&"!,2$F$"@"!D200".*O$"7FPE+'=!2&0'!"$=%&$"'

3'RO0?%!2$".5'?00=5SSD".2#D2;"!,"OS<%=>!21?0S";#D=-"$S#1!""D"&0T7'O$"9'('&")'1#"*O$"9FPE:'AO$"!L7A:'A/&G"D#&.C0!"#DA:'AE&0"!&"0A:'

Ahttp://my.tv/video.ogv#t=60,100A:'AFCA:'A9L76KLNK7QA+'!"#$%&"!,-%#.*O$"9+'!"$=%&$"'('!"#$%&"!,2$F$"@"!D200".*O$"9FPE+'=!2&0'!"$=%&$"'

3'@!%?2U20".5'?00=5SSD".2#D2;"!,"OS<%=>!21?0S";#D=-"$S=%-2<>T7'

License Checking Implementation

06/06/13 Slide 22 of 28

@prefix co: <http://rhizomik.net/ontologies/2013/05/copyrightonto.owl#> .@prefix odrlv: <http://w3.org/ns/odrl/vocab#> . !@prefix owl: <http://www.w3.org/2002/07/owl#> . !!<http://example.com/policy> a co:Agree ;! co:agent <http://example.com/billie>, <http://example.com/sony> ;! co:theme [ a owl:Class ;! co:end "2012-12-31" ;! co:start "0001-01-01" ;! owl:intersectionOf (!

[ owl:intersectionOf (!odrlv:play [ owl:complementOf odrlv:mobile_ringtone ] ) ]

[ a owl:Restriction ;!owl:hasValue <http://example.com/billie> ;owl:onProperty co:agent ][ a owl:Restriction ;owl:hasValue <http://example.com/music> ;owl:onProperty co:theme ] ) ] .

!<http://example.com/sony:10> a co:Person .!<http://example.com/music> a co:Creation .!<http://example.com/billie> a co:Person .!

ODRL Example from LCC Rights Reference Model Spec!

()*+'$,-./01.)2),+/2$3"14"/.)/."&.5/[F),/!3,2!N-II-1!ER511!.#3.!!N-II-1!%I3/!3,2!,).!*)N-I1!5-,R.),1!!*+F-(\0!

06/06/13 Slide 23 of 28

Creative Commons "BY-SA Model!

!"#$%�%&'()*'+,)-%�%

!"..,/'01)-%�%213-451'61+6-

2$21/'7-()1)'"/

!"#$#

.$8"*3

4))*'+,)-

%&'()!)&'

.-

!"#$#

*#%)+)#'!

49*--

!"#$#

&-*'5-

!"#$#

!"#$#

:-;21/'7-()1)'"/

/-;8"*3

*#,-.!

),/0')1#,!0!)&'21

),3#*)40!)&'21

!"#$%�%&'()*'+,)-%�%

!"..,/'01)-%�%213-451'61+6-

!"#$#

!"#$#%&'()!)&'

),/0')1#,!0!)&'21

! "

#

06/06/13 Slide 24 of 28

•! Following Linked Data principles, "starting from just content ID (URI)…!

Long term vision…!

24!

W),.1,.!!AKV!

AKV!3!1N+()51&V*3R1!!!!2(.&L.I1![?@A!^_\!!!!2(.&()%/5-R#.!678/!!!T!!!!!

HTTP GET image/jpeg

06/06/13 Slide 25 of 28 06/06/13 Slide 25 of 28

Linked Data for Worldwide "Copyright Management?!

25!

W),.1,.!!

AKV!

W),.1,.

AKV!3!1N+()51&V*3R1!!!!2(.&L.I1![?@A!^_\!!!!2(.&()%/5-R#.!AKV!!!!T!!!!!

HTT

P G

ET

ER511*1,.!!

AKV!

ER511*1,.

AKV!3!()&ER511*1,.!!!!()&3R1,.!678!!!!()&.#1*1!T/!!!T!!!!!

HTT

P G

ET

O15F),!!

AKV!

URI a foaf:Person foaf:name “…” vcard:address … vcard:country URI

HTT

P G

ET

ER511*1,.ER511*1,.

AKV

ER511*1,.

URI a foaf:Person foaf:name “…” vcard:address …

vcard:country

HTT

P G

ET

06/06/13 Slide 26 of 28

Get involved!•! Join MediaMixer community to know more "

(http://community.mediamixer.eu):!–! discussions, use cases, demonstrators, tutorials,

presentations, software,…!

06/06/13 Slide 26 of 28

06/06/13 Slide 27 of 28

Get more involved!•! Want to experience semantic technologies?

Apply to become core partner"(http://mediamixer.eu/contact) :!–!Contribute use case, sample data and just

some time…!–!…selected partners get a demo showcasing

semantic technologies for their problem!

More details: http://community.mediamixer.eu http://rhizomik.net/copyrightonto/

Roberto García Universitat de Lleida, Spain

http://rhizomik.net/~roberto

Thank you for your attention