1

SelectingUser-GeneratedVideosforAugmentedRealityApplications

IEEEBigMM 2016

HienTo,Hyerim Park,Seon HoKim,CyrusShahabi.

IntegratedMediaSystemsCenterUniversityofSouthernCalifornia

2

Motivationü Popularityofaugmented-reality(AR)

ü Scarcity ofARcontent

ü Lackofusercontextscausesimpreciseregistration

DifficulttocreatecontentinARbrowsersAdoptmultimediacontentfromsocialmedia

ARbrowsers,e.g.,Layar,Wikitude,andJunaio

Needcontentsourcewithrichcontextinformation

3

Motivation(Cont…)Mobilecamerasareeverywhere!

AnywaytousemobilevideosasARcontent?

4

Motivation(Cont…)

• Currentmostimportantkeywordsinvideomarketareasfollows:(byKleiner PerkinsCaufield &Byers (KPCB), 2015)–User-generated– Tagged– Curated– Indexed&Searchable

5

Challenges

Knownissuesofmobilevideos

1.OutofControl 2.HardtoSearch 3.HardlySystematic

HardtosearchforinterestingvideosforARcontent

6

RelatedWorkü Content-basedretrieval• Extensivecomputationrequired,stillinaccurate

ü Spatialmetadata(cameralocation,orientation)• Geo-taggingrequired,sensorsonmobiledevice

𝑁𝑑

pR𝜃

p:cameralocationd:cameradirection𝜃:viewableangleR:viewabledistancet:timestamp

FieldOfView(FOV)

MediaQ:mobilemediamanagementsystemusingspatialmetadatafromsensors(http://mediaq.usc.edu)

7

Geo-taggedMobileVideos

üSelectingvideosforacertainareaisnowfast!

8

ProblemStatementüSimplegeospatialrelevanceisnotenough.Forexample,selecting“interesting”videosforARapplicationsfrommanyatalocation?

üIdentifyinteresting(orsignificant)videostoARusersasasequenceofFOVsthatfollowaparticularcamerashootingpattern usedinfilmingsuchastracking,zooming,panning,andarchingscenes

9

ModelingSignificantVideoSegmentsSingle Multiple

SingleZoom Pan

MultipleTrack Arch

PositionDirection

1d2d

3d

4d5d

1d2d 3d 4d

5dθ

1d

2d3d

4d

5d1d

2d3d

4d 5d

3θ4θ 5θ

0

< 𝑡'(), 𝑑'+, >

< 𝑡'(), 𝑑'+, , 𝑟'+, > < 𝑡'(), 𝑑'(), 𝑟'+, >

< 𝑡'(), 𝑑'+, >

10

Statistics ValueNumberofvideoswithgeo-metadata 2,397Averagelengthper videowithcontent(sec) 72.14Averagecameramovingspeed(km/h) 4.5Averagecamerarotationspeed(degrees/sec) 10Numberofusers 289Numberofvideos byeachuser 8.29NumberofFOVs 208,978NumberofFOVpersecond 1.03NumberofFOVpervideo 74.16

[ACMMMSys 2016DatasetPaper,USCMediaQMobileVideoData]

ExperimentUsingRealDatasets

11

SignificantVideoSegments

Numberofscenesfound Searchingtime(onevideo)

0

500

1000

1500

2000

Track Arch Zoom Pan

Num

bero

fscenes

051015202530

Track Arch Zoom Pan

Runtime(m

s)

𝑡'() = 15, 𝑑'+, = 15, 𝑑'() = 120

Trackingisthemostpopularwayofcapturingmobilevideos

Overallsearchtimeisfast

12

ZoomingScene

TrackingScene

PanningScene

ArchingScene

13

DetectingHotSpots

Hotspotscoveredbymanyvideoframes

0

200

400

600

800

1000

100 90 80 70 60 50 40 30 20 10Totalnum

berofhotspots

MinimumFOVsperhotspot

14

ConclusionandFutureWork

üProposedmodelsandalgorithmstoidentifyinterestingvideosegmentsandhotspots

üExperimentsonrealgeo-taggedvideodatashowthatthealgorithmsarefastandabletoidentifyinterestingscenes

üWillconsiderusermobilityandrankingofvideosegments

15

Thanks

SeonHoKim,Ph.D.UniversityofSouthernCalifornia

seonkim@usc.edu