DominantCodewordsSelec2onwithTopicModelforAc2onRecogni2on

HirokatsuKATAOKA,MasakiHayashi†,YoshimitsuAOKI†,KenjiIWATA,YutakaSATOH,SlobodanIlic‡

Na2onalIns2tuteofAdvancedIndustrialScienceandTechnology(AIST)†KeioUniversity

‡TechnischeUniversitätMünchen

hPp://www.hirokatsukataoka.net/

HumanSensing• Severaltasksareincludedinvision-basedhumansensing–  Detec2on,tracking,facerecogni2on–  Posturees2ma2on,ac2onanalysis(eventrecogni2on)–  Ac2onrecogni2onisabletoextendhumansensingapplica2ons

Mentalstate

BodySitua2on

APen2on

Ac2onAnalysis

shakinghands

Lookatpeople

Detec2on GazeEs2ma2on

Ac2onRecogni2on

PostureEs2ma2on

FaceRecogni2on

Trajectoryextrac2on

Tracking

UnderstandingHumanAc2ons•  Ac2onisdefinedassomethingthatpeopledoorcausetohappen–  e.g.walking,running, si^ng

Thisimagecontainsamanwalking•  Ac2onrecogni2on

Classifica2on•  Ac2ondetec2on

Classifica2on&localiza2on

Walking

Walking

Ac2onRecogni2on

Ac2onDetec2on

CurrentAc2onRecogni2on• Trajectory-basedrepresenta2on

ImprovedDenseTrajectories(IDT)[Wang+,ICCV13] Trajectory-pooledDeep-convolu2onalDescriptors(TDD)[Wang+,CVPR15]

•  “Dense”representa2onisimportant•  Trajectory-basedhand-craied

descrip2on(HOG/HOF/MBH/Traj.)

•  Intersec2onofhand-craiedanddeep-learnedfeatures

•  Intui2vely,IDTfeatureisreplacedbyConvMaps

•  CNNisbasedonTwo-streamConvNet[Simonyan+,NIPS14]

TypicalAc2onRecogni2onPipeline

Trajectories/keypointsextrac2on Featuredescrip2on

Codewordpooling Classifica2on

e.g.BoF,VLAD,FisherVectors hPp://www.analy2calway.com/images/SMV/svmFeatureSpace2.gif

e.g.SVM

TypicalAc2onRecogni2onPipeline• Twostrategiesformoresophis2catedfeaturevector

Trajectoryextrac2on Trajectorydescrip2on

1.Trajectoryrefinement!(maincontribu2on)

2.BePerfeature!

Ourproposal• 1.Dominantcodewordsselec2on(DCS)–  Significantac2onrepresenta2onbyusingtopicmodel(typicalLDA)–  Topic-basedgroupingfromdensetrajectories–  Noiseelimina2onateachtopic

• 2.Co-occurrencefeaturerepresenta2onforbePerfeat.–  Asanaddi2onalfeatureintotypicalHOG/HOF/MBH/Traj.–  Thefeatureisbasedon[Kataoka+,ACCV14]whichisimprovedco-occurrence

featurefocusingonextrac2ngsubtlemo2on

××××

×××

××

×

×

×

×× ×

Therearesomenoiseeveninthesta2cscene!

Flowchartofdominantcodewordsselec2on(DCS)• 1.Featureextrac2on–  DenseTrajectories

• 2.Topicmodelingtodivideprimi2vemo2on–  Eachtopicisapproxima2ngeachprimi2vemo2on

• 3.Noisecancelingateachtopic• 4.Dominantdensetrajectories(DDT)

Featureextrac2on•  Improveddensetrajectories[Wang+,ICCV13]–  HOG/HOF/MBHx,/MBHy–  CoHOG,ExtendedCoHOG[Kataoka+,ACCV14]–  Bag-of-features(BoF)toinputoftopicmodel

ExtendedCoHOG

TopicModel•  LatentDirichletalloca2on(LDA)[Blei+,JMLR03]–  Weappliedsimplifiedmodel[Griffiths+,04]–  TypicalLDA–  Parameters•  Probabilitydistribu2onoftopic:Θ•  Topic:T•  Hyperparameter:α,β•  DT-BoFfromvideo:v

TopicModelforDCS• Noiserejec2onateachtopic–  Eachtopicindicateseach“primi2vemo2on”–  In-topicadap2vethresholding

×××××××

××

×

×

“cutoffends” Topic1 Topic2 Topic3 Topic4

MPIIcooking

Eachtopicindicateseach“primi2vemo2on”

× ××× ××× ×

×××××××

××

×

×

“cutoffends”

××× × ×××

××

×

××× Unsophis2catedrejec2onisnotdesirable

× Eliminatedvector

Dominantdensetrajectoies(DDT)• Allrefinedtopicsareintegrated–  DDT

–  DDTisamixtureofdominantcodewordsusingANDopera2on e.g.•  T1={vw1,vw3,vw5}=>T1’={vw1,vw5}•  T2={vw1,vw2,vw7}=>T2’={vw1,vw2,vw7}•  T3={vw3,vw6,vw7}=>T3’={vw7}

•  DDT={vw1,vw2,vw5,vw7}

• DDTissophis2catedrepresenta2onwithdominantcodewords

Experiments• Fourdatasetsforac2onrecogni2on

【INRIA surgery】 View: 4 Activity: 4

view1 view2

view3 view4

【IXMAS】 View: 5 Activity: 11

【MPII cooking activities】 View: 1 Activity: 65

view1 view2

view3 view4

view5

【NTSEL traffic】 View: 1 Activity: 4

Parameters• DT,classifica2on&LDAarebasedonthepreviousworks–  # ofcodeword: 4,000–  # oftrajectorypooling:15frame–  TheparametersarefollowingoriginalDT–  αandβofLDAaresetat1.0and0.01–  1,000Gibbssampleritera2ons–  Thetopicmodelingparamsarebasedon[GriffithsandSteyvers,04]

• Dominantcodewordsselec2on(DCS)–  Topic-basedcodewordsaresetas1%ofthefrequency–  #oftopic:#ofviewx#ofac2on

ComparisonofDT• DominantDTvsDT–  ByusingHOG,HOF,MBHx,MBHyandcombinedfeatures

–  Dominantcodewordsselec2oniseffec2veontheDT

Co-occurrencefeatureinDDT• Onfine-graineddataset–  MPIIcooking[Rohrbach+,CVPR12]–  Extendedco-occurrencefeatureisimprovedbyDCS–  DDTisimprovedwithextendedco-occurrencefeature

–  [Ni+,ECCV14]and[Ni+,CVPR15]arestate-of-the-artwithmid-levelfeatures

TopicVisualiza2on

××

×

×

×

×

×

××× ××××× ×

××××××

×××

×

××× ×××

×× ×

×

× ×

×××××

××

×

××××××

×××

×××× ×××××

×××

×××

×

×

××

×××

××

× ×

×

×

××××

×× ××

××××××

×

×

×

××

×

×××

×××××××

××××

Topic1(View2)

“sit”View1

“sit”View2

Topic1(View1)

Topic2(View2)

Topic2(View1)

Topic3(View2)

Topic3(View1)

Topic4(View2)

Topic4(View1)

“getup”View2 Topic1(View2) Topic2(View2) Topic3(View2) Topic4(View2)

“crossing” Topic1 Topic2 Topic3 Topic4

“cutoffends” Topic1 Topic2 Topic3 Topic4

INRIAsurgery

INRIAsurgery

IXMAS

NTSELtraffic

MPIIcooking

Conclusion• Dominantcodewordsselec2on(DCS)forac2onrecogni2on–  Weappliedtopicmodelfornoisecancelingintrajectory-basedrepresenta2on–  Extrafeature(co-occurrencefeature)

• Futureworks–  Exploringparametersandmoreeffec2vevisualiza2on(on-going…)–  Seman2cflowwithtopicmodels