Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
MachineLearning
Slides: James Hays, Isabelle Guyon, Erik Sudderth, Mark Johnson, Derek Hoiem Photo: CMU Machine Learning
Department protests G20
Clustering:grouptogethersimilarpointsandrepresentthemwithasingletoken
KeyChallenges:1)Whatmakestwopoints/images/patchessimilar?2)HowdowecomputeanoverallgroupingfrompairwisesimilariCes?
Slide: Derek Hoiem
Howdowecluster?
• K-means– IteraCvelyre-assignpointstothenearestclustercenter
• AgglomeraCveclustering– StartwitheachpointasitsownclusteranditeraCvelymergetheclosestclusters
• Mean-shiHclustering– EsCmatemodesofpdf
• Spectralclustering– Splitthenodesinagraphbasedonassignedlinkswithsimilarityweights
ClusteringforSummarizaConGoal:clustertominimizevarianceindatagivenclusters– PreserveinformaCon
( )∑∑ −=N
j
K
ijiN ij
21
,
** argmin, xcδcδc
δ
Whether xj is assigned to ci
Cluster center Data
Slide: Derek Hoiem
K-meansalgorithm
Illustration: http://en.wikipedia.org/wiki/K-means_clustering
1. Randomly select K centers
2. Assign each point to nearest center
3. Compute new center (mean) for each cluster
K-meansalgorithm
Illustration: http://en.wikipedia.org/wiki/K-means_clustering
1. Randomly select K centers
2. Assign each point to nearest center
3. Compute new center (mean) for each cluster
Back to 2
BuildingVisualDicConaries1. Samplepatchesfrom
adatabase– E.g.,128dimensional
SIFTvectors
2. Clusterthepatches– Clustercentersare
thedicConary
3. Assignacodeword(number)toeachnewpatch,accordingtothenearestcluster
Examplesoflearnedcodewords
Sivic et al. ICCV 2005 http://www.robots.ox.ac.uk/~vgg/publications/papers/sivic05b.pdf
Most likely codewords for 4 learned “topics” EM with multinomial (problem 3) to get topics
AgglomeraCveclustering
AgglomeraCveclustering
AgglomeraCveclustering
AgglomeraCveclustering
AgglomeraCveclustering
AgglomeraCveclusteringHowtodefineclustersimilarity?- Averagedistancebetweenpoints,
maximumdistance,minimumdistance- Distancebetweenmeansormedoids
Howmanyclusters?- Clusteringcreatesadendrogram(atree)- Thresholdbasedonmaxnumberofclusters
orbasedondistancebetweenmergesdi
stan
ce
AgglomeraCveclusteringdemohZp://home.dei.polimi.it/maZeucc/Clustering/tutorial_html/AppletH.html
Conclusions:AgglomeraCveClusteringGood• Simpletoimplement,widespreadapplicaCon• ClustershaveadapCveshapes• ProvidesahierarchyofclustersBad• Mayhaveimbalancedclusters• SCllhavetochoosenumberofclustersorthreshold
• Needtousean“ultrametric”togetameaningfulhierarchy
• VersaCletechniqueforclustering-basedsegmentaCon
D. Comaniciu and P. Meer, Mean Shift: A Robust Approach toward Feature Space Analysis, PAMI 2002.
MeanshiHsegmentaCon
MeanshiHalgorithm• Trytofindmodesofthisnon-parametric
density
KerneldensityesCmaCon
Kernel density estimation function
Gaussian kernel
Region of interest
Center of mass
Mean Shift vector
Slide by Y. Ukrainitz & B. Sarel
MeanshiH
Region of interest
Center of mass
Mean Shift vector
Slide by Y. Ukrainitz & B. Sarel
MeanshiH
Region of interest
Center of mass
Mean Shift vector
Slide by Y. Ukrainitz & B. Sarel
MeanshiH
Region of interest
Center of mass
Mean Shift vector
MeanshiH
Slide by Y. Ukrainitz & B. Sarel
Region of interest
Center of mass
Mean Shift vector
Slide by Y. Ukrainitz & B. Sarel
MeanshiH
Region of interest
Center of mass
Mean Shift vector
Slide by Y. Ukrainitz & B. Sarel
MeanshiH
Region of interest
Center of mass
Slide by Y. Ukrainitz & B. Sarel
MeanshiH
Simple Mean Shift procedure: • Compute mean shift vector
• Translate the Kernel window by m(x)
2
1
2
1
( )
ni
ii
ni
i
gh
gh
=
=
⎡ ⎤⎛ ⎞⎢ ⎥⎜ ⎟
⎜ ⎟⎢ ⎥⎝ ⎠= −⎢ ⎥⎛ ⎞⎢ ⎥⎜ ⎟⎢ ⎥⎜ ⎟⎝ ⎠⎣ ⎦
∑
∑
x - xx
m x xx - x
g( ) ( )kʹ= −x x
CompuCngtheMeanShiH
Slide by Y. Ukrainitz & B. Sarel
• AZracConbasin:theregionforwhichalltrajectoriesleadtothesamemode
• Cluster:alldatapointsintheaZracConbasinofamode
Slide by Y. Ukrainitz & B. Sarel
AZracConbasin
AZracConbasin
MeanshiHclustering• ThemeanshiHalgorithmseeksmodesofthe
givensetofpoints1. Choosekernelandbandwidth2. Foreachpoint:
a) Centerawindowonthatpointb) Computethemeanofthedatainthesearchwindowc) CenterthesearchwindowatthenewmeanlocaCond) Repeat(b,c)unClconvergence
3. Assignpointsthatleadtonearbymodestothesamecluster
• Computefeaturesforeachpixel(color,gradients,texture,etc)• SetkernelsizeforfeaturesKfandposiConKs• IniCalizewindowsatindividualpixellocaCons• PerformmeanshiHforeachwindowunClconvergence• MergewindowsthatarewithinwidthofKfandKs
SegmentaConbyMeanShiH
http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html
MeanshiHsegmentaConresults
http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html
Mean-shiH:otherissues• Speedups
– BinnedesCmaCon– Fastsearchofneighbors– UpdateeachwindowineachiteraCon(fasterconvergence)
• Othertricks– UsekNNtodeterminewindowsizesadapCvely
• LotsoftheoreCcalsupportD.ComaniciuandP.Meer,MeanShiH:ARobustApproachtowardFeatureSpaceAnalysis,PAMI2002.
MeanshiHprosandcons
• Pros– Goodgeneral-pracCcesegmentaCon– Flexibleinnumberandshapeofregions– Robusttooutliers
• Cons– Havetochoosekernelsizeinadvance– Notsuitableforhigh-dimensionalfeatures
• Whentouseit– Oversegmentatoin– MulCplesegmentaCons– Tracking,clustering,filteringapplicaCons
Spectralclustering Grouppointsbasedonlinksinagraph
A B
Cutsinagraph
A B
Normalized Cut • a cut penalizes large segments • fix by normalizing for size of segments
• volume(A) = sum of costs of all edges that touch A
Source: Seitz
NormalizedcutsforsegmentaCon
VisualPageRank• Determiningimportancebyrandomwalk
– What’stheprobabilitythatyouwillrandomlywalktoagivennode?
• Createadjacencymatrixbasedonvisualsimilarity• EdgeweightsdetermineprobabilityoftransiCon
Jing Baluja 2008
Whichalgorithmtouse?• QuanCzaCon/SummarizaCon:K-means
– Aimstopreservevarianceoforiginaldata– Caneasilyassignnewpointtoacluster
Quantization for computing histograms
Summary of 20,000 photos of Rome using “greedy k-means”
http://grail.cs.washington.edu/projects/canonview/
Whichalgorithmtouse?• ImagesegmentaCon:agglomeraCveclustering
– Moreflexiblewithdistancemeasures(e.g.,canbebasedonboundarypredicCon)
– AdaptsbeZertospecificdata– Hierarchycanbeuseful
http://www.cs.berkeley.edu/~arbelaez/UCM.html
Thingstoremember• K-meansusefulforsummarizaCon,buildingdicConariesofpatches,generalclustering
• AgglomeraCveclusteringusefulforsegmentaCon,generalclustering
• Spectralclusteringusefulfordeterminingrelevance,summarizaCon,segmentaCon
ClusteringKeyalgorithm• K-means