Sample Segmentation Results Use of Image Segmentationcvlab.postech.ac.kr/~bhhan/class/cse441_2017f/csed441_lecture14.pdf · §Requirement of high-level inferences 7 ... #1107: 20

Lecture14:ImageSegmentation

Bohyung HanCSE,POSTECH

[email protected]

CSED441:IntroductiontoComputerVision(2017F)

ImageSegmentation

• Whatisimagesegmentation?§ Processofpartitioninganimageintomultiplehomogeneoussegments§ Processofassigningalabeltoeverypixelinanimagesuchthatpixels

withthesamelabelsharecertainvisualcharacteristics§ Compactrepresentationforimagedataintermsofasetofcomponents

• Generalframework§ Tokens:whateverweneedtogroup(pixels,points,surfaceelements)§ Bottomupsegmentation:Tokensbelongtogetherbecausetheyare

locallycoherent§ Topdownsegmentation:Tokensbelongtogetherbecausetheylieon

thesameobject

2

SampleSegmentationResults

3http://www.cs.berkeley.edu/~fowlkes/BSE/

UseofImageSegmentation

• Primitivesforothertasks§ Grouptogethersimilar-lookingpixelsforefficiencyoffurther

processing§ Unsupervisedandbottom-upprocess

4


• High-levelunderstandingofobjectorscene§ Separateanimageintocoherent“objects”or“region”§ Semanticsegmentation§ Objectdetection

5


• Imagemanipulation(incomputergraphics)

6

Challenges

• Imagesegmentationisaninherentlydifficulttaskdueto§ Subjectivityofproblem§ Weakandinconsistentfeatures§ Requirementofhigh-levelinferences

7

Challenges

• Subjectivityofproblem§ Eachpersonhasadifferentconceptofsegmentinanimage.§ Evenapersonhasdifferentconceptsofsegmentdependingonthe

situation

8

Inputimage

Annotationsbyhuman

Challenges

• Weakandinconsistentfeatures§ Unclearobjectboundaries§ Imagenoisesandartifacts§ Backgroundclutter

9

Challenges

• Requirementofhigh-levelinferences§ Recoveringmissingboundaries§ Consideringscenecontext§ Handlingrepetitivepatterns

10

Evaluation

• Howtoevaluateimagesegmentationalgorithm?§ Weneedtoconstructdatasetsandevaluationprotocols.

• Datasets§ Berkeleysegmentationdataset§ COCO

• Evaluationmethods§ Precisionandrecallofregionboundaries§ Othermeasuresforclusteringalgorithmevaluation

• (Adjusted)RANDindex• Normalizedmutualinformation

1112

DatasetsandBenchmarks

BerkeleySegmentationDatasetandBenchmark

• BSD300§ Originaldataset§ 200trainingand100testingimages§ 12,000hand-labeledsegmentationsof1,000Coreldatasetimages

from30humansubjects.§ http://www.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/§ Relatedpaper

• D.MartinandC.FowlkesandD.TalandJ.Malik.ADatabaseofHumanSegmentedNaturalImagesanditsApplicationtoEvaluatingSegmentationAlgorithmsandMeasuringEcologicalStatistics.ICCV2001

13


• Anexample§ Trainingimage#159029§ 6annotations

14

#1107:20segments #1109:13segments




• BSD500§ Anextendedversion:200additionaltestingimages§ Newimagesweresegmentedbyfivedifferentsubjectsonaverage.§ Performanceisevaluatedbymeasuringprecision/recallondetected

boundariesandthreeadditionalregion-basedmetrics.§ http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/reso

urces.html§ Relatedpaper

• P.Arbelaez,M.Maire,C.FowlkesandJ.Malik.ContourDetectionandHierarchicalImageSegmentation. TPAMI2011.

§ Downloads• Pre-compiledMatlab package• Sourcecode(forLinux/Mac,32/64bits)• Pre-computedresultsontheBSD500andPASCALVOC2012

15

• WhatisCOCO?

§ CommonObjectsinCOntext§ Anewimagerecognitionandsegmentationdatasetthatwasreleasedin

Summer2014.

• Resources§ http://cocodataset.org/§ Paper

• Tsung-YiLin,MichaelMaire,SergeBelongie,JamesHays,PietroPerona,DevaRamanan,PiotrDollár,C.LawrenceZitnick:MicrosoftCOCO:CommonObjectsinContext. ECCV(5)2014:740-755

• arXiv link:http://arxiv.org/abs/1405.0312

COCO

16

COCO

• Characteristics§ Morethan70categories§ Objectsegmentation§ Recognitionincontext§ Multipleobjectsperimage§ Morethan300,000images§ Morethan2Millioninstances

17

COCO

• Tools§ InPythonandMatlab (notyetreadythough)§ Downloadandset-uppackage§ CommonAPIinPythonandMatlab withminordifferencesforreadingand

visualizingCOCO

• Annotation§ Instanceannotation

• Instances:storinganarrayofinstanceobjectthatcontainscategoryidandsegmentationofaninstance

• Categories:mappingofcategoryidtocategoryname§ Sentenceannotation

• Anarrayofsentenceannotationthatdescribesanimage• Eachimagehasatleastfivesentences(fewhasmorethanfive).

18

SomeExampleImages

19

COCO

20

http://cocodataset.org/

21

ImageSegmentationAlgorithms

ImageSegmentationAlgorithms

• Segmentationasgrouping§ k-meansclustering§ Mean-shift

• Segmentationasgraphpartitioning§ Spectralclustering§ Normalizedcut

• Segmentationbyothermethods§ Boundarydetection:Watershed§ Labeling:Graph-cut

22

WatershedAlgorithm

• Mainidea§ Imageasatopographicrelief:thegreylevelofapixelisinterpretedasits

altitudeintherelief.§ Thewatershedofareliefcorrespondtothelimitsoftheadjacent

catchmentbasinsofthedropsofwater.

23

WatershedSegmentation

24

Image Gradient Watershedboundaries

seg = watershed(bnd_im)

Implementation

• Procedure§ Chooselocalminimaasregionseeds§ Addneighborstopriorityqueue,sortedbyvalue§ Taketopprioritypixelfromqueue

• Ifalllabeledneighborshavethesamelabel,assignittothepixel• Addallnon-markedneighbors

§ Repeatstep3untilfinished

25

Thenon-labeledpixelsarethewatershedlines.

Results

26

Results

27

Characteristics

• Pros§ Fast(<1secfor512x512image)§ Amongthebestmethodsforhierarchicalsegmentation

• Cons§ Onlyasgoodasthesoftboundaries§ Noteasytogetavarietyofregionsformultiplesegmentations§ Notop-downinformation

• Usage§ Preferredalgorithmforhierarchicalsegmentation

28

Graph-Cut

• Averygeneraltoolthatcanbeappliedto§ Stereodepthreconstruction§ Texturesynthesis§ Videosynthesis§ Imagedenoising§ Foreground/backgroundsegmentation§ Matching

• Generalcharacteristics§ Formulatedusingagraph§ Usedforbinarylabelingproblem§ Solvedbyenergyminimization

29

[Boykov04]Y.Boykov,V.Kolmogorov:AnExperimentalComparisonofMin-Cut/Max-FlowAlgorithmsforEnergyMinimizationinVision.TPAMI26(9),2004

ImageRepresentationbyGraph-Cut

• Representationasagraph§ Anodeiscreatedforeachpixel.§ Anedgerepresentsthe

relationshipbetweenadjacentpixels.

§ Twospecialnodes:source(!)andsink(")

• Largesearchspace§ #ofnodesforlabeling:#§ thenumberofallpossible

binarysegmentations:2%

§ Thisisintractable.

30

t

s


ImageRepresentationbyGraph-Cut

• Representationasagraph§ Anodeiscreatedforeachpixel.§ Anedgerepresentsthe

relationshipbetweenadjacentpixels.

§ Twospecialnodes:source(!)andsink(")

• Largesearchspace§ #ofnodesforlabeling:#§ thenumberofallpossible

binarysegmentations:2%

§ Thisisintractable.

31

B

F


EnergyMinimizationProblem

• Energyfunction

• Dataterm§ Definedforeachnode§ Howsimilariseachlabelednodetotheforegroundorbackground?§ Probabilitythatthisfeaturebelongstoforeground(resp.background)

• Smoothnessterm(a.k.a.regularizationterm)§ Definedforeachedge§ Penaltyforhavingdifferentlabels§ Encouragingspatiallycoherentsegments§ Penaltyisdown-weightedifthetwopixelcolorsareverydifferent.

32

& ' = &) ' + +&, '

dataterm smoothnessterm


TwoTerms

• Dataterm

§ Definedforeachnode

• Smoothnessterm

§ Definedforeachedge§ -./:visualsimilaritybetweentwonodes

(pixels),0 and1

33

&, ' = 2 -./ ' 0 − ' 14

.,/ ∈ℇ

&) ' =

289(;)4

=∈>

if' ; = B, FGenergy

28J(;)4

=∈>

if' ; = K, (BGenergy)

MN(O)

MP(O)


Solution

• Approach§ Weshouldminimizeenergy

bycuttingthegraphintotwopartitions.

§ Solvedbymax-flow/min-cutalgorithm

• Max-flow/min-cutalgorithm§ Energyoptimizationisequivalent

tographmincut.§ Cut:removeedgestodisconnect

tworegions§ Minimum:minimizesumof

cutedgeweight

34

B

F


35

Documents

Sample Segmentation Results Use of Image Segmentationcvlab.postech.ac.kr/~bhhan/class/cse441_2017f/csed441_lecture14.pdf · §Requirement of high-level inferences 7 ... #1107: 20