Upload
shea-mcconnell
View
45
Download
2
Embed Size (px)
DESCRIPTION
Performance Evaluation of Grouping Algorithms. Vida Movahedi Elder Lab - Centre for Vision Research York University Spring 2009. Overview. Grouping and evaluation methods Region-based measures Boundary-based measures Mixed measures Alignment measure. Overview. - PowerPoint PPT Presentation
Citation preview
Performance Evaluationof Grouping Algorithms
Vida MovahediElder Lab - Centre for Vision Research
York University
Spring 2009
Centre for Vision Research, York University 2
Overview Grouping and evaluation methods
Region-based measures
Boundary-based measures
Mixed measures
Alignment measure
Centre for Vision Research, York University 3
Overview Grouping and evaluation methods
Region-based measures
Boundary-based measures
Mixed measures
Alignment measure
Centre for Vision Research, York University 4
Grouping Edge segments: Example
Centre for Vision Research, York University 5
Perceptual Organization/ Grouping
A process of assembling features into groups which are perceptually significant based on various cues (Lowe, 1985)
The problem of aggregating primitive image features that project from a common structure in the visual scene (Elder, 2002)
Centre for Vision Research, York University 6
Evaluation Measure How good is each grouping?
Which algorithm has a better performance?
What is the best grouping that can be achieved?
Note differences with regional segmentation evaluation
Centre for Vision Research, York University 7
Evaluation Methods Three main categories (Zhang, 1996)
Analytical methodsConsider the algorithms themselves, e.g. based on the a priori knowledge they use (not based on output of the algorithms)
Empirical goodness methodsBased on the outputs of the algorithms, e.g. based on the intra-region uniformity of the segments, or the inter-region contrast between the segments.
Empirical discrepancy methodsA reference segmentation or ground truth is assumed, to compare the outputs with
Centre for Vision Research, York University 8
Goal: Measure Discrepancy
Centre for Vision Research, York University 9
SOD: Salient Object Dataset Based on Berkeley Segmentation Dataset
(BSD)
300 images
7 subjects
1
1
1
1
1
Centre for Vision Research, York University 10
Overview Grouping and evaluation methods
Region-based measures
Boundary-based measures
Mixed measures
Alignment measure
Centre for Vision Research, York University 11
Region-based Discrepancy (Young, 2005),(Ge, 2006), (Goldmann, 2008)
A and B two boundaries
RB the region corresponding to a boundary B and |RB| the area of this region
1: maximum discrepancy,
0: maximum consistency
BA
BAAB RR
RRError
1
Centre for Vision Research, York University 12
Interpretation
BA
BAB
BA
BAAAB RR
RRR
RR
RRRError
||||
Centre for Vision Research, York University 13
Evaluation by this measure
Not sensitive to spikes, wiggles, shape
>=
(more error)
Centre for Vision Research, York University 14
Examples of near-optimal cases
Centre for Vision Research, York University 15
Overview Grouping and evaluation methods
Region-based measures
Boundary-based measures
Mixed measures
Alignment measure
Centre for Vision Research, York University 16
• Distance of one point a from B
• Distance Signature of all a in A
• One directional Hausdorff
• Two directional Hausdorff
Boundary-based Distance
),(,),(),( abdMinMaxbadMinMaxMaxBAHAaBbBbAa
}),({),( AaadBASD BB
),(min)( badadBb
B
)(max)),(max(),( adBASDBAh BAa
B
),(),,(max),( ABhBAhBAH
Centre for Vision Research, York University 17
Evaluation by this measure
Not sensitive to wiggles, shape
Not sensitive to the distance distribution, but only to the maximum value
Centre for Vision Research, York University 18
Geodesic Distance: the min. distance between two points a and b without cross
Euclidean vs. Geodesic Distance
Euclidean Distance: the min. distance between two points a and b
),(min),( ieAa
e abdAbdi
),(min),( igAa
g abdAbdi
Centre for Vision Research, York University 19
Evaluation by this measure
Almost the same
by De
Almost the same by De & Dg
Centre for Vision Research, York University 20
Overview Grouping and evaluation methods
Region-based measures
Boundary-based measures
Mixed measures
Alignment measure
Centre for Vision Research, York University 21
A mixture of boundary-based and region-based Penalizing the over-detected and under-detected
regions by their Euclidean or Geodesic distances
fpfn N
kkB
fp
N
jjA
fn
qdN
pdN
BAD11
)(1
)(1
2
1),(
pj, j=1..Nfp are pixels in the false negative region (RB-RA)
qk, k=1..Nfn are pixels in the false positive region (RA-RB)
Centre for Vision Research, York University 22
Evaluation by this measure Not penalizing effectively, e.g. narrow false
positives below
Centre for Vision Research, York University 23
Correspondence Problem The false negative and false positive regions can be
very small, yet the boundaries be very different
Segments on one boundary should correspond to segments on the other
Centre for Vision Research, York University 24
Alignment The order of matching points on the two boundaries
should be monotonically non-decreasing.
nmbaMatchbaMatchji njmi )( and ,)( , If
)(),( :sother wordin
B,,,),(minarg)(
*
**
adbad
bAabadaMatchb
B
Bb
Centre for Vision Research, York University 25
Correspondence (Cont.) Note that if correspondence is maintained,
De will work almost like Dg!
Centre for Vision Research, York University 26
Overview Grouping and evaluation methods
Region-based measures
Boundary-based measures
Mixed measures
Alignment measure
Centre for Vision Research, York University 27
Alignment Distance Main idea: We need to find the ‘alignment’ that leads to
minimum total distance.
Method: Use N samples on each boundary (equally spaced) Find the NxN matrix of Euclidean distances. The diagonals show correspondences with some rotations The one with min sum of distances is the best
correspondence and its sum is our measure of discrepancy.
Centre for Vision Research, York University 28
Alignment Measure (cont.)
4321
4321
4321
4321
4321
4321
4321
4321
432 14 3 2 1
dddd
cccc
bbbb
aaaa
dddd
cccc
bbbb
aaaa
dddd
dddd
dddd
dddd
dddd
dddd
dddd
dddd
d
c
b
a
Note: Order of both samples increases clockwise
Nirotii
NrotbadBAD
..11..0
),(min),(
Centre for Vision Research, York University 29
Evaluation by this simple measure Samples falling out of phase
Solution: finer sampling on one boundary
Reference handgrouping with error=1177.8474
A better curve with less error=872.5259
>=
(more error)
Centre for Vision Research, York University 30
Bimorphism (Tagare, 2002)
A method to let correspondence of 1 to many and many to 1 symmetric
Centre for Vision Research, York University 31
A symmetric Alignment Distance Edit cost of changing one string to another
Edit operation, cost of operation
A sequence of operations taking A to B
Symmetric:
naaaA ....21
mbbbB ...21
bas : edit ofcost :)(s),()( :choose we badba
ksssS ...21
k
iisS
1
)()(
BASSBA to takingsequenceedit an is |)(min:),(
A)(B,B)(A, then ,)()( If abba
Centre for Vision Research, York University 32
Example
4321 aaaaA
321 bbbB
),,
,,(
342423
1211
bababa
babaS
)3,4(),2,4(),2,3(),1,2(),1,1(T
5|| T
),(),(),(),(),()( 3424231211 badbadbadbadbadS
)(),( SBA
Centre for Vision Research, York University 33
Cyclic shifts Cyclic shifts
Alignment Distance
Dynamic programming
Complexity:
(Maes, 1990) Complexity:
])[],([
||
1),( BA
TBADA
])[,(])[],([ BABA
mlnkBABA lk 0,0|)(),(min:])[],([
AAnkaaaaaaa knknk )( and ,1,......)...( 0
1121
).( 2 nmO
)log.( mmnO
Centre for Vision Research, York University 34
Examples
Alignment Distance=7.73
Centre for Vision Research, York University 35
Examples
Alignment Distance=3.25
Centre for Vision Research, York University 36
Evaluation by this measure Note: If using Euclidean distance, there is
no sensitivity to region elite curve with HD8-Mae
Centre for Vision Research, York University 37
References
(Elder, 2002) J. H. Elder and R. M. Goldberg (2002), "Ecological statistics of Gestalt laws for the perceptual organization of contours." J Vis, vol. 2, pp. 324-353.
(Zhang, 1996) Y. J. Zhang. (1996), “A survey on evaluation methods for image segmentation”, Pattern recognition 29(8), pp. 1335.
(BSD) D. Martin (2001), "A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics," Proceedings of the 8th IEEE International Conference on Computer Vision, vol. 2, pp. 416-423.
(Ge, 2006) F. Ge, S. Wang and T. Liu (2006), "Image-Segmentation Evaluation From the Perspective of Salient Object Extraction," Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 1, pp. 1146-1153.
(Goldmann, 2008) L. Goldmann. (2008), Towards fully automatic image segmentation evaluation. Lecture notes in computer science 5259 LNCS, pp. 566.
(Young, 2005) D. P. Young (2005), "PETS Metrics: On-line performance evaluation service," Proceedings - 2nd Joint IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, VS-PETS, vol. 2005, pp. 317, 2005.
(Huttenlocher, 1993) D. P. Huttenlocher (1993), “Comparing images using the Hausdorff distance”, IEEE transactions on pattern analysis and machine intelligence 15(9), pp. 850.
(Tagare, 2002) H. D. Tagare. (2002), “Non-rigid shape comparison of plane curves in images”, Journal of mathematical imaging and vision 16(1), pp. 57.
(Maes, 1990) M. Maes (1990), “On a cyclic string-to-string correction problem”, Information processing letters 35(2), pp. 73.