ETISEO

ETISEO

François BREMOND

ORION Team, INRIA Sophia Antipolis, France

2

Fair Evaluation

Unbiased and transparent evaluation protocol

Large participation

Meaningful evaluation

Tasks evaluatedTasks evaluated

GT & Metrics are designed to evaluate GT & Metrics are designed to evaluate tasks all tasks all along the video processing chain:along the video processing chain:

Task 1Task 1: : Detection Detection of physical objects,of physical objects,

Task Task 22: : LocalisationLocalisation of physical objects,of physical objects,

Task Task 33: : ClassificationClassification of physical objects, of physical objects,

Task Task 44: : TrackingTracking of physical objects, of physical objects,

Task Task 55: : EventEvent recognition. recognition.

Matching ComputationMatching Computation

To evaluate the matching between a candidate result and a reference data, we may use following distances:

D1-The Dice coefficient: Twice the shared, divided by the sum of both intervals: 2*card(RDC) / (card(RD) + card(C)). D2-The overlapping: card(RDC) / card(RD). D3-Bertozzi and al. metric: (card(RDC))^2 / (card(RD) * card(C)). D4-The maximum deviation of the candidate object or target according to the shared frame span: Max { card(C\RD) / card(C), card(RD\C) / card(RD) }.

RD

C

metrics (1)metrics (1)

T1- DETECTION OF PHYSICAL OBJECTS OF INTERESTC1.1 Number of physical objectsC1.2 Number of physical objects using their bounding box

T2- LOCALISATION OF PHYSICAL OBJECTS OF INTEREST C2.1 Physical objects area (pixel comparison based on BB) C2.2 Physical object area fragmentation (splitting) C2.3 Physical object area integration (merge) C2.4 Physical objects localisation

2D and 3D Centroïd or bottom centre point of BB

metrics (2)metrics (2)

T4- CLASSIFICATION OF PHYSICAL OBJECTS OF INTEREST

C4.1 Object Type over the sequenceC4.2 Object classification per typeC4.3 Time Percentage Good Classification

card{ RDC, Type(C) = Type(RD) } / card(RDC) T5- EVENT RECOGNITION

C5.1 Number of Events recognized over the sequenceC5.2 Scenario parameters

T3- TRACKING OF PHYSICAL OBJECTS OF INTEREST C3.1 Frame-To-Frame Tracking: Link between two frames C3.2 Number of object being tracked during time C3.3 Detection time evaluation C3.4 Physical object ID fragmentation C3.5 Physical object ID confusion criterion C3.6 Physical object 2D trajectory C3.7 Physical object 3D trajectory

7

Metric Evaluation

Distance for matching groundtruth and algorithms results Similar measures: D1, D2, D3, D4.

Few main metrics measure general trends Discriminant and meaningful Detection M1.2.1: CNumberObjectsBoundingBox Localization M2.4.3: CCentroid2DLocalisationPix. Tracking M3.3.1: CtrackingTime Object Classification M4.1.3: CobjectTypeOverSequenceBBoxID

Event Recognition M5.1.2: CNumberNamedEvents

8

Metric Evaluation (cont’d)

Secondary metrics: Complementary information

Pixel-based (M2.1.1) versus object-based (M1.2.1) metrics

Potential algorithm errors. Example: M3.3.1 complemented (eg., about stability) by

M3.2.1, M3.4.1 and M3.5.1. Non-informative Metrics:

Add noise to the evaluation or non-discriminative Example: M1.1.1 CNumberObjects gives the

object number per frame without position information.

The same for M4.1.1 and M5.1.1.

9

Evaluation on ETI-VS2-BE-19-C1

10

Global Results: Video

T1-Detection

0

1

2

3

4

5

6

0

0,06

0,12

0,18

0,24 0,3

0,36

0,42

0,48

0,54 0,6

0,66

0,72

0,78

0,84 0,9

0,96

scoresGlobal MO-1-C1 RD-6-C7 RD-10-C4

AP-11-C4 AP-11-C7 BE-19-C1 BE-19-C4

Remarks:

For similar scenes, very dissimilar results!

For different scenes, results can spread over a large range or concentrate in a narrow range.

11

Detection of Physical Objects (ETI-VS2-BE-19-C1.xml)

Group

Precision (global

)

Sensitivity (global

) F-Score

12 0,937775 0,8353440,8836008

41

17 0,755875 0,9439190,8394955

79

8 0,95565 0,7306810,8281592

38

1 0,985819 0,7026410,8204835

74

14 0,998807 0,545810,7058822

33

13 0,989374 0,5464620,7040534

21

3 0,564533 0,8999020,6938162

17

19 0,921622 0,5559180,6935125

4

28 1 0,5288560,6918323

24

29 1 0,5167920,6814276

45

32 0,999311 0,4727750,6418772

52

9 1 0,452560,6231205

6

20 1 0,4388650,6100155

33

15 1 0,3625690,5321844

25

23 0,548913 0,0987940,1674501

31

Group

Precision (globa

l)

Sensitivity (globa

l) F-Score

1 0,963403 0,6866640,8018270

26

8 0,854584 0,6534070,7405762

6

14 0,995221 0,5432020,7028054

54

13 0,98031 0,5357030,6928106

92

29 0,979811 0,5063580,6676698

79

12 0,706442 0,6292790,6656316

93

28 0,945129 0,4998370,6538706

71

19 0,836757 0,5047280,6296524

93

9 0,971182 0,4395170,6051624

04

20 0,976226 0,4284320,5955135

81

32 0,9 0,4254970,5778169

25

15 0,998201 0,3619170,5312273

07

17 0,429243 0,5360290,4767292

45

3 0,278175 0,443430,3418799

49

23 0,354049 0,0612980,1045029

61

M1.1.1: NumberObjects M1.2.1: NumberObjectsBoundingBoxD1

12


Group

Precision

(global)

Sensitivity

(global) F-Score

10,96340

30,68666

40,8018270

26

80,85458

40,65340

70,7405762

6

140,99522

10,54320

20,7028054

54

13 0,980310,53570

30,6928106

92

290,97981

10,50635

80,6676698

79

120,70644

20,62927

90,6656316

93

280,94512

90,49983

70,6538706

71

190,83675

70,50472

80,6296524

93

90,97118

20,43951

70,6051624

04

200,97622

60,42843

20,5955135

81

32 0,90,42549

70,5778169

25

150,99820

10,36191

70,5312273

07

170,42924

30,53602

90,4767292

45

30,27817

5 0,443430,3418799

49

230,35404

90,06129

80,1045029

61

M1.2.1: NumberObjectsBoundingBoxD1

Group

Precision

(global)

Sensitivity

(global)

Specificity

(global)

F-Score

10,84832

7 0,9726450,99284

40,906

24

80,77460

1 0,9362180,98878

90,847

78

120,86070

1 0,8152010,99457

10,837

33

190,87554

5 0,7674140,99551

10,817

92

280,72805

3 0,9143770,98594

50,810

65

140,74924

5 0,8625540,98813

40,801

92

130,78581

6 0,7952390,99116

80,790

5

30,76832

1 0,7811950,99030

70,774

7

29 0,8918 0,6299870,99685

50,738

37

200,63255

8 0,7863170,98120

40,701

11

150,67417

7 0,6964510,98614

90,685

13

90,86975

2 0,5538470,99658

70,676

75

170,49587

9 0,8664460,96375

30,630

76

320,76591

4 0,4252880,99465

70,546

9

230,27053

4 0,0867960,99103

90,131

43

M2.1.1: ObjectsArea

13


Group Splitting value

15 1

13 0,999734

14 0,998935

29 0,998915

9 0,998647

20 0,996937

32 0,996753

28 0,992038

23 0,989362

12 0,972667

1 0,969471

19 0,928911

8 0,919553

17 0,822301

3 0,352762

M2.2.1: SplittingD5 M2.3.1: MergingD2Group Merging value

19 0,862026

15 0,859442

28 0,841472

14 0,834221

3 0,822119

17 0,820084

29 0,814503

13 0,805644

12 0,799294

8 0,786615

1 0,783842

20 0,776757

23 0,760934

9 0,747107

32 0,700614

14

Summary on Detection of Physical Objects

Main metric measures: Detection M1.2.1:

CNumberObjectsBoundingBox Problems: static objects, contextual objects,

background, masks… Advantages: objects vs pixels, large objects

and bounding boxes Secondary metrics:

M2.1.1 (area): indication on the precision and handling shadows

Split/Merge measures (M2.2.1, M2.3.1): Advantage: indicate potential merge Inconvenients: threshold-dependent, non-

detected objects not taken into account

15

Localisation of Physical Objects

T2-Localisation

0123456789

10111213

0

0,06

0,12

0,18

0,24 0,3

0,36

0,42

0,48

0,54 0,6

0,66

0,72

0,78

0,84 0,9

0,96


AP-11-C4 AP-11-C7 BE-19-C1 BE-19-C4

16

Localisation of Physical Objects (ETI-VS2-BE-19-C1.xml)

Group value

20 0,032379

15 0,022457

13 0,021595

23 0,021043

28 0,020659

17 0,020328

32 0,019422

3 0,019145

14 0,018123

29 0,017538

8 0,015945

12 0,012152

9 0,012141

1 0,010686

19 0,009591

M2.3.1: MergingD2 M2.4.3: Centroid2DLocalisationPixD1Group Value (pixel)

20 19,583632

15 14,954423

13 13,282747

23 13,255976

28 13,135057

17 13,06773

3 12,369682

14 12,31259

32 11,936302

29 11,463824

8 9,851749

9 7,693683

12 7,628216

1 6,658153

19 5,964608

17


Group value

20 0,032379

15 0,022457

13 0,021595

23 0,021043

28 0,020659

17 0,020328

32 0,019422

3 0,019145

14 0,018123

29 0,017538

8 0,015945

12 0,012152

9 0,012141

1 0,010686

19 0,009591

M2.4.1: Centroid2DLocalisationD1M2.4.3: Centroid2DLocalisationPixD1Group Value (pixel)

20 19,583632

15 14,954423

13 13,282747

23 13,255976

28 13,135057

17 13,06773

3 12,369682

14 12,31259

32 11,936302

29 11,463824

8 9,851749

9 7,693683

12 7,628216

1 6,658153

19 5,964608

18


Group value

9 9,544993

23 8,881964

1 8,441473

8 8,107458

15 7,347973

14 6,725589

29 6,613178

28 6,344898

19 6,032267

32 10,068101

13 1,429808

20 1,279278

17 1,265509

12 1,214232

3 1,100734

M2.4.2.: Centroid3DLocalisationD1

19

Summary on Localisation of Physical Objects

M2.4.1, M2.4.2, M2.4.3, main metrics: Problems: low utilisation of 3D info and

calibration Good performance: good precision on

reliable TP (handling shadow and merge)

Advantages: complementary to the Detection; normalised, pixel or meter metrics

20

Tracking of Physical Objects

T3-Tracking

0

1

2

3

4

5

0

0,06

0,12

0,18

0,24 0,3

0,36

0,42

0,48

0,54 0,6

0,66

0,72

0,78

0,84 0,9

0,96


AP-11-C4 AP-11-C7 BE-19-C1 BE-19-C4

21

Tracking of Physical Objects (ETI-VS2-BE-19-C1.xml)

Group PrecisionSensitivit

y F-Score

1 1 0,833333 0,909091

12 0,75 1 0,857143

32 0,833333 0,833333 0,833333

9 1 0,666667 0,8

14 1 0,666667 0,8

13 0,8 0,666667 0,727273

17 0,545455 1 0,705882

28 0,666667 0,666667 0,666667

20 0,666667 0,666667 0,666667

19 0,5 0,666667 0,571429

29 0,6 0,5 0,545455

8 0,375 1 0,545455

15 0,4 0,666667 0,5

3 0,024096 1 0,047059

M3.2.1: NumberObjectTrackedD1 M3.3.1: TrackingTime

GroupTracking

Time

1 0,473258

12 0,365271

14 0,332216

19 0,319806

9 0,312904

32 0,297917

29 0,297554

13 0,296646

17 0,29289

8 0,26263

28 0,246822

20 0,20839

15 0,12739

3 0,110475

23 0,018877

22


Group Persistence

9 0,875

14 0,833333

20 0,777778

1 0,766667

29 0,75

32 0,7

19 0,666667

12 0,666667

13 0,625

28 0,611111

8 0,59

17 0,5625

15 0,520833

23 0,4375

3 0,113126

M3.4.1: PhysicalObjectIdFragmentation

M3.5.1: PhysicalObjectIdConfusionGroup Confusion

3 0,985106

15 0,875

14 0,875

17 0,791667

20 0,766667

19 0,761905

28 0,75

32 0,75

8 0,733333

13 0,666667

23 0,638889

29 0,633333

9 0,583333

12 0,569444

1 0,5625

23



y F-Score

9 0,75 0,5 0,6

1 0,6 0,50,54545454

5

12 0,375 0,50,42857142

9

14 0,5 0,333333 0,39999976

8 0,25 0,6666670,36363641

3

13 0,4 0,3333330,36363616

5

29 0,4 0,3333330,36363616

5

20 0,333333 0,333333 0,333333

28 0,333333 0,333333 0,333333

32 0,333333 0,333333 0,333333

19 0,25 0,3333330,28571416

3

17 0,181818 0,3333330,23529388

2

15 0,1 0,1666670,12500009

4

3 0,008032 0,3333330,01568602

9

23 0 0 0

M3.6.1: PhysicalObject2DTrajectoriesGroup value

20 0,032379

15 0,022457

13 0,021595

23 0,021043

28 0,020659

17 0,020328

32 0,019422

3 0,019145

14 0,018123

29 0,017538

8 0,015945

12 0,012152

9 0,012141

1 0,010686

19 0,009591

M2.4.1: Centroid2DLocalisationD1

24

Summary on Tracking of Physical Objects

M3.3.1, main metric: Problems: propagation of detection errors Advantages: good global overview

M3.2.1, secondary metric: Good performance: consistent TP over time for

few TPs Problems: not taking into account of complete FN

Fragmentation/confusion (M3.4.1, M3.5.1): Advantage: indicate potential ID switching Inconvenients: not discriminative; favoring under-

detection (few IDs); over-detection (multiple IDs)

25

Object Classification

T4-Classification

0

1

2

3

4

5

6

7

8

0

0,06

0,12

0,18

0,24 0,3

0,36

0,42

0,48

0,54 0,6

0,66

0,72

0,78

0,84 0,9

0,96

scoresGlobal MO-1-C1 RD-6-C7 RD-10-C4AP-11-C4 AP-11-C7 BE-19-C1 BE-19-C4

26

Object Classification (ETI-VS2-BE-19-C1.xml)

GroupPrecision(global)

Sensitivity(global) F-Score

1 0,902562 0,64330,7511901

25

28 1 0,5288560,6918323

24

12 0,72694 0,6475380,6849455

19

29 1 0,5167920,6814276

45

8 0,78209 0,5979780,6777530

01

19 0,870811 0,5252690,6552776

68

20 0,992571 0,4356050,6054840

45

13 0,828808 0,457776 0,5897919

9 0,943804 0,4271270,5881027

87

32 0,842178 0,3984350,5409474

05

17 0,39765 0,4965760,4416410

31

15 0,70054 0,2539940,3728163

83

3 0,215995 0,344310,2654598

42

28 1 0,994720,9973530

12

29 1 0,9736010,9866239

43

1 0,881468 0,989440,9323384

13

3 0,187147 0,9102430,3104625

46

M4.1.1: ObjectTypeOverSequence

GroupPrecision (global)

Sensitivity

(global) F-Score

1 0,831656 0,5927620,692176

136

29 0,968454 0,5004890,659931

085

28 0,945129 0,4998370,653870

671

8 0,735181 0,5621130,637102

765

19 0,835135 0,503750,628432

25

20 0,976226 0,4284320,595513

581

12 0,625915 0,5575480,589756

767

9 0,907781 0,4108250,565656

655

13 0,784534 0,4333220,558285

778

32 0,746382 0,3531140,479415

902

17 0,330026 0,4121290,366536

061

15 0,634892 0,2301920,337879

464

3 0,215791 0,3439840,265208

883

M4.1.1b: ObjectTypeOverSequenceBoun

dingBoxD1

Subtype

27

Object Classification (ETI-VS2-BE-19-C1.xml)


Sensitivity

(global) F-Score

1 0,831656 0,5927620,6921761

36

29 0,967823 0,5001630,6595011

88

28 0,94328 0,4988590,6525913

49

8 0,731343 0,5591780,6337764

61

19 0,834595 0,5034240,6280256

91

20 0,97474 0,427780,5946072

46

12 0,625183 0,5568960,5890670

79

9 0,907781 0,4108250,5656566

55

13 0,783353 0,432670,5574456

12

32 0,746382 0,3531140,4794159

02

17 0,330026 0,4121290,3665360

61

15 0,633993 0,2298660,3374009

76

3 0,21395 0,341050,2629464

77

M4.1.3: ObjectTypeOverSequenceBoundingBoxIdD1


Sensitivity

(global) F-Score

1 0,831656 0,5927620,692176

136

29 0,968454 0,5004890,659931

085

28 0,945129 0,4998370,653870

671

8 0,735181 0,5621130,637102

765

19 0,835135 0,503750,628432

25

20 0,976226 0,4284320,595513

581

12 0,625915 0,5575480,589756

767

9 0,907781 0,4108250,565656

655

13 0,784534 0,4333220,558285

778

32 0,746382 0,3531140,479415

902

17 0,330026 0,4121290,366536

061

15 0,634892 0,2301920,337879

464

3 0,215791 0,3439840,265208

883

M4.1.2: ObjectTypeOverSequenceBoundingBoxD1

28


M4.1.2 Precision

0

1

2

3

4

5

0

0,06

0,12

0,18

0,24 0,

3

0,36

0,42

0,48

0,54 0,

6

0,66

0,72

0,78

0,84 0,

9

0,96

scores

MO-1-C1 RD-6-C7 RD-10-C4

AP-11-C4 AP-11-C4 BE-19-C1

M4.1.2 Sensitivity

0

1

2

3

4

5

6

7

8

9

0

0,06

0,12

0,18

0,24 0,

3

0,36

0,42

0,48

0,54 0,

6

0,66

0,72

0,78

0,84 0,

9

0,96

scores

MO-1-C1 RD-6-C7 RD-10-C4

AP-11-C4 AP-11-C4 BE-19-C1

29


Group 1 3 8 9 12 13 17 19 20 28 29 32

Precision (meanby frame)

0,67 0,17 0,61 0,50 0,48 0,63 0,24 0,63 0,63 0,77 0,66 0,48

Sensitivity

(mean by frame)

0,53 0,38 0,52 0,31 0,48 0,43 0,38 0,43 0,36 0,48 0,40 0,30

F-Score 0,592 0,235 0,561 0,384 0,482 0,513 0,293 0,511 0,462 0,589 0,498 0,367

30

Summary on Object Classification

M4.1.2, M4.1.3, same main metrics: Problems: low classification of

subtypes (doors, bikes, bags), favoring a few good quality TPs.

Advantage: reliable. M4.1.1 (without BBox):

Inconvenients: wrong evaluation result in case of double errors (classified noise and FN)

Advantage: indicate potential double errors.

31

Event Recognition

T5-Event recognition

0

1

2

3

4

0

0,06

0,12

0,18

0,24 0,3

0,36

0,42

0,48

0,54 0,6

0,66

0,72

0,78

0,84 0,9

0,96


AP-11-C4 AP-11-C7 BE-19-C1 BE-19-C4

32

Event Recognition (ETI-VS2-BE-19-C1.xml)

M5.1.1 Event recognition on BE-19-C1

0,58

0,76

0,94

1,00

1,00

1,00

1,00

0,80

1,00

0,90

0,79

0,85

0,80

0,69

1,00

0,73

0,82

0,57

4

7

6

5

6

3

6

5

7

0,00 1,00 2,00 3,00 4,00 5,00 6,00 7,00 8,00

closes

enters_zone

exits_zone

gets_out

go_down_stairs

go_up_stairs

inside_zone

opens

stopped

Sensitivity Precision Number of participant

33


BE-19-C1 1 8 12 14 15 19 20 28 29 Mean Var Min Max

Precision 0,33 0,25 0,29 0,40 0,04 0,13 0,13 1,00 0,00 0,290,299

40,00 1,00

Sensitivity 0,60 0,25 0,25 0,10 0,05 0,15 0,15 0,35 0,05 0,22 0,175 0,05 0,60

F-Score 0,43 0,25 0,27 0,16 0,04 0,14 0,14 0,52 0,00 0,250,220

90,00 0,75

34


Group Precision

Sensiti

vity F-Score

12 0,88 0,75 0,8108

8 0,65 0,65 0,65

1 0,5 0,9 0,6428

19 0,56 0,65 0,6046

28 1 0,35 0,5185

20 0,42 0,5 0,4545

14 1 0,25 0,4

15 0,04 0,05 0,0454

29 0,01 0,75 0,0282

M5.1.1: NumberEvents

Group

Precision

Sensitivity

F-Score

28 1 0,35 0,51

1 0,38 0,7 0,50

12 0,47 0,4 0,43

8 0,25 0,25 0,25

14 0,4 0,1 0,16

19 0,13 0,15 0,14

20 0,125 0,15 0,14

15 0,04 0,05 0,045

29 0,00 0,05 0,002

M5.1.2: NumberNamedEventsD1

35

Summary on Event Recognition

M5.1.2 (with time), main metrics: Problems: lack of understanding of

ground truth definition Advantages: good global overview per

scenario type. M5.1.1, secondary metric:

Problems: not taking into account of occurrence time

36

Evaluation Results

BE-19-C1

0,00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

0,80

0,90

1,00

T1-Detection T2-Localisation T3-Tracking T4-Classification

T5-Eventrecognition

1

3

8

9

12

13

14

15

17

19

20

28

29

32

37

Evaluation on ETI-VS2-BE-19-C4

38


Group

Precision

(global)

Sensitivity

(global) F-Score

10,95518

5 0,544044 0,6932

90,92263

6 0,241742 0,3831

170,29739

8 0,44044 0,3550

14 1 0,164915 0,2831

130,81855

4 0,15015 0,2537

80,31076

1 0,167668 0,2178

120,98850

6 0,107608 0,1940

290,44619

4 0,085085 0,1429

190,27830

2 0,059059 0,0974

230,61471

9 0,035536 0,0671


M2.1.1: ObjectsArea

Group

Precision

(global)

Sensitivity

(global)

Specificity

(global) F-Score

10,75849

3 0,557398 0,984318 0,64258

80,77391

2 0,308128 0,992046 0,44077

170,56776

4 0,344377 0,976834 0,42872

90,92807

8 0,22665 0,998448 0,36433

120,74372

1 0,21956 0,993315 0,33903

140,90129

5 0,196442 0,998101 0,32258

130,68104

6 0,193248 0,992445 0,30107

290,49834

2 0,100903 0,991025 0,16783

190,57232

6 0,083559 0,994483 0,14583

230,60959

4 0,049188 0,997217 0,09103

39


M3.2.1: NumberObjectTrackedD1

Group Precision Sensitivity F-Score

8 0,5 1 0,666667

14 1 0,5 0,666667

9 0,75 0,5 0,6

12 1 0,333333 0,5

13 0,375 0,5 0,428571

29 0,4 0,333333 0,363636

1 0,190476 0,666667 0,296296

19 0,25 0,333333 0,285714

17 0,101695 1 0,184615

M3.3.1.D1: TrackingTime

Group Tracking Time

9 0,203304

8 0,146138

12 0,124236

1 0,118673

14 0,110668

29 0,079072

17 0,064894

13 0,062049

19 0,020582

23 0,007754

20 0

40




1 0,264151 0,378378 0,3111

12 0,75 0,162162 0,2666

14 0,444444 0,108108 0,1739

8 0,266667 0,108108 0,1538

19 0,75 0,081081 0,1463

20 0,125 0,081081 0,0983

29 0,01845 0,135135 0,0324


Group Precision

Sensit

ivity F-Score

8 0,866667 0,351351 0,4999

20 0,625 0,405405 0,4918

1 0,358491 0,513514 0,4222

14 1 0,243243 0,3913

12 1 0,216216 0,3555

19 1 0,108108 0,1951

29 0,110701 0,810811 0,1948

41

Evaluation Results

BE-19-C4

0,00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

0,80

0,90


T5-Eventrecognition

1

8

9

12

13

14

17

19

29

42

Evaluation on ETI-VS2-MO-1-C1

43

Detection of Physical Objects (ETI-VS2-MO-1-C1.xml)

Group

Precision (global)

Sensitivity

(global) F-Score

14 1 0,7755910,8736144

75

12 0,998173 0,7169730,8345214

81

8 0,677875 0,7760280,7236383

45

1 0,51055 0,7515310,6080341

15

28 1 0,3118990,4754923

97

19 0,340185 0,7095360,4598812

53

3 0,271405 0,4978130,3512890

68

26 1 0,1692910,2895617

94

17 0,148108 0,539370,2324001

99

29 0,179036 0,2712160,2156900

04

20 0,041295 0,5892390,0771810

07

23 0,244898 0,0314960,0558138

56


M2.1.1: ObjectsArea

Group

Precision

(global)

Sensitivity

(global)

Specificity

(global)

F-Score

14 0,75582 0,759066 0,9825880,7574

4

120,65330

9 0,819678 0,969115 0,7271

80,86741

7 0,570867 0,9938050,6885

7

10,30862

9 0,860968 0,8710950,4543

8

190,26086

7 0,812689 0,8484560,3949

6

30,21917

2 0,8702 0,7956190,3501

5

200,20755

3 0,755444 0,8082180,3256

4

170,18934

8 0,725661 0,7923590,3003

3

290,35698

5 0,249236 0,9703280,2935

3

260,85940

2 0,024007 0,9997210,0467

1

28 0,99999 0,023868 10,0466

2

230,23220

2 0,023804 0,9948240,0431

8

44

Tracking of Physical Objects (ETI-VS2-MO-1-C1.xml)


Group

Precision Sensitivity F-Score

14 1 1 1

8 1 1 1

12 1 1 1

28 1 0,333333 0,5

26 1 0,333333 0,5

10,09090

9 1 0,166667

290,06818

2 1 0,12766

190,05454

5 1 0,103448

170,03409

1 1 0,065934

30,02608

7 1 0,050847

200,01807

2 1 0,035503


GroupTracking

Time

8 0,776399

14 0,758954

12 0,647392

19 0,393109

29 0,276964

17 0,254244

28 0,244262

1 0,202911

26 0,13258

3 0,103148

20 0,011822

23 0,002927

45

Event Recognition (ETI-VS2-MO-1-C1.xml)



8 0,428571 0,50,46153821

3

19 0,285714 0,333333 0,307692

12 0,285714 0,333333 0,307692

28 0,5 0,1666670,25000037

5

14 0,5 0,1666670,25000037

5

1 0,076923 0,1666670,10526315

2


Group Precision

Sensitivity F-Score

8 0,571429 0,6666670,61538500

6

12 0,571429 0,6666670,61538500

6

28 1 0,3333330,49999962

5

14 1 0,3333330,49999962

5

19 0,428571 0,50,46153821

3

1 0,076923 0,1666670,10526315

2

46

Evaluation Results

MO-1-C1

0,00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

0,80

0,90

1,00


T5-Eventrecognition

1

3

8

9

12

9

12

12

14

17

19

20

26

28

29

47

Evaluation on ETI-VS2-RD-6-C7

48

Detection of Physical Objects (ETI-VS2-RD-6-C7.xml)

Group

Precision (global)

Sensitivity (global) F-Score

10,99976

70,53691

2 0,69863244

14 0,992960,42354

90,59380803

8

80,99969

8 0,414790,58631071

2

280,98089

40,41754

30,58574740

7

120,99839

60,38938

90,56026606

4

190,91379

9 0,355480,51184533

7

30,67203

60,39151

70,49478214

7

320,93442

60,32807

80,48564537

3

90,99481

50,31206

20,47509284

9

130,97494

10,31156

20,47221742

6

200,65444

80,36636

6 0,46975746

290,90567

60,27152

2 0,41779031

150,99475

90,23748

70,38343371

5

17 1 0,160410,27647124

7

23 0,90085 0,039790,07621368

7

M1.2.1: NumberObjectsBoundingBoxD1M2.1.1: ObjectsArea

Group

Precision (global)

Sensitivity (global)

Specificity (global) F-Score

10,91206

10,77622

30,99350

90,8386

8

80,80462

30,46430

90,99022

20,5888

3

280,81049

60,45950

60,99068

2 0,5865

12 0,77810,44392

6 0,989020,5653

2

30,81934

30,43041

10,99176

90,5643

6

140,73925

50,41910

8 0,987190,5349

4

90,87281

50,37745

60,99522

9 0,527

130,91612

70,36424

90,99710

80,5212

5

320,79612

10,37632

70,99164

10,5110

7

200,55164

40,45169

40,96815

90,4966

9

150,63877

60,40453

40,98015

90,4953

6

19 0,802820,35761

10,99238

20,4948

1

170,78908

70,35568

30,99175

40,4903

4

290,89217

90,31672

9 0,996680,4674

9

230,58301

40,03780

20,99765

5 0,071

49

Detection of Physical Objects: Reference Data Filtering

(ETI-VS2-RD-6-C7.xml: M1.2.1 - NumberObjectsBoundingBoxD1)

Group

Precision (global)

Sensitivity (global) F-Score

10,99976

70,53691

2 0,69863244

14 0,992960,42354

90,59380803

8

80,99969

8 0,414790,58631071

2

280,98089

40,41754

30,58574740

7

120,99839

60,38938

90,56026606

4

190,91379

9 0,355480,51184533

7

30,67203

60,39151

70,49478214

7

320,93442

60,32807

80,48564537

3

90,99481

50,31206

20,47509284

9

130,97494

10,31156

20,47221742

6

200,65444

80,36636

6 0,46975746

290,90567

60,27152

2 0,41779031

150,99475

90,23748

70,38343371

5

17 1 0,160410,27647124

7

23 0,90085 0,039790,07621368

7

No filtering With filtering

Group

Precision

(global)

Sensitivity

(global) F-Score

80,88975

7 0,701397 0,7844

14 0,99296 0,423549 0,5938

280,98089

4 0,417543 0,5857

120,99839

6 0,389389 0,5602

10,52143

5 0,529955 0,5256

320,93442

6 0,328078 0,4856

90,99481

5 0,312062 0,4750

130,97494

1 0,311562 0,4722

290,90567

6 0,271522 0,4177

30,31004

9 0,5991 0,4086

150,99475

9 0,237487 0,3834

190,38606

7 0,329387 0,3554

17 1 0,16041 0,2764

23 0,90085 0,03979 0,0762

50


(ETI-VS2-RD-6-C7.xml: M2.1.1 - ObjectsArea)

Group

Precision (global)

Sensitivity (global)


10,91206

10,77622

3 0,9935090,8386

8

80,80462

30,46430

9 0,9902220,5888

3

280,81049

60,45950

6 0,990682 0,5865

12 0,77810,44392

6 0,989020,5653

2

30,81934

30,43041

1 0,9917690,5643

6

140,73925

50,41910

8 0,987190,5349

4

90,87281

50,37745

6 0,995229 0,527

130,91612

70,36424

9 0,9971080,5212

5

320,79612

10,37632

7 0,9916410,5110

7

200,55164

40,45169

4 0,9681590,4966

9

150,63877

60,40453

4 0,9801590,4953

6

19 0,802820,35761

1 0,9923820,4948

1

170,78908

70,35568

3 0,9917540,4903

4

290,89217

90,31672

9 0,996680,4674

9

230,58301

40,03780

2 0,997655 0,071


Group

Precision

(global)Sensitivity (global)


280,81049

6 0,459506 0,990682 0,5865

12 0,7781 0,443926 0,989020,5653

2

140,73925

5 0,419108 0,987190,5349

4

90,87281

5 0,377456 0,995229 0,527

130,91612

7 0,364249 0,9971080,5212

5

320,79612

1 0,376327 0,9916410,5110

7

150,63877

6 0,404534 0,9801590,4953

6

170,78908

7 0,355683 0,9917540,4903

4

290,89217

9 0,316729 0,996680,4674

9

230,58301

4 0,037802 0,997655 0,071

51

Tracking of Physical Objects (ETI-VS2-RD-6-C7.xml)



y F-Score

28 1 0,7058820,82758

6

14 1 0,7058820,82758

6

9 1 0,6470590,78571

4

12 1 0,6470590,78571

4

15 0,785714 0,6470590,70967

7

17 1 0,5294120,69230

8

8 0,769231 0,5882350,66666

7

13 0,769231 0,5882350,66666

7

29 0,714286 0,5882350,64516

1

32 0,611111 0,6470590,62857

1

19 0,458333 0,6470590,53658

5

1 0,5 0,4117650,45161

3

3 0,059783 0,6470590,10945

3


GroupTracking

Time

14 0,603553

28 0,584835

12 0,503574

32 0,42068

9 0,419863

29 0,387679

13 0,352851

15 0,322041

17 0,265919

23 0,030452

52

Tracking of Physical Objects: Reference Data Filtering

(ETI-VS2-RD-6-C7.xml: M3.2.1 - NumberObjectTrackedD1)


y F-Score

28 1 0,7058820,82758

6

14 1 0,7058820,82758

6

9 1 0,6470590,78571

4

12 1 0,6470590,78571

4

15 0,785714 0,6470590,70967

7

17 1 0,5294120,69230

8

8 0,769231 0,5882350,66666

7

13 0,769231 0,5882350,66666

7

29 0,714286 0,5882350,64516

1

32 0,611111 0,6470590,62857

1

19 0,458333 0,6470590,53658

5

1 0,5 0,4117650,45161

3

3 0,059783 0,6470590,10945

3


Group

Precision

Sensitivity F-Score

10,92857

1 0,764706 0,83871

28 1 0,7058820,82758

6

14 1 0,7058820,82758

6

12 1 0,6470590,78571

4

9 1 0,6470590,78571

4

80,84615

4 0,6470590,73333

3

150,78571

4 0,6470590,70967

7

20 0,65 0,7647060,70270

3

17 1 0,5294120,69230

8

130,76923

1 0,5882350,66666

7

190,64705

9 0,6470590,64705

9

290,71428

6 0,5882350,64516

1

320,61111

1 0,6470590,62857

1

30,30769

2 0,9411760,46376

8

53


(ETI-VS2-RD-6-C7.xml: M3.3.1.D1 - TrackingTime)

Group Tracking Time

14 0,603553

28 0,584835

12 0,503574

32 0,42068

9 0,419863

29 0,387679

13 0,352851

15 0,322041

17 0,265919

23 0,030452


Group Tracking Time

14 0,603553

28 0,584835

1 0,534872

12 0,503574

8 0,48195

20 0,443102

19 0,42611

32 0,42068

9 0,419863

29 0,387679

3 0,380634

13 0,352851

15 0,322041

17 0,265919

23 0,030452

54

Event Recognition (ETI-VS2-RD-6-C7.xml)


Group

Precision

Sensitivity F-Score

1 1 0,25 0,4

28 1 0,25 0,4

12 1 0,25 0,4

19 0,5 0,25 0,33

8 0,5 0,125 0,2

15 0,017 0,125 0,03


Group Precision

Sensitivity F-Score

19 1 0,5 0,67

1 1 0,25 0,4

8 1 0,25 0,4

28 1 0,25 0,4

12 1 0,25 0,4

14 1 0,25 0,4

15 0,070175 0,5 0,12

55

Event Recognition (ETI-VS2-RD-6-C7)

M5.1.1 Event recognition on RD-6-C7

1,00

0,25

1,00

0,00

0,21

1,00

0,25

1,00

0,00

0,51

1

4

1

0

7

0,00 1,00 2,00 3,00 4,00 5,00 6,00 7,00 8,00

closes

gets_ out

opens

pulling

stopped

Sensitivity Precision Number of participant

56

Evaluation Results

RD-6-C7

0,00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

0,80

0,90

1,00


T5-Eventrecognition

1

3

8

9

12

13

14

15

17

19

20

28

29

32

57

Evaluation on ETI-VS2-RD-10-C4

58

Detection of Physical Objects (ETI-VS2-RD-10-C4.xml)

Group

Precision (global)

Sensitivity

(global) F-Score

140,99556

5 0,4085530,5793545

38

10,97214

8 0,3620560,5276134

93

130,97258

6 0,3583260,5237053

25

190,78370

5 0,3903550,5211371

91

170,58857

9 0,422020,4915740

26

200,55994

9 0,3620560,4397652

84

290,84887

4 0,2708830,4107061

37

120,93189

6 0,2477710,3914609

87

230,79656

2 0,0252960,0490348

27


Group

Precision

(global)

Sensitivity

(global)

Specificity

(global) F-Score

10,68833

10,83874

50,98366

90,7561

3

130,75540

30,69947

4 0,990260,7263

6

140,88343

40,60511

40,99656

70,7182

5

190,68303

4 0,562480,98877

50,6169

2

200,48923

9 0,574190,97422

20,5283

2

120,47324

60,57900

50,97228

60,5208

1

290,82493

60,37494

20,99657

80,5155

6

170,31605

70,75563

20,92968

30,4456

9

230,48493

90,04821

10,99789

7 0,0877

59

Tracking of Physical Objects (ETI-VS2-RD-10-C4.xml)


Group

Precision

Sensitivity F-Score

14 1 0,5789470,73333

3

290,84615

4 0,578947 0,6875

200,45714

3 0,8421050,59259

3

12 0,5 0,5263160,51282

1

190,35897

4 0,7368420,48275

9

130,33333

3 0,5789470,42307

7

170,24590

2 0,789474 0,375

10,24137

9 0,3684210,29166

7


Group Tracking Time

14 0,481446

17 0,320325

19 0,286615

13 0,27612

29 0,271027

1 0,27099

20 0,270541

12 0,21312

23 0,010886

60

Event Recognition (ETI-VS2-RD-10-C4.xml)



y F-Score

120,53333

3 0,2 0,2909

190,18604

7 0,2 0,1927

1 0,4 0,1 0,16

14 0,25 0,025 0,0454


Group Precision

Sensitivity F-Score

12 0,933333 0,35 0,5090

19 0,372093 0,4 0,38556

1 0,5 0,125 0,2

14 1 0,1 0,1818

61

Evaluation Results

RD-10-C4

0,00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

0,80

0,90


T5-Eventrecognition

1

12

13

14

17

19

20

62

Evaluation on ETI-VS2-AP-11-C7

63

Detection of Physical Objects (ETI-VS2-AP-11-C7.xml)

Group

Precision

(global)

Sensitivity

(global) F-Score

10,99855

6 0,975094 0,9866

8 0,98951 0,930921 0,9593

280,98428

1 0,617951 0,7592

14 1 0,594455 0,7456

12 0,98065 0,595395 0,7409

200,97378

6 0,593515 0,7375

90,98647

6 0,582707 0,7326

290,98794

2 0,577538 0,7289

190,95267

6 0,577068 0,7187

15 0,99159 0,554041 0,7108

170,97256

6 0,516447 0,6746

320,87981

9 0,546992 0,6745

130,92889

7 0,478853 0,6319

30,49633

7 0,382049 0,4317

230,89051

1 0,057331 0,1077


Group

Precision (global)

Sensitivity

(global)

Specificity

(global) F-Score

1 0,89095 0,9790690,99571

2 0,93293

80,92081

9 0,8227380,99746

9 0,86902

190,86711

5 0,4375840,99760

1 0,58164

28 0,7995 0,4505020,99595

8 0,57628

200,77782

6 0,4467610,99543

4 0,56754

140,82211

4 0,4328930,99665

3 0,56715

120,90404

1 0,4124740,99843

3 0,56649

170,75837

2 0,4410030,99497

2 0,5577

150,81834

5 0,4049780,99678

3 0,54182

90,94762

1 0,377360,99925

4 0,53977

320,88523

7 0,3673860,99829

6 0,51927

290,91978

6 0,3569960,99888

6 0,51436

30,87499

6 0,342374 0,99825 0,49217

130,98045

2 0,3271670,99976

7 0,49062

230,52525

5 0,0446250,99855

7 0,08226

64


(ETI-VS2-AP-11-C7.xml: M1.2.1 - NumberObjectsBoundingBoxD1)

GroupPrecision

(global)

Sensitivity (globa

l) F-Score

1 0,998556 0,975094 0,9866

8 0,98951 0,930921 0,9593

28 0,984281 0,617951 0,7592

14 1 0,594455 0,7456

12 0,98065 0,595395 0,7409

9 0,986476 0,582707 0,7326

29 0,987942 0,577538 0,7289

19 0,952676 0,577068 0,7187

15 0,99159 0,554041 0,7108

17 0,972566 0,516447 0,6746

32 0,879819 0,546992 0,6745

13 0,928897 0,478853 0,6319

3 0,496337 0,382049 0,4317

23 0,890511 0,057331 0,1077


GroupPrecision

(global)Sensitivity

(global) F-Score

1 0,998556 0,975094 0,9866

8 0,98951 0,930921 0,9593

28 0,984281 0,617951 0,7592

14 1 0,594455 0,7456

12 0,98065 0,595395 0,7409

20 0,973786 0,593515 0,7375

9 0,986476 0,582707 0,7326

29 0,987942 0,577538 0,7289

19 0,952676 0,577068 0,7187

15 0,99159 0,554041 0,7108

17 0,972566 0,516447 0,6746

32 0,879819 0,546992 0,6745

13 0,928897 0,478853 0,6319

3 0,496337 0,382049 0,4317

23 0,890511 0,057331 0,1077

65


(ETI-VS2-AP-11-C7.xml: M2.1.1 - ObjectsArea)

GroupPrecision

(global)

Sensitivity

(global)

Specificity

(global) F-Score

1 0,89095 0,9790690,99571

2 0,93293

80,92081

9 0,8227380,99746

9 0,86902

190,86711

5 0,4375840,99760

1 0,58164

28 0,7995 0,4505020,99595

8 0,57628

200,77782

6 0,4467610,99543

4 0,56754

140,82211

4 0,4328930,99665

3 0,56715

120,90404

1 0,4124740,99843

3 0,56649

170,75837

2 0,4410030,99497

2 0,5577

150,81834

5 0,4049780,99678

3 0,54182

90,94762

1 0,377360,99925

4 0,53977

320,88523

7 0,3673860,99829

6 0,51927

290,91978

6 0,3569960,99888

6 0,51436

30,87499

6 0,342374 0,99825 0,49217

130,98045

2 0,3271670,99976

7 0,49062

230,52525

5 0,0446250,99855

7 0,08226


Group

Precision (global)

Sensitivity

(global)

Specificity

(global) F-Score

1 0,890950,97906

9 0,995712 0,93293

80,92081

90,82273

8 0,997469 0,86902

190,86711

50,43758

4 0,997601 0,58164

28 0,79950,45050

2 0,995958 0,57628

140,82211

40,43289

3 0,996653 0,56715

120,90404

10,41247

4 0,998433 0,56649

170,75837

20,44100

3 0,994972 0,5577

150,81834

50,40497

8 0,996783 0,54182

90,94762

1 0,37736 0,999254 0,53977

320,88523

70,36738

6 0,998296 0,51927

290,91978

60,35699

6 0,998886 0,51436

30,87499

60,34237

4 0,99825 0,49217

130,98045

20,32716

7 0,999767 0,49062

230,52525

50,04462

5 0,998557 0,08226

66

Tracking of Physical Objects (ETI-VS2-AP-11-C7.xml)


Group

Precision

Sensitivity F-Score

10,83333

3 10,90909

1

14 1 0,80,88888

9

80,71428

6 10,83333

3

28 0,8 0,8 0,8

29 0,8 0,8 0,8

9 0,8 0,8 0,8

20 0,8 0,8 0,8

170,66666

7 0,80,72727

3

120,66666

7 0,80,72727

3

320,66666

7 0,80,72727

3

150,44444

4 0,80,57142

9

190,36363

6 0,8 0,5

130,21052

6 0,80,33333

3

30,06779

7 0,8 0,125

M3.3.1.D1: TrackingTimeGroup Tracking Time

1 0,842267

8 0,776723

14 0,764944

20 0,764574

28 0,66347

12 0,646376

9 0,626901

29 0,612043

17 0,536711

15 0,440843

19 0,434929

32 0,432908

3 0,265107

13 0,03841

23 0,037081

67


(ETI-VS2-AP-11-C7.xml: M3.2.1 - NumberObjectTrackedD1)


y F-Score

10,83333

3 1 0,909091

14 1 0,8 0,888889

80,71428

6 1 0,833333

28 0,8 0,8 0,8

29 0,8 0,8 0,8

9 0,8 0,8 0,8

120,66666

7 0,8 0,727273

170,66666

7 0,8 0,727273

320,66666

7 0,8 0,727273

150,44444

4 0,8 0,571429

190,36363

6 0,8 0,5

130,21052

6 0,8 0,333333

30,06779

7 0,8 0,125


Group Precision

Sensitivity F-Score

10,83333

3 10,90909

1

14 1 0,80,88888

9

80,71428

6 10,83333

3

28 0,8 0,8 0,8

29 0,8 0,8 0,8

9 0,8 0,8 0,8

120,66666

7 0,80,72727

3

170,66666

7 0,80,72727

3

320,66666

7 0,80,72727

3

150,44444

4 0,80,57142

9

190,27272

7 0,6 0,375

130,21052

6 0,80,33333

3

30,05084

7 0,6 0,09375

68


(ETI-VS2-AP-11-C7.xml: M3.3.1.D1 - TrackingTime)

Group Tracking Time

1 0,842267

8 0,776723

14 0,764944

20 0,764574

28 0,66347

12 0,646376

9 0,626901

29 0,612043

17 0,536711

15 0,440843

19 0,434929

32 0,432908

3 0,265107

13 0,03841

23 0,037081

No filteringWith filtering

GroupTracking

Time

1 0,842267

8 0,776723

14 0,764944

28 0,66347

12 0,646376

9 0,626901

29 0,612043

17 0,536711

15 0,440843

19 0,434929

32 0,432908

3 0,265107

13 0,03841

23 0,037081

69

Event Recognition (ETI-VS2-AP-11-C7.xml)



y F-Score

1 1 1 1

12 1 0,625 0,7692

140,71428

6 0,625 0,6666

8 1 0,5 0,6666

28 0,4 0,75 0,52173

200,36363

6 0,5 0,4210

290,22727

3 0,625 0,3333

19 0,4 0,25 0,3076


Group

Precision Sensitivity F-Score

1 1 1 1

14 1 0,875 0,9333

12 1 0,625 0,7692

280,53333

3 1 0,6956

8 1 0,5 0,6666

200,54545

5 0,75 0,6315

19 0,8 0,5 0,6153

290,36363

6 1 0,5333

15 0,1 0,25 0,1428

70

Event Recognition (ETI-VS2-AP-11-C7.xml)



y F-Score

120,53333

3 0,2 0,2909

190,18604

7 0,2 0,1927

1 0,4 0,1 0,16

14 0,25 0,025 0,0454


Group Precision

Sensitivity F-Score

12 0,933333 0,35 0,5090

19 0,372093 0,4 0,38556

1 0,5 0,125 0,2

14 1 0,1 0,1818

71

Evaluation Results

AP-11-C7

0,00

0,20

0,40

0,60

0,80

1,00

1,20


T5-Eventrecognition

1

3

8

9

12

13

14

15

17

19

20

28

29

32

72

Understanding versus Competition

ETISEO Goal Not a competition nor benchmarking Emphasis on gaining insight into video analysis

algorithms Better understanding of evaluation methodology

Why? ETISEO limitations: Algorithm results depend on time and manpower

(parameter tuning), format understanding (XML), objective definition

(ground truth), and algorithm capacities (static, occluded, portable and contextual objects)

previous similar experiences, number of processed videos, frame rate, start frame Metrics and parameters (split/merge) learning stage required or not.

73

Understanding versus Competition (cont’d)

Warmest thanks to the 16 teams: 8 teams achieved high quality results 9 teams performed event recognition 10 teams produced results on all

priority sequences Special thanks to teams 1, 8, 12, 14 and

28 : Stable and high-quality results on a

large video set More evaluation results…

74

Conclusions

Good performance comparison per video: automatic, reliable, consistent metrics.

A few insights into video surveillance algorithms. For example,

Shadows merge

A few limitations: Lack of understanding of the evaluation rules (output

XML, time-stamp) Data subjectivity: video, background, masks Metrics and evaluation parameters

Future improvements: flexible evaluation tool Filters for reference data Selection of metrics and parameters Selection of videos

Documents

ETISEO