INRIA, Nice. December 7 th -8 th 2006 Evaluation protocol Evaluation process

INRIA, Nice. December 7th-8th 2006

EvaluationEvaluation protocolprotocol

Evaluation processEvaluation process

INRIA, Nice. December 7th-8th 2006

EvaluationEvaluation protocolprotocol

ScheduleSchedule

AgreementAgreement

ResultsResults submissionsubmission

File format transfer toolFile format transfer tool

FeedbackFeedback

INRIA, Nice. December 7th-8th 2006

Results collection

AprilJan July Oct

Participants test

Corpus diffusion

First evaluationSeminar

DecJune

Results analysis

Evaluation review

CYCLE 1

Corpus, metrics & Gt review

Results analysis

Evaluation review

Data Collection

CYCLE 2 Seminar

Results collection

Participants test

Corpus diffusion

Etiseo evaluation

ScheduleSchedule

INRIA, Nice. December 7th-8th 2006

ETISEO ProjectETISEO Project

ScheduleSchedule

AgreementAgreement

Results submissionResults submission

FileFile formatformat transfertransfer tooltool

FeedbackFeedback

INRIA, Nice. December 7th-8th 2006

AgreementAgreement

- Actors :

The participant & the coordinator (Silogic)- To participate to the ETISEO project evaluation cycle, the participant will receive specific resources containing video sequences and meta-data files.

The agreement describes the terms of the exploitation of the resources, accepted by the participant.

The coordinator will transmit the resources to the participant after the signature of the agreement.

INRIA, Nice. December 7th-8th 2006

AgreementAgreement

- The participant agrees : - To use the resources only for research and development of video surveillance evaluation purpose.- Not to use the resources for commercial purposes.

- Not to diffused the resources to others outside the participant’s research group.

- To carry out the tests of its algorithms on the resources. - To send to the coordinator the results of the tests, in due time, in a suitable XML format using the defined ontology.

INRIA, Nice. December 7th-8th 2006

AgreementAgreement

- The participant is authorised :

- To make scientific publications based on the exploitation of the resources, under the condition of mentioning the ETISEO project.

- Small part of the resources can be displayed in a research environment, under the condition of mentioning the ETISEO project.

INRIA, Nice. December 7th-8th 2006

AgreementAgreement

- The participant authorises : - The coordinator to exploit the results in comparison with the ETISEO ground-truth to compute the evaluation. - The coordinator to diffused the evaluation results in an anonymous way.

- The coordinator to diffuse the evaluation results to INRIA-ORION team, which is the ETISEO scientific leader, for the evaluation analysis.

- The coordinator has to send back to the participant the results of the evaluation concerning its own results.

INRIA, Nice. December 7th-8th 2006

ETISEO ProjectETISEO Project

ScheduleSchedule

AgreementAgreement

ResultsResults submissionsubmission

FileFile formatformat transfertransfer tooltool

FeedbackFeedback

INRIA, Nice. December 7th-8th 2006

Results Results format -1format -1

Result Submission Format :

• Based on PETS-metrics XML schema, with the addition of events.

• Some attributes are not processed (segmentation = shape).

• Documentation on this format is available on the web site also with examples.

INRIA, Nice. December 7th-8th 2006

Results Results format -1format -1

Result Submission Format :

Extended PETS metrics

developper details

algorithm details

run date

camera used

results

frame

tracked_target

info_2D

segmentation

row_seg

object

info_3D

parent target

event_target

physical_object

contextual_object

speed

Must occur once and only once in the parent element.

Must occur at least once.

Optional but can occur once and only once in the parent element.

Optional to occur but can occur numerously.

INRIA, Nice. December 7th-8th 2006

Results Results format -2format -2

Based on VIPER using ETISEO attributes.<?xml version="1.0" encoding="UTF-8"?><viper xmlns="http://lamp.cfar.umd.edu/viper#" xmlns:data="http://lamp.cfar.umd.edu/viperdata#">

<config> <…></config><data>

<sourcefile filename="file:ETI-VS1-CR-4-C1.avi"><object framespan="1:100" id="4" name="physical_object">

<attribute name="nature"><data:svalue value="physical_object_of_interest"/>

</attribute><attribute name="type">

<data:svalue value="person"/></attribute><attribute name="info2D">

<data:bbox framespan="1:1" height="182" width="86" x="138" y="59"/><data:bbox framespan="2:2" height="183" width="86" x="139" y="59"/>

<data:bbox framespan="3:3" height="185" width="86" x="141" y="60"/> <data:bbox framespan="4:17" height="187" width="87" x="143" y="60"/> <…>

<data:bbox framespan="40:100" height="212" width="10" x="3" y="10"/></attribute><…>

</sourcefile></data>

</viper>

INRIA, Nice. December 7th-8th 2006

Results Results format -2format -2

• Id• Start time• Type (ex: person,

vehicle…)• Subtype (ex: car,

van…)• Info2D• Info3D

• Links• Speed• GT Nature• GT Status• GT Occluded (partially or

completely)• GT Connected with

Physical object

GT : specific to annotation purpose

INRIA, Nice. December 7th-8th 2006

Results Results format -2format -2

• Id• Name• Start and end time• Status• Involved objects (physical object of interest

and contextual object)

Event

INRIA, Nice. December 7th-8th 2006

Results Results contentcontent

Submission content :

- For each sequence you processed, its xml results file using the naming convention:

sequence-name.xml

- The questionnaire describing how your system proceeded on all sequences.

INRIA, Nice. December 7th-8th 2006

Results submissionResults submission

Submission method :

- by email (<10Mo) to: results-etiseo@silogic.fr

- on ftp www.silogic.fr/etiseo. Connection parameters will be provide to all participants. Send also a notification to results-etiseo@silogic.fr. Warning : other participants can have access to your results.- by mail to:

Silogic, ETISEO project.Attention of Luc Barthelemy6, rue Camboulives BP 113331036 Toulouse Cedex 1France

INRIA, Nice. December 7th-8th 2006

Results submissionResults submission

Submission method :

In any case, you received an acknowledgement to your submission.

INRIA, Nice. December 7th-8th 2006

ETISEO ProjectETISEO Project

ScheduleSchedule

AgreementAgreement

Results submissionResults submission

File format transfer toolFile format transfer tool

Evaluation feed-backEvaluation feed-back

INRIA, Nice. December 7th-8th 2006

Evaluation feedbackEvaluation feedback

Evaluation feedback :

Metrics results are sent back to participants:• res file•Example: xls file1, xls file2, xls file3

• Multiple submission is allowed.

• Statistics on metrics for each sequence available on the web-site.

• Global feedback to all participants.