Dataset Production and Performance Evaluation for Event Detection and Tracking

Scientific DevelopmentBranch

Dataset Production and Performance Evaluation for Event Detection and Tracking

Paul Hosmer Detection and Vision Systems Group

SCIENTIFIC DEVELOPMENT BRANCHPAUL HOSMER BMVA Performance Evaluation Symposium 2007

Outline

• Defining a requirement

• What to include in datasets

• Constraints

• Evaluation and Metrics

• Case Study


Intelligent Video

– Started in early 1990’s – FABIUS – Amethyst– Through to 2000’s – VMD capability study– Standards-based evaluations

Background


What did we want to achieve?

• Test systems in short period of time

• Provide data and requirements to research community

Dataset production

Problem: what to include?


Scenario definition• What is an event?

• Where does the scenario take place?

• What challenges are posed by the environment?

Ask end users / gauge demand

Conduct capability study

Monitor environment, apply a priori knowledge


Scenario definition

• Abandoned Baggage

• When is an object

abandoned?

• What types of object?

• Attributes of person?


Scenario definition“Abandoned object”

• During the current clip, a person has placed an object which was in

their possession when they entered the clip onto the floor or a seat

in the detection area &

• That person has left the detection area without the object &

• Over sixty seconds after they left the detection area, that person has

still not returned to the object &

• The object remains in the detection area.


Scenario definition• Key environmental factors:

• Lighting changes – film

dawn and dusk

• Rain and snow

• Night – head lights and low

SNR


How much data?

• Need to demonstrate

performance on wide range of

imagery

• Statistical significance

• Need large training and test

corpus – 100’s of events

• Unseen data for verification


Constraints• You can’t always capture the event you want – simulation

• Make simulations as close to the requirement as possible

• Storage vs image quality – what will you want to do with

the data at a later time?

• Cost – try to film as much variation/events as you can


Performance Evaluation• Importance of metrics – consistency across different evaluations

• When is an event detected?

• Real time evaluation, 10x real time, offline… which is most useful?

• Statistically significant unseen dataset:

Performance on training data does not tell me anything useful about

robustness


How HOSDB does it• Simulate real analogue

CCTV system

• ~ 215,000 frames per

scenario evaluation

• ~ Evaluation 300 events

• 60s to alarm after GT alarm

condition is satisfied

• One figure of merit for

ranking


F1 score for event detection

F1 = (α + 1)RP R + αP

α ranges from 0.35 to 75 depending on scenario and

application

R = TP

TP+FNP = TP

TP+FP

where


What about Tracking?55thth i-LIDS scenario i-LIDS scenario– Multiple Camera Tracking– Increasing interest from end users – Significant potential to enhance operator effectiveness and

aid post event investigation

The ProblemThe Problem– Unifying tracking labelling across multiple camera views

Dataset and Evaluation ProblemDataset and Evaluation Problem– Synchronisation


Operational RequirementCamera Requirements:Camera Requirements:– Existing CCTV systems– Cameras are a mixture of overlapping and non-overlapping– Internal cameras are generally fixed and colour

Scene Contents:Scene Contents:– Scenes are likely to contain rest points– Varying traffic densities

Target Description:Target Description:– There may be multiple targets– Targets from wide demographic


Imagery Collection


Imagery CollectionLocationLocation– Large Transport Hub

(airport)

TargetsTargets– Varied Targets– Differing target behaviour– Varying crowd densities

EnvironmentEnvironment– Lighting changes– Filmed at Dawn, Day, Dusk

and Night

VolumeVolume– 5 cameras– 1.35 Million frames– Single and multiple target– 1000+ target events– 1TB external HDD



Dataset structure

Target Event Set

MCT01

MCT02

MCT03

TES Properties

DaytimeHigh density

Night timeLow density

EtcEtc

Overlapping Stage Non-Overlapping Stage

Mixed Stage


Performance Metric

F1 = 2RP R+P

– F1 must be greater than or equal to 0.25 for the track to be a True Positive

P = Overlapping Pixels Total Track Pixels

R = Overlapping Pixels Total Ground Truth Pixels


– Performance evaluation is important– Evaluations need to use more data– With richer content– With widely accepted definitions and

metrics

– Demonstrate improved performance

Documents

Dataset Production and Performance Evaluation for Event Detection and Tracking