Outline National Research Council of Canada · 9. Video Recognition projects and interests (Dr. Dmitry Gorodnichy) DTO VACE, Phase III meeting - October 2006, Reston, USA A closer

IIT-NRC Video Recognition Systemsprogram

http://iit-iti.nrc-cnrc.gc.cahttp://iit-iti.nrc-cnrc.gc.ca/about-sujet/cv-vi_e.html

http://synapse.vit.iit.nrc.ca (www.perceptual-vision.com)

Leader: Dr. Dmitry Gorodnichy

US DTO VACE

Phase III Meeting

Washington DCOctober 2006

NRC-IIT Video Recognition Systemsprogram

+

Canadian national interests in deploying video technologies

Leader: Dr. Dmitry Gorodnichy

US DTO VACE

Phase III Meeting

Washington DC1/XI/2006

3. Video Recognition projects and interests (Dr. Dmitry Gorodnichy)DTO VACE, Phase III meeting - October 2006, Reston, USA

Outline

Part 1: - Who we are: NRC IIT CVG VRS

- VRS projects and results

Part 2:- Canadian National/Regional interests in Video Technology– Government (OGD) contacts and programs– Universities, Industry references

(in particular wrt national security)


National Research Council of Canada (NRC-CNRC, or NRC)• Over 30 institutes

• Across Canada, Divided in clusters • Logo: “From Discovery to Innovation…”• Mandate: “Make science work for Canada”

(i.e. provide solutions for OGD and industry)• Funding: government & revenue


Social values

driver

impact

NRCUniversitiesOGDsIndustry

Technology/Services

Knowledge/ Discovery

Industry

IIT

NRC’s role & place

ENG

IMTI

IMI

IOT

IAR

PBI

IBDIMB

BRI

HIAIBS INMS

NINT

IIT

ICPET

IMS

IRC

IFCI

SIMS

LS PS

GHIINH

IIT


CBSA, RCMPDRDC, DND/CF

CSIS, PPTC, CIC, PCOPSERC, TC, CATSA

CPRC

- VRS program

Flight Facility

papers,conferencesCVPR, CRV

Social values

driver

impact

NRCUniversitiesOGDsIndustry

wrt. Video Technologies

Technology/Services

Knowledge/ Discovery

Industry

IIT

Other clients:HealthMedia

Education

NRC’s role & place

…


A closer look


A closer look


A closer look

FF

IIT


Institute for Information Technology (IIT-ITI, or IIT)

Revenue & Forecast

$0

$200,000

$400,000

$600,000

$800,000

$1,000,000

$1,200,000

$1,400,000

$1,600,000

$1,800,000

$2,000,000

FY 03-04 FY 04-05 FY 05-06 (Forecast)

Fee forServiceRevenueCollaborativeRevenue

LicensingRevenue

Level of Activityby Site

Fredericton

Gatineau

Moncton

Ottawa

Saint JohnSydney

Total: $18,976M

NRC-IIT generates most of its revenues through licensing its enabling technologies

• Funding: Govern + Revenues• Location: Ottawa + Atlantic (e-IT)

Make smth that works (and sales)!


IIT:Areas of Specialty

•3D Imaging, Modeling and Visualization•Data and Text Mining •Computational Video

•Natural Language Processing •Information Security, Privacy and Trust•Human Computer Interaction

•Intelligent Internet Applications•Software Engineering


IIT: Some technologies

• From Space Shuttle (Canadarm & Canadarm2), to Hollywood, to Fine Arts, to Security applications


Video Recognition Systems(VRS)

• Part of Computational Video Group (formed in 2001)• Our Mandate: Develop Video Recognition

technologies for Canadian companies and OGD’s• Formerly called Perceptual Vision Technology

• Higher mission:

Computer Vision allows computers to see.Perceptual Vision allows computers to understand

what they see.™

Test-bed and Criteria:If you are able to recognize him / it,

why computer can’t ?..


What we do

• Research, coding…• Working with companies: consulting, making prototypes• Working with OGD: consulting, joint proposals

• “Outreach & influence”: raising awareness about Video Recognition Technology – as opposed to Video Technology– public (as you see surveill. cam in the hall or watch TV), – scientists (from diff. dept: CS,Phys,NeuroBio,Psych)– government (as they allocate money)

• Organizing conferences – CRV VideoRec’07

What we don’t do (vs. Universities): teaching (except specialized lectures)


A bit of history

• First projects: Canadarm2• Then: Nouse™, Surveilance


VRS now• 4 permanent Dr.’s, • 3-10 contract / guest /

coop / student workers• 2 Tech.Support

• Collaborate with Ottawa U. (VIVA lab)

• Works closely with OGD that require Video Tech.

• Lead: D. Gorodnichy

• Collaborators: M. Fiala:- Markers Tracking algorithms - Panoramic Surveillance systems

G.Roth:- Feature detection, - Projective Vision algorithms


Application directions

Domain 1: Security, Surveillance and Monitoring.

Domain 2: Visually-enabled computer-human interaction (inc. hand-free user interfaces)

Domain 3: Visually-enhanced communication and Intelligent video processing (inc. annotation of video)

• Canada Borders Services Agency (CBSA)• Canadian Police Research Center (CPRC, RCMP)

• Ottawa Health Center

• U of Ottawa Music Dept.• Several Canadian/Ontario private companies

(And partners)


Research & ResultsTechnologies developed

1. Neuro-associative memorization/ recognition Face Recognition from (low-res) Video(Better than correlogram) Object learning

2. (Multiple) object detection and tracking Automated tele-operating, …

3. Nose (convex-shape) tracking Hands-free interfaces

4. Critical Evidence Snapshot extraction Intelligent Surveillance

5. Fingers, hands detection trackingPiano playing annotation


1. Neuro-associative memorization/ recognition Face Recognition from (low-res) Video(Better than correlogram) Object learning

2. (Multiple) object detection and tracking Automated tele-operating, …

3. Nose (convex-shape) tracking Hands-free interfaces

4. Critical Evidence Snapshot extraction Intelligent Surveillance

5. Fingers, hands detection trackingPiano playing annotation

Research & ResultsTechnologies developed

(X,Y)

name


1. Face Recognition in VideoGoal:

Recognize faces from regular video quality:

Bad news: low-resolution (TV:320x260) data i.o.d = 12 pixels !Good news: face detectors (Viola,CMU,MIT,Pitts) can detect such faces !• So, real problem is: Different recognition modality!• So, for solution: make use of temporal domain! (“Super-resolution” comment)

• Applications: (not for identification (yet), but …) for user recognition, TV annotation, soft biometrics, multiple-people tracking, multiple-camera tracking, back-tracking


12 pixels between the eyes should be sufficientNominal face resolution for video-based recognition

1) Efficient visual attention mechanisms (motion etc)2) Decision based on accumulating results over t.3) Three main principles of neuro-processing

1. non-linear processing, 2. massively distributed collective decision making3. synaptic plasticity.

Allows: a) to accumulate learning data in time by adjusting

synapses, b) to associate a visual stimulus to a semantic

meaning based on the computed synaptic values

Keys principles we adhere to(from human vision recognition system)

(X,Y)

“Paul”


Face Recognition in Video

Solution:- 12 pixel i.o.d. face model +

accumulation over time while tracking (both in training and recognition)

- combination of neuro-biological and computer vision approaches

Tested: on TV shows, on computer user recognition


See also:

Image and Vision Computing Journal, Special Issue on “Face Processing in Video Sequences”.

(Vol. 24, No. 6). June 1, 2006. – Six-page Editorial…

– On what makes video processing special– On nominal (optimal for video) face resolution– On promising directions and future trends– On video test-sets: Why not to use “Gone with the wind”?

Also• CVPR’04 Workshop on Face Processing in Video (FPiV’04), Washington

(www.vision-interface.net/fpiv04 )• CRV’05 Workshop on Face Processing in Video (FPiV’05), Victoria

(www.vision-interface.net/fpiv05 )


Goals: - make surveillance

data manageable, - be suitable for

long-range monitoring, - be suitable for

low quality video (which they often are), - be affordable (time-wise, space-wise)Challenge: foreground is not easy to detect/track in low-quality

data: thru glass, blurred, against sun (24h), etc..Solution:- Automated extraction of A.C.E. (Annotated Critical Evidence)- Integrates recent advances in video detection and tracking

2. A.C.E. Surveillance


A.C.E. Surveillance• Is currently used to monitor

several premises 24/7 (long-term)

• Fast/Succinctsummarization

obtained(in terms of ACE

snapshots)

• Little space

• Browsableby time

by objectby similarity*


From 22:00 to 7:00with web-cam

See also:CRV’06 Workshop on Video Processing for Security (VP4S-06), Quebec.

(www.computer-vision.org/4security )


3. Automated Tele-operator• Replaces manual video operator:

automatically switches cameras• Code taken from surveillance monitoring application

(another test-bed comment on Test Data)


4. Nouse™ (Nose as Mouse)

• Goal: To enable hands-free control

• Problem: sub-pixel precision,robustness and convenience is needed.

+ For truly hand-free user face recognition is required.

x y , z α , β γPUI

monitor

binary eventON

OFF

recognition /memorization

Unknown User!


5. Pianist playing annotationWhich hand/finger played a note ?

• Problem: Detect/Track hands and fingers. Multi-object, deformable object tracking is hard.

• Unbiased test-bed for hands tracking – very different from sign language + pianist hands (flexible and fast)


Image Search for Security Apps.(Gerhard Roth)

• UAV takes large number of images–Returns to same environment at a later time–Find images matching ones previously taken

• City has a host of cameras at intersections–Given a single view of a vehicle–Want to find this vehicle at other intersections

• Take many views of an object or vehicle–Many images of vehicle at different viewpoints–Want to find this vehicle in surveillance video


•bi-tonal (only black and white)•4 corners: for 6-DOF camera/pattern pose determination•Digital Methods: Error Correction, CRC-16 Checksum

ARTag Fiducial Marker System (Mark Fiala)


•Augmented Reality

•General Purpose Tracker

•Robot Navigation and control

Marker Detection Applications(Fiala, Roth)


Part 2:Interests in Video Technology from OGD

Receptor capacity for Video Technology: users and developers

National/Regional security programs and intentions


Ottawa sector and national(by interest as customers vs. R&D)

* - expressed interest in participation in external evaluation for DTO VACE technologies

* NRC-IITCVG-VRS program

* NRC – FRL(Flight

Facilities)

* CBSA -LSSD

Border Services

* DRDC – AIS(UAV)

DRE S/V…

* RCMP(Police)

DRDC – NI“DARPA”

(Biometrics)

Industry - CRC(Communication

Research Center )

* CF / DND(Forces)

Classified only

CSIS“CIA”

Also: PPTC, PSERC

Capacity/Readiness for R&D

Capacity/Readiness for testing/using as customer


Companies to know

• Point Grey Research (Vancouver, UBC) – firewirecams & video processing

• i3DVR – “Intelligent” DVR• Deep Development Corp / Gatekeeper Systems

“High Resolution for Tough Environments” – DVR (won US Force contract for Boeing Military planes)

• March Networks – DVRs


Universities to know

• UBC (Vancouver)• Ottawa U, VIVA Lab works closely with NRC• York U (Toronto)• …

(see PC for CRV VideoRec’07 workshop)(also www.vision-interface.net – for complete list)


17 slides omitted for online version


Summary

• How can we contribute to VACE:– Science-wise: Reviewing, Testing– Client-wise: Testing– Liaisoning with OGD and

national/regional/municipal surveillance programs– Consulting, Tips, Feedback, Cheering-up :)– Inviting (Making) everyone here to participate in:

CRV Workshop on Video Processing and Recognition(VideoRec’07)

Montreal, Canada, May 28-30

Documents

Outline National Research Council of Canada · 9. Video Recognition projects and interests (Dr. Dmitry Gorodnichy) DTO VACE, Phase III meeting - October 2006, Reston, USA A closer