Haojie LiJinhui TangSi WuYongdong ZhangShouxun Lin

Automatic Detection and Analysis of Player Action in Moving Background Sports

Video Sequences

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 20, NO. 3, MARCH 2010

IntroductionGlobal motion estimationPlayer body shape segmentationAnalysis of actionExperimental results

OUTLINE

Presents a system for automatically detecting and analyzing complex player actions in moving background sports video sequences

Providing kinematic measurements for coach assistance and performance improvement

Video-based approach : low cost, no interference to the performance of players, can analyze the rich archived video clips

INTRODUCTION

Block diagram of the proposed system

INTRODUCTION

Video Sequenc

e

Global motion

estimation

Action clips detection

Action recognition

Player shape

segmentation Visual

analysis

Kinematic analysis

Highlights library

1. The detected highlights are stored into library as video summaries for user’s quick browsing2. Action recognize using CHMM ( continuous hidden markov models )

Action-based video indexing

Kinematic parameters

The result accuracy of most global motion estimation methods are influenced by outliers.

Some methods used experimentally determined or manually specified thresholds to remove outliers, thus are not adaptive to other data.

In this paper 6-parameter affi ne model & Fisher linear discriminant analysis are used. is point pair.

GLOBAL MOTION ESTIMATION

𝑢𝑖=𝐻 𝑖 𝐴

(𝑥𝑖𝑦 𝑖) (𝑥 𝑖

❑ ′ 𝑦 𝑖❑′ 1

0 0 00 0 0𝑥𝑖

❑ ′ 𝑦 𝑖❑ ′ 1) (𝑎 ,𝑏 ,𝑐 ,𝑑 ,𝑒 , 𝑓 )𝑇

: Point p in current frame

: Point p in frame

Global motion parameter

Finding point pairs in and Calculate the global standard variance of pixel values in Scan and check each n*n block. If standard variance of a

block is large enough ( > ) the upper left corner of the block is selected as

is obtained by searching nearby blocks in

GLOBAL MOTION ESTIMATION

𝐼𝑘 𝐼𝑘−1

By solving , we can obtain initial solution Since is the approximate solution and motion of

outliers is not consistent with GM according to residual errors, we can separate point pairs into inliers and outliers

We can use inliers to refine A

GLOBAL MOTION ESTIMATION

𝑟 𝑖=𝑢𝑖−𝐻 𝑖𝐴∗

𝑢𝑖∈ {𝑜𝑢𝑡𝑙𝑖𝑒𝑟 ,|𝑟 𝑖|2≥𝑇

𝑖𝑛𝑙𝑖𝑒𝑟 ,|𝑟 𝑖|2<𝑇

 

[24] N. Ostu, “A threshold selection method from gray level histogram,” IEEETrans. Syst. Man. Cybern., vol. 9, no. 1, pp. 62–66, Jan. 1979.

GLOBAL MOTION ESTIMATION

𝐼𝑘𝐼𝑘−1Global motion vectors between two frames

Outlier filtering Aligned image using estimated GME parameters

Difference image between (b) (e)

Background is accurately aligned

An algorithm has been proposed[29] (algo1)Main steps of Algorithm 1

Global motion estimation Foreground separation : three-frame-difference Background construction :

is aligned to GM Using temporal median

Object segmentation Background subtraction Significance test[30] is used to decide threshold to binarize Connected component analysis Snack model[31] is adopted to smooth each remaining

component’s boundary

PLAYER BODY SHAPE SEGMENTATION

𝐼𝑘 𝐼𝑘+1 𝐼𝑘+𝐿𝐼𝑘−𝐿 … …𝐼𝑘−1Consecutive 2L-1 frames

𝐼𝑘−1 𝐼𝑘 𝐼𝑘+1

d1 d2

𝐷1 𝐷2∩ :

An algorithm has been proposed[29] (algo1)Main steps of Algorithm 1

Global motion estimation Foreground separation : three-frame-difference Background construction :

is aligned to GM Using temporal median

Object segmentation Background subtraction Significance test[30] is used to decide threshold to binarize Connected component analysis Snack model[31] is adopted to smooth each remaining

component’s boundary

Results of Algo1

PLAYER BODY SHAPE SEGMENTATION

Problem:Work well only when object has apparent motion

Reason:Doesn’t consider the object motion between frames

Improved version (Algo2) take object motion between frames into consideration select frames with apparent object motion to construct background image

We use global motion between frames as the measurement of object motion

Key-frame selection: Neighboring frames with global motion > Th1 A frame’s cumulative global motion to the nearest key-frame >

Th1 When the cumulative global motion from to key-frame > TH2 no more frames are needed!!

PLAYER BODY SHAPE SEGMENTATION

Only selected key-frames are used to construct background

𝐼𝑘 𝐼𝑘+1 𝐼𝑘+𝐿𝐼𝑘−𝐿 … …𝐼𝑘−1Consecutive 2L-1 frames

 … …kf1 kfL1Kf-1kfL2

L1L2

𝐿𝑖=min❑ (𝐿 , argmin𝐽 𝐶𝑀 𝑖 (𝑘 , 𝐽 )≥ h𝑇 2)

Results of Algo2

PLAYER BODY SHAPE SEGMENTATION

Th1 = 4TH2 = 50L = 11

Kinematic analysis : we present an automatic method through 2-D articulated human body model fi tting, to get the joint angles. S=( x, y, θ, θ1, θ2, θ3, d )

THE ANALYSIS OF ACTION

Human body model

Test body shape

Edge mapDistance transform map [35]

The initial parameter is refine by searching with annealed particle algorithm[34]

Global position & rotation parameter

neck, hip, knee angle

Visual analysisMotion Panorama

Overlay composition

THE ANALYSIS OF ACTION

Temporal median filtering

• Compare actions performed by different players or by the same player at different time

• No constraint that two clips should be of same scene

Global motion estimation

EXPERIMENTAL RESULTS

( Interframe transformation fidelity )

Aligned image to Ik by global motion compensated on Ik-1

RANSAC & LTS with refinement procedures

Player body segmentation

EXPERIMENTAL RESULTS

Action recognition

EXPERIMENTAL RESULTS

Kinematic analysis

EXPERIMENTAL RESULTS

Visual analysis

EXPERIMENTAL RESULTS

Experiments on jump videos

EXPERIMENTAL RESULTS

EXPERIMENTAL RESULTS