Noisy Video Super-Resolution

Feng Liu, JinjunWang,ShenghuoZhu (MM’08)University of Wisconsin-Madison, NEC Laboratories America, Inc.

第一組： 資訊四 B95902105 黃彥達 資訊碩一 R98922046 蔡旻光 網媒碩二 R97944012 鄒志鴻

2

Outline

Introduction Goal File Format

Noise Reduced Image Proposed Approach Motion Estimation & Estimated Super-

Resolution Result Implementation Result Conclusion

3

Introduction

Low-quality videos often not only have limited resolution but also suffer from noise In fact, the requirements of de-noising & super-

resolution is quite similar

This paper present a unified framework which achieves simultaneous video de-noising and super-resolution algorithm by some measurements of visual quality

Goal

Refine low-quality videos from YouTube, and make the video better effects, which has better quality by human eyes.

Input is low-quality and noise-included (block effects or somewhat noise) videos

5

File Format

.3gp file Frame rate: 15(our video) or 25 Frame size: 176(w) * 144(h) MPEG-4 Part 12 It is used on 3G mobile phones

also can be played on2G and 4G phones.

Our video: 867 KB/ 98 sec

Noise-Reduced Image

mv-SAD Gaussian-space

Gaussian-time

| p(I,j) – p(i’, j’) | > threshold

Gaussian Space

Frame t

Pixel(I,j)

Standard deviation

Set Mean = 0

Motion Vector

Frame tPixel ( i , j , t)

Frame t+1

Pixel ( i + mv_i , j + mv_j , t+1)

(mv_i , mv_j)

Gaussian Time

Fram

e t

- 2Fram

e t

- 1Fram

e t

Space Gaussian

Time Gaussian

Pixel(I,j)

Fram

e

t+1Fram

e

t+2Fram

e t

Shot Detecti

on

Noise-Reduced ImageBefore After

11

Proposed Approach – 1 / 4 Consider the visual quality with respect to

the following 3 aspects: Fidelity Preserving▪ To achieve similar high-resolution result

Detail Preserving▪ Enhanced details (edge)

Spatial-Temporal Smoothness▪ Remove undesirable high-frequency contents (e.g. jitter)

12

Proposed Approach – 2 / 4 Fidelity Preserving

Conventional metrics:▪ Measure fidelity by the difference between Ih & Il would

be problematic & waste useful time-space information in video

Proposed metrics:▪ Estimate an approximation of super-resolution results

from space-time neighboring pixels▪ The fidelity measurement:

see next page for details

noised

13

Proposed Approach – 3 / 4 Detail Preserving

Enhanced details (edge)

Contrast preserving▪ Human visual system is more sensitive to contrast

than pixel values▪ Gradient fields of Ih & should be close

,where Wk is one or zero if the patchk with/o edges (canny detector)

14

Proposed Approach – 4 / 4 (Spatial-Temporal) Smoothness

Smooth results are often favored by the human system Encourage to minimize:

A 2-D Laplace filter may be

Spatial-temporal Laplacian

OR

15

An Optimization Problem

Proposed Measurements

A quadratic minimization problem to solve (AX = b):

Contrast

Similarity

Detail Information(edge)

Spatial-Temporal Smoothness

16

Implementation – 1 / 3

Inputlow hI~

hI~

hG~

hI~

)1,()1,(~~

tIwtIw hmv

hmv

= X

I

6 -1 … -1-1 6 -1 … -1 -1 6 -1 … -1

Laplacian

Gradient-1 0 1 … 1 -1 0 1 … 1 -1 0 1 … 1

EdgeMinimize

Motion Estimation

+

Result (X)

Fidelity

Gaussian filter

mvw

fidv

fidg

dt

sm

17

Implementation – 2 / 3

....,5001111 slowtoosbxA nnnn

hI~

hG~

hI~

)1,()1,(~~

tIwtIw hmv

hmv

= X

I

6 -1 … -1-1 6 -1 … -1 -1 6 -1 … -1

Laplacian

Gradient-1 0 1 … 1 -1 0 1 … 1 -1 0 1 … 1

EdgeMinimize

Fidelityfidv

fidg

dt

sm

!!!2'1 sbAxAxAA tnnn

t

by sparse least square solver

18

Implementation – 3 / 3

Adjustments for the weight terms The measurement term is more emphasized if

the weight is larger

By iteratively experiments for our test data, we took

However, we found that the best weight set may be different for different videos

3.0,1.0,1,1 smdtfidgfidv

19

Result

352 x 288 Result

20

Result

352 x 288 Result

21

Result

352 x 288 Result

22

Conclusion The proposed framework formulates noisy video

super-resolution as an optimization problem, aiming to maximize the visual quality

By exploiting the space-temporal information, we can estimate a better baseline than conventional fidelity measurement

The properties include fidelity-preserving, detail-preserving and smoothness are considered to achieve the best visual quality results

23

Thank you!!