Upload
bunny
View
49
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Super-Resolution. Authors: Wagner, Waagen and Cassabaum Presented By: Mukul Apte. Introduction. Definition: Create High Resolution visual output from Low Resolution visual input. Mathematical assistance to features viz. motion detection, face recognition, person detection. - PowerPoint PPT Presentation
Citation preview
Super-Resolution
Authors: Wagner, Waagen and Cassabaum
Presented By: Mukul Apte
Introduction
Definition: Create High Resolution visual output from Low Resolution visual input.
Mathematical assistance to features viz. motion detection, face recognition, person detection. Action Packed Sports Images Astronomy Medical Imaging Surveillance
Range of resources: low (single camera, polaroid lenses) to high (super-resolution chips, androids-ASIMO)
Basic ideaGiven: A set of degraded (warped, blurred, decimated, noised) images:
Required: Fusion of the measurements into a higher resolution image:
Types
Static Super-Resolution (SSR) - The creation of a single improved image, from the finite measured sequence of images
Dynamic Super-Resolution (SSR) - Low Quality Movie In - High Quality Movie Out
Simple Example
Concepts I: Raster, CCD, Image in mathematical domain, Matlab
Concepts II: Bresenham’s line algorithm, non-uniform interpolation, frequency domain
Simple Example
D
For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough (pixel density = twice highest static frequency), perfect reconstruction is possible.
D
Due to our limited camera resolution, we sample using an insufficient 2D grid
2D
2D
Simple Example
However, we are allowed to take a second picture and so, shifting the camera ‘slightly to the right’ we obtain
2D
2D
Simple Example
Similarly, by shifting down we get a third image
2D
2D
Simple Example
And finally, by shifting down and to the right we get the fourth image
2D
2D
Simple Example
Simple Example - Conclusion
It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed.
This is Super-Resolution in its simplest form
Uncontrolled Displacements
In the previous example we counted on exact movement of the camera by D in each direction.
What if the camera displacement is uncontrolled?
Uncontrolled Displacements
It turns out that there is a sampling theorem due to Yen (1956) and Papoulis (1977) covering this case, guaranteeing perfect reconstruction for periodic uniform sampling if the sampling density is high enough (1 sample per each D-by-D square).
Uncontrolled Rotation/Scale/Disp.
In the previous examples we restricted the camera to move horizontally/vertically parallel to the photograph object.
What if the camera rotates? Gets closer to the object (zoom)?
Uncontrolled Rotation/Scale/Disp.
There is no sampling theorem covering this case
Further Complications
Sampling is not a point operation – there is a blur
Motion may include perspective warp, local motion, etc.
Samples may be noisy – any reconstruction process must take that into account.
Static Super-Resolution
Static Super-Resolution
Algorithm
N321 Y,,Y,Y,YfX̂
1Y
X̂
2Y NY
Low Resolution Measurements
High Resolution Reconstructed Image
t
3YStatic Super-Resolution
Low Resolution Measurements
High Resolution Reconstructed Images
1tX̂,tYftX̂
ttY
ttX̂
Dynamic Super-Resolution
Algorithm
t
t
Dynamic Super-Resolution
Approach
Image Registration Motion Estimation Projection onto High-Resolution Grid
Non-Uniform Interpolation Frequency domain – alias correction
Registration Projection
High Res GridLow-res Images Registration (sub-pixel grid)
•Rotation Calculation
•Correlate 1st LR image with all LR images at all angles
OR
•Calculate energy at all angles for all LR images. Correlate energy vector to find the rotation angle
Anglei = max index(correlation(I1(θ), Ii (θ)))
Energy at angle Ii(θ)
LR image 1
Energy at angle I2(θ)
LR image 2
i = 2,3,..,N (number of LR images)
1.1 Registration (angle)
Shift Calculated using Frequency Domain Method
Δs [Δx Δy]T
u [fx fy]
Fi (uT) = ej2πuΔsF1(uT)
Δs = angle( Fi (uT) / F1(uT) )
2πu
• Used only 6% lower u (high freq could be aliased)
• Used least square to calculate Δs
1.1 Registration (shift)
Input: Down-sampled aliased images Goal I: Correct the low-freq aliased data Goal II: Predict the lost high freq values
2.1 Frequency Domain
π-πOriginal High-Res
π-πDown-sampled
π/2-π/2 π
Aliased (fix it)
Lost (find it)
Up-sampledπ-π
Desired High-Res
I (known pixel positions) = Known Values
I_fft = fft2(I)
I_fft(higher Freq) = 0
I= ifft2 (I_fft)
2.2 Projection onto High-res grid
Papoulis-Gerchberg AlgorithmProjection onto convex sets
Known pixel values Known Cut-off freq in the HR image
Algorithm:
N
1k
1kkkkkkk ,0~V ,VXY
WNFHD
X
High-Resolution
ImageH
H
Blur
1
N
F =I1
FN
Geometric Warp
D
D1
N
Decimation
V1
VN
Additive Noise
Y1
YN
Low-Resolution
Images
SSR – The Model
F[j,i]=1
X Z
Per every point in X
find a matching point in Z
N
j
1
N
j
1
z
z
z
1
1
1
x
x
x
00
Warp – Linear Operation
We assume that the images Yk and the operators Hk, Dk, Fk,& Wk are known to us, and we use them for the recovery of X.
Yk – The measured images (noisy, blurry, down-sampled ..)
Hk – The blur can be extracted from the camera characteristics
Dk – The decimation is dictated by the required resolution ratio
Fk – The warp can be estimated using motion estimation
Wk – The noise covariance can be extracted from the camera
characteristics
Model Assumptions
X
Y
Y
Y
NNN
222
111
N
2
1
FHD
FHD
FHD
Clearly, this linear system of equations should have more equations than unknowns in order to make it
possible to have a unique solution.
Special Condition - Noiseless
SSR – Handling Problems
VXY
N
1kkkk VXY F
VXY D
VXY H
YIX̂1T
WSS
YX̂ T1TT HWSSHH
YX̂ T1TT DWSSDD
N
1kk
Tk
1
TN
1kk
Tk YX̂ FWSSFF
XXXA TT WSSUsing
Single image de-noising
Single image restoration
Single image scaling
Motion compensation average
Simulated error
Weighted edges
Back projection
j
N
1k
Tjkkkkk
Tk
Tk
Tkj1j X̂X̂YX̂X̂
WSSFHDWDHF
SSR Standard Equation
All the above operations can be interpreted as iterative operations performed on images.
(Typically 15-20 iterations are performed.)
Thumb Rule on Desired Resolution
Assume that we have N images of M-by-M pixels, and we would like to produce an image X of size L-by-L. Then –
MNL
Taj Mahal – Low-res image I
Initial Setup
FFT (Initial image)
Papoulis – Gerchberg Algorithm
Image at iteration 0
FFT
I(high freq) =0
Known PixelValues
Image after 1st iteration
Papoulis – Gerchberg Algorithm
Image at iteration 1
FFT
I(high freq) =0
Known PixelValues
Image after 10 iterations
Papoulis – Gerchberg Algorithm
After 50 iterations
SR Reconstructed imageBilinear Interpolation Bicubic Interpolation
Papoulis – Gerchberg Algorithm
Original Low-res images(Courtesy: Patrick Vandewalle)
Results (Images - I)• Input: 4 snaps using a high-res digital camera• Cropped the same part of each image • SR algorithm compared with bicubic interpolation
Bicubic Interpolation
Results (Images - I)
Super-resolution
Results (Images - I)
Low-Res Image I Low-Res Image II
• SR Algorithm didn’t work as expected !!!
• Reason:
• Motion was not restricted to shifts & rotation
• Images had affine mapping
• Rule I - Need Correct Registration
Results (Images - II)
Original Bicubic SR
Why didn’t SR work???• Low-res images were created by forcing shifts at
critical velocities• Rule II - If low-res frames are at critical velocities,
can’t create good high-res frame
Results (Frames - I)
Original Bicubic SR
Why did SR work so well???• Low-res images were created by forcing shifts at
non-critical velocities• Rule III If low-res images have all the info
about high-res then HR image can be perfectly constructed
Results (Frames - II)
• Superresolution with multiple motions between frames - create high res video
• Predict the high-res frequency components using wavelet methods
Predict Predict
Predict
Future Work
Acknowledgements
Prof John Apostolopoulos Prof Susie Wee Patrick Vandewalle
THANK YOU!!!
QUESTIONS?
COMMENTS?