Using Photographs to Enhance Videos of a Static

ScenePravin Bhat1, C. Lawrence Zitnick2, Noah Snavely1, Aseem

Agarwala3, Maneesh Agrawala4, Michael Cohen1,2, Brian Curless1, Sing Bing Kang2

EGSR 2007

University of Washington1, Microsoft Research Redmond2 University of California3, Adobe Systems4

An overview of Spacetime Fusion

Motivation

• Low quality video

Input Video

Motivation

• Low quality video• Reconstructed video

Input Video

Reconstructed Video

Motivation

• Low quality video• Reconstructed video

– Reconstructed from photos– Good spatial reconstruction– Bad temporal reconstruction

Input Video

Reconstructed Video

Motivation

• Spacetime Fusion result

Input Video

Spacetime Fusion Result

Motivation

• Spacetime Fusion result– Spatial properties of reconstruction– Temporal properties of input video

Input Video

Spacetime Fusion Result

• Define a 3D gradient field

Spacetime Fusion

• Define a 3D gradient field – Spatial gradients from reconstruction– Temporal gradients from input video

Spacetime Fusion

• Define a 3D gradient field – Spatial gradients from reconstruction– Temporal gradients from input video– Key Idea

• Temporal gradients defined betweenmotion compensated temporal neighbors

Spacetime Fusion

• Define a 3D gradient field – Spatial gradients from reconstruction– Temporal gradients from input video– Key Idea

• Temporal gradients defined betweenmotion compensated temporal neighbors

Video frame: t Video frame: t - 1

Spacetime Fusion

• Define a 3D gradient field – Spatial gradients from reconstruction– Temporal gradients from input video– Key Idea

• Temporal gradients defined betweenmotion compensated temporal neighbors

Video frame: t Video frame: t - 1

Gt

Gt(x, y, t) = V(x, y, t) - V(x, y, t - 1)

Spacetime Fusion

• Define a 3D gradient field – Spatial gradients from reconstruction– Temporal gradients from input video– Key Idea

• Temporal gradients defined betweenmotion compensated temporal neighbors

Video frame: t Video frame: t - 1

Gt

Gt(x, y, t) = V(x, y, t) - V(x - u, y - v, t - 1)

Spacetime Fusion

• Define a 3D gradient field – Spatial gradients from reconstruction– Temporal gradients from input video– Key Idea

• Temporal gradients defined betweenmotion compensated temporal neighbors

• Increases compatibility betweentemporal gradients and spatial gradients

Spacetime Fusion

• Define a 3D gradient field – Spatial gradients from reconstruction– Temporal gradients from input video– Key Idea

• Temporal gradients defined betweenmotion compensated temporal neighbors

• Increases compatibility betweentemporal gradients and spatial gradients

• Integrate the 3D gradient field

Spacetime Fusion

Spacetime Fusion

• Integrating the gradient field

Solve linear system:Av = b

• Integrating the gradient field

Solve linear system:Av = b

Constraints:vx, y, t – vx-1, y, t = Gx(x, y, t)vx, y, t – vx, y-1, t = Gy(x, y, t)vx, y, t – vx-u, y-v, t = Gt(x, y, t)

Spacetime Fusion

Applications

Enhanced Exposure

Input Video

Edit Propagation

Edit Propagation

User Edits

Edit Propagation

User Edits

Edit Propagation

User Edits

Edit Propagation

User Edits

Edit Propagation

User Edits

User Edits

Edit Propagation

Edited Video

Edit Propagation

Super-Resolution

Conclusion

• Spacetime fusion

Conclusion

• Spacetime fusion – Combines spatial and temporal gradients

from two different sources

Conclusion

• Spacetime fusion – Combines spatial and temporal gradients

from two different sources– Requires motion vectors for temporal

source• stereo (static scenes)• flow (dynamic scenes)

Conclusion

• Spacetime fusion – Combines spatial and temporal gradients

from two different sources– Requires motion vectors for temporal

source• stereo (static scenes)• flow (dynamic scenes)

– Major applications• Enforcing temporal coherence• Transferring lighting information