Neural Rendering - dvl.in.tum.de · Photo-realistic Image Synthesis The Rendering Equation [Kajiya...

Neural Rendering

1Prof. Leal-Taixé and Prof. Niessner

Rendering

3D Scene:- Material- Lighting- Geometry

(incl. animation)

Camera View Point- Extrinsics- 6 DoF (3rot, 3trans)

Camera Def.- Intrinsics- Often:

- focal length- principal point)

Prof. Leal-Taixé and Prof. Niessner

Photo-realistic Image Synthesis

The Rendering Equation [Kajiya 86]

Need 3D Content for Rendering

Textures Material & LightingGeometry

Computer Vision for Reconstruction

ICCV’09 [Agarwal et al.]: Building Rome in a Day5Prof. Leal-Taixé and Prof. Niessner

Computer Graphics

3D Digitization

Computer Vision

Traditional Graphics vs Deep Learning

3D Model + Textures + Shading -> Synthetic Image

Star Wars Rogue One

Discriminator loss

Generator loss

[Karras et al. 18]

Generative Adversarial Networks

Idea of Neural RenderingNovel View point synthesis:

Neural Network-> Encodes entire scene description, lighting, materials,

6 DoF Camera Pose / View Point

Neural Rendering with Pix2PixGround truth for training- Pose + Target Image (e.g., observed from real world)- Constrain with re-rendering loss

Testing- Given unseen pose, generate image

Prof. Leal-Taixé and Prof. Niessner 9

Neural Rendering with Pix2Pix

Other Neural Rendering- Conditioned on Faces (Deep Video Portraits)

- Conditioned on Human Skeleton (Everybody Dance Now)

Neural Rendering with Pix2Pix

Deep Voxels

[Sitzmann et al. CVPR’19] Deep Voxels13Prof. Leal-Taixé and Prof. Niessner

Deep Voxels• Main idea for video generation:

– Why learn 3D operations with 2D Convs !?!?

– We know how 3D transformations work• E.g., 6 DoF rigid pose [ R | t ]

– Incorporate these into the architectures• Need to be differentiable!

– Example application: novel view point synthesis• Given rigid pose, generate image for that view

Deep Voxels

2D U-Net

Rendering

Lifting Layer2D 3D

2D U-Net

2D FeatureExtraction

Source View R, t

Projection Layer

OutputSourceTarget

View R, t3D U-Net

3D Features

Simplified overview for novel view synthesis

Deep Voxels

Issue: we don’t know the depth for the target!-> Per-pixel softmax along the ray-> Network learns the depth

Occlusion Network:

Deep Voxels

[Sitzmann et al. ’18] Deep Voxels18Prof. Leal-Taixé and Prof. Niessner

Deep Voxels

Deep Voxels: Insights• Lifting from 2D to 3D works great

– No need to take specific care for temp. coherency!

• All 3D operations are differentiable

• Currently, only for novel view-point synthesis– I.e., cGAN for new pose in a given scene

• But: limited resolution due to dense 3D voxel grid

Importing 3D structure from CG

Representa

Multi-Plane Images Voxelgrids Image-based Point Clouds Implicit Function

Renderer(Alpha) compositing Volumetric

Ray-basedRasterization Splatting Sphere-Traced

Volumetric

Both Scene Representation and Differentiable Renderer often adapted from traditional computer graphics.

Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)21Prof. Leal-Taixé and Prof. Niessner

Representa

Volumetric

Representa

Volumetric

Representa

Volumetric

Fast renderingHigh qualityGeneralizes

Only 2.5DSize

Representa

Volumetric

Only 2.5DSize

Representa

Volumetric

Only 2.5DSize

“True 3D”High quality

No reconstruction priors

Memory O(n3)

Representa

Volumetric

Only 2.5DSize

Memory O(n3)

Representa

Volumetric

Only 2.5DSize

Memory O(n3)

High quality

Requires good SFMNo compact

representation

Representa

Volumetric

Only 2.5DSize

Memory O(n3)

High quality

representation

Representa

Volumetric

Only 2.5DSize

Memory O(n3)

High quality

representation

High quality

Requires good SFM

Representa

Volumetric

Only 2.5DSize

Memory O(n3)

High quality

representation

High quality

Requires good SFM

Scene Representation NetworksSitzmann et al., Neurips 2019

Representa

Renderer

ℝ3՜ ℝ𝑛

ReLU MLP Generalized (learned) sphere-tracing

Generalizati

Hypernetwork

Scene Representation NetworksSitzmann et al., Neurips 2019

Full 3D Reconstruction from single image!

NERF: Neural Radiance FieldsMildenhall et al., arXiv 2020

Representa

Renderer

ℝ6՜ ℝ3

ReLU MLP +Positional Encoding

View Direction

Volumetric,stratified sampling

Generalizati

NERF: Neural Radiance FieldsMildenhall et al., arXiv 2020

Photorealistic, including view-dependence!(~100 images)

Requirements

Representa

Ray-basedRasterization Splatting Sphere-Tracing

Volumetric

Only 2.5DSize

Memory O(n3)

High quality

representation

High quality

Requires good SFM

High qualityMay generalize!

Expensive rendering, training

Neural Textures: Features on 3D Mesh

Siggraph’19 [Thies et al.]: Neural Textures

3D Geometry

Neural Texture

UV-Map

Sampled Texture

3D Geometry

Rendering

View R, t

Neural Texture

UV-Map

Renderer Output Image

Sampled Texture

3D Geometry

Rendering

View R, t

Neural Texture

Deferred Neural RenderingA

Deferred Renderer

Handcrafted ”Feature Maps“

Siggraph’19 [Thies et al.]: Neural Textures41Prof. Leal-Taixé and Prof. Niessner

Deferred Neural RenderingA

Deferred Renderer

Handcrafted ”Feature Maps“Learned

Neural

UV-Map

Renderer Output Image

Sampled Texture

3D Geometry

Rendering

View R, t

Neural Texture

Deferred Neural Rendering

Input UV-Map Ours

Novel View-Point Synthesis

Ground Truth Ours

Novel View-Point Synthesis

Scene Editing

Facial Animation

Neural Voice Puppetry

[Hannun et al.] DeepSpeech RNN

Output of the RNN of DeepSpeech:- Logits of alphabet (|alphabet|=29)

We use a time window (n=16)

Person-specificBlendshape Expression Model

Neural Voice PuppetryAudio2Expression Training

Hundreds of commentator videos available-- all with ‘neutral’ talking style --

Flame Model Basel Model

with pose

Big Open Challenges

Photo-realistic Reconstruction

Big Open Challenges: How much can AI do?

Big Open Challenges: 3D in NetworksWhy learn 3D operations, such as transformations?

-> differentiate known operators

Capsule networks are motivated by inverse graphics [Sabour et al. 17]

See you next week

Some Extra Slides:

Facial Reenactment

Dense Sparse

Neural Rendering and Reenactment ofHuman Actor Videos

[Liu et al. 19] Neural Rendering and Reenactment of Human Actor Videos 83Prof. Leal-Taixé and Prof. Niessner

Body ReenactmentDense Sparse

Open Challenges

• Motion Capturing

• Person-specific Motions/Expressions

• Temporal Stability

• Image Quality

Neural Rendering - dvl.in.tum.de · Photo-realistic Image Synthesis The Rendering Equation [Kajiya...

Documents

Rendering 101: A Primer on the Rendering Industry

Non-Photorealistic Rendering RENDERING + COMPOSITION

FREEDOM POINT PLAZA - LoopNet€¦ · Proposed Proposed Rendering Rendering Rendering Proposed Rendering Proposed Rendering. DEMOGRAPHICS 1-MILE 3-MILE 5-MILE 2017 Total ... Suites

Tribhuwan University Institute of Engineering Pulchowk Campus · mesh rendering, sprite and billboard rendering, depth-map rendering, skybox rendering etc. Audio System The audio

Signaling with conserved quantities: two realizations in CMOS and SFQ Logic Jim Kajiya Microsoft Research

GPU-Accelerated Path Renderingdeveloper.download.nvidia.com/GTC/PDF/GTC2012/... · 3D Rendering vs. Path Rendering Characteristic GPU 3D rendering Path rendering Dimensionality Projective

Prof. Stefan Schlechtweg-Dorendorf · Rendering-Pipeline Beleuchtungsmodelle und -berechnungen Texturen Computergraphik-Hardware Animation ... Rendering-System REYES (" Rendering

Lecture 3 recap · Neural Network Credit: Li/Karpathy/Johnson 3 Prof. Leal-Taixé and Prof. Niessner. Neural Network Depth Width 4 Prof. Leal-Taixé and Prof. Niessner. Neural Network

THE RENDERING EQUATION - Unicamp · Dallas, August 18-22 Volume 20, Number 4, 1986 THE RENDERING EQUATION James T. Kajiya California Institute of Technology Pasadena, Ca

VU Rendering SS 2015 186 - Institut für Computergraphik ... · Topics. Rendering theory Basic optics, rendering equation, filtering Rendering algorithms Ray tracing, radiosity, (bi-directional)

Revit Rendering Performance and Lighting - IMAGINiTportal.imaginit.com/...Revit_Rendering_Performance_and_Lighting.pdfFor the Revit rendering tool, ... Page 3 • Revit Rendering Performance

Volume Rendering Volume Modeling Volume Rendering Volume Modeling Volume Rendering 20 Apr. 2000

252.eon, Kajiya command line Datasets profile vs ... file252.eon, Kajiya command line Datasets profile vs. Reference Dataset The following are the profiles for the 252.eon benchmark,

Surface Detail and Non-PhotoRealistic Rendering Lecture 3 (Rendering)

Parallel Physically Based Path-tracing and Shading Part 1 of 2 · Path-tracing Algorithm •Solves the rendering equation, which was ﬁrst proposed by James Kajiya in 1986. •Generalizes

Toshiyuki Kajiya Japan Electronics & Information Technology Industry Association (JEITA) Conformity Assessment System Committee Product Safety Committee

Real-time Rendering of Procedural Multiscale Materials · Materials with anisotropic highlights were ﬁrst studied by Kajiya and Kay [1989], followed by works by Poulin and Fournier

Rendering Revolution - Nvidia · Rendering Revolution Ken Pimentel, ... Image-space vs. Ray-tracing rendering ... A few companies involved in the rendering revolution: Mental images:

Rendering Fur with Three Dimensional Textures James T. Kajiya and Timothy L. Kay

Rendering Revolution - NVIDIAon-demand.gputechconf.com/...Rendering-Revolution.pdf · Rendering 101: Image-space vs. Ray-tracing rendering Image-space 1. Project polygons onto an