Deep Convolutional Inverse Graphics Network

Tejas D. Kulkarni*1, Will Whitney*2, Pushmeet Kohli3, Joshua B. Tenenbaum4

1,2,4Computer Science and Artificial Intelligence Laboratory, MIT1,4Brain and Cognitive Sciences, MIT3Microsoft Research Cambridge, UK

1tejask@mit.edu 2wwhitney@mit.edu 3pkohli@microsoft.com 4jbt@mit.edu* First two authors contributed equally to this work.

Abstract

This paper presents the Deep Convolution In-verse Graphics Network (DC-IGN) that aims tolearn an interpretable representation of imagesthat is disentangled with respect to various trans-formations such as object out-of-plane rotations,lighting variations, and texture. The DC-IGNmodel is composed of multiple layers of convolu-tion and de-convolution operators and is trainedusing the Stochastic Gradient Variational Bayes(SGVB) algorithm [12]. We propose trainingprocedures to encourage neurons in the graphicscode layer to have semantic meaning and forceeach group to distinctly represent a specific trans-formation (pose,light,texture,shape etc.). Given astatic face image, our model can re-generate theinput image with different pose, lighting or eventexture and shape variations from the base face.We present qualitative and quantitative results ofthe model’s efficacy to learn a 3D rendering en-gine. Moreover, we also utilize the learnt rep-resentation for two important visual recognitiontasks: (1) an invariant face recognition task and(2) using the representation as a summary statis-tic for generative modeling.

1 Introduction

Deep learning has led to remarkable breakthroughs in au-tomatically learning hierarchical representations from im-ages. Models such as Convolutional Neural Networks(CNNs) [16, 13], Restricted Boltzmann Machines (RBM)based generative models [9, 25, 17], and Auto-encoders [1,29, 20] have been successfully applied to produce multi-ple layers of increasingly abstract visual representations.However, there is relatively little work on characterizingwhat is the optimal representation of the data. While re-searchers such Cohen et al. [6] have considered this prob-lem by proposing a theoretical framework to learn irre-

ducible representations having both invariances and equiv-ariances, coming up with the best representation for anygiven task is an open question.

The Vision as inverse graphics approach offers an interest-ing perspective to various desiderata for a good represen-tation [2, 6, 8]: invariance, meaningfulness of representa-tions, abstractions and disentanglement. Computer graph-ics is a function to go from compact description of scenes(graphics code) to images. This graphics code based rep-resentation is often disentangled to allow rendering sceneswith fine-grained control over transformations such as ob-ject location, pose, lighting, texture and shape. Therefore,it naturally produces representations satisfying the above-mentioned properties.

Recent work in inverse graphics [11, 19, 18, 14] followsa general strategy of first defining a probabilistic or non-probabilistic model with latent parameters, followed by aninference or optimization algorithms to find the most ap-propriate set of latent parameters given observations. Re-cently, Tieleman et al. [27] moved beyond this two stagepipeline by using a generic encoder network with a domain-specific decoder network to approximate a 2D renderingfunction. However, none of these approaches have beendemonstrated to automatically produce a semantically in-terpretable graphics code and to learn an approximate 3Drendering engine to re-produce images.

In this paper, we present an approach for learning inter-pretable graphics codes. Given a set of images, we usea hybrid encoder-decoder model (trained using unsuper-vised and/or semi-supervised learning) to learn a represen-tation that is disentangled with respect to various trans-formations such as object out-of-plane rotations, lightingvariations, texture and subtle changes in 3D geometry. Toachieve this, we employ a deep directed graphical modelwith many layers of convolution and de-convolution opera-tors that is trained using the Stochastic Gradient VariationalBayes (SGVB) algorithm [12]. Without the loss of gener-ality, we present 3D face analysis as a working examplethroughout this paper. We choose faces due to their richintrinsic shape and appearance variability. Moreover, ac-

arX

iv:1

503.

0316

7v3

[cs

.CV

] 1

7 M

ar 2

015

observed image

Filters = 96kernel size (KS) = 5

150x150

Convolution + Pooling

graphics code

x

Q(zi|x)

Filters = 64KS = 5

Filters = 32KS = 5

7200poselight

shape

....Filters = 32

KS = 7Filters = 64

KS = 7Filters = 96

KS = 7

P (x|z)

Encoder(De-rendering)

Decoder(Renderer)

Unpooling (Nearest Neighbor) + Convolution

{µ200,⌃200}

Figure 1: Model Architecture: Deep Convolutional Inverse Graphics Network (DC-IGN) has an encoder and a decoder.We follow the variational autoencoder[12] architecture with several variations. The encoder consists of several layersof convolutions followed by max-pooling and the decoder has several layers of unpooling (upsampling using nearestneighbors) followed by convolution. (a) During training, data x is passed through the encoder to produce the posteriorapproximation Q(zi|x), where zi consists of scene latent variables such as pose, light, texture or shape. In order to learnparameters in DC-IGN, gradients are backpropagated using stochastic gradient descent using the following variationalobject function: −log(P (x|zi)) + KL(Q(zi|x)||P (zi)) for every zi. We can force DC-IGN to learn a disentangled rep-resentation by showing mini-batches with a set of inactive and active transformations (eg face rotating, light sweeping insome direction etc). (b) During test, data x can be passed through the encoder to get latents zi. Images can be re-renderedto different viewpoints, lighting conditions, shape variations etc by just manipulating the appropriate graphics code group(zi), which is how one would manipulate an off the shelf 3D graphics engine.

cess to large number of training data allows us to carefullytease apart the effects of training size, learning algorithmand modeling biases.

We propose various training procedures to encourage neu-rons in the graphics code layer to have semantic mean-ing and forces each group to distinctly represent a specifictransformation (pose,light,texture,shape etc.). To learn adisentangled representation, we train with random varia-tions on different transformations in each mini-batch ofdata. Another approach we adopt is to train using datawhere each mini-batch has a set of active and inactive trans-formations (but we do not provide target values as in super-vised learning). For example, a nodding face would havethe 3D elevation transformation active but its shape, textureand other affine transformations would be inactive. We ex-ploit this type of a learning procedure to force chosen neu-rons in the graphics code layer to specifically represent ac-tive transformations, thereby automatically creating a dis-entangled representation. Given a static face image, ourmodel can re-generate the input image with different pose,lighting or even texture and shape variations from the baseface. We present qualitative and quantitative results of themodel’s efficacy to learn a 3D rendering engine. Moreover,we also utilize the learnt representation for two importantvisual recognition tasks: (1) an invariant face recognitiontask and (2) using the representation as a summary statistic

for generative modeling.

2 Related Work

As mentioned before, a number of generative models havebeen proposed in the literature to obtain abstract visual rep-resentations. Unlike most RBM based models[9, 25, 17],our approach is trained using back-propagation from thedata reconstruction term and the variational bound.

Relatively recently, Kingma et al. [12] proposed the SGVBalgorithm to learn generative models with continuous la-tent variables. In this work, a feed-forward neural network(encoder) is used to approximate the posterior distributionand a decoder network serves to enable stochastic recon-struction of observations. In order to handle fine-grainedgeometry of faces, we work with relatively large scale im-ages (150x150). Therefore, our approach extends and ap-plies the SGVB algorithm to jointly train and utilize manylayers of convolution and de-convolution operators for theencoder and decoder network respectively. The decodernetwork is a function that transform a compact graphicscode ( 200 dimensions) to a 150x150 image. We proposeusing unpooling (nearest neighbor sampling) followed byconvolution to handle the massive increase in dimensional-ity while keeping the number of parameters be bounded.

Generative stochastic networks [3] which learn the transi-tion operator of a Markov chain whose stationary distribu-tion samples from the data distribution are also relevant toour work. Recently, [7] proposed using CNNs to gener-ate images given object-specific parameters in a supervisedsetting. As their approach requires ground-truth labels forthe graphics code layer, it cannot be directly applied toimage interpretation tasks. Our work is similar to Ran-zato et al.[24], who were amongst the first people to use ageneric encoder-decoder architecture for feature learning.However, in comparison to our proposal their model wastrained layer-wise, the intermediate representations werenot disentangled like a graphics code and their approachdoes not use the variational autoencoder loss to approxi-mate the posterior distribution. Our work is also similar inspirit to [26], but in comparison our model does not assumea lambertian reflectance model and implicitly constructsthe 3D representations. Another piece of related work isthe deconvolution network by Zeiler et al. [32] which isused to produce image representations using convolutionaldecomposition of images under a sparsity constraint.

In comparison to existing approaches, it is important tonote that our encoder network produces interpretable anddisentangled representations, which is necessary to learna meaningful 3D graphics engine. A number of inversegraphics inspired methods have recently been proposed inthe literature [11, 19, 18]. However, most such methodsrely on hand-crafted rendering engines. The exception tothis is work by Hinton et al. [10] (transforming autoen-coders) and Tieleman [27] which uses a domain specificdecoder to reconstruct input images. Our work is similarin spirit to these works but has some key differences: (a)It uses a very generic convolutional architecture in the en-coder and decoder networks to enable efficient learning onlarge datasets and image sizes, (b) it can handle single staticframes as opposed to pair of images required in [10], and(c) it is generative.

3 Model

As shown in figure 1, the basic structure of the Deep Con-volutional Inverse Graphics Network (DC-IGN) consistsof: the encoder network that captures distribution overgraphics codes Z given data x and a decoder networkwhich learns the conditional distribution to produce x givenZ. Z can be a disentangled representation containing sev-eral factored set of latent variables zi ∈ Z such as pose,light and shape. This is important to learn a meaningful ap-proximation of a 3D graphics engine and helps tease apartthe generalization capability of the model with respect todifferent types of transformations on static images.

Let us denote the encoder output of DC-IGN to be ye =encoder(x). The encoder output is used to parametrize thevariational approximation Q(zi|ye), where Q is chosen to

𝜙1

𝛼1

𝛼

𝜙L1

𝜙L

z[4,n]z = z3z2z1

𝜙corresponds to

Output

first sample in batch x1

from encoder to encoder

intrinsic properties (shape, texture, etc)

same as output for x1

z[4,n]z3z2z1

later samples in batch xi z[4,n]z3z2z1

unique for each xi in batch

zero error signalfor clamped outputs

zero error signalfor clamped outputs

error signalfrom decoder

∇zki = zk

i - mean zk

Backpropagation

z[4,n]z3z2z1

z[4,n]z3z2z1

Backpropagation with invariance targeting

z[4,n]z3z2z1

k ∈ batch

k ∈ batch

Caption: Training on a minibatch in which only 𝜙, the azimuth angle of the face, changes.During the forward step, the output from each component z_k != z_1 of the encoder is forced to be the same for each sample in the batch. This reflects the fact that the generating variables of the image which correspond to the desired values of these latents are unchanged throughout the batch. By holding these outputs constant throughout the batch, z_1 is forced to explain all the variance within the batch, i.e. the full range of changes to the image caused by changing 𝜙.

During the backward step, backpropagation of gradients happens only through the latent z_1, with gradients for z_k != z_1 set to zero. This corresponds with the clamped output from those latents throughout the batch.

Caption: In order to directly enforce invariance of the latents corresponding to properties of the image which do not change within a given batch, we calculate gradients for the z_k != z_1 which move them towards the mean of each invariant latent over the batch. This is equivalent to regularizing the latents z_{[2,n]} by the L2 norm of (zk - mean zk).

Figure 3: Structure of the representation vector. φ isthe azimuth of the face, α is the elevation of the face withrespect to the camera, and φL is the azimuth of the lightsource.

be a multivariate normal distribution. There are two rea-sons for using this parametrization: (1) Gradients of sam-ples with respect to parameters θ of Q can be easily ob-tained using the reparametrization trick proposed in [12],and (2) Various statistical shape models trained on 3D scan-ner data such as faces have the same multivariate normallatent distribution (see section 4.1). Given that model pa-rameters We connect ye and zi, the distribution parametersθ = (µzi ,Σzi) and latents Z can then be expressed as:

µzi = We ∗ ye (1)Σzi = diag(exp(We ∗ ye)) (2)∀i, zi ∼ N (µzi ,Σzi) (3)

We present several training procedures for DC-IGN to learnrepresentations with different characteristics:

3.1 Random Transformations

For our simplest training procedure, we construct mini-batches of randomly selected training samples and train theDC-IGN network in figure 1 according to the SGVB proce-dure. This training leads to networks with extremely strongreconstruction performance. We explore several uses of therepresentations generated by these networks in section 4.

3.2 Specific Transformations

One of the main goals of this work is for the DC-IGNsto learn a representation of the data which consists of dis-entangled and semantically interpretable latent variables.Following Bengio et al.[2] we would like only a small sub-set of the latent variables to change for sequences of inputscorresponding to real-world events.

One natural choice of target representation for informationabout scenes is that already designed for use in graphicsengines. If we can deconstruct a face image by splittingit into variables for pose, light, and shape, we can triviallyrepresent the same kinds of transformations that these vari-ables are used for in graphics applications. Figure 3 depictsthe representation which we will attempt to learn.

To achieve this goal, we perform a training procedurewhich directly targets this definition of disentanglement.We organize our data into minibatches corresponding to

Output

first sample in batch x1

from encoder to encoder

same as output for x1

z[4,n]z3z2z1

later samples in batch xi z[4,n]z3z2z1

unique for each xi in batch

zero error signalfor clamped outputs

zero error signalfor clamped outputs

Backpropagation

z[4,n]z3z2z1

z[4,n]z3z2z1

Figure 2: Training on a minibatch in which only φ, the azimuth angle of the face, changes. During the forward step,the output from each component zk 6= z1 of the encoder is forced to be the same for each sample in the batch. This reflectsthe fact that the generating variables of the image which correspond to the desired values of these latents are unchangedthroughout the batch. By holding these outputs constant throughout the batch, z1 is forced to explain all the variance withinthe batch, i.e. the full range of changes to the image caused by changing φ. During the backward step, backpropagation ofgradients happens only through the latent z1, with gradients for zk 6= z1 set to zero. This corresponds with the clampedoutput from those latents throughout the batch.

changes in only a single scene variable (azimuth angle, el-evation angle, and azimuth angle of the light source); theseare transformations which might occur in the real world.We will term these the extrinsic variables. To train this rep-resentation, we adopt a procedure that is similar to SGVB,with the key differences being:

1. Select at random a latent variable ztrain which wewish to correspond to one of {azimuth angle, eleva-tion angle, azimuth of light source}

2. Select at random a minibatch in which that extrinsicscene variable changes

3. Show the network the first example in the minibatchand capture its latent representation z1

4. For all subsequent examples in the minibatch, hold allzi 6= ztrain constant and equal to z1i (these latents are”clamped”)

5. Backpropagate error as usual in the decoder, but passzero gradients for all zi 6= ztrain. The gradient at zishould be passed through unchanged.

6. Continue backpropagation through the encoder usingthe modified gradient.

Along with these we generate minibatches in which thethree extrinsic scene variables are held fixed but all otherproperties of the image (variables controlling the identityand appearance of the face) change randomly. These intrin-sic properties of the model are represented by the remain-der of the latent variables. Training of these remaining la-tent variables is similar to the procedure above, but instead

of a single element being trained, ztrain = z[4,n]. That is,we clamp the latent variables corresponding to the extrinsicproperties of pose and lighting and only allow the intrinsiclatents z[4,n] to change within a batch; they are similarly theonly latents given nonzero gradients. These minibatchesvarying intrinsic properties are interspersed stochasticallywith those varying the extrinsic properties. The probabilityof selecting an intrinsic batch is proportional to a hyperpa-rameter we term the ”shape bias”, for it represents a ”bias”in the training set towards examples which vary the under-lying shape of the face.

This training method leads to networks whose latent vari-ables have a strong equivariance with the generating pa-rameters, as shown in figure 9. This allows the value ofthe true generating parameter to be trivially extracted fromthe latent representation. However, this training methoddoes not directly enforce the invariance of the clamped la-tents to transformations to which one might like them tobe insensitive. Instead, it gets its invariances from the ten-dency of neural networks’ representations to approach lo-cally optimal compression. Since information about, forexample, the elevation of the face is explicitly required tobe stored in z2, any information about the elevation storedin any other zi would be redundant and wasteful of that zi’sinformation-carrying capacity.

3.2.1 Invariance targeting

In order to further improve the invariance of the proceduredefined in section 3.2, we propose a procedure for explic-itly training invariance of the latent variables to orthogonaltransformations. As depicted in figure 4, we replace thezero gradients of the zi 6= ztrain from section 3.2 with an

𝜙1

𝛼1

𝛼

𝜙L1

𝜙L

z[4,n]z = z3z2z1

𝜙corresponds to

Output

first sample in batch x1

from encoder to encoder

intrinsic properties (shape, texture, etc)

same as output for x1

z[4,n]z3z2z1

later samples in batch xi z[4,n]z3z2z1

unique for each xi in batch

zero error signalfor clamped outputs

zero error signalfor clamped outputs

error signalfrom decoder

∇zk = zk - mean zk

Backpropagation

z[4,n]z3z2z1

z[4,n]z3z2z1

Backpropagation with invariance targeting

z[4,n]z3z2z1

k ∈ batch

k ∈ batch

Caption: Training on a minibatch in which only 𝜙, the azimuth angle of the face, changes.During the forward step, the output from each component z_k != z_1 of the encoder is forced to be the same for each sample in the batch. This reflects the fact that the generating variables of the image which correspond to the desired values of these latents are unchanged throughout the batch. By holding these outputs constant throughout the batch, z_1 is forced to explain all the variance within the batch, i.e. the full range of changes to the image caused by changing 𝜙.

During the backward step, backpropagation of gradients happens only through the latent z_1, with gradients for z_k != z_1 set to zero. This corresponds with the clamped output from those latents throughout the batch.

Caption: In order to directly enforce invariance of the latents corresponding to properties of the image which do not change within a given batch, we calculate gradients for the z_k != z_1 which move them towards the mean of each invariant latent over the batch. This is equivalent to regularizing the latents z_{[2,n]} by the L2 norm of (zk - mean zk).

i i

Figure 4: Backpropagation with invariance targeting.In order to directly enforce invariance of the latents cor-responding to properties of the image which do not changewithin a given batch, we calculate gradients for each zk

predicted latent in the batch which moves it towards themean of each invariant latent over the batch. This is equiv-alent to regularizing the latents z[2,n] by the L2 norm ofzk − mean

k∈batchzk.

error gradient pointing in the direction of greatest differ-ence from the mean of the latent variables we desire to beinvariant in this minibatch. During training, this pushes thevalues which we desire to be constant throughout the batchtowards one another, decreasing the variation that these la-tents experience due to orthogonal variables. This regular-izing force needs to be scaled to be much smaller than thetrue training signal; empirically, a factor of 1/100 workswell.

4 Experiments

We experimented with two common visual perceptiontasks: (1) invariant recognition task to predict whethertwo faces are same or equal, and (2) using the learnt DC-IGN representation as a distance metric in likelihood-freebayesian inference. These tasks stress the importance ofa disentangled representation – given a factored set ofzi’s, invariance follows automatically by ignoring subsetsof zi’s. For face recognition, if we denote zidentity ∈{zshape, ztexture} to be latents corresponding to facialidentity and {zpose, zlight} to be rest of the variables, thenwe can extract an invariant represent of any given face byusing zidentity and ignoring all other zi’s. Similarly, forusing Z as a distance metric, we may wish to preserveall transformations to be equivariant by keeping all Z’saround.

Vetter et al.[5] first proposed using 3D laser scanned facedata to learn a deformable 3D face model (3DMM). Duringtraining of 3DMM model, 3D faces are manually aligned tobe in a canonical pose and a statistical shape based modelis obtained with tunable vector based representation. In allour experiments, we train using the synthetic data obtainedfrom the 3DMM model and test it out on real face datawith ground truth 3D annotations (geometry and texture)obtained from [23]. This model allows us to generate syn-thetic images of faces by systematically varying pose, light,

shape and texture. Moreover, 3DMM also gives us a goodground truth metric to test the generalization capabilities ofour learnt 3D rendering function. We trained our model onabout 10000 batches of faces, where each batch consists ofabout 30 faces on average with random variations on faceidentity variables (shape/texture), pose and lighting. Weused rmsprop[27] learning algorithm during training andset the meta learning rate to be equal to 0.0005, the mo-mentum decay to be 0.1 and weight decay to be 0.01.

4.1 3D Statistical Shape Modeling

In order to test the equivariance power of DC-IGN’s learntrepresentation, we use Z as a summary statistic in a gener-ative model based on the 3DMM face model. Given a pre-trained DC-IGN, we are interested in the following ques-tion – how well can we recover latents of a compositional3D shape model by using latent representations Z? Afterconvergence, this inference procedure produces 3D verticesand reflectance values for each vertex given a single staticimage. Intuitively, we are essentially using DC-IGN repre-sentations along with a rich domain-specific face decodermodel for coarse-to-fine inference. Alternatively, we canalso use the learnt representations from our model to learndata-driven proposals similar to [11].

Bayesian inference in our setting is difficult as the likeli-hood is unavailable in closed form due to the presence ofthe rendering pipeline and subsequent DC-IGN computa-tions. However, there has been considerable progress in thestatistics community to perform likelihood free Bayesianinference, which is often termed as Approximate BayesianComputation (ABC). Therefore, ABC provides a naturalformulation to describe inverse graphics in likelihood-freesettings. We define a simple distance function on the sum-mary statistics obtained from the DC-IGN-encoder net-works and apply a variant of the probabilistic approximateMCMC algorithm [30] to do inference.

We used a pre-trained DC-IGN encoder-decoder network(from any one of the training procedures) on the syntheticface dataset, followed by stripping the decoder part of thenetwork. The DC-IGN encoder can now be used as a sum-mary statistic function ν : x → Z, which calculates thelatent representation Z for any given image x. We replacethe DC-IGN decoder network by a richer generative modelbased on the 3DMM face model. This is similar in spiritto Nair et al.[22], where a domain-specific decoder is usedalong with a generic encoder to do analysis-by-synthesisbased generative modeling.

The 3DMM face model consists of a set of two facerelated latent variables (Sshape, Stexture,), each with200 dimensions and sampled from {N (0, 1)}200i=1.Additionally, there are scene variables including:(Slight, Selevation) ∼ Uniform(−90, 90) and{Sjcamera}j∈{x,y,z} ∼ Uniform(0, 1). Let us de-

Stochastic Scene Generator

GraphicsSimulator

DC-IGNENCODER

dataID

S ⇠ P (S)

IR ⇠ g(S)

Distance Metric(Likelihood-free)

ABC distance function (eg. L1)⇢(⌫(ID), ⌫(IR))

⌫(ID) ⌫(IR)

DECODER

Figure 5: Deep DC-IGN encoder with a domain-specificdecoder: In order to test the equivariance of DC-IGNslearnt representation, we can use DC-IGN encoder out-put as summary statistic for likelihood-free bayesian in-ference. We use the 3DMM Basel face model as prior on3D faces. The prior P (S) is a multivariate gaussian distri-bution which generates a set of latent variables Si denot-ing pose, lighting, shape and texture. The scene S can bepassed to a graphics simulator which produces the hypothe-sis image IR. Given IR and observation image ID, the taskis to infer P (S|ID). Since the likelihood function is notavailable in closed-form, we can simply resort to approxi-mate Bayesian computation (ABC) based MCMC[30]. TheDC-IGN encoder produces summary statistics for both theobservation ν(ID) and current hypothesis ν(IR), where thecurrent hypothesis is accepted or rejected.

note IB(.) as the indicator function of the set B andAε,ID = {IR|ρ(ν(ID), ν(IR)) ≤ ε}, where ρ can be asuitable distance metric such as L1. As shown in figure 5,g(S) denotes the graphics simulator which generatesrendering primitives (images, mesh, etc.) given a sceneS. We can now formulate the image interpretation task asapproximately sampling the posterior distribution of sceneS given observations (ID):

π(S|ID) ≈ πε(S|ID), (4)where πε(S|ID) (5)

=

∫πε(S, IR|ID)dIR (6)

∝π(S)δg(S)(IR)IAε,ID (IR)∫Aε,ID×S

π(S)δg(S)(IR)dS(7)

During inference, we make use of elliptical proposalson large set of coupled continuous random variables S.

(a)0 50 100 150 200 250

Epoch0.005

0.010

0.015

0.020

0.025

0.030

0.035

0.040

Pixe

l MSE

(b)0 1 2 3 4 5 6

0.000

0.002

0.004

0.006

0.008

0.010

0.012

0.014

0.016

0.018

Pixe

l Rec

onst

ruct

ion(

MSE

)

Imagenet(Pool5)Imagenet(fc6)

Imagenet(fc7)DCIG(Code=2)

DCIG(Code=20)DCIG(Code=120)

Figure 6: 3D Face generative model results: (a) As in-ference for scheme highlighted in figure 5 proceeds, thepixel reconstruction error (MSE) between the observed im-age and the hypothesis IR decreases. We show multipleindependent chains to illustrate typical inference trajecto-ries during inference. Note how the pixel error increasesconsiderably during some of the intermediate runs but thenconverges to a lower value with more iterations, suggestingthat it escapes some of the local minimas as it is following asmoother manifold in the DC-IGN’s representation space.(b) We quantitatively compared the test pixel reconstruc-tion error for various baseline models – pre-trained Ima-genet with different layers as the chosen summary statisticand DC-IGN with varying number of neurons in the graph-ics code layer.

Neal [4] and Murray et al. [21] studied generative mod-els with Gaussian priors having zero mean and arbitrarycovariance structure. Let us assume that X ∼ N (0,Σ).Given a state S of the scene, a new state S′ can beefficiently proposed as follows: S′ =

√1− α2S +

αθ, where θ ∼ N (0,Σ) and α ∼ Uniform(−1, 1).

Based on figure 6(a), as inference proceeds the pixeldifference between the current hypothesis of the modeland observation decreases. We also quantitatively com-pare DC-IGN’s latent representation with a pre-traineddeep convolutional network (trained on Imagenet [28]version:imagenet-caffe-ref ). We used several intermediatelayers of the pre-trained network as the summary statisticsand show quantitative performance on test data in compari-son to various DC-IGN models. The DC-IGN models weretrained with different number of dimensions for Z. From

Humans CNN-Imagenet

CNN-Imagenet (supervised fine tuning) DC-IGN

78% 67% 72% 68%

Same or

different?

Figure 7: Invariance Tests: In order to test the invarianceof Z, we tested the network on predicting whether a givenpair or images has the same or different face. Due to pose-light variability, this task is hard even for human subjects(limited viewing time of about 200ms). We quantitativelycompared the prediction task with a pre-trained ImagenetCNN and also a fine-tuned version of the same CNN withthe face dataset (see section 4.2 for details). Both of thesebaselines were trained or fine-tuned with supervised learn-ing. On the other hand, DC-IGN achieves comparable ac-curacy with just unsupervised learning.

figure 6(b), it is clear that DC-IGN code with a relativelylarger dimensions is necessary to capture the rich set oftransformations on real faces. This emphasizes the fact thatto obtain a disentangled representation, data reconstructionthrough a small bottleneck graphics code layer might bemore beneficial than just tuning for recognition as is thecase with supervised CNNs.

4.2 Recognition

In order to test the invariance properties of DC-IGN, weperformed a task to predict whether two given faces aresame or different. This experiment was first proposed byYildrim et al[31] and we closely follow this procedure forall neural network baselines.

The test dataset from Yildrim et al. consists of 96 pairsof faces with varying degree of pose, lighting, shape andtexture. In order to ensure that the task is interesting andhard, they also obtained accuracy of the prediction fromhuman subjects using Amazon Turk. Obtaining an invari-ant representation is hard as the two faces can have dras-tically different transformations. We also obtained a fine-tuned CNN on the SURF-W dataset from Yildrim et al.,which was fine-tuned with 400 famous people under dif-ferent light-pose conditions. As shown in figure 7, humansperform at 78 percent which is higher than all deep learn-ing based system. It is perhaps surprising that DC-IGN per-forms competitively with other baseline models even whentrained completely unsupervised using the random trans-form training procedure in section 3.1.

(a)

Original Reconstuction Pose (Elevation) varied

(b)

Original Reconstuction Pose (Azimuth) varied

Figure 8: Manipulating pose variables: Qualitative re-sults showing the generalization capability of the learntDC-IGN decoder to render original static image with dif-ferent pose directions.(a) The latent neuron zelevation ischanged to random values but all other latents are clamped.(b) The latent neuron zazimuth is changed to random valuesbut all other latents are clamped.

In order to calculate test accuracy of the same vs differ-ent task for all baseline models, we follow the same-vs-different test procedure first proposed in [31]. We firstscaled each test image representation independently to becentered at 0 and to have a standard deviation of 1. Then,for each pair of images, we calculated the Pearson corre-lation coefficient between the two network representations.We searched for a threshold correlation ∈ [1, 1] such that

the any given model’s performance will be highest with re-spect to ground truth. The search was such that the pairs ofcorrelation values lower than the threshold were assigneddifferent, and the pairs of equal or higher correlation val-ues than the threshold were assigned same by the model.We report results based upon the threshold that gives thehighest performance for each baseline model separately.

4.3 Generalization of the rendering function

The decoder network learns an approximate rendering en-gine as shown in figure(8,10). Given a static test image,the encoder network produces the latents Z depicting scenevariables such as light, pose, shape etc. Similar to an off-the-shelf rendering engine, we can selectively clamp zi’sand vary the rest. For example, as shown in figure 10, giventhe original test image, we can vary zlight by clamping allother nodes and conditionally generating the image throughthe decoder network. It is perhaps surprising that the learntdecoder network is able to approximately function like a3D rendering engine. We performed similar analysis onother transformations as shown in figure 8.

We also quantitatively measured the decoder’s generaliza-tion capability to render across arbitrary viewpoints andlighting conditions. We generated 100 test batches fromthe 3DMM model where each batch contained a distinctface with varying pose (azimuth in range -1.5 to 1.5 with0.05 step size). Within each batch, we compute the la-tent representation Z for every image and extracted the la-tents corresponding to face identity zidentity ∈ z[4, n] asshown in figure 3. Within each batch, there is a face im-age with pose azimuth angle equal to zero (frontal image).For same face identity, we ideally expect zidentity of allimages within a batch to be equal. We calculated the co-sine distance between the latent representation zidentity ofevery image with the frontal image and averaged the scoreacross all batches. As shown in figure 9(d), images closeto the frontal face have almost identical latent representa-tions but the similarity decays as azimuth goes away fromzero. In other words, the invariance characteristics grad-ually degrades with viewpoint extrapolation. We performsimilar analysis by generating separate datasets by varyingelevation and lighting conditions.

We tested our model with three different DC-IGN net-work configurations – (1) DC-IGN with shape bias (seesection3.2), (2) DC-IGN without shape bias and (3) DC-IGN without shape bias but with targeted invariance (seesection 3.2.1). All three networks do reasonably well inpredicting azimuth, light and elevation (figure 9(a,b,c). In-terestingly, as shown in figure 9(a), the encoder networkseems to have learnt a switch node to separately processleft and right profile faces. However, as shown in fig-ure 9(d,e,f), the network with high shape bias seems tohave the strongest invariance properties. This intuitively

Original Reconstuction Light direction varied

Figure 10: Manipulating light variables: Qualitative re-sults showing the generalization capability of the learntDC-IGN decoder to render original static image with dif-ferent light directions. The latent neuron zlight is changedto random values but all other latents are clamped.

makes sense as this network saw relatively larger variationsin facial identities. It is also interesting to note (figure 9(a)and figure8(b)) that in comparison to other transformations,generalization across the azimuth suffers most from radicalextrapolation. This is likely because the lateral out-of-planerotation transformation is probably one of the hardest trans-formations to learn for faces. However, this may improveby training the model with additional training data.

5 Discussion

We have shown that it is possible to train a deep convolu-tional inverse graphics network with interpretable graphicscode layer representation just from raw images. By uti-lizing a deep convolution and de-convolution architecturewithin a variational autoencoder formulation, our modelcan be jointly trained using back-propagation using thestochastic variational objective function [12]. We proposedvarious training procedures to force the network to learndisentangled representations. Using 3D face analysis asa working example, we have demonstrated the invariantand equivariant characteristics of the learnt representation.Moreover, as highlighted in section 4.1, the DC-IGN de-convolutional network based decoder can be replaced by adomain-specific decoder for fine-grained model-based in-ference.

To scale our approach to handle more complex scenes, itwill likely be important to experiment with deeper archi-tectures in order to handle large number of object cate-

(a)2.0 1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0

Ground Truth30

20

10

0

10

20

30

Infe

rred

Pose (Azimuth)

(b)0.4 0.2 0.0 0.2 0.4

Ground Truth20

10

0

10

20

30

Infe

rred

Pose (Elevation)

(c)100 50 0 50 100

Ground Truth15

10

5

0

5

10

15

Infe

rred

Light

(d)1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0

Value0.0

0.2

0.4

0.6

0.8

1.0

1.2

Cosi

ne D

ista

nce

Pose (Azimuth)

(e)0.4 0.3 0.2 0.1 0.0 0.1 0.2 0.3 0.4

Value0.5

0.6

0.7

0.8

0.9

1.0

1.1

Cosi

ne D

ista

nce

Pose (Elevation)

(f)80 60 40 20 0 20 40 60 80

Value0.86

0.88

0.90

0.92

0.94

0.96

0.98

1.00

1.02

Cosi

ne D

ista

nce

Light

Figure 9: Generalization of decoder to render images in novel viewpoints and lighting conditions: We generatedseveral datasets by varying light, azimuth and elevation, and tested the invariance properties of DC-IGN’s representationZ. We show quantitative performance on three network configurations as described in section 4.3. (a,b,c) All DC-IGNencoder networks reasonably predicts transformations from static test images. Interestingly, as seen in (a), the encodernetwork seems to have learnt a switch node to separately process azimuth on left and right profile side of the face. (d,e,f)We generated 100 faces and individually varied light, azimuth and elevation. Within each face set, we calculate theaverage cosine distance between the frontal face and every other transformed image. As demonstrated above, the networkwith strong shape bias has the strongest invariance power, which is intuitive as this network was trained using relativelyhigher facial variations. On the other hand, the target invariance training (without any shape bias) scheme gives strongerinvariance in comparison to network trained without it.

gories within a single network architecture. It is also veryappealing to design a spatio-temporal based convolutionalarchitecture to utilize motion in order to handle compli-cated object transformations. The current formulation ofSGVB is restricted to continuous latent variables. How-ever, real-world visual scenes contain unknown number ofobjects that move in and out of frame. Therefore, it mightbe necessary to extend this formulation to handle discretedistributions. There has been recent work[15] on trying toapproximate discrete distribution by using particle basedvariational approximations. An interesting future directionis to construct a mixed continuous-discrete variational ap-proximationQmixed and to iteratively inter-leave optimiza-tion of the discrete parameters with SGVB.

We hope that our work motivates further research into auto-matically learning disentangled representations using vari-ants of different encoder and decoder networks. The mostgeneral form of a decoder network can be thought of as astochastic graphics program that conditions images givenits latent graphics code. In the future, we aim to explorea hierarchy of coarse-to-fine encoder-decoder programs,where each encoder reasonably predicts a small set of la-tent variables and passes a noisy graphics code for the sub-sequent encoder-decoder program for refinement. This ex-plicit hierarchy to sequentially handle different transforma-tions may allow us to learn the network using relatively

fewer training examples.

Acknowledgements

We thank Thomas Vetter for giving us access to the Baselface model. T. Kulkarni was graciously supported by theLeventhal Fellowship. This research was supported byONR award N000141310333, ARO MURI W911NF-13-1-2012 and CBMM. We would like to thank Ilker Yildrim,Max Kleiman-Weiner, Karthik Rajagopal and GeoffreyHinton for helpful feedback and discussions.

References[1] Y. Bengio. Learning deep architectures for ai. Foundations

and trends R© in Machine Learning, 2(1):1–127, 2009.

[2] Y. Bengio, A. Courville, and P. Vincent. Representa-tion learning: A review and new perspectives. PatternAnalysis and Machine Intelligence, IEEE Transactions on,35(8):1798–1828, 2013.

[3] Y. Bengio, E. Thibodeau-Laufer, G. Alain, and J. Yosinski.Deep generative stochastic networks trainable by backprop.arXiv preprint arXiv:1306.1091, 2013.

[4] J. Bernardo, J. Berger, A. Dawid, A. Smith, et al. Regressionand classification using gaussian process priors. In BayesianStatistics 6: Proceedings of the sixth Valencia internationalmeeting, volume 6, page 475, 1998.

[5] V. Blanz and T. Vetter. A morphable model for the synthesisof 3d faces. In Proceedings of the 26th annual conference on

Computer graphics and interactive techniques, pages 187–194. ACM Press/Addison-Wesley Publishing Co., 1999.

[6] T. Cohen and M. Welling. Learning the irreducible rep-resentations of commutative lie groups. arXiv preprintarXiv:1402.4437, 2014.

[7] A. Dosovitskiy, J. Springenberg, and T. Brox. Learn-ing to generate chairs with convolutional neural networks.arXiv:1411.5928, 2015.

[8] I. Goodfellow, H. Lee, Q. V. Le, A. Saxe, and A. Y. Ng.Measuring invariances in deep networks. In Advancesin neural information processing systems, pages 646–654,2009.

[9] G. Hinton, S. Osindero, and Y.-W. Teh. A fast learning algo-rithm for deep belief nets. Neural computation, 18(7):1527–1554, 2006.

[10] G. E. Hinton, A. Krizhevsky, and S. D. Wang. Transformingauto-encoders. In Artificial Neural Networks and MachineLearning–ICANN 2011, pages 44–51. Springer, 2011.

[11] V. Jampani, S. Nowozin, M. Loper, and P. V. Gehler. Theinformed sampler: A discriminative approach to bayesianinference in generative computer vision models. arXivpreprint arXiv:1402.0859, 2014.

[12] D. P. Kingma and M. Welling. Auto-encoding variationalbayes. arXiv preprint arXiv:1312.6114, 2013.

[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenetclassification with deep convolutional neural networks. InNIPS, pages 1106–1114, 2012.

[14] T. D. Kulkarni, V. K. Mansinghka, P. Kohli, and J. B. Tenen-baum. Inverse graphics with probabilistic cad models. arXivpreprint arXiv:1407.1339, 2014.

[15] T. D. Kulkarni, A. Saeedi, and S. Gershman. Variationalparticle approximations. arXiv preprint arXiv:1402.5715,2014.

[16] Y. LeCun and Y. Bengio. Convolutional networks for im-ages, speech, and time series. The handbook of brain theoryand neural networks, 3361, 1995.

[17] H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng. Convolu-tional deep belief networks for scalable unsupervised learn-ing of hierarchical representations. In Proceedings of the26th Annual International Conference on Machine Learn-ing, pages 609–616. ACM, 2009.

[18] M. M. Loper and M. J. Black. Opendr: An approximatedifferentiable renderer. In Computer Vision–ECCV 2014,pages 154–169. Springer, 2014.

[19] V. Mansinghka, T. D. Kulkarni, Y. N. Perov, and J. Tenen-baum. Approximate bayesian image interpretation usinggenerative probabilistic graphics programs. In Advances inNeural Information Processing Systems, pages 1520–1528,2013.

[20] J. Masci, U. Meier, D. Ciresan, and J. Schmidhuber. Stackedconvolutional auto-encoders for hierarchical feature extrac-tion. In Artificial Neural Networks and Machine Learning–ICANN 2011, pages 52–59. Springer, 2011.

[21] I. Murray, R. P. Adams, and D. J. MacKay. Elliptical slicesampling. arXiv preprint arXiv:1001.0175, 2009.

[22] V. Nair, J. Susskind, and G. E. Hinton. Analysis-by-synthesis by learning to invert generative black boxes. InArtificial Neural Networks-ICANN 2008, pages 971–981.Springer, 2008.

[23] P. Paysan, R. Knothe, B. Amberg, S. Romdhani, and T. Vet-ter. A 3d face model for pose and illumination invariantface recognition. Proceedings of the 6th IEEE InternationalConference on Advanced Video and Signal based Surveil-lance (AVSS) for Security, Safety and Monitoring in SmartEnvironments, 2009.

[24] M. Ranzato, F. J. Huang, Y.-L. Boureau, and Y. LeCun. Un-supervised learning of invariant feature hierarchies with ap-plications to object recognition. In Computer Vision andPattern Recognition, 2007. CVPR’07. IEEE Conference on,pages 1–8. IEEE, 2007.

[25] R. Salakhutdinov and G. E. Hinton. Deep boltzmann ma-chines. In International Conference on Artificial Intelli-gence and Statistics, pages 448–455, 2009.

[26] Y. Tang, R. Salakhutdinov, and G. Hinton. Deep lambertiannetworks. arXiv preprint arXiv:1206.6445, 2012.

[27] T. Tieleman. Optimizing Neural Networks that Generate Im-ages. PhD thesis, University of Toronto, 2014.

[28] A. Vedaldi and K. Lenc. Matconvnet – convolutional neuralnetworks for matlab. CoRR, abs/1412.4564, 2014.

[29] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A.Manzagol. Stacked denoising autoencoders: Learning use-ful representations in a deep network with a local denois-ing criterion. The Journal of Machine Learning Research,11:3371–3408, 2010.

[30] R. D. Wilkinson. Approximate bayesian computation (abc)gives exact results under the assumption of model error.Statistical applications in genetics and molecular biology,12(2):129–141, 2013.

[31] I. Yildrim, T. Kulkarni, W. Freiwald, and J. Tenenbaum. Ex-plaining monkey face patch system as deep inverse graphics.In Computational and Systems Neuroscience, 2015.

[32] M. D. Zeiler, D. Krishnan, G. W. Taylor, and R. Fergus.Deconvolutional networks. In Computer Vision and Pat-tern Recognition (CVPR), 2010 IEEE Conference on, pages2528–2535. IEEE, 2010.