NeuralRestoration

NeuralReconstructionApplyingA.I.toArtRestoration

PresentationbyEdChinDecember13,2016

ARTWORLD

!  Goal:Restoreunseendamagedimages/photographswithacontent-awareneuralnet

CanArtificialIntelligencerivalhumansinSemanticImageReconstruction?

OriginalImage DamagedImage RestoredImage

ArtRestorationwithDeepLearning

!  Howwouldyoufillinthemissinginformation?

!  Canonetrainacomputertohavethenecessarycognitiontofillinthemissingstructural,textualandcontentinformation?

WeighingVariousArchitectures

Non-ParametricImageMatching

VariationalSampling

ConvolutionalNeuralNet

AdversarialNetworks

ContextualAwareness ✔ ✔ ✔ ✔PerceptualAwareness ✗ ✔ ✔ ✔HighResolution ✔ ✗ ✗ ✔LargeMissingPatches ✗ ✗ ✔ ✔EasytoTrain ✔ ✔ ✗ ✗

DataMiningandImageAugmentation

Scrapeandaugment200kartsy.netimages

1 Retrieveand

pre-processimages

2

Trainautoencoderasimagegenerator&

extractlatentfeatures

3 Simultaneouslytrainan“adversarial”neuralnettodiscriminate

realvs.“fake”images

4 ImageRestoration

Application

5

RetrievalandPre-ProcessingPipeline

!  Imagesareresizedandnormalizedforzeromean/unitvariance

!  1,000imagesaresetasideeachforvalidationandtestset

TargetImage TrainingImage Arbitrarilysizedmaskisplacedrandomlyto“corrupt”thetrainingimage

ImageGeneratorArchitecture

4 hidden convolution layers

4 hidden deconvolution layers

1024 8 x 10 feature maps

MSE only

Replace final dense layers. Connect output units to input units as autoencoder

Not good!

DeepConvolutionalGenerativeAdversarialNetworks(DCGANs)

G(MSE) + D(Fake_BCE)

Real Image

Fake Image

D(Real_BCE)

D(Fake_BCE)

Generator, G() Discriminator, D()

No heuristic cost functions

needed!!!

KeyModelInsights

!  GenerativeAdversarialNetworks,sinceitsintroductionbyIanGoodfellowin2014,isknowntobeverytrickytotrain

!  Anumberofstabilizingmeasureswereintroduced,basedonthegroundbreakingpaper“UnsupervisedRepresentationLearningwithDeepConvolutionalGenerativeAdversarialNetworks”byAlecRadford,LukeMetzandSoumithChintala

!  Themainbenefitforusingadversarialnetworksisthemodel’sabilitytolearnitsowncostfunctioninanunsupervisedfashion

SampleOutputsfromTestSet

DamagedImage RestoredImage







ArtWorld–ImageRestorationApp

Step 1 Step 2 Step 3

Drag or Upload flawed image

to website

Place user defined mask on damaged area

Hit repair button. Receive repaired image instantly

Whichonewouldyouprefer?

DamagedImageA.I.

RestoredImageHuman

RestoredImage

OtherCommercialApplicationsoftheDCGANsArchitecture

!  Learnhigh-orderfunctionslikereasoning,planningandprediction

!  Dimensionreduction/LatentFeatureExtraction(morepowerfulthanPCA)

!  Generatesuperresolutionorup-sampledimages

!  Forwardvideoprediction

ContactInformation

EdChin

! Email:[email protected]

! Linkedin:!  https://www.linkedin.com/in/edwin-chin-62392b1

! Repo:github.com/echin6/my_recent_projects

Documents

NeuralRestoration