Ladder Network

Semi-supervised learningwith Ladder NetworksNIPS, 2016/1/20Preferred Networks, @mattya1089

2

(@mattya1089)Preferred NetworksChainer-goghChainer-DCGAN

Deep Generative ModelsSemi-supervised learning with deep generative models (Kingma et al., 2014) Improving Semi-Supervised Learning with Auxiliary Deep Generative Models(Maaloe et al., 2015) Virtual Adversarial TrainingDistributional smoothing with virtual adversarial training (Miyato et al., 2015)Ladder Networks ()Semi-supervised learning with Ladder network (Rasmus et al., 2015) Deconstructing the ladder network architecture (Mohammad et al., 2016)

3

(semi-supervised learning)4

(semi-supervised learning)5

()

(semi-supervised learning)6

(semi-supervised learning)7

50%()

(semi-supervised learning)8

50%()

(semi-supervised learning)9

()

x: y: 16.4%60000 : 0.82 -> 0.82%x33

Ladder NetworkLateral connection

xxx~ytReconstruction lossSupervised lossOKxdenoisingCombination

denoising

Ladder Network

xxx~ytReconstruction lossSupervised lossReconstructionSemi-supervisedh1h2h1~h2~100 label: 16.4 -> 1.86%60000 : 0.82 -> 0.73%y~

Ladder Networkxxx~ytReconstruction lossSupervised lossVATh1h2h1~h2~100 label: 1.86 -> 1.69%60000 : 0.73 -> 0.61%NoiseNoiseNoiseNoisey~

Ladder NetworkReconstruction lossLadder Networkxxx~ytReconstruction lossSupervised lossh1h2h1~h2~100 label: 1.69 -> 1.09%60000 : 0.61 -> 0.61%NoiseNoiseNoiseNoisexxyh1h2Encodery~DecoderPriorLossweightFully-supervised

Ladder Network

Feed-forward NNDecoderReconstructionLateral connectionReconstruction loss100 labels25.8% 23.0%16.4%1.86%1.69%1.09%60000 labels1.18% 0.82%0.82%0.73%0.61%0.61%NN

Generative models, VAT, Ladderdeep unsupervised32LadderLateral connectionReconstructionLateral connectionGenerative modelsMNIST The proposed model is simple and easy to implement with many existing feedforward architectures(`)

Semi-supervised learning with Ladder network (Rasmus et al., 2015) Deconstructing the ladder network architecture (Mohammad et al., 2016) Semi-supervised learning with deep generative models (Kingma et al., 2014) Improving Semi-Supervised Learning with Auxiliary Deep Generative Models(Maaloe et al., 2015) Distributional smoothing with virtual adversarial training (Miyato et al., 2015)

https://github.com/mattya/chainer-semi-supervisedVATLadder Network (1.4%1.1%)ChainerLadder34

Ladder NetworkEncoderBatch Normalization

Ladder NetworkLateral connectionDecoderEncoderDecoderDecoderBNscale, shift

Ladder NetworkEncoderz~DecoderuCNNDenoisingz~iui

Ladder NetworkReconstruction LosscleanzreconstructzLateralzBatch normalizationminibatchEncoderCleanNoisyreconstructionzcleanNormalizationcleanz

Ladder NetworkAdamLearning rateepoch1000.0021500[0,1]DecodertopsoftmaxLinearBiasClean encoderBackpropWeight decay