Sung Whan Yoon (presenter), Jun Seo and Jaekyun Moon

Jun 11 @ ICML 2019

TapNet: Neural Network Augmented with Task-Adaptive Projection for Few-Shot Learning

Korea Advanced Institute of Science and Technology

Query set 𝑸Support set 𝑺 𝑸𝑺

Few-shotlearner

updateFew-shotlearner

𝑺 𝑸

Few-shotlearner*

Given 𝑺, classify 𝑸

Few-shotlearner

update overtraining episodes

𝑺 𝑸

⋯

⋯

Few-Shot Learning Handling previously unseen classification tasks (episodes)

• Training model with widely varying episodes (episodic training) [Vinyals et al., 2016]

Meta-training stage Test stage

Given 𝑺, classify 𝑸 Given 𝑺, classify 𝑸 Given 𝑺, classify 𝑸

update

TapNet: Task-Adaptive Projection Network Model description (three key elements)

• Feature extractor 𝑓𝜃• Learnable reference vectors 𝚽 = 𝝓1;⋯ ;𝝓𝑁

• Task-adaptive projection space 𝐌» Project references 𝚽 and embedded queries to 𝐌, and apply metric-based classification

Support 𝑺

Query 𝑸

feature extractor(CNN, 𝑓𝜃)

𝚽Construct

task-adaptiveprojection space 𝐌

per-classaverages

feature extractor(CNN, 𝑓𝜃)

embeddedqueries Classify queries in 𝐌

𝑑 𝐌 𝝓𝑘 , 𝐌 𝑓𝜃 𝒙𝑞

referencevectors

𝐌

task-specific conditioning

loss ℒupdate 𝑓𝜃 update 𝚽

How to Construct Projection Space 𝐌

𝒄5

𝒄4

𝒄3

𝒄2

𝒄1

𝝓1

𝝓2

𝝓3

𝝓5

𝝓4

Support 𝑺 with 5 shots Query 𝑸 embedding space

Construction of projection space via linear nulling• Error vector between per-class average 𝒄𝑘 and reference 𝝓𝑘 should be zero-forced in 𝐌.

• For every episode, compute 𝐌 = null 𝝐1; 𝝐2; 𝝐3; 𝝐4; 𝝐5 anew

𝝐1

𝝐2

𝝐3

𝝐4

𝝐5feature extractor(CNN, 𝑓𝜃)

task-adaptive projection space

𝐌

Classification and Learning

𝝓1

𝝓2

𝝓3

𝝓5

𝝓4

Support 𝑺 with 5 shots Query 𝑸 embedding space

Classifying in the task-adaptive space• Project 𝚽 and embedded queries to 𝐌 → Classify the projected queries with projected 𝚽.

• Loss based on distance in 𝐌 is used to update 𝑓𝜃 and 𝚽

task-adaptive projection space

𝐌

feature extractor(CNN, 𝑓𝜃)

Observations Projection space gives better separation of classes

Reference vector tips actually grow apart with training

Learning trend of 𝚽embedding space projection space

t-SNE plot

Results and Conclusions Non-learning-based and explicit task-adaptation method

Excellent generalization performance

Visit Poster #4 in Pacific Ballroom for further results!