Nearest neighbor
CSE 291
A: Finding similar items from the past
Application: Novelty detection
• Past observations: x1, x2, . . . , xn from some space X• Now you see x
• Is it something familiar? Or something new that warrants attention?
Nearest neighbor approach:
• Fix a distance function d on X• Find mini d(xi , x)
• If this distance is large: x is something new
Application: Implicit classification problems
• Database of past medical patients: x1, . . . , xn
• A new patient x arrives
• Will this patient get disease y?
Nearest neighbor approach:
• Fix a distance function d on the space of patient records
• Find the k nearest patients xi using this distance function
• Find out how many of these patients got disease y
Can do this with any number of different y ’s!
B: Picking the right distance function
Example: MNIST digits
• Each image is 28× 28 and grayscale
• Can represent as a vector in R784
MNIST and Euclidean distance
Euclidean distance between two images, represented by x , x ′ ∈ R784:
‖x − x ′‖ =
√√√√ 784∑i=1
(xi − x ′i )2.
An experiment:
• Remember a set of 60,000 images.
• Then 10,000 more appear.
• What fraction of the time are their nearest neighbors the same digit? 96.91%
Failures
Query
NN
Euclidean distance doesn’t respect basic invariances for images, e.g.:
Another distance function for images: Shape contextFrom D’Arcy Thompson’s On Growth and Form:
• Identify a set of “keypoints” in each image
• Summarize each by its position relative to other keypoints in the image
• Match up keypoints between the two images
• Find a transformation between the images
• Distance is the cost of this transformation
MNIST “accuracy”: 99.37%
C: Picking the right representation
From Herbert Simon, Sciences of the Artificial:
Solving a problem simply means representing it so as to make thesolution transparent.
Representations for text
• Bag of words
• Latent semantic indexing
• Brown clustering
• Topic models
• Word2Vec and Glove
• BERT and beyond
Representations for images
• Principal component analysis, for images or image-patches
• Wavelets
• Sparse coding
• SIFT
• HOG
• Deep belief nets
• Self- or fully-supervised deep representations
Example: Histogram of oriented gradients (HoG)
1 Normalize gamma and color
2 Compute gradients
3 Spatial/orientation binning
4 Contrast normalization
5 Concatenation
Other domains
• Audio: speech, music, etc.
• Genetic sequences
• What else?