Near Neighbor Problem Made Fairmahabadi/slides/FairNN.pdfAll existing algorithms for this problem...

Near Neighbor Problem Made Fair

Sariel Har-PeledUIUC

Sepideh MahabadiTTIC

Nearest Neighbor Problems

• Nearest Neighbor: Given a set of objects, find the closest one to the query object.

• Near Neighbor: given a set of objects, find one that is close enough to the query object.

There are many applications of NN

Searching for the closest object

Near Neighbor

Dataset of 𝑛𝑛 points 𝑃𝑃 in a metric space, e.g. ℝ𝑑𝑑, and a parameter 𝑟𝑟

Near Neighbor

Dataset of 𝑛𝑛 points 𝑃𝑃 in a metric space, e.g. ℝ𝑑𝑑, and a parameter 𝑟𝑟A query point 𝑞𝑞 comes online

𝑞𝑞

𝑟𝑟

Near Neighbor

Goal: • Find a point 𝑝𝑝∗ in the 𝑟𝑟-neighborhood

𝑞𝑞𝑝𝑝∗

𝑟𝑟

Near Neighbor

Goal: • Find a point 𝑝𝑝∗ in the 𝑟𝑟-neighborhood• Do it in sub-linear time and small space

𝑞𝑞𝑝𝑝∗

𝑟𝑟

Near Neighbor

Goal: • Find a point 𝑝𝑝∗ in the 𝑟𝑟-neighborhood• Do it in sub-linear time and small space

All existing algorithms for this problem• Either space or query time depending exponentially on 𝑑𝑑• Or assume certain properties about the data, e.g., bounded

intrinsic dimension

𝑞𝑞𝑝𝑝∗

𝑟𝑟

Approximate Near Neighbor

Goal: • Find a point 𝑝𝑝∗ in the 𝑟𝑟-neighborhood• Do it in sub-linear time and small space• Approximate Near Neighbor

─ Report a point in distance c𝑟𝑟 for c > 1

𝑞𝑞𝑝𝑝

𝑟𝑟𝑐𝑐𝑟𝑟

Approximate Near Neighbor

Goal: • Find a point 𝑝𝑝∗ in the 𝑟𝑟-neighborhood• Do it in sub-linear time and small space• Approximate Near Neighbor

─ Report a point in distance c𝑟𝑟 for c > 1─ For Hamming (and Manhattan) query time is 𝑛𝑛𝑂𝑂(1/𝑐𝑐) [IM98] ─ and for Euclidean it is 𝑛𝑛𝑂𝑂( 1

𝑐𝑐2) [AI08]

𝑞𝑞

𝑟𝑟𝑐𝑐𝑟𝑟𝑝𝑝

Fair Near Neighbor

Sample a neighbor of the query uniformly at random

Individual fairness: every neighbor has the same chance of being reported. Remove the bias inherent in the NN data structure

Fair Near Neighbor

Sample a neighbor of the query uniformly at random

Individual fairness: every neighbor has the same chance of being reported. Remove the bias inherent in the NN data structure

Applications:Removing noise, k-NN classificationAnonymizing the dataCounting the neighborhood size

Fair Near Neighbor

Goal: • Return each point 𝑝𝑝 in the neighborhood of 𝑞𝑞 with uniform

probability• Do it in sub-linear time and small space

𝑞𝑞12

𝑟𝑟 12

Approximate Fair Near Neighbor

Goal of Approximate Fair NN─ Any point 𝑝𝑝 in 𝑁𝑁(𝑞𝑞, 𝑟𝑟) is reported with “almost uniform”

probability, i.e., 𝜆𝜆𝑞𝑞(𝑝𝑝) where

11 + 𝜖𝜖 𝑁𝑁 𝑞𝑞, 𝑟𝑟

≤ 𝜆𝜆𝑞𝑞(𝑝𝑝) ≤1 + 𝜖𝜖𝑁𝑁 𝑞𝑞, 𝑟𝑟

𝑞𝑞12

𝑟𝑟 12

Results on (1 + 𝜖𝜖)-Approximate Fair NN

𝑆𝑆𝐴𝐴𝐴𝐴𝐴𝐴 and 𝑇𝑇𝐴𝐴𝐴𝐴𝐴𝐴 are the space and query time of standard ANN

Domain Guarantee Space Query

Exact Neighborhood𝑁𝑁(𝑞𝑞, 𝑟𝑟)

w.h.p 𝑂𝑂(𝑆𝑆𝐴𝐴𝐴𝐴𝐴𝐴) �𝑂𝑂(𝑇𝑇𝐴𝐴𝐴𝐴𝐴𝐴 ⋅𝑁𝑁 𝑞𝑞, 𝑐𝑐𝑟𝑟𝑁𝑁 𝑞𝑞, 𝑟𝑟

𝑞𝑞

𝑟𝑟

Approximate neighborhood: a set 𝑆𝑆 such that 𝑁𝑁 𝑞𝑞, 𝑟𝑟 ⊆ 𝑆𝑆 ⊆ 𝑁𝑁 𝑞𝑞, 𝑐𝑐𝑟𝑟

Approximate Neighborhood𝑁𝑁 𝑞𝑞, 𝑟𝑟 ⊆ 𝑆𝑆 ⊆ 𝑁𝑁(𝑞𝑞, 𝑐𝑐𝑟𝑟)

In expectation 𝑂𝑂(𝑆𝑆𝐴𝐴𝐴𝐴𝐴𝐴) �𝑂𝑂(𝑇𝑇𝐴𝐴𝐴𝐴𝐴𝐴)

𝑞𝑞

𝑟𝑟𝑞𝑞

𝑟𝑟𝑐𝑐𝑟𝑟

Dependence on 𝜖𝜖 is O(log(1𝜖𝜖))

Experiments

Recent paper [Aumuller, Pagh, Silvestry’19] defining the same notion

Locality Sensitive Hashing (LSH)One of the main approaches to solve the Nearest Neighbor problems

Hashing scheme s.t. close points have higher probability of collision than far points

Locality Sensitive Hashing (LSH)

Hashing scheme s.t. close points have higher probability of collision than far pointsHash functions: 𝑔𝑔1 , … ,𝑔𝑔𝐿𝐿

• 𝑔𝑔𝑖𝑖 is an independently chosen hash function

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

Hashing scheme s.t. close points have higher probability of collision than far pointsHash functions: 𝑔𝑔1 , … ,𝑔𝑔𝐿𝐿

• 𝑔𝑔𝑖𝑖 is an independently chosen hash function

If 𝑝𝑝 − 𝑝𝑝′ ≤ 𝑟𝑟 , they collide w.p. ≥ 𝑃𝑃ℎ𝑖𝑖𝑖𝑖ℎIf 𝑝𝑝 − 𝑝𝑝′ ≥ 𝑐𝑐𝑟𝑟 , they collide w.p. ≤ 𝑃𝑃𝑙𝑙𝑙𝑙𝑙𝑙

For 𝑃𝑃ℎ𝑖𝑖𝑖𝑖ℎ ≥ 𝑃𝑃𝑙𝑙𝑙𝑙𝑙𝑙

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

Retrieval: [Indyk, Motwani’98]• The union of the query buckets is roughly

the neighborhood of 𝑞𝑞

• ⋃𝑖𝑖 𝐵𝐵𝑖𝑖 𝑔𝑔𝑖𝑖 𝑞𝑞 is roughly the neighborhood

𝑞𝑞

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

• How to report a uniformly random neighbor from union of these buckets?

𝑞𝑞

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

• How to report a uniformly random neighbor from union of these buckets?

• Collecting all points might take 𝑂𝑂(𝑛𝑛) time

𝑞𝑞

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

Approaches

How to output a random neighbor from ⋃𝑖𝑖 𝐵𝐵𝑖𝑖 𝑔𝑔𝑖𝑖 𝑞𝑞 :

1. Choose a uniformly random bucket2. Choose a uniformly random point in the

bucket

Approach 1: Uniform/Uniform

𝑞𝑞

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

1. Choose a random bucket proportional to its size

2. Choose a random point in the bucket

Approach 2: Weighted/Uniform

𝑞𝑞

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

2. Choose a random point in the bucket Each point 𝑝𝑝 in the neighborhood is picked

w.p. proportional to its degree 𝑑𝑑𝑝𝑝

Approach 2: Weighted/Uniform

𝑞𝑞

Number of buckets that 𝒑𝒑appears in

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

3. Keep 𝑝𝑝 with probability 1𝑑𝑑𝑝𝑝

, o.w. repeat

Approach 3: Optimal

𝑞𝑞

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

, o.w. repeat

Uniform probability

𝑞𝑞

Approach 3: Optimal

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

, o.w. repeat

Uniform probability Need to spend 𝑂𝑂(𝐿𝐿) to find the degree

𝑞𝑞

Approach 3: Optimal

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

, o.w. repeat

Uniform probability Need to spend 𝑂𝑂(𝐿𝐿) to find the degree Might need 𝑂𝑂 𝑑𝑑𝑚𝑚𝑚𝑚𝑚𝑚 = 𝑂𝑂(𝐿𝐿) samples Total time is 𝑂𝑂(𝐿𝐿2)

𝑞𝑞

Approach 3: Optimal

𝑔𝑔1

𝑔𝑔2

𝑔𝑔3

𝑔𝑔𝐿𝐿

Approximate the degree 𝑑𝑑𝑝𝑝Sample 𝑂𝑂( 𝐿𝐿

𝑑𝑑𝑝𝑝⋅𝜖𝜖2) buckets out of 𝐿𝐿 buckets to (1 + 𝜖𝜖)-approximate the degree.

Still if the degree is low this takes 𝑂𝑂(𝐿𝐿) samples.

Case 1: Small degree 𝒅𝒅𝒑𝒑:• More samples are required to estimate• Reject with lower probability -> Fewer queries of this type

Case 2: Large degree 𝒅𝒅𝒑𝒑:• Fewer samples are required to estimate• Reject with higher probability -> More queries of this type

Case 1: Small degree 𝒅𝒅𝒑𝒑:• More samples are required to estimate• Reject with lower probability -> Fewer queries of this type

Case 2: Large degree 𝒅𝒅𝒑𝒑:• Fewer samples are required to estimate• Reject with higher probability -> More queries of this type

This decreases 𝑂𝑂(𝐿𝐿2) runtime to �𝑂𝑂(𝐿𝐿) Large dependency on 𝜖𝜖 of the form 𝑂𝑂( 1

𝜖𝜖2)

Via a different sampling approach we show how to reduce the dependency to logarithmic 𝑂𝑂(log 1

𝜖𝜖).

Experiments

Setup• Take MNIST as the data set• Ask a query several times and compute the empirical distribution of the

neighbors.• Compute the statistical distance of the empirical distribution to the uniform

distribution

Experiments

Setup• Take MNIST as the data set• Ask a query several times and compute the empirical distribution of the

neighbors.• Compute the statistical distance of the empirical distribution to the uniform

distributionComparison• Our algorithm performs 2.5 times worse than the optimal algorithm, but the

other two perform 7 and 10 times worse than the optimal.• Four times faster than the optimal but 15 times slower than the other two

Conclusion

The dependence on the parameter 𝑛𝑛 matches the standard Nearest Neighbor.We get an independent near neighbor each time we draw a sample. More generally the approach works for sampling form a sub-

collection of sets.

Conclusion

collection of sets.Open Problem:

o Finding the optimal dependency on the density parameter: 𝐴𝐴 𝑞𝑞,𝑐𝑐𝑐𝑐𝐴𝐴 𝑞𝑞,𝑐𝑐

Conclusion

ThanksQuestions?

collection of sets.Open Problem:

o Finding the optimal dependency on the density parameter: 𝐴𝐴 𝑞𝑞,𝑐𝑐𝑐𝑐𝐴𝐴 𝑞𝑞,𝑐𝑐

Near Neighbor Problem Made Fairmahabadi/slides/FairNN.pdfAll existing algorithms for this problem...

Documents

Guard Zones and the Near-Far Problem in DS-CDMA Ad Hoc …community.wvu.edu/~mcvalenti/documents/72presentation.pdf · 2017-01-06 · • Near-far problem exists when an interferer

arxiv.orgarXiv:1807.07527v1 [cs.DS] 19 Jul 2018 OPTIMAL LAS VEGAS APPROXIMATE NEAR NEIGHBORS IN ℓp ALEXANDER WEI Harvard University Abstract. We show that approximate near neighbor

Approximate Nearest Neighbor Search for Low Dimensional ... · Approximate Nearest Neighbor Search for Low ... [KR02]. The problem of ANN in spaces of low doubling dimension was studied

Near Neighbor Search in High Dimensional Data (1)

SIMP: Accurate and Efﬁcient Near Neighbor Search in High ...€¦ · nearest neighbor queries. Solutions supporting a set of query ranges suffer from prohibitive space cost. There

Quantifying the Noisy Neighbor Problem in Openstack · Quantifying the Noisy Neighbor Problem in Openstack Ajay Gulati, Nodir Kodirov, Gautam Kulkarni OpenStack Summit - Austin, TX

Near Neighbor Searching with K Nearest References - …gnavarro/ps/is15.pdf · · 2015-02-03Near Neighbor Searching with K Nearest References E. Ch aveza,1, M. Gra c,1, ... Many

Lecture 4 Divide and Conquer for Nearest Neighbor Problem Shang-Hua Teng

Identification of Burkholderia pseudomallei Near-Neighbor ... · Charles Darwin University Identification of Burkholderia pseudomallei Near-Neighbor Species in the Northern Territory

K-Nearest Neighbor Categorization Problem over ... · K-Nearest Neighbor Categorization Problem over Semantically Protected Encrypted Relational Database . Ms. Radhika S. Morey Computer

Near Neighbor Join - storage.googleapis.com › pub-tools-public... · Near Neighbor Join Herald Kllapi #+1, Boulos Harb 2, Cong Yu 3 # Dept. of Informatics and Telecommunications,

1 Near-Neighbor Search Applications Matrix Formulation Minhashing

Near-Neighbor Methods in Random Preference Completionhomepages.rpi.edu/~liua6/docs/NN_presentation.pdf · 2/27/2019 · Near-Neighbor Methods in Random Preference Completion Settings

FOCUSFOCUS ONON FAITH€¦ · 10.11.2018 · Love thy Neighbor: thy homeless neighbor, thy gay neighbor, thy Muslim neighbor, thy black neighbor, thy immigrant neighbor, thy Jewish

Guide to Neighborhood Services - Concord Park · Step 3. Make Your Request to Your Neighbor in Writing If the problem persists, make your request to your neighbor in writing. If you

Quantifying the Noisy Neighbor Problem in Openstack · 2020-02-09 · Quantifying the Noisy Neighbor Problem in Openstack Ajay Gulati, Nodir Kodirov, Gautam Kulkarni OpenStack Summit

Near Neighbor Search in High Dimensional Data (1)Finding Similar Documents • Locality-Sensitive Hashing (LSH) is a general method to find near-neighbors in high-dimensional data

SOLVING A SUPPLY CHAIN MANAGEMENT PROBLEM TO NEAR ... · Optimisation to yield near optimal results across a large set of problem ... holds a major importance since primary transports

1 Nearest Neighbor Search Problem: what's the nearest restaurant to my hotel?

Near-Optimal Discretization of the Brachistochrone Problem - CCRMA