SPN2: Single-Sided Privacy Preserving Nearest Neighborand its Application to Face Recognition

Djamila Aouada Dalia KhaderInterdisciplinary Centre for Security, Reliability, and Trust

University of Luxembourgdjamila.aouada@uni.lu, daliakhader@googlemail.com

Abstract

We address the privacy concerns that raise when run-ning a nearest neighbor (NN) search on confidential datain a surveillance system composed of a client and a server.The proposed privacy preserving NN search uses Boneh-Goh-Nissim encryption to hide both the query data capturedby the client and the database records stored in the server.As opposed to state–of–the–art approaches which rely ona large number of interactions, this encryption enables theclient to fully outsource the NN computation to the server;hence, ensuring a single-sided private computation, and re-sulting in a one–round protocol between the server and theclient. We analyze the practical feasibility of this algorithmon a face recognition problem. We formally prove and ex-perimentally show that the resulting system maintains therecognition rate while fully preserving the privacy of boththe database and the acquired faces. 1

1. IntroductionWhile the proliferation of surveillance systems in our dailylife enforces security, privacy is becoming a serious con-cern. Given that the captured data often contain sensi-tive information, there has been a growing public con-cern about loosing control over privacy. Efforts of find-ing a balance between security and privacy have been un-dertaken in the last couple of years [1–6]. A typical sce-nario is surveillance systems deployed in public buildingssuch as airports, used for instance for biometrics recogni-tion/identification. These systems may be decomposed intoa client part which captures the query data; and a server partthat stores the database used for system training. The recog-nition/identification task consists in comparing the querydata to the database. An important tool to perform thiscomparison is the nearest neighbor (NN) algorithm. It is

1This work was supported by the National Research Fund, Luxem-bourg, under the CORE project C11/BM/1204105/FAVE/Ottersten.

used to find the closest neighbors to the query data in thefeature space defined by the database and equipped with adistance metric. A privacy preserving NN (PPNN) is theNN algorithm run using cryptographic primitives such thatthe queried data remains confidential to the client and therecords of the database of the server are not revealed. Theconcept of PPNN was first introduced in 2006 by Shanecket al. in [7]. They proposed to use secure multiparty compu-tation (MPC) [8] to compute an NN search collaborativelybetween the client and the server without disclosing any in-formation to any external entity or to each other, other thanthe final result. In MPC, this collaboration is achieved viaa protocol that guides all interactions between the two par-ties while keeping their input private and secure, usuallyvia having them encrypted. More MPC protocols for PPNNhave emerged since [7] as surveyed in [2]. The methodsthat have been mostly researched rely on MPC and all re-quire a large number of interactions between the two par-ties of the system. These interactions and the amount ofcommunicated data that they induce unavoidably cause anincrease in the algorithm complexity as detailed in [9]. Theaim of this paper is to reduce the number of interactions.To that end one needs to go back to the basic propertiesof the encryption schemes used so far in PPNN. These areknown as partially homomorphic encryptions [10, 11]. Asopposed to fully homomorphic encryptions which allow ad-dition and multiplication operations over encrypted data,partially homomorphic encryptions only allow one of thetwo operations. One operation is not enough for computingNN. While fully homomorphic encryption schemes wouldallow more operations directly on encrypted data, they arestill unpractical and remain a theoretical solution [12, 13].There exists a special type of encryption schemes that en-ables the encryption of quadratic expressions, e.g., Boneh-Goh-Nissim’s encryption (BGN) [14].

We herein show that by using such an encryptionscheme, it is possible to design a PPNN protocol that onlyrequires a single round of interactions. This results in a sim-plified, still secure algorithm, with a largely reduced amount

2014 11th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)

978-1-4799-4871-0/14/$31.00 ©2014 IEEE 31

of transmitted data. We have implemented this algorithmfor PP face recognition. For the sake of simplicity, and inorder to directly compare with [6], we choose the same facerecognition algorithm, namely, eigenface [15] and considera squared Euclidean distance as a metric. The contributionof this paper is a single round PPNN algorithm in which theonly interactions happening are for the transmission of theencrypted query and the transmission of the encrypted dis-tances. The privacy preserving (PP) distance computationhappens at the server level without any interaction. The PPminimum search is computed at the server level and final-ized at the client level and executed thanks to a one–timerandomization.

The remainder of the paper is organized as follows. InSection 2, we describe briefly the NN search on a featurespace and give the considered trust model. In Section 3,we explain the proposed PPNN including the encryption ofsquared Euclidean distances at the server level by BGN–like encryption schemes and the proposed PP minimization.In Section 4, the PP face recognition system is summa-rized as applied on a principal component analysis (PCA)–based algorithm. Section 5 provides a performance analysisof the proposed system computationally, security–wise andreliability/accuracy–wise. Finally, a discussion and insightsare given in Section 6.

2. Privacy Preserving Nearest Neighbor

2.1. Nearest Neighbor Search

Let us consider a general NN problem where f ∈ Rn is ann-dimensional feature point and gi ∈ Rn, i = 1, · · · ,M ,are M feature vectors spanning the feature space. The pur-pose of NN is to find the nearest neighbor(s) of f in the fea-ture space within a distance τ called threshold. The mostcommon choice of the distance metric is Euclidean or, forfaster computations, its squared version. It is this squareddistance that will be considered throughout the paper. Thethreshold τ corresponds then to ε2, where ε is the radius ofan ε–ball in Rn centered at f as illustrated in Figure 1.The NN search takes two steps; first, computation of all theM distances di between f and gi as follows:

di = ‖f − gi‖2 =n∑j=1

(f(j)− gi(j)

)2. (1)

Second, given (1), the objective is to find the indexes ofthe closest feature vectors, i.e., S = {i∗} = {i | di ≤τ}, or further refining the problem by finding the in-dex of the closest vector, which can be written as i∗ =argmini{di} s.t. di ≤ τ.

2.2. Trust Model

The NN search involves the following two entities; client,which captures f , and server, which stores the databaseg = [g1, · · · ,gM ]. We assume that the trust model ishonest–but–curious. This means that on one hand, these en-tities are concerned about the confidentiality of their data.The client does not want to reveal the data it captures whilethe server does not want to reveal its entries. On the otherhand, both entities follow the protocol steps honestly and donot deviate from it. This assumption of “honesty” is reason-able since both entities are interested in knowing whether ornot any of the captured features matches one of the recordsin the server.To respect the conditions of the given trust model, com-putation of di, S and i∗ needs to be done over encrypteddata. An encryption is a function that is easy to compute,yet hard to compute the inverse of (i.e. decryption) unlessone has an extra information referred to as the key. We referto the data to be encrypted as “plaintext” and the output as“ciphertext”. In this paper, we use asymmetric encryptionwith homomorphic properties. An asymmetric encryptionscheme is the one that has a publicly known key used forencrypting and a corresponding secret key used in decrypt-ing. In other words any entity can encrypt and only theowner of the secret key can decrypt. Homomorphic encryp-tion is an encryption scheme which allows specific types ofoperations to be carried out on a set of ciphertexts to obtaina valid ciphertext which, decrypted, matches the result of aset of operations performed on the plaintext.In privacy preserving NN search, the client generates apair of keys (pk, sk) using a key generator (pk, sk) =KeyGen(λ), where pk is a public key communicated tothe server, and sk is a secret key. The security param-eter λ is a variable that determines the requirements ofthe encryption scheme (e.g. size of the key and size ofthe plaintext space) such that an attacker’s probability ofbreaking the security is negligible. For a given plaintextp and its corresponding ciphertext c, both system entitiescan encrypt and only the client can decrypt. We may writeE(p, pk) = c and D(c, sk) = p. To improve readability,we drop the keys from the encryption E(·) and decryptionD(·) functions throughout the rest of the paper. We alsouse the notation E(p), where p is a vector of plaintexts,to imply the encryption of every individual element of thatvector. Thus the client captures f , encrypts it as E(f), andsends the result to the server.

2.3. Problem Formulation

The server receives E(f), and applying the homomorphicproperties of E on (1), it attempts to compute the encrypteddistances E(di), i = 1, · · · ,M . Notice that the values ofdi remain unknown to the server. Existing privacy preserv-

32

Figure 1. Illustration of the Nearest Neighbor search in a 2–dimensional feature space distributed in a uniform hexagonalVoronoi diagram. Recall that ε2 = τ .

ing face recognition systems proposed in the literature useadditive homomorphic cryptosystems. In these encryptionschemes, for two integer plaintexts p1 and p2, the followingtwo properties are satisfied:

E(p1 + p2) = E(p1) · E(p2) and E(p1 · p2) = E(p1)p2 .(2)

The side effect of the chosen schemes is that they are lim-ited to multiplication by known constants. This means thatin (1), the encryption of the terms f2(j) cannot be done di-rectly by the server, and an interactive MPC protocol withthe client involving multiple rounds of communication isrequired. This imposes, in turn, a high throughput betweenthe client and the server [2]. Moreover, the server usuallyhas a higher computational power than the client. Imposinglarge computations on the client is constraining in most ap-plications. If the client is a camera, for instance, it wouldhave to be equipped with a powerful processor instead ofsimply collecting images. The two questions we ask are:

• Can we achieve PPNN while reducing the communicationoverhead between the client and the server to a strict min-imum, i.e., one round of communication?

• Can we achieve PPNN while outsourcing computations tothe server?

In the following section we show how these two questionscan be answered positively and propose a new single sidedPPNN (SSPPNN) protocol that we will refer to as SPN2.

3. Proposed Single Sided PPNNIn order to alleviate the limitation of MPC, we propose touse the BGN scheme [14] which is an additive homomor-phic encryption with a special property. It allows a sin-gle homomorphic multiplication operation of two cipher-texts while maintaining the additive homomorphic proper-

ties of the earlier cryptosystem. To facilitate our discus-sion, let us assume two encryption schemes E1 : Z → G1,and E2 : Z → G2; both additive with the properties in (2)with G1 and G2 being the relative ciphertext spaces. TheBGN scheme utilizes properties of “pairing based cryptog-raphy” [16] and uses the functionality P : G1 × G1 → G2

to perform:

P(E1(p1), E1(p2)) = E2(p1 · p2). (3)

With these new properties, the two parts of NN search canbe privately computed by the server as detailed below.

3.1. PP Distance Computation

To compute the encryption of the squared Euclidean dis-tances d = [d1, · · · , dM ], the server encrypts the featurevectors gi and E1(−1 · f(j)) and using the properties (2) wefind:

E2(di) = E2

n∑j=1

(f(j)− gi(j)

)2=

n∏j=1

E2(f(j)− gi(j)

)2. (4)

Given the property (3), the distance computation in (4) canbe further computed as:

E2(di) =n∏j=1

P(E1(f(j)− gi(j)), E1(f(j)− gi(j))

)(5)

=n∏j=1

P(E1(−f(j))E1(gi(j)), E1(−f(j))E1(gi(j))).

3.2. PP Minimization

The server would not be able to compare the distances sincethey are encrypted in the client’s key sk. Furthermore, itcannot send the encrypted distances E2(d) to the client asthey may reveal information about the feature space oncedecrypted.As a solution, we start by redefining the minimization prob-lem as a topological data analysis problem. To that end, westart by defining the following sign distance function:

si = −sign (di − τ) ={

+1 if (di − τ) < 0−1 otherwise. (6)

The objective in classical NN can be rewritten as finding theindexes S = {i∗} = {i | si = +1}. Topologically, this isequivalent to finding the indexes of the features gi that areinside the ε–ball around f with τi∗ = ε2. Finding the indexof the closest neighbor i∗ relies on setting the parameter τsuch that the cardinality of S is equal to one. Note that this

33

operation only uses the plaintext values of g and is thereforeinformation available at the server.Estimating the right value of τ is a classical M class–separation problem where gi are considered to be class rep-resentatives. This task boils down to finding the Voronoi di-agram defining M regions Ri, i = 1, · · · ,M , whose sitesare the M feature vectors gi. Formally, we may write:

Ri = {h ∈ Rn | ‖h−gi‖ ≤ ‖h−gj‖ for all j 6= i}. (7)

For simplicity of discussion, we assume that every regionRk is contained in a minimum enclosing ε–ball centeredat its site gi and whose squared radius is τi = ε2. Thecorresponding sign distance function si becomes a func-tion of τi instead of having a fixed τ for all classes. SeeFigure 1 for an example of a 2–dimensional feature space,e.g., (x, y) location, with a uniform hexagonal Voronoi di-agram where τi =

(12min

j‖gi − gj‖

)2. Thus, the server

computes E2(di − τi), generates a positive random vectorr = [r1, · · · , rM ], and sends E2 (ri · (di − τi)) to the client.

E2 (ri · (di − τi)) = (E2(di)E2(−τi))ri , ∀i. (8)

The randomization is needed in order to keep the distancesd private while transmitting the signed distance defined in(6). The random values ri are taken from a positive uni-form random distribution whose interval is chosen to belarge enough to avoid guessing attacks, but small enoughfor the values to remain within the plaintext space. Theclient receives the vector E2 (r� (d− τ )), where τ =[τ1, · · · , τM ] and � is a Hadamard product. This vectoris directly decrypted using sk. The set S is found as theindexes corresponding to the positive elements in the vectors = −sign (r� (d− τ )). For an appropriately chosen τ ,the index of the closest point is the only element of S.Note that all computations have been done on one side only,i.e., the server. The client only had to encrypt f and decryptr� (d− τ ).

4. Application to PCA–based Face RecognitionWe apply the proposed SPN2 algorithm to eigenface, an al-gorithm based on PCA. The procedure on plaintext can bedivided into two main phases [15]:Training Phase. Let x1, . . . ,xM be the training set of fa-cial images, in column vector form, of the same sizeR. The

average image is computed as x̄ =1

M

M∑j=1

xj . The covari-

ance matrix is computed as C =1

M

M∑j=1

(xj−x̄)(xj−x̄)T ,

where T is the matrix transpose. PCA is then applied onC, and the K largest eigenfaces constitute a basis B =[e1, . . . , eK ] corresponding to theK largest eigenvalues are

found such that K � M . The features gj ∈ RK corre-sponding to each image xj are computed by a simple pro-jection on B such that gj = (xj − x̄)TB, j = 1, · · · ,M.Identification Phase. The client, an acquisition system,captures a new facial image y and sends it to the server. Atthe server, y is projected onto the face space spanned by theeigenfaces ei. The feature vector f ∈ RK corresponding toy has its elements computed as:

f(i) = (y − x̄)T

ei, i = 1, . . . ,K. (9)

Given gj and f(i), the face recognition problem becomessimilar to an NN search the way described in Section 2.1.The distance between the captured new face and the facesin the database is obtained by computing the distance be-tween their respective features as in (1). Since the trainingphase is done on the server side only, a privacy preserv-ing PCA–based face recognition concerns the identificationphase where both parties, client and server, are involved.All steps in SPN2 are followed exactly as described in Sec-tion 3. There is, however, one additional operation requiringcryptography, namely, the extraction of the feature vector f .This is achieved as follows:

E1(f(i)) = (E1(y).E1(−x̄))eiT

, i = 1, · · · ,K. (10)

The database features have been computed in the trainingphase, however, they need to be encrypted in order to com-pute the distances in (4). The server computes E1(−gj(i)).

5. Experimental EvaluationWe analyze the performance of SPN2 for face recognitionusing eigenface in terms of complexity, security, and accu-racy. We implement the SPN2 on eigenface in Java usingthe JPBC library version 1.2.1 [17]. Testing was done usinga 2.80GHz Intel Core 2 Duo processor, and a 4GB of RAMrunning on a 32-bit Windows 7 machine. The server andclient were implemented on the same computer thus any re-ported performance data does not include network latency.Furthermore, for testing reliability of SPN2, we evaluate theaccuracy of eigenface on the ORL database of faces2, whichcontains 10 images of 40 different subjects. All images areof size (112 × 92) and are grey scaled. We choose to im-plement Freeman’s version of BGN encryption [18] given itis known to be the most efficient in today’s literature. Dif-ferent security parameters offered by the JPBC library havebeen analyzed, however, the time measures in Table 1 cor-respond to the ones of Type A1 elliptic curve [17].

5.1. Accuracy Analysis

To be able to use the homomorphic properties of an asym-metric encryption scheme such as BGN, the information en-

2http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html

34

Figure 2. Accuracy of privacy preserving binary classification inthe case of the No–Fly–List scenario.

Figure 3. Privacy preserving face recognition accuracy vs. thresh-old for different values of the scaling factor α.

crypted needs to be integers rather than decimals as theypresent group operations. Recall from Section 4, we en-crypt two types of data, the images as pixels and the fea-ture vectors as weights. Therefore, when normalizing theimage we scale it with a factor α = 10` and truncate theremaining digits. Moreover, we scale the feature vectors toβ = 10κ. It is important to evaluate the extent to whichthe reliability of SPN2 is affected by rounding and scalingthe images by the factors α and β. Our experiments showthat β hase no effect. In order to test the effect of α, weconsider two different PP face recognition scenarios. Thefirst one is an airport No–Fly–List scenario where the facerecognition system is required to give a binary answer; pos-itive when the query image is part of this list, i.e., containedin the database, and negative if it is not. To that end werun a 5–fold cross validation on the 400 faces contained inthe ORL database where 50% of the data is never used inthe training phase. We show in Figure 2 the binary classi-fication results for increasing values of the threshold τ andconsidering three different values for α. We find that the ac-

BGN Opera-tion

Time Server Client

Encryption E1 171 O(R+KM)∗ O(R)MultiplyingConstant(E1, E2)

< 1 O(RK +M) —

Addition in E1 375 O(R+RK +KM) —Addition in E2 826 O(KM +M) —MultiplyingEncryption E1

1016 O(KM) —

DecryptionD2

(Lookup 107)110 — O(M)

Table 1. The BGN-computation overhead used in SPN2 for thePCA–based face recognition algorithm. Time is indicated in mil-liseconds.

curacy of SPN2 is the same as the accuracy of the originaleigenface algorithm as run on plaintext data, and reacheda maximum recognition rate of about 95% for the consid-ered experiment and data. The only drop in performancehappens when there is no scaling, i.e., α = 1, which is ex-plained by losing information when rounding. This case canhowever very easily be avoided in practice. The second sce-nario we consider, is a full face recognition where the NNis to be found each time. To that end, all the 40 subjects inthe ORL database were used in the training within a 5–foldcross validation. The same results are observed in Figure 3where accuracy is maintained for all scaling factors exceptwhen α = 1.

5.2. Complexity Analysis

Table 1 summarizes all cryptographic functionalities ofBGN used in our application of face recognition. The com-putation and communication overhead depends on systemparameters such as size of database M , number of eigen-faces K, and the resolution of the images used R. Noticethat for running any NN search over encrypted data, we donot need to encrypt in E2 or Decrypt in D1; therefore, weomit these operations from Table 1. Furthermore, the timein Table 1 is measured in milliseconds. The ∗ indicates thatthese operations can be done beforehand, and offline3.The BGN encryption scheme was designed in such a waythat decrypting a message is done in two phases. Tradi-tionally, a decryption using a secret key retrieves the plain-text p, while in BGN-type of schemes the decryption re-trieves Bp to a known base B. The p is then computedusing algorithms such as Pollard’s Rho, Baby–step/Giant–step, etc, [19]. These algorithms are generally computation-ally expensive and require a small message space. To speedthe decryption, we propose using a pre–computed lookup

3The detailed explanation of the table will be provided in an extendedversion of this paper.

35

table as it is faster than using [20]. Table 1 computes thetime needed to decrypt a lookup table of size 107.

5.3. Security Analysis

An acquired facial image is private to the clients, whilethe database records are private to the server. The knowl-edge revealed at the end is a boolean value that expresseswhether a match was found or not. Furthermore, the cardi-nality of the database is revealed. Given that all informa-tion exchanged between the server and client are sent overan encrypted channel, an external attacker, referred to inthe literature as man–in–middle attacks, cannot retrieve anyinformation without breaking the BGN encryption which isknown to be computationally hard. The randomization donein the last step of the SPN2 protocol guarantees that the dis-tances are information theoretic secure which implies thatthe recovery of the distances is statistically hard. Recall ourtrust model is based on the honest–but–curious assumption.We thus focus on confidentiality of the data.

6. ConclusionLately, there has been a growing interest in designing aprivacy preserving NN search. Existing solutions rely onmulti–party computations causing several rounds of com-munication between the server and the client. In this paperwe proposed a solution that reduces this communication tothe ultimate minimum and that is one round. Moreover, oursolution outsources the computations to the server thus theclient’s computations become lighter. Our solution relies onthe BGN encryption scheme which is an additive encryptionscheme with one single multiplication over the ciphertexts.This multiplication is all what we need to achieve the one–round NN search. We analyzed the practical feasibility ofthis algorithm on a face recognition problem. The recogni-tion rate is shown to be the same as for the original PCAalgorithm. The size of the image data, however, makes thedeployment of this NN application in real-time still difficulttoday. Our experiments lead to insights on how to choosethe dimensionality of the feature space which is part of ourongoing work.

References[1] Z. Erkin, M. Beye, T. Veugen, and R. L. Lagendijk,

“Efficiently computing private recommendations, inICASSP’11.

[2] S. Rane, P. T. Boufounos, “Privacy-preserving nearestneighbor methods: comparing signals without reveal-ing them,” IEEE Sig. Proc. Mag., Mar. 2013.

[3] W. Lu, A. Varna, A. Swaminathan, and M. Wu, “Se-cure image retrieval through feature protection, inICASSP’09.

[4] A. Sadeghi, T. Schneider, and I. Wehrenberg, “Efficientprivacy-preserving face recognition, in ICISC’09.

[5] L. Wenjun, A. L. Varna, W. Min, “Security analysis forprivacy preserving search of multimedia,” in ICIP’10.

[6] Z. Erkin, M. Franz, J. Guajardo, S. Katzenbeisser, R.Lagendijk, and T. Toft,“Privacy-preserving face recog-nition,in PET’09.

[7] M. Shaneck, Y. Kim, V. Kumar, “Privacy PreservingNearest Neighbor Search,” in ICDMW’06.

[8] B. Schneier, “Applied Cryptography: Protocols, Algo-rithms, and Source Code in C,” Wiley; 2nd edition.

[9] R.L. Lagendijk, Z. Erkin, M. Barni, “Encrypted signalprocessing for privacy protection: Conveying the utilityof homomorphic encryption and multiparty computa-tion,” IEEE Sig. Proc. Mag., IEEE, Jan. 2013.

[10] T. ElGamal, “A public key cryptosystem and asignature scheme based on discrete logarithms,”CRYPTO’84.

[11] P. Paillier, “Public-Key Cryptosystems Based onComposite Degree Residuosity Classes,” EURO-CRYPT’99.

[12] C. Gentry, “A fully homomorphic encryption scheme,”PhD thesis, Stanford University, 2009.

[13] N. P. Smart and F. Vercauteren, “Fully homomor-phic encryption with relatively small key and ciphertextsizes,” PKC’10.

[14] D. Boneh, E.-J. Goh, and K. Nissim, “Evaluating 2-DNF formulas on ciphertexts,in TCC’05.

[15] M. Turk and A. Pentland, “Eigenfaces for Recogni-tion,” J. of Cognitive Neuroscience, 1991.

[16] D. Boneh and M. Franklin, “Identity-Based Encryp-tion from the Weil Pairing,” CRYPTO’01.

[17] A. De Caro and V. Iovino, “jPBC: Java pairing basedcryptography”, in ISCC’11.

[18] D. Freeman, “Converting Pairing-Based Cryptosys-tems from Composite-Order Groups to Prime-OrderGroups,” EUROCRYPT’10.

[19] J. Katz and Y. Lindell,“ Introduction to ModernCryptography: Principles and Protocols,” Chapman &Hall/CRC, Cryptography and Network Security Series.

[20] D. J. Bernstein and T. Lange, “Computing Small Dis-crete Logarithms Faster,” INDOCRYPT’12.

36