IEEE COMMUNICATIONS LETTERS, VOL. 17, NO. 4, APRIL 2013 629

Bloom-Filter Aided Two-Layered Structured Overlay forHighly-Dynamic Wireless Distributed StorageKengo Sasaki, Member, IEEE, Shinya Sugiura, Senior Member, IEEE,Satoshi Makido, Member, IEEE, and Noriyoshi Suzuki, Member, IEEE

Abstract—In this letter, we propose a novel two-layered struc-tured overlay, called a distributed Bloom filter table (DBFT),which is capable of substantially reducing overhead imposed byhighly-dynamic peer-to-peer (P2P) distributed storage systems.More specifically, in our DBFT scheme, each node’s ID remainsunchanged, when the associated cluster changes due to thephysical movement of the node. This provides us with an exclusivebenefit of maintaining the rule of structured overlay withoutimposing any additional overhead to reconfigure index betweensaved information and its stored node, which cannot be achievedby the previous P2P techniques designed for mobile ad hocnetworks. Moreover, another merit of the proposed scheme isthat stored packets per node become more equally distributedthan in the conventional schemes, because each node maintains tohave its own addressing- and content-data when it moves withina storage area.

Index Terms—Bloom filter, distributed cache system, DHT,MANET, P2P, structured overlay, wireless sensor network.

I. INTRODUCTION

RECENTLY, a new concept of area-based collaborativedistributed cache systems has been proposed in [1],

where local useful information can be stored in a wirelessdistributed storage system, which is situated in a specificgeographic area of interest. More specifically, such a dis-tributed storage system consists of a collection of wirelessmobile nodes in an infrastructure-less manner, which allowsus to operate diverse location-specific applications. For exam-ple, file/message sharing and advertising distribution are thepromising candidates. We note, nevertheless, that it is quitea challenging task to manage the connection between storeddata and its saved node in a decentralized manner.

By contrast, in the field of the Internet, the structured peer-to-peer (P2P) network [2] has been of significant interest,owing to its beneficial capability of exploiting a globallyconsistent protocol to ensure that a route to a destinationpeer is efficiently obtained. Hence, this enables building avariety of scalable and robust distributed applications over theInternet. In particular, one of the most relevant techniques isa distributed hash table (DHT) [3, 4], which employs a hashtable function in order to map keys on network nodes, rather

Manuscript received November 17, 2012. The associate editor coordinatingthe review of this letter and approving it for publication was A. Vinel.

K. Sasaki and S. Makido are with Toyota Central R&D Laboratories, Inc.,Nagakute, Aichi, 480-1192, Japan (e-mail: sasaki-ken@mosk.tytlabs.co.jp).

S. Sugiura is with the Department of Computer and Information Sciences,Tokyo University of Agriculture and Technology, Tokyo, 184-8588, Japan(e-mail: sugiura@ieee.org).

N. Suzuki is with DENSO AUTOMOTIVE Deutschland GmbH, FreisingerStrasse 21, 85386, Eching, Germany.

Digital Object Identifier 10.1109/LCOMM.2013.020513.122557

than on memory slots. This allows peers to cooperativelymaintain a distributed database, which is indexed with files anduser locations. Considering that both the distributed storagesystems and the structured overlay networks share several keycharacteristics, namely the lack of a central infrastructure, adynamic network topology, and the need for self-organization,it was a logical direction to incorporate the DHT philosophyinto distributed storage systems of [1, 5, 6], as documented in[7–9]. To be more specific, in [7] the first geographic basedrouting, called geographic hash table (GHT), was proposedfor the sake of constituting a table, which indexes a keywith static stored information by using hash functions of [3].Furthermore, in [8] mobile ad-hoc pastry (MADPastry) wasproposed to exploit location-based addressing for the sake ofadapting the overlay to the underlying network.

Typically, a coverage area of geographic routing [7–9] isdivided into diverse subareas or clusters, each supportingmultiple component nodes. Unfortunately, the previous mo-bile ad hoc network (MANET)-based DHT schemes, suchas GHT and MADPastry, cannot be efficiently carried outin the scenario, where component nodes frequently movefrom a cluster to another, since information packets thatthe nodes have need to be passed to a node belonging tothe original cluster. Naturally, this induces additional hightraffic, especially in highly-dynamic MANET scenarios, wherea topology of vehicles rapidly changes [10]. This problemmay be ignorable in the scenario of either a static distributedstorage network or a static clustering topology, where com-ponent nodes associated with a cluster head do not change.Furthermore, in [11] the distributed storage system relying onBloom filter was proposed, where a problem of probabilisticselection of a backup node is considered, while assuming thepresence of a central coordinator. In this letter, however, wefocus our attentions on the scenarios of highly-dynamic area-based distributed cache systems [1], hence the cluster headassociated with each node changes frequently. To the best ofour knowledge, there has been no literature that considered theeffects of nodes’ movements from a cluster to another withinthe location-specific range of distributed storage systems.

Against this background, the novel contributions of thisletter are as follows. We first propose a two-layered structuredoverlay, called a distributed Bloom filter table (DBFT), whichis capable of substantially reducing overhead imposed byhighly-dynamic distributed storage systems. More specifically,in our DBFT scheme, each node’s ID remains unchanged,when the associated cluster changes due to the physicalmovement of the node. This provides us with an exclusive

1089-7798/13$31.00 c© 2013 IEEE

630 IEEE COMMUNICATIONS LETTERS, VOL. 17, NO. 4, APRIL 2013

Fig. 1. An example of our DBFT structure under the assumption of M = 2clusters and K = 2 hash functions.

benefit of maintaining the rule of structured overlay withoutimposing any additional overhead to reconfigure index be-tween saved information and its stored node, which cannotbe achieved by the previous P2P techniques designed forMANETs. Another merit of the proposed scheme is that storedpackets per node become more equally distributed than in theconventional schemes, because each node maintains to haveits own addressing- and content-data when it moves froma cluster to another, assuming that the node’s movement islimited within a storage area.

The remainder of this letter is organized as follows. SectionII provides our DBFT algorithm, while Section III comparesthe performance of DBFT with GHT [7] and MADPastry [8].Finally, our conclusions are presented in Section IV.

II. PROPOSED METHOD

Let us consider that a distributed-storage area A is com-posed of multiple location-specific Clusters Sm (m =1, · · · ,M), each supporting Nm mobile component nodes.In the scenario of uploading an information packet1 to thisstorage system, the information packet is stored at a single outof the

∑Mm=1 Nm nodes. Here, it is assumed that the storage

area A and its clusters Sm are preassigned in advance of thebeginning of the storage service. Furthermore, we also assumethat the Nm nodes, which are located in the same Cluster Sm,are able to stably communicate with each other, hence knowall the Nm nodes’ IDs.

Our DBFT is the two-layered structured overlay, whichindexes, without any ambiguity, any information packets withtheir stored nodes, by introducing two parameters, namelythe cluster-specific Bloom filters (BFs) and the information-packet-specific DBFT weights. Below, we elaborate more onthe structure of our DBFT scheme. As shown in Fig. 1, eachof the M Clusters has an L-length binary BF sequence ofkm = [km,1, · · · , km,L]

T (m = 1, · · · ,M). To be morespecific, the mth cluster’s BF sequence km is calculated asfollows. Firstly, assume that the mth cluster’s nth node’s ID isgiven by ID(m,n) (1 ≤ n ≤ Nm) and that we have K integervalues hk[ID(m,n)] (k = 1, · · · ,K) for each node, wherehk[•] denotes the well-known hash function of [3], which hasthe range of 1 ≤ hk[•] ≤ L. Finally, the BF sequence km isobtained, such that the total (Nm ·K) hash functions’ outputscorresponds to unity-positions km. Otherwise, zeros are setto the rest of the km’s elements. Fig. 1 shows an example

1In this letter, the term “information packet” is used interchangeably with“chunk of data”.

of the BF realization, where each of the M = 2 clusters hasa L = 24-length BF sequence and K = 2 hash functions.We assume that the BF sequences, rather than nodes’ IDs, areperiodically exchanged between the neighboring clusters.

As a means of identifying a stored information packet’slocation, we also introduce the DBFT weights w(X ) =[w1(X ), · · · , wL(X )]T ∈ Z

L×1, where each L-length weightvector is calculated from the ID of a stored information packetX as well as from another set of hash functions. Let us assumethat we have G > K number of independent hash functionsgi[•] (i = 1, · · · , G), which is used for the entire storage areaA. Then, w(X ) is calculated, such that the ‘gi[X ]’th value isset to 2i−1 for 1 ≤ i ≤ G. Otherwise, zeros are set to therest of (L−G) elements in w(X ). For example, in Fig. 2 wedescribe the relationship of DBFT’s BF- and weight-vectors.

Having completed the setup of our DBFT scheme, wenow highlight the way of identifying the location of a storedinformation packet, having the data ID of X . Firstly, let uscalculate M integer values from the BF sequences and theDBFT weights as follows:

fi = wT (X )ki =

L∑

j=1

wj(X ) · ki,j . (1)

Then, we arrive at the cluster ID corresponding to X ’s storedlocation, according to

IDc(X ) = arg maxi

fi. (2)

Since we find the cluster ID of X from Eq. (2), we similarlyidentify a specific node ID ‘IDn(X )’ in Cluster ‘IDc(X )’,which stores the packet X . More specifically, the node IDIDn(X ) can be expressed as

IDn(X ) = arg maxi

wT (X )k(i)IDc(X ), (3)

where k(j)m represents a subset of BF km, which only

contains the elements of the jth node’s hash tablesh1[ID(IDc(X ),j)], · · · , hK [ID(IDc(X ),j)].2 This structure hasthe beneficial characteristics that the rule of the DBFT struc-ture remain unchanged if a node having information packetsmove from a cluster to another in the same storage area A.This also implies that a node is capable of moving in thearea A without imposing either any additional overhead orinformation-packet exchanges between the nodes.

In order to expound a little further, in Fig. 2 we provide asimple example of calculating the location of information X ,which corresponds to the system model of Fig. 1. Here, weconsidered M = 2 clusters, supporting N1 = 4 and N2 = 5nodes, while we have the BF size of L = 24. As shownin Fig. 2(a), the weight w can be calculated from G hashfunctions as well as the ID of the information packet X . Then,the X ’s location can be identified as IDc(X ) = 2, since wehave f1 = 5 and f2 = 15 according to Eq. (1).

As a further example, let us consider the scenario, wherethe node ID(2,5), having a information packet X , moves fromCluster 2 to Cluster 1. As shown in Fig. 2(b), the DBFT

2In order provide further insights, we have the relationship of ki =∑Nm

j=1 k(j)i .

SASAKI et al.: BLOOM-FILTER AIDED TWO-LAYERED STRUCTURED OVERLAY FOR HIGHLY-DYNAMIC WIRELESS DISTRIBUTED STORAGE 631

BF

BF

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 01 11 1 1 1 1

0 0 0 0 0 0 0 0 0 0 0 0 0 01 1 1 1 1 1 1 1 1 1

weights

8 2 4 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

(a)

BF

BF

0

0

0 0

0

0 0 0 0 0 0 0 0 0 0 0 01 11 1 1 1 1

0 0 0 0 0 0 0 0 0 0 0 0 0 0

1

1

1

1 1 1 1 1 1 1

weights

8 2 4 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

(b)

Fig. 2. (a) Structure of DBFT and its weights. (b) Changes in BFs inaccordance with node’s movement.

Fig. 3. The network model considered in our simulations, where the storagearea A is divided into location-specific M = 5 clusters.

metrics f1 and f2 becomes f1 = 15 and f2 = 5, respectively,therefore the X ’s location can be correctly identified usingthe same DBFT rule, which is achieved without any packetexchanges between the nodes.

By contrast to the above-mentioned merits, there may be acouple of potential limitations in our DBFT scheme. Firstly,a low BF-length L may induce unexpected errors as wellas limited storage capacity, due to the unavoidable statisticalslot collisions. On the other hand, a sufficiently-high L valueincreases the overhead, which is imposed by the BF-exchangesbetween neighboring clusters. Similar to the BF length L, alow G-value also degrades the our scheme’s reliability, whichreduces the maximum supportable information packets stored,while a high G-value imposes an increased calculation costfor the weights w(X ). Furthermore, in this letter we do notevaluate the overhead required for exchanges of BFs betweenneighboring clusters, which may reduce the merit of our DBFTscheme. However, this does not significantly affect the totaloverhead when each packet size, including the address and theinformation load, is sufficiently higher than the BF’s size.

III. PERFORMANCE EVALUATION

In this section, we present our performance results forcharacterizing the proposed DBFT scheme, while providingthe comparisons with GHT and MADPastry. Fig. 3 illus-trates our simulated model considered, where a distributed-storage area A is composed of five location-specific clustersSm ∈ {S1,S2,S3,S4,S5}.3 As shown in Fig. 3, each square-

3Although the cluster configuration may largely affect the achievableoverhead performance of distributed storage systems, its detailed evaluationis out of scope of this letter and it is left for future study.

TABLE IBASIC SYSTEM PARAMETERS EMPLOYED IN OUR SIMULATIONS

Number of clusters M 5Number of nodes N 100Number of nodes Nm in each cluster (N1, N2, N3, N4, N5)

= (13, 13, 48, 13, 13)Size L of BF km L = 50, 100Number of hash functions hi K = 2Number of hash functions gi G = L/2Number of information packets NP 100 ≤ NP ≤ 10, 000Number of Monte Carlo simulations 400

shaped cluster has the area of 200× 200 m2. The other basicsystem parameters employed in our simulations are listed inTable I. Here, we assumed that the total number of nodes inthe distributed-storage area A is N = 100, while we haveN1 = 13, N2 = 13, N3 = 48, N4 = 13 and N5 = 13.Also, we assumed that each node’s transmission range wasgiven by 200 m. Furthermore, NP information packets wererandomly assigned to the N = 100 nodes, where NP wasvaried in the range of 10 ≤ NP ≤ 10, 000. For the sake ofsimplicity, we focused our attention on the scenario, where thestored information packets as well as the component nodes aremaintained to exist within the storage area A.

Here, we considered a scenario, where a node situatedin Cluster 3 moves to one of the neighboring clusters, i.e.Clusters 1, 2, 4 and 5, while in turn a node in the cluster alsomoves to Cluster 3. Here, in order to evaluate the overheadrequired for maintaining structured overlay in the DHT, MAD-Pastry and DBFT schemes, we counted the average numberof transmission hops. More specifically, the hop count wasdefined as the number of transmission hops, between a nodethat originally has a stored packet and a node that receivesit. This also implies that in this definition upon increasing thenumber of packets per node, then the hop count simultaneouslyincreases. For example, the number of hops becomes threewhen the distance between source and destination nodes is500 m. The number of Monte Carlo (MC) simulations wasset to 400, where a destination node was randomly selectedfrom all the nodes in the network in each realization.

Table II compares the overhead, i.e. the average number ofhops per node and packet, which was required to maintain thestructured overlay. Here, GHT and MADPastry were selectedas the benchmark schemes. We considered two scenarios forour proposed schemes of DBFT(50) and DBFT(100), i.e. thosehaving the BF sizes of L = 50 and 100, respectively.4 Observein Table II that the proposed schemes attained several timeslower overhead imposed by maintaining the structured overlay.This result exhibited the benefit of our DBFT scheme’sunified table structure, which is capable of managing storedinformation and packets over the entire storage area A, ratherthan each component cluster.5

Next, in Fig. 4 we examined the effects of the BF-size L

4In our simulations, the BF size was determined based on our extensivesimulation results, such that it represents a good performance in terms ofoverhead. The detailed optimization is left for our future investigations.

5On the other hand, in the GHT-based benchmark scheme a node movingto another cluster has to send its own data to a node of the original cluster, inorder to maintain the structured overlay. This induces a substantial overhead,which is specific to the location-based distributed storage systems consideredin this letter.

632 IEEE COMMUNICATIONS LETTERS, VOL. 17, NO. 4, APRIL 2013

TABLE IICOMPARISON OF THE OVERHEAD

GHT MADPastry DBFT(50) DBFT(100)

Average no. hops 14.29 1.92 0.92 0.49

Size of BF [byte]

Average number of hops per node

0

5

10

15

20

25

30

0 50 100 150 200 250 300

DBFT

MADPastry

Fig. 4. The effects of the BF size on the average number of hops per node,where information packets NP was set to 1,000.

Information packets per node

Cumulative distribution function

0

0.2

0.4

0.6

0.8

1

0 50 100

GHT

MadPastry

DBFT(50)

DBFT(100)

6 %11 %

Fig. 5. The cumulative distribution functions (CDFs) of our DBFT(50) andDBFT(100) schemes as well as the above-mentioned two benchmark schemes,while the average information packets per node was set to 1,000.

on the average number of hops per node, where informationpackets NP was set to 1,000. Also, we drew the associated90% confidence interval in each point. It can be seen in Fig. 4that upon increasing the size of BF used for our DBFT scheme,the overhead size monotonically decreased. This is because asmentioned above, the more collision events happen, the lessBF size is. In the simulated scenario, the BF size has to behigher than 25 bytes, in order to achieve a lower overheadthan MADPastry.

Finally, Fig. 5 shows the cumulative distribution func-tions (CDFs) of stored information packets per node in ourDBFT(50) and DBFT(100) schemes as well as the above-mentioned two benchmark schemes, while the total informa-tion packets per node was set to NP = 1, 000. It was foundthat the two CDFs of our DBFT arrangements were almostidentical, which exhibited a lower variation than those of the

benchmark schemes. Let us observe in Fig. 5 that in GHTand MADPastry schemes longer tails are found in the CDFscompared with the DBFT counterparts, which implies thatthere exists a fraction of the nodes, which have more packetsthan others. In particular, in the GHT scheme, 11% of thenodes had more than 20 information packets, while those of

the proposed DBFT schemes were less than 3%. Similarly, inMADPastry 6% of the nodes were in charge of managing morethan 50 information packets. This ensures that the proportionof nodes that have many packets is lower in our DBFT schemethan in the benchmark schemes. Hence, our scheme has thepositive potential of equally distributing nodes’ packets andoverhead load.

IV. CONCLUSIONS

In this letter, we have proposed the BF-assisted structuredoverlay, which is invoked for highly-dynamic location-baseddistributed storage systems. The proposed scheme has themerit of substantially reducing the overhead, which is imposedby maintaining the function of P2P networks. Our simplifiedsimulation results demonstrated that the DBFT scheme iscapable of decreasing unnecessary application-layer traffic andof equally-distributing stored information packets over thecomponents nodes, while the classic GHT and MADPastryschemes are not.

REFERENCES

[1] H. Narimatsu, H. Kasai, and R. Shinkuma, “Area-based collaborativedistributed cache system using consumer electronics mobile device,”IEEE Trans. Consum. Electron., vol. 57, no. 2, pp. 564–573, May 2011.

[2] D. Chopra, H. Schulzrinne, E. Marocco, and E. Ivov, “Peer-to-peeroverlays for real-time communication: security issues and solutions,”IEEE Commun. Surveys & Tutorials, vol. 11, no. 1, pp. 4–12, 2009.

[3] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan,“Chord: a scalable peer-to-peer lookup service for internet applications,”Computer Commun. Rev., vol. 31, no. 4, pp. 149–160, Aug. 2001.

[4] J. Yu, P. Xiao, Z. Li, and Y. Zhou, “Toward an accurate snapshot ofDHT networks,” IEEE Commun. Lett., vol. 15, no. 1, pp. 97–99, 2011.

[5] A. Dimakis, V. Prabhakaran, and K. Ramchandran, “Decentralizederasure codes for distributed networked storage,” IEEE/ACM Trans.Networking, vol. 14, pp. 2809–2816, 2006.

[6] S. Sugiura, “Decentralized-precoding aided rateless codes for wirelesssensor networks,” IEEE Commun. Lett., vol. 16, no. 4, pp. 506–509,2012.

[7] S. Ratnasamy, B. Karp, L. Yin, F. Yu, D. Estrin, R. Govindan, andS. Shenker, “GHT: a geographic hash table for data-centric storage,” inProc. 2002 ACM International Workshop on Wireless Sensor Networksand Applications, pp. 78–87.

[8] T. Zahn and J. Schiller, “MADPastry: a DHT substrate for practicablysized MANETs,” in Proc. 2005 Workshop on Applications and Servicesin Wireless Networks.

[9] L. Caviglione and F. Davoli, “Peer-to-peer middleware for bandwidthallocation in sensor networks,” IEEE Commun. Lett., vol. 9, no. 3, pp.285–287, 2005.

[10] H. Hartenstein and K. Laberteaux, “A tutorial survey on vehicular adhoc networks,” IEEE Commun. Mag., vol. 46, no. 6, pp. 164–171, 2008.

[11] C. Jardak, E. Osipov, and P. Mahonen, “Distributed information storageand collection for WSNs,” in Proc. 2007 IEEE Internatonal Conferenceon Mobile Adhoc and Sensor Systems, pp. 1–10.