An enhancement towards dynamic grid-based DDM protocol for distributed simulation using multiple levels of data filtering

Parallel Computing 32 (2006) 902–919

www.elsevier.com/locate/parco

An enhancement towards dynamic grid-based DDMprotocol for distributed simulation using multiple levels

of data filtering q

Azzedine Boukerche *, Caron Dzermajko, Lu Kaiyuan

PARADISE Research Laboratory, SITE, University of Ottawa, Canada

Abstract

In large-scale distributed simulation systems, participating federates may receive and send a large number of messages toshare state data and update subscription and publication regions. In an effort to offer a scalable data distribution managementsolution (DDM), we propose two variants of DDM scheme, which we refer to as an optimized dynamic grid-based DDMand, grid-filtered region-based DDM. The main focus of data distribution management (DDM) is to limit and control thevolume of data exchanged between participating simulated entities (hosts) in large scale distributed simulation systems. Inthe first variant, we propose an enhancement of a dynamic grid-based DDM scheme, which we refer to as an optimizeddynamic grid-based DDM, to further reduce irrelevant data that might be received by simulation entities. In the second var-iant, we propose a grid-filtered region-based DDM scheme that determines the percentage of grids covered by the subscrip-tion or publication region using a grid overlay on the virtual space, and further filters irrelevant exchanged data, based on apercentage threshold, using a matching technique, like that of the region-based DDM scheme. We present our DDMschemes, discuss their implementation and report on their performance evaluation using an extensive set of simulation results.� 2006 Published by Elsevier B.V.

Keywords: HLA/RTI; Interest management; Data distribution management

1. Introduction

The High Level Architecture (HLA) was originally developed by the Defense Modeling and SimulationOffice (DMSO) of the Department of Defense (DoD) to meet the increasing needs of defense-related projects[1]. HLA aims to provide highly reusable and interoperable software architecture for the construction ofcomponent-based distributed interactive simulations (DIS). In 2000, HLA was accepted as IEEE Std 1516-2000, the Standard for Modeling and Simulation High Level Architecture. Run Time Infrastructure (RTI),

0167-8191/$ - see front matter � 2006 Published by Elsevier B.V.

doi:10.1016/j.parco.2006.09.008

q This work was partially sponsored by the Canada Research Chair Program, NSERC, Canada Foundation for Innovation Funds, andOIT/Ontario Distinguished Researcher Award.

* Corresponding author.E-mail address: [email protected] (Prof. A. Boukerche).

mailto:[email protected]

A. Boukerche et al. / Parallel Computing 32 (2006) 902–919 903

the middleware software defined in HLA Interface Specification, provides necessary services to coordinate theinter-operation among the distributed simulation entities.

Data distribution management (DDM) is one of the six services provided by RTI. It is responsible for pro-viding efficient, scalable, interest-based mechanisms for distributing state updates and interaction informationin HLA based distributed simulations. Specifically, DDM will throttle the number of messages that are beingplaced on the network and relief the processing burden on the simulation entities (or federates). While design-ing DDM algorithms for large-scale systems, more issues need to be considered, such as, dynamic grouping ofparticipants according to their communication interests, efficient data multicasting with bounded end-to-endnetwork latency, and optimized signaling overhead on multicast group membership management.

The remainder of this paper is organized as follows. Section 2 presents an overview of DDM. In Section 3,we discuss previous DDM schemes. Section 4 introduces our proposed DDM variants and discusses theirimplementation followed by proposed DDM variants. Section 5 presents our simulation environment as wellas the results we have obtained in order to evaluate the performance of our DDM schemes. Section 6 con-cludes the paper.

2. Overview of DDM

Before we move forward with a discussion about DDM, there are a few terms we need to define.

• Dimension: a named coordinate axis.• Extent: a sequence of ranges, one for each dimension.• Federate: a simulator or processor participating in a federation simulation.• Federation: a distributed simulation.• Interest region: a general term referring to publication and subscription regions.• Intersecting region: the intersection of a publication and subscription region.• Name space: a set of values used to define and describe information of interest.• Publication region: a defined set of data for which messages deliver state updates to subscribing federates.• Range: a set of ordered pairs defining a continuous interval on a dimension.• Region: a set of extents that are bound to declare a routing space.• Routing space: a multi-dimensional coordinate system.• Subscription region: a defined subset of the name space that describes the data in which a federate is

interested.

The role of data distribution management (DDM) is to limit and control the volume of data exchangedduring a simulation, and reduce the processing requirements of participating hosts by relaying events and stateinformation only to those applications that require them [6–18]. To accomplish this task we use a multi-dimen-sional coordinate system known as a routing space, defined areas of interest to federates, called publicationand subscription regions and the identification of intersection regions, which are the intersections of publica-tion and subscription regions. Fig. 1 illustrates a two-dimensional routing space with two regions.

Fig. 1. Two-dimensional routing space.

904 A. Boukerche et al. / Parallel Computing 32 (2006) 902–919

2.1. Routing space

The multi-dimensional coordinate system, known as a routing space, is a grid without partitions withinwhich a federate declares interest in receiving or sending state updates and interaction notifications. Wedeclare routing spaces through the HLAs federation execution data interchange format (FED DIF), whichis a standard file exchange format to facilitate the use of tools for data file transfer and exchange between fed-erates. This declared routing space, is a normalized coordinate system consisting of at least two dimensions(usually more) and representing the simulation environment or virtual world of the simulation.

To put this in perspective, we will draw an example. Imagine a simulation of a military tank opposing anenemy fortress. At the least, this simulation requires a routing space consisting of three dimensions. Two spa-tial dimensions are required, to represent tank movement (North–South and East–West) and a team or coun-try designation is needed, to indicate opposition. The team determines an entity’s ability to detect or bedetected by an entity on the opposite team, which brings us to the next subsection, describing publicationand subscription regions.

2.2. Publication and subscription regions

A subscription region is the area of interest declared by a federate, within a routing space. A publication

region is the set of data made available to other federates by a simulated entity, or federate. An interest region

is a general term that refers to both publication and subscription regions An example of a subscription regionmight be the 100 sq. mile region surrounding a tank, where the maximum range of visibility, of that tank, isdefined as the boundaries of the subscription region. This tank would require information about any objects,on the opposite team, being simulated within that hundred square mile region, but is not interested in infor-mation about any objects being simulated outside of that region. In our scenario, we use point publicationregion, where the publication region of the opposing tanks are represented by a two-dimensional coordinate.Where these subscription and publications regions overlap, is called an intersection region, which we describefurther in the following section.

2.3. Intersection region

The intersection of defined publication and subscription regions is called an intersection region. An inter-section region exists when corresponding region sets overlap. When an intersection region exists, dataexchange occurs between the subscribing federate and the publishing federate. How to identify these intersec-tion regions and the method in which publishing and subscribing federates exchange data, are the basic con-cerns of DDM.

3. Previous and related work

In the past few years, a considerable amount of research has been focus on data distribution manage-ment in the HLA/RTI context. Based on how to perform the matching to detect intersection regionsbetween publication regions and subscription regions of all participating simulation entities, DDM meth-ods can be divided into two categories, region-based DDM methods and grid-based DDM approaches.Depending on the manner of multicast group assignment, grid-based scheme can be further divided intofixed grid-based DDM and dynamic grid-based DDM. Now, let us have a closer look at each of the threeapproaches.

3.1. Grid-based DDM

The motivation behind the grid-based approach is to reduce the computational and communication over-head of the region-based method. While the region-based approach necessitates matching between all publi-cations and all subscriptions, the grid-based technique avoids these extensive and costly operations. Instead ofexplicitly, comparing publication and subscription regions, an RTI maps each interest region onto a multi-

Fig. 2. Grid overlay.


dimensional grid, which represents the routing space, as shown in Fig. 2. The grid will generally have the samenumber of dimensions as the routing space.

Referring to Fig. 2, each grid block, or cell, is numbered one through four. The subscription of a spy plane,located in cell 1, is mapped to cell 1 and cell 3. These cells represent the spy plane’s area of interest. The pub-lication of planes in one squadron, which are the three planes located in cell 3, is mapped to cell 3. The fourplanes, or squadron of planes, located in cell 4, have a publication mapped to cell 4. Cells that are part of botha publication and subscription represent the intersection. In Fig. 2, the intersection occurs in cell 3. Therefore,entities with subscriptions to cell 3, such as the spy plane, require updates about entities publishing to cell 3,like the three squadron planes.

An RTI component on each host performs the process of overlaying a grid on the terrain. Each host’s RTIcomponent performs the region-to-cells mapping independently of the other hosts since we initialize the num-ber and size of the grid cells before the simulation begins and these values remain constant throughout beforethe simulation begins and these values remain constant throughout the federation execution. This methoddoes not require a centralized DDM coordinator and does not incur the overhead involved in the complexoperation of matching. While this potential reduction in communication overhead offers advantages in scala-bility, the major drawback of the grid-based method is that its accuracy in identifying intersection regionstends to be lower, compared to the region-based method. The mapping of the interest regions to the grid cellsmay not be exact, since region boundaries tend not to fall on cell boundaries, in which case, the area coveredby the cells, representing the interest region, will be slightly larger than the region itself. This may cause super-fluous intersections in some cells, leading to publishers sending subscribers unneeded data.

3.2. Region-based DDM

The region-based DDM approach, implemented in the RTI 1.3 compares the subscription and publicationregions, directly, in order to find which ones overlap we refer to this process as matching. Once the DDMmechanism identifies intersections, a second type of communication must be enabled, and that is the sendingof data from publishers to subscribers. This second kind of inter-host communication generally uses multicasttechnology which we will discuss further in Section 3.

Now we will review how the region-based scheme uses the information, provided by matching, to assignhosts to multicast groups and facilitate data transfer. Fig. 3, represents a federation with four regions, Region1, Region 2, Region 3 and Region 4, and five federates, subscribing federates S1 and S2 and publishing fed-erates P1, P2 and P3. Note that S2 and P3 intersect and that the DDM mechanism in a region-based DDMscheme would detect a match as a result of that intersection. The detection of that match would trigger Region2 to join a multicast group, we will call MG-P and the DDM mechanism would trigger federates S2 and P3 tojoin MG-P. P3 then begins sending update data to all subscribing federates on MG-P, or, in our example, toS2. Subscriptions that intersect with Region 2 also join multicast group MG-P and their federates receive trig-gers to join MG-P, thereby receiving any data intended for that group. This facilitates the transfer of data

Fig. 3. Region-based DDM.


between P3 and all other federates interested in the data P3 publishes in Region 2. After the DDM mechanismperforms additional matching, due to the addition, modification, or deletion of regions, suppose Region 2 isno longer part of an intersection. Then, all federates that belong to MG-P, including P3, receive triggers toleave the group, terminating the data exchange between publisher(s) and subscriber(s).

Another way of determining intersecting regions with the region-based method is by using a centralizedcoordinator. All federates send their interest information, or subscriptions to the DDM coordinator, whichis a dedicated processor that collects the publication and subscription information in a database. TheDDM coordinator is responsible for performing the matching between publications and subscriptions andtriggering federates to join or leave multicast groups based on the results of the matching. One of the limitingfactors of using a DDM coordinator is the potential for a bottleneck, since all messaging performed in a sim-ulation, using a DDM coordinator, must be triggered and processed by the single processor that is designatedas the coordinator.

The grid-filtered region-based DDM method uses matching, rather than a DDM coordinator, to determineintersection regions. The following section presents this new algorithm and describes the differences andadvantages of this algorithm over the region-based DDM method.

3.3. Our motivation and weakness of dynamic grid-based DDM algorithm

In this section, we wish to highlight some of the weaknesses of the dynamic grid-based DDM algorithmbefore we propose our solution to overcome these weaknesses. As we already discussed earlier, the dynamicgrid-based DDM approach [3] exhibits a few appealing characters when compared to the other two methodsmentioned above. Whereas it still has some weakness, which are the motivation behind the enhanced dynamicgrid-base DDM algorithm. We wish to optimize its performance by enforcing it with a second level of sender-side filtering.

Consider the situation of a grid cell i shown in Fig. 2. Besides the overlapped publication region u1 andsubscription region s1, there still irrelevant publication region u2 and subscription region s2. According todynamic grid-based approach, a multicast group is assigned to this cell in that publishing and subscribingregions co-exist and all the pub/sub federates must have joined the multicast group. In this case (1) the pub-lisher of u2 has to send data that no data consumer is interested and the subscriber of s2 has to receive datathat it is not interested in. Even within the overlapping regions, (2) there are some needless data being sent,such as the area of U1 minus the shade area. Sending and receiving these needless data put communication(in case 1 and 2) and computation (in case 2) overhead to DDM services, which degrade the performanceof DDM in large-scale distributed simulations.

Furthermore, the dynamic grid-based approach assigns unnecessary multicast groups, such as the caseshown in Fig. 3. Since the publishing and subscribing regions (u1, s1) co-exist in this cell, a multicast groupis assigned to this cell even there is no intersection existing in this cell, which means the communication band-width is occupied with no data transmission. This is another drawback that needs to be eliminated for large-scale distributed simulations.


4. Proposed data distribution management (DDM) algorithms

Realizing the weakness discussed earlier, we make a modification to the data distribution managementschemes approach to limit and control further these needless data from being sent and prohibit the unneces-sary multicast group from being created. Our solution is to incorporate precise filtering mechanism with thedynamic grid-based DDM scheme.

4.1. Optimized dynamic grid-based DDM algorithm

Generally speaking, filtering can occur in two general locations: at the sender or at the receiver. If a pub-lisher knows that another federate is not interested in an update, then that publisher could avoid sending theupdate to the federate and thus save the expense of sending the data. This is sender-side filtering. If the pub-lisher does not know whether or not a federate needs an update, then the publisher must send the data and thesubscribing federate’s LRC must determine locally whether to deliver the update to the federate or not. This isa receiver-side filtering. The sender-side filtering paradigm is more desirable but is also more difficult andcostly to compute. Often, DDM services using sender-side filtering gain higher performance. When updatesand interactions are filtered out before they are sent, there are savings involved in the cost of sending the data(e.g., sender processing and network bandwidth), costs in receiving the data (e.g., receiver processing and fil-tering the data at the receiver), and finally the callback to the federate. When an update is filtered out at thereceiver’s side, only the callback to the federate is saved. Therefore, our modification is about sender-sideusing a more precise filtering mechanism.

In our enhanced DDM algorithm, each cell has an owner responsible for collecting publication regions andsubscription regions information like the region-based method, and stores them in a special data structure.The information includes which federates are publishing or subscribing to the cell and the correspondingregions, which are used to detect the intersection regions. Cell owner allocates multicast group to a cell ifand only if an intersection exists, thus reduces the number of multicast group created. This is different fromthe original dynamic grid-based approach, which assigns multicast group to a cell when the pub/sub regionsco-exist in the cell. The information of detected intersection regions and the related federates are stored in alocal publish/subscribe table. Whenever a change to the table occurs during the run time, the cell owner willsend out join/leave message to trigger the corresponding federates to join or leave the multicast group. Thismethod, which is complementary to the weakness mentioned above of dynamic grid-based approach, makesDDM only send out messages that are absolutely necessary and optimizes the utility of network resources.

In what follow, we shall present the details of the enhanced dynamic grid-based DDM algorithm. The localmatching process for each cell, multicast group assignment and specific data structure to be used are included.

4.1.1. Matching step of the optimized algorithmFirstly, cells are evenly distributed among the nodes that are participating in the simulation. The node (or

processor) to which a cell is distributed is said to be the owner of that cell. Cell owner keeps track of federatesthat are publishing and subscribing to each cell during the execution of the simulation.

The major change to the execution steps of the enhanced algorithm, as shown in Fig. 4, lies in the imple-mentation details of the matching process. It adopts the exact matching in every cell like in the region-basedDDM. To realize the exact matching in a cell, the cell owner needs the information of publishing and subscrib-ing regions. We use two bit-arrays and a pointer list Interest_Region list to store those data at the cell ownernode like shown in Fig. 5.

The cell owner is responsible for matching the publishing regions and the subscribing regions in the pointerlist to find intersection regions. Once intersection region (or regions) is found, the related information is storein the data structure Intersection_Info list shown in Fig. 5 for the later use.

4.1.2. Multicast group assignment

Multicast group is allocated to the cell, where pub/sub intersection occurs. The cell owner is responsible forcalculating the intersection regions when a change of regions in the cell occurs and store the information ofintersection regions such as the publishers, subscribers, and the extents of a certain intersection region in a

Fig. 4. A grid cell with multiple regions.

Fig. 5. A cell with sub/pub but no intersection.


local table, of which the intersection region_ID should be the index. A multicast group is allocated to a cell ifand only if there is at least one intersection, and no more than one multicast group can be allocated to thesame cell. After additional matching has been performed, if no intersection region exits any longer, the cellowner will release the assigned multicast group.

Once the cell owner detects any publisher/subscriber intersection in a cell, it creates a multicast group andassigns it to the cell. All federates publishing or subscribing to the intersection region (or regions) must be toldto join the group. The cell owner is responsible for sending them a join_group message. The join_group messagefor publishing federates includes the bounded ranges of the intersection region. Based on the bounded rangesinformation, publishing federates only send out data of absolutely necessary. Whereas the information aboutbounded ranges is not included in the join_group message to subscribing federates. Similarly, when the ownerdetects the cessation of a publisher/subscriber intersection due to unpublisher/unsubscriber, it sends a lea-

ve_group message. Each time a join_group or leave_group message is sent, an update message is sent to themanager of the multicast group, which is responsible for updating the group membership list.

4.2. Grid-filtered region-based DDM

We propose a new method of DDM we call grid-filtered region-based DDM. This scheme utilizes a gridoverlay on the simulated environment, determines the percentage of the grid covered by the publication orsubscription regions of federates responsible for objects positioned within that grid, then, based on a percent-age threshold, which is a concept we will describe in more detail a little later in this section, the DDM mech-anism determines if matching should be performed to further filter data before a message is sent from the hostsimulating the publishing entity to the host simulating the subscribing entity. In the subsections that follow, wewill define the data structures used to implement this DDM method and discuss the potential advantages overa region-based DDM implementation.


4.2.1. Data structures

• Threshold: cut off value to trigger matching. If a region covers a percentage of a cell higher than threshold,the region will automatically match the entire cell. Otherwise, each entity in the cell is explicitly checkedagainst the region for overlap.

• Cells: two-dimensional array of cells. Indexed by row and column• Cellij.ID: unique integer, identifying cell at position (1, j) in cells.• Cellij.P_fullcoverage: set of publishers whose coverage value exceeds threshold.• Cellij.S_fullcoverage: set of subscribers whose coverage value exceeds threshold.• Cellij.P_partcoverage: set of publishers whose coverage value is less than threshold.• Cellij.S_partcoverage: set of subscribers whose coverage value is less than threshold.

4.2.2. Initialization

Initialization distributes the responsibility for managing cells among the available federates. This is done byusing a mod operation. For example, if there are N federates, federate 0 will be responsible for managing cells0, N, 2N,. . .

For all cij 2 Cells

Make federate (cij.ID mod NumFederates) manager of cell cij

Initialize Simulation.

4.2.3. Object Oi on federate Fm updates its publication or subscription region in routing space

When a simulated entity’s publication or subscription region changes, the set of cells that entity subscribesor publishes to changes as well. The cell manager has to be updated with the proper information, including thepercent of cell covered by the region. First comparison of cell boundaries and region boundaries determine theset of cells, Rcells, where an overlap occurs. Secondly, the percent of cell area overlapped by the region is cal-culated. This is done by finding the smallest area defined by the region and cell boundaries. The calculationassumes a coordinate system such that (0,0) is the low left corner of the simulation space. The calculationdetermines the vertical range by using the minimum of the upper coordinate of the region and the upper cor-ner of the cell and the maximum of the lower coordinate of each. The horizontal range is found using the min-imum of the right edge of both cell and region boundaries and the maximum of the left edge of each. Theseranges are multiplied to find the area of the overlap which is then divided by the cell area. (Note: IF cell areasare uniform and constant, this last division is unnecessary. Threshold can be measured in space rather thanpercentage.) Once this is calculated a notification of cell entry is sent to the federate managing the cell, includ-ing the percent covered and the region’s boundaries.

Fm:Map region to set of cells Rcells such that if any part of the region overlaps a cell cij, cij is in Rcells.

Let R_lft = region.leftboundLet R_rght = region.rightbound

Let R_upr = region.upperbound

Let R_lwr = region.lowerbound

For all cij 2 Rcells

{cell_areaij = (cij.upper � cij.lower) * (cij.right � cij.left)percent coveredij ¼ ððminðcij :upper;R uprÞ�maxðcij:lower;R lwrÞÞ � ðminðcij:right;R rghtÞ�maxðcij:left;R lftÞÞÞ

cell areaij

Identify owner Fk of cellijSend cell_entry (R_lft, R_rght, R_upr, R_lwr, cij, and percent_coveredij, type, Fm) to Fk

}


4.2.4. Federate Fk receives cell_entry(R_lft, R_rght, R_upr, R_lwr, cij and percent_coveredij, type, Fm)

When a federate receives notification of an entity entering a cell, it uses the information sent with the notifi-cation to add the entity to one of four lists. If the type is publisher it adds the entity to one of its two publisherlists. If the type is subscriber its added to a subscriber list. The two publisher lists divide the entities based on per-cent of the cell covered by the region. If the percent exceeds the threshold value, it is added to the list indicating fullcoverage, otherwise its added to the partial coverage list. The same logic is applied to subscribers.

Fk:if type is publisher then

{new publisher Pub (Fm, R_lft, R_rght, R_upr, R_lwr)if percent_covered P threshold then

cij.P_fullcoverage = cij.P_fullcoverage + Pub

else

cij.P_fullcoverage = cij.P_partcoverage + Pub

}else if type is subscriber then

{new subscriber Sub (Fm, R_lft, R_rght, R_upr, R_lwr)if percent_covered P thresholdelse

cij.S_fullcoverage = cij.S_partcoverage + Sub

}

4.2.5. Matching on Federate Fk

When a federate must perform matching on its cells, it performs the following steps. For each cell it firsttakes each subscriber in its partial covered list and performs an exact matching algorithm against each pub-lisher in its partial covered list of publishers. It concatenates the returned list of matches with all elements in itsfull coverage list of publishers. It then triggers these federates to join the proper multicast groups. The algo-rithm assumes multicast groups are uniquely identified by the publisher writing to the group. Then the algo-rithm creates a matched list of all publishers in the cell and instructs all subscribers in its full coverage list tojoin multicast groups for each publisher in the matched list.

For all cij managed by Fk

{For all s 2 S_partcoverage

{matched = match (s,P_partcoverage) + P_fullcoverage

s.F.trigger_federate_join (matched)}matched = P_partcoverage + P_fullcoverage

For all s 2 S_fullcoverage

{s.F.trigger_federate_join (matched)

}}

4.2.6. Match (s,P_partcoverage)

Performs an exact matching of subscriber s and publishers in P_partcoverage. If an overlap between regionsexists, the publisher is added to a list which is returned when all matches are found. If a match exists there arethree cases


1. The publication region completely encloses the subscription region.2. The subscription region completely encloses the publication region.3. At least one edge of the publication region will intersect with at least one edge of the subscription region.

This is the case when there is only partial overlap between the two regions.

exact_match = ;For all p 2 P_partcoverage

{if (p.left 6 s.left) ^ (p.right P p.right) ^ (p.upper P s.upper) ^ (p.lower 6 s.lower) then

exact_match = exact_match + p

else if (s.left 6 p.left) ^ (s.right P p.right) ^ (s.upper P p.upper) ^ (s.lower 6 p.lower) then

exact_match = exact_match+pelse

{if ((s.left 6 p.left 6 s.right) _(s.left 6 p.right 6 s.right)) then

{if (p.lower 6 s.lower) ^ (p.upper P s.upper) then


else if (s.lower 6 p.lower 6 s.upper) _(s.lower 6 p.upper 6 s.upper) thenexact_match = exact_match + p

}else if ((s.lower 6 p.lower 6 s.upper) _ (s.lower 6 p.upper 6 s.upper)) then

{if (p.left 6 s.left) ^ (p.right P s.right) then


else if (s.left 6 p.left 6 s.right) _ (s.left 6 p.right 6 s.right) then


}}

}

4.2.7. s.F.trigger_federate_join (matched)Causes the federate modelling subscribers s to join multicast groups indicated by the matched set. s.F indi-

cates the federate controlling entity s. The algorithm assumes multicast groups are named based on thepublisher.

For each publisher p in matched

s.F.join_multicast_group (multicastp)

5. Performance evaluation

In this section, we present the simulation experiments we have carried out to evaluate the performance ofour optimized dynamic grid-based DDM algorithm. During our experiments we wish to be able to study clo-sely the behavior of our optimized protocol through comparing it with the original dynamic grid-based DDMand the region-based DDM which has been implemented in RTI 1.3 [2].

5.1. Simulation environment

The optimized algorithm is implemented and incorporated with Grid-Kit [3], which was previously devel-oped by our research lab using the Georgia Tech. RTI-Kit [5]. The RTI-Kit contains a set of libraries designed


to support the development of the run-time infrastructures (RTIs) for parallel and distributed simulations,especially federated simulations running on high performance computer platforms. The Grid-Kit is a separatecomponent integrated with the RTI-Kit performing DDM services with several DDM schemes, namelyregion-based, grid-filtered-region-based, fixed grid-based, dynamic grid-based and the newly added optimizeddynamic grid-based methods. It is envisioned that new DDM algorithms can be added to the Grid-Kit makingit more flexible to different simulation DDM requirements.

5.2. Mini-RTI Kit

In this section we describe the relationships between our mini-RTI and the Georgia Tech. RTI Kit The pri-mary purpose of our mini-RTI Kit is to serve as an interface to the Grid-Kit For this reason, our mini-RTI Kitsupports only the data distribution management service. We implement a component called Grid-Kit thatintegrates with the RTI Kit and performs DDM using the grid-based DDM schemes. A user may call uponthe various components of the RTI Kit together, or individually, depending on their simulation needs. In thiscontext, the user is an RTI builder or researcher.

Our mini-RTI uses the Grid-Kit that we developed and utilizes, for the purpose of this research, the region-based DDM and the grid-filtered region-based DDM methods to communicate data in a collaborative virtualenvironment, simulating a shopping mall setting. In addition to using the Grid-Kit, the mini-RTI also usescertain components of the RTI Kit Of the several RTI Kit components completed, the one that is most impor-tant to our work is the MCAST library (MCAST-Lib).

5.3. Multicast group communications

The MCAST-Lib component of the Georgia Tech. RTI Kit provides our mini-RTI with group communi-cation services, using the multicast paradigm. Currently the MCAST-Lib implements multicasting in software,meaning that the network does not need to be multicast-enabled in order to use it. MCAST-Lib achieves com-munication over the network as follows. It uses another component of the Georgia Tech RTI-Kit, the FM-Lib, which is communication layer software, API based on the Illinois Fast Messages (FM). FM-Lib canbe compiled using various transport protocols. We use the TCP/IP in our implementation. Note that optimi-zations made to the MCAST-Lib will not affect the mini-RTI, as long its API remains unchanged. If addi-tional functionality is added to the MCAST-Lib, we can modify our mini-RTI accordingly to takeadvantage of these properties. Delete multicast groups is foremost among the enhancements to theMCAST-Lib that could benefit our work. There is currently no way to indicate that a particular multicastgroup is no longer being used.

To circumvent the inability to delete multicast groups once they are no longer needed, the mini-RTI recy-cles such groups. Once an allocated group is no longer in use, the mini-RTI places it on a free list. Before cre-ating a group, the mini-RTI checks the free list and removes a group from that list and reuses it. A new groupis only created and initialized if the free list is empty.

It is the mini-RTI, by way of the Grid-Kit, that determines when federates should join and leave groups.However, the MCAST-Lib manages the actual membership lists for each group. In order to test the Grid-Kit,we develop a mini-RTI that uses the Grid-Kit to perform DDM. This RTI does not perform all of the servicesspecified by HLA, and is not meant to be a complete RTI.

In what follows, we describe how the mini-RTI uses the Grid-Kit to carry out DDM. Recall that a Grid-Kitcomponent is present on each node on which the mini-RTI runs. Since the mini-RTI links to each federateexecutable program, the mini-RTI, and the Grid-Kit, runs on all nodes that participate in the simulation.One of the primary functions of the Grid-Kit is to represent the grid cells and distribute these cells amongthe Grid-Kit components running on each node. There is no centralized representation of the grid, whichis one of the strengths of our approach.

Note that both the cells that the Grid-Kit manages and the federate simulated entities reside on each node.However, the federate is not aware of the grid cells, since a federate should not make assumptions about howan RTI is performing DDM. Just as the federate is designed to work with different RTIs, we design the Grid-Kit to easily integrate into different RTIs. For this reason, the Grid-Kit is not aware of individual entities, only


federates. The RTI is responsible for managing information about the federate simulated entities, if desired,and the mini-RTI performs this task. Federates express interest in terms of regions and the mini-RTI trans-lates these regions into the cells they overlap. For each cell, the mini-RTI sends a request to the Grid-Kit man-aging that cell, directing it to publish, subscribe, unpublish, or unsubscribe that federate to or from that cell.We represent the interface between the federate, mini-RTI, and the Grid-Kit on a particular node in Table 1.The Grid-Kit is technically part of the mini-RTI, although, in this context, the term mini-RTI refers to allmini-RTI components, except those that are part of the Grid-Kit. Referring again to Table 1, the federatesand mini-RTI involved in steps 1 and 2 are both on the same Node X. However, the Grid-Kit, that performsstep 3, is not necessarily on Node X. For example, the region used in step 1 is mapped to one or more cells instep 2. These cells are either managed locally, by the Grid-Kit on Node X, or remotely, by the Grid-Kit onother nodes. For remote cells, the mini-RTI on Node X sends a message to the remote node requesting theappropriate operation be performed on the specified cell. For local cells, the mini-RTI simply makes a func-tion call that invokes the local Grid-Kit (see Table 2).

Let us now turn to our simulation results. We simulated the Tank dogfight scenario [3,4], which is typicallyemployed in researches on data distribution management in parallel and distributed simulations. Initially, theobjects are randomly placed in the two-dimensional battlefield. Each tank moves with a constant velocitytowards a randomly chosen direction at each time step. All tank objects are both publishers and subscribers(see Table 3).

For the experiments, we run the simulations on 3.0 GHz PentiumIV IBM computers connected by 100 MbEtherent, running Linux. Six federates are spawned in each simulation run, and each federate simulates thesame number of tank objects. Half federates model red tank objects and half model blue ones, therefore,the routing space has three dimensions: North–South, East–West, and Team. The battlefield is 500 km2. Sim-ulations consisted of 100 time steps. Table 1 summarizes the parameters that were held constant throughoutour experiments.

We have evaluated our optimized DDM protocol using the following performance metrics:

5.4. Optimized dynamic grid based DDM – experimental results

In section, we wish to investigate the performance of our optimized dynamic grid-based DDM algorithm bycomparing it with region-based and the initial dynamic grid-based DDM. Here we use the terminology UPSD(the number of units per spatial dimension) to express the division of the routing space. In our experiments thefollowing equation holds:

Table 1Component interface

Federate Mini-RTI Grid-Kit

Responsibility Simulates objects Maps regions to cells Manages cellsSteps (objects) 1-Performs operation

with region2-Performs operation foreach cell region

3-Updates data structurebased on operation

Table 2Constant parameters for the simulations

Parameters Values

Number of federates 6Battlefield size (km2) 500Sensor range (km) 8.0Number of routing space dimensions 3Object moving speed (km/time step) 4.0Number of time steps 100

Table 3Performance metrics

Metrics Discription

DDM time The total time consumed by the DDM servicesDDM messages The total number of messages generated to perform the task of DDM schemeMulticast groups The number of multicast groups created by the DDM scheme used during the simulation


Number of grid cells ¼ UPSD2 � 2

Basically, we conduct the experiments in two stages. In the first stage, we assign a constant number 200 toUPSD which mean we divided the routing space to 80,000 cells, while vary the number of tank objects in eachsimulation run from 100 to 1000. Figs. 6–8 give the experimental results of the first stage simulations.

As shown in Fig. 6, our optimized scheme has lower message overhead than the dynamic approach. This isthe result of the introduction of the light-level-matching in the optimized scheme, which filters out further theunnecessary data to be transmitted on the network. Region-based DDM has the lowest message overhead dueto its exact matching mechanism between publications and subscribe regions, which however, causes highertime overhead. Fig. 7 shows DDM time of the three schemes increase as the number of object grows.Region-based DDM has better time performance when the number of objects is below 400, while dynamicand optimized schemes outperform region-based DDM. The DDM time of the optimized scheme is slightlyhigher than that of dynamic DDM. As to the number of multicast groups, our optimized DDM lies betweenthe dynamic and region-based ones as shown in Fig. 8.

The second stage of experiments is as following: the number of UPSD is changing from 100 to 800 for eachsimulation run, while the number of tank objects is fixed at 200. We are trying to investigate the effect ofchanging number of grid cells on the performance of our optimized DDM. Figs. 9–11 give the experimentalresults of the second stage simulations.

From Figs. 9–11, we notice that varying the number of grid cells do not affect the performance of theregion-based DDM scheme. For optimized and dynamic DDM, two metrics, DDM time and DDM message

Fig. 6. Execution steps of the proposed algorithm.

Fig. 7. The data structure used in our DDM.

1500k

1200k

900k

600k

300k

0 1000 900 800 700 600 500 400 300 200 100

DD

M M

essa

ges

Number of Oblects

RegionDynamic

Optimized

Fig. 8. Number of messages vs. number of objects.


increase in a linear pattern showing that the two DDM schemes scale in terms of number of grid cells. TheMulticast Groups used decrease somewhat in both optimized and dynamic DDM. This behavior is not sur-prising, because the decrease of cell size allows greater accuracy in detecting intersections, meanwhile, fewerintersections detected, and fewer multicast groups created. We can also observe that similar to the first set ofexperiments, the performance of our optimized DDM lies between that of region-based and dynamic schemes(see Figs. 12 and 13).

The experimental results from both the two stages demonstrate that our proposed optimized dynamic grid-based DDM algorithm has lower message overhead compared with the dynamic grid-based DDM approach,while lower time overhead and less multicast groups created when compared with the region-based DDM.This is due to that a light level of exact matching is employed in the optimized algorithm, which makes itachieve certain trade-off between the two DDM schemes, namely region-based and dynamic grid-based DDM.

1000

800

600

400

200

0 1000 900 800 700 600 500 400 300 200 100

DD

M T

ime(

s)

Number of Oblects

RegionDynamic

Optimized

Fig. 9. DDM time vs. number of objects.

Mul

ti-ca

st G

roup

s

Number of Oblects

RegionDynamic

Optimized

1000

800

600

400

200

0 1000 900 800 700 600 500 400 300 200 100

Fig. 10. Multicast groups vs. number of objects.


5.5. Grid-filtered region-based DDM – experimental results

For the experiments, we run the simulator on 4 and 10 linux machines, to measure performance differences,based on the number of participating federates.

While the grid-filtered region-based algorithm (Hybrid) is the one of primary interest, we repeated all exper-iments, using the dynamic grid-based (Dynamic), fixed grid-based (Fixed), and region-based (Region)approaches, for comparison. All the grid-filtered region-based trials use a constant threshold value of 60%,to determine whether a grid match is enough to trigger a subscription or whether the match needs to be con-firmed with region-based matching. This value was selected to help differentiate the grid-filtered region-basedapproach from the region-based in trials with large cells, and from the grid-based approaches when the cellsare small.

3500k

2800k

2100k

1400k

700k

0 800 600 450 350 300 250 200

DD

M M

essa

ges

UPSD

RegionDynamic

Optimized

Fig. 11. DDM time vs. number of objects.

1200

900

600

300

100

0 800 600 450 350 300 250 200

DD

M T

ime(

s)

UPSD

RegionDynamic

Optimized

Fig. 12. DDM time vs. UPSD.


All experiments are of the Tank dogfight scenario type so all objects on all processors are both publishersand subscribers. Half the processors model red objects and half model blue objects.

In our experiments, on the UNT mini-RTI kit, running the tank dogfight scenario, we looked at our grid-filtered region-based DDM method (Hybrid), compared to the region-based (Region), dynamic grid-based(Dynamic), and fixed grid-based (Fixed) DDM methods, when we varied the number of participating feder-ates, the number of objects simulated, the grid size over the terrain and the percentage threshold. Specifically,we looked at the number of messages generated during the simulation time, the number of multicast groupsused and total processing time to see which DDM method produced a reduction in resources used and effi-ciency in processing. The results of our experiments showed favorably toward a reduction in the number ofmessages, number of multicast groups used and more efficient processing time using the grid-filteredregion-based DDM method, compared to the region-based, dynamic grid-based and fixed grid-based DDMmethods. An analysis of the following figures supports this assessment.

Mul

ti-ca

st G

roup

s

UPSD

RegionDynamic

Optimized

100

80

60

40

20

0 800 600 450 350 300 250 200

Fig. 13. Multicast groups vs. UPSD.


First we will discuss the results of our comparison between the number of DDM messages when we vary thenumber of objects and the number of processors. Fig. 5 shows the results of simulation runs with the UPSDset to 60, varying the objects from 80 to 400 and to 1000 and measuring the number of DDM messages gen-erated during the simulation. Reviewing Fig. 5, we see a steady increase in the number of DDM messages gen-erated when the region-based, and grid-based DDM methods are used, while the line representing the grid-filtered region-based DDM method has very close to a slope of zero. Fig. 5 shows similar and slightlyimproved results using ten processors, as we increase the number of objects. These two figures together suggestthat an increase in the number of processors and the implementation of the grid-filtered region-based DDMmethod will improve on the number of DDM messages generated by more accurately filtering data to sub-scribing federates, thereby reducing the number of spurious and unnecessary messages in a distributed simu-lation.

0

50000

100000

150000

200000

250000

300000

0 100 200 300 400 500 600 700 800 900 1000

DD

M M

essa

ges

Objects

DDM Messages vs Objects on Four Processors and 60 UPSD

FixedDynamic

HybridRegion

0

50000

100000

150000

200000

250000

300000

350000

400000

0 100 200 300 400 500 600 700 800 900 1000

DD

M M

essa

ges

Objects

DDM Messages vs Objects on Ten Processors and 60 UPSD

FixedDynamic

HybridRegion


6. Conclusion

Data distribution management plays an important role in lager scale distributed simulations. Therefore, itis desired to provide efficient, scalable and cost effective DDM mechanisms to meet the high performancerequirement.

As an outgrowth of our work in large-scale distributed interactive simulation, we have proposed two var-iant to enhance the performance of DDM for distributed simulations, i.e., an optimized dynamic grid-basedscheme and filtered grid region based technique, in which irrelevant information will be further filtered outusing a sender-side-filter to effectively extend the accuracy of the filter capability of dynamic grid-basedDDM. It saves the network resources and reduces the receiving-side processing load, which optimizes theDDM services further to meet the high performance requirement in large-scale distributed simulation.

The experimental results demonstrate that our optimized dynamic grid-based DDM achieves certain trade-off between the two DDM schemes, namely region-based and dynamic grid-based DDM,due to the use of thelight level of exact matching in our new DDM scheme.

References

[1] IEEE Std 1516-2000, 1516.1-2000, 1516.2-2000, Standard for Modeling and Simulation High Level Architecture (HLA), 2000.[2] HLA Run-Time Infrastructure RTI 1.3-Next Generation Programmer’s Guide Version 5.[3] A. Boukerche, A. Roy, Dynamic grid-based approach to data distribution management, Journal of Parallel and Distributed

Computing 62 (2002) 366–392.[4] A. Boukerche, C. Dzermajko, Alternative approaches for data distribution management in large-scale distributed systems, in:

Proceedings of the International Symposium on Performance Evaluation of Computer and Telecommunication Systems, 2003.[5] R. Fujimoto, T. McLean, K. perumalla, I. Tacic, Design of high performance RTI Software, in: Fourth IEEE International

Workshop on Distributed Simulation and Real-Time Applications (DS-RT’00), San Francisco, California, August 25–27, 2000.[6] R. Fujimoto, Parallel and Distributed Simulation Systems, Wiley, NewYork, 2000.[7] K.L. Morse, M. Petty, Data distribution management migration from DoD 1.3 to IEEE 1516, in: Proceedings of DS-RT, 2001.[8] G. Tan, L. Xu, F. Moradi, Y. Zhang, An agent-based DDM filtering mechanism, in: Proceedings of the Eighth IEEE/ACM

Workshop on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, 2000, pp. 374–381.[9] D.J. Van Hook, S.J. Rak, C. James, Approaches to RTI implementation of HLA data distribution management services, in:

Proceedings of DIS/SIW, 1996, 96-14-084.[10] S.J. Rak, Evaluation of grid-based relevance filtering for multicast group assignment, in: Proceedings of DIS/SIW, 1996, 96-14-106.[11] M. Hyett, R. Wuerfel, Implementation of the data distribution management services in the RTI-NG, in: Simulation Interoperability

Workshop, Spring 2002.[12] S.T. Bachinsky, J.R. Noseworthy, F.J. Hodum, Implementation of the next generation RTI, in: Simulation Interoperability

Workshop, Spring 1999.[13] S.T. Bachinsky, L. Mellon, G. Tarbox, R. Fujimoto, RTI 2.0 Architecture, in: Simulation Interoperability Workshop, Spring, 1998.[14] G. Tan, Y. Zhang, R. Ayani, A hybrid approach to data distribution management, in: Proceedings of IEEE DS-RT, 2000.[15] H. Abram, K. Watsen, M. Zyda, Three-tiered interest management for large-scale virtual environments, in: Proceedings of ACM

Symposium on Virtual Reality Software and Technology, 1998, pp. 125–129.[16] A. Boukerche, K. Lu, Optimized dynamic grid-based DDM protocol for large-scale distributed simulation systems, in: Proceedings of

IPDPS/PMEO, Denver April 2005, pp. 4–8.[17] M.R. Macedomia, M.J. Zyda, D.R. Pratt, D.P. Brutzman, P.T. Barham, Exploiting reality with multicast groups: a network

architecture for large-scale virtual environments, in: Proceedings of the Virtual Reality Annual International Symposium (VRAIS’95),March 11–15, 1995, p. 2.

[18] K.L. Morse, J.S. Steinman, Data distribution management in the HLA: multidimensional regions and physically correct filtering, in:Proceedings of SIW, Spring, 1997.

Documents

An enhancement towards dynamic grid-based DDM protocol for distributed simulation using multiple levels of data filtering