GridSAT: Grid enabled Satellite Architecturefor Reliable Transmissions

Giovanni Aloisiol, Massimo Cafaro', Simone Molendini', Marco Tana2, Franco Tommasi1, Andrea Tricco2IDept. of Innovation Engineering, University of Lecce, Italy

2Innovative Materials and Technologies School - ISUFI, University of Lecce, Italy{giovanni.aloisio, massimo.cafaro, simone.molendini, marco.tana, franco.tommasi, andrea.tricco}@unile.it

Via Monteroni, Lecce, ITALY +390832297310

Abstract - The paper defines an architecture that uses thesatellite as a communication medium for efficient and reliablelarge file distribution across WANs, and so disseminate large filesto multiple receivers in geographically dispersed grids, exploitingthe SRDP protocol.

I. INTRODUCTION

Nowadays it is usual to find situations in which there is theneed to transmit large files from one producer to manyconsumers simultaneously in a grid environment [1]. Wedefine such a situation as Data Intensive Multicasting (DIM)(i.e., the one to many transmission of a huge quantity of data).Examples include distributed computing, remote databaseupdate, coherent management of replicas, software and mirrorsite updates and, images and multimedia content distribution,generally.

In this paper we propose GridSAT as a general purpose(customizable) powerful architecture for reliable data transfersusing satellite connections. GridSAT architecture is a solutionto present Data Grid [2] problems. The challenges arisingfrom the size of data, geographical distances and the numberof entities involved mean that Data Grids can not alwaysguarantee efficient distribution. Moreover, in Gridenvironments, access to distributed data is as important asaccess to distributed computational resources [3]. For thesereasons we have decided to adopt satellite as a transmissionmedia because of its intrinsic characteristics of easyreachability and scalability, particularly in those situations inwhich DIM has to be performed. With regard to reachabilityand scalability, satellites provide broadcast transmissions inthe fullest sense, they can cover large areas of the earth. Thesatellite (which is not as expensive a solution as one mightthink if the potential number of receivers and the coveredgeographic area are taken into account) is an almost idealmedium for wide-scale broadcasting. Today satellites reach anumber of households and businesses far beyond that oftraditional broadcast methods [4].To date, the only feasible solution to the transmission of

files in a grid environment is GridFTP [5,6] but we believethat there is the need for a complementary mechanism focusedon DIM. In the GridFTP context a grid user must performmultiple GridFTP sessions even though the networkingresources are not utilized efficiently. Multicasting via satelliteyields appreciable improvements. In the proposed GridSAT

architecture the Satellite Reliable Distribution Protocol(SRDP) [7] is used to allow many receivers to receive datathrough the satellite. This is a suitable solution and permitsgood scalability in terms ofthe number of receivers.

In Sec. II we present the requisites of grids for reliablemulticast protocols introducing benefits of the use of thesatellite for the grids and the requisites of reliable multicastuseful for this purpose. Sec. III shows an architecture whichintroduces the satellite link within the grid environment. Sec.IV discusses problems of the reliable multicast in general andthen as regards satellite links. Sec. V briefly shows the SRDPprotocol. Sec. VI provides results of the performance of suchan architecture gathered from a simulation. Conclusions,future work and references are also provided.

II. GRIDS AND MULTICAST

This section is aimed at defining the set of grid applicationsrequired for transmission in multicast. In the following wediscuss the most relevant requirements.The first requirement is related to multicast transmission:

i.e. a sender Data Node needs to transmit similar data toseveral receiving Data Nodes, each node requiring the wholeor part of the transmitted data. The applications that ask forthis are often called MISD, Multiple Instruction Single Data.The second requirement is related to the transmission time:

multicast is justified if the application needs an upperbound tothe transmission time. The applications that require such abound are often named real-time applications in generic terms,since this limit is applied to all datagrams during thetransmission instead of to the overall transmission.

Since in the grid environment transmission reliability isrequired, Sec. IV discusses the state of the art of reliablemulticast trasnmission.The classic approach to transmitting the same data to

several nodes is to allow the sender to replicate multiplesessions of a unicast connection to each individual receivingnode, by means of the TCP, for instance GridFTP. Analternative approach is to allow the sender to open a singlesession in multicast to all receivers, to transmit the superset ofdata and, if needed, to allow each receiver tailor its ownportion ofthe data.

Multicast transmission is an attractive alternative to unicasttransmission because the sender need must not wait for thetransmission of the data to all the receivers before ending the

0-7803-9206-X/05/$20.00 ©2005 IEEE

634

session. However, multicast is only economically useful forthe grids when the multiplication of the number of receivers,the percentage of transmission time in respect to the overallcalculation time, the average of each node's file size in respectto the overall superset size and the efficiency of the reliablemulticast protocol in respect to TCP is favorable. This meansthat the use of the multicast yields well only if the number ofreceivers is large enough to justify more network engineeringthan usual and if the scalability of the transmission time isadequate.

Recently, the need to perform multicast in a gridenvironment was pointed out [8, 9]. A multicast TCP approachhas been proposed: TCP-XM is a TCP adaptation in order tosupport multicast transfer by means of a TCP engine run onUDP as a userspace transport protocol. There is also a relatedimplementation of this multicast TCP approach as a GlobusXIO transport driver [10]. In [11] the authors present acomparison of some protocol implementations for reliablemulticast in the Grid. It is shown that the usage ofTCP in one-to-many transfers can become very prohibitive for networkresources, even if only a few receivers are involved.

In fact in grid environments it is very usual to send largefiles. On the other hand, reliable multicast protocols takinginto account (MDP and NORM), show a very good scalabilityin terms of goodput and network overhead. In fact multicastgroup increase does not affect these two parameters.Nevertheless, these reliable multicast protocols are not able toachieve appreciable performances in terms of goodput. This isa severe limitation for Grid applications.

In active networking, routers themselves play an active roleby executing application-dependent services on incomingpackets [12]. Recently, the use of active network conceptswhere routers themselves can contribute to enhancing thenetwork services by customized functions have been proposedin the multicast research community and these could be verybeneficial to the grid community. These active reliablemulticast protocols open new perspectives for achieving highthroughput and low latency on wide-area network by helpingwith feedback implosion problems, retransmission scopingand cache of data. In this project, the benefits a computinggrid can obtain from an underlying active reliable multicastservice are investigated. The Dynamic Replier Active ReliableMulticast protocol [13] for reducing end-to-end latency isproposed. However, these active services introduce additionalprocessing costs that must be carefully evaluated, especiallywith regard to deployment in a large scale grid testbed [14,15].Moreover, several factors currently hinder large-scale

deployment of terrestrial multicast services; it is particularlydifficult to support delivery to large groups of users [16].

Satellites offer a natural way of extending the multicastservice to this large number of users (scalability with regard tothe number of users). They can exhibit high capacity(especially using next generation satellite systems) and alsoeliminate the need for a large number of intermediate routinghops.

Figure 1. The Elements

Bearing in mind all these considerations, we decided tochoose a satellite transport protocol to perform reliablemulticast in Grids. The description of the protocol itself andthe motivations for choosing it are presented in Sec. IV.

III. THE ARCHITECTUREIn this section we discuss the main elements of the

GridSAT architecture, as shown in Fig. 1.At the application layer, GridSAT elements are represented

by the Data Nodes (DNs). For each multicast transmission aDN can be either a source or a sink of file resources.Moreover, unicast transmissions between DNs are allowed.At the network layer, SRDP elements are one Satellite

Sender (SS) which transmits data to the satellite and manySatellite Receivers (SRs) that receive those data. In order totake advantage of the SRDP, it is important to have SS andSRs directly connected to the satellite apparatus. In an idealsituation the path between the SS and a SR is one-hop length,although some intermediate routers between them and thesatellite are allowed at the cost of some reduction in theoverall performance.

Moreover, application layer elements would like to use, ingeneral terms, many-to-many transmissions but a many-to-many efficient reliable multicast protocol does not exist todaysince the complexity of its design would be prohibitive. Ifmore than one contemporary transmission is observed it is theresponsibility of the SS to queue up occurring requests or toassign them a set of lots in the satellite bandwidth.

IV. EXPERIENCE ON SATELLITE RELIABLE MULTICASTTRANSPORT

In this section we focus the attention on the efficiency ofreliable multicast protocols.The attempt by both the Intemet Research and the Internet

Engineering Task Force (IRTF and IETF) working groups todesign reliable multicast transport led to the conclusion thatsuch a general-purpose transport does not exist. This isbecause the problems of multicast are much more difficultthan those of unicast.

In fact, at a simple level, working out the reliability ofseveral receivers in a generic intemet should result in thereduction of resources and capabilities (calculatingtransmission rate and dealing with congestion control, just to

635

mention a few) to the minimum of the resources in the set ofreceivers. Of course, performance of the transmissionexponentially decreases when the number of receiversincreases. Elaborated algorithms alleviate this situation, butthe main issues, especially their scalability, still exist.

The generally accepted design-space document [17]distinguishes between total and semi-reliable transports. TheGrid High Performance Networking Research Group (GHPN-RG) of the Global Grid Forum asserts that most gridapplications need total reliability, that is if any of theapplication data unit is missing, none of the received portionof the application data unit is useful. We introduce a harderdefinition, full reliability, that is if any received portion doesnot obtain total reliability, none of the total transmission isuseful.

For example, for database replication total reliability shouldbe enough while for distributed computing full reliability is arequirement that must be met.

This requirement excludes several protocols that use packet-level Forward Error Correction (FEC) without any otherpacket recovery methods.Today research gets to the bottom of these problems by

relaxing the requirement that the Reliable Multicast Transport(RMT) will use a generic internet. By introducing the satellitelink into the network we will significantly simplify severalproblems.

Even though rain and other atmospheric attenuation presenta severe problem for Ka-band satellites and transmitting viasatellite is still expensive, even if the cost can be spreadbetween the number of receivers and this cost is decreasing,satellite peculiarities which are particularly useful in thecontext of grid applications are:* As a satellite is a natural broadcast media, it is simple to

disseminate in multicast* Many satellites, or constellations of them, cover entire

continents* The multicast topology is a star-based one (the satellite is

the hub), and so a multicast routing protocol is not needed* The receiving apparatus is cheap, the number ofhops from

the satellite to the receiver is short* The capacity is flexible* Today satellite technology introduces on-board processing

and buffering (in the hub of the topology) that greatlyhelps the management at network layer - see for examplefor Hughes SPACEWAY [18]The hierarchy of the topology is simple and congestion

control problems are limited. Moreover, it is simple to managethe network from the satellite service center, and then to applyquality of service and traffic engineering in an easier way thanfor a generic network.

This paper does not discuss or compare those severalsatellite RMT (Sat-RMT) protocols whose taxonomy ispresented in several papers, for example in [19, 20].

However, in [20] it has been shown that one of them, theSatellite Reliable Distribution Protocol (SRDP), allows hybridretransmissions via satellite and terrestrial paths - see Fig. 2.

Packet recovery is an important component of a Sat-RMTprotocol [21]. SRDP trades between satellite and terrestrialsolutions, separating both correlated and uncorrelated errors.We believe that a correct approach must trade betweensolutions related to correlated and uncorrelated losses. In thecase of receivers experiencing correlated losses, missingpackets could be retransmitted via the satellite channel.Otherwise packets could be unicast to single receivers via aterrestrial link. In the former case there is an appreciableimprovement in the transmission efficiency of the satellitebecause its communication channel has a broadcast nature.Retransmission of individual packets triggered by individualrequests over the satellite channel does not make good use ofthe broadcast channel and implies high recovery latency aswell. If applications aim at full reliability and high scalabilityat the same time, multicast transmissions are useful for, letssay, a 95% of file size, while for the rest of data unicastcorrections are more comfortable.

Moreover, the SRDP use of satellite duration time isflexible because the duration of the satellite phase isconfigurable. The reminder of the operations can betransmitted via terrestrial path.

V. THE SRDP

SRDP is designed for the networks where the satellite isused as a means of distributing data to a group of receivers.From this point of view, the SRDP protocol aims attransmitting reliable multicast information in an efficient andscalable way for the star-based topologies that are intrinsic tothe satellite transmissions. SRDP profits from feedbackcoming from receivers provided with an outbound connectionbut it can also be used to transmit data to receivers providedonly with an incoming satellite connection.We do not show in this paper the algorithms at the basis of

the SRDP, already discussed in another paper [7].The SRDP manages its transmissions in three phases, each

one approaching a better level of reliability. We just mentionthe fact that applications are able to select phase 1 only, phase1 and 2, phase 1 and 3 or phase 1, 2 and 3 - see Fig. 2.During phase 1, also called the "Satellite phase", a file is

transmitted through the satellite internet and a reasonabledegree of redundancy is added. SRDP adopts Reed-SolomonFEC codes that use the algebraic properties of finite fields,even if other algorithms, for example Tornado codes [22],should be used since both the syntax and the coding of theprotocol are modular.

_ _SatReduldi c'aSe

Satellite Phase Hybrid Phase

Figure 2. The three phases

e PktGestFslP

Terrestrial Phase

636

During phase 2, also known as the "Hybrid phase", a set ofredundant packets are retransmitted to attempt the recovery ofpackets lost during phase 1. To this purpose, the senderperiodically polls the receivers for feedback about the numberof lost packets (Stat Req and Stat_Rep messages). Duringphase 3, on the "Terrestrial phase", each receiver is allowed torequest all its missing packets by opening a TCP connection tothe Sender. Phase 3 is then aimed at eliminating the remaininglosses. These three phases are separated in time and identifiedby SRDP synchronization messages.SRDP has been designed to be used with the satellite

networks where an important goal is to reduce the waste ofresources. In these environments the time that is dedicated tothe transmission is often limited and it constrains the totalnumber of redundant packets that can be added. The typicalprocess of bandwidth purchase can be outlined as follows. Acertain amount of data packets K must be transmitted. If thechannel bandwidth available is B (packets/s), one has todecide a priori (based on experience and/or on the results ofour simulations) the amount of redundancy packets, H, to addto the data. The time slot to be purchased will then be at least(K+H)/B seconds. For this reason, SRDP assumes theparameter H as fixed.

In order to understand how the SRDP protocol usesredundancy, let H be the number of redundant packets that areavailable to the sender, and K be the number of originalpackets obtained from the file. The choice of the H parameterinfluences the performances of the protocol. If H is large, thereliability increases but the goodput of the protocoldiminishes. If H is small, the protocol has a betterperformance in terms of goodput but the reliability is reduced.

VI. SIMULATIONS

The emphasis of our simulations is to study how thereceivers use the return channel to receive the missing packets.The more the receivers use the terrestrial phase to receivemissing packets, the more the behaviour of the protocolbecomes similar to a unicast connection (for instanceGridFTP). For this purpose, Rcv-Phase3 was defined as thepercentage of receivers that need to use phase 3 in order toreceive the missing packets; and Pkt-Phase3 as the percentageof data packets exchanged through the terrestrial returnchannel during phase 3.

Simulations was based on the ns2 -Network Simulator [23].GEO satellite was used and to the satellite up-link and down-links was assigned a bandwidth of 2 Mbps with 125 ms ofdelay. The transfer of a 100 MB length file from a SRDPSatellite Sender to R Satellite Receivers was examined.During all the simulations the length of the IP-packets wasassumed to be 1500 bytes (the maximum ofan Ethemet LAN).The number of the data packets k in each transmission groupwas equal to 100 and a Vandermonde matrix was used togenerate the redundant packets from the originals.

In the following, a set of simulations are presented toanalyse the behaviour of the protocol in conditions of randomlosses. To simulate packet losses for each of the two sets R+1error distributions were used: R distributions were used to

simulate uncorrelated errors, which occured in the Receivingpath (the subpath from the satellite to the receiving DN); onedistribution was used to simulate correlated errors, whichoccured in the Reaching path (the subpath from the senderDN to the satellite). All the error distributions werecharacterized by a Bit Error Rate b that was assumed to beconstant over the time of a transmission for all links.Admittedly this is a simplification (in general b on the up-linkis different from b on the down-link) but it is needed to controlthe complexity of the simulation work. Moreover, the R+Idistributions were assumed to be mutually independent.Which implied an upper bound on the packet loss probability.A whole packet was assumed to be lost when at least one bitwas corrupted. The packet loss probability P is hencedependent on the packet size S in octets, and was calculated asP= 1 - (1-b)8S (e.g. a b of 10-6 gives a P, packet lossprobability of about 0,012).

Fig. 3 shows the percentage of receivers Rcv-Phase3 as afunction of the percentage of redundant packets H added bythe SRDP protocol during the transmission via satellite. In thefigure different values of packet error rate P are compared.When the percentage of redundant packets H increases, thepercentage of receivers that need phase 3 decreases. Forinstance, with P = 0.1 and H = 10, all receivers must use theterrestrial phase to correctly complete the reception.

Figure 4 shows the percentage of packets Pkt-Phase3 asfunction of the packet error rate P and shows that Pkt-Phase3increases with P.

100 -0000

90

80-.--P=0.0001

70-P=0. 001

60 -+P=0o080 P=0.1

-40

30

20

10

010 15 20

Redundant Packets H (%)

Figure 3. Rcv-Phase3 as function of H, R = 100.

0m

a-

a-

0,0001 0,001 0,01Packet error rate P

Figure 4. Pkt-Phase3 as function of P, R = 100.

0,1

637

VII. CONCLUSIONS AND FUTURE WORK

Efficient scientific exploration and collaboration requiresthe exchange of large amounts of data between scientificlaboratories across the world. Future research will beincreasingly inter-disciplinary in its nature, and thus specificdata sets that are produced by independent labs will need to beshared between a number of distributed sites supportingheterogeneous networks, computing and storage resources.

Grid computing involves a plethora of applications, devices,distributed computing contexts, experimentation tools, datamanipulation software, large-scale database access and dataexploration, VR tools, transmission of continuous mediaflows, use of heterogeneous networks, transmission platforms,and services (archiving, security, caching, etc.). Grid trafficincludes bulk data transfer, high priority access to remotedatabases, grid control traffic, exchange of audio and video forcommunication and visualization purposes, and interactivedata visualization.

In this paper we presented the GridSAT system highlightingour goal, its architecture and the middleware exploited. Wehave gathered experiences from both grid and networkingcommunities.

Future work is multifaceted and related to several topics.We plan to add security mechanisms for confidential datatransmission. We are also investigating the possibility ofenabling DNs to use the Replica Location Service (RLS) ofthe Globus Toolkit [24] for purposes related to replicamanagement in the data grid area. Finally, we want toinvestigate some specific case studies in order to show howthe GridSAT system can be applied to real situations such asremote and large scale database updates that need a secure,reliable, efficient multicast transfer of data across wide areanetworks.

At the moment preliminary tests are on going within theSPACI grid infrastructure [25]. Related experimental results inthe SPACI context (that are out ofthe scope ofthis paper) willbe presented in a future work.

VIII. REFERENCES

[1] I. Foster, C. Kesselman, The Grid: Blueprint for a New ComputingInfrastructure, Morgan Kaufmann, 1998.

[2] A. Chervenak, I. Foster, C. Kesselman, C. Salisbury, S. Tuecke, TheData Grid: Towards an Architecture for the Distributed Management andAnalysis of Large Scientific Datasets. Journal ofNetwork and ComputerApplications, 23:187-200, 2001.

[3] A. Chervenak, I. Foster, C. Kesselman, S. Tuecke, Protocols andservices for distributed data-intensive science. ACAT2000 Proceedings(2000) 161-163.

[4] G. Maral and M. Bousquet, "Satellite Communication Systems", 4ndEdition ed. Toulouse: John Wiley & Sons, 2001.

[5] B. Allcock, Bester, J., Bresnahan, J., Chervenak, A.L., Foster, I.,Kesselman, C., Meder,S., Nefedova, V., Quesnel, D. and Tuecke, S.,Secure, Efficient Data Transport and Replica Management for High-Performance Data-Intensive Computing. In Mass Storage Conference,2001.

[6] GridFTP Protocol. URL: [http://www-fp.mcs.anl.gov/dsl/GridFTP-Protocol-RFC-Draft.pdfl

[7] F. Tommasi, S. Molendini, A. Tricco: "Design of the Satellite ReliableDistribution Protocol (SRDP)", in Proceedings of the IEEE Globecomm2003, 1-5 December 2003, San Francisco, USA.

[8] V. Sander, W. Allcock, P. CongDuc, I. Monga, P. Padala, M. Tana, F.Travostino, J. Crowcroft, M. Gaynor, D. B. Hoang, P. Vicat-BlancPrimet, M. Welzl, Networking Issues for Grid Infrastructure, GlobalGrid Forum Document, November 2004, GFD.37.

[9] K. Jeacle, J. Crowcroft, Reliable high-speed Grid data delivery using IPmulticast, Proceedings ofUK All Hands Meeting 2003, Nottingham, UK

[10] K. Jeacle, J. Crowcroft, Extending Globus to support MulticastTransmission, UK e-Science All Hands Meeting 2004

[11] M. P. Barcellos, M. Nekovee, M. Daw, J. Brooke, S. Olafsson, ReliableMulticast for the Grid: a comparison of protocol implementations, UK e-Science All Hands Meeting 2004

[12] M. Maimour, C. Pham, "An Active Reliable Multicast Framework forthe Grids", Proceedings of the International Conference onComputational Science (ICCS 2002), LNCS 2329&2330&2331, April21-24 2002, Amsterdam, The Nederlands, pp588-597

[13] M. Maimour, C. Pham, "Dynamic Replier Active Reliable Multicast(DyRAM)", Proceedings of 7th IEEE Symposium on Computers andCommunications (ISCC 2002), July 1-4 2002, Taormina, Italy

[14] M. Maimour, J. Mazuy, C. Pham, The Cost of Active Services in ActiveReliable Multicast, Proceedings of IEEE 4th Annual InternationalWorkshop on Active Middleware Services (AMS 2002), July 24-26,2002, Edinburg, UK, pp67-72

[15] T. Wolf, J. Turner, Design Issues for High Performance Active Routers,IEEE Journal of Selected Areas of Communications, 2001, 19(3):404-409.

[16] K. Obraczka, Multicast transport protocols: a survey and taxonomy,IEEE Communications Magazine, 1998, 36(l):94-102

[17] M. Handley, S. Floyd, B. Whetten, R. Kermode, L. Vicisano, M. Luby,The Reliable Multicast Design Space for Bulk Data Transfer, InternetSociety Request for Comments, August 2000, RFC 2887.

[18] E.J. Fitzpatrick, SPACEWAY system summary". SpaceCommunications 1995; 13:7 23.

[19] M. W. Koyabe, G. Fairhurst, Reliable multicast via satellite: acomparison survey and taxonomy, International Journal of SatelliteCommunications, 2001, 19:3-28.

[20] G. Akkor, M. Hadjitheodosiou, J. S. Baras, Transport protocols inmulticast via satellite, International Journal of Satellite Communicationsand Networking, 2004, 22:611-627

[21] Z. Sun, M. P. Howarth, H. Cruickshank, S. Iyengar, L. Claverotte,Networking issues in IP multicast over satellite, International Journal ofsatellite Communications and Networking, 2003, 21:489-507

[22] Luby M, Mitzenmacher M, Shokrollahi A, Spielman D. Effcient erasurecorrecting codes. IEEE Transactions on Information Theory 2001;47(2):569 584.

[23] The Network Simulator - ns-2, http://www.isi.edu/nsnam/ns/[24] The Globus Alliance, URL: [http://www.globus.org/].[25] SPACI, URL: [http://www.spaci.it]

638