7
IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, VOL. 17, NO. 2, MARCH/APRIL 2011 377 Energy-Efficient Design of a Scalable Optical Multiplane Interconnection Architecture Odile Liboiron-Ladouceur, Member, IEEE, Isabella Cerutti, Pier Giorgio Raponi, Nicola Andriolli, and Piero Castoldi, Member, IEEE Abstract—As the power dissipation of data centers challenges their scalability, architectures for interconnecting computers, or servers must simultaneously achieve high throughput at peak uti- lization and power consumption proportional to utilization levels. To achieve this goal, this paper proposes the use of an optical mul- tiplane interconnection network, named space-wavelength (SW) switched architecture, able to route and switch packets between servers (on cards) and between processors within a card (or card ports). SW architecture exploits the space domain to address the destination card and the wavelength domain to address the desti- nation port on a per-packet basis. Scalability and energy efficiency of the considered architecture are quantified and compared to typ- ical single-plane architectures. Not only can the SW multiplane architecture achieve higher throughput by exploiting two switch- ing domains, but its performance is shown to be highly scalable with network utilization. More importantly, higher performance is reached with an energy efficiency superior to single-plane archi- tectures. The excellent energy efficiency is achieved using optical devices with low idle power. Index Terms—Computer network performance, optical inter- connection, optical switch, power consumption, semiconductor op- tical amplifier (SOA). I. INTRODUCTION D ATA centers are experiencing a skyrocketing increase in the amount of traffic to store and process. To face this chal- lenge, the preferred architectural solution is a cluster of com- puters or servers interconnected in parallel [1]. Nevertheless, the challenge imposes serious issues concerning the scalability of the interconnections and the power drainage behind handling tremendous amount of traffic. Indeed, not only are the costs con- siderable for supplying megawatts of power to data centers and cooling them down, but the associated greenhouse gas emission is a threatening concern [2]. Similar issues are faced by inter- connection networks used by high-end multiprocessor systems, Internet routers, server blades, and on-chip networks. While energy efficiency has been one of the main design re- quirements in battery-run mobile computers, the nonportable Manuscript received March 1, 2010; revised April 21, 2010; accepted April 28, 2010. Date of publication June 20, 2010; date of current version April 6, 2011. O. Liboiron-Ladouceur is with the Department of Electrical and Com- puter Engineering, McGill University, Montreal H3W 2M1, Canada (e-mail: [email protected]). I. Cerutti, P. G. Raponi, N. Andriolli, and P. Castoldi are with the Scuola Supe- riore Sant’Anna, Pisa 56124, Italy (e-mail: [email protected]; [email protected]; [email protected]; [email protected]). Digital Object Identifier 10.1109/JSTQE.2010.2049733 devices and their interconnection have been exempted from this requirement so far. Indeed, the ultimate goal of interconnection networks has always been low latency and high throughput. This requirement has led to the design of electronic interconnection networks with the switch fabric consuming substantial power, up to 10% of the overall network power at peak utilization [3]. The switch fabric alone exhibits poor energy efficiency, draining more than 50% of its total power even when the interconnection link is not utilized or idle [4], [5]. Hence, electronic intercon- nection networks drain a considerable amount of power at peak utilization, while large amount of power is wasted at low utiliza- tion level. A typical operating range for data centers is between 10% and 50% of their maximum utilization levels [1]. While optical transmissions undoubtedly demonstrated the capability to handle tremendous amount of data traffic [6], it is now nec- essary to design scalable optical interconnection architectures able to achieve both a high throughput at peak utilization and a low-power consumption proportional to the interconnection utilization levels [4]. Architectures for optical interconnections can be realized by exploiting space, time, or wavelength domain to route (or switch) data between interconnected processors or servers. Single-plane interconnection architectures use one of the three domains, whereas recently proposed multiplane architectures exploit two switching domains (i.e., space and wavelengths) to overcome the throughput limitations of a single domain [7]. By exploiting two switching domains, a greater flexibility is achieved as it becomes possible to transfer data packet from any input port to any output port, as both cards and ports are inter- connected. The challenge remains in ensuring practical power dissipation at peak network utilization while keeping it propor- tional to network utilization. This new paradigm requires novel design of multiplane network interconnection architectures that exploits the idle mode of optical devices to reduce power con- sumption. One such device is the semiconductor optical ampli- fier (SOA). When not routing data, the SOA in idle mode drains low power, enabling energy-efficient interconnection networks with power dissipation proportional to the computing load of the data centers. In this paper, the space-wavelength (SW) switching is intro- duced as an energy-efficient multiplane optical interconnection architecture for data centers. The proposed multiplane architec- ture is presented in Section III and compared to state-of-the-art single-plane optical architectures discussed in Section II. Tech- nological implementation and scheduling issues are addressed to evaluate the scalability of the architectures. In Sections IV and V, an analysis of the delay experienced by the packets 1077-260X/$26.00 © 2010 IEEE

Energy-Efficient Design of a Scalable Optical Multiplane Interconnection Architecture

  • Upload
    p

  • View
    212

  • Download
    0

Embed Size (px)

Citation preview

IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, VOL. 17, NO. 2, MARCH/APRIL 2011 377

Energy-Efficient Design of a Scalable OpticalMultiplane Interconnection Architecture

Odile Liboiron-Ladouceur, Member, IEEE, Isabella Cerutti, Pier Giorgio Raponi, Nicola Andriolli,and Piero Castoldi, Member, IEEE

Abstract—As the power dissipation of data centers challengestheir scalability, architectures for interconnecting computers, orservers must simultaneously achieve high throughput at peak uti-lization and power consumption proportional to utilization levels.To achieve this goal, this paper proposes the use of an optical mul-tiplane interconnection network, named space-wavelength (SW)switched architecture, able to route and switch packets betweenservers (on cards) and between processors within a card (or cardports). SW architecture exploits the space domain to address thedestination card and the wavelength domain to address the desti-nation port on a per-packet basis. Scalability and energy efficiencyof the considered architecture are quantified and compared to typ-ical single-plane architectures. Not only can the SW multiplanearchitecture achieve higher throughput by exploiting two switch-ing domains, but its performance is shown to be highly scalablewith network utilization. More importantly, higher performanceis reached with an energy efficiency superior to single-plane archi-tectures. The excellent energy efficiency is achieved using opticaldevices with low idle power.

Index Terms—Computer network performance, optical inter-connection, optical switch, power consumption, semiconductor op-tical amplifier (SOA).

I. INTRODUCTION

DATA centers are experiencing a skyrocketing increase inthe amount of traffic to store and process. To face this chal-

lenge, the preferred architectural solution is a cluster of com-puters or servers interconnected in parallel [1]. Nevertheless,the challenge imposes serious issues concerning the scalabilityof the interconnections and the power drainage behind handlingtremendous amount of traffic. Indeed, not only are the costs con-siderable for supplying megawatts of power to data centers andcooling them down, but the associated greenhouse gas emissionis a threatening concern [2]. Similar issues are faced by inter-connection networks used by high-end multiprocessor systems,Internet routers, server blades, and on-chip networks.

While energy efficiency has been one of the main design re-quirements in battery-run mobile computers, the nonportable

Manuscript received March 1, 2010; revised April 21, 2010; accepted April28, 2010. Date of publication June 20, 2010; date of current version April 6,2011.

O. Liboiron-Ladouceur is with the Department of Electrical and Com-puter Engineering, McGill University, Montreal H3W 2M1, Canada (e-mail:[email protected]).

I. Cerutti, P. G. Raponi, N. Andriolli, and P. Castoldi are with the Scuola Supe-riore Sant’Anna, Pisa 56124, Italy (e-mail: [email protected]; [email protected];[email protected]; [email protected]).

Digital Object Identifier 10.1109/JSTQE.2010.2049733

devices and their interconnection have been exempted from thisrequirement so far. Indeed, the ultimate goal of interconnectionnetworks has always been low latency and high throughput. Thisrequirement has led to the design of electronic interconnectionnetworks with the switch fabric consuming substantial power,up to 10% of the overall network power at peak utilization [3].The switch fabric alone exhibits poor energy efficiency, drainingmore than 50% of its total power even when the interconnectionlink is not utilized or idle [4], [5]. Hence, electronic intercon-nection networks drain a considerable amount of power at peakutilization, while large amount of power is wasted at low utiliza-tion level. A typical operating range for data centers is between10% and 50% of their maximum utilization levels [1]. Whileoptical transmissions undoubtedly demonstrated the capabilityto handle tremendous amount of data traffic [6], it is now nec-essary to design scalable optical interconnection architecturesable to achieve both a high throughput at peak utilization anda low-power consumption proportional to the interconnectionutilization levels [4].

Architectures for optical interconnections can be realizedby exploiting space, time, or wavelength domain to route (orswitch) data between interconnected processors or servers.Single-plane interconnection architectures use one of the threedomains, whereas recently proposed multiplane architecturesexploit two switching domains (i.e., space and wavelengths) toovercome the throughput limitations of a single domain [7].By exploiting two switching domains, a greater flexibility isachieved as it becomes possible to transfer data packet from anyinput port to any output port, as both cards and ports are inter-connected. The challenge remains in ensuring practical powerdissipation at peak network utilization while keeping it propor-tional to network utilization. This new paradigm requires noveldesign of multiplane network interconnection architectures thatexploits the idle mode of optical devices to reduce power con-sumption. One such device is the semiconductor optical ampli-fier (SOA). When not routing data, the SOA in idle mode drainslow power, enabling energy-efficient interconnection networkswith power dissipation proportional to the computing load ofthe data centers.

In this paper, the space-wavelength (SW) switching is intro-duced as an energy-efficient multiplane optical interconnectionarchitecture for data centers. The proposed multiplane architec-ture is presented in Section III and compared to state-of-the-artsingle-plane optical architectures discussed in Section II. Tech-nological implementation and scheduling issues are addressedto evaluate the scalability of the architectures. In Sections IVand V, an analysis of the delay experienced by the packets

1077-260X/$26.00 © 2010 IEEE

378 IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, VOL. 17, NO. 2, MARCH/APRIL 2011

Fig. 1. Switching domains in interconnection architectures.

waiting to be switched and the power consumption is carriedout to evaluate the scalability and the energy efficiency of theproposed optical multiplane interconnection architecture.

II. OPTICAL SWITCHING DOMAINS

The throughput of an interconnection network is achieved byexploiting different domains to carry the data. Each connectedport can communicate with each other through an allocated timeinterval to switch data packets (time domain), on a designatedlink (space), or using a specific wavelength (bandwidth). Thethree domains are illustrated as a three-axis graph in Fig. 1 anddiscussed in this section.

A. Bandwidth Domain

To exploit the parallelism of the optical domain, wavelengthsin the C-band (1530–1570 nm) can be used for the optical inter-connection (see Fig. 1). The wavelength spacing determines thenumber of channels that can be used to encode the data, whichis also strongly dependent on the modulation format and datarate, as well as the wavelength shifting tolerance at the physicallayer. In order to keep the pace with the development of fastercomponents for Petaflop computer clusters [8], and technol-ogy development of 100 Gb/s components [9], it is reasonableto assume that the line rate will enable optical transmissionsat 100 Gb/s. With component integration advances, the heatdissipated by the electronics will reach the optical devices lead-ing to wavelength drifts. Hence, a wider wavelength spacingis required [10]. Spacing can be shrunk when using a complexthermal regulation, which ends up draining more power. Whenbalancing these adverse effects for an energy-efficient switch-ing, the optimal spacing is found to be 3.2 nm. With such spac-ing, the C-band can accommodate 12 channels, which representa realistic technological limit of the wavelength domain, as rep-resented by the z-axis in Fig. 1.

B. Space Domain

Space switching exploits the ability to broadcast an opticalsignal channel to multiple destinations. This dimension is ex-ploited in broadcast-and-select type of switches, where a space-switch is used to select the appropriate destination [11], [12].

The optical power losses suffered in the optical switch can becompensated through amplification. SOAs are appropriate de-vices to use as they offer a broad gain bandwidth, while ex-hibiting fast-switching time. The scalability limitation of thespace domain is set by the minimum optical signal-to-noise ra-tio (OSNR) required by the photodetector for a specified datarate and modulation format. The space domain and its limitationboundary is represented by the x-axis in Fig. 1.

C. Time Domain

Time is an additional dimension that can be exploited withoutany theoretical physical limitation, but at the cost of increasedlatency. This dimension is represented by the y-axis in Fig. 1.One way to mitigate the delay increase is to “speed up” thebit rate. In [13], the optical transmission data rate is increasedwith respect to the line rate of the incoming data. However, theincrease of the line rate of the I/O off-chip processor interfacerequires the use of complex modulation formats, making theimplementation more difficult.

An alternative solution consists in simultaneously exploitingthe bandwidth domain using a wavelength-striped approach,where serial packets are mapped onto multiple parallel channels(wavelengths) [14], [15]. By doing so, the transmission time ofmultiwavelength packets is reduced, without requiring a “speedup” of the transmission bit rate. In this solution, time domainsuffers from the same limitations as of the wavelength domain.Therefore, in this paper, switching using the time domain is notconsidered.

D. Single-Plane Architectures: State of the Art

Single-plane architectures that exploit either the wavelengthdomain or the space domain are surveyed. In the single-planearchitectures, N input ports are connected to N output ports.The architectures are assumed to operate synchronously andcan be reconfigured at each time slot, according to the schedulerdecision. Thus, at each time slot, packets buffered at the inputqueues (IQ) can be transmitted and switched from any inputport to any output port.

1) Wavelength-Switched Architecture: Wavelength-switched(W) architecture exploits wavelength division multiplexing(WDM). The wavelength switching is achieved by transmittingthe packets at the input ports on different wavelengths, selectedaccording to the desired destination port. It can be implementedas follows. Each input port requires one array of fixed lasers (orone fast tunable laser). An arrayed waveguide grating (AWG)combines the modulated signals coming from the N ports, anda second AWG demultiplexes the signal to N output ports. Themaximum number of ports that can be supported is limited bythe bandwidth domain [16].

2) Space-Switched Architecture: The space-switched (S) ar-chitecture is based on a space-switch permitting multiple packetsto be sent along disjoint physical routes to avoid contentions.It can be implemented as follows. Each input port requires onefixed laser. The modulated signals are sent to the output portsthrough a nonblocking N × N optical switch based on SOA,as proposed in [11]. A subset (1 × N ) of the optical switch is

LIBOIRON-LADOUCEUR et al.: ENERGY-EFFICIENT DESIGN OF A SCALABLE OPTICAL MULTIPLANE INTERCONNECTION ARCHITECTURE 379

Fig. 2. Multiplane SW.

shown in Fig. 3, where an SOA is gated for each output. Thespace switch can be scaled by increasing the binary tree structurewith amplification stages to maintain the optical signal powerafter a cascade of five power splitters. The space switch allowsthe synchronous switching of all the signals coming from thecards, by properly gating the SOAs. The maximum number ofports N that can be supported is given by the OSNR degrada-tion experienced by the optical signal after passing through oneSOA for up to N = 32, or through two SOAs for larger spaceswitches up to N = 1024.

III. MULTIPLANE SW SWITCHED ARCHITECTURE

By exploiting both the wavelength domain and the spacedomain, as shown in Fig. 1, a multiplane architecture can berealized. Fig. 2 shows the space-wavelength (SW) switched ar-chitecture, which is composed of up to M independent cardswith N input ports and N output ports each. The connectivityfrom any input ports to any output ports is achieved through theswitch fabric. The architecture operates synchronously and canbe reconfigured in both domains at each time slot.

As shown in Fig. 2, each card is equipped with an ar-ray of N fixed lasers, each one transmitting on a differentwavelength. Each laser is connected to a modulator (E/O),and then, each modulated wavelength is directed to a 1 ×M optical space switch (see Fig. 3) controlled by the inter-card scheduler. On each card, an intracard scheduler connectsthe input ports to the modulators operating on distinct wave-lengths. Each output port on each card is equipped with afixed receiver, tuned on a distinct wavelength. Each receiver(O/E) is preceded by a M :1 coupler (see Fig. 4) that is con-nected to all the 1 × M space-switches operating on thesame wavelength, i.e., the wavelengths outgoing from the first1 ×M space-switch of each card (switching the wavelength λ1)are connected to the first M :1 coupler of each card (collectingall outputs on λ1), etc.

Fig. 3. SOA-based optical space switch structure with a gating SOA at eachoutput controlled by the intercard scheduler.

Fig. 4. Scalable coupler structure with one or two amplification stages de-pending on the size.

To switch a packet from an input port to its output port,the destination card is selected by setting the 1 × M space-switch, while its destination port on the card is selected byselecting the wavelength to be modulated. It is worth notingthat each wavelength uniquely identifies an output port on acard, since receivers are fixed. Multiple packets can be switchedfrom different input ports to different output ports within thesame time slot, by properly scheduling the transmissions. Theintracard schedulers and the intercard scheduler are describedin the following section.

A. Intracard and Intercard Schedulers

The intracard and the intercard schedulers are in charge ofdeciding—at each time slot—the packets to switch among thosewaiting in the IQ, while ensuring that the architectural con-straints are satisfied. In particular, for each time slot each input(output) port can transfer (receive) packets to (from) at most oneoutput (input) port, and packets departing from different portson the same input card must be assigned different wavelengths.The latter constraint is due to the switching domain and thelimited connectivity in the switch fabric. As illustrated in Fig. 5,the scheduling problem consists in selecting a packet from eachinput port and placing it into a matrix representing the card and

380 IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, VOL. 17, NO. 2, MARCH/APRIL 2011

Fig. 5. Scheduling in multiplane architecture.

port domains, so that no more than one packet coming from thesame card is present in each row.

The intracard and the intercard schedulers address thescheduling problem using the two-step scheduling algorithm [7],as follows. The intracard schedulers run independently on eachcard, based only on the status of the IQ at each input port(no global or intercard information is required). Each intracardscheduler assigns a distinct wavelength (i.e., a distinct row inthe matrix of Fig. 5) to each input port. For the assignment, Ncounters are stored for each input port, i.e., one counter for eachwavelength, recording the total number of packets waiting tobe transmitted on the corresponding wavelength. Then, the N 2

counters present on the card (N for each input port) are sorted.The intracard scheduler selects the largest one and connects theassociated input port to the corresponding wavelength, and soon until all the N wavelengths have been assigned to differentinput ports.

Based on the decisions taken by the intracard scheduler, theintercard scheduler matches the output cards to each input port,i.e., decides the column in the matrix of Fig. 5, for each inputport. This step can be solved by running a matching algorithm,such as iSLIP [17].

IV. PERFORMANCE ANALYSIS

Performance of the intra- and intercard schedulers for multi-plane architecture is evaluated and compared to single-plane per-formance. Note that in single-plane architectures, the schedulingproblem is reduced to a single dimension and can be solved bythe intercard scheduler alone.

The maximum number of packets that can be switched in atime slot, or maximum throughput, of multiplane architectureis M × N, and is M times higher than single-plane throughput(i.e., N ). This improvement is achieved due to the possibility ofexploiting two switching domains. By doing so, it is possible toovercome the domain limitations discussed in Section II (i.e.,N ≤ 12 for W architecture or N limited by the physical layerperformance for S architecture) and to considerably scale themaximum throughput.

Fig. 6. Queuing delay versus network utilization.

The average queuing delay, i.e., average number of time slotswaited by the packets in the IQ, is quantified for single- and mul-tiplane architectures through simulations. In the simulations, thenumber of iterations of iSLIP algorithm is set to 3. Simulationsare run until the confidence interval of the average delay is be-low 5% at 95% confidence level. The packet input ports areuniformly selected. A bursty statistical model is used to charac-terize the process of packets generated in data centers [18].Bursty packet arrivals are modeled as an ON–OFF Markovchain: packets arrive at input ports during the geometricallydistributed ON periods, whose average duration is 32 time slots.The destination port is the same for all packets belonging to thesame burst.

The queuing delay experienced by SW architecture is evalu-ated for different architecture sizes (i.e., M ) with N = 12, andis compared against the one experienced by single-plane archi-tecture supporting the same maximum throughput. In Fig. 6, thenumber of cards in the SW architecture is M = 32, 64, 1024. Thenumber of ports in single-plane architectures is shown for N =384 and 768 for a throughput corresponding to the 32 × 12 and64 × 12 SW architectures, respectively. Larger single-plane ar-chitectures are not practical from the physical layer perspective.The curves of queuing delay are shown in Fig. 6 for increasingrates of packet arrivals. The simulation results for the single-plane architecture are very similar and overlap, as well as thesimulation results for the three multiplane implementation sizes.This is due to the fact that, when the switch fabric is sufficientlylarge, the scheduling algorithm behavior is not affected by thearchitecture size [17]. When the average probability of a packetarrival in a time slot at a given port is below 90%, no packetsare dropped from the IQ, and thus, the achieved throughput is100% of the arrival rate. The average probability of a packet ar-rival in a time slot is equivalent to the time-average probabilitythat a packet is switched by the network architecture, or whatis referred to as network utilization. As a result, performance isplotted as a function of network utilization.

In Fig. 6, lower queuing delays are experienced by the mul-tiplane architectures with respect to single-plane architectures.The queuing delay is reduced by more than 40% for any network

LIBOIRON-LADOUCEUR et al.: ENERGY-EFFICIENT DESIGN OF A SCALABLE OPTICAL MULTIPLANE INTERCONNECTION ARCHITECTURE 381

TABLE IPOWER DISSIPATION OF THE COMPONENTS

utilization. These results indicate the superior performance andthe excellent scalability of multiplane architecture.

V. POWER DISSIPATION AND ENERGY EFFICIENCY ANALYSIS

The power dissipation of the SW architecture is quantified andcompared to single-plane space-switched architectures based onrecent advances in photonic integration. To reduce the overallpower dissipation, optical components with large optical band-width, such as couplers, modulators, receivers, and SOAs areused. The envisioned data rate requires external modulation of acontinuous wavelength (CW) source, which heavily contributesto the overall power dissipation. The lasers are included in thepower dissipation calculation.

For energy efficiency, the optical switch architecture ex-ploits the idle mode of SOA. The SOA has been shown tobe an optimum switching element in an optical interconnec-tion due to its broad bandwidth, fast-switching time, and re-duced manufacturing cost due to integration [19]–[21]. In theproposed multiplane architecture, the SOA is in idle modewhen no packet is switched. When a packet is being switched,the SOA is either enabled by the packet itself or by the in-tercard scheduler. In the 1 × M space switch (see Fig. 2),the first SOA in the routing path is enabled by monitoringthe optical power at its input to enable the SOA, whereas thegating SOA is enabled by the intercard scheduler. In theM :1 coupler (see Fig. 3), the SOA is enabled by the switchedpacket. This design implementation exploits the idle mode ofthe SOA to achieve better energy efficiency.

To calculate the power dissipation of the multiplane archi-tecture, the scalability of the physical layer is first determined.Simulation results confirm that the space switch and couplerstructures shown in Figs. 3 and 4 can scale up to 1024 ports(bit error rate (BER) < 10−9). The physical layer simulationis performed based on SOA specifications reported in [19]. Be-yond 1024 ports, additional amplification stages using SOAs arerequired. However, the amplified spontaneous emission (ASE)noise accumulates, which results in a reduction of the OSNRof the signal. The relative comparison of the single-plane andmultiplane architectures is then carried using reported powerdissipation from recent device implementations as stated inTable I.

As mentioned in Section IV, network utilization is a measureof the time-averaged number of packets switched during a timeslot, which is equal to the time-averaged number of packetsarriving at input buffers in a time slot, referred as network uti-lization U (0 < U < 1). The total power dissipation P of the

Fig. 7. Power consumption per bits per second versus network utilization.

Fig. 8. Normalized power (dashed lines) and energy efficiency (solid lines)versus network utilization of the multiplane (blue lines) and the single-plane(gray lines) network architectures.

space switch is a function of the network utilization

P = USPON + (1 − U) SPIDLE +(

M

25 − 1)

PIDLE

+ (M − 1) PIDLE (1)

where S is the number of amplification stages. One stage isused for a 1 × 32 space switch and two stages are requiredfor larger space switches. The SOA compensates any 15 dBloss experienced by the packet after a cascade of five powersplitters. In (1), the first two terms of the equation correspond tothe time-average power associated with the SOAs in the routingpath. The last two terms take into account the remaining SOAsin idle mode. The coupler exhibits a similar time-average powerdissipation relationship, with the exception of the last term,which is simply nonexistent.

In Fig. 7, the total power is normalized to the throughputto compare the single-plane (S) and multiplane (SW) architec-tures. While the power increases with network size, it drasticallydrops by 77% to 86% at high network utilization levels for allnetwork sizes. Moreover, the SW architecture exhibits lower

382 IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, VOL. 17, NO. 2, MARCH/APRIL 2011

normalized power dissipation with respect to the single-planefor the same total number of ports. At 50% network utilization,the normalized powers of the 32 × 12 and 768 × 1 multiplanearchitecture implementations are approximately 30% and 40%,respectively, of the normalized power of their single-plane im-plementations with the same maximum throughput (384 × 1and 768 × 1, respectively). The largest multiplane architecture(1024 × 12) has been included even if it cannot be comparedwith a realistic single-plane implementation, to demonstrate thescalability of the proposed architecture. Indeed, the normalizedpower dissipation is only slightly higher than a single-plane net-work implementation (768 × 1), while achieving a maximumthroughput that is 16 times greater.

In Fig. 8, the power dissipation of the switch fabric is analyzedwith an emphasis on the energy efficiency of the space switchand coupler. The power dissipation is normalized to the peakpower. The first observation that can be made is that the normal-ized power increases with the network utilization (dashed lines).The multiplane architecture exhibits a normalized power that ismore proportional to the network utilization compared to thesingle-plane architecture with approximately 50% of the peakpower when the network is not being used. This is also illustratedthrough the energy efficiency, which is calculated by dividingthe network utilization by the normalized power, a metric pro-posed in [1] and used here to demonstrate the power efficiencyof scalable optical interconnection architectures (solid lines).An energy efficiency of 100% means that the network utiliza-tion (as a percentage of its maximum) is the same as the powerdissipation (as a percentage of the peak power). The energy ef-ficiency of the 32 × 12 multiplane architecture is superior tothe 384 × 1 single-plane architecture, since it is closer to 100%.The higher efficiency is partly due to the fact that multi plane re-quires less optical components to achieve the same throughput.For the same reasons, the energy efficiency decreases for largermultiplane architectures. Although the multiplane architectureis scalable up to 1024 × 12 from the point of view of the physi-cal layer as well as its latency performance, improvement in theidle power dissipation of optical devices would highly benefitthe energy efficiency of larger interconnection networks.

VI. CONCLUSION

The paper presented a multiplane optical interconnection ar-chitecture based on SW switching. The architecture intercon-nects multiple cards each one having multiple ports, and per-mits flexible switching of packets from any port to any port.An intracard scheduler combined with an intercard schedulerwas considered for a scalable scheduling of packet switching.An implementation based on recent advancements in photonicintegration was considered for achieving high energy efficiency.Moreover switches and couplers are realized using SOAs con-trolled by either the scheduler or the packet. The control permitsto perform the space-switching operations as well as to turn offSOAs when unused, in order to save power.

Scalability and energy efficiency were evaluated and com-pared to those of single-plane architectures. Multiplane archi-tecture offers a higher throughput by exploiting two switching

domains. Also, the delay experienced by packets is reduced by40% and does not increase with the number of cards. Simul-taneously, the performance improvements are achieved usinglower power dissipation compared to single-plane architectures.Moreover, the power per bits drops by more than 77% at highnetwork utilization. Overall, the energy efficiency of its corre-sponding switch fabric is superior at any network utilization,due to the use of optical devices (such as SOA) used in idlemode. The achievements of a high throughput at peak utiliza-tion and power dissipation proportional to network utilizationare, thus, making the multiplane architectures suitable towardmore energy-efficient data centers.

REFERENCES

[1] L. A. Barroso and U. Holzle, “The case of energy-proportional comput-ing,” IEEE Comput., vol. 40, no. 12, pp. 33–37, Dec. 2007.

[2] Intergovernmental Panel on Climate Change (IPCC), “Climate change2007: Synthesis report,” IPCC, Geneva, Switzerland, 2007, Tech. Rep.,2007.

[3] S. Aleksic, “Analysis of power consumption in future high-capacity net-work nodes,” IEEE/OSA J. Opt. Commun. Netw., vol. 1, no. 3, pp. 245–258, Aug. 2009.

[4] V. Soteriou and L.-S. Peh, “Exploring the design space of self-regulatingpower-aware on/off interconnection networks,” IEEE Trans. Parallel Dis-trib. Syst., vol. 18, no. 3, pp. 393–408, Mar. 2007.

[5] J. Luo, N. K. Jha, and L.-S. Peh, “Simultaneous dynamic voltage scalingof processors and communication links in real-time distributed embeddedsystems,” IEEE Trans. Very Large Scale Integr. (VLSI) Syst., vol. 15,no. 4, pp. 427–437, Apr. 2007.

[6] A. H. Gnauck, G. Charlet, P. Tran, P. J. Winzer, C. R. Doerr, J. C. Centanni,E. C. Burrows, T. Kawanishi, T. Sakamoto, and K. Higuma, “25.6-Tb/sWDM transmission of polarization-multiplexed RZ-DQPSK signals,” J.Lightw. Technol., vol. 26, no. 1, pp. 79–84, Jan. 2008.

[7] P. G. Raponi, N. Andriolli, P. Castoldi, and A. Bianchi, “Complexityanalysis and performance evaluation of a two-step scheduler for modularoptical packet switches,” in Proc. BROADNETS 2009, Sep., pp. 1–7.

[8] K. J. Barker, K. Davis, A. Hoisi, D. J. Kerbyson, M. Lang, S. Pakin, andJ. S. Sancho, “Entering the Petaflop era: The architecture and performanceof roadrunner,” in Proc. ACM/IEEE Conf. Supercomput., 2008, pp. 1–11.

[9] J. H. Sinsky, “Integration and packaging of devices for 100-Gb/s trans-mission,” in OFC/NFOEC 2009 Tech. Dig., Mar., pp. 1–3.

[10] F. Xia, M. Rooks, L. Sekaric, and Y. Vlasov, “Ultra-compact high orderring resonator filters using submicron silicon photonic wires for on-chipoptical interconnects,” Opt. Expr., vol. 15, no. 19, pp. 11934–11941, Sep.2007.

[11] T. Lin, K. A. Williams, R. V. Penty, I. H. White, and M. Glick, “Capacityscaling in a multihost wavelength-striped SOA-based switch fabric,” J.Lightw. Technol., vol. 25, no. 3, pp. 655–663, Mar. 2007.

[12] R. Hemenway, R. R. Grzybowski, C. Minkenberg, and R. Luijten,“Optical-packet-switched interconnect for supercomputer applications,”OSA J. Opt. Netw., vol. 3, no. 12, pp. 900–913, Dec. 2004.

[13] I. Elhanany and D. Sadot, “DISA: A robust scheduling algorithm forscalable crosspoint-based switch fabrics,” IEEE J. Sel. Areas Commun.,vol. 21, no. 4, pp. 535–545, May 2003.

[14] O. Liboiron-Ladouceur, H. Wang, A. S. Garg, and K. Bergman, “Low-power, transparent optical network interface for high bandwidth off-chipinterconnects,” Opt. Expr., vol. 17, no. 8, pp. 6550–6561, Apr. 2009.

[15] M. Glick, “Optical interconnects in next generation data centers; An endto end view,” in Proc. IEEE HOTI 2008, Aug., pp. 178–181.

[16] R. Gaudino, G. A. G. Castillo, F. Neri, and J. M. Finochietto, “Can simpleoptical switching fabrics scale to terabit per second switch capacities?,”IEEE/OSA J. Opt. Commun. Netw., vol. 1, no. 3, pp. B56–B69, Aug. 2009.

[17] N. McKeown, “The iSLIP scheduling algorithm for input-queuedswitches,” IEEE/ACM Trans. Netw., vol. 7, no. 2, pp. 188–201, Apr.1999.

[18] T. Benson, A. Anand, A. Akella, and M. Zhang, “Understanding datacenter traffic characteristics,” in Proc. ACM WREN, pp. 65–72, Aug.2009.

[19] S. Tanaka, S.-H. Jeong, S. Yamazaki, A. Uetake, S. Tomabechi, M. Ekawa,and K. Morito, “Monolithically integrated 8:1 SOA gate switch with large

LIBOIRON-LADOUCEUR et al.: ENERGY-EFFICIENT DESIGN OF A SCALABLE OPTICAL MULTIPLANE INTERCONNECTION ARCHITECTURE 383

extinction ratio and wide input power dynamic range,” IEEE J. QuantumElectron., vol. 45, no. 9, pp. 1155–1162, Sep. 2009.

[20] S. Yoshida, Y. Kai, G. Nakagawa, K. Sone, Y. Aoki, and S. Kinoshita,“High-speed SOA gate switch driver with equal-length wiring and peakingtechniques,” in Proc. OECC/ACOFT 2008, Jul., pp. 1–2.

[21] N. Sahri, D. Prieto, S. Silvestre, D. Keller, F. Pommereau, M. Renaud,O. Rofidal, A. Dupas, F. Dorgeuille, and D. Chiaroni, “A highly integrated32-SOA gates optoelectronic module suitable for IP multi-terabit opticalpacket routers,” in Proc. OFC 2001 Tech. Dig., PD32, 2001, pp. PD32-1–PD32-3.

[22] M. Chacinski, U. Westergren, B. Stoltz, L. Thylen, R. Schatz, andS. Hammerfeldt, “Monolithically integrated 100 GHz DFB-TWEAM,”J. Lightw. Technol., vol. 27, no. 16, pp. 3410–3415, Aug. 2009.

[23] Y. Baeyens, N. Weimann, V. Houtsma, J. Weiner, Y. Yang, J. Frackoviak,P. Roux, A. Tate, and Y. K. Chen, “Submicron InP D-HBT single-stagedistributed amplifier with 17 dB gain and over 110 GHz bandwidth,” inProc. Microw. Symp. Digest, IEEE MTT-S Int., Jun. 11–16, 2006, pp. 818–821.

[24] (2010). [Online]. Available: www.u2t.de, 100 GHz PhotodetectorXPDV4120R

[25] V. E. Houtsma, N. G. Weimann, A. Tate, J. Frackoviak, and Y. K. Chen,“InP single-ended transimpedance amplifier with 92-GHz transimpedancebandwidth,” in Proc. Compound Semicond. Integr. Circuit Symp., Oct. 14–17, 2007, pp. 1–4.

[26] S. Shahramian, A. Chan Carusone, P. Schvan, and S. P. Voinigescu, “An81 Gb/s, 1.2 V TIALA-Retimer in 65 nm CMOS process,” in Proc. Com-pound Semicond. IC Symp., Oct. 2008, pp. 1–4.

Odile Liboiron-Ladouceur (M’95) received theB.Eng. degree in electrical engineering from McGillUniversity, Montreal, QC, Canada, in 1995, and theM.S. and Ph.D. degrees in electrical engineering fromColumbia University, New York, in 2003 and 2007,respectively. Her doctoral research work focused onthe physical layer of optical interconnection networksfor high-performance computing. She was recipientof a Postdoctoral Fellowship from the Natural Sci-ences and Engineering Research Council of Canadain 2007.

She is currently an Assistant Professor in the Electrical and Computer Engi-neering Department, McGill University. She is the author or coauthor of morethan 40 papers in peer-reviewed journals and conferences. Her research interestsinclude energy-efficient photonic interconnection for high-performance opticalsystems.

Isabella Cerutti received the Laurea degree fromthe Politecnico di Torino, Turin, Italy, in 1998, andthe Ph.D. degree from the University of Texas atDallas, Richardson, TX, in 2002, both in electricalengineering.

From 2002 to 2006, she was a Postdoctoral Re-search Associate at the University of Texas at Dallas,and then at the Scuola Superiore Sant’Anna, Pisa,Italy, where she is currently an Assistant Professor.She has authored or coauthored more than 50 publi-cations in international journals and conference pro-

ceedings. Her research interests include performance evaluation and design ofoptical and wireless networks, with special emphasis on energy efficiency.

Pier Giorgio Raponi received the Laurea (B.Eng.)degree and the Laurea Specialistica (M.S.) degreeboth in telecommunications engineering from theUniversity of Pisa, Pisa, Italy, in 2004 and 2009, re-spectively, and the Diploma degree with honors fromthe Scuola Superiore Sant’Anna, Pisa, in 2004. Heis currently working toward the Ph.D. degree at theCentre of Excellence for Information and Communi-cation Engineering, Scuola Superiore Sant’Anna.

From 2001 to 2008, he was with the Scuola Superi-ore Sant’Anna. His current research interests include

fast-switching architectures, scheduling algorithms and design, and analysis ofpacket-switched optical networks and nodes.

Nicola Andriolli received the Laurea degree intelecommunications engineering from the Universityof Pisa, Pisa, Italy, in 2002, and the Diploma andPh.D. degrees from the Scuola Superiore Sant’Anna,Pisa, in 2003 and 2006, respectively.

He was a Visiting Student at the Budapest Uni-versity of Technology and Economics, Budapest,Hungary, and in the Department of Communica-tions, Optics and Materials, Technical University ofDenmark, Copenhagen, Denmark. He was a VisitingTrainee at the National Institute of Information and

Communications Technology, Tokyo, Japan. He is currently an Assistant Pro-fessor at the Scuola Superiore Sant’Anna. His research interests include networkmodeling and simulation, QoS and fault tolerance in optical networks, general-ized multiprotocol label switching architectures, and optical packet switching.

Piero Castoldi (S’93–A’96–M’97) received theMaster’s degree in electronic engineering from theUniversity of Bologna, Bologna, Italy, in 1991, andthe Ph.D. degree in information technology from theUniversity of Parma, Parma, Italy, in 1996.

He is currently an Associate Professor at theScuola Superiore Sant’Anna, Pisa, Italy, where he isan Area Leader of the networks and services area andalso the Director of the National Laboratory of Pho-tonic Networks, National Interuniversity Consortiumfor Telecommunications since January 2005. During

1996–1997, he was a Visiting Postdoctoral Researcher in the Department ofElectrical Engineering, Princeton University, Princeton, NJ, where he was alsoa Visiting Professor during 1999–2000. From December 1997 until February2001, he was an Assistant Professor at the Faculty of Engineering, Universityof Parma. He is author or coauthor of more than 170 publications in inter-national journals and conference proceedings. His research interests includecover reliability, switching paradigms and control for optical networks, includ-ing application-network cooperation mechanisms.