SPRACE KyaTera / UltraLight Proposal

VI D0SAR Workshop

São Paulo, Brazil

September 16, 2005

Rogério L. Iope

Universidade de Sao Paulo

(Grad. Research Assistant for SPRACE)

e-Science: Data Gathering, Analysis, Simulation, Collaboration

Scientific discoveries increasingly driven by data collection• Computationally intensive analyses

• Massive data collections

• Data distributed across networks of varying capability

• Internationally distributed collaborations

New approaches to enquiry based on• Deep analysis of huge quantities of data

• Interdisciplinary collaboration

• Large-scale simulation

• Smart instrumentation

e-Science methods no longer optional but now vital to scientific competitiveness

e-Science: Driving Global Cyberinfrastructure

TOTEM

LHCb: B-physics

ALICE : HI

ATLAS

CMS

e-Science is about providing significantly enhanced research infrastructure by utilizing distributed resources such as computers, storage devices, scientific instruments, and experts using information technology

e-Science: Driving Global Cyberinfrastructure

The enormous speedups of computers and networks have enabled simulations of far more complex systems and phenomena, as well as visualizing the results from many perspectives

Advanced computing no longer restricted to a few research groups in a few fields, but pervades scientific and engineering research

New data-intensive applications are driving seemingly insatiable demand for more bandwidth

Groups collaborate across institutions and time zones, sharing data, complementary expertise, ideas, and access to special facilities without traveling

Optical Networks are key to this vision• Massive scalable bandwidth• Protocol and bit-rate independence• The ability to launch and scale new services on demand

Photonic Networking: the way to cope with IP traffic explosion

Overview of the UltraLight Project

UltraLight is• A collaboration of experimental physicists, computer scientists, and network

engineers from BNL, Caltech, CERN, UF, FIU, FNAL, Internet2, UM, MIT, SLAC...• …to provide the network advances required to enable petabyte-scale analysis

of globally distributed data• An application-driven network R&D program to explore the integration of

cutting-edge network technology with the Grid computing and data infrastructure of HEP/Astronomy

• A non-standard core network with dynamic and varying bandwidth interconnecting globally distributed nodes

• An NSF-funded 4 year program to deliver a new, high-performance, network-integrated infrastructure

Two primary, synergistic activities (source: S. McKee)• Network “Backbone”: Perform R&D / engineering• Application “Driver”: System Services R&D / engineering

Overview of the UltraLight Project

Main goals• Engineer and operate a trans- and intercontinental optical network testbed• Promote the network as an actively managed component• Develop and deploy prototype global services which broaden existing Grid

computing systems• Enable physics analysis and discoveries by integrating and testing UltraLight in

Grid-based physics production and analysis systems currently under development in ATLAS and CMS

A three-phased plan• Phase 1 (12 months): Implementation of network, equipment and initial services• Phase 2 (18 Months): Integration and footprint expansion• Phase 3 (18 Months): Transition to production (LHC physics + eVLBI astronomy)

Overview of the UltraLight Project

Project Management Team• PI: Harvey Newman (Caltech)

• Project Coordinator: Rick Cavanaugh (UF)• Network Coordinator: Shawn McKee (UM)• Applications Coordinator: Frank van Lingen (Caltech)• Education&Outreach Coordinator: Laird Kramer (FIU)• Physics Analysis User Community Coordinator: Dimitri Bourilkov (UF)• “Wan-In-Lab”: Steven Low (Caltech)

Project Coordination activities

• Regularly scheduled phone and video meetings• Periodic face-to-face focus workshops (semi-annually or quarterly)• Persistent VRVS room for collaboration• Mail-lists• Web-page portal (first prototype)

Overview of the UltraLight Project

Some important UltraLight R&D goals

• Basic Network Services

• Data transport protocols

• MPLS/QoS Services and Planning

• Optical Path Management Plans

• Optical Testbed

• Optical Exchange Point

• Network Monitoring

• Network Management and AAA

• Disk-to-disk data transfers

• Wan-In-Lab / DISUN

• HEP Application Services

Connectivity Diagram for UltraLight

Source: http://ultralight.caltech.edu/

The KyaTera Project

A cooperative program proposed by FAPESP, as part of the TIDIA Program

Main goal: The establishment of an optical fiber network infrastructure connecting

laboratories for research, development and demonstrations of technologies for advanced Internet applications

Network infrastructure based upon the concept of dark fibers reaching directly to the research laboratories (FTTLab)

The name KyaTera comes from• Kya (“net” in Tupi-Guarani)• Tera (greek teras = monster)

The KyaTera Project

Composed by a dark fiber mesh spread over several cities among the State of São Paulo

• A large, geographically distributed laboratory facility for experimental tests of new network concepts and optical devices, new network protocols and services

• A platform for developing and deploying new high performance e-Science applications

A stable, high performance network always co-exists with the experimental network• new developments in the last do not interfere with the operation of the first

The KyaTera Project

Research subjects for KyaTera organized in 3 layers

• Physical Layer optical communications, new developments on fiber infrastructure

• Transport Layer protocols, interface standards, maanagement, monitoring, interoperability,

etc, in optical networks

• Applications Layer automation and computer control of scientific instruments, Grid

applications, HDTV, etc

WDM Fundamentals

Wavelength-Division Multiplexing – WDM• An approach that can exploit the huge bandwidth available on fiber optic links

• Can manyfold the capacity of existing networks by transmitting many channels simultaneously on a single fiber optic line

• The optical transmition spectrum is carved up into a number of non-overlapping wavelength (or frequency) bands

• Multiple WDM channels from different end-users may be multiplexed on the same fiber

Each wavelength supports a single communication channel operating at peak electronic speed

By allowing multiple WDM channels to coexist on a single fiber, one can tap into the huge fiber bandwidth

A more cost-effective alternative compared to laying more fibers

WDM - Parallelism on Optical Networking

(WDM)

Source: Steve Wallach, Chiaro Networks

“Lambdas”Parallel lambdas will drive this decade

the way parallel processors drove the 1990s !

WDM Fundamentals

WDM building blocks

• Light sources (laser diodes) and detectors (photodetectors, filters)• Optical fibers (single-mode, multi-mode)• Multiplexers and Demultiplexers• Optical Add/Drop Multiplexers• Optical amplifiers (e.g. EDFA)• Photonic cross-connect switches• Transponders

WDM Fundamentals

A wavelength-routed optical WDM network consists of a photonic switching fabric comprising active optical switches connected by fiber links forming any arbitrary physical topology

Each node equipped with a set of transmitters and receivers (which may be “wavelength tunable”)

The basic mechanism of communication in such a network is a lightpath• Lightpath: an all-optical communication channel (a path) between 2 nodes (it can

span more than one fiber link!)

The intermediate nodes in this fiber path route the lightpath in the optical domain using their active optical (photonic) switches

The end-nodes of the lightpath access the signal with transmitters and receivers that are tuned to the wavelength on which the lightpath operates

WDM Fundamentals

Photonic switches & protocols like GMPLS are key elements to address new goals, and implement a multi-tiered and scalable IP/Optical network

WDM Fundamentals

In wavelength-routed WDM networks, a control mechanism is needed to set up and take down the optical connections (lightpaths)

A successful data transfer event between 2 nodes has three phases• Connection establishment• Data transfer• Connection release

During first phase, a few control signaling packets are exchanged between network resources, aiming to establish a lightpath with an assigned wavelength

If it succeeds, a lightpath is established, and data transfer occur through this circuit from source to destination

When the transfer is completed, control packets are again exchanged between the nodes, and the resources are released and made ready to be assigned for another connection

WDM Fundamentals

A challenging networking problem is that, given a set of lightpaths that need to be established on the network, and given a constraint on the number of wavelengths,

• determine the routes over which these lightpaths should be set up

• determine the wavelengths that should be assigned to them so that the maximum number of lightpaths may be established

If any switching/router node is also equipped with a wavelength-converter facility, then lightpaths can be established using diferent wavelengths on their routes from origin to destination

• This problem is referred to as the RWA problem

WDM Systems: General layout

Transmissor

TransponderDWDM

...

1

n

CoreRouter

CoreRouter

1

n

MUX DWDM

fiber

DEMUXDWDM

EDFAEDFA

fiber

OXC

GB

IC 1

GB

IC n

OA

DM

1

c1

cn

...

c1

Border

Router

. . . OADM n

cn

Border

Router

OA

DM

1

c1

cn

. . .

c1

Border

Router

...

OADM n

cn

Border

Router

GB

IC 1

GB

IC n

MUXCWDM

DEMUXCWDM

MUXCWDM

DEMUXCWDM

CWDM CWDM

...

DWDM

TransponderCWDM

TransponderCWDM

(Source: M. Stanton - GIGA Project)

WDM Systems: R-OADM Conceptual Architecture

AddWavelengths

DropWavelengths

Pass-Through WavelengthsSplitter

AddWavelengths

SoftwareControlled

DEMUX

Pass-Through WavelengthsSplitter

1NetworkElement

3NetworkElement

Software Controlled Selectors(Pass-through/Add/Block)

DWDMSignal

TransponderModule

West

East

DWDMSignal

DropWavelengths

drop block blockdrop

dropblock block drop

SoftwareControlled

DEMUX

Add

Pass

Add

Pass

NetworkElement

NetworkElement

TransponderModule

Pass

Pass

Add

Add

Software Controlled Selectors(Pass-through/Add/Block)

13

The KyaTera testbed: Reference Architecture

MUX/DEMUX&

R-OADM

MUX/DEMUX&

R-OADM

MUX/DEMUX&

R-OADM

MUX/DEMUX&

R-OADM

MUX/DEMUX&

R-OADM

MUX/DEMUX&

R-OADM

EthernetAggregation

Switch

EthernetAggregation

Switch

EthernetAggregation

Switch

Photonic Switch

Photonic Switch

Photonic Switch

IP Router10 GbE <->

IP Router10 GbE <->

IP Router10 GbE <->

The KyaTera testbed (example of a proposed solution)

Enabling e-Science: The KyaTera / UltraLight Proposal

Network support: a critical aspect of Grid-enabled environments

Commodity Internet is based on a best-effort delivery model, a vehicle excessively slow and unreliable for the huge masses of data being generated in emerging e-Science applications

Deployment of Grids on wide-area scales is being severely restricted

Enabling e-Science: The KyaTera / UltraLight Proposal

Optical networing: a promising solution to these limitations• Emerging lightpaths technologies are becoming more and more popular in

the Grid community• They can include the network resources as an integral Grid component,

controlled by Grid schedulers in the same way as computing elements and storage resources

The challenge:• A new management technology is needed to allow end-users to acquire

network resources on demand, control end-to-end interconnections between peers (lightpaths), and share unused bandwidth in a flexible and collaborative way

The KyaTera / UltraLight Proposal

Our project proposal:

• To work on the problem of monitoring, managing, and optimizing the use of the networking resources present in next-generation user-controlled optical networks in real time

• To work in close partnership with the UltraLight Project and the KyaTera Project

• To use the optical networking infrastructure that is being made available by the KyaTera Project

The KyaTera network insfrastructure, enhanced by an intelligent optical control plane middleware, will provide the basement for the deployment of the Grid-enabled Analysis Environment Service Architecture (GAE), a project being developed at Caltech and University of Florida, coordinated by Prof. Harvey Newman

The KyaTera / UltraLight Proposal

São Paulo Campinas

PXCWANSwitch

Cluster

Cluster

WANSwitch

PXC

GMPLS Control Plane

End-to-end lightpath

São Paulo Campinas

PXCWANSwitch

Cluster

Cluster

WANSwitch

PXC

GMPLS Control Plane

End-to-end lightpath

Research will be done on provisioning end-to-end survivable optical connections in the testbed, as in a Grid environment, with an innovative use of the GMPLS control plane

(this will be accomplished in a close partnership with the OptiNet lab experts)

(Drawing and text courtesy of Gustavo Pavani – OptiNet / UNICAMP)

Project Planning: Milestones and Timeframe

Milestones

I. Provisioning of end-to-end optical connections between pairs of nodes

II. Provisioning multilayer protocols and intelligent monitoring software agents, and research on RWA algorithms

III. Deployment of routing/switching and control protocols to locate suitable lightpaths and schedule the networking resources

IV. Deployment of Grid Analysis Environment

V. Job submissions and data transfers between sites over the distributed computing infrastructure looking for failures, malfunctioning and bottlenecks

Project Planning: Milestones and Timeframe

TasksFirst Year Second Year

1st 2nd 3rd 4th 5th 6th 7th 8th

1

2

3

4

5

6

7

8

9

10

11

12

13

14

I

IV

V

III

II