GT4 as an Implementation Kernel for NAREGI Grid MW beta 1 Satoshi Matsuoka Professor, Global Scientific Information and Computing Center, Deputy Director,

GT4 as an Implementation Kernel for NAREGI Grid MW

beta 1

Satoshi MatsuokaProfessor, Global Scientific Information and

Computing Center, Deputy Director, NAREGI Project

Tokyo Institute of Technology / NII

National Research Grid Infrastructure (NAREGI) 2003-2007

• Petascale Grid Infrastructure R&D for Future Deployment– $45 mil (US) + $16 mil x 5 (2003-2007) = $125 mil total– PL: Ken Miura (FujitsuNII)

• Sekiguchi(AIST), Matsuoka(Titech), Shimojo(Osaka-U), Aoyagi (Kyushu-U)…– Participation by multiple (>= 3) vendors, Fujitsu, NEC, Hitachi, NTT, etc.– NOT AN ACADEMIC PROJECT, ~100FTEs– Follow and contribute to GGF Standardization, esp. OGSA

AIST

Grid Middleware Grid Middleware R&DR&D

SuperSINETSuperSINET

Grid R&D Infrastr.Grid R&D Infrastr.15 TF => 100TF15 TF => 100TF

National AAA Infr.National AAA Infr.

““NanoGrid”NanoGrid”IMS ~10TFIMS ~10TF

(BioGrid(BioGridRIKEN)RIKEN)

OtherOtherInst.Inst.

National ResearchNational ResearchGrid Middleware R&DGrid Middleware R&D

NanotechNanotechGrid AppsGrid Apps

(Biotech(BiotechGrid Apps)Grid Apps)

(Other(OtherApps)Apps)

Titech

Fujitsu

NECOsaka-U

U-Kyushu Hitachi

Focused “Grand Challenge” Grid Apps Areas

IMS

FY2003 FY2004 FY2005 FY2006 FY2007

UNICORE/Globus/Condor-based R&D Framework

Globus4/OGSA/WSRF-based R&D Framework

Utilization of NAREGI NII-IMS Testbed Utilization of NAREGI-Wide Area Testbed

PrototypingNAREGI Middleware

Components

Development andIntegration of

αrelease

Development and Integration of

βrelease

Evaluation on NAREGI Wide-area

Testbed Development ofOGSA-based MW

beta 1, 2,

Verification& Evaluation

Of Ver. 1

Apply ComponentTechnologies toNano Apps and

Evaluation

Evaluation of αrelease in NII-IMS

Testbed

Evaluation of βReleaseBy IMS and other

Collaborating Institutes

Development and deployment of β1, 2, 3,…

αRelease(internal)

OGSA(subset)/Unicore+GT2/3/3.9

β1 Release(public)

OGSA/WSRF/GT4GGF17

Version1.0

Release

Mid

po

int

Eva

lua

tion

NAREGI Middleware Roadmap

NAREGI Software Stack (Alpha 2005)

Computing Resources

NII IMS Research Organizations etc

（（ UNICORE4, GT2/3/3.9, CondorUNICORE4, GT2/3/3.9, CondorOGSA)OGSA)

SuperSINET

Grid-Enabled Nano-Applications

Grid PSEGrid 　 Programing

-Grid RPC -Grid MPI

Grid Visualization

Grid VM

Packag

ing

DistributedInformation Service

Grid Workflow

Super Scheduler

High-Performance & Secure Grid Networking

NAREGI Software Stack (beta 1 2006)

Computing Resources and Virtual Organizations

NII IMS Research Organizations

Major University Computing Centers

（（WSRF (GT4+Fujitsu WP1) + GT4 and other services)WSRF (GT4+Fujitsu WP1) + GT4 and other services)

SuperSINET

Grid-Enabled Nano-Applications (WP6)

Grid PSEGrid 　Programming (WP2)

-Grid RPC -Grid MPI

Grid Visualization

Grid VM (WP1)

Packag

ing

DistributedInformation Service

(CIM)

Grid Workflow (WFML (Unicore+ WF))

Super Scheduler

Grid Security and High-Performance Grid Networking (WP5)

Data (W

P4)

WP3

WP1WP1

Highlights of NAREGI Beta (May 2006, GGF17/GridWorld)

• Professionally developed and tested• “Full” OGSA-EMS incarnation

– Full C-based WSRF engine (Java -> Globus 4)– OGSA-EMS/RSS WSRF components– Full WS-Agreement brokering and co-allocation– GGF JSDL1.0-based job submission, authorization, etc.– Support for more OSes (AIX, Solaris, etc.) and BQs

• Sophisticated VO support for identity/security/monitoring/accounting (extensions of VOMS/MyProxy, WS-* adoption)

• WS- Application Deployment Support via GGF-ACS• Comprehensive Data management w/Grid-wide FS• Complex workflow (NAREGI-WFML) for various coupled simulatio

ns• Overall stability/speed/functional improvements• To be interoperable with EGEE, TeraGrid, etc. (beta2)

List of NAREGI “Standards”(beta 1 and beyond)

• GGF Standards and Pseudo-standard Activities set/employed by NAREGI

GGF “OGSA CIM profile” GGF AuthZ GGF DAIS GGF GFS (Grid Filesystems) GGF Grid CP (GGF CAOPs) GGF GridFTP GGF GridRPC API (as Ninf-G2/G4)GGF JSDL GGF OGSA-BES GGF OGSA-Byte-IO GGF OGSA-DAI GGF OGSA-EMS GGF OGSA-RSS GGF RUS GGF SRM (planned for beta 2) GGF UR GGF WS-I RUS GGF ACS GGF CDDLM

• Other Industry Standards Employed by NAREGI

ANSI/ISO SQL DMTF CIM IETF OCSP/XKMS MPI 2.0 OASIS SAML2.0 OASIS WS-Agreement OASIS WS-BPEL OASIS WSRF2.0 OASIS XACML

• De Facto Standards / Commonly Used Software Platforms Employed by NAREGI

GangliaGFarm 1.1Globus 4 GRAMGlobus 4 GSI Globus 4 WSRF (Also Fujitsu WSRF for C binding)IMPI (as GridMPI)Linux (RH8/9 etc.), Solaris (8/9/10), AIX, …MyProxy OpenMPI Tomcat (and associated WS/XML standards) Unicore WF (as NAREGI WFML)VOMS

Role of GT4 in NAREGI Grid MW stack beta 1

GT4 as Implementation Kernel• As underlying WSRF engine for OGSA and othe

r WS components• Various Globus services used throughout to i

mplement OGSA and other GGF “standard” components and functionalities in all WPs– GSI, WS-GRAM, GridFTP, RFT– OGSA-DAI, MyProxy

• In a way, NAREGI MW “glues” GT4 services and some GGF OGSA “standards”

NAREGI beta 1 SSS ArchitectureAn extended OGSA-EMS Incarnation

WFML2BPEL

SS

NAREGI JM(SS) 　 Java I/F module

NAREGI-WP3 WorkFlowTool, PSE, GVS

JM-Client

SubmitStatusDeleteCancel

EPS

CSG

IS

Cancel StatusSubmit

BPEL2WFST

CreateActivity(FromBPEL)GetActivityStatusRequestActivityStateChanges

CES

OGSA-DAI

CIM

DB PostgreSQL

JSDLJSDL

JSDLJSDL

MakeReservationCancelReservation

globus-url-copy

uber-ftp

CES

CES

S

SR

R

BPEL (include JSDL)BPEL (include JSDL)

Invoke EPS

Invoke SC

JSDLJSDLJSDL

JSDLJSDLJSDL

NAREGI-WFML

NAREGI-WFML

JSDL

Co-allocation FileTransfer

CreateActivity(FromJSDL)GetActivityStatusRequestActivityStateChanges

MakeReservationCancelReservation

JSDLJSDLJSDL

JSDLJSDLJSDL

JSDLJSDLJSDL

JSDLJSDLJSDL

JSDLJSDL

JSDLJSDL

JSDLJSDL

Delete

JSDLJSDL

NAREGI JM (BPEL Engine)

S

JSDLJSDL

JSDLJSDLJSDL

JSDLJSDLJSDL

JSDLJSDL

AbbreviationSS: Super SchedulerJSDL: Job Submission Description DocumentJM: Job ManagerEPS: Execution Planning ServiceCSG: Candidate Set GeneratorRS: Reservation ServiceIS: Information ServiceSC: Service ContainerAGG-SC: Aggregate SCGVM-SC: GridVM SCFTS-SC: File Transfer Service SCBES: Basic Execution Service I/FCES: Co-allocation Execution Service I/F (BES+)CIM: Common Information ModelGNIS: Grid Network Information Service

SelectResourceFromJSDL

GenerateCandidateSet

GenerateSQL QueryFrom JSDL

AGG-SC/RS

SC

CES

PBS, LoadLeveler

S

GRAM4 specific

SC(GVM-SC)

SC

CES

PBS, LoadLeveler

S

GRAM4 specific

SC(GVM-SC)

R R

WS-GRAM

GridVMWS-GRAM

GridVM

FTS-SC

Fork/Exec

is-query

Fork/Exec

globusrun-wsglobusrun-ws

Fork/ExecFork/Exec

GFarmserver

GNIS

GetGroups-OfNodes

WSRF Interoperability• Most of NAREGI-SS is implemented as C WS

RF component on NAREGI WSRF ToolKit (NWK), which complies to the latest WSRF spec

• Other NAREGI WSRF components are typically Java and runs on top of GT4 WSRF, which complies to the previous candidate spec

• Fortunately, NWK and GT4 WSRFs interoperate quite well, with very minor workarounds (!)

NAREGI Info Service (beta) Architecture

Client(Resource Broker etc.)

ClientLibrary

Java-API

RDB

Light-weightCIMOMService

AggregatorService OS

Processor

File System

CIM Providers

●　●

ResourceUsageService

RUS::insertURs

ChargeableService

(GridVM etc.)

Job Queue

Cell Domain Information Service

Node B

Node A

Node C

ACL

GridVM


Information Service Node

… Hierarchical filtered aggregationParallel Query …

PerformanceGanglia

DataService

ViewerUserAdmin.

・ CIMOM Service classifies info according to CIM based schema. ・ The info is aggregated and accumulated in RDBs hierarchically.・ Client library utilizes OGSA-DAI client toolkit.・ Accounting info is accessed through RUS.

Client(publisher)

NAREGI IS: Standards Employed in the Architecture

GT4.0.1DistributedInformation Service

Client(OGSA- RSS etc.)

　 Clientlibrary

Java-API

RDB Light-weightCIMOMService

AggregatorService

OS

Processor

File System

CIM Providers

●　●

WS-IRUS

RUS::insertURs

GridVM(Chargeable

Service)

Job Queue


Node B

Node A

Node C

ACL

GridVM


Information Service Node Information Service Node

… Hierarchical filtered aggregation... Distributed Query …

PerformanceGanglia

OGSA-DAI WSRF2.1

ViewerUserAdmin.

OGSA-DAIClient toolkit

CIM Schema2.10

/w extension

GGF/UR

GGF/UR

APP

CIM spec.CIM/XML

Tomcat 5.0.28

Client(OGSA- BES etc.)

APP

NAREGI GridVM (beta) Architecture

Virtual execution environment on each site•Virtualization of heterogeneous resources•Resource and job management services with unified I/F Super

SchedulerSuper

SchedulerInformation

ServiceInformation

Service

AIX/LoadLeveler

GridVM SchedulerGridVM Scheduler

GRAM4GRAM4 WSRF I/FWSRF I/F

Local SchedulerLocal Scheduler

GridVM EngineGridVM Engine

Linux/PBSPro





GridMPI

sitePolicy

Advance reservation, Monitoring, Control

Accounting

ResourceInfo.

SandboxJob Execution site

Policy

NAREGI GridVM: Standards Employed in the ArchitectureSuper

SchedulerSuper

SchedulerInformation

ServiceInformation

Service









GridMPI

sitePolicy

xacml-like access control policy

CIM-based resource info. provider

GT4 GRAM-integration andWSRF-based extension services

UR/RUS-based job accounting

Job submission based on JSDLand NAREGI extensions

sitePolicy

GT4 GRAM-GridVM Integration

GridVMscheduler

RFT File Transfer

Local scheduler

GridVMEngine

SS

globusrun RSL+JSDL’

Delegate

Transferrequest

GRAMservices

DelegationScheduler

Event Generator

GRAMAdapter

GridVMJobFactory

GridVMJob

SUDO

Extension Service

Basic job management＋ Authentication, Authorization

Site

Integrated as an extension module to GT4 GRAM Aim to make the both functionalities available

Ninf-G: A Reference Implementation of the GGF GridRPC API

Large scale computing across supercomputers on the Grid

user

① Call remote procedures

② Notify results

Utilization of remotesupercomputers

Call remote libraries

Internet

• What is GridRPC?Programming model using RPCs on a GridProvide easy and simple programming interfaceThe GridRPC API is published as a proposed recommendation (GFD-R.P 52)

• What is Ninf-G?A reference implementation of the standard GridRPC APIBuilt on the Globus ToolkitNow in NMI Release 8 (first non-US software in NMI)

• Easy three steps to make your program Grid aware– Write IDL file that specifies interface of your library– Compile it with an IDL compiler called ng_gen– Modify your client program to use GridRPC API

Architecture of Ninf-G

Client

WS/Pre-WS GRAMNAREGI-SS

Invoke Executable

Connect back

IDL file NumericalLibrary

IDL Compiler

Remote LibraryExecutable

Generate Interface Request

Interface Reply

Server side

Client side

MDS2/4 Interface InformationLDIF Fileretrieve

Globus-IO

Grid Application Environment（WP3）

Workflow Engine &Super Scheduler

FileTransfer

(RFT)

UnderlyingGrid Services

DistributedInformation Service ・・・・・・

CFDVisualization

Service

CFDVisualizer

MolecularVisualization

Service

MolecularViewer

ParallelVisualization

Service

ParallelVisualizer

DeploymentService

ApplicationContentsService

compile deploy un-deploy Application

Repository

(ACS)

File

/ExecutionManager

WorkflowService

GridFile System

VOMSMyProxy

Portal GUI

Visualization GUIRegister UIDeployment UI Workflow GUI

Grid VisualizationGrid WorkflowGrid PSE

　 NAREGI PortalPortal GUIBio VOBio VO Nano VONano VO

Gateway Services

Core Grid Services

NAREGI-WFML

JM I/F module

BPEL+JSDL

WSRF

Data 1 Data 2 Data n

Grid-wide File System

MetadataManagement

Data Access Management

Data ResourceManagement

Job 1

Meta-data

Meta-data

Data 1

Grid Workflow

Data 2 Data n

NAREGI Data Grid beta1 Architecture (WP4)

Job 2 Job n

Meta-data

Job 1

Grid-wide Data Sharing Service

Job 2

Job n

Data Grid Components

Import data into workflow

Place & register data on the Grid

Assign metadata to data

Store data into distributed file nodes

Data 1 Data 2 Data n

Gfarm 1.2 PL4(Grid FS)

Data Access Management

NAREGI WP4: Standards Employed in the Architecture

Job 1

Data SpecificMetadata DB

Data 1 Data n

Job 1 Job nImport data into workflow

Job 2

Computational Nodes

FilesystemNodes

Job n

Data ResourceInformation DB

OGSA-DAI WSRF2.0

OGSA-DAI WSRF2.0

Globus Toolkit 4.0.1

Tomcat5.0.28

PostgreSQL 8.0

PostgreSQL 8.0

Workflow (NAREGI WFML =>BPEL+JSDL)

Super Scheduler (SS) (OGSA-RSS)

Data Staging

Place data on the Grid

Data Resource Management

Metadata Construction

OGSA-RSSFTS SC

GridFTP

GGF-SRM (beta2)

NAREGI WP4 Standards• GGF Standards we help set within a WG

– Grid FileSystems WG (discussion about functionality and usecase scenario)

• GGF and related Standards we employ– OGSA-DAI– OGSA-RSS– GridFTP– WSRF 2.0– JSDL– SRM (planned for beta 2)

• Other industry standards we employ– BPEL

• Other de-facto “standards” we employ– Globus 4– Tomcat (and associated WS/XML standards)

NAREGI-beta1 Security Architecture(WP5)

NAREGICA

Client EnvironmentClient Environment

Portal WFT

PSE

GVM SS

clie

nt

ProxyCertificatewith VO


Super Scheduler



log-inlog-in

Request/GetRequest/Get CertificateCertificate

VOMSVOMS

CertificateManagement Server

CertificateManagement Server

ProxyCertificate

withVO

ProxyCertificate

withVO

UserCertificate

PrivateKey

Put ProxyCertificaPut ProxyCertificatete

with VOwith VO

Get VOMS Get VOMS AttributeAttribute

MyProxyMyProxyProxy

Certificatewith VO


voms-myproxy-initvoms-myproxy-init

InformationService

Resources InfoResources Infoincl. VOincl. VO

ResourceResource

GridVMlocal Info.local Info.incl. VOincl. VO

CA Service

ssh + ssh + voms-myproxy-initvoms-myproxy-init

ProxyCertificateProxyCertificatewith VOwith VO

queryquery(requirements(requirements+VO info)+VO info)

resourcesresourcesin the VOin the VO

globusrun-wsglobusrun-ws　　　　 GridVM seGridVM se

rvicesrvices(incl. GSI)(incl. GSI)

Resource Info.Resource Info.(Incl. VO info)(Incl. VO info)

Signed Signed Job Job

DescriptionDescription

VO 、 Certificate

Management Service

VO InfoVO Info 、、 Execution Info,Execution Info,Resource InfoResource Info

CP/CPS

GRID CP(GGF CAO

Ps)

Audit Criteria

Subset of WebTrust

Programs for CA

InformationService

NAREGI Application Mediator (WP6) for Coupled Applications

Job 1

WorkflowNAREGI WFT

co-allocated jobs

Super Scheduler

GridVM GridVM GridVM

Simulation ASimulation ASimulation A

Mediator

Simulation ASimulation ASimulation B

MediatorA

MediatorA

MediatorA

MediatorB

MediatorB

Sim.B

Sim.B

Sim.A

Sim.A

Sim.A

Support data exchange between coupled simulation

ＭＰＩ

OGSA-DAI WSRF2.0

SQL

JNI

・ Semantic transform- ation libraries for different simulations

Data transfer management

・ Synchronized 　　　　　 file transfer

Mediator Components

・ Multiple protocol GridFTP/MPI

Job n

ＭＰＩ

Data transformationmanagement

・ Coupled accelerator

ＭＰＩ

GridFTP

MediatorMediator

Globus Toolkit 4.0.1

GridFTP

APIAPI

Data1

Data2

Data3

*SBC: Storage-based communication

SBC* -XML・ Global Job ID・ Allocated nodes・ Transfer Protocol etc.

Summary and Future• NAREGI beta 1 will be publicly released at GGF17@Tokyo, M

ay 2006 • GT4 has served as robust, scalable, and reliable implement

ation kernels for OGSA and other GGF standards services• Plan to work with the GT team to enhance OGSA features, ex

ploit advanced functionalities (already budgeted for FY06)– Enhance ES-GRAM, in particular default JSDL integration, reservatio

n/co-scheduling– CIM/GLUE interoperable information Model and MDS4– Workspace Service and GridVM– Etc…

Documents

GT4 as an Implementation Kernel for NAREGI Grid MW beta 1 Satoshi Matsuoka Professor, Global Scientific Information and Computing Center, Deputy Director,