©STFC/Keith G JefferyThe Future of GRIDs: A European Perspective 20080410 1 The Future of GRIDs: A European Perspective Keith G Jeffery Science and Technology

©STFC/Keith G Jeffery The Future of GRIDs: A European Perspective 20080410 1

The Future of GRIDs:

A European Perspective

Keith G JefferyScience and Technology Facilities Council

Rutherford Appleton Laboratory, OX11 0QX UKe-mail: [email protected]


Who ?

Old job – running the major IT department

– Computer operations• 360,000 users• 1100 servers

– Systems development• Corporate• Departments• External

– R&D– Library and Information Services– Photoreprographics

Director, IT & International Strategy


And…President ERCIMPresident euroCRISHonorary Visiting Professor

– University of Cardiff– Heriot Watt University Edinburgh– Masaryk University Czech Republic

Fellow BCS and GS, Honorary Fellow ICS, CEng, CITP


CCLRC-RAL Site


PPD: CERN: LHC


PPD : CMS


ISIS: Neutrons


SSTD: Earth

Gulf Stream

Etna


SSTD: Mars

23 January 2004This picture was taken by the High Resolution Stereo Camera (HRSC) onboard ESA's Mars Express orbiter, in colour and 3D, in orbit 18 on 15 January 2004 from a height of 273 km. The location is east of the Hellas basin at 41° South and 101° East. The area is 100 km across, with a resolution of 12 m per pixel, and shows a channel (Reull Vallis) once formed by flowing water. The landscape is seen in a vertical view, North is at the top.


SSTD: Earth

Dartford, UK (with inset of the Queen Elisabeth Bridge) from 680km up

First image from RAL camera on TOP-SAT


Lasers: Vulcan


Diamond: Synchrotron Radiation


Computing

STFC runs HPC(X)5th fastest computer when purchasedIBM Power seriesUsed by UK R&D Community


VR: EISCAT Control

Problem: cost and data loss when training scientists to use EISCAT

Answer: VR system at RAL to train before going to Norway


18 European countries - major labs or consortia of universities12000 ICT researchers

– Working groups– Fellows programme– Cor Baayen Award

Strategy documents for EC and national governmentsR&D projects, networks of excellence etc> 100 spin-out companiesHost of W3C EuropeEuropean Office(s) of W3CERCIM Newswww.ercim.org

ERCIMEuropean Research Consortiumfor Informatics and Mathematics


Linking together systems in each country managing research information

– Funders of research– Organisations performing research

For– Strategic decision-making about ewhat research to fund /do– Finding research partners and competitors– Finding innovative ideas for technology transfer / exploitation– Informing the media / public

www.eurocris.org

CERIF: an EU Recommendation to member states


So?This background gives you some idea of ‘where I’m coming from’

– Advanced research problems requiring ICT solutions– Research not ‘blue sky’ but practical– Management and administrative systems for Research

Support– International working– Strategic thinking for / using blue sky research to plan

roadmaps for ICT R&D


So?This background gives you some idea of ‘where I’m coming from’

– Advanced research problems requiring ICT solutions– Research not ‘blue sky’ but practical– International working– Strategic thinking for / using blue sky research to plan roadmaps for ICT R&D

And what I am going to talk about is the ICT of the future that we shall all be using and/or developingAnd the research challenges we have to overcome to make it happen


STRUCTURE

The Original UK IdeaWhere We Are NowThe R&D Required to Achieve GRIDsNGG: Next Generation GRIDsCoreGRIDChallengersConclusion


In the beginning…..In 1999 the UK Research Councils (which fund university R&D) were undergoing their Strategic Review Exercise for funding beyond 2000

– Grand challenge science projects

The DGRC (John Taylor) unhappy that plans– had too little IT– the IT proposed was incoherent

So he asked CCLRC CEO (Bert Westwood) to have someone generate an IT planAnd Bert asked me


The GRIDs Vision

The end-user interacts with the GRIDs environment to clarify the request

– using a ‘device’ or ‘appliance’

The GRIDs environment proposes a ‘deal’ to satisfy the request

– which may or may not involve money

The user accepts or rejects the ‘deal’


The GRIDs Vision

The GRIDs environment is such that– A user can interact with it intelligently – It provides transparent access to

• data, information, knowledge• computation • instrumentation / detectors


The GRID Bible


The GRIDs Architecture

Knowledge Layer

Information Layer

Computation / Data LayerDat

a to

Kno

wle

dge

Control


The GRIDs ArchitectureD

ata

to K

now

ledg

eC

ontrol

Par

ticl

e P

hysi

cs A

ppli

cati

on

Gen

omic

s A

ppli

cati

on

Env

iron

men

tal A

ppli

cati

on

E-B

usin

ess

App

lica

tion


The Big Idea:What it Provides

UserAppliance

‘The Wall’

The GRIDsEnvironment

Plug-in

PCPalmtopMobile..

Personal CommunicationPersonal Shopping

Hobbies, family activitiesBusiness Communication

Business DealingBusiness Information


Ambient, pervasive,

mobileThe user appliance may well be mobile and requires pervasive connectivityIt may have interesting capabilities such as attachment of detectors / instruments

– Scientific research– paramedics, firefighters– Even ‘road warriors’


A POSSIBLE ARCHITECTURE

U:USER

S:SOURCE R:RESOURCE

Rm:ResourceMetadata

Ra:ResourceAgent

Ua:User Agent

Um:User Metadata

Sm:SourceMetadata

Sa:Source Agent brokers

The GRIDs Environment


Classification of Metadata

data (document)

SCHEMA NAVIGATIONAL ASSOCIATIVE

how to

get it

constrain it

view to users


Representative Agents

Represent the entities {U, S, R} continuously and actively within the GRIDs environmentWith metadata represent the entity to others represented by their agentsAct on behalf of the entity


Brokers(a) authentication,

(b) clarification / precision of request,

(c) resource discovery (information and if necessary compute power, visualisation facilities etc)

(d) authorisation (rights),

(e) offer and pricing,

(f) closure of deal (U accepts (e))


Brokers (continued)(g) fusion of responses,

(h) application of any transformation / analysis / simulation / visualisation processes,

(i) presentation formatting (for variously abled devices and people using various resources),

(j) network routing, and (k) scheduling of physical resource access / usage


Monitoring Brokers

and others will monitor

– quality of service,

– utilisation of resource collections

– specialist physical resources

– etc etc.


STRUCTURE

The Original UK IdeaWhere We Are NowThe R&D Required to Achieve GRIDsNGG: Next Generation GRIDsCoreGRIDChallengersConclusions


1G: custom-made architecture machines to user– Pioneering metacomputing

2G: proprietary standards and interfaces– I-WAY GLOBUS, UNICORE, CONDOR, LEGION AVAKI

2.5G: added in FTP, SRB, LDAP, AccessGRID3G: adopted W3C concepts for open interfaces – OGSA / OGSI: note especially OGSA/DAI

– But built on 2.G foundations

A Brief History of GRIDs



1G: custom-made architecture machines to user

– Pioneering metacomputing

2G: proprietary standards and interfaces– I-WAY GLOBUS, UNICORE, CONDOR, LEGION

AVAKI

2.5G: added in FTP, SRB, LDAP, AccessGRID 3G: adopted W3C concepts for open interfaces – OGSA / OGSI: note especially OGSA/DAI


e-ScienceApps


1G: custom-made architecture machines to user– Pioneering metacomputing

2G: proprietary standards and interfaces– I-WAY GLOBUS, UNICORE, CONDOR, LEGION AVAKI

2.5G: added in FTP, SRB, LDAP, AccessGRID3G: adopted W3C concepts for open interfaces – OGSA / OGSI: note especially OGSA/DAI



e-ScienceApps

e-ScienceR&D


But…..This comes nowhere near the requirements as originally defined for GRIDsToo low-level (programmer not end-user level)

– Insufficient representativity– Insufficient expressivity– Insufficient resilience– Insufficient dynamic flexibility


STRUCTURE



1999-2000The R&D issues were proposed in late 1999Discussed and approved at a meeting in UK of representative ‘gurus’ from academia and industry in early 2000Essential Technologies needing R&D

• Ease of use• Trust• Performance


Facilitate Ease of UseMetadata

– Representation language expressivity: user, source, resource– within / across domains

Agents– Specialised or Generalised and configured by metadata– Dynamically reconfigured by events / messages

Brokers– Functional– Knowledge-based with some autonomy– Strategic knowledge


Facilitate TrustSecurity

– Wireless communications– Availability of service

Privacy– Tradeoff personal information for intelligent

system reaction

Trust– Of services / servers– Of payment for services


Facilitate Performance

Mobile code– Be able to move the code to the data rather than

data to the code– Share code among nodes active in one request

Performance and optimisationSynchronisation, consistency, reliabilityEase of management


STRUCTURE



So…..

The US GRID is metacomputing plus extensions

– In 2002 improved with OGSA using W3C Web Services ideas

European position is that GRID architecture (GLOBUS or even UNICORE) is the wrong starting point for the European vision


And…..EC persuaded of importance of GRIDs

– Started in IST/Environment (early 2000) with IT architectural framework for FP6 projects

– Set up GRID Unit under Wolfgang Boch (late 2002)

January 2003: large workshop (GRID Unit)– (~ 240 participants)– Keynotes:

• Thierry Priol (INRIA, FR) • Domenico Laforenza (CNR, IT) • Keith Jeffery (CCLRC, UK)


NGG Requirements

Transparent and reliableOpen to wide user and provider communitiesPervasive and ubiquitousSecure and provide trust across multiple administrative domainsEasy to use and to programPersistentBased on standards for software and protocolsPerson-centricScalableEasy to configure and manage


inteliGRIDSemantic Grid

based virtual organisations

ProvenanceProvenance for Grids

DataminingGridDatamining

tools & services

UniGridSExtended OGSA

Implementation based on UNICORE

K-WF GridKnowledge based

workflow & collaboration

GRIDCOORDBuilding the ERAin Grid research

European - wide virtual laboratory for longer term Grid research - creating the foundation for the next generation Grids

COREGRID

EU - driven Grid services architecture for business

and industryNEXTGRID

Mobile Grid architecture and services for dynamic

virtual OrganisationsAKOGRIMO

Grid-based generic enablingapplication technologies to

facilitate solution of industrialproblemsSIMDAT

OntoGridKnowledge Services for

the semantic Grid

HPC4UFault tolerance,dependability

for Grid

Figure 1: The Call 2 Projects as a ‘house’

Call2 (NGG1) Projects Funded


NGG2

NGG1 left some undefined research areasCall2 projects did not address all areas of research opportunityNGG2 convened to update the vision:

– Particularly security / trust– Particularly self-* properties– Particularly semantic description of components

Report September 2004


Application A Application B Application C

Grids Middleware Services Needed for A

Grids Middleware Services Needed for B

Grids Middleware Services Needed for C

Grids Foundations for Operating System X

Grids Foundations For Operating System Y

Operating System X

Operating System Y

Grids Operating System(including Foundations)Modular and dynamically loadable

NGG2 Architecture


September 2005 - January 2006 Draft report to EC DG INFSO F2 in December for EC internal discussions on FP7Final Report in January 2006Key messages

– GRIDs environment layering too complex– Use SOKU

• Service Oriented Knowledge Utility

NGG3


A utility is a directly and immediately useable service with established functionality, performance and dependability, illustrating the emphasis on user needs and issues such as trust

Services are knowledge-assisted (‘semantic’) to facilitate automation and advanced functionality, the knowledge aspect reinforced by the emphasis on delivering high level services to the user

Service-Oriented Knowledge UtilityNGG3

The architecture comprises services which may be instantiated and assembled dynamically, hence the structure, behaviour and location of software is changing at run-time;


Next Generation Grids Report 2005Next Generation Grids Report 2005NGG3

NGG1&NGG2 vision and research challenges

NextGeneration

Grids

Architectural Vision

OpenReliable Scalable

Persistent Transparent

Person-centricPervasive

Secure / trusted Standards-based

User Interface

Grid Economies Business models

Virtual Organisation

Systems Management Co-ord. and orchestration

Information representation

Research Themes

NextGeneration

Grid(s)

Adaptability, Scalability,

Dependability

Raising theLevel of

Abstraction

Trust and Security in

VirtualOrganizations

Semantic Technologies

Lifecycle Management

Pervasiveness and

Context Awareness of Services

Future for European Grids: GRIDs and Service Oriented Knowledge Utilities –Future for European Grids: GRIDs and Service Oriented Knowledge Utilities – Vision and Research Directions 2010 and Beyond, December 2006Vision and Research Directions 2010 and Beyond, December 2006

Human Factors and

Societal Issues

End-User – Business/Enterprise –Manufacturing/Industrial

Research Topics

Driving Scenarios


From Web towards SOKUNGG3


NGG3: SOKU

Interfaces

ComputingInfrastructure

Services

Non SOKU

Non SOKU

Non SOKU

Non SOKU


NGG3: SOKU

SOKU Concept – dynamically composed utility environment

SOKU components – Grid/Web services with attitude

• Semantic metadata self-description allowing self-organising, self-composing

– like an OO class with attitude– like a KE Frame with attitude– like a (rather elaborate) function signature


e-,i-,k-infrastructure

serverserver server server

detectors

e-

i-

k- Deduction & induction – human or machine

Physical

Information

Systems

server


Middleware – and as SOKUs

e-

i-

k-

Lower middleware(hides physical heterogeneity)

Upper middleware(hides syntactic heterogeneity)

K- upper middleware(resolves semantic heterogeneity)

K- lower middleware(presents declared semantics)


Services to SOKU with METADATA

A ServiceComposed Services

Functional Program

Code(to deliver the service)

Service description(descriptive metadata)

InputParameterdefinitions

OutputParameterdefinitions

Restrictions on use of service(restrictive metadata)

Multiple

Instances

Parallel

execution


STRUCTURE



Developmental Work packages

– Data and knowledge management

– Programming Model– System Architecture– Resource Management and

Scheduling– Problem Solving

Environments

Other Work Packages– Internal Dissemination– Trust & Security– Testbed for Research

Assessment– Mobility of Researchers

External– Education & Training– Industry Liaison

•NoE Submitted October 15th 200342 partners; coordinated by


Achievements

•GCM: Grid Component Model•Trust and Security•Data, Information and Knowledge management•Mobile GRIDs•Desktop GRIDs

By linking with other EC and national-funded GRIDs projectsand providing a coherent base


And after CoreGRID ?

We have a reasonably well-trained pan-European workforce in advanced ICT (GRIDs)Sustainability by using ERCIM as the centre for a CoreGRID Institute (Working Group)To ensure Europe stays at the forefront


STRUCTURE


©STFC/Keith G Jeffery The Future of GRIDs: A European Perspective 20080410 65March 2008 CHALLENGERS Consortium 65

CHALLENGERS CHALLENGERS Research Agenda and Research Agenda and

RoadmapRoadmap

IST- 034128

©STFC/Keith G Jeffery The Future of GRIDs: A European Perspective 20080410 66Brussels March 2008 CHALLENGERS Consortium 66

Applications:•Social networking and collaborative users communities:

•Collaborative environments of users' communities•Augmented reality• On-line gaming

•Sensor enriched infrastructures:•User as a sensor – Networked sensors – Internet of things•Situation/Context Monitoring (e.g. health monitoring)•Scientific Grids -> Sensors + Experiments

Applications:•Social networking and collaborative users communities:

•Collaborative environments of users' communities•Augmented reality• On-line gaming

•Sensor enriched infrastructures:•User as a sensor – Networked sensors – Internet of things•Situation/Context Monitoring (e.g. health monitoring)•Scientific Grids -> Sensors + Experiments

Applications:•Grid-enabled support of social and cognitive sciences with knowledge:

•From data/information to knowledge•Semantic metadata exist for everything

•Grid enabled mission-critical applications:•Grids in business and critical infrastructures•Knowledge based healthcare •Disaster monitoring and handling•Risk reduction

Applications:•Grid-enabled support of social and cognitive sciences with knowledge:

•From data/information to knowledge•Semantic metadata exist for everything

•Grid enabled mission-critical applications:•Grids in business and critical infrastructures•Knowledge based healthcare •Disaster monitoring and handling•Risk reduction

High Priority Component Technologies:•User interfaces

•Hide complexity•Enable personalization•Allow service composition by users

•Intelligent Networking •Network semantics•Support SOA•Enable QoS by design

•Computer Architectures•Systems on Chip •Towards Networked Sensors (Internet of Things)

•Data Engineering•Improve Data management from SOA to SOKU context•Security-Privacy (by design)

•Lifecycle Management

High Priority Component Technologies:•User interfaces

•Hide complexity•Enable personalization•Allow service composition by users

•Intelligent Networking •Network semantics•Support SOA•Enable QoS by design

•Computer Architectures•Systems on Chip •Towards Networked Sensors (Internet of Things)

•Data Engineering•Improve Data management from SOA to SOKU context•Security-Privacy (by design)

•Lifecycle Management

High Priority Non-Functional Requirements:•Usability•Mobility•Performance•Manageability•Security-Privacy

High Priority Non-Functional Requirements:•Usability•Mobility•Performance•Manageability•Security-Privacy

High Priority Component Technologies:•Self-*

•Autonomic Infrastructures•Manageability of large scale distributed platforms

•Formal methods and architectural languages•User-centric communication•Transition to knowledge-based environments

•Measuring, Metering, controlling and reasoning of NFRs •Ensure functionality and scalability

•Lifecycle management and governance

High Priority Component Technologies:•Self-*

•Autonomic Infrastructures•Manageability of large scale distributed platforms

•Formal methods and architectural languages•User-centric communication•Transition to knowledge-based environments

•Measuring, Metering, controlling and reasoning of NFRs •Ensure functionality and scalability

•Lifecycle management and governance

High Priority Non-Functional Requirements:•Flexibility - Manageability•Scalability•Dependability•Interoperability

High Priority Non-Functional Requirements:•Flexibility - Manageability•Scalability•Dependability•Interoperability

Mid term Vision: DEMOCRATISATION OF GRIDS•Ubiquity •Interconnectedness at any level•Pervasiveness•Life in the Open and "on the move"

Long term Vision: From Core Sciences to Cognition

•Knowledge-based communication•Pervasive infrastructures•Grid-enabled mission-critical applications

SOASOA SOKUSOKU

PresentPresentMid-term Future

(5-years)Mid-term Future

(5-years)Long-term Future

(10-15 years)Long-term Future

(10-15 years)

©STFC/Keith G Jeffery The Future of GRIDs: A European Perspective 20080410 67Brussels March 2008 CHALLENGERS Consortium 67

The CHALLENGERS Core Group:

• Dora Varvarigou, ICCS/NTUA

• Santi Ristol, Atos Origin

• Keith Jeffery, STFC

• Stefan Wesner, HLRS

• Colin Upstill, IT Innovation

• Domenico Laforenza, ISTI/CNR

• Michel Riguidel, ENST

• Jarek Nabrziski, PSNC

• John Barr, Group 451• Theo Dimitrakos


CHALLENGERS SSA: CHALLENGERS SSA: Towards a Research Agenda and Towards a Research Agenda and

A Roadmap for the coming A Roadmap for the coming DecadeDecade

IST- 034128

A personal contribution…


Background Rationale

– Recognises the current view of the end-point (SOKU)– Recognises the current state-of-the-art– Takes into account the outputs from previous

Challengers workshops– Takes into account what is happening in the world in the

ICT area specifically in e-infrastructure, GRIDs, SOA etc

– (with acknowledgement to NGG, EC-Funded GRIDs projects, NESSI, 3S and others)


SOKU Revisited

The key point is that SOKUs are self-standing, self-managing services that can self-compose and/or be composedThis implies they either:

– Cooperate with some composer (orchestrator, choreographer) via an interface

• discovery, composition, execution, monitoring, recomposition

or– Manage themselves dependent on parameters

via an interface (and their own intelligence)– Cooperate with others via parameters passed

through an interface (shared intelligence)– In both composition and execution phases


Note

The SOKU (services) can be ‘wrapping’:

– Fully composed services end-to-end from user to e-infrastructure (servers, data stores, communications)

– Data or information (or knowledge) sources– Software sources– Physical hardware resources (servers,

communications, detectors, user devices)– Persons -in roles- (agents)– …….


SOKU key point

Interfaces defined by metadata:– Schema (to assure correctness e.g. database schema or formal

functional signature)– Navigational (to reach it e.g. URL)– Associative (to use it)

• Descriptive (what the service can do including any functional limitations e.g. precision, accuracy)

• Restrictive (conditions to use the service like cost, rights (trust, privacy, security), performance parameters,…

– And supported by • Supportive (domain-level (not service-level) metadata such as

thesauri, domain ontologies…

The metadata is used at composition and execution timeThe metadata provides the flexibility for self-*The metadata allows intercommunication through the software stack so that service levels for NFRs can be realised This may involve dynamic recomposition (i.e. self-*)


‘Strawman Roadmap’

2006 20182012 20152009

SOKU Apps

SOKU middleware

Web & GRID Services

Ontology Services

OGSA

Web Services

G-Lite

Globus

Unicore

Linux

Windows

Linux-WS

Win-WS

Linux-GS

Note: coloured lines indicate predicted transitions to lead to the vision


Key Points

WS W/G S– (since 2002)

GLOBUS GLite– Increase usability

(EGEE EGI)– European convergence

W/G S SOKU– Improved metadata

Linux Linux WS– Compose OS functions with

application dynamically2006 20182012 20152009

SOKU Apps

SOKU middleware

Web & GRID Services

Ontology Services

OGSA

Web Services

G-Lite

Globus

Unicore

Linux

Windows

Linux-WS

Win-WS

Linux-GS

Note: coloured lines indicate predicted transitions to lead to the vision


STRUCTURE



And behind it all….CS issues

Trust– Security

– Privacy

– Trust

Ease of use (expressivity, representativity)– Metadata

– Agents

– Brokers

Conclusions


And behind it all….CS issues

Mobile code– Be able to move the code to the data rather than data to the code– Share code among nodes active in one request (distribution /

parallelism)

Performance and optimisation– Over computation, network, data source

Synchronisation, consistency, reliability– For mobile devices (PDAs, detectors, instruments…)– In a dynamic environment

Conclusions


What does this mean for ISE?

Both Data modelling and System ImplementationRepresentation of real world

– Connected graphs not only hierarchies (to represent real world)• ‘hierarchies represent a limitation of the human view of complex structures’

– Complex semantics (systems need to understand for autonomy)• Domain ontologies to provide supportive metadata for interoperability• well-formed richly-structured syntax to permit tractable programming

– Business processes (because they evolve)• general code and execute-time binding of data • Configured dynamically by representative metadata

Global state– Local state and ‘active interfaces’ between local stateful systems

Conclusions



Transactions (represent real world, not idealised CS world)– Complex, multi-level, local but with open-ended properties

• E.g. Orders, invoices…

– Messaging interfaces

Completeness– Dealing with incomplete information for decision-making– Probability, fuzziness, learning systems

Certainty– Dealing with uncertain information– Probability, fuzziness, learning systems

Optimisation– No global state so local, maybe wider with negotiation– Partitioning, approximation, much use of metadata

Conclusions



Trust– Representation of business organisations, their policies (how?)– Contracts and proposals (some progress)– Servicel level agreements (some progress)

Security– System availability and continuity under attack– Prevention of unauthorised system access– Authentication and authorisation – global or connected local systems

Privacy– Openness of personal data to data subject (right to correctness)– Security of personal data to others (right to privacy)

Unacceptable Use– What does a spam transaction look like?– Hijacking of system by ‘adult, political, racist…. transactions’

Conclusions


Conclusions

The UK e-Science programme led the wayNGG1,2,3… provides a courageous vision

With many challenging R&D topics

aims to provide the architecture of the ICT environment of the future

for business and science, for healthcare and environment, for culture and leisure

Challengers provides the roadmap to 2025


Prof. Keith G Jeffery CEng, CITP, FBCS, FGS, HFICSDirector, IT & International Strategy

STFC Rutherford Appleton Laboratory

[email protected]

Documents

©STFC/Keith G JefferyThe Future of GRIDs: A European Perspective 20080410 1 The Future of GRIDs: A European Perspective Keith G Jeffery Science and Technology