Rich Miller Surendra Reddy

Infrastructure 2.0 Working Group January 20, 2010

Lighthouse: Intercloud Metadata Service

Agenda

•  Intercloud & Lighthouse Objectives

•  Use cases (as drivers & definition)

•  Lighthouse Requirements & Concepts

•  Available technologies & standards

•  Architectural Guiding Principles

•  Call(s) to action

Intercloud & Objectives

Intercloud

Requires the dissemination & exchange of operational metadata - among clouds, - between cloud services and consumers of their services.

Lighthouse

Lighthouse

Where to start? •  Agreement on identification, location

and ID-Loc resolution •  A registry for the discovery and

description of intercloud constituents •  A mechanism for the delivery of cloud

service descriptive & operational data •  A governance structure for

admission & ejection, assurance, permissions & entitlements

Lighthouse

The concept: •  Each member takes responsibility for

its own metadata access services •  Membership in a communal registry of

metadata access services, with identification – location resolution

•  Agreement on mechanisms for - pub/sub/search/query - asynchronous message delivery

Lighthouse Scope

Scope is limited to providing the Service Access Point and related metadata to service Consumers

Use Cases

Intercloud: Use Case #1

•  Customer A, EDA company, seeks a list of IaaS services which claim to provide:

•  cloud data management •  Linux OS image management

•  Queries the Intercloud registry, returns IDs of services that meet criteria

•  Searches IaaS service metadata to make selection •  Access the Service Access Point (SAP) of a

vendor to validate claims •  Subscribes to Service Access Point for receipt of

service announcements, rate changes, etc

Intercloud: Use Case #2

•  Customer B, an insurance company, seeks a single IaaS provider to continuously satisfy service requirements (constraints)

•  E.g. latencies, geography, SLAs etc. •  Queries the Intercloud registry,

returns IDs of services that meet criteria •  Searches IaaS metadata to make selection •  Access the SAPs of vendor to create

Cloud Service Account Instance •  Subscribes to SAP for receipt of relevant

requirement-specific metadata •  Takes specific actions based on timely notifications

(near realtime alerts) via Service Provider APIs and management functions

Intercloud: Use Case #3

•  Customer C, a globally distributed online service looking for an IaaS Providers in Europe and in USA with specific SLAs. •  Using the Intercloud registry, locates services

meeting needs in two locations. •  Identifies alternative providers for the business

continuity (DR, backup, …) functions. •  Customer C’s application management system

subscribes to failure events & performance sensors from the IaaS providers.

•  Based monitored event/sensor feeds, C’s service monitoring application dynamically scales up/down the resources (computing, networking, and storage) to their applications

Intercloud: Use Case #4

•  Customer D, a financial services company, runs applications that are either (or both)

•  latency sensitive •  throughput sensitive

•  After selecting IaaS provider: •  Sets up the virtual network between on-premise

data center and the IaaS provider cloud. •  Customer D runs their own application mobility

controller within their data center. •  Application Mobility Controller subscribes to

IaaS and data center metadata related to: •  traffic flows, performance metrics •  log feeds from the IaaS cloud service.

Intercloud: Use Case #5

•  PaaS E, a security broker service, provides an anti-phishing service for e-mail: “whitelist”, analytics and forensics

•  Operates on behalf of domain holders •  A list management and forensics for multiple

receiver services (e.g. web mail services) •  After establishing service w/ receiver:

•  Each receiver establishes a metadata access point (MAP) regarding failed email

•  PaaS E publishes notifications of phishing attempts to subs, on behalf of domain holder

•  All new events and changes in state/status distributed as pub/sub metadata

Lighthouse: Requirements & Concepts

Lighthouse Requirements

•  Defines a dynamically extensible set of identifiers and metadata

•  Automatically aggregates and associates real-time info from many different sources

•  Provides real-time pub/sub/search mechanism for data regarding cloud instances, their state and their activities

•  Scales for cloud to cloud coordination

Lighthouse Concept

Autonomous Metadata Access Point

•  All interested and authenticated cloud services, acting in ‘good faith’, provide their own Metadata Access Point.

•  A Metadata Access Point publishes to the intercloud community any information about itself.

Lighthouse Concept

A Registry of Registries

•  Identity and location of individually and autonomously managed Metadata Access Services

•  Authoritatively establishes the status of any individual cloud service and its standing within the Intercloud community

Lighthouse Concept

Process / Event Coordination

•  All 'interested' consumers of a cloud’s MAP Service may subscribe to metadata updates that result in a 'property' change

•  Many systems can coordinate through a Metadata Access Protocol with no in-depth knowledge of each other's APIs

Lighthouse Concept

Share operational metadata

•  Near Real-time

•  Cloud Information Service + Cloud Operations Coordination

Intercloud Registry: Features

•  Discovery of a registry’s specific interfaces / capabilities

•  Auditable logging mechanism •  For element / value changes •  For publishing event

Intercloud Registry: Features

Forms of Search & Query •  search and report of items based on

(…) •  comparison of object to ‘checklist’ of

elements and parameters •  ‘standing’ search/query established as

subscription •  query and retrieval of items based on

published / recognized (?) data scheme

Intercloud Registry: Operational

•  Distributed MAP Servers: Each Cloud Service is responsible for establishing and administering •  its own Registry Server, or •  publication of metadata by a trusted party

•  Authoritative compilation of Registries (and, therefore, of Cloud Services) •  Unambiguous identification •  Authentication method associated with ID

Available Standards

Current Standards/Protocols

Federated UDDI Registry • Pros:

•  Federated UDDI consisting of multiple repositories that are synchronized periodically.

•  Federated UDDI is an efficient solution for service discovery in distributed service networks.

• Cons: •  too expensive to replicate frequently updated

information •  it is hard to directly utilize this approach to support

discovery of dynamic information •  Governance nightmare…

Current Standards/Protocols

Service Location Protocol (SLP) • Pros:

•  agent based service discovery framework •  designed for service discovery in for local area

network •  extensions to SLP proposed aiming to the WAN

environment • Cons:

•  Not suitable for wide area network environment •  unsuitable for Cloud environment due to the scale

and distribution complexities involved.

Current Standards/Protocols

IF-MAP • Pros:

•  Client-Server based, real-time pub/sub/search •  Designed to disseminate network security info on

objects & events (dynamic state and activity data) •  Easily extensible to components other than network

and security components •  XML-based SOAP protocol •  Supports standardized, dynamic data interchange •  Provides an uniform mechanism to securely

discover, consume, and manage a single management domain’s metadata.

Current Standards/Protocols IF-MAP (continued) •  Cons:

•  SOAP based only, heavy messaging structure •  Scale for Cloud •  Need lot of extensions to existing metadata model •  IF-MAP access point becomes a central authority

•  TBD •  Federation to support intercloud scale? •  Wider range of protocols / RESTful interface? •  A MAP-to-MAP (P2P) approach to bi-directional

pub/sub? •  Asynch messaging queues? •  “Economical” message encoding system ?

hierarchical, binary, self-describing

Current Standards/Protocols Other Standards/Protocols to Consider

•  WebDAV/DASL •  DAV Provides Versioned Metadata

Access of Resources •  DASL: Provides Searching and Location

Current Standards/Protocols And, what about asynchronous messaging? •  AMQP •  Session Initiation Protocol (SIP) •  XMPP •  HTTP •  JMS

Not to mention message encoding… •  ASN.1 •  FUDGE •  …

Lighthouse: Architectural Model

Lighthouse: Metadata Model

Lighthouse: Conceptual Architecture 1

CSP MAP

Cloud Service Provider

Metadata Access Point

CSP CSP

IC-MAP "

InterCloud Registry

IC Registry Metadata

Access Point

Lighthouse: Conceptual Architecture 2

CSP IC

MAP

Cloud Service Provider

CSP

CSP

IC-ROOT

IC Metadata

Access Point

InterCloud Registry "

IC Registry Metadata

“Root Server”

Rich Miller Surendra Reddy

Infrastructure 2.0 Working Group

Lighthouse: Call(s) to Action