Rich Miller & Surendra Reddy Lighthouse is a concept for the creation of an intercloud registry service, based on (1) access points established and maintained by cloud instances to disseminate operational metadata; and, (2) the use of publish/subscribe (pub/sub) asynchronous messaging as the dominant means of disseminating operational metadata among the constituents of the intercloud.
- 1. Lighthouse: Intercloud Metadata Service Rich Miller Surendra ReddyInfrastructure 2.0 Working GroupJanuary 20, 2010
2. Agenda Intercloud & Lighthouse Objectives Use cases (as drivers & definition) Lighthouse Requirements & Concepts Available technologies & standards Architectural Guiding Principles Call(s) to action 3. Intercloud & Objectives 4. IntercloudRequires the dissemination & exchange of operational metadata - among clouds, - between cloud services and consumers of their services. 5. Lighthouse 6. Lighthouse Where to start? Agreement on identification, locationand ID-Loc resolution A registry for the discovery anddescription of intercloud constituents A mechanism for the delivery of cloudservice descriptive & operational data A governance structure foradmission & ejection, assurance,permissions & entitlements 7. Lighthouse The concept: Each member takes responsibility forits own metadata access services Membership in a communal registry ofmetadata access services, withidentification location resolution Agreement on mechanisms for- pub/sub/search/query- asynchronous message delivery 8. Lighthouse ScopeScope is limited to providing the Service Access Point and relatedmetadata to service Consumers 9. Use Cases 10. Intercloud: Use Case #1 Customer A, EDA company, seeks a list ofIaaS services which claim to provide: cloud data management Linux OS image management Queries the Intercloud registry, returns IDs of services that meet criteria Searches IaaS service metadata to make selection Access the Service Access Point (SAP) of a vendor to validate claims Subscribes to Service Access Point for receipt of service announcements, rate changes, etc 11. Intercloud: Use Case #2 Customer B, an insurance company, seeks asingle IaaS provider to continuously satisfyservice requirements (constraints) E.g. latencies, geography, SLAs etc. Queries the Intercloud registry, returns IDs of services that meet criteria Searches IaaS metadata to make selection Access the SAPs of vendor to create Cloud Service Account Instance Subscribes to SAP for receipt of relevant requirement-specific metadata Takes specific actions based on timely notifications (near realtime alerts) via Service Provider APIs and management functions 12. Intercloud: Use Case #3 Customer C, a globally distributed onlineservice looking for an IaaS Providers in Europeand in USA with specific SLAs. Using the Intercloud registry, locates servicesmeeting needs in two locations. Identifies alternative providers for the businesscontinuity (DR, backup, ) functions. Customer Cs application management systemsubscribes to failure events & performance sensorsfrom the IaaS providers. Based monitored event/sensor feeds, Cs servicemonitoring application dynamically scales up/downthe resources (computing, networking, and storage)to their applications 13. Intercloud: Use Case #4 Customer D, a financial services company,runs applications that are either (or both) latency sensitive throughput sensitive After selecting IaaS provider: Sets up the virtual network between on-premise data center and the IaaS provider cloud. Customer D runs their own application mobility controller within their data center. Application Mobility Controller subscribes to IaaS and data center metadata related to: traffic flows, performance metrics log feeds from the IaaS cloud service. 14. Intercloud: Use Case #5 PaaS E, a security broker service, provides ananti-phishing service for e-mail:whitelist, analytics and forensics Operates on behalf of domain holders A list management and forensics for multiple receiver services (e.g. web mail services) After establishing service w/ receiver: Each receiver establishes a metadata access point (MAP) regarding failed email PaaS E publishes notifications of phishing attempts to subs, on behalf of domain holder All new events and changes in state/status distributed as pub/sub metadata 15. Lighthouse: Requirements & Concepts 16. Lighthouse Requirements Defines a dynamically extensible set ofidentifiers and metadata Automatically aggregates and associatesreal-time info from many different sources Provides real-time pub/sub/searchmechanism for data regarding cloud instances,their state and their activities Scales for cloud to cloud coordination 17. Lighthouse ConceptAutonomous Metadata Access Point All interested and authenticated cloudservices, acting in good faith, providetheir own Metadata Access Point. A Metadata Access Point publishes tothe intercloud community anyinformation about itself. 18. Lighthouse ConceptA Registry of Registries Identity and location of individually andautonomously managedMetadata Access Services Authoritatively establishes the status ofany individual cloud service and itsstanding within the Intercloudcommunity 19. Lighthouse ConceptProcess / Event Coordination All 'interested' consumers of a cloudsMAP Service may subscribe tometadata updates that result in a'property' change Many systems can coordinate througha Metadata Access Protocol with no in-depth knowledge of each other's APIs 20. Lighthouse ConceptShare operational metadata Near Real-time Cloud Information Service+Cloud Operations Coordination 21. Intercloud Registry: Features Discovery of a registrys specificinterfaces / capabilities Auditable logging mechanism For element / value changes For publishing event 22. Intercloud Registry: FeaturesForms of Search & Query search and report of items based on() comparison of object to checklist ofelements and parameters standing search/query established assubscription query and retrieval of items based onpublished / recognized (?) data scheme 23. Intercloud Registry: Operational Distributed MAP Servers: Each Cloud Service is responsible for establishing and administering its own Registry Server, or publication of metadata by a trusted party Authoritative compilation of Registries(and, therefore, of Cloud Services) Unambiguous identification Authentication method associated with ID 24. Available Standards 25. Current Standards/Protocols Federated UDDI Registry Pros: Federated UDDI consisting of multiple repositories that are synchronized periodically. Federated UDDI is an efficient solution for service discovery in distributed service networks. Cons: too expensive to replicate frequently updated information it is hard to directly utilize this approach to support discovery of dynamic information Governance nightmare 26. Current Standards/Protocols Service Location Protocol (SLP) Pros: agent based service discovery framework designed for service discovery in for local areanetwork extensions to SLP proposed aiming to the WANenvironment Cons: Not suitable for wide area network environment unsuitable for Cloud environment due to the scaleand distribution complexities involved. 27. Current Standards/Protocols IF-MAP Pros: Client-Server based, real-time pub/sub/search Designed to disseminate network security info onobjects & events (dynamic state and activity data) Easily extensible to components other than networkand security components XML-based SOAP protocol Supports standardized, dynamic data interchange Provides an uniform mechanism to securelydiscover, consume, and manage a singlemanagement domains metadata. 28. Current Standards/Protocols IF-MAP (continued) Cons: SOAP based only, heavy messaging structure Scale for Cloud Need lot of extensions to existing metadata model IF-MAP access point becomes a central authority TBD Federation to support intercloud scale? Wider range of protocols / RESTful interface? A MAP-to-MAP (P2P) approach to bi-directionalpub/sub? Asynch messaging queues? Economical message encoding system ? hierarchical, binary, self-describing 29. Current Standards/Protocols Other Standards/Protocols to Consider WebDAV/DASL DAV Provides Versioned MetadataAccess of Resources DASL: Provides Searching and Location 30. Current Standards/Protocols And, what about asynchronous messaging? AMQP Session Initiation Protocol (SIP) XMPP HTTP JMS Not to mention message encoding ASN.1 FUDGE 31. Lighthouse: ArchitecturalModel 32. Lighthouse: Metadata Model 33. Lighthouse: Conceptual Architecture 1Cloud Service ProviderCSP CSP CSPMAPMetadata Access PointIC- MAP"InterCloud Registry IC RegistryMetadataAccess Point 34. Lighthouse: Conceptual Architecture 2 Cloud Service ProviderCSPCSPCSP IC MAPInterCloud Registry " IC IC- ROOTMetadata Access Point IC Registry Metadata Root Server 35. Lighthouse: Call(s) to ActionRich Miller Surendra ReddyInfrastructure 2.0 Working Group