Cloud Portability, Lifecycle Management and more!

@mrutkowski

Wednesday, 18 May, 2016 @ 11:00 AM EDT

Matt Rutkowski IBM STSM, Cloud Open Technologies OASIS TOSCA Chair, Simple Profile WG

2

▪What is TOSCA? ▪ milestones & participation

▪What Makes TOSCA Unique? ▪ intent model

▪Key Modeling Concepts Topology, Composition, Portability, Lifecycle (management), Policy

▪ TOSCA’s Growing Eco-System ▪ in open source & standards

▪What’s Next ▪ work group activities, version 1.1

An important Open standard, that is enabling a unique Cloud eco-system supported by a large and growing number of international industry leaders…

TOSCA uses a domain-specific language (DSL) to define interoperable descriptions of :

• Cloud applications, services, platforms, infrastructure and data components, along with their relationships, requirements, capabilities, configurations and operational policies…

• …thereby enabling portability and automated management

across cloud providers regardless of underlying platform or infrastructure thus expanding customer choice, improving reliability and time-to-value while reducing costs.

3

Associated Companies • TOSCA Version 1.0 Specification approved as an OASIS Standard

— published Nov 2013, XML format

• TOSCA Simple Profile v1.0 Specification (YAML format)

— final public review, ended March 2016, towards OASIS Standard

— TOSCA Simple Profile v1.1 Specification (target: June 2016)

Supports Domain-Specific Profile Specifications:

– Network Function Virtualization (NFV) Profile v1.0

• Government and Corporate Awareness:

– OASIS: 600+ participant organizations. 5000+ participants spanning 65+ countries

– TOSCA Committee: 170+ people 45+ companies/orgs

– International Standards & Research: ISO/IEC JTC 1 liaison, EU FP7, ETSI NFV liaison, etc.

• Multi-company Interoperability Demonstrated:

– EuroCloud 2013, Open Data Center Alliance 2014, OSCON 2015, OpenStack Summit 2016 (Indigo DataCloud)

4

Includes contributors, reviewers, implementers, users or supporters of the TOSCA Standard via OASIS

incorporates both Data and Information Model features and concepts …

… but brings unique orchestration concepts focus in Lifecycle mgmt. and State

Information Models Typically, used to model a constrained

domain that can be described by a closed set of entity types, properties, relationships and operations.

Data Models Typically, describe the structure (format), enabling manipulation (via interfaces) of the data stored in data management systems assuring integrity.

• Topology • Composition • Requirements - Capabilities • State (Nodes, Relationships) • Lifecycle (Management) • Policy

Intent Model Adds:

TOSCA is an Intent Model which is declarative (integration points for imperative)

• Structure • Format • interfaces

• Types, Relationships • Properties • Operations

TOSCA is can work with imperative scripts (e.g., Ansible, Chef, Bash, Ant, etc.)

TOSCA can include other data models (e.g., JSON, YANG)

• Topology • Composition

• Portability

• Lifecycle

• Policy

Tier (Group Type)

TOSCA is used first and foremost to describe the topology of the deployment view for cloud applications and services

7

source_resource

Node_Type_A

target_resource

Node_Type_B

Requirement

connect_relationship

ConnectsTo Capability

Nodes - are the resources or components that will be materialized or consumed in the deployment topology

Relationships express the dependencies between the nodes (not the traffic flow)

Node templates to describe components in the topology structure

Relationship templates to describe connections, dependencies, deployment ordering

Requirement - Capability Relationships can be customized to match specific source requirements to target capabilities

Groups Create Logical, Management or Policy groups (1 or more nodes)

8

TOSCA Service Template (container) Application Tier

(container)

Web Server (container)

Web App

PHP Module

Database Tier (container)

DB Server (container)

Database

Service Templates provide the “container” to exchange and reuse topologies: • Reusable models extend investments by making it easy to compose more valuable

and complex apps from existing apps • Determines dependency boundaries to maximize parallelism of deployments • Models (dependencies) can be validated by automation to ensure application-aware,

policy-aligned configuration, deployment and operational semantics

Cont

ainm

ent

Connectivity

Example: a simple, 2-Tier Cloud application expressed in a TOSCA Service Template

• Topology

• Composition • Portability

• Lifecycle

• Policy

Application Tier (container)

Application Tier

(container)

10

Logging/Monitoring Tier (ELK)

nodejs

WebServer

app_server

Compute

paypal_pizza store

WebApplication

collectd

logstash SoftwareComponent

Requirements

Container

Capabilities log_endpoint

logstash_server

Compute

Capabilities

Container

elasticsearch SoftwareComponent

Requirements Container

Capabilities search_endpoint

elasticsearch _server

Compute

Capabilities

kibana SoftwareComponent

Requirements

Container

kibana_server

Compute

Capabilities

search_endpoint

ConnectsTo

HostedOn HostedOn HostedOn

ConnectsTo

mongo_dbms

DBMS

mongo_server

Compute

mongo_db

Database

rsyslog

search_endpoint

Container Container

ConnectsTo

Example: Connect a Logging / Monitoring Service composed of ElasticSearch, LogStash and Kibana (ELK)

Enabling the description of complex, multi-tier (hybrid) Cloud applications

Analytics Service

(Topology)

Cloud Application (Topology)

Orchestrators can “substitute” for abstract nodes… … as long as all declared “requirements” are met: • Monitoring Service can be substituted in Cloud Application • Analytics Service can be substituted in Monitoring Service

Abstract nodes in one TOSCA topology can be substituted with another topology

Monitoring Service

(Abstract)

Java Application

Web Application

Server

SQL Datastore

Monitoring Service (Topology)

Collector

Logger

Monitoring Framework

Analytics Service

(Abstract)

Analytics Engine

Hadoop Service Template 1

Service Template 2

Service Template 3

• Topology

• Composition

• Portability • Lifecycle

• Policy

TOSCA Service Template

Storage

Compute1

DB

Compute2

App

Network

Scaling Policy

TOSCA’s defines Normative Types for different domains, for example:

Application, IaaS Types are part of “core” specification

e.g., Web Server, Database, Compute, Block Storage, Network

Cloud Application’s declarative modelled from these normative types …

… Can be understood by any Cloud Provider

unfulfilled Application Requirements

can be exported for Orchestrators to fulfill

Templates include (or reference) all necessary configuration and Infrastructure requirements

TOSCA applications, using normative types, are portable to different Cloud infrastructures

TOSCA Meta-Model Normative Types

Nodes • Properties • Attributes

Relationships • Properties • Attributes

Capabilities

Interfaces (Operations)

Groups

Policies

Requirements

Interfaces

com

pose

d fr

om

base

d up

on

Example: TOSCA applications are portable to different Cloud infrastructures

Application Requirements

TOSCA Orchestration

TOSCA Service Template

Storage

Compute1

DB

Compute2

App

Network

Scaling Policy

Cloud Provider C

Cloud Provider A

Cloud Provider B

by expressing application Requirements…

independently from cloud provider Capabilities…

& Optimization Automatic Matching

Infrastructure Capabilities

Orchestrators concern themselves dealing with disparate cloud APIs 14

• Topology

• Composition

• Portability

• Lifecycle • Policy

TOSCA models have a consistent view of state-based lifecycle have Operations (implementations) that can be sequenced against state of any dependent resources fits into any Management Framework or Access Control System

my_resource_name

My_Resource_Type

Lifecycle.Standard

create

configure

start

stop

delete

Standardize Resource Lifecycle Standardize Relationship Lifecycle Lifecycle Customization

source_resource

Type_A

A

target_resource

Type_B

B

my_relationship

ConnectsTo

Ope

ratio

ns Lifecycle.Configure

pre_config_target

post_config_target

add_target

remove_target

pre_config_source

post_config_source

add_source

remove_source

Operations

Lifecycle.Configure.NFV

Ope

ratio

ns

Lifecycle.Standard

create

configure

start

stop

delete

Create new Lifecycles or Augment existing (via subclassing)

pre_config

pre_delete

16

my_resource_name

My_Resource_Type

Lifecycle.Standard

create

configure

start

stop

delete

Node Lifecycle

Ope

ratio

ns

Implementations (e.g., imperative scripts) can be bound to operations.

source_resource

Type_A

A

target_resource

Type_B

B

my_relationship

ConnectsTo

Lifecycle.Configure

pre_config_target

post_config_target

add_target

remove_target

pre_config_source

post_config_source

add_source

remove_source

Operations

The Orchestrator moves the nodes through their Lifecycle States by executing their Lifecycle Operations in topological order • Orchestrators can work to deploy nodes in parallel based upon node relationships

Relationship Lifecycle Nodes have their own Lifecycle

Operations which are invoked in order to achieve a target state

Relationships also have their own Lifecycle Operations to configure or allocate and de-configure or deallocate Node related resources

• Topology

• Composition

• Portability

• Lifecycle

• Policy

v1.0 includes the groundwork for Placement (Affinity), Scaling and Performance Policies ‒ Orchestrators can evaluate Conditions based on Events that trigger Automatic or Imperative Actions

Policies can be declared independently and ttached to various points in your models 1. That can be attached to Interfaces or specific Operations, 2. Nodes and 3. Groups of Nodes

my_app_1

Compute

Capabilities

Container

... Lifecycle

create

configure

...

Policy • Type • Event, Condition • Action

my_scaling_group

backend_app

Compute

Policy • Type • Event, Condition • Action

my_database

Compute web-app

Compute Policy • Type • Event, Condition • Action

1

2

3

Scaling

“Policies are non-functional Requirements independent of nodes”

TOSCA Policy Definition (e.g., Placement, Scaling, Performance) : <policy_name>: type: <policy_type_name> description: <policy_description> properties: <property_definitions> # allowed targets for policy association targets: [ <list_of_valid_target_resources> ] triggers: <trigger_symbolic_name_1>:

event: <event_type_name>

target_filter: node: <node_template_name> | <node_type> # (optional) reference to a related node # via a requirement requirement: <requirement_name> # (optional) Capability within node to monitor capability: <capability_name> # Describes an attribute-relative test that # causes the trigger’s action to be invoked. condition: <constraint_clause>

action: # implementation-specific operation name <operation_name>: description: <optional description> inputs: <list_of_parameters> implementation: <script> | <service_name> ...

Event • Name of a normative TOSCA Event Type • That describes an event based upon a

Resource “state” change. • Or a change in one or more of the

resources attribute value.

Condition Identifies: • the resource (Node) in the TOSCA

model to monitor. • Optionally, identify a Capability of the

identified node. • Describe the attribute (state) of the

resource to evaluate (condition)

1..N

Trig

gers

can

be

decl

ared

Describes: • An Operation (name) to invoke when

the condition is met • within the declared Implementation • Optionally, pass in Input parameters to

the operation along with any well-defined strategy values.

Action

• Reference by other Standards

• Open Source • OpenStack

22

Topology, Type & LCM Design http://alien4cloud.github.io/

alien4cloud

Service Orchestration & Management http://getcloudify.org/

Data/computing platform targeted at scientific communities

http://information-technology.web.cern.ch/about/projects/eu/indigo-datacloud

https://wiki.openstack.org/

Heat-Translator (IaaS, App Orchestration) Tacker (Network Function Orchestration)

http://ariatosca.org//

Multi-Cloud Orchestration (Amazon, Azure, VMware, OpenStack)

Open Sourced from Cloudify

www.seaclouds-project.eu/media.html Open, Multi-Cloud Management

Parser Deployment Template Translation

https://wiki.opnfv.org/display/parser/Parser Note: ETSI NFV ack. TOSCA can be used as an input model/format

TOSCA-Parser

TOSCA Plugin

TOSCA Integration

apps.openstack.org

OpenStack Client (OSC) Plugin

Senlin Clustering & Policy

(on roadmap)

Tacker NFV Orchestration Parser

Heat-Translator App. Orchestration

https://wiki.openstack.org/

Core

Compute (Nova)

Storage (Cinder, Swift)

Network (Neutron)

Shar

ed

Image (Glance)

Database (Trove)

Identity (Keystone) … Metering

(Ceilometer) Orchestration (Heat / HOT)

(Heat-Translator / TOSCA)

HOT Pattern Domain

TOSCA CLI App. Catalog

Library export

https://pypi.python.org/pypi/heat-translator https://pypi.python.org/pypi/tosca-parser

• Interoperability (Conformance) • Goal: Conformance test suite for v1.0; includes tests for each section of Simple Profile v1.0 specification.

• Each test is a TOSCA Service Templates with metadata describing test using the OASIS Test-Assertion (TAG) Standard • Work underway to publish in new GitHub repo., announcement (target ~May 2016)

• Container (Clustering) • Goal: Finish new Cluster capability definitions, Data Cluster use cases. for Simple Profile v1.1

• Instance Model • Goal: new schema for an Instance Model (reuse existing schema where possible)

• Discussing API potentially enabling capture, export and management of deployed application

• Monitoring • Goal: Create normative event types for basic operational events

• Focus on events types for Health, Scaling & Performance • Support basic “Red-Yellow-Green” and Percentage-based monitoring for dashboards

• Network Function Virtualization (NFV) • Expanded Scope: include Software-Defined Network (SDN) use cases • Goal: Complete v1.0 Specification, v1.0 Public Review Draft 3 Published (17 March 2016)

• Can model complete ETSI MANO specification: Network Services, Virtual Network Functions (NFV)s, Virtual Links, with Forwarding Paths,

• Orchestration demonstrated with OpenStack Tacker Project, multi-VNF use cases for next release

Specification Release Targets • Public Review Draft 01 - target June 2016

• “Final” Public Review Draft - target 3Q 2016

New Features • Metadata (completed)

• now supported in all Types (Node, Relationship, Capability, Data, etc.) • Conformance Testing metadata

• Group Type (completed) • Expanded Group Type to allow management of member resources (i.e., Lifecycle) • Has its own Capabilities and Requirements

• Policy Definition (completed) • Event-Condition-Action model • Includes Event Filters and Triggers

• Workflow (80% completed) • Intermix declarative with Imperative (e.g., Ansible, Chef, Ant, Bash) • Preserve investment in existing scripts for complex installations / configurations

• Cluster Type (75% completed) • Add support for Cluster normative type; based upon new Group Type • Will support new normative LoadBalancer , Scalable and Router Capability Types • Data Clusters (e.g., Cassandra, MongoDB, etc.) – In-Progress

25

26

• TOSCA Technical Committee Public Page (latest documents, updates, and more)

— https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=tosca

• OASIS YouTube Channel, TOSCA Playlist — https://www.youtube.com/user/OASISopen , http://bit.ly/1BQGGHm

• LinkedIn Group: “TOSCA OASIS Standard”: — https://www.linkedin.com/groups/8505536

• TOSCA Simple Profile in YAML v1.0 (final public review draft, 04, Feb. 2016) — http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.0/TOSCA-Simple-Profile-YAML-v1.0.html

• TOSCA Simple Profile for NFV v1.0 (latest public review draft, 17 March 2016)

– http://docs.oasis-open.org/tosca/tosca-nfv/v1.0/tosca-nfv-v1.0.html

• Contact the Technical Committee Co-Chairs: – Paul Lipton, paul.lipton@ca.com; Simon Moser, smoser@de.ibm.com

Q&A

Complete Source: https://github.com/openstack/heat-translator/blob/master/translator/toscalib/tests/data/tosca_single_server.yaml

tosca_definitions_version: tosca_simple_yaml_1_0 description: > Template for deploying a single server with predefined properties and input parameter topology_template: inputs: cpus: type: integer description: Number of CPUs for the server. constraints: - valid_values: [ 1, 2, 4, 8 ] node_templates: my_server: type: tosca.nodes.Compute capabilities: host: properties: num_cpus: { get_input: cpus } disk_size: 10 GB mem_size: 512 MB os: properties: architecture: x86_64 type: linux distribution: rhel outputs: server_address: description: IP address of server instance. value: { get_attribute: [server, private_address] }

Logical Network Model

AppServerTier

AppServer

App Component

DBServerTier

DBServer

APP DB

Required EndPoint

Provided EndPoint

HTTP Client

Application Architecture: Structure, dependencies, requirements, and interconnectivity

AppServerTier

DB Client Port

App Server Port

DBServerTier

DB Server Port

Private Logical

Network

Public Logical

Network

Network Technology Independent Representation

Concrete Network Model

Technology specific rendering of Logical Network

Logical Router Logical Switch 10.0.2.0/24

DB Client

Intf

Logical Router Logical Switch

X.X.X.X/24

App Server

Intf

Logical Switch

10.0.1.0/24 DB

Server Intf

Logical Switch

X.X.X.X/24

Edge Intf

Logical FW

Rules

Logical FW

Rules

HTTP Client

Edge Services Gateway

Service Topology Layered Topologies