Upload
cloudifysource
View
110
Download
2
Embed Size (px)
Citation preview
DICEHorizon 2020 Research & Innovation ActionGrant Agreement no. 644869http://www.dice-h2020.eu
Funded by the Horizon 2020Framework Programme of the European Union
DICE: quality Big Data made easy
Matej Artač @matej_artacXLAB Research @xlab_research
Michele Guerriero, Damian A. TamburriPolitecnico di Milano
2
Agenda
o Introductiono About DICE
Developing Data-Intensive Cloud Applications with Iterative Quality Enhancements
o DICERo TOSCAo Cloudifyo DICE delivery tools
3
Introduction
4
Building blocks for DIAs today
Coordinator (Kafka)
Orchestrator (Hadoop Cluster)
Data Store
Batch Layer
Speed Layer
Serving LayerServing LayerServing Layer
Data Source
Data Source
Distributedcomputation
Data streaming
HDFS
Distributed storage
Lambda architecture Cloud infrastructure
5
What problems EU SMEs face?
Traditional market: Legacy software systems
Customers with legacydata now ask for Big Data technologies
Growth in sight, but …
Learning curves
Initial prototyping
Risk of failure
(+ others…)
Fast-paced market
Traditional approach to deployment
o Spend time studying documentation
o Use trial and error to set up a working cluster
o Use incompatible public cookbooks
o Repeat for each changeo Keep the Big Data cluster
fixed for fear of breaking it
days
7
What is in the box?
o DICER model your application create a TOSCA blueprint
o Cloudify Pure-play orchestration & automation
o DICE deployment tool alternative front-end for Cloudify
o TOSCA library worries about deploying Big Data
services so that you don’t have to
8
DICEDeveloping Data-Intensive Cloud Applications with Iterative
Quality Enhancements
9
The Rapid Growth of Big Data
o Software market rapidly shifting to Big data 27% compound annual growth rate through 2017 (IDC) Popular technologies such as Spark, Hadoop, and NoSQL
boost Big Data adoption and revenues from new services
Business issue: 65% of Big data projects still fail (CapGemini’15)
Source: IDC Source: Wikibon
10
DevOps toolchains in innovation
Application Release Automation
Continuous Delivery
DevOps Toolchain
11
DICE Mission and Partners
ICT 9 Call/2014 – Software engineering 9 partners (Academia & SMEs), 7 EU countries
Mission: support SMEs in developing high-quality cloud-based data-intensive applications (DIAs)
(IEAT)
(IMP)
(PMI)
(ZAR) (NETF)
(XLAB)
(ATC)
(FLEXI)(PRO)
12
Ingredients of the DICE approach
o DevOps
o Model-Driven EngineeringDev Ops Dev Ops
Analysis
Deployment blueprint
13
DICE incremental modeling and analysis
DICE Platform Independent Model (DPIM)
DICE Technology Specific Model (DTSM)
DICE Deployment Specific Model (DDSM)
is implemented by
is deployed onto
TOSCA blueprint
Analysis
Analysis
Analysis & Optimization
M2M transformation
M2M transformation
M2T transformation
DICE Methodology
14
DICE deployment, monitoring and testing
Deployment
Testbed
Monitoring
Fault Injection
Quality Testing
Trace Checking
Enhancement
Anomaly
Detection
Running DIA
Comp
MW
VM
Running DIA
Comp
MW
VM
Configurationoptimization
TOSCA blueprint DICE M
ethodology
15
DICE architecture
IDEbased on
Eclipse
Profile
Simulation
Optimization
Verification
Repository & CI
Configuration Optimization
Delivery
Running DIA
Comp
Running DIA
Comp
Monitoring
Trace Checking
Enhancement
Anomaly Detection
Fault Injection(Resilience)
Quality Testing
MW
VMMW
VM
MW
VMRunning
DIA Comp
DICER
16
XLAB delivery tools
IDEbased on
Eclipse
Profile
Simulation
Optimization
Verification
Repository & CI
Configuration Optimization
Delivery
Running DIA
Comp
Running DIA
Comp
Monitoring
Trace Checking
Enhancement
Anomaly Detection
Fault Injection(Resilience)
Quality Testing
MW
VMMW
VM
MW
VMRunning
DIA Comp
DICER
17
DICERCreate actionable deployment diagrams of
Data Intensive Applications
18
DICER
o Assisted Component-based infrastructure designo 100% automation of model transformations
19
DICER in action: Assisted modelling!
1. <<Jerry: I’m modelling this DDSM thingie… I don’t know what I’m doing!>>
2. <<DICE: Hey Jerry! You’re missing this piece here… And there!>>
…
20
DICER in action: Assisted modelling!
3. <<DICE: ok that’s better now… Carry on…>>
21
OASIS TOSCAOASIS Topology and Orchestration Specification
for Cloud Applications
22
What is TOSCA?
o Open standardo Enabling a unique Cloud
eco-systemo Supported by a large
and growing number of international industry leaders
Associated Companies
TOSCA is an Intent Model which is declarative (integration points for imperative)
TOSCA Domain-Specific Language
Information ModelsTypically, used to model a constrained
domain that can be described by a closed set of entity types, properties,
relationships and operations.
Data ModelsTypically, describe the structure (format), enabling manipulation (via interfaces) of the data stored in data management systems assuring integrity.
• Topology• Composition • Requirements - Capabilities• State (Nodes, Relationships)• Lifecycle (Management)• Policy
Intent Model Adds:
• Structure • Format• interfaces
• Types, Relationships• Properties• Operations
TOSCA can work with imperative scripts (e.g., Ansible, Chef, Bash, Ant, etc.)
TOSCA can include other data models (e.g., JSON, YANG)
24
Tier (Group Type)
TOSCA is used first and foremost to describe the topology of the deployment view for cloud applications and services
Topology – Nodes and Relationships
source_resource
Node_Type_A
target_resource
Node_Type_B
Requirement
connect_relationship
ConnectsToCapability
Nodes - are the resources or components that will be materialized or consumed in the deployment topology
Relationships express the dependencies between the nodes (not the traffic flow)
Requirement - Capability Relationships can be customized to match specific source requirements to target capabilities
GroupsCreate Logical, Management or Policy groups (1 or more nodes)
Node templates to describe components in the topology structure
Relationship templates to describe connections, dependencies, deployment ordering
25
Application Tier (container)
Application Tier
(container)
Composition – different service templates can be “wired” together
Logging/Monitoring Tier (ELK)
nodejs
WebServer
app_server
Compute
paypal_pizzastore
WebApplication
collectd
logstashSoftwareComponent
Requirements
Container
Capabilitieslog_endpoint
logstash_server
Compute
CapabilitiesContainer
elasticsearchSoftwareComponent
RequirementsContainer
Capabilitiessearch_endpoint
elasticsearch _server
Compute
Capabilities
kibanaSoftwareComponent
Requirements
Container
kibana_server
Compute
Capabilities
search_endpoint
ConnectsTo
HostedOn HostedOn HostedOn
ConnectsTo
mongo_dbms
DBMS
mongo_server
Compute
mongo_db
Database
rsyslog
search_endpoint
ContainerContainer
ConnectsTo
Enabling the description of complex, multi-tier (hybrid) Cloud applications
26
Cloudify
27
Introducing CloudifyOpen | Extensible |
Simple
REPEATABLE
28
ApplicationBlueprint(TOSCA)
IaaSPlugins
ContainerPlugins
Conf MgmtPlugins
● Provision● Configure● Monitor● Manage
Monitoring &Alarming
29
30
Cloudify Key Aspects
Open SourceOpen source is key to drive innovation and create superb quality software.
Open StandardOpen standard-based TOSCA Spec for application blueprints allows vendor neutrality, and enables collaboration.
Future ProofTry new emerging technologies while using stable in place existing ones.
31
The only constant is change” -Unknown
32
DICE Delivery Tools
33
Components of Delivery Tools
RESTful API
IaaS
Web GUITechnology
Library
34
Delivery Service
Container 1
Blueprint A
Platform params
Container 2
Blueprint B
Blueprint B.2
Platform params
Container 15
Blueprint B.2
35
TOSCA technology library
o A plug-in for Cloudifyo A single import line in the TOSCA blueprinto Node types + Chef cookbooks for Big Data serviceso Unified across supported IaaS vendors
©DICE
36
Deploy your own Big Data services
DevOps approach:o Describe your Big Data
cluster and application in a blueprint
o Store and maintain the blueprint in your VCS with the application’s code: IasC
o Rely on orchestrators and configuration managers for executing deployments
hours
42
Demo – DevOps
43
Conclusion
44
Conclusion
o DICE tools remove barriers to Big Datao DICE technology library simplifies blueprintso TOSCA blueprints describe infrastructure as codeo Enabled Continuous Integration and Continuous
Delivery
45
Links
o Cloudify: http://getcloudify.org o DICE H2020: http://www.dice-h2020.eu/
o DICE deployment service:https://github.com/dice-project/DICE-Deployment-Service
o Big Data blueprint examples:https://github.com/dice-project/DICE-Deployment-Examples
o DICER:https://github.com/dice-project/DICER
46
Follow us
o DICE project: @diceh2020o Cloudify:
@CloudifySource – [email protected] User groups:
https://groups.google.com/forum/#!forum/cloudify-users o Webinars:
http://getcloudify.org/webinars.html
o Matej Artač: @matej_artac – [email protected] XLAB Research: @xlab_research
47
Q & A