BERLIN 2015 - Amazon Web Servicesaws-de-media.s3. Summit Berlin 2015...آ  Solution: • Stack membership

  • View
    0

  • Download
    0

Embed Size (px)

Text of BERLIN 2015 - Amazon Web Servicesaws-de-media.s3. Summit Berlin 2015...آ  Solution: • Stack...

  • BERLIN 2015

  • CI in the cloud

    Case Talend & Nordcloud

    Ilja Summala, Group CTO, Nordcloud

    Anubhav Sharma, Senior Software Architect, Talend Deutschland

  • About Nordcloud

    Nordcloud is leading European AWS

    consultancy with offices in Helsinki, Munich,

    London, Stockholm, Oslo and Malmö.

    Nordcloud is AWS Premier Consultancy

    Partner for 2015.

    We have completed over 200 cloud

    migration projects.

  • Talend Challenge

    • Talend provides data integration software

    • Introducing cloud SaaS to complement on-

    premise software

    • Nordcloud helped Talend to build secure,

    scalable and and automated service platform on

    AWS

  • AWS toolset for SaaS providers

    Elastic Capacity

    • EC2

    • ELB

    • Route 53

    • Cloudformation

    Security

    • VPC

    • Security Groups

    • Subnet NACLs

    • KMS

    • Cloudtrail

    Automation

    • Cloudformation

    /Troposphere /

    Boto

    • [Configuration

    management

    tools

    Puppet/Chef]

    • [CI tools

    (Jenkins)]

  • www.talend.com

    http://www.talend.com/

  • Key Facts

    because Data-Driven companies have…

    • 23 times greater customer acquisition

    • 6 times greater customer retention

    • 19 times more profitability

    Connecting the Data-Driven Enterprise

  • A Data-Driven world

    Big Data Cloud Connected Enterprise

    Self-Service

    Hadoop

    becomes the

    dominant

    data

    processing

    platform

    Cloud

    overtakes

    on-premises

    investment

    APIs

    Connect the

    Data-Driven

    World

    Self-Service

    becomes the

    norm

  • Unified Integration Platform

    • Lowest Cost of Ownership

    • Open, Standards Based

    • Run in the Cloud, On-premises or Hybrid

    • Big Data Leadership

  • Broad Customer Base

  • Talend Integration Cloud

    www.talend.com/products/integration-cloud

    http://www.talend.com/

  • Talend Integration Cloud

    • Simplify and accelerate cloud integration

    • Manage hybrid integration with instant, elastic and secure capacity.

    • Enable big data insight at a fraction of the cost

    • Empower anyone to integrate data using

    a self-service interface

  • • Background

    – Lacking insight into sales

    and marketing programs

    • Goals

    – 360 customer view

    – Weekly pipeline

    snapshots

    – Full lead-to-deals won

    lifecycle

    Example Use Case

    Customer Data Warehouse

    Redshift

  • Use Cases

    Analytics in the cloudOffload to the cloud

    Internet of Things

    Self-Service Integration

    Connecting SaaS-SaaS, SaaS-Ground

    Redshift

  • Talend Integration Cloud Architecture

    A secure cloud integration platform

    DESIGN · MANAGE ·

    MONITOR Talend Studio

    Web Interface

    On-Premises

    Talend Remote Engine

    Talend Cloud Engine

    • Data Integration

    • Data Quality

    • Big Data Integration

    • 800+ Components

    • Real-time Integration (Winter)

    • API Gateway (Winter)

  • Everything you can do in Studio,

    now works in the cloud!

    Integration Specialist

    Capabilities

    • Productive graphical tooling

    • Create simple or complex flows

    • Enforce governance and security

    • Publish to the cloud

    Benefits

    • Faster time-to-market

    • Lower costs

    • Establish control

  • Empower Others with Integration Self-

    Service Capabilities

    • Design anywhere

    • Extract, cleanse and load data

    • Reuse components published from Studio or Talend Exchange

    Benefits

    • Increase agility

    • Reduce error rates

    • Empower a new class of users

    • Ensure data governance and security

    Business User

  • Leverage the power of Talend’s open source community

    Speed Project Delivery with Talend Exchange

    • Hundreds of pre-packaged integration

    actions

    • SaaS

    • Big data

    • Packaged apps

    • Technology connectors

    • Data quality

    • Easily add hundreds of additional

    components from Talend Studio

  • Talend Integration Cloud – on AWS

    Manage multiple environments

  • Challenges

    • Picking up java artifacts and turning them into working stacks.

    • Handling multiple teams and Environments – Dev, QA, staging, production.

    • Single vs. Multiple AWS accounts.

    • Orchestration.

    • Dealing with AWS KMS region dependency.

    Dev Ops

    QA-Auto-RT

    Staging ProductionQA-Auto-UIDev latest

  • • Package everything into rpms.

    • Use S3 as repository to store rpm‘s.

    • Single entry point for packages.

  • • Single AMI for multiple stacks & environments.

    • One ami for every role.

    • Instance started as part of the cloud formation stack.

    • Subenv parameter is used to build multiple stacks.

    • User data is generated by troposphere and facts are included.

    • Puppet converts the userdata into custom facter facts. These facts are used for hiera lookup for that particular subenv.

  • Hiera look-up

    • What classes to load?

    • Which is my java version?

    • Where is my mongoDB ?

  • Node signature

    • Node signature is the string representation of the instance t_facts

    • The node signature variable within our configuration management system

    contains list of all instances which have the same signature

  • Problem:

    • EC2 instances needs to find each other within the same stack. (service discovery)

    • Orchestrate instance startup based on fine grained service availability.

    Solution:

    • Stack membership - serfdom.

    • Gossip based cluster membership.

    • Every instance has a tag and tfacts.

    • Serf facts are assigned using facter facts.

    • Serf agent on every instance is queried for discovery.

    • Distributed dependency and sequence of service

    startups handled using Amazon wait condition on

    cloud formation level.

  • • Hiera look-up

    • Node signature variables are interpolated in Hiera to list of nodes that have the

    same signature.

  • • Keys in KMS are region specific

    and only available in that region.

    • To handle disaster recovery and move to

    different regions the application architecture

    should be designed to incorporate this factor.

    • We use region specific master key but region

    independent data key to encrypt application

    data.

  • &

    Project Partnership

  • Talend: Nordcloud Partnership

    Scope of Engagement

    AWS Consulting /

    System Setup

    • General setup of the

    entire stack in a fully

    automated way

    • Special expertice

    (MongoDB Cluster

    config)

    • Backup / Recovery /

    High Availability

    Day to Day Operations

    • AWS Helpdesk

    • On-side support on

    day to day

    operations

    • Centralized System

    Monitoring

    • Stand-by support for

    critical incidence on

    a 24/7 base

    AWS Partner

    • Consolidated Billing

    • AWS Enterprise

    Support

    • AWS cost

    management /

    optimization (e.g

    Instance Type

    Selection,

    Reserved Instance

    Mgmt)

    &

  • Talend: Nordcloud Partnership

    Engagement Milestones

    &

    After a final selection period

    In November 2014

    We agreed to start with

    Nordcloud as our partner for

    Cloud Operations support.

    Start: December 2014

    (Primary Reasons:

    • Used to work with same tech

    stack (Cloud Formation /

    troposhere, puppet,

    mongoDB and others )

    • Senior consultants for on

    side support with a track

    record in Cloud Operations

    System Setup

    Consulting –

    Working closely

    with our DevOps

    Team to setup

    the system to a

    full production

    scale

    deployment

    (incl. 3 Beta

    Phases /

    Deployments)

    Production

    Start (End Apr

    15)

    Continues Setup

    / Config Support

    for next version

    Support our

    CloudOps team

    on day to day

    operations

    Next:

    Rolling Release

    Update from

    Spring 15 (1.0)

    version to

    Summer 15

    Version (1.1) end

    of June

    Continues day to

    day operations

    suppo