BERLIN 2015
CI in the cloud
Case Talend & Nordcloud
Ilja Summala, Group CTO, Nordcloud
Anubhav Sharma, Senior Software Architect, Talend Deutschland
About Nordcloud
Nordcloud is leading European AWS
consultancy with offices in Helsinki, Munich,
London, Stockholm, Oslo and Malmö.
Nordcloud is AWS Premier Consultancy
Partner for 2015.
We have completed over 200 cloud
migration projects.
Talend Challenge
• Talend provides data integration software
• Introducing cloud SaaS to complement on-
premise software
• Nordcloud helped Talend to build secure,
scalable and and automated service platform on
AWS
AWS toolset for SaaS providers
Elastic Capacity
• EC2
• ELB
• Route 53
• Cloudformation
Security
• VPC
• Security Groups
• Subnet NACLs
• KMS
• Cloudtrail
Automation
• Cloudformation
/Troposphere /
Boto
• [Configuration
management
tools
Puppet/Chef]
• [CI tools
(Jenkins)]
Key Facts
because Data-Driven companies have…
• 23 times greater customer acquisition
• 6 times greater customer retention
• 19 times more profitability
Connecting the Data-Driven Enterprise
A Data-Driven world
Big Data Cloud Connected Enterprise
Self-Service
Hadoop
becomes the
dominant
data
processing
platform
Cloud
overtakes
on-premises
investment
APIs
Connect the
Data-Driven
World
Self-Service
becomes the
norm
Unified Integration Platform
• Lowest Cost of Ownership
• Open, Standards Based
• Run in the Cloud, On-premises or Hybrid
• Big Data Leadership
Broad Customer Base
Talend Integration Cloud
• Simplify and accelerate cloud integration
• Manage hybrid integration with instant, elastic and secure capacity.
• Enable big data insight at a fraction of the cost
• Empower anyone to integrate data using
a self-service interface
• Background
– Lacking insight into sales
and marketing programs
• Goals
– 360 customer view
– Weekly pipeline
snapshots
– Full lead-to-deals won
lifecycle
Example Use Case
Customer Data Warehouse
Redshift
Use Cases
Analytics in the cloudOffload to the cloud
Internet of Things
Self-Service Integration
Connecting SaaS-SaaS,SaaS-Ground
Redshift
Talend Integration Cloud Architecture
A secure cloud integration platform
DESIGN · MANAGE ·
MONITORTalend Studio
Web Interface
On-Premises
Talend Remote Engine
Talend Cloud Engine
• Data Integration
• Data Quality
• Big Data Integration
• 800+ Components
• Real-time Integration (Winter)
• API Gateway (Winter)
Everything you can do in Studio,
now works in the cloud!
Integration Specialist
Capabilities
• Productive graphical tooling
• Create simple or complex flows
• Enforce governance and security
• Publish to the cloud
Benefits
• Faster time-to-market
• Lower costs
• Establish control
Empower Others with Integration Self-
ServiceCapabilities
• Design anywhere
• Extract, cleanse and load data
• Reuse components published from Studio or Talend Exchange
Benefits
• Increase agility
• Reduce error rates
• Empower a new class of users
• Ensure data governance and security
Business User
Leverage the power of Talend’s open source community
Speed Project Delivery with Talend Exchange
• Hundreds of pre-packaged integration
actions
• SaaS
• Big data
• Packaged apps
• Technology connectors
• Data quality
• Easily add hundreds of additional
components from Talend Studio
…
Talend Integration Cloud – on AWS
Manage multiple environments
Challenges
• Picking up java artifacts and turning them into working stacks.
• Handling multiple teams and Environments – Dev, QA, staging, production.
• Single vs. Multiple AWS accounts.
• Orchestration.
• Dealing with AWS KMS region dependency.
Dev Ops
QA-Auto-RT
Staging ProductionQA-Auto-UIDev latest
• Package everything into rpms.
• Use S3 as repository to store rpm‘s.
• Single entry point for packages.
• Single AMI for multiple stacks & environments.
• One ami for every role.
• Instance started as part of the cloud formation stack.
• Subenv parameter is used to build multiple stacks.
• User data is generated by troposphere and facts are included.
• Puppet converts the userdata into custom facter facts. These facts are used for hiera lookup for that particular subenv.
Hiera look-up
• What classes to load?
• Which is my java version?
• Where is my mongoDB ?
Node signature
• Node signature is the string representation of the instance t_facts
• The node signature variable within our configuration management system
contains list of all instances which have the same signature
Problem:
• EC2 instances needs to find each other within the same stack. (service discovery)
• Orchestrate instance startup based on fine grained service availability.
Solution:
• Stack membership - serfdom.
• Gossip based cluster membership.
• Every instance has a tag and tfacts.
• Serf facts are assigned using facter facts.
• Serf agent on every instance is queried for discovery.
• Distributed dependency and sequence of service
startups handled using Amazon wait condition on
cloud formation level.
• Hiera look-up
• Node signature variables are interpolated in Hiera to list of nodes that have the
same signature.
• Keys in KMS are region specific
and only available in that region.
• To handle disaster recovery and move to
different regions the application architecture
should be designed to incorporate this factor.
• We use region specific master key but region
independent data key to encrypt application
data.
&
Project Partnership
Talend: Nordcloud Partnership
Scope of Engagement
AWS Consulting /
System Setup
• General setup of the
entire stack in a fully
automated way
• Special expertice
(MongoDB Cluster
config)
• Backup / Recovery /
High Availability
Day to Day Operations
• AWS Helpdesk
• On-side support on
day to day
operations
• Centralized System
Monitoring
• Stand-by support for
critical incidence on
a 24/7 base
AWS Partner
• Consolidated Billing
• AWS Enterprise
Support
• AWS cost
management /
optimization (e.g
Instance Type
Selection,
Reserved Instance
Mgmt)
&
Talend: Nordcloud Partnership
Engagement Milestones
&
After a final selection period
In November 2014
We agreed to start with
Nordcloud as our partner for
Cloud Operations support.
Start: December 2014
(Primary Reasons:
• Used to work with same tech
stack (Cloud Formation /
troposhere, puppet,
mongoDB and others )
• Senior consultants for on
side support with a track
record in Cloud Operations
System Setup
Consulting –
Working closely
with our DevOps
Team to setup
the system to a
full production
scale
deployment
(incl. 3 Beta
Phases /
Deployments)
Production
Start (End Apr
15)
Continues Setup
/ Config Support
for next version
Support our
CloudOps team
on day to day
operations
Next:
Rolling Release
Update from
Spring 15 (1.0)
version to
Summer 15
Version (1.1) end
of June
Continues day to
day operations
support
Dec 15 Dec - Apr 15 Apr 15 Jun 15 ....
Recommended
