Upload
karen-almog
View
209
Download
1
Embed Size (px)
Citation preview
Integration Systems @ mobile.deKaren Almog Sr. Web Operations Engineer, Mobile International GmbH [email protected]
!
4
Our Platform "
mobile.de
Over 2.7M ads Approx. 50k ads per
day
eBay Kleinanzeigen
Nearly 12M ads Over 200k ads per day
+50
Developers/QA
11
Teams
1826
VMs
Our Platform
6
What Does PD Want? "
# $
%&
'Requirements
REPRODUCTION CONSISTENT TEST DATA
SIMPLE SETUP ROLLBACK
A Standard Integra setup @ mobile
7
What Does PD Need? (
!O/S Installation Debian/Solaris
"
#
Server ConfigurationTomcats, Databases, Search Indexers, DNS, Proxy Servers and more
Data Initialization Initial Test Data Sets
Application DeploymentWAR/JAR
Approx. 60 Virtual machines per environment Ñ VM Install
$
8
What Does SiteOps Want? "
[ ©
9
The Solution: Autoconf "
*
%
'
A Server Provisioning system and CMDB
Automated HW and VM server provisioning
Servers inventoried as a resource in a database%
'
Reduces provisioning time from hours to minutes)
10
Inventory Database Overview "
11
Inventory Database Overview "
insert into server set manufacturer='VMware, Inc.', productname='VMware Virtual Platform'; select last_insert_id() into @var_server_id; insert into serverdetails set server_id=@var_server_id,detailkey='datacenter_number', detailvalue='44'; insert into serverdetails set server_id=@var_server_id,detailkey='virtual', detailvalue='vmware'; insert into serverdetails set server_id=@var_server_id,detailkey='vmware_createopts_desire_hdd', detailvalue='12'; insert into serverdetails set server_id=@var_server_id,detailkey='vmware_createopts_desire_mem', detailvalue='3072'; insert into serverdetails set server_id=@var_server_id,detailkey='vmware_createopts_desire_cpu', detailvalue='1'; insert into assignment_queue set server_id=@var_server_id, serverclass_id=477, sequence_number=48, hostsuffix=2; select * from view_assignment_queue_hostdetails where server_id=@var_server_id;
12
There’s No Shame in Using A Mouse! "
13
The Solution: Genesis "
*
%
'
A “Master” Autoconf Instance for Integras
Based on Autoconf
Includes a Web-‐UI and API%
'
DB Resource for other Tools*DNS, Deployment, Monitoring, Puppet
Joined Project between PD & SiteOps+
14
Genesis: Web-‐UI "
15
Inventory Database: Integra Reference List "
16
Genesis: API "
17
Genesis: API "
root@genesis44-219:~# vm_creator.py -i graphite
Assignment Queue entry for "graphite":
{ "serverClassId": 352, "sequenceNumber": 17, "hostSuffix": "1", "hostPrefix": "graphite", "modificationTime": null, "state": "newrequest", "serverId": 243, "id": 335 }
18
Genesis: API "
19
Genesis: Data Initialization "
*
%
'
Set of BASH Scripts
Initializes Integra’s Database according to use case%
'
Sets up database replication & Initialization)
Can be launched from the Genesis GUI*
20
Application Deployment: Autodeploy "
*
%
'
Continuous Deployment System
Web interface & API
Decentralized architecture%
'
Full Deployment in 20 minutes)
Instant feedback & Logging*
Collaborative effort between PD & SiteOps+
21
Autodeploy Architecture "
eBay Inc. confidential
Architecture Overview
7
Autodeploy UI
AutoDeploy Web App
AutoDeployDB (infradb)
Apache ZooKeeper
Server running Autodeploy
Agent
Server running Autodeploy
Agent
Server running Autodeploy
Agent
22
Autodeploy Architecture: Host Agent "
eBay Inc. confidential
AutoDeploy Agent - What happens during a deployment
8
Text
XMLDEPLOY
PLAN
Autodeploy Agent
TARGET HOST 1. Autodeploy sends deploy instructions to autodeploy agent running on target host.
2. Autodeploy agent executes local deployment script on target host.
3. Deployment script executes deployment steps (autonomous)
deployment script
• Call LBAPI• Call Nagios• Download software• Create properties• Test app• update releases table on
infradb• exit with status
4. Script output and exit code sent back to autodeploy agent
Loggingback to
AutoDeploy
1.
2.
3.4.
Script outputsent to Agent
5. Logging and status reported back to Autodeploy by agent
AutoDeploy Web Application
5.
23
Autodeploy: GUI "
24
Autodeploy: GUI "
2014
25
Autodeploy: Stats ,
Production Pre-‐Production ( Integra)
Rollouts Per Day 8 21
Applica2ons Changes per day (avg) 29 2017
Deployable Ar2facts (Total) 279 279
Deployable Hosts (Total) 555 1621
26
Virtualization in Test Environment ,
0!
450!
900!
1350!
1800!
2006! 2007! 2009! 2010! 2011! 2013! 2014!
Physical! Virtual!
27
Today ,
CURRENT STATUS
• We know it well, it sbll consumes bme and there is very licle led to automate.
• IaaS is becoming ubiquitous, PaaS is our new challenge.
• Several IaaS clouds are already available at eBay.
• We already have a sort of IaaS (VMWare/Autoconf) of our own and it works well enough for the short term.
28
What Does The Future Hold? ,
,Development teams and processes are moving towards Micro-‐Services
29
What Does The Future Hold? ,
THANK YOU! :-‐)-Karen Almog Sr. Web Operations Engineer, Mobile International GmbH [email protected]