Upload
others
View
12
Download
0
Embed Size (px)
Citation preview
DevOps:
Cloud-inspired,
On-Premises
Patrick Gemme ([email protected])
Rick Nickle ([email protected])
Introductions
Patrick Gemme
● Network Engineer
● Windows Server
● Hyper-V
Rick Nickle
● Systems Integration
● Linux
● QA and Test
Plan of Presentation
● Cultural issues with migrating some applications from on-
premises to cloud suggested a DevOps approach.
● Automate our applications environment for remaining on-
premises applications, by leveraging our Cloud approach.
● Measure and monitoring our environment in order to
continuously improve our availability, performance and
capabilities.
● Share our experience.
Why Cloud-inspired?
● Cultural issues inspired our migration of apps to the Cloud
using DevOps principles.
● Automation of on-premises application deployment was built
on our success with the Cloud migration.
● Measurement is the key element which we hope to
continuously improve as we extend our embrace of DevOps.
● Sharing lessons learned, both positive and negative, with
our own team and our peers is part of the open culture we
are promoting.
Cloud
Cloud Applications Environment
● Previously hosted on-premises, moved to
Acquia and WPEngine cloud services.
● Both providers use Git to manage
content.
● Only one provided a full-featured
dashboard for managing releases.
Cloud Applications Strategy
Challenge Response Toolsets
Dev. and Comm. not familiar
with revision control and/or
Git.
Find usability tools and
develop training for release
procedures.
Git-bash, GitLab, GitKraken,
slides and Panopto videos.
Dev. and Comm. step on each
others releases.
Require change control
approval before release.
SA Manage ticketing using
REST API, integrated to
GitLab CI/CD.
Dev. and Comm. not testing or
not testing on environment
equivalent to production.
Require release to staging
environment before release to
production.
Use Merge Request to require
release to test before
production can release.
Definition of CI/CD
Continuous Integration
● Any authorized user
can checkout,
modify, checkin, and
tag code at any time.
Continuous Deployment
● Any authorized user
may use CI to trigger
automation, including
release to test or
production.
Authorization for CI/CD
● Each project has an authorization
dashboard.
● Project owner can control level of
authorization.
● Test branches are not protected.
● Production branches are protected○ Use Merge Request of tested branch to update.
Interface for CI/CD
● Accessed through git transactions, the
dashboard, or the REST API.
● All transactions are browsable.
● Webhooks are included for social media and
other apps. (e.g., Slack)
Enable an Application for Cloud
Two requirements:
● App Code
● A CI/CD file○ A simple YAML
file that defines
how to release.
code
CI/CD
.gitlab-ci.yml
An overview of Cloud CI/CD file
production:
script:
- /ci-tools/bin/check_samanage --tag $CI_BUILD_TAG
- /ci-tools/bin/ci_target --verbose
only:
- /^PROD-SA.*$/
except:
- branches
Cloud Release Flow
● CI/CD file defines release
flow.
● Pre-built Container has code
and credentials for release to
cloud.
● User checks in code and tags
code.
● CI/CD invokes Docker to
checkout code and push it to
Cloud repo.
code
Checkin
& Tag
Docker Container
Checks Code Out
Docker Container
Pushes Code to
Cloud
Cloud and CI/CD
prod
testPropose
Change Control
Ticket?
CI/CD
CI/CDTag
“TEST”
Tag
“PROD”
Create
Branch
yes
no
Branch
“A”
Branch
“master”
...coding..
.
branching tagging releasing runningthinking
merging
Applications
Legacy Applications Environment
● Virtualization environment must be shut down when
host node is patched.
● Release by many methods: sftp, Capistrano, rsync
● Release requires privileged access
● Test and Development not equivalent to production
● Single app environments for each group of application:
rails3, rails4, php5, java, tomcat.
Consolidation, Release Management
● Asked to consolidate applications and
restrict deployment ability, preferably
behind automation controlled via CI/CD.
● Used a simple rsync based prototype to
release code fragments and “gadgets.”
Refactor for Microservices and CI/CD
● Experiments with consolidating small
applications led to trying to containerize
medium sized LAMP applications.
● LAMP applications performed very well
and were relatively easy to port, quickly
put several into DevOps production.
Microservices, CI/CD Strategy
Challenge Response Toolsets
Application tied to server
configuration; Disparity
between Test and Prod
Containerize Applications Docker Swarm
Release requires privilege Authorization defined in
application repository.
GitLab CI/CD
Consolidate similar services Standardized configurations,
containers
DSC, Docker Swarm
Legacy VM Environment
● Legacy application environment, older VM technology.
● Variations in server configurations over time.
● Security standards not defined for original deployment.
● Consolidate similar services.
● Automate deployment of applications (non-privileged)
● Virtualization host not clustered.
Infrastructure as Code/Data StrategyChallenge Response Toolsets
Virtual servers unavailable
when host patched or
rebooted.
Hyperconverged Clusters Hyper-V, CSV/Storage Spaces
Variations in VM configs. VM templates, managed
configurations.
SaltStack, DSC
Security and best practices Published standards Group Policies, DSC
Infrastructure as Code/Data
● DSC not only encodes the instructions for recreating,
replicating and scaling the environment, it is also a
design document that can be referred to for
understanding, auditing and measuring the
environment.
● Virtualization provides an API for manipulating the
running environment.
Containerize an Application with DevOps
You need four things:
● Application code
● Dockerfile
● CI/CD file
● Compose file
codeDockerfile
Compose.ymlCI/CD
.gitlab-ci.yml
Container Release Flow
● CI/CD file defines Dockerfile and
Compose file to use for release.
● Dockerfile defines how to build
container from code.
● Compose file describes how to
launch container.
● User checks in and tags code.
● CI/CD invokes Docker to
checkout code, build container
with Dockerfile and launch with
Compose file.
code
Checkin
& Tag
Docker Container
Checks Code Out
Docker Container
builds and launches
new container.
Overview of Container CI/CD File
HCREG=docker-reg.holycross.edu
PROJECT=$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME
COMPOSE=docker-compose-$CI_JOB_NAME.yml
DEPLOY_OPTIONS=”--resolve-image always --with-registry-auth”
prod:
script:
- CONTAINER=$HCREG/$PROJECT:$CI_JOB_NAME
- /ci-tools/bin/check_samanage --tag $CI_BUILD_TAG
- docker build --pull --build-arg APPDEPLOY=$CI_JOB_NAME -t $CONTAINER .
- docker push $CONTAINER
- docker stack deploy -c $COMPOSE $CI_PROJECT_NAME $DEPLOY_OPTIONS
Challenges and OpportunitiesChallenge/Opportunity Response Toolsets
Some applications have very
long container build times,
slowing development cycle.
Consider alternative
development pipelines
Hybrid scripts? Jenkins?
Capistrano?
Collaboration across
organizational lines.
Encourage more of this
(DataOps)
Social media, word of mouth,
training.
Operational challenges slow
adoption: legacy, priorities,
failures.
First things first. Patience, reflection, and
lessons learned.
Monitoring
Scaling, Performance, AvailabilityChallenge Response Toolsets
Apache MPM overwhelmed by
peak loads
Change from ‘prefork’ to
‘event’, multiple servers,
ingress buffering and caching,
functional monitoring.
Apache 2, Cloudflare, Docker
Stack, SCOM, Pingdom
Current monitoring solution
unable to provide useful deep
website monitoring
Actually got help from our
security team; setup web-
based test logins
Pingdom
Monitoring and Culture
● Making decisions based
on measurements.
● If you monitor something and alert on it
but don’t have a defined response, don’t
monitor it.
Monitoring & Automation
• Common problems with common fixes
should be automated.
• Most troubleshooting starts with basic tasks
which can be auto-launched.
• Ticketing system flow and auto ticket
creation (API’s, scheduled, form filled).
Monitoring & Automation (cont.)
Common Issue
Microsoft SCOM
Automated Fix
Sharing
Our DevOps OutlookStrength Weakness Threat Opportunity
CI/CD encourages frequent
releases of enhancements,
“blameless” culture.
Basic monitoring is effective,
difficulty measuring non-
functional requirements.
Need to maintain containers
for performance and
security: DevSecOps.
Containers have built-in
monitoring hooks for
applications: stateful apps.
DSC, templates and
containers enhance
availability, DR and
scalability: infrastructure
as code/data.
Sharing of methods, tools
and results are uneven.
Build new apps with
DevOps in mind.
Docker access by
developers is privileged
access, restrict Docker
access to test.
DevSecOps.
New model for deployment
requires refactoring of
application: 12-factor app
model.
Adoption of DevOps for
some legacy tools impeded
by vendor defined
environments. LegacyOps.
Success Stories I
● When marketing team, who had
previously complained about complexity
of process, commented that they found it
empowering after getting used to it.
Mixed Results I
● When customer in another department
who previously had delegated all changes
to operations, was trained and given
access to release configuration changes
themselves, was suddenly eager to move
from on-premises to cloud.
Success Stories II
● When development team member
commented that they would like to clone an
existing application as a baseline for a new
application.
● When development team member built and
tested an application themselves by creating
a new project from scratch.
Cautionary Tale II
● When developer points out that his Ruby
release mechanism was faster albeit with
more maintenance overhead, while
acknowledging that the Docker based
mechanism has superior performance
and scaling features.
Success Stories III
● When marketing team member who
previously had thought version control
and change control was a waste of time,
commented that the development team
wasn’t properly checking in, tagging and
testing their code before release.
DevOps Toolsets To Explore● GitLab Features
○ Build Artifacts (deliverables consumed by applications)
○ Static testing (code)
○ Environments (serverless, Kubernetes)
● QA○ Locust (load testing)
○ Katalon (app testing)
○ Robot Framework (acceptance testing)
● Monitoring and Measurement○ ELK (syslog processing)
○ Grafana/Prometheus (metrics and visualization)
● Load Balancing○ Traefik (container based application load balancer)
● Cloud-like user portal (IaaS, SaaS, PaaS)○ Azure Stack, Nutanix Calm, Heroku
Q & A
Panel at 3pm
Notes
Reference
Strategic Overview for PresentationInfrastructure/Operations
● Monitoring (SCOM)
● Management (Salt)
● Group Policy (AD)
● Containers (Docker)
● Load Balancers (Apache)
● Database Servers
(MySQL/Oracle)
Applications/Development
● Continuous Integration
(GitLab)
● Continuous Deployment
(GitLab)
● Source Management (GitLab)
● QA/Test
● Twelve Factor
DevOps Integration
Token Funny Quote