Upload
trinhdieu
View
234
Download
1
Embed Size (px)
Citation preview
Containers Infrastructure for Advanced Management
Federico Simoncelli – Principal Software EngineerRed Hat
July 2015
AgendaRunning containers on an infrastructure is a “solved problem”.
Now you have to run them smoothly.
• Deployment• Metrics Technologies• Analyzing and Correlating Metrics• Smart Analysis – Fleecing• Cross-Provider Relationships• Traceability• Advanced Management
Requirements• Docker
automates the deployment of applications inside software containers
• Kubernetesmanages a cluster of Linux containers as a single system
• OpenShiftbuild, deploy, and manage your applications with Docker and Kubernetes
• Project Atomiclightweight operating system (rpm-ostree) assembled out of RPM content
• Foremancomplete lifecycle management tool for physical and virtual servers
Deployment - Choices• Deploying Atomic
• http://www.projectatomic.io (Fedora and CentOS based images)• Bootable, immutable, versioned filesystem, updates delivered in large sets• Everything that is not in the base system runs as a container
• Deploying a Traditional Distribution• More flexible in nodes content• Matrix of components is less tested
• Deployment on IaaS (OpenStack, oVirt)• Secure – Improved isolation of tenancy• “Elastic” on demand
• Deployment on Bare Metal• Automated with Foreman or Satellite
Deployment - Technologies• Kubernetes project uses Salt
• Requires a salt-master and salt-minion running on the nodes• Mostly for GCE (and Vagrant)
• OpenShift project uses Ansible• https://github.com/openshift/openshift-ansible• Supports AWS GCE libvirt OpenStack Vagrant
• OpenStack• Heat - Kubernetes and OpenShift templates can be found on github repositories• Magnum - Container orchestration engines as first class resources in OpenStack
• Self-Registration of nodes• Enabled by default - preferred pattern used by most distributions• Can be disabled for strict security policies
Metrics Collection• Containerized solution• Collecting Metrics
• Heapster• Kubelet (cAdvisor)
• Storing Metrics• InfluxDB• Hawkular
• Metrics Analisys• Metrics per Container / Pod / Service / Replica• Highly coupled with the metrics database
Kubernetes Node 1 Kubernetes Node N
Heapster Metrics DB
Kubernetes Master Metrics Analysis
IaaS Host YIaaS Host X
Kubernetes Node 1 Kubernetes Node N
Metrics Cross-Linking• IaaS metrics collection
• OpenStack Ceilometer• oVirt Data Warehouse
• Metrics collection and analysis• Nodes, Instances and Hosts
Heapster
IaaS Instance 1 IaaS Instance N
IaaS Metrics
Host load spike
Instance load spike
Metrics – Hawkular• Open Source monitoring solution - http://www.hawkular.org• Components can be re-used in applications outside of Hawkular• Capable of receiving monitoring data from various existing sources• Scalable and high performance metric storage based on Cassandra• On-premise or in the cloud (works on OpenShift out of the box)• Other features and work in progress
• Alerting• Data Visualization (in progress)• Analytics• Push notification (in progress)
Fleecing – Images Inspection• Case 1 – Strict policy, allow only specific known images
• Pre-compute images content• Case 2 – No policy, images are downloaded from any registry
• Images discovery• Docker signed images• Extract images (tricky)• Inspect images (e.g. OpenSCAP)• Cross-link with packages and errata database• Store/cache content and analysis (retention after image disappeared)
Fleecing – Images Inspection• Use the same nodes of the cluster infrastructure• Dockerized solution - runs everywhere (Atomic)• Should not execute anything from inside the image (best practice)• docker-fleece – https://github.com/simon3z/docker-fleece
• Creates a stopped container from image (fast)• Uses CopyFromContainer to extract files (linear)• Docker Graph Driver independent
• docker-mount – https://github.com/rhvgoyal/docker-mount• Creates a stopped container from image (fast)• Mounts externally the Container mountpoint (fast)• Tightly coupled with the Docker Graph Driver (device-mapper, overlayfs, etc.)
Fleecing – Docker Fleece Example• How to extract image content:
$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
docker.io/fedora 22 ded7cd95e059 6 weeks ago 186.5 MB
docker.io/fsimonce/docker-fleece latest 847c8293c5b4 7 weeks ago 217.9 MB
$ docker run -ti --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp/docker-fleece:/tmp/docker-fleece fsimonce/docker-fleece --image=fedora:22 --path=/tmp/docker-fleece
(3.38 seconds)
$ du -sh /tmp/docker-fleece
194M /tmp/docker-fleece
Fleecing – Container Inspection• Inspect the content of a running container• Analyze the “drift” of a container from its original image
• were the packages/binaries tampered with?• Fleecing a running container requires snapshots and merge capabilities
• Freeze container processes• Flush cache• Snapshot and clone container• Unfreeze container processes• Analyze cloned container• Merge back container snapshot
Kubernetes - Traceability• Report Docker Container ID in Container Status
• Allows to peek in Docker (or other runtimes) low level information
• Report Docker Image ID in Container Status• Allows to reliably identify the specific images in use
• Add Node System Info• Allows to identify reboots and version of the critical components (Kernel, Docker, etc.)
• Add Instance ID and Provider ID to Nodes• Allows to identify the instances of the Kubernetes nodes
• Improved Logging• Allows to analyze important events in the cluster and infrastructure
ManageIQ• Open Source Cloud Management Platform - http://manageiq.org• Self-Service Provisioning and Chargeback• Places Virtual Workloads according to your business priorities
• Balance your costs, performance, security, and reliability across cloud platforms
• Supports Infrastructure (oVirt, VMware) and Cloud (OpenStack) Providers• Smart Analysis of Hosts, Virtual Machines and Instances
• Drift of the instances from initial image (detect new applications, new users)• Supports definition of Compliance Policies
ManageIQ - Inventory Full Picture• Inventory collection at all layers for infrastructures, clouds and containers
• Containers, Instances, Hosts, Storage and Network
Service Pods Container Image
NodesCluster
InstanceRegion
Host
Storage Network
ManageIQ - Capacity and Utilization• Metrics correlations, and in the future cross-provider correlations
ManageIQ - Stay in Touch• Community http://talk.manageiq.org• Code https://github.com/ManageIQ/manageiq #providers/containers• Documentation http://manageiq.org/documentation• Social:
• Twitter @manageiq #manageiq
• Contact Me• Federico Simoncelli - [email protected]• Twitter @simon3z