Upload
yong-feng
View
104
Download
1
Embed Size (px)
Citation preview
Kubernetes on EGO : Bringing enterprise resource management and scheduling to KubernetesDa Ma ([email protected])Software Architect, IBMOwner of kube-incubator/kube-mesos-framework
Yong Feng ([email protected])Senior Software Architect, IBM
Why “Kubernetes on EGO”?
Computing, Storage and Network
Application Application
Container RuntimeThe container runtime packages and launches application instance in a sandbox with portable and flexible capability.Docker and rkt are container runtimes.
Workload ManagementThe workload management component manages the life cycle of an application as well as access to the application, including service compose, service discovery, load balanceKubernetes and Marathon are workload managers.
Resource ManagementThe resource management component provides an abstraction of resources (cpu, mem, …) for application and then allocates/provision resources among tenants and applications. Mesos is an open source resource manager. EGO is an IBM enterprise resource manager.
Why Kubernetes on EGO?
1992
PBS/SGE/LSF Res mgr & wld mgr
tightly coupledBatch wld only
Monolithic
2003 2016 future
??Shared state
between fws by Optimistic Offer
Shared state
Two Level Scheduling
Mesos/YARN/EGO
……
Architecture Overview
EGO Master
VEMKD
MLIM
BASE API
LIM PEM
EGO Agent
LIM PEM
EGO Agent
LIM PEM
EGO Agent
UDP SocketTCP Socket
PLUGIN
k8s-apiserver
k8s-controller-manager
k8se-scheduler
kubelet
kube-proxy
resreq alloc1. Get Pods
2. Send resource request to EGO
3. Get allocations from EGO
4. Bind Pods with Host
5. Run Pods by kubelet
1
2 3
4
5
EGO: Enterprise Resource Manager• Hierarchical consumer• Enterprise sharing policies• Smart preemption• Rich resource attributes
and resource requirement language
• Unified management console• Security• Monitor and alert• HA and multiple site• Resource usage analysis
EGO: Hierarchical Consumer
Dept-1
ORG
Dept-2
Dept-n
Team-1
Team-2
Proj-1
Proj-2
Hierarchical resource budget plan
Hierarchical role based access control
EGO: Enterprise Sharing PoliciesTime-windows based resource
plan per resource groupOwnership and one-to-one
lending/borrow policy
EGO: Enterprise Sharing PoliciesDynamic sharing from top
down to leaf consumerHybrid sharing polices
o At T0, A has a demand of 20
A = 20o At T1, B1 has a demand of 20 and reclaims its parent’s 16
A:B1 = 4:16 o At T2, A cancels all workload and becomes idle
B1 = 20o At T3, B2 has a demand of 20 thus reclaims its 12
B1:B2=8:12
/
B1
BS=1
S=3S=1
S=4
(A=4)
(B1=4) (B2=12)
20 slots in total
A
B2
Hybrid Ownership Share ratioSharing by default X xReserve slots from being shared
X X
Plan configured by absolute number
X X
Sibling first borrowing X xbalance checking X X Proportional borrowing X xProportional reclaiming X x
EGO: Enterprise Sharing Policies
Flexible framework of scheduling plugin for
customized sharing policies
EGO: Smart Preemption• Asynchronized resource negotiation protocol
Issue resource request via allocation which allows client to orchestrate multiple services from different tenants; update resource request on the fly; receive resource allocation by event;
• Grace period in resource planContract between resource lender and borrower used to decide how resources will be returned if required
• Candidate resource listAllows the borrower to optimize when making decisions on which resources to return within grace period
EGO: Rich Variety of Resource Attributes and Resource Requirement Language• Various types of resource attributes and ways to define and collect them
Static vs dynamic; integer vs Boolean vs string vs ip vs topology; user defined vs collected by script
• Resource requirement language
select(), order(), affinity(), antiaffinity(), rusage() …
Demo Video
We’re Contributorto the community !!!
Mesos: OpenSource Resource Management• Hierarchy Role (MESOS-6375)• Multi-Role Frameworks (MESOS-1763)• Scheduling (Pending)
oResource RevocationoOversubscription for reservationoQuota Chunkso…..
Kubernetes on Mesos• Sponsor: Tim Hockin (Google)• Champion: David Eads (Redhat)• Owner: Klaus Ma (IBM)• Github: kuberntes-incubator/kube-mesos-framework
Kubernetes on Mesos (kube-mesos-framework)
1. Get Pods
2. Match Pods and Offers
3. Bind Pods with Host
4. Update Pods status
5. Run Pods by kubelet
IBM Spectrum Conductor for Container
Spectrum Conductor with
Spark
Watson / Cognitive
Container Cloud
Session Scheduler
Workflow
Installer(Deploy,
Reconfigure, HA, Scale,
Rolling update)Mesos Agent
K8s executor
pod pod pod container container
containercontainer
Mesos Master
Kubernetes
GUI
Service Discovery
AuthenticationAuthorization
Distributed Key-value
Store
Image registry
Monitor
HPC
App Store
Persistent Volume
Service Load Balance
Trouble-shooting
Network Topology
Community Value IBM Value-add Customer Value
Docker Hub Registry holds a repository of 75000+ Docker images
Lots of application integrated with MesosKubernetes enable micro-service architecture
• Client unique registry available on premises• Security readiness guidance via the Vulnerability Advisor• Build-in applications of popular open source projects and IBM enterprise
products in App Store
Access to the images and application you require to deploy containers that meet your business needs and strategy
Open-source, standardized, lightweight, self sufficient container technology
• Balance workload between on-prem and off-prem• Deployment choice with openPOWER and x86_64
Flexibility to choose on-prem and off-prem or mix for your business
Build, ship, and run standardized containers• Integrated monitoring & logging• Elasticity to grow storage & container needs• Integrated CI/CD flow• Life-cycle management of containers and data volumes
Docker ease of use combined with enterprise-level integrity and confidence
Create a Container Cloud for developers supporting DevOps practices and cloud-native apps. Pre-built app catalog for fast deployment of OSS tools. Reduce developer friction, creating faster time to results
1
Improve Developer Productivity
Fine grain, dynamic allocation of resources maximizes efficiency of Spark instances sharing a common resource pool.
2Increase Resource Utilization
Proven architecture at extreme scale, with enterprise class workload management, monitoring, reporting, and security capabilities.
3Reduce Administration Costs
Mesos
Kubernetes(role = *, bigdata-daemons)
MyriadSlaves
(weight)
SparkSlaves(weight)
App Area (label: app) BigData Area (label: bigdata)role = bigdata-daemon: Reserve resources for HDFS and Yarn/Spark masterrole = bigdata-comute: Reserve resource for Yarn/Spark agents
Spark SessionSchedulerMyriad Masters
Dep 1 Dep 2 Dep 3
ns1+
quota1
ns2+
quota2
ns3+
quota3
Container service
role = *
BigData Service & Applications
(role = bigdata-comute)
ResourceSharing
Hierarchy Consumer
Smart preemption &
Sharing policies
NS/Quota
Network/DNS
Scheduling
Dream ???
Resource Requirement
Spark with kube-mesos
What’s next?• Support Sharing Policies & Smart Preemption:
Revocable resources support (#19529)Scheduling enhancement (# 31068)
• Support Hierarchical Consumer:Namespace/Quota support/integrate (#31069)Multiple roles support
• Kube-DNS integrate with external DNS (# 28453)
• …
Roadmap of kube-mesos-framework (DRAFT)
Nov, 2016 End of 2016 2017
v0.7 release new code base
v0.9 release new features
v0.8 releasek8sm refactor v1.0 release
Production Ready
Thank You !!