Upload
others
View
75
Download
0
Embed Size (px)
Citation preview
Disclaimer
©2019 VMware, Inc.
This presentation may contain product features or functionality that are current
under development.
This overview of new technology represents no commitment from VMware to d
these features in any generally available product.
Features are subject to change, and must not be included in contracts, purcha
or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new features/functionality/technology discussed
or presented, have not been determined.
The information in this presentation is for informational purposes only and may not be incorporated into any contract. There is no commitment
to deliver any items presented herein.
Public
Users
VMs, Containers, Microservices
VMware partners (VMC)
Private Data Centers
TelcoNetwork
Private Cloud(VCF)
VCN
10kcustomers
to date
82%of Fortune 100
enterprises
70%of all Fortune
global 500 telcos
Gartner
MQ Leader
WAN Edge
Infrastructure
Ties it all together.
©2019 VMware, Inc.
Virtual Cloud Networking
ResourcesHow to get started
©2019 VMware, Inc.
Design Guides
Demos
Take a Hands-on Lab Join VMUG, VM
Communities (VM
LEARN
VMware.com/go/NSXtechzone
CONNECTTRY
Agenda
©2019 VMware, Inc.
Kubernetes Overview
NSX-T Integration with Kubernetes
Demo: The new topology
Summary
Kubernetes Components
K8s Cluster Con
Master(s) and N
K8s Master Com
• API Server
• Scheduler
• Controller M
• Dashboard
K8s Node Com
• Kubelet
• Kube-Proxy
• Containers R
K8s masterK8s master
K8s
Master
Controller
Manager
K8s API
Server
Key-Value
Store
dashboard
Scheduler
K8s nodeK8s node
K8s nodeK8s node
K8s Nodes
kubelet c runtime
Kube-proxy
> _Kubectl
CLI
K8s Master(s)
©2019 VMware, Inc.
Kubernetes Namespace
Namespaces ar
divide cluster re
amongst users
They can be tho
Tenants
They are a way
Resources Quo
Networking Mu
and Name uniq
Namespace: fooBase URI: /api/v1/namespaces/foo
‚redis-master‘ Pod:/api/v1/namespaces/foo/pods/redis-master
‚redis‘ service:/api/v1/namespaces/foo/services/redis-master
©2019 VMware, Inc.
Namespace: barBase URI: /api/v1/namespaces/bar
‚redis-master‘ Pod:/api/v1/namespaces/bar/pods/redis-master
‚redis‘ service:/api/v1/namespaces/bar/services/redis-master
Kubernetes Pod
A Pod is a grou
more container
an IP address a
VolumePod
10.24.0.0/16
10.24.0.2
pause container (‘owns’ the IP stack)
nginx tcp/80
mgmt tcp/22
loggingudp/514
IPC
External IP Traffic
©2019 VMware, Inc.
K8s
MasterReplication Controller:
• The replication controller en
'desired' state of a collection
makes sure that 4 Pods area
in the cluster
Replica Set:
• Replica Set is the next-gene
Replication Controller. Repli
Based selectors while replica
controllers use Equity-Based
Kubernetes RC & RS
Kubernetes Replication Controller (rc) and Replica Set (rs
Replication Controller
/ Replica Set
Pods
©2019 VMware, Inc.
c
Daemon Sets:
• A DaemonSet ensures that a
nodes run a copy of a Pod.
• As nodes are added to the c
added to them.
• As nodes are removed from
those Pods are garbage coll
• Deleting a Daemon Set will
pods it created
• Daemon Sets are used to re
Units in a lot of cases today
Kubernetes Daemon Set
Kubernetes Daemon Set
K8s
Node
InfraPod
K8s
Node
InfraPod
K8s
Node
InfraPod
K8s
Node
InfraPod
K8s
Master
Deamon Set
©2019 VMware, Inc.
Kubernetes Service
A Kubernetes S
a logical set of
selected with m
Serves multiple
• Service Disc
• East/West lo
the Cluster (T
• External load
L4 TCP/UDPLoadBalancer)
• External acc
service throu
IPs (Type: Node
Redis Slave
Pods
redis-slave svc
10.24.0.5
ClusterIP
Web Front-End
Pods
10.24.2.7
▶kubectl describe svc redis-slave
Name: redis-slaveNamespace: default
Labels: name=redis-slave
Selector: name=redis-slave
Type: LoadBalancer
IP: 172.30.0.24
LoadBalancer Ingress: 134.247.200.20
Port: <unnamed> 6379/TCPEndpoints: 10.24.0.5:6379,
10.24.2.7:6379
DNS:
redis-slave.<ns>.cluster.local ➔172.30.0.24
ExternalIP
134.247.200.20 172.30.0.24
DNS:
redis-slave.external.com ➔ 134.247.200.20
©2019 VMware, Inc.
Kubernetes Ingress
A Kubernetes I
is a L7 LoadBal
that binds a hos
url to aService
The LoadBalan
can be impleme
external Load B
a K8s Pod
Web Front-End
Pods (shop svc)Web Front-End
Pods (special-offers svc)
http://www.bikeshop.com/shop
http://www.bikeshop.com/special-offers
LoadBalancer Datapath
(External or K8s Pods)
▶kubectl describe ingress bikeshop-ingress-shop
©2019 VMware, Inc.
Name: Namespace:
bikeshop-shop bikeshop100.64.240.9,134.247.200.1Address:
Default backend: default-http-backend:80 (<none>)
Backends--------
Rules:Host----www.bikeshop.com
Path----/shop
web-svc-1:80 (<none>)
External IP: 134.247.200.1
DNS: *.bikeshop.com ➔134.247.200.1
Key Design Goals of the NSX-T Data Center Kubernetes
Don't stand in the
way of the developer
!
Provide solutions to
map the Kubernetes
constructs to
enterprise
networking
constructs
Secure Containers,
VMs and any other
endpoints with
overarching Firewall
Policies
Provide
troubl
tools to
containe
in the
©2019 VMware, Inc.
NSX-T K8s Integration – Namespaces & PodsDynamic per Namespace Topology
admin@k8s-master:~$ kubectl create namespace foo namespace ”foo" created
admin@k8s-master:~$ kubectl create namespacebar
namespace ”bar" created
admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foo deployment "nginx-foo" created
admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bar
deployment "nginx-bar" created
NSX / K8s topology
Namespace: foo
10.24.0.0/24 10.24.1.0/24
10.24
NAT
boundary
K8s nodesK8s Masters
©2019 VMware, Inc.
NSX-T K8s Integration – Routed Namespaces
admin@k8s-master:~$ vim no-nat-namespace.yaml
apiVersion: v1kind: Namespacemetadata:
name: no-nat-namespace annotations:
ncp/no_snat: "true“
admin@k8s-master:~$ kubectl create –fno-nat-namespace.yamlnamespace ”no-nat-namespace"created
admin@k8s-master:~$ kubectl run nginx-no-nat --image=nginx –nno-nat-namespacedeployment "nginx-k8s" created
Namespace: no-nat-namespace
NSX / K8s topology
114.4.10.0/26
Direct Routing
114.4.10.6
K8s nodesK8s Masters
©2019 VMware, Inc.
NSX-T K8s Integration – Shared T1TopologyShared T1 for all Namespaces in the Cluster - Both for Policy and MP
NSX / K8s topology
10.24.0.0/24 10.24.2.0
K8s nodesK8s Masters
T1
admin@k8s-master:~$ kubectl create namespace foo namespace ”foo" created
admin@k8s-master:~$ kubectl create namespace bar namespace ”bar" created
admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foo deployment "nginx-foo" created
admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bardeployment "nginx-bar" created
AA
NAT boundary
N
©2019 VMware, Inc.
NCP is a softwa
component pro
VMware in form
container imag
run as a K8s Po
NCP is build in
way, so that ind
adapters can b
different CaaS a
systems at som
K8s / NSX-T Data Center ComponentsNSX Container Plugin (NCP)
NC
M
Infra
K8s / OS
Adapter
CloudFoundry
Adapter
NSX Container Plugin
More…
NSX
Manager
API
Client
NSX
Manager
NS: fooNS: bar
NSX/ K8s topology
K8s master
etcd
API-
Server
Scheduler
©2019 VMware, Inc.
Node
VM
DFW
eth2
Node
VM
DFW
eth0
Minion Mgmt.
IP Stack
eth0
Minion Mgmt.
IP Stack
mgmt network
OVS
mgmt network
Vla
n10
vla
n11
Sub VIF
eth2
vla
n10
vla
n11
OVS
NSX CNI
Plugin & Node Agent
Pods
PodsNSX CNI
Plugin & NodeAgent
Sub VIF
Sub VIF Sub VIF
K8s Node VMs: Most cust
©2019 VMware, Inc.
looking to deploy K8sNo
today
Nested Network-Virtualiz
terminating the overlay tu
Node VM, we areextendi
Hypervisor vSwitch into t
using VLAN tagging. The
vSwitch (OVS) is ‘standal
gets programed by the N
Benefits:
• Enhanced security thro
isolation of the Node V
Control-Plane
• Less transport-nodes in
equates to higher scale
Container Interfaces
NSX-T Sub-VIF Interfaces
Kubelet: calls the NSX CNIplugin.
NSX CNI Plugin: is a simple python script th
translates between Kubelet and NSX Node
simple private protocol to talk to the Node
socket
NSX Node Agent: runs as a DeamonSet in
Mode and is responsible for:
1. The retrieval of IP/MAC/VLAN informatio
next 2 slides) through an NSX-RPC TCP c
Hypervisor
2. The creation and configuration of the upli
the Hypervisor vSwitch inOVS
3. The creation and configuration of downlin
connecting OVS to the Pods 'pause conta
NSX Kube Proxy: Is responsible for the crea
and Load Balancing rules in OVS to do the
East/West LB (will be covered later again).
responsible to create the needed flow/NAT
for the Node-Agent to Hyperbus communi
Node Agent &CNI Plugin
Node Agent & CNI Plugin
eth0
Minion Mgmt.
IP Stack
eth2
vla
n10
vla
n11
OVS
Pods
Hostnetwork-Mode
Node Agent & NSX Kube-Proxy
DeamonSet
NSX CNI Pluginkubelet
©2019 VMware, Inc.
©2019 VMware, Inc.
Pod attachment workflow
NC
M
Infra
K8s / OS
Adapter
CloudFoundr
y Adapter
Libnetwork
Adapter
NSX Container Plugin
More…
NSX
Manager
API
Client
NSX
Manager
K8s master
etcd
API-
Server
Schedule
r
NS: foo NS: bar
NSX/ K8s topology
1)2)
3)
4)
1. NCP creates a ‚watch‘ on any Pod events
2. A user creates a new K8s
3. The K8s API Server notifiethe change (addition) of P
4. NCP creates a logical port
a) Requests an IP from theSubnet
b) Request a MAC from the pool in NSX
c) Assigns a VLAN for theP d) Creates a logical port (Su
the Namespace LS and aMAC and VLAN to the lo
e) Adds all K8s Pod Labels logical port
Shared under NDA
K8s / NSX Workflows (1/2)Pod attachment workflow
©2019 VMware, Inc.
K8s / NSX Workflows (2/2)Pod attachment workflow Pod attachment workflow
Hypervisor
(ESXi &
KVM)
Node
VM
Vla
n4
09
4
Vla
n2
cif
kubelet
NSX
Manager
NSX
Controllers
NSX LCP
5)
6)
NSX
Hyperbus
Node Agent /
CNI Plugin
7)
8)
9)
5. NSX LCP will create the LPon
6. Hyperbus monitors LCP for neinterfaces and learns the CIF’s Id/IP/MAC/VLAN binding
7. Kubelet sees a new ‘PodSpec’Master and starts a new Pod. Ithe NSX cni plugin binary to d ‘network wiring’ of the Pod –proxy'd to the NSX Node Age
8. The NSX Node Agent gets theId/IP/MAC/VLAN binding dataHyperbus over the isolated anchannel (one-way connection e
9. The Node Agent creates theOright VLAN, and configures theinterface to connect to OVS witIP/MAC. After this, Kubelet isuthe Pod creation succeeds
Tenancy / Topology MappingPersistent IPs for K8s Namespaces
With NSX-T each Tenant (Kubernetes Namespace)either gets its own SNAT IP (NATMode), or is directly identifiable by itssource subnet (No NAT Mode)
Node VM
OpenvSwitch
10.12.5.5/2410.12.1.8/24
172.16.1.11/24
mgmt IP
vnic
PAS VMsT1router
VLAN Trunk
NSX-T Logical Switch
172.16.1.1/24
Namesp. T1router
10.12.1.1/24
Pods
Database
(VM based or Physical)
Physical DC Firewall
A new Son the TTenant f
Tenant: fTenant: bar
In NoA-TNMAoTdMeo,tdhee, ethxeteerxntaelrDnaCl FDiCreFwiarellwall aanndd tthhee DDBBccaann ddiissttiinngguuiisshh tteennaanntt ''ffoooo'' aanndd tteennaanntt ''bbaarr'' uussiinngg tthhee ssoouurrccee ISPNAT IP SthuabtnisetatllhoactaitseadlltoocatsepdetcoifaicsTpeencaifnict.Tenant.
©2019 VMware, Inc.
Firewall rules in exis
Firewalls to allow tr
workloads in K8s
The K8s user / Dev
applications that are
identifiable in the ph
With this feature as
Workloads (Pods) c
use a specific IP or
to source their traffi
Feature
Benefits
Infrastructure Team
Persistent SNAT IP per K8s ServiceSpecifying the source IP Kubernetes Workloads using the K8s service
Tier0 LR
Corporate network
DB
allow – from: 134.247.100.10 (App)
to: 134.247.200.9 (DB)
Tier1 LR
Kubernetes Namespace:
Foo
Web-Frontend
Pods
App Logic
Pods
K8s Svc for AppK8s Svc for Web
Namespace LS(s)
SNAT App Svc Pods to:
134.247.100.10For all other Pods
use namespace SNAT IP
©2019 VMware, Inc.
Kubernetes Metadata / NSX Logical Port Mapping
▶kubectl get pod nsx-demo-rc-c7x65 -o yaml
apiVersion: v1
kind: Pod
metadata:creationTimestamp: 2018-07-25T12:05:56Z
generateName: nsx-demo-rc-
labels:
app: nsx-demo
name: nsx-demo-rc-c7x65
namespace: nsx-ujo
Metadata within Kubernetes like Namespace, Pod names, Labels allge copied to the NSX Logical Port as Por
©2019 VMware, Inc.
NSX can be configured to collect ports and switches in dynamic security group
Tags (Kubernetes Metadata) and apply Firewall rules on them
Pre-Created Security Groups / Firewall rules (admin rule
Match on Port Tags
MaPoof t
GroFiresrc
©2019 VMware, Inc.
Support of Kubernetes Network Policy---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: ps-ing-to-fe
spec:
podSelector:
matchLabels:
app: planespotter-frontend
policyTypes:
-Ingress
ingress:
- from:
- ipBlock:
cidr: 100.64.0.0/16
ports:
- protocol: TCP
port: 80
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: ps-fe-to-app
spec:
podSelector:
matchLabels:
app: planespotter-app
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: planespotter-frontend
ports:
- protocol: TCP
port: 80©2019 VMware, Inc.
©2019 VMware, Inc.
Policy Support – Security per Category
CONFIDENTIA
Environ
Health-checks Admin Rules
Applica
Kubernetes Network Default rule:1.Allow Cluster2. Allow Namespace
3. None
Built-in Load BalancingBuilt-in support for Ingress (L7 HTTP/HTTPS) and Svc Type LB (L4 TCP/UDP) i
K8s integration. Most other K8s networking choice don't support Svc Type LB
you need an additional technology like NGINX from Ingress (L7).
NCM
Infra
K8s / OS
Adapter
CloudFoundry
Adapter
Libnetwork
Adapter
NSX Container Plugin
More…
NSX
Manager
API
Client
NSX
Manager
K8s master
etcd
API-
Server
Scheduler
Virtual Server10.114.209.209HTTP and/or
HTTPStraffic
Server Pool 1
Server Pool 2Rule 2/bar/
Rule 1/ f oo/
LB Service
NCM
Infra
K8s / OS
Adapter
CloudFoundry
Adapter
Libnetwork
Adapter
NSX Containe
More…
NSX
Manager
K8s master
etcd
API-
Server
Scheduler
Virtual Server10.114.209.212TCP and/or
UDP traffic
Server Pool
LB Service
©2019 VMware, Inc.
---
apiVersion: v1
kind: Service
metadata:
name: planespotter-frontend
labels:
app: planespotter-frontend
spec:
loadBalancerIP: 78.11.24.19
type: LoadBalancer
ports:
# the port that this service should serve on
- port: 80
selector:
app: planespotter-frontend
[root@master1 ~]# oc describe service/planespotter-frontend
Name:
Namespace:
Labels:
Annotations:
planespotter-frontend
default
app=planespotter-frontend
kubectl.kubernetes.io/ last-applied-
nnotations":{},"labels":{"app":"planespotter-configuration={"apiVersion":"v1","kind" :"Service" ,"metadata" :{"a
frontend"},"name":"planespotter-frontend","namespace":"d...
ncp/internal_ip_for_policy=100.64.64.1
Selector:
Type:
IP:
IP:
app=planespotter-frontend
LoadBalancer
172.30.253.255
78.11.24.19
LoadBalancer Ingress: 78.11.24.19, 100.64.64.1
Port:
TargetPort:
NodePort:
Endpoints:
Session Affinity:
<unset> 80/TCP
80/TCP
<unset> 32688/TCP
10.4.3.4:80,10.4.3.6:80
None
External Traffic Policy: Cluster
Events: <none>
master1 ]#
Persistent IP for Service of type LB
K8s / NSXWorkflows
©2019 VMware, Inc.
Installation ImprovementContainerize OVS and NCP Bootstrap initContainer
nsx-ncp-bootstrap-xkfgm 1/1 Running 0 131m 10.114.209.215 node2 <none> <none>
nsx-ncp-bootstrap-zrnq5 1/1 Running 0 131m 10.114.209.212 master2 <none> <none>
nsx-node-agent-4zfrj 3/3 Running 0 131m 10.114.209.212 master2 <none> <none>
nsx-node-agent-7gr6t 3/3 Running 0 131m 10.114.209.214 node1 <none> <none>
nsx-node-agent-g25v5 3/3 Running 0 131m 10.114.209.213 master3 <none> <none>
nsx-node-agent-n2z4p 3/3 Running 0 131m 10.114.209.211 master1 <none> <none>
nsx-node-agent-z5q87 3/3 Running 0 131m 10.114.209.215 node2 <none> <none>
root@master1:~#
root@master1:~# kubectl get pods -n nsx-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS
GATESnsx-ncp-bcf5c8778-q67wg 1/1 Running 0 120m 10.114.209.215 node2 <none> <none>
Bootst
initContainer installs:1. Installs/Upgrades t 2.Loads the ncp-app3. Installs/Upgrades/
nsx-ncp-bootstrap-4xq5f 1/1 Running 0 131m 10.114.209.214 node1 <none> <none> OVS kernel module ifnsx-ncp-bootstrap-grdqs nsx-ncp-bootstrap-pmhcx
1/1
1/1
Running Running
0
0
131m
131m
10.114.209.211
10.114.209.213
master1 master3
<none>
<none>
<none>
<none>
4. Stops OVS user sprunning on the host m
NSX node
Containers:1. nsx-node-age2. nsx-kube-pro
3. nsx-ovs
©2019 VMware, Inc.
Hipster Shop
©2019 VMware, Inc.
Cloud-Native Microservices Demo Application
https://github.com/GoogleCloudPlatform/microservices-demo
Why NSX for Kubernetes?
Muti-tenancy Load Balancing and
services
Secure Containers,
VMs and any other
endpoints with
overarching Firewall
Policies
Provide
troubl
tools to
containe
in the
©2019 VMware, Inc.
Join the NSX VMUG Community
vmug.com/nsx
Connect with yourPeers
communities.vmware.com
Embrace the NSXMindset
nsxmindset.com
Find NSX Resources
vmware.com/products/nsx
Read the Network Virtualization Blog
blogs.vmware.com/networkvirtualization
©2019 VMware, Inc.
Where to Get Started
Showcases, breakouts, quick talks & gro
discussions
Visit the VMwareBooth
Product overviews, use-case demos
Visit Technical Partner Booths
Integration demos – Infrastructure,secuoperations, visibility, and more
Meet the Experts
Join our experts in an intimate roundtab
Free Hands-on Labs
Test drive NSX with expert-led or self-paces hands-on
labs
labs.hol.vmware.com
VMware Education - Training and Certific
vmware.com/go/nsxtraining
Free NSX Training on Coursera
vmware.com/go/coursera
Engage and Learn Experience
Attend the Networking and Security Sess
Try Take