Containers and Kubernetes Put a Lid on It: Securing for ... · Put a Lid on It: Securing Containers and Kubernetes on vSphere and AWS Steve Hoenisch, VMware, Inc. Nolan Karpinski,

#vmworld

Put a Lid on It: Securing Containers and Kubernetes

on vSphere and AWSSteve Hoenisch, VMware, Inc.Nolan Karpinski, VMware, Inc.

CNA1656BE

#CNA1656BE

VMworld 2018 Content: Not for publication or distribution

Disclaimer

2©2018 VMware, Inc.

This presentation may contain product features orfunctionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.


Agenda


The Cloud-Native Stack

Container Security Risks and Threats

Security in Pivotal Container Service on vSphere

Security in VMware Cloud PKS on AWS

Security with AppDefense and Demo

Summary



The Cloud-Native StackLayers and Processes Requiring Security

Applications

Infrastructure

Orchestration Platform

Container Registry

Container Runtime

Container Host

Container Images

Container Management



Container LifecycleSecurity for Developers Takes Place in this Workflow

DeployBuild Run



Risks and ThreatsAttack Vectors throughout the Stack Identified by NIST



Just like traditional apps, containerized apps are vulnerable to typical flaws.

Vulnerability can be exploited.

Containers give you a prescribed way to apply security principles with depth and breadth.

Core protection strategies help secure cloud workloads.

The Application

Applications

Infrastructure


Container Registry

Container Runtime

Container Host

Container Images




Images with vulnerabilities, malware, embedded secrets, etc.

Images that are outdated or unpatched or of unknown origin

Insecure connectivity

Inadequate access control

Insecure configuration (running as privileged user or with SSH)

The rogue containers of hasty developers

Operating systems with a large attack surface, shared kernel, or package vulnerabilities

Container ManagementPortability and Reuse Heighten Risks

Applications

Infrastructure


Container Registry

Container Runtime

Container Host

Container Images




Unsecured components, like the Kubernetes Dashboard, on the Internet

Lack of standard directory service and RBAC

Unencrypted data

Apps sharing the same virtual networks

Mixing workloads with different sensitivity levels and threat postures

Risks to the Orchestration SystemUnsecure Components, Lack of Access Control, and Mixed Workloads Pose Threats

Applications

Infrastructure


Container Registry

Container Runtime

Container Host

Container Images




Unprotected data in transit and in storage heightens risks of a breach

Lack of logging, monitoring, and visibility can make it hard to identify intrusions and respond quickly

Infrastructure

Applications

Infrastructure


Container Registry

Container Runtime

Container Host

Container Images




Security in VMwarePivotal Container Service Container Images

Access Control

Micro-Segmentation

Logging and Monitoring



Architecture



Harbor: Image Registry Imposes CountermeasuresScans, Secures, and Signs Images and then Prescribes Use with Policies

DevelopmentTeam

RBAC

UAA AUTH REPLImage Pull

K8s Cluster deployed by PKS

Build Image Push Image Scan ImageFor CVEs

Sign Trusted Image

kubectl run

ImageRegistry

Image Scanning

Image Signing

HarborProjects

AUDIT LOGGING



Identity and Access ManagementControlling Access to Kubernetes Clusters

Operator admin

Developers Developers

Kubernetes

Namespace Namespace

Namespace NamespaceKubernetes

Namespace Namespace

Namespace Namespace

PKS Broker

UAA PKS API

BOSH CredHub

Authentication and RBAC for PKS CLI and Kubernetes API

Centralized credential generation and management with CredHub

PKS secures the Kubernetes Dashboard by default with user authentication.



PKS with NSX-T Provides Strong Isolation for Kubernetes ClustersEach Cluster Isolated on its own Network Segment

Virtual Switch

K8 Worker

K8 Worker

K8 Worker

Kubernetes Cluster

T1 Router

Virtual Switch

K8 Worker

K8 Worker

K8 Worker

Kubernetes Cluster

T1 Router

T0 RouterLB LB

Cloud Provider Uplinks Easily create clusters and then use a cluster as a unit of tenancy in a multi-tenant context.

Tenants get separate clusters. NSX ensures isolation is

logically enforced in the network.

NSX isolates orchestrator traffic from workload traffic.

Cluster Networking



NSX-T Isolates Namespaces with Logical Switches and RoutersProvides Stronger Isolation than Default K8s Namespaces

admin@k8s-master:~$ kubectl create namespace foonamespace ”foo" created

admin@k8s-master:~$ kubectl create namespace barnamespace ”bar" created

admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foodeployment "nginx-foo" created

admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bardeployment "nginx-bar" created

Namespace: foo Namespace: bar

K8s Topology with NSX

10.24.0.0/24 10.24.1.0/24 10.24.2.0/24

NAT boundary

K8s nodesK8s Masters

• Within each cluster, each namespace gets its own network segment for strong isolation.



NSX-T Can Use K8s Network Policies for Micro-SegmentationSegments Pods by Applying Dynamic Security Groups and Policies

admin@k8s-master:~$ vim nsx-demo-policy.yamlapiVersion: extensions/v1beta1kind: NetworkPolicymetadata:name: nsx-demo-policy

spec:podSelector:matchLabels:app: web

ingress:- from:

- namespaceSelector:matchLabels:ncp/project: db

ports:- port: 80protocol: TCP

admin@k8s-master:~$ kubectl create -f nsx-demo-policy.yaml

Namespace: foo Namespace: bar

NSX / K8s topology

10.24.0.0/24 10.24.1.0/24 10.24.2.0/24

NAT boundary

DBLabel: app=db

WebLabel: app=web

• Assign policies in Kubernetes that get translated into to NSX policies and firewall rules.

• Policies are defined as part of app’s deployment for portability.



Pod Micro-SegmentationNSX Uses Network Policy to Dynamically Create Security Groups and Policy

$ kubectl create -f nsx-demo-policy.yaml

Dynamic Creation of Security Groups

Dynamic Creation of Security Policy based on K8s Network PolicyVMworld 2018 Content: Not for publication or distribution


Getting Visibility Across the StackTraceflow with NSX-T

Monitor network traffic

Trace packets from containers to physical networks

Visualize traffic flows across the stack

Other tools:

Port Mirroring

Port Connection Tool

Spoofguard

Syslog

Port Counters

IPFIX

NSX-T Traceflow



vRealize Log Insight and vRealize Operations

Gain comprehensive visibility across apps and infrastructure

Analyze logs for suspicious activity

Monitoring, Logging, and AnalyticsOperational Visibility and Auditing

vRLI vROPS

Logs Metrics



Lifecycle Management Maintains SecurityBOSH Patches and Repairs Nodes

VMware GCP

AvailabilityZone

master

etcd

worker

etcd etcd

master

worker worker

master

etcd

worker

etcd etcd

master

worker worker

AvailabilityZone

AvailabilityZone

BOSH

Health Monitor

Health Monitor

worker workerPatch

K8sNewVer

Release

Repair

Deploy

ScaleUpgradePatchRepair

Day 1

Day 2

workerworker

PKS

Con

trol

Pla

ne

CVE



Operator admin

PKS Security Overview

InfrastructureStorageCompute Networking

Container Mgmt.

Namespace

vSphere Google Cloud Platform

Hybrid

Namespace

Namespace Namespace

Container Mgmt.

Namespace Namespace

Namespace

Developers

IAM

Events & Monitoring

Build Pipelines

Namespace

Platform LCM

Kubernetes Cluster Kubernetes Cluster

Apps Apps

Micro-service

Micro-service

Micro-service

Micro-service

Micro-service

Micro-service

Micro-service

Micro-service

Platform LCM


Infrastructure

LDAP or AD

IAM

VNFM

vRealizeOperations

Monitoring

Logging

vRealize Log Insight



Security in VMware Cloud PKS

Overview

Access Control Policies (Lightwave)

Linux Container Host (Photon OS)



VMware Cloud PKS OverviewHighly Secure and Available Kubernetes Service on AWS

Fully managed for you by VMware

Smart Clusters Elastic cluster size HA configuration Automatic recovery from

failure Granular multi-tenant access

policies Multi-cloud ready Differences between

VMware Cloud PKS and PKS

US west

US east

Europe West



Manage user identities centrally in VMware Cloud Services

Access VMware Cloud Services, VMware Cloud PKS, and all your Kubernetes clusters with your identity

Single Sign-On with Unified Identity at cloud.vmware.com

25

Part of VMware Cloud Services



Applications

Infrastructure


Container Registry

Container Runtime

Container Host

Container Images


Access Control Policies



Multi-Tenant Access PoliciesAccess Control Encompasses Entire Service

ENGINEERING FOLDER QUOTAPOLICY

DEV/TEST PROJECT

SMART CLUSTER

NAME SPACE

SMART CLUSTER

PRODUCTION PROJECT

SMART CLUSTER SMART CLUSTER

AI PROJECT

SMART CLUSTER

FINANCE FOLDER

EU-West-1US-West-2 US-East-1 US-West-2

SMART CLUSTER

EU-West-1

ACCESS POLICIES

ACCESS POLICIES

QUOTAPOLICY

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE



Role-Based Access Control (RBAC) Tenants can organize clusters

with folders and projects.

Roles grant a configurable set of permissions, such as administering or viewing a cluster or a namespace.

Access polices are sets of roles bound to users or groups.

Roles are inherited down the tree.

VMware Cloud PKS pushes policies to Kubernetes and translates them into Kubernetes RBAC.

Policies can apply two key principles: Separation of dutiesand principle of least privilege.

Organization

Folder 1 Folder 2

Project 1 Project 2

Cluster 1 Cluster 2

Namespace 1 Namespace 2

SmartCluster.Admin: ClusterAdmin1


SmartCluster.Admin: ClusterAdmin2SmartCluster.Admin: ClusterAdmin1



Project 1

Cluster 1



Lightwave in VMware Cloud PKS Directory Service, Certificate Authority, Secure Token Service

31

LIGHTWAVEDIRECTORY SERVICE

LW D

C1

LW D

C2 LW

DC

1

LW D

C2

ESXiHOST

ESXiHOST

ESXiHOST

ESXiHOST

ESXiHOST

C

K8SK8S

NNN

NNN

Cluster

NN

NN

Cluster

NNN

NNN

Cluster

T1T4 T2N

NN

Cluster

T3

K8SK8S

AWS US-Region

N

K8S K8S

N N N

N N

Cluster

N N

N N

Cluster

N N N

N N N

Cluster

T1 T4T2 N

N N

Cluster

T3

K8S K8S

CC

1 NSXManager

CC

2

CC

3

CC

3NSXManager

CC

2

CC

1

AWS EU-RegionLW

DC

3

LW D

C3

N

Photon OS

https://github.com/vmware/lightwaveVMworld 2018 Content: Not for publication or distribution


Minimalist: The number of packages is limited to the minimum necessary for hosting containers.

Security-hardened Linux: The kernel is configured according to the recommendations of the Kernel Self-Protection Project (KSPP).

Curated packages and repositories: Packages are built with hardened security flags.

Advanced lifecycle management: There are timely security patches and updates to container packages, such as Docker and Kubernetes.

Project Lightwave integration: Lightwave clients are installed on Photon OS by default, which let it join to Lightwave domain and be managed by Lightwave consistently

Photon OSA Linux Container-Optimized Operating System

https://github.com/vmware/photonVMworld 2018 Content: Not for publication or distribution


Security Managed for You

Data encryption at rest (EBS volumes are encrypted; keys are managed by Amazon)

Data encryption in motion (TLS)

Encrypted K8s secrets in etcd

Data Encryption

Kubernetes

Automatic secure OS updates on Kubernetes nodes

Patches and Upgrades



Each Organization is mapped to an AWS account managed by VMware Cloud PKS

Each production Smart Cluster is deployed on a separate Network segment

Infrastructure Isolation on AWS

VMware Cloud PKS

Acme Globex

user@Acme user@Globex

VPC VPC VPC

AWS cloud

AWS Account(vke.tenant.acme) AWS Account(vke.tenant.globex)

Smart Cluster Smart Cluster Smart Cluster



Container Security at RuntimeAppDefense and Aqua Security



Applications

Infrastructure


Container Registry

Container Runtime

Container Host

Container Images


Application Layer



Opportunity to Focus on Core Protection StrategiesGartner Market Guide for Cloud Workload Protection Framework

AV

Deception

HIPS withVulnerability Shielding

Server Workload EDRBehavioral Monitoring

IaaS Data at Rest Encryption

Exploit Prevention / Memory Protection

Application Control / Whitelisting

System Integrity Monitoring / Management

Network Firewalling, Segmentation and Visibility

Hardening, Configuration and Vulnerability ManagementFoundational

Less Critical

Optional Protection Strategies

Core Protection Strategies

Important, but often provided outside of CWPP

Figure 1. Cloud Workload Protection Controls Hierarchy, © 2018 Gartner, Inc.

Source: Gartner, Market Guide for Cloud Workload Protection Platforms, Neil MacDonald, March 26th 2018. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document.



Gartner Market Guide for Cloud Workload Protection Framework

AV

Deception

HIPS withVulnerability Shielding

Server Workload EDRBehavioral Monitoring

IaaS Data at Rest Encryption

Exploit Prevention / Memory Protection

Application Control / Whitelisting

System Integrity Monitoring / Management

Network Firewalling, Segmentation and Visibility

Hardening, Configuration and Vulnerability ManagementFoundational

Less Critical

Optional Protection Strategies

Core Protection Strategies

Important, but often provided outside of CWPP

Figure 1. Cloud Workload Protection Controls Hierarchy, © 2018 Gartner, Inc.

Source: Gartner, Market Guide for Cloud Workload Protection Platforms, Neil MacDonald, March 26th 2018. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document.

Post-deployment Controls

Opportunity to Focus on Core Protection Strategies



Cyber ThreatsResidual Risk

Apps Data

Cyber HygieneAttack Surface

EncryptionMicro-Segmentation

RepavingLeastPrivilege

Multi-FactorAuthentication


40

vSphere

AppDefense Manager

AppDefenseAppliance

vCenter

SaaS

Guest OS

Guest Module

Host Module

Customer Data Center

Guest OS

Guest Module

Guest OS

Guest Module

1. Application Context2. Central Intelligence

Architecture

ContainerContainerContainerContainerContainerContainer

ContainerContainerContainer



1001010101001100101001010101010110100101010100110010100101010101011010010101010011001010010101010101101001010101001100101001010101010110100101010100110010100101010101011010010101010011001010010101010101101001010101001100101001010101010110100101010100110010100101010101011010010101010011001010010101010101101001010101001100101001010101010110100101010100110010100101010101011010010101010011001010010010101011010010101010011001010010101010101101001010101001100101001010101010110100101010100110010100101010101011010010101010011001010010101010101010010101010011001010010101010101101

Changing the Application Security ModelFrom chasing bad to ensuring good

Pro

cess

es

Pro

cess

es

Pro

cess

es

OSXX,000,000 XX

Chasing Bad Ensuring Good



Ensuring Known Good

Detect Deviations RespondCapture & Analyze

Learn Intended State Protect Intended State

ManifestApp ManifestApp



LearnCapture the purpose and intended state of a container

Intended State Engine

vCenter IaaS/PaaS

AppScope

Off-the-shelf images Custom imagesCI/CD pipeline

Machine Learning!

Learn Protect

Detect RespondCapture & Analyze

ManifestApp

ManifestApp

ManifestApp



DetectRuntime application attestation and secure manifest store

Co

ntai

ner

Co

ntai

ner

Co

ntai

ner

OS

AppDefenseMonitor

Co

ntai

ner

Co

ntai

ner

Co

ntai

ner

OS

AppDefenseMonitor

Co

ntai

ner

Co

ntai

ner

Co

ntai

ner

OS

AppDefenseMonitor

Protected zone

Learn Protect

Detect RespondCapture & Analyze

ManifestApp ManifestAppManifestApp



RespondOrchestrated incident response routines for the SOC

Secureinfrastructure

IntegratedEcosystem

Learn Protect

DetectRespond

Capture & Analyze

Add Behavior Block/Alarm

Quarantine Repave!


Integrated Solution Architecture

46

ContainerContainerContainer

Hypervisor

AppDefense Manager

Aqua Manager

Aqua Agent

SaaS

Guest OS

Container Host

Integration Container



Demo: AppDefense with Aqua



Applications

Infrastructure


Container Registry

Container Runtime

Container Host

Container Images


Summary



NIST Application Container Security Guide

NIST Security Assurance Requirements for Linux Application Container Deployments

Cloud-Native Stack Security: How VMware Pivotal Container Service Secures Containers and Kubernetes

Control Access with VMware Cloud PKS

Containers on Virtual Machines or Bare Metal? Deploying and Securely Managing Containerized Applications at Scale

Glossary of Cloud-Native Terms

Securing Cloud Platforms with Project Lightwave

Photon OS: A Linux Container-Optimized Operating System

A Dash of Security: Locking Down Kubernetes Admin Access

References and Other ResourcesGuides and White Papers


https://doi.org/10.6028/NIST.SP.800-190

https://doi.org/10.6028/NIST.IR.8176

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/whitepaper/vmware-pks-security-white-paper.pdf

https://assets.cloud.vmware.com/v3/assets/blt719094f4883f620b/blt0e1847c6dbf03990/5b30443d073818f857728872/download

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/whitepaper/vmw-wp-conatiner-on-vms-a4-final-web.pdf

https://assets.cloud.vmware.com/v3/assets/blt719094f4883f620b/bltcee5618b0aca3766/5a8da0f511ef86064e4106d9/download?disposition=inline

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/cloudnative/vmware-securing-cloud-platforms-with-project-lightwave.pdf

https://vmware.github.io/photon/assets/files/photon-os-datasheet.pdf

https://blogs.vmware.com/cloudnative/2018/07/20/a-dash-of-security-locking-down-kubernetes-admin-access-to-thwart-crytojacking/


Summary: Best PracticesImplement Container-Specific Countermeasures

Integrate countermeasures into life cycle and pipeline

Monitor containers thru life cycle & stack for full visibility

Enforce security with policies for RBAC and image use

Use only the latest patched, scanned, and signed images

Run images as non-privileged, immutable containers without SSH and manage through the orchestrator

Securely store secrets, encrypted, in the orchestrator, not the image

Connect to registries, dashboards over secure channels

Control access to registries, orchestrators, and dashboards with RBAC using principles of least privilege and separation of duties

Provide single sign-on with a single, standard directory

Log, monitor, and audit access to registry and orchestrator

Encrypt data at rest using container-specific methods; see NIST 800-111

Segment orchestrator traffic into discrete virtual networks by sensitivity level

Only mix workloads of the same sensitivity level and threat posture on the same host

Use a patched, up-to-date, CVE-monitored runtime

Constrain network access from containers

Profile and protect apps at runtime to ensure known good

Use an up-to-date, minimalist container OS to narrow the attack surface; see NIST SP 800-123

Set the root file system to read-only

Limit, log, and audit host OS access to detect anomalies and privileged operations


DON’T FORGET TO FILL OUT YOUR SURVEY.

#vmworld #CNA1656BE


THANK YOU!

#vmworld #CNA1656BE


Documents

Containers and Kubernetes Put a Lid on It: Securing for ... · Put a Lid on It: Securing Containers and Kubernetes on vSphere and AWS Steve Hoenisch, VMware, Inc. Nolan Karpinski,