VMware Validated Design - Cloud Panda...Controllers, vCenter Server instances, ESXi hosts, and NSX components with syslog protocol. vRealize Log Insight also integrates with vRealize

VMware Validated Designfor Software-Defined Data Center 4.0

@tenthirtyam | vmware.com/go/vvd-docsCopyright © 2017 VMware, Inc. All rights reserved.

Reference

NFS Storage

SSD PCIe

Read and Write Cache

Capacity

NVMeCaching

Tier

SD

DataPersistence

Tier

VMware Validated Designs use vSAN Ready Nodes to ensure seamless compatibility and support.The configuration and assembly for each node is standardized with all components installed the same manner to

eliminate system variability. vSAN enables both hybrid and all-flash architectures.

The design used NFS storage as a secondary storage tier for management and compute pods. NFS is used as the target for vSphere Data Protection backups and vRealize Log Insight log archives in the management pod.

NFS is also used to host the virtual machine templates in the compute pods. vSphere Data Protection is interchangable with any vSphere APIs for Data Protection compatible solution.

Storage

Internet orEnterprise

WAN/MPLS

172.16.11.0/24

192.168.11.0/24

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

Region BRegion A

ECMPNSX Edge

Services Gateways

NSXMOS

VDPOS

PSCOS

VCOS

Top-of-RackLeaf Switches

L3

L2

BGP Peering

SpineSwitches

NSX Edge Services GatewayOne-Arm Load Balancer

To Shared Edge and Compute Pod

To Additional Compute Pods

Inte

rnet

or

Ent

erp

rise

WA

N/M

PLS

192.168.11.0/24


Internet orEnterprise

WAN/MPLS

ECMPNSX Edge

Services Gateways

Top-of-RackLeaf Switches

L3

L2

BGP Peering

SpineSwitches

Inte

rnet

or

Ent

erp

rise

WA

N/M

PLS

172.17.11.0/24

PSCOS

VDPOS

VCOS

NSXMOS

Management Universal Distributed Logical Router

192.168.10.0/24 192.168.10.0/24

192.168.31.0/24

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

192.168.32.0/24

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

Universal Transit NetworkUniversal Logical Switch / VXLAN Segment

Reserved for Disaster Recovery

To Shared Edge and Compute Pod

To Additional Compute Pods

IWS

IMS

VRO

VRA

VIP: 192.168.11.56192.168.11.54 > Active Node192.168.11.55 > Active Node

VIP: 192.168.11.59192.168.11.57 > Active Node192.168.11.58 > Passive Node

VIP: 192.168.11.65192.168.11.63 > Active Node

192.168.11.64 > Active Node

VIP: 192.168.11.53192.168.11.51 > Active Node192.168.11.52 > Active Node

Region B

192.168.11.0/24NSX Edge Services Gateway

One-Arm Load Balancer

ECMPNSX Edge

Services Gateways

Region Independent Application Virtual Network

192.168.10.0/24

192.168.32.0/24

IAS

APP

OS

IAS

APP

OS

BUC

APP

OS

192.168.31.0/24

IAS

APP

OS

IAS

APP

OS

BUC

APP

OS

Region Independent Application Virtual Network

192.168.11.0/24

Region AECMPNSX Edge

Services Gateways


192.168.10.0/24

VRA

IWS

IMS

DEM

VRO

IWS

IMS

DEM

VRO

APP

OS

APP

OS

APP

OS

APPOS

APP

OS

APP

OS

OS

APP

OS

APP

APP

OS

SQL BUSAPP

OSAPP

OS

APP

OSVRA

IWS

IMS

DEM

VRO

VRA

IWS

IMS

DEM

VRO

APP

OS

APP

OS

APP

OS

APPOS

APP

OS

APP

OS

OS

APP

OS

APP

APP

OS

SQL BUSAPP

OSAPP

OS

APP

OS


VRA

Distributed Logical Routing and Application Virtual Networks for Management, Operations and Automation Solutions

vRealize Automation ApplianceVRA

vRealize Automation IaaS Web ServerIWSvRealize Automation IaaS Manager ServiceIMSvRealize Automation IaaS vSphere Proxy AgentIASvRealize Automation Distributed Execution ManagerDEM

vRealize Orchestrator ApplianceVRO

vRealize Business ApplianceBUSvRealize Business Data CollectorBUC

Microsoft SQL Server DatabaseSQL

Networks Notable Acronyms

Management Application Virtual Network VXLAN

Universal Transit Network VXLAN

External Transit Network(s)

Management Distributed Port Group

Logical Component Architecture

In a dual-region SDDC, the design instantiates two Platform Service Controllers and two vCenter Server instances in the appliance form factor. This includes a vCenter Server for the management pod and a vCenter Server for the shared edge and compute pods.

Each vCenter Server instance is connected to a load-balanced pair of Platform Services Controllers using an NSX Edge Services Gateway. To enable enhanced linked mode, the design joins the Platform Services Controller instances into a unified Single Sign-On domain

Region A

Common vCenter Single Sign-On Domain(Ring Topology)

Region B

Platform ServicesControllerAppliance

Management StackvCenter Server

Appliance


Compute StackvCenter Server

Appliance



vSphere Update Manager Download

Service


Appliance


Appliance

In a dual-region SDDC, two separate NSX Manager instances are deployed in Region A. One instance for the management pod and one instance for the shared edge and compute pods, along with associated NSX Universal Controller Clusters.

In Region B the secondary NSX Manager instances automatically import the configurations of the NSX Universal Controller Clusters from Region A.

Region B Management PodRegion A Management Pod

Region B Shared Edge and Compute Pod(Edge Resource Pool)

Region A Shared Edge and Compute Pod(Edge Resource Pool)

NSX Edge Services Gateways

(N/S Routing)

NSX Edge Services Gateway w/ HA

(One-Arm Load Balancer)


(N/S Routing)


(One-Arm Load Balancer)

Management StackNSX Manager

(Primary)

Compute StackNSX Manager

(Primary)


Appliance


Appliance

Management Stack NSX Universal

Controller Cluster

Management StackNSX Manager(Secondary)

Compute StackNSX Manager(Secondary)


Appliance


Appliance

Import of Management StackNSX Controller Configurationfrom Primary NSX Manager

Compute Stack NSX Universal Controller

Cluster

Import of Compute Stack NSX Controller Configuration

from Primary NSX Manager

NSX Manager Pairing

NSX Manager Pairing

Region A Region B


(N/S Routing)


(N/S Routing)

In a dual-region SDDC, a vRealize Log Insight cluster is deployed in each region and consists of three nodes, enabling continued availability and increased log ingestion rates. vRealize Log Insight collects log data from Platform Service Controllers, vCenter Server instances, ESXi hosts, and NSX components with syslog protocol. vRealize Log Insight also

integrates with vRealize Operations Manager to send notication events and facilitate root cause analysis.

Region A Region B

Management / Compute

vCenter Servers

NSX

vSAN

vRealize Log Insight Cluster vRealize Log Insight Cluster NSX

MasterNode

WorkerNode

WorkerNode

MasterNode

WorkerNode

WorkerNode

vRealizeAutomation


vCenter Servers

vRealizeOperations

vSAN

Primary Storage

NFS

Log Archives

vSAN

Primary Storage

NFS

Log Archives

Event Forwarding

via Ingestion API

vSAN

vRealize Log Insight

VRO

VRO

vRealizeOrchestrator

BUS

BUC

vRealizeBusiness

VRA IWS IMS DEM IAS

VRA IWS IMS DEM IAS

SQL

vRealizeAutomation

BUC

vRealizeBusiness

IAS

IAS

vRealizeAutomation

BGPPeering

BGPPeering

BGPPeering

BGPPeering

BGPPeering

BGPPeering

Refer to the design release notes for products and versions included in the design.

Replicated for Disaster RecoveryvRealize Automation / vRealize Orchestrator vRealize Business for Cloud

Core vSphere Management NSX

vRealize Automation, vRealize Orchestrator and vRealize Business for Cloud

Distributed Logical Routingand Application Virtual Networks

vRealize Operationsand vRealize Log Insight

vRealize Automation, vRealize Orchestrator and vRealize Business for Cloud

vSAN Ready Nodes

Core and Pod Architecture

Management Distributed Switch

Universal Management Transport Zone

plus NFS

vSAN

Management Cluster

Minimum 4 vSAN Ready Nodes Recommended | vSAN EnabledvSphere HA and DRS Enabled

ESXi

VTEP VTEP

ESXi

VTEP VTEP

ESXi

VTEP VTEP

ESXi

VTEP VTEP

Management Stack

Management Pod

The management pod hosts the infrastructure components used to instantiate, manage and monitor the SDDC. This includes the core infrastructure

components, such as the Platform Services Controllers, vCenter Server instances, NSX Managers, NSX Controllers for the management stack, vSphere Replication,

Site Recovery Manager, as well as the SDDC monitoring and automation solutions like vRealize Operations, vRealize Log Insight and vRealize Automation.

Managed by Management Stack vCenter Server

Workloads running in the SDDC do not have direct access to external networks.To access external networks, tra c is routed through distributed routing

to the NSX Edge Services Gateways in the shared edge and compute pod. Expansions beyond the initial shared pod are simply compute pods.

plus NFS

Shared Edge and Compute Cluster& Compute Cluster n

Minimum 4 Nodes | vSAN Ready Nodes RecommendedvSphere HA and DRS Enabled | Business Workload Requirements

Compute Distributed Switch

ESXi ESXi ESXi ESXi

Any Supported Storage

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Shared Edge and Compute Pod& Compute Pod n

Compute StackManaged by Compute Stack vCenter Server

The design uses a small set of common, standardized building blocks called pods.

L3

L2

The physical network architecture in the design is tightly coupled with the pod-and-core architecture and uses a Layer 3 leaf-and-spine network model for an e cient, resilient, and distributed core.

Spine Spine Spine

vSphere Update Manager Download

Service

vRealizeAutomation

Proxy Agents

vRealizeOperations

RemoteCollectors

vSphere Update Manager Download Service, vRealize Operations Analytics Cluster and Remote Collectors, Regional vRealize Log Insight Cluster,

Distributed vRealize Automation and Proxy Agents, and vRealize Business for Cloud Server and Collector.

vSphere Update Manager Download Service,vRealize Operations Remote Collectors, Regional vRealize Log Insight Cluster,vRealize Automation Proxy Agents and vRealize Business for Cloud Collector.

Disaster Recovery vRealize Operations Analytics Cluster, Distributed vRealize Automation, and vRealize Business for Cloud Server.

Application Virtual Networks for SDDC Management Solutions in Region A Application Virtual Networks for SDDC Management Solutions in Region B




Universal Logical Switch / VXLAN Segment Universal Logical Switch / VXLAN Segment

Region Dependent Application Virtual NetworkUniversal Logical Switch / VXLAN Segment


Region Independent Application Virtual NetworkUniversal Logical Switch / VXLAN Segment




Server

10 GigE

L3

L2

L3

L2

10 GigE

Additional Compute Pods(Up to 19 2RU Hosts or 19 vSAN Ready Nodes)

Shared Edge and Compute Pod(4+ Hosts)

Management Pod(4 vSAN Ready Nodes)

LeafLeafLeafLeafLeafLeaf

IGMP IGMP IGMP IGMP IGMP IGMP

Leaf Leaf

IGMP IGMP

Host Connectivity

Management Distributed Switch

Management Pod ESXi Host

nic0 nic1

VLAN NFS

VLAN Management

VLAN vMotion

VLAN VTEP (VXLAN)

VLAN vSphere Replication

VLAN vSAN

VLAN External Management

VLAN Uplink 02

VLAN Uplink 01

MTU9000

MTU9000

MTU9000

MTU9000

MTU9000

MTU9000

10 GigE 10 GigE

MTU 9000

Compute Distributed Switch

Shared Edge and Compute Pod ESXi Host

nic0

MTU 9000

nic110 GigE 10 GigE

VLAN NFS

VLAN Management

VLAN vMotion

VLAN VTEP (VXLAN)

VLAN vSAN

MTU9000

MTU9000

MTU9000

MTU9000

MTU9000

VLAN Uplink 02

VLAN Uplink 01

Layer 3 ToR Switch

VLAN 1611 VLAN 1612 VLAN 1613 VLAN 1614

Management172.16.11.0/24

DGW:172.16.11.253

vMotion172.16.12.0/24

DGW:172.16.12.253

VXLAN172.16.13.0/24

vSAN172.16.14.0/24

ESXi Host

Routed Uplinks (ECMP)

VLAN Trunk (802.1Q)

L2

L3

Span

of

VLA

Ns

Span

of

VLA

Ns

The leaf switches of each rack acts as the Layer 3 interface for the corresponding subnet. All pods are provided with externally accessible VLANs for access to the Internet and corporate networks.

The two 10GbE NICs on each host are connected across the top-of-rack leaf switches and teamed on the vSphere Distributed Switch via an active-active configuration.All port groups, except for the ones that carry VXLAN tra c, are configured for the 'Route based on physical NIC load' teaming algorithm.

VTEP kernel ports and VXLAN tra c use the ’Route based on SRC-ID' algorithm. The vSphere Distributed Switch has a MTU of 9000 configured for Jumbo Frames along with with necessary VMkernel ports.

Spine Spine

40 GigE 40 GigE

Leaf-and-Spine Network

Region Protection and Disaster Recovery

Pod and Clusters

SRM

Region ANon-Replicated


vRealize Automation vRealize Orchestrator

vRealize Business for CloudvRealize Operations

(using vSphere Replication) SRM

vRealize AutomationvRealize Orchestrator

vRealize Business for CloudvRealize Operations

(using vSphere Replication)

Region BNon-Replicated


Region A Infrastructure Management

vSphereNSX

Site Recovery ManagervSphere Data Protection

Region B Infrastructure Management

vSphereNSX

Site Recovery ManagervSphere Data Protection

Region A Replicated Region B Replicated

Pods

Management

North/South Uplink(s)

vMotion

vSAN

Region Dependent VXLAN

vSAN

ESXi-MGMT-01 ESXi-MGMT-02 ESXi-MGMT-03VTEPs VTEPs VTEPs

Management

North/South Uplink(s)

vMotion

vSAN

VXLAN xxxx

Any SupportedStorage

ESX-COMP-01VTEPs


Region A Management Pod Region B Management Pod

Region A Management Pod Region B Management Pod

NSX Controllers(Compute)

Region A

Export(vRealize

Automation)

Export(vRealize

Log Insight)

Export(vSphere

Data Protection)

NFS Storage Array

Volume 2Volume 1

Region B

Export(vRealize

Automation)

Export(vRealize

Log Insight)

Export(vSphere

Data Protection)

NFS Storage Array

Volume 2Volume 1

vCenter(Compute)

PSC (Compute)

NSX Manager(Compute)

NSX Manager(Management)

PSC(Management)

vCenter(Management)

VDP(Management)

SRM(Management)

VR(Management)

ExternalNetworks

N/S NSX EDGE(Compute)

vRealize AutomationBusiness Groups & Reservations

The design uses several VMware solutions for network, storage, and cloud management. You can monitor and perform diagnostics on all of them by using vRealize Operations and solution management packs. vRealize Operations

Region Independent VXLAN VXLAN xxxx

Management Pod Edge/Compute Pod

L3

L2

L3

L2

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

Universal Logical Switch

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS


APP

OS

APP

OS

APP

OS

APP

OS


APP

OS

APP

OS

APP

OS


APP

OS

L2

L2

UDLRUDLRUDLR

EdgeResource

Pool

UDLR

UDLR

UDLR UDLR & DLR

N/S NSX EDGE(Management)

UDLR & DLR

L3

L3

L2

L2

L3

L3

L3 L3L3 L3

Dis

trib

uted

Swit

ches

Uni

vers

alT

rans

po

rt Z

one

sC

ore

Pla

tfo

rmSe

rvic

esA

pp

licat

ion

Vir

tual

Net

wo

rks

for

SDD

C S

olu

tio

ns

Wo

rklo

ad V

irtu

al N

etw

ork

s

No

rth/

So

uth

Ro

utin

g

NSX Controllers(Management)

Man

agem

ent

Cus

ter

Ed

ge/

Co

mp

ute

Clu

ster

VMware Validated Designs establish a Cloud Management Platform with vRealize Automation to provide the service catalog and self-service portal to deploy, update, and manage the workloads. vRealize Orchestrator

provides a repository of extensible workflows and integrations. vRealize Business for Cloud provides visibility into the financial aspects of the cloud infrastructure, allowing cost to be tracked and optimized.

The design implements a single vRealize Automation tenant. Business groups can be created to fit your needs. Within each business group the tenant administrators are able to manage users and groups, apply tenant-specific

branding, enable notifications, configure business policies, and manage the service catalog.

One region is designated as the “primary” region and the other as the “secondary” region. The SDDC management, automation and operations solutions are deployed in the primary region and configured to migrate to the secondary region in the event of a disaster. All regions actively run business workloads.

https://my.sddc.local/vcac/org/company

Sign In

BusinessGroupManager

BusinessGroupManager

TenantAdmin

IaaSAdmin

TenantAdmin

IaaSAdmin

Business GroupReservation


EdgeReservation

Region B Data Center Infrastructure Fabric

Region B Fabric GroupFabricAdmin

Additional Compute Pod(s)

UDLR & DLR

UDLR & DLR

Shared Edge/Compute Pod

EdgeReservation



Region A Data Center Infrastructure Fabric

Region A Fabric GroupFabricAdmin

Additional Compute Pod(s)Shared Edge/Compute Pod

Universal Compute Transport Zone

Platform Services ControllerPSCNSX ManagerNSXM

Site Recovery ManagerSRMUniversal Distributed Logical RouterUDLR

VXLAN Tunnel EndpointVTEP

vSphere Data ProtectionVDPvSphere ReplicationVR

192.168.11.0/24

Region AECMPNSX Edge

Services Gateways


192.168.10.0/24

192.168.31.0/24

MasterNode

APP

OS

APP

OSAPP

OS

ReplicaNode

DataNode

APP

OS

APP

OS

CollectorNode

CollectorNode

vRealize Operations

vRealize Operations


ClusterVIP

APP

OSAPP

OS

APP

OS

MasterNode

WorkerNode

WorkerNode

Region B

192.168.11.0/24


ECMPNSX Edge

Services Gateways

192.168.10.0/24

192.168.32.0/24

APP

OS

APP

OS

CollectorNode

CollectorNode

vRealize Operations


Replicated for Disaster Recovery

ClusterVIP

APP

OSAPP

OS

APP

OS

MasterNode

WorkerNode

WorkerNode

MasterNode

APP

OS

APP

OSAPP

OS

APP

OS

ReplicaNode

DataNode


Local Compute Transport Zone






(Load Balancer)


(Load Balancer)

SharedStorageSystems

Region A Region B


vCenter Servers

NSX Remote Collectors Remote Collectors NSX

SharedStorageSystems

Analytics Cluster

MasterNode

ReplicaNode

DataNode

ClctrNode

ClctrNode

ClctrNode

ClctrNode

vRealizeAutomation


vCenter Servers




Documents

VMware Validated Design - Cloud Panda...Controllers, vCenter Server instances, ESXi hosts, and NSX components with syslog protocol. vRealize Log Insight also integrates with vRealize