Upload
jody-donna-carpenter
View
224
Download
1
Tags:
Embed Size (px)
Citation preview
File Storage Strategies for Private CloudJose BarretoPrincipal Program ManagerFile Server and Clustering teamMicrosoft
WS-B309 Blog: http://smb3.info
• Describe the basics of the Hyper-V over SMB scenario, including the main reasons to implement it.
• Enumerate the most common performance bottlenecks in Hyper over SMB configurations.
• Outline a few Hyper-V over SMB configurations that can provide continuous availability, including details on networking and storage.
Session Objectives
Sample Configurations
Agenda
Hyper-V over SMB - Overview
Performance Considerations
Basic Configurations
Overview
Hyper-VHyper-VHyper-VHyper-VHyper-VHyper-V
Hyper-V over SMBWhat is it?• Store Hyper-V files in shares over the SMB 3.0
protocol(including VM configuration, VHD files, snapshots)
• Works with both standalone and clustered servers
(file storage used as cluster shared storage)
Highlights• Increases flexibility• Eases provisioning, management and migration• Leverages converged network• Reduces CapEx and OpEx
Supporting Features• SMB Transparent Failover - Continuous availability• SMB Scale-Out – Active/Active file server clusters• SMB Direct (SMB over RDMA) - Low latency, low
CPU use• SMB Multichannel – Network throughput and
failover• SMB Encryption - Security• VSS for SMB File Shares - Backup and restore• SMB PowerShell - Manageability
File Server
File Server
SharedStorage
Hyper-V
SQLServer
IIS
VDIDesktop
Hyper-V
SQLServer
IIS
VDIDesktop
Hyper-V
SQLServer
IIS
VDIDesktop
Hyper-V Cluster
File Server Cluster
File Server Cluster
SMB Transparent FailoverFailover transparent to server application• Zero downtime – small IO delay during failover
Supports planned and unplanned failovers• Hardware/Software Maintenance• Hardware/Software Failures• Load Rebalancing
Resilient for both file and directory operations
Requires:• File Servers configured as Windows Failover
Cluster• Windows Server 2012 on both the servers
running the application and file server cluster nodes
• Shares enabled for “continuous availability” (default configuration for clustered file shares)
• Works for both classic file server clusters (cluster disks) or scale-out file server clusters (CSV)
Hyper-V
Failover share - connections and handles lost,temporary stall of IO
2
2
Normal operation1
Connections and handles auto-recoveredApplication IO continues with no errors
3
1 3
File Server Node A
File Server Node B
\\fs\share \\fs\share
SMB Scale-OutTargeted for server app storage• Example: Hyper-V and SQL Server• Increase available bandwidth by adding
nodes• Leverages Cluster Shared Volumes (CSV)
Key capabilities:• Active/Active file shares• Fault tolerance with zero downtime• Fast failure recovery• CHKDSK with zero downtime• Support for app consistent snapshots• Support for RDMA enabled networks• Optimization for server apps• Simple management
Single File System Namespace
Cluster Shared Volumes
Single Logical File Server (\\FS\Share)
Hyper-V Cluster(Up to 64 nodes)
File Server Cluster
(Up to 8 nodes)
Datacenter Network(Ethernet, InfiniBand or
combination)
SMB Client SMB Server
SMB ServerSMB Client
User
Kernel
Application
DiskR-NIC
Network w/RDMA support
NTFSSCSI
Network w/RDMA support
R-NIC
SMB Direct (SMB over RDMA)
Advantages• Scalable, fast and efficient storage access• High throughput with low latency• Minimal CPU utilization for I/O processing• Load balancing, automatic failover and
bandwidth aggregation via SMB Multichannel
Scenarios• High performance remote file access for
application servers like Hyper-V, SQL Server, IIS and HPC
• Used by File Server and Clustered Shared Volumes (CSV) for storage communications within a cluster
Required hardware• RDMA-capable network interface (R-NIC)• Three types: iWARP, RoCE and Infiniband• RDMA NICs should not be teamed (use SMB
Multichannel)
Multiple RDMA NICsMultiple 1GbE NICsSingle 10GbE RSS-capable NIC
SMB Server
SMB Client
SMB Multichannel
Full Throughput
• Bandwidth aggregation with multiple NICs
• Multiple CPUs cores engaged when NIC offers Receive Side Scaling (RSS)
Automatic Failover
• SMB Multichannel implements end-to-end failure detection
• Leverages NIC teaming (LBFO) if present, but does not require it
Automatic Configuration
• SMB detects and uses multiple paths
SMB Server
SMB Client
SMB Server
SMB Client
Sample Configurations
Multiple 10GbE in LBFO team
SMB Server
SMB ClientLBFO
LBFO
Switch10GbE
NIC10GbE
NIC10GbE
Switch10GbE
NIC10GbE
NIC10GbE
NIC10GbE
NIC10GbE
Switch1GbE
NIC1GbE
NIC1GbE
Switch1GbE
NIC1GbE
NIC1GbE
Vertical lines are logical channels, not cables
Switch10GbE/IB
NIC10GbE/
IB
NIC10GbE/
IB
Switch10GbE/IB
NIC10GbE/
IB
NIC10GbE/
IB
Switch10GbE
SMB Encryption
End-to-end encryption of SMB data in flight• Protects data from eavesdropping or
snooping attacks on untrusted networks
Zero new deployment costs• No need for IPSec, specialized
hardware, or WAN accelerators
Configured per share or for the entire server
Can be turned on for a variety of scenarios where data traverses untrusted networks
• Application workload over unsecured networks
• Branch Offices over WAN networks
ServerClient
SMB Encryption
VSS for SMB File Shares
Application consistent shadow copies for server application data stored on Windows Server 2012 file shares
Backup and restore scenarios
Full integration with VSS infrastructure
Volume Shadow Copy Service
\\fs\fooData volume
\\fs\foo@t1Shadow
Copy
Backup Server
Application Server File Server
File Share Shadow Copy Agent
Coordinate Shadow Copy
Create Shadow Copy
Create Shadow Copy
Request Shadow Copy
VSS Providers
Backup A
B
C
D
E
Read fromShadow CopyShare
G
Relay Shadow Copy
request
Backup Agent
Volume Shadow Copy Service
File Share Shadow Copy Provider
F
Basic Configurations
File Server ConfigurationsSingle-node File Server
• Lowest cost for shared storage• Shares not continuously available
Dual-node File Server• Low cost for continuously available
shared storage• Limited scalability
(up to a few hundred disks)
Multi-node File Server• Highest scalability
(up to thousands of disks)• Higher cost, but still lower than
connecting all Hyper-V hosts with FC
Hyper-V Parent 1
Child 1Config
VHD Disk
Hyper-V Parent N
Child NConfig
VHD Disk
File Server
Share1 Share2
Disk Disk
Hyper-V Parent 1
Child 1Config
VHD Disk
Hyper-V Parent N
Child NConfig
VHD Disk
File Server 1
Share1 Share2
File Server 2
Share1 Share2
Shared SAS Storage
Disk DiskDisk Disk
Hyper-V Parent 1
Child 1Config
VHD Disk
Hyper-V Parent N
Child NConfig
VHD Disk
FS 1
Share1
Fibre Channel Storage Array
Disk Disk Disk DiskDisk Disk Disk
FS 2
Share2
FS 3
Share3
FS 4
Share4
A B C
Network Configurations1GbE
NetworksMixed 1GbE/10GbE 10GbE or InfiniBand Networks
Hyper-V 1
File Server 1
Hyper-V 2
File Server 2
1GbE 1GbE
1GbE 1GbE
Hyper-V 1
File Server 1
Hyper-V 2
File Server 2
10GbE / IB 10GbE / IB
1GbE 1GbE
Hyper-V 1
File Server 1
Hyper-V 2
File Server 2
10GbE / IB 10GbE / IB
10GbE / IB 10GbE / IB
Clients Clients Clients
File Server 1
File Server 2
10GbE / IB 10GbE / IB
Clients
B CA D
Hyper-V 1 Hyper-V 2
Permissions for Hyper-V over SMB
• Full permissions on NTFS folder and SMB share for• Hyper-V Administrator• Computer Account of Hyper-V hosts• If Hyper-V is clustered, the Hyper-V Cluster Account (CNO)
1. Create FolderMD F:\VMFolder
2. Create ShareNew-SmbShare -Name VMShare -Path F:\VMFolder -FullAccess Dom\HAdmin, Dom\HV1$, Dom\HV2$, Dom\HVC$
3. Apply Share permissions to NTFS Folder permissions
(Get-SmbShare –Name VMShare).PresetPathAcl | Set-Acl
How to use it: simply type a UNC pathNew-VHD -Path \\FS1\VMShare\VM1.VHDX -Dynamic -SizeBytes 100GBNew-VM -Name VM1 -Path \\FS1\VMShare -VHDPath \\FS1\VMShare\VM1.VHDX -Memory 4GB
Important notes on Hyper-V over SMB
• Hyper-V supports SMB version 3.0 onlyo The Hyper-V Best Practices Analyzer (BPA)
will check the version of SMBo Third-party SMB 3.0 implementations from
storage partners like EMC and NetApp
• Continuously Available shares are recommended
• Active Directory is requiredo Computer accounts, which are required for
configuring proper permissions, only exist in a domain
• Loopback configurations are not supportedo File Server and Hyper-V must be separate
serverso If using Failover Clusters, File Server and
Hyper-V must be on separate clusters
• Virtual Machine Manager 2012 SP1 supports Hyper-V over SMBo Available since January 2013
• Remote Managemento Use PowerShello Use Server Manager (for file
shares)o Use Remote Desktop (RDP) o Use VMM 2012 SP1 o If using Hyper-V Manager
remotely, Constrained Delegation is required
Performance
Hyper-V
ClientClient
Hyper-VHyper-VHyper-V
Host
FileServer
2
Typical Configuration for Hyper-V over SMB
FileServer
1
SAS HBASAS HBA
R-NIC
R-NIC
R-NIC
R-NIC
Client
Storage
Spaces
SMB 3.0Server
SMB 3.0Client
Switch5
Switch6
NIC
NIC
NIC Teaming
vSwitch
Switch4
Switch1
NIC
RouterSwitch
2
ClientClient
Client
NIC
VMVMVMVirtual
Machine
vNIC vDiskFile
ShareSpaceFile
Share Space
……
…NIC
NIC
Switch3
FileServerDHCP
DC/DNSManagement
…
NIC NIC
File ServerCluster
JBODsClientsHyper-VCluster
SAS JBOD
SASModule
SASModule
Disk
Disk
Disk
Disk
SAS JBOD
SASModule
SASModule
Disk
Disk
Disk
Disk
SAS JBOD
SASModule
SASModule
Disk
Disk
Disk
Disk
R-NIC
R-NIC
NIC
NICSAS HBASAS HBA
SAS JBODSAS JBOD
Performance considerations
Hyper-V
ClientClient
Hyper-VHyper-VHyper-V
Host
FileServer
FileServer
SAS HBASAS HBA
R-NIC
R-NIC
R-NIC
R-NIC
Client Storage
Spaces
SMB 3.0ServerSMB 3.0
ClientSwitch
5
Switch6
NIC
NIC
NIC Teaming
vSwitch
Switch4
Switch1
NIC
RouterSwitch
2
ClientClient
Client
NIC
VMVMVMVirtual
Machine
vNIC vDisk
FileShare
SpaceFileShare Space
……
…
NIC
NIC
Switch3
FileServerDHCP
DC/DNSManagement
…
NIC NIC
VMs per hostVirtual processes per
VMRAM per VM
R-NICs per Hyper-V host
Speed of R-NICs
SAS ports per module
SAS Speed
SAS HBAs per File Server
SAS Speed
R-NICs per file server,Speed of R-NICs
NICs per Hyper-V host
Speed of NICs
Disks per JBODDisk SpeedSAS Speed
Number of SpacesColumns per spaceCSV cache config
Hyper-V hostsCores per Hyper-V
hostRAM per Hyper-V host
Number of clientsSpeed of client NICs
SAS JBOD
SASModule
SASModule
Disk
Disk
Disk
Disk
Designing a solution
• The workload• 500 VDI VMs, 2GB RAM, 1 virtual proc• ~50GB per VM, ~30 IOPS per VM
• Some building blocks• Disks: 900 GB HDD @ 10Krpm, ~140
IOPS• JBOD: 24 or 60 disks per JBOD• Hyper-V host, 16 cores, 128GB RAM
• Working out the storage• IOPS: 30 * 500 /140 = ~107 disks• Capacity: 50GB * 2 * 500 / 900 = ~56
disks
• Rounding up• 107 disks for IOPS, twice the required
capacity• 2 JBODs x 60 disks 120 disks (some
spares)
• Working out the Hyper-V hosts• 2GB VM/128GB ~ 50 VM/host (some RAM for
host)• 50 VMs * 1 virtual procs / 16 cores ~ 3:1 ratio• 500 VMs/50 ~ 10 hosts (+1 as a spare)
• Networking• 500 VMs*30 IOPS*64KB = 937 MBps required• Single 10GbE = 1100 MBps . 2 for fault
tolerance.• Single 4-lane SAS 6Gbps = 2200 MBps. 2 for FT.
• File Server • 500 * 25 = 12,500 IOPS. Single file server. 2 for
FT.• RAM = 64GB, good size CSV cache (up to 20%
of RAM)
• Now let’s draw this out…
VDI workload (sample only, your requirements may vary)
~4
.4 G
B/s
ec
2 x
10
Gb
E x
2Hyper-V
ClientClient
Hyper-VHyper-VHyper-V
Host
FileServer
FileServer
R-NIC
R-NIC
R-NIC
R-NIC
Client Storage
Spaces
SMB 3.0ServerSMB 3.0
Client
Switch5
Switch6
NIC
NIC
NIC Teaming
vSwitch
Switch4
Switch1
NIC
RouterSwitch
2
ClientClient
Client
NIC
VMVMVMVirtual
Machine
vNIC vDisk
FileShare
SpaceFileShare Space
……
…
NIC
NIC
Switch3
FileServerDHCP
DC/DNSManagement
…NIC NIC
2GB per VM50 VMs per host
500 VMs total50GB VHD per VM
2 R-NIC @ 10Gbps
4 SAS ports @ 6 Gbps
2 SAS HBAs @ 6Gbps2 SAS ports/HBA
2 R-NIC @ 10Gbps
2 NICs @ 10Gbps60 disks/JBOD120 disks total
900GB @ 10Krpm
8 mirrored spaces16 columns/space12 GB CSV cache
11 Hyper-V hosts16 cores/host
128GB RAM/host
500 clients1 Gbps NICs
SAS JBOD
SAS HBASAS HBA SAS JBOD
SASModule
SASModule
Disk
Disk
Disk
Disk
8.8
GB
/sec
2 x
6G
b S
AS
x4
x
2
Speeds and Feeds – Maximum Theoretical Throughput
NIC Throughput
1Gb Ethernet ~0.1 GB/sec
10Gb Ethernet ~1.1 GB/sec
40Gb Ethernet ~4.5 GB/sec
32Gb InfiniBand (QDR)
~3.8 GB/sec
56Gb InfiniBand (FDR) ~6.5 GB/sec
HBA Throughput
3Gb SAS x4 ~1.1 GB/sec
6Gb SAS x4 ~2.2 GB/sec
4Gb FC ~0.4 GB/sec
8Gb FC ~0.8 GB/sec
16Gb FC ~1.5 GB/sec
Bus Slot Throughput
PCIe Gen2 x4 ~1.7 GB/sec
PCIe Gen2 x8 ~3.4 GB/sec
PCIe Gen2 x16 ~6.8 GB/sec
PCIe Gen3 x4 ~3.3 GB/sec
PCIe Gen3 x8 ~6.7 GB/sec
PCIe Gen3 x16 ~13.5 GB/sec
Memory Throughput
DDR2-400 (PC2-3200) ~3.4 GB/sec
DDR2-667 (PC2-5300) ~5.7 GB/sec
DDR2-1066 (PC2-8500)
~9.1 GB/sec
DDR3-800 (PC3-6400) ~6.8 GB/sec
DDR3-1333 (PC3-10600)
~11.4 GB/sec
DDR3-1600 (PC3-12800)
~13.7 GB/sec
DDR3-2133 (PC3-17000)
~18.3 GB/sec
Intel QPI Throughput
4.8 GT/s ~9.8 GB/sec
5.86 GT/s ~12.0 GB/sec
6.4 GT/s ~13.0 GB/sec
7.2 GT/s ~14.7 GB/sec
8.0 GT/s ~16.4 GB/sec
Only a few common configurations listed. Numbers are rough approximations.Actual throughput in real life will be lower than these theoretical maximums.
Numbers provided are for one way traffic only (double for full duplex). One interface/port only.Numbers use base 10 (1 GB/sec = 1,000,000,000 bytes per second)
Potential Variations
Hyper-V
ClientClient
Hyper-VHyper-VHyper-V
Host
FileServer
FileServer
R-NIC
R-NIC
R-NIC
R-NIC
Client Storage
Spaces
SMB 3.0ServerSMB 3.0
ClientSwitch
5
Switch6
NIC
NIC
NIC Teaming
vSwitch
Switch4
Switch1
NIC
RouterSwitch
2
ClientClient
Client
NIC
VMVMVMVirtual
Machine
vNIC vDisk
FileShare
SpaceFileShare Space
……
…
NIC
NIC
Switch3
FileServerDHCP
DC/DNSManagement
…
NIC NIC
Regular NICs insteadof RDMA NICs
Fibre Channel oriSCSI instead of SAS
Third-party SMB 3.0 NAS
Instead of Windows File Server Cluster
SAS JBODSAS JBOD
SAS HBASAS HBA SAS JBOD
SASModule
SASModule
Disk
Disk
Disk
Disk
Traditional SAN instead of JBODS
Demo
Hyper-V over SMB
Demo: Summary
Workload BWGB/sec
IOPSIOs/sec
%CPUPrivileged
Physical host, 512KB IOs, 100% read, 2t, 12o ~16.8 ~32K ~16%
Physical host, 32KB IOs, 100% read, 8t, 4o ~10.9 ~334K ~52%
12 VMs, 4VP, 512KB IOs, 100% read, 2t, 16o ~16.8 ~32K ~12%
12 VMs, 4VP, 32KB IOs, 100% read, 4t, 32o ~10.7 ~328K ~62%
File Server(SMB 3.0)
Hyper-VHost VM1
RDMA NIC
RDMA NIC
RDMA NIC
RDMA NIC
RDMA NIC
RDMA NIC
SA
S
SASHBA
JBODSSDSSDSSDSSDSSDSSDSSDSSD
SA
S
SASHBA
JBODSSDSSDSSDSSDSSDSSDSSDSSD
SA
S
SASHBA
JBODSSDSSDSSDSSDSSDSSDSSDSSD
SA
S
SASHBA
JBODSSDSSDSSDSSDSSDSSDSSDSSD
SA
S
SASHBA
JBODSSDSSDSSDSSDSSDSSDSSDSSD
SA
S
SASHBA
JBODSSDSSDSSDSSDSSDSSDSSDSSD
Storage Spaces
VM2 VM3 VM12…
Results from EchoStreams FlacheSAN2 server, with 2 Intel CPUs at 2.40 Ghz. Hyper-V Server with 2 Intel CPUs at 2.70 GHz. Both using three Mellanox ConnectX-3 network interfaces on PCIe Gen3 x8 slots.
Data goes to 6 LSI SAS adapters and 48 Intel SSDs, attached directly to the EchoStreams server. This demo could use further optimization of the VM configuration – Work in progress still…
Sample Configurations
Sample Configurations
Sharing a collection of sample configurations coming from internal testing, demo systems, conferences:
• All are using SMB, not all of them use Hyper-V• Includes standalone, classic cluster and scale-out
clusters• Includes both SAS and Fibre Channel• Includes both Storage Spaces and SAN • Includes performance details when available
Standalone, 1GbE, FC Array
• Random IO bandwidth actually below 1Gbps• SMB only 3% slower than DAS (in the last 19min)• 1GbE nearly meets 2 x 4GbFC, bottleneck is disk• Details in Dan Lovinger’s SDC 2012 presentation• White paper coming soon
Client Server
CPU 2 sockets, 8 cores total, 2.26 GHz
Memory 24 GB RAM
Network 1 x 1GbE NIC (onboard)
Storage adapter
N/A 1 FC adapter2 x 4Gbps links
Disks N/A 24 x 10Krpm HDD20 used for data
2 used for log
Dell Servers, Standalone, 10GbE, SAS, Storage Spaces
• 2 Dell R910 servers running Windows Server 2012, 10GbE Chelsio NICs, 6Gbps LSI SAS HBA.• 1 RAID Inc. JBOD, 60 600GB 15K SAS HDD, configured using Storage Spaces as mirrored Spaces.• 8 Hyper-V VMs running Windows Server 2012, 4 vCPUs, 12GB of RAM, SQL Server (OLTP
workload)• Remote (over SMB 3.0) compares well to local (~5% difference). Bottleneck are the HDDs.• Details in white paper by ESG
VMsLocalIOPS
RemoteIOPS
Remote/Local
1 900 850 94.4%
2 1,750 1,700 97.1%
4 3,500 3,350 95.7%
6 5,850 5,600 95.7%
8 7,000 6,850 97.9%
Intel, Standalone, 56GbIB, FusionIO
Results from two Intel Romley machines with 2 sockets each, 8 cores/socketBoth client and server using a single port of a Mellanox network interface PCIe Gen3 x8 slot
Data goes all the way to persistent storage, using 4 FusionIO ioDrive 2 cards
Configuration BWMB/sec
IOPS512KB IOs/sec
%CPUPrivilege
d
Non-RDMA (Ethernet, 10Gbps)
1,129 2,259 ~9.8
RDMA (InfiniBand QDR, 32Gbps)
3,754 7,508 ~3.5
RDMA (InfiniBand FDR, 54Gbps) 5,792 11,565 ~4.8
Local 5,808 11,616 ~6.6
Workload: 512KB IOs, 8 threads, 8 outstanding
SMB 3.0 + RDMA (InfiniBand FDR)
SMBClient
SMB Server
Fusion IOFusion IOFusion IOFusion IO
IO Micro Benchmark
IB FDR
IB FDR
SuperMicro, Standalone, 2 x 56GbIB, SAS, LSI RAID
Results from SuperMicro servers, each with 2 Intel E5-2680 CPUs at 2.70 GhzBoth client and server using two Mellanox ConnectX-3 network interfaces on PCIe Gen3 x8 slotsData goes to 4 LSI 9285-8e RAID controllers and 4 LSI JBODs, each with 8 OCZ Talos 2 R SSDs
Data on the table is based on a 60-second average. Performance Monitor data is an instant snapshot.
Configuration BWMB/sec
IOPS512KB IOs/sec
%CPUPrivileged
Latencymilliseconds
1 – Local 10,090 38,492 ~2.5% ~3ms
2 – Remote 9,852 37,584 ~5.1% ~3ms
3 - Remote VM 10,367 39,548 ~4.6% ~3 ms
Workload: 256 KB IOs, 2 threads, 16 outstanding
Hyper-V (SMB 3.0)
File Server(SMB 3.0)
VM
RDMA NIC
RDMA NIC
RDMA NIC
RDMA NIC
SQLIO
SA
S
RAIDControlle
r
JBODSSDSSDSSDSSDSSDSSDSSDSSD
SA
S
RAIDControlle
r
JBODSSDSSDSSDSSDSSDSSDSSDSSD
SA
S
RAIDControlle
r
JBODSSDSSDSSDSSDSSDSSDSSDSSD
SA
S
RAIDControlle
r
JBODSSDSSDSSDSSDSSDSSDSSDSSD
Shown at TechEd 2012
File Server(SMB 3.0)
File Client(SMB 3.0)
EchoStreams, Standalone, 3 x 56GbIB, Storage Spaces
SQLIO
RDMA NIC
RDMA NIC
RDMA NIC
RDMA NIC
RDMA NIC
RDMA NIC
SA
S
RAIDControlle
r
JBODSSDSSDSSDSSDSSDSSDSSDSSD
SA
S
SASHBA
JBODSSDSSDSSDSSDSSDSSDSSDSSD
SA
S
SASHBA
JBODSSDSSDSSDSSDSSDSSDSSDSSD
SA
S
SASHBA
JBODSSDSSDSSDSSDSSDSSDSSDSSD
SA
S
SASHBA
JBODSSDSSDSSDSSDSSDSSDSSDSSD
SA
S
SASHBA
JBODSSDSSDSSDSSDSSDSSDSSDSSD
Storage Spaces
WorkloadBW
MB/secIOPSIOs/sec
%CPUPrivileged
Latency
milliseconds
512KB IOs, 100% read, 2t, 8o 16,778 32,002 ~11% ~ 2 ms
8KB IOs, 100% read, 16t, 2o 4,027 491,66
5 ~65% < 1 ms
Results from EchoStreams FlacheSAN2 server, with 2 Intel E5-2620 CPUs at 2.00 GhzBoth client and server using three Mellanox ConnectX-3 network interfaces on PCIe Gen3 x8 slotsData goes to 6 LSI SAS adapters and 48 Intel SSDs, attached directly to the EchoStreams server.
Data on the table is based on a 60-second average. Performance Monitor data is an instant snapshot.
Shown via TechNet RadioWhite paper coming…
X-IO, Standalone, FC SAN, 3 x 56GbIB
Full-sized server rack with• 10 X-IO ISE-2 storage units (FC attached)• 200 10Krpm 2.5” SAS HDDs (20 per unit)• 2 HP DL380 G8 + 2 DL360 G7 (3 servers, 1
client)• 2 QLogic 5800 8GB FC switches • 6 Qlogic QLA2564 (quad-port) HBAs (2 per
server)• 6 Mellanox ConnectX-3 adapters (56Gbps)
Performance highlights:• Over 15 Gbytes/sec throughput• Over 120,000 I/Os per second Details in this partner team blog post
Wistron, Cluster-in-a-box, 10GbE, SAS, Storage Spaces
• 2 File Server cluster nodes, single enclosure
• LSI 6Gbps SAS controllers, one per node
• Internal shared SAS JBOD, 24 x SAS HDDs
• Storage Spaces • Dual 10GbE ports, plus dual 1GbE ports• Shown in TechEd 2012 demo
HP StoreEasy 5000, Cluster-in-a-box, 10GbE, SAS
• 2 cluster nodes in single enclosure (right side)• Intel Xeon at 2.4 GHz, 4 cores (per
node)• 24GB RAM, upgradable to 96GB (per
node)• Internal shared SAS JBOD tray (left side)
• 16 x 3.5” SAS drives or 36 x 2.5” SAS drives
• External SAS JBOD option for additional drives
• HP controller with Failover Cluster support
• Two 10GbE ports, plus 1GbE ports (per blade)
• HP iLO (Integrated Lights-Out) for management
Quanta, Cluster-in-a-box, 56GbIB, SAS, LSI HA-DAS
Quanta Cluster-in-a-box, 2 Intel Romley CPUs, 16GB RAMLSI HA-DAS MegaRAID ® and SAS Controllers on PCIe Gen2 x8 slots
Mellanox ConnectX-3 network interfaces on PCIe Gen3 x8 slots24 OCZ Talos 2 SSDs in a Quanta JBOD
• Around 2.5 GBytes/sec throughput• Shown at TechEd 2012 demo
Violin Memory Prototype, Cluster-in-a-box, 56GbIB
Configuration• Cluster-in-a-box configuration• Fits in only 3 Rack Units (3U unit)• Up to 32TB of raw flash capacity• 2 x 56Gbps InfiniBand network
ports
Results using real SQL Server workloads• Over 4.5+ Gbytes/sec with 256KB
IOs(loading database to memory)
• Over 200,000 IOPS with 8KB IOs(transaction processing)
• Very low latency• Shown at TechEd 2012
VDI boot storm, Scale-Out (File Server test team)Comparison 1:• Local: 8 hosts running Hyper-V directly• Remote: 8 hosts using Hyper-V over SMB• 2,560 VMs (320 VMs per host). 8GB CSV
cache.
Comparison 2:• 16 hosts using Hyper-V over SMB• Comparing 8GB CSV cache enabled/disabled• 5,120 VMs (320 VMs per host)
On both setups, using Hitachi SAN back-end. 2 x 8GbFC Server to Fabric, 4 x 4GbFC Fabric to SAN.For Hyper-V over SMB configurations, using 2 x 10GbE between Hyper-V cluster nodes and File Server cluster nodes.For all setups, using parent VHDX with differential VHDX files. Time measured from VM state change to user logon complete.
holSystems: Dell Servers, Cluster, 10GbE, FC Arrays
HV 1
FS 1
FC Array1
FS 2 FS 3 FS 4
FC Array2
FC Array 7
…
FS 5
FC Array2
HV 2 HV 3 HV 4 HV 5 HV 50…
FC Switch 1
10GbE Switch 1
10GbE Switch 2
10GbE Switch 3
10GbE Switch 4
FC Switch 2
Failover Cluster 2Failover Cluster 1
Configuration• 50 Hyper-V Hosts (Dell, 72GB-400GB RAM each, 10TB
total)• 5 File Servers (Dell, 1 standalone, 2 x two-node
clusters)• 7 Fibre Channel arrays of varying sizes. 100TB total.• 10GbE between Hyper-V and File Servers (120 ports)• 8GbFC between File servers and FC arrays (2 switches)
Workload • Running Windows Server 2012 Virtual Labs and HOL/ILL
labs for major Microsoft events including TechEd, TR, Lync Ignite, SharePoint Ignite, SQL Server Labs, Convergence
• Each user spins up a new set of VMs in just seconds!• Commonly used by 500 users at 16GB each.• Capacity tested for up to 7,000 VMs.• Over 800,000 VMs just in the last 3 years.• New set of VMs deployed every 5-7 minutes (in
average).• Try it yourself using the link above
Thanks to Corey Hynes for the details!
Microsoft IT: HP Servers, Cluster, 6 x 10GbE, FC Array
HV 1
EMC VMAX
FS 1 FS 2 FS 3 FS 4
HV 2 HV 3 HV 28…
FC Switch 1
10GbE Switch 1
10GbE Switch 2
10GbE Switch 3a
10GbE Switch 3b
FC Switch 2
Failo
ver
Clu
ster
Single Stamp Configuration
• Microsoft IT Scale Unit v3 (Network and Infrastructure Services team)
• 28 Hyper-V Hosts (HP BL660, 1TB RAM each, 28TB total)
• 4 File Servers (HP BL660, 1TB RAM, 4-node scale-out file server)
• EMC VMAX 40K in a 2+1 configuration (400 disks, 3 tiers)
Networking
• 2 x 10GbE Broadcom between Hyper-V and File Servers
• 4 x 10GbE Broadcom between every server and the outside world
• 2 x Cisco 3064 for East-West traffic, 2 x Cisco 5548 for North-South traffic
• 2 x 8GbFC Emulex HBAs per server, 24 FC ports + 8 iSCSI to VMAX
• 2 x Brocade 6510 FC Switches with 48 ports (up to 16GbFC)
Workload
• Private cloud environment with LOB apps (SQL, IIS, others)
• Average VM size is 16 virtual processors with 32GB of RAM
• Largest VM supported is 32 virtual processors and 128GB of RAM
• Built for high performance networking, including SR-IOV support
• 1,000 VMs per stamp projected
Thanks to Jeromy Statia for the details!
NTTX: Dell servers, Clusters, 10GbE, SAS JBODs Hardware
• 48 Hyper-V hosts (10 R910, 18 R720, 24 M1000e)• Hyper-V hosts divided into 7 distinct clusters in 2
sites• 10 File Servers, 5 clusters in 2 sites (4 R710, 6 Dell
R320)• 38 JBODs (Dell MD1200, 12 drives each)• Mostly 15K rpm 600GB SAS HDDs. Some 7.2K 2TB
SAS for DPM. ~300TB total raw (~150TB usable in mirrored spaces).
Networking• 1GbE: 4 PowerConnect 5548. 10GbE: PowerConnect
8164• Site 1 Scale-Out File Servers use 8x1GbE NICs, in a
team• Site 2 Scale-Out File Servers use 4x10GbE NICs, in a
team• Between sites, 4 NetASQ U450 (2 per site), 400
MbpsWorkload • Mixed workload (hosting) includes SQL, VDI,
Exchange, Lync, SharePoint, CRM, other app servers, Backup.
• ~3,000 VMs total with varying storage, RAM, processors
• Hyper-V Replica between the 2 sites, heavily customized
Highlights• Cost reduction. Efficiency. High scalability. Full HA
(CA).• Delivers on varying and evolving workloads.• Built on commodity hardware. 100% Microsoft stack.
Thanks to Philip Moss for the details!
Cluster 14 x Dell R910
8 JBODs96 HDDs
FS10FS9
8 JBODs96 HDDs
FS8FS7
6 JBODs72 HDDs
FS6FS5
8 JBODs96 HDDs
FS4FS3
8 JBODs96 HDDs
FS2FS1
Cluster 24 x Dell R910
1GbE Switch
1GbE Switch
1GbE Switch
1GbE Switch
Cluster 46 x Dell R720
Cluster 312 x Dell M1000e
10GbE Switch
Cluster 54 x Dell R910
Cluster 712 x Dell
R720
Cluster 612 x Dell M1000e
NetASQ
U450
NetASQ
U450NetASQ
U450
NetASQ
U450
Site 1 Site 2
In Review: Session Objectives
• Describe the basics of the Hyper-V over SMB scenario, including the main reasons to implement it.
• Enumerate the most common performance bottlenecks in Hyper over SMB configurations.
• Outline a few Hyper-V over SMB configurations that can provide continuous availability, including details on networking and storage.
Evaluation
Complete your session evaluations today and enter to win prizes daily. Provide your feedback at a CommNet kiosk or log on at www.2013mms.com.Upon submission you will receive instant notification if you have won a prize. Prize pickup is at the Information Desk located in Attendee Services in the Mandalay Bay Foyer. Entry details can be found on the MMS website.
We want to hear from you!
Resources
http://channel9.msdn.com/Events
Access MMS Online to view session recordings after the event.
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.