Window Server 2019 Hyper-V Persistent...

Window Server 2019 Hyper-V&

Persistent Memory

Didier Van HoyeTechnical Architect & StrategistMicrosoft Cloud & Datacenter Management MVPVeeam VanguardPublic & private speaker

Twitter @workinghardinitOne of the Hyper-V Amigos http://hyper-v-amigos.nethttps://blog.workinghardinit.work

Storage at the speed of memory!!!!!

What Is Persistent Memory?

Didier Van Hoye - WorkingHardInIT

➢ Persistent memory (PMEM) = a solid-state high-performance byte-addressable memory device that resides on the memory bus

➢ PMEM has DRAM-like access to data➔ almost the same speed & latency of DRAM but persistent like NAND flash

Persistent Memory … different types

➢Non-Volatile DIMM (NVDIMM)

➢Storage Class Memory (SCM)

… but don’t get caught up in the semantics too much

… lots to happen in this space

Didier Van Hoye - WorkingHardInIT

Persistent Memory … differences!

Non-Volatile DIMM (NVDIMM-N)➢ Combines DRAM & NAND to deliver persistent storage in a

DIMM slot. The DRAM is the storage but data is persisted to NAND at shutdown, crash or power loss. At boot the data is made available again in the DRAM which increased boot times. This requires BIOS configuration & OS support or the modules are seen as normal memory module. Provides the fastest & lowest latency, in small capacity modules (16-32-64GB).

Didier Van Hoye - WorkingHardInIT

Persistent Memory … differences!

Storage Class Memory (SCM)➢ Extends the capacity of a “regular” DIMM by using NAND

as a capacity store and DRAM as the cache for active data. This allows for high DIMM capacity servers at reduced cost, SCM feel and act like regular memory DIMMS so they don’t require special BIOS or Operating system support (*ahum). Less fast due to higher latency, but still very good but with higher capacities (512GB right now).

Didier Van Hoye - WorkingHardInIT

Examples

➢NVDIMM-N

➢3D-Xpoint (Intel Optane)

Both can act as PMEM (storage) or “Normal” Memory … but there are subtle differences …Take away: use as non-persistent memory or as persistent memory (storage: persistent, for DAX or Block address)

Didier Van Hoye - WorkingHardInIT

NVDIMM variants N, F, P

NVDIMM-PWork in Progress(wait for DDR5?)

NVDIMM-NStandard

Available today

NVDIMM-FVendors Specific

• Memory mapped DRAM, Flash is not mapped, used ony for persistence• Access: byte or block mode to DRAM• DIMM like latency: 10s of nanoseconds• Capacity like DIMM: 8-32 GB now, maybe 64-128GB later on• Battery backup for backup / persisting of data

• Memory is system mapped FLASH, DRAM is not system mapped but a cache• Access: block mode to FLASH/NAND• FLASH like latency: 10s of microseconds• Capacity like FLASH: 128GB to X TB

• Both FLASH and DRAM are memory mapped• Access: block mode to FLASH (like NVDIMM-F) & byte or block to DRAM (like NVDIMM-N)• FLASH NVME like latency: 100’s of nanoseconds• Capacity like FLASH NVMe: 128GB to X TB

Didier Van Hoye - WorkingHardInIT

NVDIMM-N

Didier Van Hoye - WorkingHardInIT

NVDIMM-N

Didier Van Hoye - WorkingHardInIT

NVDIMM-N

NVDIMM battery is dead or set as read-only (BIOS)?

Didier Van Hoye - WorkingHardInIT

NAND NAND NAND NANDNANDNANDController

DDR interface (JEDEC DDR4 for now)

NVDIMM-F (vendor specific)

Difference with SCM => no DRAM cache on the moduleBasically NAND on DDR4 / Slow – slowness hidden by regular RAM caching (that part is not persistent!)

Didier Van Hoye - WorkingHardInIT

NAND NAND NANDController

DDR interface (JEDEC DDR4 with extensions, and in DDR 5 time frame)

NVDIMM-P Combines N and F

DRAM DRAM DRAM DRAM DRAM DRAM

NAND NAND NAND

Didier Van Hoye - WorkingHardInIT

No battery requiredRAM is cache not storage

Intel® OPTANE™ DC PERSISTENT MEMORY

Didier Van Hoye - WorkingHardInIT

Cascade Lake

FLASH FLASH FLASH

Controller

DDR interface (JEDEC DDR4 for now)

Storage Class Memory

DRAM DRAM DRAM

FLASH = High performance NAND or 3D-Xpoint (Intel Optane DC)

No battery requiredRAM is cache not storage

FLASH FLASH FLASH

DRAM DRAM DRAM

Didier Van Hoye - WorkingHardInIT

NVDIMM / Storage Class Memory

Didier Van Hoye - WorkingHardInIT

Semantics: when is SCM really SCM and why wouldNVDIMM-P not be SCMI do not so see anything that prevents NVDIMM-P from being SCM is NAND type is fast enough

NVDIMM-P

SCM

PMEM <> Intel® OPTANE™ PX4800X

Didier Van Hoye - WorkingHardInIT

Opportunity zone for PMEM/SCMPMEM/SCM100 ns – 1000 ns

SSD100-200 us

DRAM100 ns

L1-L3 cache1-40 ns

HDD1-10 ms

NVMe10-20 us

Pri

ce/C

apac

ity

Latency (1 million delta in latency)

Break the storage latency/speed barrier …

Didier Van Hoye - WorkingHardInIT

Exponential data & analytics growth …

Didier Van Hoye - WorkingHardInIT

© 2016 Micron Technology, Inc,

It’s about value optimization!

A modern cloud OS & hardware

Didier Van Hoye - WorkingHardInIT

Didier Van Hoye - WorkingHardInIT

BIOS Persistent Memory Configuration

Didier Van Hoye - WorkingHardInIT

Get-PmemUnusedRegion | New-PmenDisk -AtomicityType None

Using Persistent Memory on Windows

Didier Van Hoye - WorkingHardInIT

Persistent Memory Interleaving

Get-PmemUnusedRegion | New-PmenDisk -AtomicityType None

Consume PMEM Storage: Host

Initialize-Disk 6 –PartitionStyle GPT

New-Partition -DiskNumber 6 -UseMaximumSize -driveletter P | Format-Volume

-NewFileSystemLabel PMEM01 -IsDAX $True

Didier Van Hoye - WorkingHardInIT

Consume PMEM Storage: VM

New-VHD P:\DEMOVM-PMDisk01.vhdpmem -Fixed -SizeBytes 30GB

Some things to take notice off:

1. Generation 2 virtual machines only.2. The .vhdpmem extension specifies the virtual

disk is a persistent memory disk3. You have to create it on an NTFS DAX volume4. For now, only fixed size persistent memory

virtual disks are supported

Use a normal virtual machine with a VHDX OS disk and potentially one or more data virtual disk (VHDX).

Didier Van Hoye - WorkingHardInIT

Consume PMEM Storage: VM (2/2) Add-VMPmemController PMEMVM

Add-VMHardDiskDrive -VMName PMEMVM PMEM -ControllerLocation 1 -Path P:\DEMOVM-

PMDisk01.vhdpmem

New-Partition -DiskNumber 1 -UseMaximumSize -driveletter P | Format-Volume -

NewFileSystemLabel VDISKVMPMEM01 -FileSystem NTFS -AllocationUnitSize 65536

Get-Disk | Where Bustype -eq SCM

Inside the virtual machine after startup:

Didier Van Hoye - WorkingHardInIT

Optimal workloads …

Didier Van Hoye - WorkingHardInIT

4K24 workers100 Read100 Sequential

Less optimal workloads …

Didier Van Hoye - WorkingHardInIT

8K24 workers70/30 Read/Write70/30 Sequential/Random

Diskspd.exe for the win ;-)

8K24 workers70/30 Read/WriteRandom IO

diskspd.exe -b8K -d20 -L -F24 -W5 -Sh -r -w30 P:\iobw.tst Q:\iobw.tst

Didier Van Hoye - WorkingHardInIT

Hyper-V PMEM Support Info (1/2)

Didier Van Hoye - WorkingHardInIT

• Only with Generation 2 VMs only (Windows & Linux)• New virtual disk type: VHDPMEM➢Convert-VHD converts between VHDPMEM & VHDX➢Select Atomicity at creation time (none or BTT)➢VHDPMEM are read/write mountable on host (SCSI)

• one vPMEM device = one VDHPMEM file

Hyper-V PMEM Support Info (2/2)

Didier Van Hoye - WorkingHardInIT

• DAX/BTT is supported in guest• Use large pages automatically (2MB)• No support for:➢ Live migration➢ Checkpoints➢ Backup➢ Save/Restore

PMEM used as storage (tiers)➢ ReFS➔ so no DAX, only block access➢ Leverage PMEM individually or aggregated via interleaving for higher

capacity and performance

PMEM usage in S2D

PMEM used as cache ➢ No need to worry about no support for ReFS for DAX or

overhead of block level access➢ Interleaving recommend (high number of cache devices

not practical/possible)

Didier Van Hoye - WorkingHardInIT

PMEM usage in S2D

Didier Van Hoye - WorkingHardInIT

In both cases (Cache/ Storage):➢ You treat the PMEM disk as a normal

fallible disk➢ It is S2D that will provide protection

against disk failures with known mechanisms

PMEM usage in S2D (2/2)

Didier Van Hoye - WorkingHardInIT

Some observations (1/2)

Didier Van Hoye - WorkingHardInIT

Some observations (2/2)

Didier Van Hoye - WorkingHardInIT