The next generation certainty in shared storage environments

The Next Generation - FOSDEM 04.02.2017

THE NEXT GENERATIONCERTAINTY IN SHARED STORAGE ENVIRONMENTS

Adam Litke - Senior So�ware Engineer - Red Hat

FOSDEM 2017 - 04 February 2017

[email protected]

mailto:[email protected]


AGENDAoVirt shared storage architecture😱 MAYHEM!😌 Order.Examples


OVIRT SHARED STORAGE

NFS

GlusterGluster

iSCSI

Ceph

VM x 1000s

x 100s

x 1Engine

FibreChannel


OVIRT IMAGE

Volumes

A

B

C

D


OVIRT VOLUME

Metadata Lease

Data


STORAGE OPERATIONS

VM Datapath

Metadata


STORAGE JOBSEncapsulate a singular storage operation on a hostEngine selects a host and submits the jobHost performs work asynchronouslyEngine polls host for job statusJob status only available on host while job is active


THE WORLD IS A CHAOTIC PLACEPower outagesNetwork outagesHardware failureSo�ware bugs


FAILED HOST

Engine

Host A

Host B

Vol 1

Vol 2

Vol 3

Storage


RESTORING ORDERDetermine the status of outstanding storage jobsThe answers to all questions must come from storageWait or abort active jobs on unresponsive hostsCheck final status of jobs that have ended


VOLUME LEASESImplemented using Lockspace is on shared storagealongside the volumesA lease grants a host exclusiveaccess to a volumeFailing hosts will be fenced bySanlock if they hold leases

SanlockMetadata Lease

Data

https://fedorahosted.org/sanlock


VOLUME GENERATIONSMonotonically increasing valueStored in volume metadata areaChangeable only while holding thevolume leaseAllows sequencing of storage jobs

Metadata Lease

Data


STORAGE JOB STRUCTURE1. Acquire volume lease2. Validate volume generation3. Do work4. Increment volume generation5. Release volume lease


EXAMPLE: NORMAL FLOW


SCHEDULE JOB

Lease

Engine

Host A

Host B

JobJob

Vol 1

Vol 2

Vol 3

Storage


ACQUIRE VOLUME LEASE

Engine

Host A

Host B

JobJob

Vol 1

Vol 2

Vol 3

Storage

Lease


VALIDATE VOLUME GENERATION

Engine

Host A

Host B

JobJob

Vol 1

Vol 2

Vol 3

Storage

Lease

11 11


WRITE VOLUME

Engine

Host A

Host B

JobJob

Vol 1

Vol 2

Vol 3

Storage

Lease! 50%


INCREMENT VOLUME GENERATION

Engine

Host A

Host B

JobJob

Vol 1

Vol 2

Vol 3

Storage

Lease

22


RELEASE VOLUME LEASE

Lease

Engine

Host A

Host B

JobJob

Vol 2

Vol 3

Storage

Vol 1



DONE EVENT

Lease

Engine

Host A

Host B

JobJob

Vol 2

Vol 3

Storage

Vol 1!


SCENARIO: UNRESPONSIVE HOST


JOB SCHEDULED ON HOST A

Engine

Host A

Host B

JobJob

Vol 1

Vol 2

Vol 3

Storage

1


HOST A BECOMES UNRESPONSIVE

EngineHost B

Vol 2

Vol 3

Storage

???

Host A

Lease

? Vol 1?


VOLUME RECONNAISSANCESpecial storage job that resolves volume statusChecks if a job is runningOption to use sanlock fencing to free the leaseIf volume is free, uses generation to check statusIncrements generation to preempt pending jobs


SELECT ANY AVAILABLE HOST

EngineHost B

Vol 2

Vol 3

Storage

???

Host A

Lease

? Vol 1?

JobJob

1


ACQUIRE OR REQUEST LEASE

Host B

JobJob

Vol 1Lease

! JobRunning

Host B

JobJob

Vol 1Lease ...

or


COMPARE AND BUMP GENERATION

Host B

JobJob

Vol 1

Host B

JobJob

Vol 1 ! JobSucceeded

or

Vol 11122

! JobFailed11

2211

Lease

Lease



PREEMPTED JOB

Engine

Host A

Host B

JobJob

Vol 1

Vol 2

Vol 3

Storage

11 22Lease


FUTURE WORKShared lease supportParallel job schedulingIntegrate with VM leases


JOIN US!http://www.ovirt.orgirc://irc.o�c.net/ovirthttp://lists.ovirt.org/mailman/listinfo/develhttp://lists.ovirt.org/mailman/listinfo/users

http://www.ovirt.org/

irc://irc.oftc.net/ovirt

http://lists.ovirt.org/mailman/listinfo/devel

http://lists.ovirt.org/mailman/listinfo/users


QUESTIONS?

Technology

The next generation certainty in shared storage environments