Gluster and Kubernetes

GlusterFS & Kubernetes

The State of GlusterFS Support in Kubernetes

Joe Julian

● Long time community member.● Senior Staff Engineer with Samsung SDS/CNCT.● Certified Kubernetes Administrator (my first cert ever, hope it’s worth the

paper it’s printed on)(I haven’t printed it).● Insufferable know-it-all with an opinion about everything.

What is Kubernetes

Kubernetes is ancient Greek for "Helmsman". Root of the word "Governor", "Cybernetics".

Kubernetes is a "Container Orchestrator" or "Cluster Manager".

● Places containers on nodes● Recovers automatically from failure● Basic monitoring, logging, health checking● Enables containers to find each other.

How can gluster be used with Kubernetes

● Persistent state Containers can be restarted at any time due to failed liveness check or deployment.

How can gluster be used with Kubernetes

● Persistent state● Shared Storage

Multiple containers may need to operate on the same data set.

How can gluster be used with Kubernetes

● Persistent state● Shared Storage● Cats

How can gluster be used with Kubernetes

● Persistent state● Shared Storage● Cats

GlusterFS volume driver

GlusterFS Volume Driver

volumes: - name: glusterfsvol glusterfs: endpoints: glusterfs-cluster path: kube_vol readOnly: true

kind: Endpointsmetadata: name: glusterfs-clustersubsets:- addresses: - ip: 10.0.0.1- addresses: - ip: 10.0.0.1

GlusterFS Volume Driver

containers: - name: glusterfs image: nginx volumeMounts: - mountPath: "/mnt/glusterfs" name: glusterfsvol

Mounts the volume to /mnt/glusterfs in the container’s chroot.

GlusterFS Volume Driver

Pros:

● Works with existing volumes● Shared data

Cons:

● Not dynamic● Must be managed externally● Only shared data

Local-volume storage class

Local-volume storage class

kind: StorageClassapiVersion: storage.k8s.io/v1metadata: name: myglustervol namespace: kube-systemprovisioner: kubernetes.io/host-path

The code says:// This Provisioner is meant for// development and testing only and // WILL NOT WORK in a multi-node// cluster.

Local-volume storage class

kind: StorageClassapiVersion: storage.k8s.io/v1metadata: name: myglustervol namespace: kube-systemprovisioner: kubernetes.io/host-path

The code says:// This Provisioner is meant for// development and testing only and // WILL NOT WORK in a multi-node// cluster.

Local-volume storage class

● Can only use /tmp/hostpath_pv● Awkward mount● No actual allocation restriction● Works great!

/tmp/hostpath_pv glusterfs gluster:/myvol ...

/tmp/hostpath_pv/{{ uuid }}

kind: PersistentVolumeClaimapiVersion: v1metadata: name: deluge-configspec: accessModes: - ReadWriteMany resources: requests: storage: 50Mi

HeketiGlusterFS storage class

GlusterFS+Heketi

Heketi provides a REST api for the configuration of GlusterFS which can be (and is) used by several infrastructure tools including Kubernetes. Support for GlusterFS using Heketi is part of the main kubernetes tree and has strong downstream acceptance.

Heketi is the Taíno (a Caribbian aboriginal) word for “One”.

Heketi

● GlusterFS in pods● Strongly opinionated cluster design● Allows metered private PVCs● Provides an excellent api interface to GlusterFS

Heketi

● GlusterFS in pods (privileged)● Strongly opinionated cluster design

○ Whole disk○ volume-per-PVC

● Requires prior knowledge● Doesn’t follow best practices wrt hostnames vs IP addresses

Rook

Rook

Rook is a completely cloud-native toolset to provide turnkey storage to kubernetes. It currently only does so with Ceph, but they want to add GlusterFS support and are seeking help.

Rook

What I like about rook:

$ helm repo add rook-alpha https://charts.rook.io/alpha

$ helm install rook-alpha/rook

Rook

● Established provider of ceph storage to kubernetes● Downstream user = community contributions!● Enhance the user experience

Rook

● Automated decision making● Too easy?● Small team, needs contributions

Glusterd V2?

Questions?

Joe Julian, Samsung SDS

me@joejulian.name

Think about where we’re going with respect to kubernetes.Contribute to Kubernetes, Rook, Kraken.

Be part of the community!

● Join the sig-storage meetings● Hang out in #sig-storage on the

kubernetes slack● Join the rook slack● Hang out with me in #gluster on IRC

github.com/samsung-cnct/krakengithub.com/kubernetes/kubernetesgithub.com/rook/rook

Comment on the Rook + GlusterFS design document! https://goo.gl/L62Bi2