GoalsDesign a HA production quality OSE architecture that leverages native Azure Cloud infrastructure and Services

3 x Masters - With HA Load Balancer

3 x Infra Nodes - With HA Load Balancer

N x Nodes

Bastion for Safety and Security

Shared Performance Storage

Simple and Flexible

Expandable

Usable in further automations

Provisioning and Automation OverviewSeveral Choices -

Ansible - Ansible

Azure Resource Manager - ARM

Ansible - ARM - Ansible

In order to use the full function of Azure, a Azure Resource Manager Template was found to be the best way to fully leverage Azure.

Azure Resource Manager - OverviewResource Manager template - A JavaScript Object Notation (JSON) file that defines one or more resources to deploy to a resource group. It also defines the dependencies between the deployed resources.

resource group - A container that holds related resources for an application. The resource group can include all of the resources for an application, or only those resources that you group together.

ARM TemplateARM Templates are JSON files

ARM Templates are nestable

They can provision the majority of Azure Resources

Microsoft recommended methodology for Cloud Orchestration

Openshift Enterprise on Azure Template

https://github.com/glennswest/azure-openshift Moving to openshift contrib directory soon.

Running Azure Openshift ARM TemplateSupply the following:

Openshift User Name and Password (No @)

SSH Public and Private(base64) Key

RHN User Name and Password

PoolId for subscription to use

Number of Nodes you want - 3-30 Currently

Azure Machine Sizing for:

Master

Infra

Node

Storage

Running It

Template ComponentsREADME.md Links to Deployment Template

azuredeploy.json ARM Template - Orchestration

bastion.json ARM Template - Bastion Host

master.json ARM Template - Master(s)

node.json ARM Template - Nodes(s)

infranode.json ARM Template - Infra

logging.json ARM Template - Logging

store.json ARM Template - Storage Node(s)

azuredeploy.parameters.json ARM Template - Common Parameters

{hostttype}.sh Bash Script for VM Setup

Naming and Inventory - InternalMasters master1,master2,master3

Infranode Infranode (1 and 2 comming soon)

Nodes Node01--32 (99+ coming soon)

Bastion basion

Storage Store1 (more coming)

Naming and Inventory - ExternalMasters {resourcegroupname}m1...

Infranode Determined by user

Nodes No public Ip

Bastion {resourcegroupname}b1

Storage No Public Ip

Masters and Load BalancingAzure Traffic Manager = Load Balancer

Load Balance - Round Robin

Health Checks

DNS Level

Survives complete data center loss

Considered more reliable than Azure Load Balancer.

BastionUsing a ARM Extension Launched Script

Sets up ssh keys

Gets bastion subscriptions setup

Builds Host Inventory /etc/ansbile/hosts

Setup ansible settings

Build Ansible script to setup subscriptions

Setup PostInstall Script

Build openshift-install.sh

Launch openshift-install.sh

Openshift-install.sh

Turn off .ssh key checking

Run ansible subscribe playbook

Run ansible openshift byo playbook

Run Postinstall playbook

Azure / OSE Storage - OverviewStorage is needed in a few categories

RHEL System/Boot Disk

Docker Container Storage

Persistent Storage

Registry

Azure Has:

No NFS

No Native Iscsi

No FC

Upstream coming for azure block

Azure StorageLessons Learned/Problems:

Azure Standard Storage is really slow

Minimal Config of Script could take 5Hrs

Questionable for Apps with Db/MsgQ

Most apps today in data center are SSD

What we want:

Full HA Redundance

Support for Database Apps

MySQL/MongoDB

Easy to add more storage

Supported with existing storage plugin

Azure Storage SolutionChoose VM types that support Premium Storage

Implement Persistent Volumes based on ISCSI

Use RHEL Iscsi target support

Created automation to automatically create lvm backed ISCSI targets

Iscsi

Quota Enforced by size of volume

Use LVM Striped Volumes

Azure 3x redundancy

Expand more by adding another appliance

Only needs standard RHEL

https://github.com/glennswest/ose_pvcreatelun

Store1 ServerProvisioned automatically as part of AzureDeploy

Start with 8 Data Drives in one volume group

Auto Partition and Format drives

Executes 3x ose_pvcreate

Ose_pvcreate

Auto create iscsi target device

Auto create lun

Auto share the lun

Set acl

Auto create yml pv definition

Register pv with ose

Two (Current) Separate Objectives

● Create A Reference Architecture on best practice to Install OSE on Azure

● To create automations that make it easy to deploy

Current status

Team for OSE / Azure

● One Systems Design and Engineering resource

● PM support

● Engineering mgr support

● Trello board

● Upstream github repo

● Container Network

○ Openshift-sdn

● Load Balancer - HA is Default

○ Azure Traffic Manager for Masters

○ Azure Traffic Manager for Infra

● OpenShift "router" deployed

● Local registry deployed

● DNS

● Authentication

Current status - OSE on Azure

● Auto - Scaling

○ Auto Scaling current not in scope

● Iscsi for Persistent Volumes

○ Docker Registry storage

○ OpenShift Application storage

● Authentication

○ Authenticate based on htpasswd

● Deployment Environments

○ OpenShift deployment via Packages

Current status - OSE on Azure (cont.)● Target OS

○ RHEL 7

● Packages

○ RHEL GA Repos

● QE

● Docs

○ Reference architecture WIP

● Short term:

○ We should ship support these at some point

○ Reference Architecture being worked on by Glenn West

● Medium term:

○ OSE on Azure wishlist:

■ Ansible template to Deploy ARM Template

● Long term:

○ Plugable - Click deployment of additional nodes and storage○ Ansible Tower Integration

Plans

ParticipatingCurrently under active drevelopment.

Current Git Hub:

https://github.com/glennswest/azure-openshift

Soon in upstream. (Active Development above, pushed to contrib soon for stable)

Demo 2

https://youtu.be/NsGYugBX6EM

With active discussion and walkthru.

Demo(s)