Deep DiveOpenShift on Azure

&.NET Core on OpenShift

Takayoshi Tanaka @TanakaTakayoshi

Red Hat K.K. (Japan) tatanaka@redhat.com

Notes:This slide is available online.

As I have tested at OCP 3.5 and .NET Core 2.0 preview2,something will be changed at the latest OCP 3.6 and .NET Core 2.0 RTM.

If you have any question or comments, feel free to contact me:

email: tatanaka@redhat.com

Twitter: @TanakaTakayoshi

BackgroundRed Hat K.K. (Japan)◦ Software Maintenance Engineer

◦ OpenShift◦ Red Hat solutions on Azure◦ .NET Core on RHEL

Personal◦ Microsoft MVP for VSDT

◦ C# Lang, .NET Core on Linux

◦ Blogs:◦ Red Hat Developers◦ Personal Blog “Silver light and Blue sky”

VSDT: Visual Studio & Development Technologies

Goal◦ Learn about OpenShift on Azure Reference Architecture

◦ How to integrate Azure Features with OpenShift

◦ .NET Core 2.0/integrating OpenShift features with ASP.NET Core

Reference ArchitectureDocument is now available◦ Deploying Red Hat OpenShift Container Platform 3 on Microsoft Azure

OpenShift Ansible - Azure ARM Template

Installation Summary

1• ARM Template for Azure Resources (VM, LB, NW…)

2

• Custom Script Extension with ARM

• generate config. files & execute ansible

3• Ansible Installer for OpenShift

Known Issue

Available only in the Azure Marketplace VM

• duplicated billing. Custom image (.vhd) is on the roadmap.

No official Red Hat is available (self-support only)

• You should troubleshoot by yourself.

The OpenShift VM configuration is fixed

• 3 masters with etcd (same hosts), 3 infra nodes, 3+ nodes, 1 bastion

構成図

構成図

構成図

構成図

構成図

VMs1 bastion

3 masters with etcd

3 infra nodes (router/docker registry)

3+ nodes

Support request required for increasing cpu core limit.

This limitation is due to design of ARM template.

You can install all-in-one OpenShift on 1 host (not supported)

Examples: Integrating Azure FeaturesAvailability Set

Azure Load Balancer◦ master endpoint

◦ backend is a group of masters

◦ routing endpoint◦ backend is a group of infra nodes (routers)

Azure VHD for Persistent Volume (PV)◦ Virtual Hard Disk for Azure VM (VHD)

◦ Dynamic provisioning Available at OCP 3.5+

How does Azure VHD for PV work?

node service

/etc/azure.conf

1. node service receivesVolume Mount request

2. Load azure.conf(API auth etc)

3. (if dynamic provisioning)Create an empty VHD

4. Mount VHD to Azure VM5. Create filesystem if needed6. Mount filesystem to container

Depending on kubernetess Azure Volume Plugin

VHD

attach

How to configure azure.conf

See the document for more detail.

Easy 3 steps with Azure CLI 2.0

$ az account list -o json//Retrieve tenantID & id

$ az group show --name <ResourceGroupName> -o json//Retrieve id & location

$ az ad sp create-for-rbac --name <ResourceGroupName> --role contributor \--scopes "<Resource Id>“ -o json//Retrieve appId, password

Azure VHD for PV NotesManaged Disk is unavailable◦ kubernetes Azure Disk plugin is not supported Managed Disk

Be sure to confirm VM name to hostname◦ Also specification of kubernetes plugin

Configure DNS yourself◦ VMs can be communicate with their VM name.

◦ If not using Azure internal DNS

◦ If using VNET peering or other

More Azure FeaturesAzure Active Directory Open ID Connect◦ authentication for master◦ LDAP integration with AAD+AAD DS or AD is also available.

Azure Blob Storage for OpenShift internal docker registry◦ object storage is suitable for docker registry storage

Azure File Storage◦ File storage is also available for PV◦ Linux kernel CIFS module with SMB 3 is still experimental

Operation Management Suite integration◦ Log Analysis also available for containers

How to set up OpenID connect?Create Azure AD App using the Microsoft Azure portal

How to set up LDAP auth with AD?Option A) AAD + AAD DS + (VNET peering or VNET-to-VNET VPN)

* AAD DS only supports Classic VNET and requires private network from ARM VNET.

AAD AAD DS

classic VNET ARM VNET

OCP

master

ldap://xx.xx.xx.xx/

AAD DS configuration example- name: "aad_ds_provider"

challenge: true login: true mappingMethod: claim provider:

apiVersion: v1kind: LDAPPasswordIdentityProviderattributes:

id: - dnemail: - userPrincipalNamename: - cnpreferredUsername: - cn

bindDN: "cn=adadmin,ou=AADDC Users,DC=example,DC=onmicrosoft,DC=com" bindPassword: "<password>" insecure: trueurl: "ldap://XXX.XX.XX.XX/OU=AADDC Users,DC=example,DC=onmicrosoft,DC=com?

userPrincipalName?sub?(memberof=CN=ocpgroup,OU=AADDC Users,DC=example,DC=onmicrosoft,DC=com)"

master-config.yaml

ou: AADDC UsersAAD default OU

userPrincipalName will be email

How to set up LDAP auth with AD?Option B) on premise AD + VPN

Connect on premise Network and ARM Network with VPN.

AD

on premise NW ARM NW

OCP

master

ldap://xx.xx.xx.xx/

Storage Technology ComparisonType References Notes

Azure Blob Storage

Object storage

Extended Registry ConfigurationMicrosoft Azure storage driverDeploying Your Own Private Docker Registry on AzureAzure Blob Storage

Only Available for docker registry storage

Azure VHD Filesystemon external Disk

Persistent Storage Using Azure DiskConfiguring for AzureAbout disks and VHDs for Azure Linux VMs

Depends on k8s plugin

Azure File Storage

NFS Persistent Storage Using Azure FileConfiguring for AzureHow to use Azure File Storage with Linux

Depends on k8s pluginExperimental.

External NFS service

NFS N/A Should maintain yourself or buy 3rd

party service.

Operation Management Suite (OMS)Log Analysis & other features for on-premised to cloud

Containers (Preview) solution in Log Analytics now support OpenShift

Installing OMS AgentAdding OMS agent directly on Linux Host

Or, install agent as a OpenShift daemonset

Container solution

Future: Windows Container?No roadmap: Windows Container

kubernets has roadmap for working with Windows Container

“Capability” is existing.

.NET Application Model

35

.NET Core Inside

36

.NET Core App (C#/VB)

IL Assembly (exe, dll)

Roslyn CoreFX(.NET Core

Class Library)

ManagedUnmanaged

OS

Native ABI etc

(F#)compile

r

CoreCLR(.NET Core Runtime)

.NET Core SDK T

oo

ls

.NET Core on OpenShifts2i build◦ “Source code in the Git repo” To “docker Image”

◦ can run out of OpenShift

s2i image◦ parameters for simple customize

◦ more customization are available with s2i scripts

Template project◦ Start .NET Core on OpenShift with few clicks at the portal

◦ All in one: deploymentconfig, service, route etc…

s2i build & deploy flow

builder pod

SCM(git) internal registry

deployer pod

pod

deploymentConfigbuildConfig

$ dotnet build$ dotnet publish

$ dotnet <dll>

.NET Core 2.0 launch start today!rh-dotnet supports csproj at .NET Core 2.0◦ rpm version will be available

◦ s2i for .NET Core 2.0 & ASP.NET Core 2.0

◦ Runtime image & s2i image (s2i image only at 1.x)

More new features coming◦ Announcing .NET Standard 2.0◦ Announcing .NET Core 2.0

◦ Introducing ASP.NET Core 2.0

◦ Announcing Entity Framework Core 2.0

Use Case Examples◦Schedule Jobs with .NET Core◦Switching Configuration for Dev & Prod Environment◦Razor Page & C# 7.1◦Redis for HTTP Session storage with multi pods

Notes:All examples are built on .NET Core 2.0 preview.We’re actively working on it now.

Schedule Job with .NET Core

Run .NET Core Console App as a cron job: Cron JobsExample Repository

Web portal does not support cron jobs, so use the CLI.

$ oc create imagestream cronjobexample$ oc create -f cronjob-buildconfig.yaml$ oc create -f cronjob.yaml

Schedule Jobs with .NET Corespec:

containers:- command:

- /opt/rh/rh-dotnet20/root/bin/dotnet- bin/Release/netcoreapp2.0/CronJobExample.dllimage: 172.30.142.2:5000/london-openshift/cronjobexample:latestimagePullPolicy: Alwaysname: lipsum-cronresources: {}

restartPolicy: NeversecurityContext: {}terminationGracePeriodSeconds: 30

schedule: '*/1 * * * *'

command to execute:should be the full path

command to execute:*scl should be enabled.

To be fixed in my example

image should be specified with full URLOCP 3.6 will support imagestreamtag.Replace 172.30.142.2:5000 with your

internal registry’s IP and port

Switching Configuration for Dev & Prod Environment

How to treat different environments with one code◦ Connect to different database

◦ Use Redis as a cache - only in a production environment

◦ Integrate with a different OpenID account

Use Environment feature in ASP.NET Core◦ Specified by environment variables.

Configuration can be injected specific to each environment.

Switching with Environment

Startup class• constructor• Configure method• ConfigurreService method

Check IHostingEnvironment

Can’t inject IHostingEnvironment into ConfigerService method

Switching ConfigureXXXServicesConfigureXXXServices

Switching ConfigureXXXConfigureXXX

Inject configuration from Environment Variable

Loading configuration from Secretuse OpenShift secret feature.

Razor Page + C# 7.1RazorPage:◦ Simpler application than original MVC: “Page-focused scenarios”

◦ WebMatrix like easy development◦ Razor Page is enabled with MVC

C# 7.1:◦ available at .NET Core 2.0 & ASP.NET Core 2.0

◦ C# 7.1 in Razor page is not working at Preview 2 by bug (see issue)◦ It should be fixed at 2.0 RTM.

HTTP session for multi pods By default:◦ Sticky session: request goes to the same pod in same user session ◦ HTTP session is stored in the memory of each pod◦ HTTP session is encrypted by pod specific key

It means:

When a pod has died, a user session will be lost.

How to keep HTTP session

IDistributedCache & IDataProtectionIDistributedCache◦ Provide distribution cache

◦ Available for storing session

◦ ASP.NET Core team provides SQLServer and Redis

IDataProtection◦ Provide key management for encryption

◦ Encrypt http session

◦ By default, generate machine (=pod) specific key and store in local file

◦ ASP.NET Core team provides NFS, Redis and AzureStorage (Preview)

machine A

pod B

pod C

ASP.NET Core

ASP.NET Core

ASP.NET Core

Session Data A’

Each pod has a different key.Can’t decrypt session data When loading another pod from a different sessionload with same id.

~/.aspnet

default implementation of IDataProtection

machine A

machine B

machine C

ASP.NET Core

ASP.NET Core

ASP.NET Core

Session Data

Use DataProtection.Redis

Configuration for Redispublic void ConfigureServices(IServiceCollection services){

// You can retrieve this connection string from Azure Portal.var conn = Configuration["REDIS_CONNECTION_STRING"];

var redis = ConnectionMultiplexer.Connect(conn);

services.AddDataProtection().PersistKeysToRedis(redis, "DataProtection-Keys");

services.AddDistributedRedisCache(option =>{

option.Configuration = conn;option.InstanceName = "master";

});services.AddSession();

}

httpsession.redis.cache.windows.net:6380,password=<password>,ssl=True,abortConnect=False

High Level Debugging .NET Core

58

GDB/LLDB

MICore

MIDE/Engine.Impl

MIDE(*)/AD7.Impl

VSCode Debugger

AD7 Interface

*MIDE: MIDebugEngine: GitHub repository*vsdbg can be used only in VS products and might not be distributed.

MIText

VS Debugger

VS Debugger Engine vsdbg

(closed license*)

windbg

See more:Architecture of MIEngine

Remote Debugging .NET Corevsdbg provided by Microsoft◦ only trusted communication is required

◦ SSH is generally available

◦ VS remote debugger tools is also available on Windows

◦ Due to the license limitation, VS products (VS, VS Code, VS for mac) are only available for debugging.

* Low level debugger is provided by Red Hat◦ sos

◦ Not providing graphical debugger interface

59

Remote debugging toa container on OpenShift“oc rsh” is available instead of ssh

vsdbg should be manually installed◦ install script is unavailable as s2i image doesn’t have unzip

◦ download vsdbg on local and rsync

see more detail in my wiki

Remote debug from Visual Studio Code.vscode/launch.json

61

{"name": ".NET Core Docker Remote Attach","type": "coreclr","request": "attach",“processId”: “1”, "pipeTransport": {

"pipeProgram": “oc",“pipeArgs”: [ “rsh”, “-T”, “tanaka733@centos.example.com”], “quoteArgs”:false, “debuggerPath”: “/opt/app-root/src/vsdbg/vsdbg”, "pipeCwd": "${workspaceRoot}"

},"sourceFileMap": {

"/opt/app-root/src": "${workspaceRoot}“}

}

SummaryOpenShift on Azure◦ Reference Architecture is a good place to start.

◦ More Azure features available-- Authenticating with OpenID and others

.NET Core 2.0/ASP.NET Core 2.0 on OpenShift◦ csproj support

◦ cronjob for .NET Core console app

◦ OpenShift secret & configuration. ASP.NET Core environment

◦ Remote debugging