Architecting world class azure resource manager templates

Marc Mercuri Rama Ramani Arsen VladimirskyPrincipal PM Senior PM Senior SDEAzure CAT Azure CAT TED

Architecting World Class Azure Resource Manager Templates

ARCCT308 #TR21ARCCT308

Presented 2015

Session Objectives And TakeawaysSession Objective(s): Understand key customer scenarios for ARM templates across Enterprises, SIs, and CSVsUnderstand key considerations and how to architect ARM templatesUnderstand how ARM relates to Powershell DSC, Chef, and Puppet and how to use them together

Consistent Management Layer

AZURE RESOURCE MANAGER API

Areas of Focus

Deploy Organize Control

template-driven declarative idempotent multi-service multi-region extensible

Deploying with Azure Resource Manager

Resource Group container for multiple resources resources exist in one* resource

group resource groups can span

regions resource groups can span

services

RESOURCE GROUP

*and only one

imperative and/ordeclarativeYou decide

New-AzureVM –VM $myVMNew-AzureStorageAccount –StorageAccountName $acctSet-AzureVNetConfig –ConfigurationPath -Path

{ "$schema": "https://../deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": {}, "variables": {}, "resources": [], "outputs": {}}

Deployment tracks template execution created within a resource

group allows nested deployments

RESOURCE GROUP

Demo

Enterprises and System Integrators Internal Software Development Teams

Delivering an application Corporate IT

Delivering a capability or cloud capacity Cloud Service Vendors (CSVs)

Support different multi-tenancy approaches Distinct deployments per customer

Within the CSV’s subscription “Bring Your Own Subscription” model that uses customer subscriptions

Scale units within a central multi-tenant system Support ability to make available via the marketplace

All deploy known configurations/skus/t-shirt sizes

Common Use Cases for ARM Templates

Lots of variables makes free form less desirable Potentially different VM types, different number of disks, and instances

Hadoop with 8 master nodes w/ 4 attached disks and 16 disks pooled on each data node would have 208 VMs and 3232 disks.

Resource constraints need to be factored on the fly No math capabilities in ARM language Harder to price, manage, support for the Enterprise, SI, or CSV

T-shirt Sizes / SKUs are the common approach Known configuration makes testing easier Easier to model cost/price Easier to support Easier to manage Allows for better density within a subscription

Known Configurations/SKUs vs. Free Form

Capacity May be one or more linked templates with a scoped purpose Delivers an environment, with appropriate security and auditing for

compliance Capability

May be one or more linked templates with a scoped purpose Delivers a technology or a capability that can be utilized post-

deployment Examples include delivering capabilities such as SQL Server,

Cassandra, Elasticsearch, Hadoop, a Web Server, etc. End to End Solution

Compositions of one or more capability scoped templates Examples include end to end data pipeline

Common Template Scopes

Template describes the topology (outside the box)

Template extensions can initiate state configuration (inside the box)

Multiple extensions available DSC Chef Puppet Custom Scripts AppService + WebDeploy SQLDB + BACPAC

Inside the Box vs. Outside the Box

Note – at launch, extensions running longer than 90 minutes trigger a timeout error

base64encode(‘stringtoencode’)concat(‘string’,’to’,’encode’)copyIndex(offset)listKeys(storageAccountResourceId, apiVersion)padLeft(stringToPad,targetLength,paddingCharacter)parameters(‘parameterName’)providers(namespace, resourceType)reference(resourceId,apiVersion)resourceGroup()resourceId(‘namespace/resourceType', ‘resourceName’)subscription()variables(‘variables’)

@ a glance - template language expressions

{ "apiVersion": "2015-05-01-preview", "type": "Microsoft.Compute/virtualMachines", "name": "[concat(parameters('vmNamePrefix'), copyindex())]", "location": "[parameters('location')]", "copy": { "name": "virtualMachineLoop", "count": "[parameters('numberOfInstances')]" }, "dependsOn": [ "[concat('Microsoft.Network/networkInterfaces/', 'nic', copyindex())]" ], "properties": { "hardwareProfile": { "vmSize": "[parameters('vmSize')]" }, "osProfile": { "computername": "[concat('vm', copyIndex())]", "adminUsername": "[parameters('adminUsername')]", "adminPassword": "[parameters('adminPassword')]" }, "storageProfile": { "osDisk": { "name": "[concat(parameters('vmNamePrefix'),'-osDisk',copyindex())]", "osType": "[parameters('osType')]", "caching": "ReadWrite", "image": { "uri": "[variables('userImageName')]" }, "vhd": { "uri": "[concat(variables('osDiskVhdContainer'),parameters('vmNamePrefix'),copyindex(),'osDisk.vhd')]" } } },

Resource loops deploy n instances

Fixed or parameterdriving instance count

Concat + Parameter Prefix + CopyIndex() for dynamic naming

Deploying Multiple Instances

Provides the ability to link multiple templates

Can be used to facilitate decomposition Template URIs can be dynamic Allows for re-use

Template Linking

{ "name": "cluster-nodes", "type": "Microsoft.Resources/deployments", "apiVersion": "2015-01-01", "dependsOn": [ "[concat('Microsoft.Resources/deployments/', 'shared')]" ], "properties": { "mode": "Incremental", "templateLink": { "uri": "[variables('clusterNodesTemplateUrl')]", "contentVersion": "1.0.0.0" },

Execution engine builds a state machine dependsOn and reference() establish

dependencies

Template Execution

StartApp

Service Plan

EndAuto Scale Settin

g

WebSite Alert

Rule

App Insight

s

MS Deploy PKG

After App Service Plan Completes

After Website Completes Once All

Complete

Multiple types of state Parameters Static Variables Dynamic Variables

Templates accept parameters Templates return variables as output

values Simple or Complex object types are

supported

Passing State In and Out of Templates

Easier to pass a number of related values with a single variable

Object.Property approach provides additional context when reading the template

Examples

Passing State - Complex Objects

"networkSettings": { "vnetName": "[parameters('virtualNetworkName')]", "addressPrefix": "10.0.0.0/16", "subnets": { "dmz": { "name": "dmz", "prefix": "10.0.0.0/24", "vnet": "[parameters('virtualNetworkName')]" }, "data": { "name": "data", "prefix": "10.0.1.0/24", "vnet": "[parameters('virtualNetworkName')]" } } }

"osSettings": { "imageReference": { "publisher": "Canonical", "offer": "UbuntuServer", "sku": "14.04.2-LTS", "version": "latest" }

"tshirtSizeSmall": { "vmSize": "Standard_A1", "diskSize": 1023, "vmTemplate": "[concat(variables('templateBaseUrl'), 'database-2disk-resources.json')]", "vmCount": 2, "storage": { "name": "[parameters('storageAccountNamePrefix')]", "count": 1, "pool": "db", "map": [0,0], "jumpbox": 0 } },

"availabilitySetSettings": { "name": "pgsqlAvailabilitySet", "fdCount": 3, "udCount": 5 }

A template can return values to its caller via the outputs section

These values can then be used by the caller

Passing State – Output variables

"outputs": {"masterip": {"value": "[reference(concat(variables('nicName'),0)).ipConfigurations[0].properties.privateIPAddress]","type":"string"}}

"masterIpAddress": {"value":"[reference('master-node').outputs.masterip.value]" } }

Passing State - Common ParametersName Value DescriptionLocation String The location where the resources will be deployed from a constrained list

of Azure regions.storageAccountNamePrefix

String Unique DNS name for the Storage Account where the VM’s disks will be placed

virtualNetworkName

String For deployments that create a new Virtual Network, the name to use for creating that resource. For deployments that use an existing Virtual Network, the name of the VNet to deploy into.

username String User name for the virtual machine(s) and potentially the application(s). More than one user name can be requested from the end user, but at least one must be prompted.

password String Password for the virtual machine(s) and potentially the application(s). More than one password can be requested from the end user for different VMs or applications, but at least one must be prompted.

tshirtSize String The named scale unit size to provision from a constrained list ofoffered t-shirt sizes. For example, “Small”, “Medium”, “Large”

enableJumpbox String Parameter that identifies whether to enable a jumpbox for the environment. Values: “enabled”, “disabled”

No control flow logic built into ARM template language

An approach with parameters, variables, and linked templates Use provides parameter value that provides context, e.g. tshirtSize

parameter is passed in as a value of ‘small’ Using concat and a pre-defined variable, a new variable value is

created which points to the specific , e.g. ‘tshirtSize-small.json’ Template linking is incorporated into the template and uses this new

value to identify which template to deploy. Common examples are “tshirt sizes” and optional features for a

deployment, e.g. “enableJumpbox”

Control Flow

UserNamePasswordstorageAccountnamelocationvirtualNetworkNameaddressPrefixsubnetNamesubnetPrefixjumpboxtshirtSizeosFamily

Capability or Capacity Scoped Template

Capability or Capacity Scoped TemplateUserNamePasswordstorageAccountnamelocationvirtualNetworkNameaddressPrefixsubnetNamesubnetPrefixjumpboxtshirtSizeosFamily







Widely Re-UsableScript(s)


Optional CapabilityTemplate

setup.sh



Member ResourcesTemplate


End to End Solution Scoped TemplateUserNamePasswordstorageAccountnamelocationvirtualNetworkNameaddressPrefixsubnetNamesubnetPrefixjumpboxtshirtSizeosFamily

Templates can be made available via the marketplace

SKU/known configuration is the marketplace scope Small, Medium, Large t-shirt sizes Community or Enterprise skus

End to End Solution or Capability scoped templates work with 2 simple edits Remove SKU / known configuration / t-shirt size parameter from Core

Template Use static variable assignment within the core template, e.g. tshirtSize

= “small” to take the place of the parameter

Marketplace



setup.sh



Modifying the template (small t-shirt size)

“variables:{ “tshirtSize”:“small“



setup.sh



Modifying the template (medium t-shirt size)

“variables:{ “tshirtSize”:“medium“

role based access control audit logs resource locks

Control with Azure Resource Manager

Role Based Access Control Allows secure access with granular

permissions

Assignable to users, groups, or service principals

Built-in roles make it easy to get startedNOTE – RBAC in ARM applies to resources exposed via ARM. Software inside of VMs which may have it’s own security mechanisms and should be considered during design.

Two Key ConceptsRole Definitions

• describes the set of permissions (e.g. read actions)

• can be used in multiple assignments

Role Assignments• associate role definitions

with an identity (e.g. user/group) at a scope (e.g. resource group)

• always inherited – subscription assignments apply to all resources

Role Based Access Control

Granular Scopes

/subscriptions/{id}/resourceGroups/{name}/providers/…/sites/{site}

subscription level – grants permissions to all resources in the sub

resource group level – grants permissions to all resources in the group

resource level – grants permissions to the specific resource

Encrypt keys and small secrets like passwords Import or generate your keys in HSMs

(certified to FIPS 140-2 level 2 standards) Grant permission for your own and third party applications to use as

needed Monitor and audit key use with Azure logging

Pipe logs into HDInsight or your SIEM for additional analysis and threat detection (coming soon)

Provision new vaults or keys in minutes Scales to meet peak demand w/o having to deploy dedicated HSMs Provision vaults across Azure global DCs for global redundancy Microsoft does not see or extract your keys

Key management with Key Vault

Azure VMs, ARM and Key Vault are fully integrated

Benefits of using them together – ARM templates contain only URI references to secrets

Actual secrets are not in code, config or source code repositories Secrets stored in Key Vault are under full RBAC control of a trusted

operator Full compartmentalization of assets Loading of secrets into the VM at deployment occurs via direct channel

between Azure Fabric and the Key Vault within the Microsoft data center.

Key vaults are always regional, so the secrets always have locality (and sovereignty) with the VMs. There are no global key vaults.

Proven practice is to separate ARM templates for Creation of Vaults (which will contain the key material) Deployment of the VMs (with URI references the keys contained in the

vaults)

Secrets and Certificates

Each Network Security Group Has a name, is associated with a Region and has a descriptive label Has two types of rules, Inbound and Outbound that control traffic to

one or more VM instances Inbound rules are applied on the incoming packets to a VM Outbound rules are applied to outgoing packets from the VM Incoming or outgoing packets must match an ‘Allow’ rule to be permitted;

otherwise, it’s dropped Rules are processed in order of priority, with lower numbers taking precedence over

higher ones Once a match is found, no more rules are processed.

Can be associated with a VNET, Subnet or a VM within a VNET A VM or subnet can be associated with only 1

NSG, but each NSG can contain up to 200 rulesNOTE – endpoint-based ACLs and NSGs are not supported on the same instance.NOTE – NSGs are not compatible with VNETS associated with an affinity group

Network Security Groups

A rule specifies the following: Name: A unique identifier for the rule Type: Inbound/Outbound Priority: An integer between 100 and 4096 Source IP Address: CIDR of source IP range Source Port Range: An integer or range between 0 and 65536 Destination IP Range: CIDR of the destination IP Range Destination Port Range: An integer or range between 0 and 65536 Protocol: TCP, UDP or ‘*’ Access: Allow/Deny

ICMP cannot be specified but is allowed within a virtual network

A range of ports can be specified with a hyphen, e.g. 100-500 Rules can be changed at any time

Network Security Groups - Rules

Network Security Groups - Default Rules

Default Outbound Rules

Default Inbound Rules

NSG Examples { "apiVersion": "2015-05-01-preview", "type": "Microsoft.Network/networkSecurityGroups", "name": "[parameters('FENSGName')]", "location": "[parameters('location')]",

"properties": { "securityRules": [ { "name": "rdp_rule", "properties": { "description": "Allow RDP", "protocol": "Tcp", "sourcePortRange": "*", "destinationPortRange": "3389", "sourceAddressPrefix": "Internet", "destinationAddressPrefix": "*", "access": "Allow", "priority": 100, "direction": "Inbound" } }, { "name": "web_rule", "properties": { "description": "Allow WEB", "protocol": "Tcp", "sourcePortRange": "*", "destinationPortRange": "80", "sourceAddressPrefix": "Internet", "destinationAddressPrefix": "*", "access": "Allow", "priority": 101, "direction": "Inbound" } } ] } }

{ "apiVersion": "2015-05-01-preview", "type": "Microsoft.Network/networkSecurityGroups", "name": "[parameters('AppNSGName')]", "location": "[parameters('location')]",

"properties": { "securityRules": [{ "name": "Allow_FE", "properties": { "description": "Allow FE Subnet", "protocol": "Tcp", "sourcePortRange": "*", "destinationPortRange": "443", "sourceAddressPrefix": "10.0.0.0/24", "destinationAddressPrefix": "*", "access": "Allow", "priority": 100, "direction": "Inbound" } }, { "name": "Block_RDP_Internet", "properties": { "description": "Block RDP", "protocol": "tcp", "sourcePortRange": "*", "destinationPortRange": "3389", "sourceAddressPrefix": "Internet", "destinationAddressPrefix": "*", "access": "Deny", "priority": 101, "direction": "Inbound" } }, { "name": "Block_Internet_Outbound", "properties": { "description": "Block Internet", "protocol": "*", "sourcePortRange": "*", "destinationPortRange": "*", "sourceAddressPrefix": "*", "destinationAddressPrefix": "Internet", "access": "Deny", "priority": 200, "direction": "Outbound" } }] } }

Resource Tags Name-value pairs assigned to

resources or resource groups

Subscription-wide taxonomy

Each resource can have up to 15 tags

x

Tagging Tips Tag by environment, e.g.

dev/test/prod

Tag by role, e.g. web/cache/db

Tag by department, e.g. finance/retail/legal

Tag by responsible party, e.g. Bob

x

Audit Logs journals all write/delete/actions

central location

common format

Resource Locks Accidents happen. Resource locks help

prevent them :)

Resource locks allow administrators to create policies which prevent write actions or prevent accidental deletion.

Key Concepts Resource lock

Policy which enforces a "lock level" at a particular scope Lock level

Type of enforcement; current values include CanNotDelete and ReadOnly

Scope: The realm to which the lock level is applied. Expressed as a URI;

can be set at the resource group, or resource scope.

Testing Templates are code, manage in source

control Test, test, test

Test different distros/versions Different Linux distros may have different default behaviors mke2fs, which formats disks on Linux, has different default

behaviors across distros/versions. Fast formatting is not turned on by default in CentOs 6.5

Test different regions Different service availability in different regions can impact

deployments Test different VM SKUs / Instance Counts

If you offer the customer the ability to choose different VM SKUs or different instance counts of resources, test. Identify potential issues with performance, resource constraints, script

timeouts, et. Decomposition allows for isolating key aspects for granular levels of

testing (solution, building block, linked template within either of these)

Websites SQL Azure DB Redis Cache 40+ other resource types v1 Virtual Machines, Network, Storage v2 Virtual Machines, Network, Storage

Resource Group Supported Resources

52

v1: Not fully integrated: no RBAC, tagging, templates Clunky network modelling

v2: Fully integrated into Resource Manager Revamped control plane with asynchronous, parallel operations Network resource types are separate from compute

Part of Azure-consistent private cloud

What’s New About the v2 Resource Providers?

MICROSOFT CONF IDENTIAL – INTERNAL ONLY

Many examples available @ https://github.com/Azure/azure-quickstart-templates

Documentation available @ http://azure.microsoft.com/en-us/documentation/articles/resource-group-overview/

Getting started with templates

Getting StartedAzure Resource Manager OverviewUsing Windows PowerShell with Resource ManagerUsing the Azure Cross-Platform Command-Line Interface with the Resource ManagerUsing the Azure Portal to manage your Azure resources Creating and Deploying ApplicationsAuthoring Azure Resource Manager TemplatesDeploy an application with Azure Resource Manager templateTroubleshooting Resource Group Deployments in AzureAzure Resource Manager Template FunctionsAdvanced Template Operations Organizing Resources Using tags to organize your Azure resources Managing and Auditing Access Managing and Auditing Access to ResourcesAuthenticating a Service Principal with Azure Resource ManagerCreate a new Azure Service Principal using the Azure classic portal

Next Steps

http://azure.microsoft.com/en-us/documentation/articles/resource-group-overview/

http://azure.microsoft.com/en-us/documentation/articles/powershell-azure-resource-manager/

http://azure.microsoft.com/en-us/documentation/articles/xplat-cli-azure-resource-manager/

http://azure.microsoft.com/en-us/documentation/articles/resource-group-portal/

http://azure.microsoft.com/en-us/documentation/articles/resource-group-authoring-templates/

http://azure.microsoft.com/en-us/documentation/articles/resource-group-template-deploy/

http://azure.microsoft.com/en-us/documentation/articles/resource-group-deploy-debug/

http://azure.microsoft.com/en-us/documentation/articles/resource-group-template-functions/

http://azure.microsoft.com/en-us/documentation/articles/resource-group-advanced-template/

http://azure.microsoft.com/en-us/documentation/articles/resource-group-using-tags/

http://azure.microsoft.com/en-us/documentation/articles/resource-group-using-tags/

http://azure.microsoft.com/en-us/documentation/articles/resource-group-rbac/

http://azure.microsoft.com/en-us/documentation/articles/resource-group-rbac/

http://azure.microsoft.com/en-us/documentation/articles/resource-group-authenticate-service-principal/

http://azure.microsoft.com/en-us/documentation/articles/resource-group-create-service-principal-portal/

In Review: Session Objectives And TakeawaysSession Objective(s): Understand key customer scenarios for ARM templates across Enterprises, SIs, and CSVsUnderstand key considerations and how to architect ARM templatesUnderstand how ARM relates to Powershell DSC, Chef, and Puppet and how to use them together

© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.