OpenShift Container Platform 4...1.8.14. Creating the worker nodes in AWS OpenShift Container Platform 4.6 Installing on AWS 4. OpenShift Container Platform 4.6 Installing on AWS 6

OpenShift Container Platform 46

Installing on AWS

Installing OpenShift Container Platform AWS clusters

Last Updated 2021-03-31

OpenShift Container Platform 46 Installing on AWS

Legal Notice

Copyright copy 2021 Red Hat Inc

The text of and illustrations in this document are licensed by Red Hat under a Creative CommonsAttributionndashShare Alike 30 Unported license (CC-BY-SA) An explanation of CC-BY-SA isavailable athttpcreativecommonsorglicensesby-sa30 In accordance with CC-BY-SA if you distribute this document or an adaptation of it you mustprovide the URL for the original version

Red Hat as the licensor of this document waives the right to enforce and agrees not to assertSection 4d of CC-BY-SA to the fullest extent permitted by applicable law

Red Hat Red Hat Enterprise Linux the Shadowman logo the Red Hat logo JBoss OpenShiftFedora the Infinity logo and RHCE are trademarks of Red Hat Inc registered in the United Statesand other countries

Linux reg is the registered trademark of Linus Torvalds in the United States and other countries

Java reg is a registered trademark of Oracle andor its affiliates

XFS reg is a trademark of Silicon Graphics International Corp or its subsidiaries in the United Statesandor other countries

MySQL reg is a registered trademark of MySQL AB in the United States the European Union andother countries

Nodejs reg is an official trademark of Joyent Red Hat is not formally related to or endorsed by theofficial Joyent Nodejs open source or commercial project

The OpenStack reg Word Mark and OpenStack logo are either registered trademarksservice marksor trademarksservice marks of the OpenStack Foundation in the United States and othercountries and are used with the OpenStack Foundations permission We are not affiliated withendorsed or sponsored by the OpenStack Foundation or the OpenStack community

All other trademarks are the property of their respective owners

Abstract

This document provides instructions for installing and uninstalling OpenShift Container Platformclusters on Amazon Web Services

Table of Contents

CHAPTER 1 INSTALLING ON AWS11 CONFIGURING AN AWS ACCOUNT

111 Configuring Route 531111 Ingress Operator endpoint configuration for AWS Route 53

112 AWS account limits113 Required AWS permissions114 Creating an IAM user115 Supported AWS regions116 Next steps

12 INSTALLING A CLUSTER ON AWS WITH CUSTOMIZATIONS121 Prerequisites122 Internet and Telemetry access for OpenShift Container Platform123 Generating an SSH private key and adding it to the agent124 Obtaining the installation program125 Creating the installation configuration file

1251 Installation configuration parameters1252 Sample customized install-configyaml file for AWS

126 Deploying the cluster127 Installing the OpenShift CLI by downloading the binary

1271 Installing the OpenShift CLI on Linux1272 Installing the OpenShift CLI on Windows1273 Installing the OpenShift CLI on macOS

128 Logging in to the cluster by using the CLI129 Logging in to the cluster by using the web console1210 Next steps

13 INSTALLING A CLUSTER ON AWS WITH NETWORK CUSTOMIZATIONS131 Prerequisites132 Internet and Telemetry access for OpenShift Container Platform133 Generating an SSH private key and adding it to the agent134 Obtaining the installation program135 Creating the installation configuration file

1351 Installation configuration parameters1352 Network configuration parameters1353 Sample customized install-configyaml file for AWS

136 Modifying advanced network configuration parameters137 Configuring an Ingress Controller Network Load Balancer on a new AWS cluster138 Cluster Network Operator configuration

1381 Configuration parameters for the OpenShift SDN default CNI network provider1382 Configuration parameters for the OVN-Kubernetes default CNI network provider1383 Cluster Network Operator example configuration

139 Configuring hybrid networking with OVN-Kubernetes1310 Deploying the cluster1311 Installing the OpenShift CLI by downloading the binary

14 INSTALLING A CLUSTER ON AWS INTO AN EXISTING VPC141 Prerequisites

10181819

2020202122232432353636373738383939404041

424244525355575859606061

62646465656666676767

Table of Contents

142 About using a custom VPC1421 Requirements for using your VPC1422 VPC validation1423 Division of permissions1424 Isolation between clusters

143 Internet and Telemetry access for OpenShift Container Platform144 Generating an SSH private key and adding it to the agent145 Obtaining the installation program146 Creating the installation configuration file

1461 Installation configuration parameters1462 Sample customized install-configyaml file for AWS1463 Configuring the cluster-wide proxy during installation

15 INSTALLING A PRIVATE CLUSTER ON AWS151 Prerequisites152 Private clusters

1521 Private clusters in AWS15211 Limitations

154 Internet and Telemetry access for OpenShift Container Platform155 Generating an SSH private key and adding it to the agent156 Obtaining the installation program157 Manually creating the installation configuration file

16 INSTALLING A CLUSTER ON AWS INTO A GOVERNMENT REGION161 Prerequisites162 AWS government regions163 Private clusters

164 About using a custom VPC1641 Requirements for using your VPC

686870717171727374758486888990909191

929393939394949494979797979899

100101

109111

113114114115115116116117117117118118119119119

1642 VPC validation1643 Division of permissions1644 Isolation between clusters

1681 Installation configuration parameters1682 Sample customized install-configyaml file for AWS1683 AWS regions without a published RHCOS AMI1684 Uploading a custom RHCOS AMI in AWS1685 Configuring the cluster-wide proxy during installation

17 INSTALLING A CLUSTER ON USER-PROVISIONED INFRASTRUCTURE IN AWS BY USINGCLOUDFORMATION TEMPLATES

171 Prerequisites172 Internet and Telemetry access for OpenShift Container Platform173 Required AWS infrastructure components

1731 Cluster machines1732 Certificate signing requests management1733 Other infrastructure components1734 Required AWS permissions

174 Obtaining the installation program175 Generating an SSH private key and adding it to the agent176 Creating the installation files for AWS

1761 Optional Creating a separate var partition1762 Creating the installation configuration file1763 Configuring the cluster-wide proxy during installation1764 Creating the Kubernetes manifest and Ignition config files

177 Extracting the infrastructure name178 Creating a VPC in AWS

1781 CloudFormation template for the VPC179 Creating networking and load balancing components in AWS

1791 CloudFormation template for the network and load balancers1710 Creating security group and roles in AWS

17101 CloudFormation template for security objects1711 RHCOS AMIs for the AWS infrastructure

17111 AWS regions without a published RHCOS AMI17112 Uploading a custom RHCOS AMI in AWS

1712 Creating the bootstrap node in AWS17121 CloudFormation template for the bootstrap machine

1713 Creating the control plane machines in AWS17131 CloudFormation template for control plane machines

1714 Creating the worker nodes in AWS17141 CloudFormation template for worker machines

1715 Initializing the bootstrap sequence on AWS with user-provisioned infrastructure

122122122123123124125126134136137139141

142142143143144144145

146146146147148149149155163163164165167168170172172174180183191

194203204204206

211215221

228232235

Table of Contents

1716 Installing the OpenShift CLI by downloading the binary17161 Installing the OpenShift CLI on Linux17162 Installing the OpenShift CLI on Windows17163 Installing the OpenShift CLI on macOS

1717 Logging in to the cluster by using the CLI1718 Approving the certificate signing requests for your machines1719 Initial Operator configuration

17191 Image registry storage configuration171911 Configuring registry storage for AWS with user-provisioned infrastructure171912 Configuring storage for the image registry in non-production clusters

1720 Deleting the bootstrap resources1721 Creating the Ingress DNS Records1722 Completing an AWS installation on user-provisioned infrastructure1723 Logging in to the cluster by using the web console1724 Additional resources1725 Next steps

18 INSTALLING A CLUSTER ON AWS THAT USES MIRRORED INSTALLATION CONTENT181 Prerequisites182 About installations in restricted networks

1821 Additional limits183 Internet and Telemetry access for OpenShift Container Platform184 Required AWS infrastructure components

185 Generating an SSH private key and adding it to the agent186 Creating the installation files for AWS

18101 CloudFormation template for security objects1811 RHCOS AMIs for the AWS infrastructure1812 Creating the bootstrap node in AWS

18121 CloudFormation template for the bootstrap machine1813 Creating the control plane machines in AWS

18131 CloudFormation template for control plane machines1814 Creating the worker nodes in AWS

18141 CloudFormation template for worker machines1815 Initializing the bootstrap sequence on AWS with user-provisioned infrastructure1816 Logging in to the cluster by using the CLI1817 Approving the certificate signing requests for your machines1818 Initial Operator configuration

236236237237237238241242242243243244247247248248249249250250250251251

253253259266267268270272273275276278283287295297306307312316321

329333335336337340341341

19 UNINSTALLING A CLUSTER ON AWS191 Removing a cluster that uses installer-provisioned infrastructure

342343345346347347348348

Table of Contents

CHAPTER 1 INSTALLING ON AWS

11 CONFIGURING AN AWS ACCOUNT

Before you can install OpenShift Container Platform you must configure an Amazon Web Services(AWS) account

111 Configuring Route 53

To install OpenShift Container Platform the Amazon Web Services (AWS) account you use must have adedicated public hosted zone in your Route 53 service This zone must be authoritative for the domainThe Route 53 service provides cluster DNS resolution and name lookup for external connections to thecluster

Procedure

1 Identify your domain or subdomain and registrar You can transfer an existing domain andregistrar or obtain a new one through AWS or another source

If you purchase a new domain through AWS it takes time for the relevant DNSchanges to propagate For more information about purchasing domains throughAWS see Registering Domain Names Using Amazon Route 53 in the AWSdocumentation

2 If you are using an existing domain and registrar migrate its DNS to AWS See Making AmazonRoute 53 the DNS Service for an Existing Domain in the AWS documentation

3 Create a public hosted zone for your domain or subdomain See Creating a Public Hosted Zonein the AWS documentationUse an appropriate root domain such as openshiftcorpcom or subdomain such as clustersopenshiftcorpcom

4 Extract the new authoritative name servers from the hosted zone records See Getting theName Servers for a Public Hosted Zone in the AWS documentation

5 Update the registrar records for the AWS Route 53 name servers that your domain uses Forexample if you registered your domain to a Route 53 service in a different accounts see thefollowing topic in the AWS documentation Adding or Changing Name Servers or Glue Records

6 If you are using a subdomain add its delegation records to the parent domain This givesAmazon Route 53 responsibility for the subdomain Follow the delegation procedure outlined bythe DNS provider of the parent domain See Creating a subdomain that uses Amazon Route 53as the DNS service without migrating the parent domain in the AWS documentation for anexample high level procedure

1111 Ingress Operator endpoint configuration for AWS Route 53

If you install in either Amazon Web Services (AWS) GovCloud (US) US-West or US-East region theIngress Operator uses us-gov-west-1 region for Route53 and tagging API clients

The Ingress Operator uses httpstaggingus-gov-west-1amazonawscom as the tagging APIendpoint if a tagging custom endpoint is configured that includes the string us-gov-east-1

For more information on AWS GovCloud (US) endpoints see the Service Endpoints in the AWSdocumentation about GovCloud (US)

IMPORTANT

Private disconnected installations are not supported for AWS GovCloud when you installin the us-gov-east-1 region

Example Route 53 configuration

Route 53 defaults to httpsroute53us-govamazonawscom for both AWS GovCloud (US)regions

Only the US-West region has endpoints for tagging Omit this parameter if your cluster is inanother region

112 AWS account limits

The OpenShift Container Platform cluster uses a number of Amazon Web Services (AWS) componentsand the default Service Limits affect your ability to install OpenShift Container Platform clusters If youuse certain cluster configurations deploy your cluster in certain AWS regions or run multiple clustersfrom your account you might need to request additional resources for your AWS account

The following table summarizes the AWS components whose limits can impact your ability to install andrun OpenShift Container Platform clusters

Component

Number ofclustersavailable bydefault

Default AWSlimit

Description

platform aws region us-gov-west-1 serviceEndpoints - name ec2 url httpsec2us-gov-west-1amazonawscom - name elasticloadbalancing url httpselasticloadbalancingus-gov-west-1amazonawscom - name route53 url httpsroute53us-govamazonawscom 1 - name tagging url httpstaggingus-gov-west-1amazonawscom 2

InstanceLimits

Varies Varies By default each cluster creates the followinginstances

One bootstrap machine which is removedafter installation

Three master nodes

Three worker nodes

These instance type counts are within a newaccountrsquos default limit To deploy more workernodes enable autoscaling deploy large workloadsor use a different instance type review your accountlimits to ensure that your cluster can deploy themachines that you need

In most regions the bootstrap and worker machinesuses an m4large machines and the mastermachines use m4xlarge instances In some regionsincluding all regions that do not support theseinstance types m5large and m5xlarge instancesare used instead

Elastic IPs(EIPs)

0 to 1 5 EIPs peraccount

To provision the cluster in a highly availableconfiguration the installation program creates apublic and private subnet for each availability zonewithin a region Each private subnet requires a NATGateway and each NAT gateway requires aseparate elastic IP Review the AWS region map todetermine how many availability zones are in eachregion To take advantage of the default highavailability install the cluster in a region with at leastthree availability zones To install a cluster in aregion with more than five availability zones youmust increase the EIP limit

IMPORTANT

To use the us-east-1 region youmust increase the EIP limit for youraccount

VirtualPrivateClouds(VPCs)

5 5 VPCs perregion

Each cluster creates its own VPC

Component

Default AWSlimit

Description

ElasticLoadBalancing(ELBNLB)

3 20 per region By default each cluster creates internal andexternal network load balancers for the master APIserver and a single classic elastic load balancer forthe router Deploying more Kubernetes Serviceobjects with type LoadBalancer will createadditional load balancers

NATGateways

5 5 per availabilityzone

The cluster deploys one NAT gateway in eachavailability zone

ElasticNetworkInterfaces(ENIs)

At least 12 350 per region The default installation creates 21 ENIs and an ENIfor each availability zone in your region Forexample the us-east-1 region contains sixavailability zones so a cluster that is deployed inthat zone uses 27 ENIs Review the AWS region mapto determine how many availability zones are in eachregion

Additional ENIs are created for additional machinesand elastic load balancers that are created bycluster usage and deployed workloads

VPCGateway

20 20 per account Each cluster creates a single VPC Gateway for S3access

S3 buckets 99 100 buckets peraccount

Because the installation process creates atemporary bucket and the registry component ineach cluster creates a bucket you can create only99 OpenShift Container Platform clusters per AWSaccount

SecurityGroups

250 2500 peraccount

Each cluster creates 10 distinct security groups

Component

Default AWSlimit

Description

113 Required AWS permissions

When you attach the AdministratorAccess policy to the IAM user that you create in Amazon WebServices (AWS) you grant that user all of the required permissions To deploy all components of anOpenShift Container Platform cluster the IAM user requires the following permissions

Example 11 Required EC2 permissions for installation

tagTagResources

tagUntagResources

ec2AllocateAddress

ec2AssociateAddress

ec2AuthorizeSecurityGroupEgress

ec2AuthorizeSecurityGroupIngress

ec2CopyImage

ec2CreateNetworkInterface

ec2AttachNetworkInterface

ec2CreateSecurityGroup

ec2CreateTags

ec2CreateVolume

ec2DeleteSecurityGroup

ec2DeleteSnapshot

ec2DeregisterImage

ec2DescribeAccountAttributes

ec2DescribeAddresses

ec2DescribeAvailabilityZones

ec2DescribeDhcpOptions

ec2DescribeImages

ec2DescribeInstanceAttribute

ec2DescribeInstanceCreditSpecifications

ec2DescribeInstances

ec2DescribeInternetGateways

ec2DescribeKeyPairs

ec2DescribeNatGateways

ec2DescribeNetworkAcls

ec2DescribeNetworkInterfaces

ec2DescribePrefixLists

ec2DescribeRegions

ec2DescribeRouteTables

ec2DescribeSecurityGroups

ec2DescribeSubnets

ec2DescribeTags

ec2DescribeVolumes

ec2DescribeVpcAttribute

ec2DescribeVpcClassicLink

ec2DescribeVpcClassicLinkDnsSupport

ec2DescribeVpcEndpoints

ec2DescribeVpcs

ec2GetEbsDefaultKmsKeyId

ec2ModifyInstanceAttribute

ec2ModifyNetworkInterfaceAttribute

ec2ReleaseAddress

ec2RevokeSecurityGroupEgress

ec2RevokeSecurityGroupIngress

ec2RunInstances

ec2TerminateInstances

Example 12 Required permissions for creating network resources during installation

ec2AssociateDhcpOptions

ec2AssociateRouteTable

ec2AttachInternetGateway

ec2CreateDhcpOptions

ec2CreateInternetGateway

ec2CreateNatGateway

ec2CreateRoute

ec2CreateRouteTable

ec2CreateSubnet

ec2CreateVpc

ec2CreateVpcEndpoint

ec2ModifySubnetAttribute

ec2ModifyVpcAttribute

If you use an existing VPC your account does not require these permissions forcreating network resources

Example 13 Required Elastic Load Balancing permissions for installation

elasticloadbalancingAddTags

elasticloadbalancingApplySecurityGroupsToLoadBalancer

elasticloadbalancingAttachLoadBalancerToSubnets

elasticloadbalancingConfigureHealthCheck

elasticloadbalancingCreateListener

elasticloadbalancingCreateLoadBalancer

elasticloadbalancingCreateLoadBalancerListeners

elasticloadbalancingCreateTargetGroup

elasticloadbalancingDeleteLoadBalancer

elasticloadbalancingDeregisterInstancesFromLoadBalancer

elasticloadbalancingDeregisterTargets

elasticloadbalancingDescribeInstanceHealth

elasticloadbalancingDescribeListeners

elasticloadbalancingDescribeLoadBalancerAttributes

elasticloadbalancingDescribeLoadBalancers

elasticloadbalancingDescribeTags

elasticloadbalancingDescribeTargetGroupAttributes

elasticloadbalancingDescribeTargetHealth

elasticloadbalancingModifyLoadBalancerAttributes

elasticloadbalancingModifyTargetGroup

elasticloadbalancingModifyTargetGroupAttributes

elasticloadbalancingRegisterInstancesWithLoadBalancer

elasticloadbalancingRegisterTargets

elasticloadbalancingSetLoadBalancerPoliciesOfListener

Example 14 Required IAM permissions for installation

iamAddRoleToInstanceProfile

iamCreateInstanceProfile

iamCreateRole

iamDeleteInstanceProfile

iamDeleteRole

iamDeleteRolePolicy

iamGetInstanceProfile

iamGetRole

iamGetRolePolicy

iamGetUser

iamListInstanceProfilesForRole

iamListRoles

iamListUsers

iamPassRole

iamPutRolePolicy

iamRemoveRoleFromInstanceProfile

iamSimulatePrincipalPolicy

iamTagRole

Example 15 Required Route 53 permissions for installation

route53ChangeResourceRecordSets

route53ChangeTagsForResource

route53CreateHostedZone

route53DeleteHostedZone

route53GetChange

route53GetHostedZone

route53ListHostedZones

route53ListHostedZonesByName

route53ListResourceRecordSets

route53ListTagsForResource

route53UpdateHostedZoneComment

Example 16 Required S3 permissions for installation

s3CreateBucket

s3DeleteBucket

s3GetAccelerateConfiguration

s3GetBucketAcl

s3GetBucketCors

s3GetBucketLocation

s3GetBucketLogging

s3GetBucketObjectLockConfiguration

s3GetBucketReplication

s3GetBucketRequestPayment

s3GetBucketTagging

s3GetBucketVersioning

s3GetBucketWebsite

s3GetEncryptionConfiguration

s3GetLifecycleConfiguration

s3GetReplicationConfiguration

s3ListBucket

s3PutBucketAcl

s3PutBucketTagging

s3PutEncryptionConfiguration

Example 17 S3 permissions that cluster Operators require

s3DeleteObject

s3GetObject

s3GetObjectAcl

s3GetObjectTagging

s3GetObjectVersion

s3PutObject

s3PutObjectAcl

s3PutObjectTagging

Example 18 Required permissions to delete base cluster resources

autoscalingDescribeAutoScalingGroups

ec2DeleteNetworkInterface

ec2DeleteVolume

elasticloadbalancingDeleteTargetGroup

elasticloadbalancingDescribeTargetGroups

iamDeleteAccessKey

iamDeleteUser

iamListAttachedRolePolicies

iamListInstanceProfiles

iamListRolePolicies

iamListUserPolicies

s3DeleteObject

s3ListBucketVersions

tagGetResources

Example 19 Required permissions to delete network resources

ec2DeleteDhcpOptions

ec2DeleteInternetGateway

ec2DeleteNatGateway

ec2DeleteRoute

ec2DeleteRouteTable

ec2DeleteSubnet

ec2DeleteVpc

ec2DeleteVpcEndpoints

ec2DetachInternetGateway

ec2DisassociateRouteTable

ec2ReplaceRouteTableAssociation

If you use an existing VPC your account does not require these permissions to deletenetwork resources

Example 110 Additional IAM and S3 permissions that are required to create manifests

iamCreateAccessKey

iamCreateUser

iamDeleteAccessKey

iamDeleteUser

iamDeleteUserPolicy

iamGetUserPolicy

iamListAccessKeys

iamPutUserPolicy

iamTagUser

iamGetUserPolicy

iamListAccessKeys

s3PutBucketPublicAccessBlock

s3GetBucketPublicAccessBlock

s3PutLifecycleConfiguration

s3HeadBucket

s3ListBucketMultipartUploads

s3AbortMultipartUpload

Example 111 Optional permission for quota checks for installation

servicequotasListAWSDefaultServiceQuotas

114 Creating an IAM user

Each Amazon Web Services (AWS) account contains a root user account that is based on the emailaddress you used to create the account This is a highly-privileged account and it is recommended touse it for only initial account and billing configuration creating an initial set of users and securing theaccount

Before you install OpenShift Container Platform create a secondary IAM administrative user As youcomplete the Creating an IAM User in Your AWS Account procedure in the AWS documentation set thefollowing options

Procedure

1 Specify the IAM user name and select Programmatic access

2 Attach the AdministratorAccess policy to ensure that the account has sufficient permission tocreate the cluster This policy provides the cluster with the ability to grant credentials to eachOpenShift Container Platform component The cluster grants the components only thecredentials that they require

While it is possible to create a policy that grants the all of the required AWSpermissions and attach it to the user this is not the preferred option The clusterwill not have the ability to grant additional credentials to individual componentsso the same credentials are used by all components

3 Optional Add metadata to the user by attaching tags

4 Confirm that the user name that you specified is granted the AdministratorAccess policy

5 Record the access key ID and secret access key values You must use these values when youconfigure your local machine to run the installation program

IMPORTANT

You cannot use a temporary session token that you generated while using amulti-factor authentication device to authenticate to AWS when you deploy acluster The cluster continues to use your current AWS credentials to create AWSresources for the entire life of the cluster so you must use key-based long-livedcredentials

Additional resources

See Manually creating IAM for AWS for steps to set the Cloud Credential Operator (CCO) tomanual mode prior to installation Use this mode in environments where the cloud identity andaccess management (IAM) APIs are not reachable or if you prefer not to store an administrator-level credential secret in the cluster kube-system project

115 Supported AWS regions

You can deploy an OpenShift Container Platform cluster to the following public regions

af-south-1 (Cape Town)

ap-east-1 (Hong Kong)

ap-northeast-1 (Tokyo)

ap-northeast-2 (Seoul)

ap-south-1 (Mumbai)

ap-southeast-1 (Singapore)

ap-southeast-2 (Sydney)

ca-central-1 (Central)

eu-central-1 (Frankfurt)

eu-north-1 (Stockholm)

eu-south-1 (Milan)

eu-west-1 (Ireland)

eu-west-2 (London)

eu-west-3 (Paris)

me-south-1 (Bahrain)

sa-east-1 (Satildeo Paulo)

us-east-1 (N Virginia)

us-east-2 (Ohio)

us-west-1 (N California)

us-west-2 (Oregon)

The following AWS GovCloud regions are supported

us-gov-west-1

us-gov-east-1

116 Next steps

Install an OpenShift Container Platform cluster

Quickly install a cluster with default options on installer-provisioned infrastructure

Install a cluster with cloud customizations on installer-provisioned infrastructure

Install a cluster with network customizations on installer-provisioned infrastructure

Installing a cluster on user-provisioned infrastructure in AWS by using CloudFormationtemplates

12 INSTALLING A CLUSTER ON AWS WITH CUSTOMIZATIONS

In OpenShift Container Platform version 46 you can install a customized cluster on infrastructure thatthe installation program provisions on Amazon Web Services (AWS) To customize the installation youmodify parameters in the install-configyaml file before you install the cluster

121 Prerequisites

Review details about the OpenShift Container Platform installation and update processes

Configure an AWS account to host the cluster

IMPORTANT

If you have an AWS profile stored on your computer it must not use a temporarysession token that you generated while using a multi-factor authenticationdevice The cluster continues to use your current AWS credentials to create AWSresources for the entire life of the cluster so you must use long-lived credentialsTo generate appropriate keys see Managing Access Keys for IAM Users in theAWS documentation You can supply the keys when you run the installationprogram

If you use a firewall you must configure it to allow the sites that your cluster requires access to

If you do not allow the system to manage identity and access management (IAM) then a clusteradministrator can manually create and maintain IAM credentials Manual mode can also be usedin environments where the cloud IAM APIs are not reachable

122 Internet and Telemetry access for OpenShift Container Platform

In OpenShift Container Platform 46 you require access to the Internet to install your cluster TheTelemetry service which runs by default to provide metrics about cluster health and the success ofupdates also requires Internet access If your cluster is connected to the Internet Telemetry runsautomatically and your cluster is registered to the Red Hat OpenShift Cluster Manager (OCM)

Once you confirm that your Red Hat OpenShift Cluster Manager inventory is correct either maintainedautomatically by Telemetry or manually using OCM use subscription watch to track your OpenShiftContainer Platform subscriptions at the account or multi-cluster level

You must have Internet access to

Access the Red Hat OpenShift Cluster Manager page to download the installation program andperform subscription management If the cluster has Internet access and you do not disableTelemetry that service automatically entitles your cluster

Access Quayio to obtain the packages that are required to install your cluster

Obtain the packages that are required to perform cluster updates

IMPORTANT

If your cluster cannot have direct Internet access you can perform a restricted networkinstallation on some types of infrastructure that you provision During that process youdownload the content that is required and use it to populate a mirror registry with thepackages that you need to install a cluster and generate the installation program Withsome installation types the environment that you install your cluster in will not requireInternet access Before you update the cluster you update the content of the mirrorregistry

See About remote health monitoring for more information about the Telemetry service

123 Generating an SSH private key and adding it to the agent

If you want to perform installation debugging or disaster recovery on your cluster you must provide anSSH key to both your ssh-agent and the installation program You can use this key to access thebootstrap machine in a public cluster to troubleshoot installation issues

In a production environment you require disaster recovery and debugging

You can use this key to SSH into the master nodes as the user core When you deploy the cluster thekey is added to the core userrsquos ~sshauthorized_keys list

You must use a local key not one that you configured with platform-specific approachessuch as AWS key pairs

Procedure

1 If you do not have an SSH key that is configured for password-less authentication on yourcomputer create one For example on a computer that uses a Linux operating system run thefollowing command

Specify the path and file name such as ~sshid_rsa of the new SSH key

Running this command generates an SSH key that does not require a password in the locationthat you specified

2 Start the ssh-agent process as a background task

Example output

$ ssh-keygen -t ed25519 -N -f ltpathgtltfile_namegt 1

$ eval $(ssh-agent -s)

3 Add your SSH private key to the ssh-agent

Example output

Specify the path and file name for your SSH private key such as ~sshid_rsa

Next steps

When you install OpenShift Container Platform provide the SSH public key to the installationprogram

124 Obtaining the installation program

Before you install OpenShift Container Platform download the installation file on a local computer

Prerequisites

You have a computer that runs Linux or macOS with 500 MB of local disk space

Procedure

1 Access the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site If youhave a Red Hat account log in with your credentials If you do not create an account

2 Select your infrastructure provider

3 Navigate to the page for your installation type download the installation program for youroperating system and place the file in the directory where you will store the installationconfiguration files

IMPORTANT

The installation program creates several files on the computer that you use toinstall your cluster You must keep the installation program and the files that theinstallation program creates after you finish installing the cluster Both files arerequired to delete the cluster

IMPORTANT

Deleting the files created by the installation program does not remove yourcluster even if the cluster failed during installation To remove your clustercomplete the OpenShift Container Platform uninstallation procedures for yourspecific cloud provider

4 Extract the installation program For example on a computer that uses a Linux operating

Agent pid 31874

$ ssh-add ltpathgtltfile_namegt 1

Identity added homeltyougtltpathgtltfile_namegt (ltcomputer_namegt)

4 Extract the installation program For example on a computer that uses a Linux operatingsystem run the following command

5 From the Pull Secret page on the Red Hat OpenShift Cluster Manager site download yourinstallation pull secret as a txt file This pull secret allows you to authenticate with the servicesthat are provided by the included authorities including Quayio which serves the containerimages for OpenShift Container Platform components

125 Creating the installation configuration file

You can customize the OpenShift Container Platform cluster you install on Amazon Web Services(AWS)

Prerequisites

Obtain the OpenShift Container Platform installation program and the pull secret for yourcluster

Procedure

1 Create the install-configyaml file

a Change to the directory that contains the installation program and run the followingcommand

For ltinstallation_directorygt specify the directory name to store the files that theinstallation program creates

IMPORTANT

Specify an empty directory Some installation assets like bootstrap X509certificates have short expiration intervals so you must not reuse aninstallation directory If you want to reuse individual files from another clusterinstallation you can copy them into your directory However the file namesfor the installation assets might change between releases Use caution whencopying installation files from an earlier OpenShift Container Platformversion

b At the prompts provide the configuration details for your cloud

i Optional Select an SSH key to use to access your cluster machines

For production OpenShift Container Platform clusters on which you wantto perform installation debugging or disaster recovery specify an SSHkey that your ssh-agent process uses

ii Select AWS as the platform to target

$ tar xvf openshift-install-linuxtargz

$ openshift-install create install-config --dir=ltinstallation_directorygt 1

iii If you do not have an Amazon Web Services (AWS) profile stored on your computerenter the AWS access key ID and secret access key for the user that you configured torun the installation program

iv Select the AWS region to deploy the cluster to

v Select the base domain for the Route 53 service that you configured for your cluster

vi Enter a descriptive name for your cluster

vii Paste the pull secret that you obtained from the Pull Secret page on the Red HatOpenShift Cluster Manager site

2 Modify the install-configyaml file You can find more information about the availableparameters in the Installation configuration parameters section

3 Back up the install-configyaml file so that you can use it to install multiple clusters

IMPORTANT

The install-configyaml file is consumed during the installation process If youwant to reuse the file you must back it up now

1251 Installation configuration parameters

Before you deploy an OpenShift Container Platform cluster you provide parameter values to describeyour account on the cloud platform that hosts your cluster and optionally customize your clusterrsquosplatform When you create the install-configyaml installation configuration file you provide values forthe required parameters through the command line If you customize your cluster you can modify the install-configyaml file to provide more details about the platform

After installation you cannot modify these parameters in the install-configyaml file

IMPORTANT

The openshift-install command does not validate field names for parameters If anincorrect name is specified the related file or object is not created and no error isreported Ensure that the field names for any parameters that are specified are correct

Table 11 Required parameters

Parameter Description Values

apiVersion The API version for the install-configyamlcontent The current version isv1 The installer may alsosupport older API versions

String

baseDomain The base domain of yourcloud provider The basedomain is used to createroutes to your OpenShiftContainer Platform clustercomponents The full DNSname for your cluster is acombination of the baseDomain and metadataname parametervalues that uses the ltmetadatanamegtltbaseDomaingt format

A fully-qualified domain or subdomain name such as examplecom

metadata Kubernetes resource ObjectMeta from which onlythe name parameter isconsumed

Object

metadataname The name of the cluster DNSrecords for the cluster are allsubdomains of metadatanamebaseDomain

String of lowercase letters hyphens (-) and periods() such as dev

platform The configuration for thespecific platform upon whichto perform the installation aws baremetal azure openstack ovirt vsphereFor additional informationabout platformltplatformgtparameters consult thefollowing table for yourspecific platform

Object

pullSecret Get this pull secret fromhttpscloudredhatcomopenshiftinstallpull-secret toauthenticate downloadingcontainer images forOpenShift Container Platformcomponents from servicessuch as Quayio

auths cloudopenshiftcom authb3Blb= emailyouexamplecom quayio authb3Blb= emailyouexamplecom

Table 12 Optional parameters

additionalTrustBundle

A PEM-encoded X509 certificatebundle that is added to the nodestrusted certificate store This trustbundle may also be used when a proxyhas been configured

String

compute The configuration for the machinesthat comprise the compute nodes

Array of machine-pool objects Fordetails see the following Machine-pool table

computearchitecture

Determines the instruction setarchitecture of the machines in thepool Currently heteregeneousclusters are not supported so all poolsmust specify the same architectureValid values are amd64 (the default)

String

computehyperthreading

Whether to enable or disablesimultaneous multithreading or hyperthreading on computemachines By default simultaneousmultithreading is enabled to increasethe performance of your machinescores

IMPORTANT

If you disablesimultaneousmultithreading ensurethat your capacityplanning accounts forthe dramaticallydecreased machineperformance

Enabled or Disabled

computename Required if you use compute Thename of the machine pool

worker

computeplatform Required if you use compute Use thisparameter to specify the cloudprovider to host the worker machinesThis parameter value must match the controlPlaneplatform parametervalue

aws azure gcp openstack ovirt vsphere or

computereplicas The number of compute machineswhich are also known as workermachines to provision

A positive integer greater than or equalto 2 The default value is 3

controlPlane The configuration for the machinesthat comprise the control plane

Array of MachinePool objects Fordetails see the following Machine-pool table

controlPlanearchitecture

Determines the instruction setarchitecture of the machines in thepool Currently heterogeneousclusters are not supported so all poolsmust specify the same architectureValid values are amd64 (the default)

String

controlPlanehyperthreading

Whether to enable or disablesimultaneous multithreading or hyperthreading on control planemachines By default simultaneousmultithreading is enabled to increasethe performance of your machinescores

IMPORTANT

Enabled or Disabled

controlPlanename Required if you use controlPlaneThe name of the machine pool

master

controlPlaneplatform

Required if you use controlPlaneUse this parameter to specify the cloudprovider that hosts the control planemachines This parameter value mustmatch the computeplatformparameter value

controlPlanereplicas

The number of control plane machinesto provision

The only supported value is 3 which isthe default value

credentialsMode The Cloud Credential Operator (CCO)mode If no mode is specified theCCO dynamically tries to determinethe capabilities of the providedcredentials with a preference for mintmode on the platforms where multiplemodes are supported

Not all CCO modesare supported for allcloud providers Formore information onCCO modes see theCloud CredentialOperator entry in theRed Hat Operatorsreference content

Mint Passthrough Manual or anempty string ()

fips Enable or disable FIPS mode Thedefault is false (disabled) If FIPSmode is enabled the Red HatEnterprise Linux CoreOS (RHCOS)machines that OpenShift ContainerPlatform runs on bypass the defaultKubernetes cryptography suite and usethe cryptography modules that areprovided with RHCOS instead

false or true

imageContentSources

Sources and repositories for therelease-image content

Array of objects Includes a sourceand optionally mirrors as describedin the following rows of this table

imageContentSourcessource

Required if you use imageContentSources Specify therepository that users refer to forexample in image pull specifications

String

imageContentSourcesmirrors

Specify one or more repositories thatmay also contain the same images

Array of strings

networking The configuration for the pod networkprovider in the cluster

Object

networkingclusterNetwork

The IP address pools for pods Thedefault is 101280014 with a hostprefix of 23

Array of objects

networkingclusterNetworkcidr

Required if you use networkingclusterNetwork The IPblock address pool

IP network IP networks arerepresented as strings using ClasslessInter-Domain Routing (CIDR) notationwith a traditional IP address or networknumber followed by the forward slash() character followed by a decimalvalue between 0 and 32 that describesthe number of significant bits Forexample 1000016 represents IPaddresses 10000 through 100255255

networkingclusterNetworkhostPrefix

Required if you use networkingclusterNetwork Theprefix size to allocate to each nodefrom the CIDR For example 24 wouldallocate 2^8=256 addresses to eachnode

Integer

networkingmachineNetwork

The IP address pools for machines Array of objects

networkingmachineNetworkcidr

Required if you use networkingmachineNetwork TheIP block address pool The default is 1000016 for all platforms otherthan libvirt For libvirt the default is 192168126024

networkingnetworkType

The type of network to install Thedefault is OpenShiftSDN

String

networkingserviceNetwork

The IP address pools for services Thedefault is 172300016

Array of IP networks IP networks arerepresented as strings using ClasslessInter-Domain Routing (CIDR) notationwith a traditional IP address or networknumber followed by the forward slash() character followed by a decimalvalue between 0 and 32 that describesthe number of significant bits Forexample 1000016 represents IPaddresses 10000 through 100255255

publish How to publish or expose the user-facing endpoints of your cluster suchas the Kubernetes API OpenShiftroutes

Internal or External To deploy aprivate cluster which cannot beaccessed from the internet set publish to Internal The defaultvalue is External

sshKey The SSH key or keys to authenticateaccess your cluster machines

For productionOpenShift ContainerPlatform clusters onwhich you want toperform installationdebugging or disasterrecovery specify anSSH key that your ssh-agent processuses

One or more keys For example

sshKey ltkey1gt ltkey2gt ltkey3gt

Table 13 Optional AWS parameters

computeplatformawsamiID

The AWS AMI used to bootcompute machines for thecluster This is required forregions that require a customRHCOS AMI

Any published or custom RHCOS AMI that belongsto the set AWS region

computeplatformawsrootVolumeiops

The InputOutput OperationsPer Second (IOPS) that isreserved for the root volume

Integer for example 4000

computeplatformawsrootVolumesize

The size in GiB of the rootvolume

computeplatformawsrootVolumetype

The instance type of the rootvolume

Valid AWS EBS instance type such as io1

computeplatformawstype

The EC2 instance type for thecompute machines

Valid AWS instance type such as c59xlarge

computeplatformawszones

The availability zones wherethe installation programcreates machines for thecompute machine pool If youprovide your own VPC youmust provide a subnet in thatavailability zone

A list of valid AWS availability zones such as us-east-1c in a YAML sequence

computeawsregion

The AWS region that theinstallation program createscompute resources in

Any valid AWS region such as us-east-1

controlPlaneplatformawsamiID

The AWS AMI used to bootcontrol plane machines for thecluster This is required forregions that require a customRHCOS AMI

controlPlaneplatformawstype

The EC2 instance type for thecontrol plane machines

controlPlaneplatformawszones

The availability zones wherethe installation programcreates machines for thecontrol plane machine pool

controlPlaneawsregion

The AWS region that theinstallation program createscontrol plane resources in

Valid AWS region such as us-east-1

platformawsamiID

The AWS AMI used to boot allmachines for the cluster Ifset the AMI must belong tothe same region as thecluster This is required forregions that require a customRHCOS AMI

platformawsserviceEndpointsname

The AWS service endpointname Custom endpoints areonly required for cases wherealternative AWS endpointslike FIPS must be usedCustom API endpoints can bespecified for EC2 S3 IAMElastic Load BalancingTagging Route 53 and STSAWS services

Valid AWS service endpoint name

platformawsserviceEndpointsurl

The AWS service endpointURL The URL must use the https protocol and the hostmust trust the certificate

Valid AWS service endpoint URL

platformawsuserTags

A map of keys and values thatthe installation program addsas tags to all resources that itcreates

Any valid YAML map such as key value pairs in the ltkeygt ltvaluegt format For more informationabout AWS tags see Tagging Your Amazon EC2Resources in the AWS documentation

platformawssubnets

If you provide the VPCinstead of allowing theinstallation program to createthe VPC for you specify thesubnet for the cluster to useThe subnet must be part ofthe same machineNetwork[]cidrranges that you specify For astandard cluster specify apublic and a private subnet foreach availability zone For aprivate cluster specify aprivate subnet for eachavailability zone

Valid subnet IDs

1252 Sample customized install-configyaml file for AWS

You can customize the install-configyaml file to specify more details about your OpenShift ContainerPlatform clusterrsquos platform or modify the values of the required parameters

IMPORTANT

This sample YAML file is provided for reference only You must obtain your install-configyaml file by using the installation program and modify it

apiVersion v1baseDomain examplecom 1credentialsMode Mint 2controlPlane 3 4 hyperthreading Enabled 5 name master platform aws zones - us-west-2a - us-west-2b rootVolume iops 4000 size 500 type io1 6 type m5xlarge replicas 3compute 7- hyperthreading Enabled 8 name worker platform aws rootVolume iops 2000 size 500 type io1 9 type c54xlarge zones - us-west-2c replicas 3metadata name test-cluster 10networking clusterNetwork - cidr 101280014 hostPrefix 23 machineNetwork - cidr 1000016 networkType OpenShiftSDN serviceNetwork - 172300016platform aws region us-west-2 11 userTags adminContact jdoe costCenter 7536 amiID ami-96c6f8f7 12 serviceEndpoints 13

1 10 11 14

Required The installation program prompts you for this value

Optional Add this parameter to force the Cloud Credential Operator (CCO) to use the specifiedmode instead of having the CCO dynamically try to determine the capabilities of the credentialsFor details about CCO modes see the Cloud Credential Operator entry in the Red Hat Operatorsreference content

If you do not provide these parameters and values the installation program provides the defaultvalue

The controlPlane section is a single mapping but the compute section is a sequence of mappingsTo meet the requirements of the different data structures the first line of the compute sectionmust begin with a hyphen - and the first line of the controlPlane section must not Although bothsections currently define a single machine pool it is possible that future versions of OpenShiftContainer Platform will support defining multiple compute pools during installation Only onecontrol plane pool is used

Whether to enable or disable simultaneous multithreading or hyperthreading By defaultsimultaneous multithreading is enabled to increase the performance of your machines cores Youcan disable it by setting the parameter value to Disabled If you disable simultaneousmultithreading in some cluster machines you must disable it in all cluster machines

IMPORTANT

If you disable simultaneous multithreading ensure that your capacity planningaccounts for the dramatically decreased machine performance Use larger instancetypes such as m42xlarge or m52xlarge for your machines if you disablesimultaneous multithreading

To configure faster storage for etcd especially for larger clusters set the storage type as io1 andset iops to 2000

The ID of the AMI used to boot machines for the cluster If set the AMI must belong to the sameregion as the cluster

The AWS service endpoints Custom endpoints are required when installing to an unknown AWSregion The endpoint URL must use the https protocol and the host must trust the certificate

Whether to enable or disable FIPS mode By default FIPS mode is not enabled If FIPS mode isenabled the Red Hat Enterprise Linux CoreOS (RHCOS) machines that OpenShift ContainerPlatform runs on bypass the default Kubernetes cryptography suite and use the cryptographymodules that are provided with RHCOS instead

You can optionally provide the sshKey value that you use to access the machines in your cluster

- name ec2 url httpsvpce-idec2us-west-2vpceamazonawscompullSecret auths 14fips false 15sshKey ssh-ed25519 AAAA 16

For production OpenShift Container Platform clusters on which you want to performinstallation debugging or disaster recovery specify an SSH key that your ssh-agentprocess uses

126 Deploying the cluster

You can install OpenShift Container Platform on a compatible cloud platform

IMPORTANT

You can run the create cluster command of the installation program only once duringinitial installation

Prerequisites

Configure an account with the cloud platform that hosts your cluster

Procedure

1 Change to the directory that contains the installation program and initialize the clusterdeployment

For ltinstallation_directorygt specify the location of your customized install-configyaml file

To view different installation details specify warn debug or error instead of info

If the cloud provider account that you configured on your host does not havesufficient permissions to deploy the cluster the installation process stops andthe missing permissions are displayed

When the cluster deployment completes directions for accessing your cluster including a link toits web console and credentials for the kubeadmin user display in your terminal

Example output

$ openshift-install create cluster --dir=ltinstallation_directorygt 1 --log-level=info 2

INFO Install completeINFO To access the cluster as the systemadmin user when using oc run export KUBECONFIG=homemyuserinstall_dirauthkubeconfigINFO Access the OpenShift web-console here httpsconsole-openshift-

The cluster access and credential information also outputs to ltinstallation_directorygtopenshift_installlog when an installation succeeds

IMPORTANT

The Ignition config files that the installation program generates containcertificates that expire after 24 hours which are then renewed at that time If thecluster is shut down before renewing the certificates and the cluster is laterrestarted after the 24 hours have elapsed the cluster automatically recovers theexpired certificates The exception is that you must manually approve thepending node-bootstrapper certificate signing requests (CSRs) to recoverkubelet certificates See the documentation for Recovering from expired controlplane certificates for more information

IMPORTANT

You must not delete the installation program or the files that the installationprogram creates Both are required to delete the cluster

2 Optional Remove or disable the AdministratorAccess policy from the IAM account that youused to install the cluster

The elevated permissions provided by the AdministratorAccess policy arerequired only during installation

127 Installing the OpenShift CLI by downloading the binary

You can install the OpenShift CLI (oc) in order to interact with OpenShift Container Platform from acommand-line interface You can install oc on Linux Windows or macOS

IMPORTANT

If you installed an earlier version of oc you cannot use it to complete all of the commandsin OpenShift Container Platform 46 Download and install the new version of oc

1271 Installing the OpenShift CLI on Linux

You can install the OpenShift CLI (oc) binary on Linux by using the following procedure

Procedure

1 Navigate to the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site

2 Select your infrastructure provider and if applicable your installation type

consoleappsmyclusterexamplecomINFO Login to the console with user kubeadmin and password 4vYBz-Ee6gm-ymBZj-Wt5ALINFO Time elapsed 36m22s

3 In the Command-line interface section select Linux from the drop-down menu and clickDownload command-line tools

4 Unpack the archive

5 Place the oc binary in a directory that is on your PATHTo check your PATH execute the following command

After you install the CLI it is available using the oc command

1272 Installing the OpenShift CLI on Windows

You can install the OpenShift CLI (oc) binary on Windows by using the following procedure

Procedure

3 In the Command-line interface section select Windows from the drop-down menu and clickDownload command-line tools

4 Unzip the archive with a ZIP program

5 Move the oc binary to a directory that is on your PATHTo check your PATH open the command prompt and execute the following command

1273 Installing the OpenShift CLI on macOS

You can install the OpenShift CLI (oc) binary on macOS by using the following procedure

Procedure

3 In the Command-line interface section select MacOS from the drop-down menu and clickDownload command-line tools

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

4 Unpack and unzip the archive

5 Move the oc binary to a directory on your PATHTo check your PATH open a terminal and execute the following command

128 Logging in to the cluster by using the CLI

You can log in to your cluster as a default system user by exporting the cluster kubeconfig file The kubeconfig file contains information about the cluster that is used by the CLI to connect a client to thecorrect cluster and API server The file is specific to a cluster and is created during OpenShift ContainerPlatform installation

Prerequisites

You deployed an OpenShift Container Platform cluster

You installed the oc CLI

Procedure

1 Export the kubeadmin credentials

For ltinstallation_directorygt specify the path to the directory that you stored theinstallation files in

2 Verify you can run oc commands successfully using the exported configuration

Example output

129 Logging in to the cluster by using the web console

The kubeadmin user exists by default after an OpenShift Container Platform installation You can loginto your cluster as the kubeadmin user by using the OpenShift Container Platform web console

Prerequisites

You have access to the installation host

You completed a cluster installation and all cluster Operators are available

$ echo $PATH

$ oc ltcommandgt

$ export KUBECONFIG=ltinstallation_directorygtauthkubeconfig 1

$ oc whoami

systemadmin

Procedure

1 Obtain the password for the kubeadmin user from the kubeadmin-password file on theinstallation host

Alternatively you can obtain the kubeadmin password from the ltinstallation_directorygtopenshift_installlog log file on the installation host

2 List the OpenShift Container Platform web console route

Alternatively you can obtain the OpenShift Container Platform route from the ltinstallation_directorygtopenshift_installlog log file on the installation host

Example output

3 Navigate to the route detailed in the output of the preceding command in a web browser andlog in as the kubeadmin user

See Accessing the web console for more details about accessing and understanding theOpenShift Container Platform web console

1210 Next steps

Validating an installation

Customize your cluster

If necessary you can opt out of remote health reporting

13 INSTALLING A CLUSTER ON AWS WITH NETWORKCUSTOMIZATIONS

In OpenShift Container Platform version 46 you can install a cluster on Amazon Web Services (AWS)with customized network configuration options By customizing your network configuration your clustercan coexist with existing IP address allocations in your environment and integrate with existing MTU andVXLAN configurations

You must set most of the network configuration parameters during installation and you can modify only kubeProxy configuration parameters in a running cluster

$ cat ltinstallation_directorygtauthkubeadmin-password

$ oc get routes -n openshift-console | grep console-openshift

console console-openshift-consoleappsltcluster_namegtltbase_domaingt console https reencryptRedirect None

131 Prerequisites

IMPORTANT

If you have an AWS profile stored on your computer it must not use a temporarysession token that you generated while using a multi-factor authenticationdevice The cluster continues to use your current AWS credentials to create AWSresources for the entire life of the cluster so you must use key-based long-livedcredentials To generate appropriate keys see Managing Access Keys for IAMUsers in the AWS documentation You can supply the keys when you run theinstallation program

IMPORTANT

Procedure

Example output

Agent pid 31874

Next steps

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

1352 Network configuration parameters

You can modify your cluster network configuration parameters in the install-configyaml configurationfile The following table describes the parameters

You cannot modify these parameters in the install-configyaml file after installation

Table 17 Required network parameters

Parameter Description Value

The default Container Network Interface (CNI)network provider plug-in to deploy

Either OpenShiftSDN or OVNKubernetes Thedefault value is OpenShiftSDN

networkingclusterNetwork[]cidr

A block of IP addresses from which pod IP addressesare allocated The OpenShiftSDN network plug-insupports multiple cluster networks The addressblocks for multiple cluster networks must not overlapSelect address pools large enough to fit youranticipated workload

An IP address allocation inCIDR format The defaultvalue is 101280014

networkingclusterNetwork[]hostPrefix

The subnet prefix length to assign to each individualnode For example if hostPrefix is set to 23 theneach node is assigned a 23 subnet out of the given cidr allowing for 510 (2^(32 - 23) - 2) pod IPaddresses

A subnet prefix The defaultvalue is 23

networkingserviceNetwork[]

A block of IP addresses for services OpenShiftSDNallows only one serviceNetwork block The addressblock must not overlap with any other network block

networkingmachineNetwork[]cidr

A block of IP addresses assigned to nodes created bythe OpenShift Container Platform installationprogram while installing the cluster The addressblock must not overlap with any other network blockMultiple CIDR ranges may be specified

IMPORTANT

apiVersion v1baseDomain examplecom 1credentialsMode Mint 2controlPlane 3 4 hyperthreading Enabled 5 name master platform aws zones - us-west-2a - us-west-2b rootVolume iops 4000 size 500 type io1 6

1 10 12 15

3 7 11

If you do not provide these parameters and values the installation program provides thedefault value

The controlPlane section is a single mapping but the compute section is a sequence of mappingsTo meet the requirements of the different data structures the first line of the compute sectionmust begin with a hyphen - and the first line of the controlPlane section must not Although bothsections currently define a single machine pool it is possible that future versions of OpenShiftContainer Platform will support defining multiple compute pools during installation Only one

type m5xlarge replicas 3compute 7- hyperthreading Enabled 8 name worker platform aws rootVolume iops 2000 size 500 type io1 9 type c54xlarge zones - us-west-2c replicas 3metadata name test-cluster 10networking 11 clusterNetwork - cidr 101280014 hostPrefix 23 machineNetwork - cidr 1000016 networkType OpenShiftSDN serviceNetwork - 172300016platform aws region us-west-2 12 userTags adminContact jdoe costCenter 7536 amiID ami-96c6f8f7 13 serviceEndpoints 14 - name ec2 url httpsvpce-idec2us-west-2vpceamazonawscompullSecret auths 15fips false 16sshKey ssh-ed25519 AAAA 17

control plane pool is used

IMPORTANT

136 Modifying advanced network configuration parameters

You can modify the advanced network configuration parameters only before you install the clusterAdvanced configuration customization lets you integrate your cluster into your existing networkenvironment by specifying an MTU or VXLAN port by allowing customization of kube-proxy settingsand by specifying a different mode for the openshiftSDNConfig parameter

IMPORTANT

Modifying the OpenShift Container Platform manifest files directly is not supported

Prerequisites

Create the install-configyaml file and complete any modifications to it

Procedure

1 Change to the directory that contains the installation program and create the manifests

For ltinstallation_directorygt specify the name of the directory that contains the install-configyaml file for your cluster

2 Create a file that is named cluster-network-03-configyml in the ltinstallation_directorygtmanifests directory

For ltinstallation_directorygt specify the directory name that contains the manifestsdirectory for your cluster

After creating the file several network configuration files are in the manifests directory asshown

Example output

3 Open the cluster-network-03-configyml file in an editor and enter a CR that describes theOperator configuration you want

The parameters for the spec parameter are only an example Specify your configurationfor the Cluster Network Operator in the CR

The CNO provides default values for the parameters in the CR so you must specify only theparameters that you want to change

$ openshift-install create manifests --dir=ltinstallation_directorygt 1

$ touch ltinstallation_directorygtmanifestscluster-network-03-configyml 1

$ ls ltinstallation_directorygtmanifestscluster-network-

cluster-network-01-crdymlcluster-network-02-configymlcluster-network-03-configyml

apiVersion operatoropenshiftiov1kind Networkmetadata name clusterspec 1 clusterNetwork - cidr 101280014 hostPrefix 23 serviceNetwork - 172300016 defaultNetwork type OpenShiftSDN openshiftSDNConfig mode NetworkPolicy mtu 1450 vxlanPort 4789

4 Save the cluster-network-03-configyml file and quit the text editor

5 Optional Back up the manifestscluster-network-03-configyml file The installation programdeletes the manifests directory when creating the cluster

For more information on using a Network Load Balancer (NLB) on AWS see ConfiguringIngress cluster traffic on AWS using a Network Load Balancer

137 Configuring an Ingress Controller Network Load Balancer on a new AWScluster

You can create an Ingress Controller backed by an AWS Network Load Balancer (NLB) on a new cluster

Prerequisites

Procedure

Create an Ingress Controller backed by an AWS NLB on a new cluster

2 Create a file that is named cluster-ingress-default-ingresscontrolleryaml in the ltinstallation_directorygtmanifests directory

Example output

3 Open the cluster-ingress-default-ingresscontrolleryaml file in an editor and enter a CR thatdescribes the Operator configuration you want

$ touch ltinstallation_directorygtmanifestscluster-ingress-default-ingresscontrolleryaml 1

$ ls ltinstallation_directorygtmanifestscluster-ingress-default-ingresscontrolleryaml

cluster-ingress-default-ingresscontrolleryaml

apiVersion operatoropenshiftiov1kind IngressController

4 Save the cluster-ingress-default-ingresscontrolleryaml file and quit the text editor

5 Optional Back up the manifestscluster-ingress-default-ingresscontrolleryaml file Theinstallation program deletes the manifests directory when creating the cluster

138 Cluster Network Operator configuration

The configuration for the cluster network is specified as part of the Cluster Network Operator (CNO)configuration and stored in a CR object that is named cluster The CR specifies the parameters for the Network API in the operatoropenshiftio API group

You can specify the cluster network configuration for your OpenShift Container Platform cluster bysetting the parameter values for the defaultNetwork parameter in the CNO CR The following CRdisplays the default configuration for the CNO and explains both the parameters you can configure andthe valid parameter values

Cluster Network Operator CR

Specified in the install-configyaml file

Configures the default Container Network Interface (CNI) network provider for the clusternetwork

metadata creationTimestamp null name default namespace openshift-ingress-operatorspec endpointPublishingStrategy loadBalancer scope External providerParameters type AWS aws type NLB type LoadBalancerService

apiVersion operatoropenshiftiov1kind Networkmetadata name clusterspec clusterNetwork 1 - cidr 101280014 hostPrefix 23 serviceNetwork 2 - 172300016 defaultNetwork 3 kubeProxyConfig 4 iptablesSyncPeriod 30s 5 proxyArguments iptables-min-sync-period 6 - 0s

The parameters for this object specify the kube-proxy configuration If you do not specify theparameter values the Cluster Network Operator applies the displayed default parameter values If

The refresh period for iptables rules The default value is 30s Valid suffixes include s m and hand are described in the Go time package documentation

Because of performance improvements introduced in OpenShift Container Platform43 and greater adjusting the iptablesSyncPeriod parameter is no longernecessary

The minimum duration before refreshing iptables rules This parameter ensures that the refreshdoes not happen too frequently Valid suffixes include s m and h and are described in the Go timepackage

1381 Configuration parameters for the OpenShift SDN default CNI network provider

The following YAML object describes the configuration parameters for the OpenShift SDN defaultContainer Network Interface (CNI) network provider

Specify only if you want to override part of the OpenShift SDN configuration

Configures the network isolation mode for OpenShift SDN The allowed values are Multitenant Subnet or NetworkPolicy The default value is NetworkPolicy

The maximum transmission unit (MTU) for the VXLAN overlay network This is detectedautomatically based on the MTU of the primary network interface You do not normally need tooverride the detected MTU

If the auto-detected value is not what you expected it to be confirm that the MTU on the primarynetwork interface on your nodes is correct You cannot use this option to change the MTU value ofthe primary network interface on the nodes

If your cluster requires different MTU values for different nodes you must set this value to 50 lessthan the lowest MTU value in your cluster For example if some nodes in your cluster have an MTUof 9001 and some have an MTU of 1500 you must set this value to 1450

The port to use for all VXLAN packets The default value is 4789 If you are running in a virtualizedenvironment with existing nodes that are part of another VXLAN network then you might berequired to change this For example when running an OpenShift SDN overlay on top of VMwareNSX-T you must select an alternate port for VXLAN since both SDNs use the same defaultVXLAN port number

On Amazon Web Services (AWS) you can select an alternate port for the VXLAN between port

defaultNetwork type OpenShiftSDN 1 openshiftSDNConfig 2 mode NetworkPolicy 3 mtu 1450 4 vxlanPort 4789 5

On Amazon Web Services (AWS) you can select an alternate port for the VXLAN between port 9000 and port 9999

1382 Configuration parameters for the OVN-Kubernetes default CNI network provider

The following YAML object describes the configuration parameters for the OVN-Kubernetes defaultCNI network provider

Specify only if you want to override part of the OVN-Kubernetes configuration

The maximum transmission unit (MTU) for the Geneve (Generic Network VirtualizationEncapsulation) overlay network This is detected automatically based on the MTU of the primarynetwork interface You do not normally need to override the detected MTU

The UDP port for the Geneve overlay network

1383 Cluster Network Operator example configuration

A complete CR object for the CNO is displayed in the following example

Cluster Network Operator example CR

defaultNetwork type OVNKubernetes 1 ovnKubernetesConfig 2 mtu 1400 3 genevePort 6081 4

apiVersion operatoropenshiftiov1kind Networkmetadata name clusterspec clusterNetwork - cidr 101280014 hostPrefix 23 serviceNetwork - 172300016 defaultNetwork type OpenShiftSDN openshiftSDNConfig mode NetworkPolicy mtu 1450

139 Configuring hybrid networking with OVN-Kubernetes

You can configure your cluster to use hybrid networking with OVN-Kubernetes This allows a hybridcluster that supports different node networking configurations For example this is necessary to runboth Linux and Windows nodes in a cluster

IMPORTANT

You must configure hybrid networking with OVN-Kubernetes during the installation ofyour cluster You cannot switch to hybrid networking after the installation process

Prerequisites

You defined OVNKubernetes for the networkingnetworkType parameter in the install-configyaml file See the installation documentation for configuring OpenShift ContainerPlatform network customizations on your chosen cloud provider for more information

Procedure

1 Create the manifests from the directory that contains the installation program

Example output

vxlanPort 4789 kubeProxyConfig iptablesSyncPeriod 30s proxyArguments iptables-min-sync-period - 0s

$ ls -1 ltinstallation_directorygtmanifestscluster-network-

3 Open the cluster-network-03-configyml file and configure OVN-Kubernetes with hybridnetworking For example

The parameters for the spec parameter are only an example Specify your configurationfor the Cluster Network Operator in the custom resource

Specify the CIDR configuration used when adding nodes

Specify OVNKubernetes as the Container Network Interface (CNI) cluster networkprovider

Specify the CIDR configuration used for nodes on the additional overlay network The hybridClusterNetwork CIDR cannot overlap with the clusterNetwork CIDR

4 Optional Back up the ltinstallation_directorygtmanifestscluster-network-03-configyml fileThe installation program deletes the manifests directory when creating the cluster

For more information on using Linux and Windows nodes in the same cluster seeUnderstanding Windows container workloads

IMPORTANT

apiVersion operatoropenshiftiov1kind Networkmetadata creationTimestamp null name clusterspec 1 clusterNetwork 2 - cidr 101280014 hostPrefix 23 externalIP policy serviceNetwork - 172300016 defaultNetwork type OVNKubernetes 3 ovnKubernetesConfig hybridOverlayConfig hybridClusterNetwork 4 - cidr 101320014 hostPrefix 23status

Prerequisites

Procedure

Example output

IMPORTANT

INFO Install completeINFO To access the cluster as the systemadmin user when using oc run export KUBECONFIG=homemyuserinstall_dirauthkubeconfigINFO Access the OpenShift web-console here httpsconsole-openshift-consoleappsmyclusterexamplecomINFO Login to the console with user kubeadmin and password 4vYBz-Ee6gm-ymBZj-Wt5ALINFO Time elapsed 36m22s

IMPORTANT

Procedure

$ tar xvzf ltfilegt

Procedure

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

$ echo $PATH

Prerequisites

Procedure

Example output

Prerequisites

Procedure

$ oc ltcommandgt

$ oc whoami

systemadmin

Example output

1314 Next steps

14 INSTALLING A CLUSTER ON AWS INTO AN EXISTING VPC

In OpenShift Container Platform version 46 you can install a cluster into an existing Amazon VirtualPrivate Cloud (VPC) on Amazon Web Services (AWS) The installation program provisions the rest ofthe required infrastructure which you can further customize To customize the installation you modifyparameters in the install-configyaml file before you install the cluster

141 Prerequisites

IMPORTANT

142 About using a custom VPC

In OpenShift Container Platform 46 you can deploy a cluster into existing subnets in an existingAmazon Virtual Private Cloud (VPC) in Amazon Web Services (AWS) By deploying OpenShiftContainer Platform into an existing AWS VPC you might be able to avoid limit constraints in newaccounts or more easily abide by the operational constraints that your companyrsquos guidelines set If youcannot obtain the infrastructure creation permissions that are required to create the VPC yourself usethis installation option

Because the installation program cannot know what other components are also in your existing subnetsit cannot choose subnet CIDRs and so forth on your behalf You must configure networking for thesubnets that you install your cluster to yourself

1421 Requirements for using your VPC

The installation program no longer creates the following components

Internet gateways

NAT gateways

Subnets

Route tables

VPC DHCP options

VPC endpoints

If you use a custom VPC you must correctly configure it and its subnets for the installation program andthe cluster to use The installation program cannot subdivide network ranges for the cluster to use setroute tables for the subnets or set VPC options like DHCP so you must do so before you install thecluster

Your VPC must meet the following characteristics

The VPCrsquos CIDR block must contain the NetworkingMachineCIDR range which is the IPaddress pool for cluster machines

The VPC must not use the kubernetesiocluster owned tag

You must enable the enableDnsSupport and enableDnsHostnames attributes in your VPC sothat the cluster can use the Route 53 zones that are attached to the VPC to resolve clusterrsquosinternal DNS records See DNS Support in Your VPC in the AWS documentation

If you use a cluster with public access you must create a public and a private subnet for each availabilityzone that your cluster uses The installation program modifies your subnets to add the kubernetesiocluster shared tag so your subnets must have at least one free tag slot available forit Review the current Tag Restrictions in the AWS documentation to ensure that the installationprogram can add a tag to each subnet that you specify

If you are working in a disconnected environment you are unable to reach the public IP addresses forEC2 and ELB endpoints To resolve this you must create a VPC endpoint and attach it to the subnetthat the clusters are using The endpoints should be named as follows

ec2ltregiongtamazonawscom

elasticloadbalancingltregiongtamazonawscom

s3ltregiongtamazonawscom

Required VPC components

You must provide a suitable VPC and subnets that allow communication to your machines

Component

AWS type Description

VPCAWSEC2VPC

AWSEC2VPCEndpoint

You must provide a public VPC for thecluster to use The VPC uses an endpointthat references the route tables for eachsubnet to improve communication withthe registry that is hosted in S3

Publicsubnets AWSEC2Subnet

AWSEC2SubnetNetworkAclAssociation

Your VPC must have public subnets forbetween 1 and 3 availability zones andassociate them with appropriate Ingressrules

Internetgateway AWSEC2InternetGateway

AWSEC2VPCGatewayAttachment

AWSEC2RouteTable

AWSEC2Route

AWSEC2SubnetRouteTableAssociation

AWSEC2NatGateway

AWSEC2EIP

You must have a public Internet gatewaywith public routes attached to the VPCIn the provided templates each publicsubnet has a NAT gateway with an EIPaddress These NAT gateways allowcluster resources like private subnetinstances to reach the Internet and arenot required for some restricted networkor proxy scenarios

Networkaccesscontrol

AWSEC2NetworkAcl

AWSEC2NetworkAclEntry

You must allow the VPC to access thefollowing ports

Port Reason

80 Inbound HTTPtraffic

443 Inbound HTTPStraffic

22 Inbound SSHtraffic

1024 - 65535 Inboundephemeral traffic

0 - 65535 Outboundephemeral traffic

Privatesubnets AWSEC2Subnet

AWSEC2RouteTable

Your VPC can have private subnets Theprovided CloudFormation templates cancreate private subnets for between 1 and3 availability zones If you use privatesubnets you must provide appropriateroutes and tables for them

Component

1422 VPC validation

To ensure that the subnets that you provide are suitable the installation program confirms the followingdata

All the subnets that you specify exist

You provide private subnets

The subnet CIDRs belong to the machine CIDR that you specified

You provide subnets for each availability zone Each availability zone contains no more than onepublic and one private subnet If you use a private cluster provide only a private subnet for eachavailability zone Otherwise provide exactly one public and private subnet for each availabilityzone

You provide a public subnet for each private subnet availability zone Machines are notprovisioned in availability zones that you do not provide private subnets for

If you destroy a cluster that uses an existing VPC the VPC is not deleted When you remove theOpenShift Container Platform cluster from a VPC the kubernetesiocluster shared tag is removedfrom the subnets that it used

1423 Division of permissions

Starting with OpenShift Container Platform 43 you do not need all of the permissions that are requiredfor an installation program-provisioned infrastructure cluster to deploy a cluster This change mimicsthe division of permissions that you might have at your company some individuals can create differentresource in your clouds than others For example you might be able to create application-specific itemslike instances buckets and load balancers but not networking-related components such as VPCssubnets or ingress rules

The AWS credentials that you use when you create your cluster do not need the networking permissionsthat are required to make VPCs and core networking components within the VPC such as subnetsrouting tables Internet gateways NAT and VPN You still need permission to make the applicationresources that the machines within the cluster require such as ELBs security groups S3 buckets andnodes

1424 Isolation between clusters

If you deploy OpenShift Container Platform to an existing network the isolation of cluster services isreduced in the following ways

You can install multiple OpenShift Container Platform clusters in the same VPC

ICMP ingress is allowed from the entire network

TCP 22 ingress (SSH) is allowed to the entire network

Control plane TCP 6443 ingress (Kubernetes API) is allowed to the entire network

Control plane TCP 22623 ingress (MCS) is allowed to the entire network

Once you confirm that your Red Hat OpenShift Cluster Manager inventory is correct either maintained

IMPORTANT

Procedure

Example output

Next steps

Prerequisites

Procedure

IMPORTANT

Agent pid 31874

IMPORTANT

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

apiVersion v1baseDomain examplecom 1credentialsMode Mint 2controlPlane 3 4 hyperthreading Enabled 5 name master platform aws zones - us-west-2a - us-west-2b

1 10 11 15

rootVolume iops 4000 size 500 type io1 6 type m5xlarge replicas 3compute 7- hyperthreading Enabled 8 name worker platform aws rootVolume iops 2000 size 500 type io1 9 type c54xlarge zones - us-west-2c replicas 3metadata name test-cluster 10networking clusterNetwork - cidr 101280014 hostPrefix 23 machineNetwork - cidr 1000016 networkType OpenShiftSDN serviceNetwork - 172300016platform aws region us-west-2 11 userTags adminContact jdoe costCenter 7536 subnets 12 - subnet-1 - subnet-2 - subnet-3 amiID ami-96c6f8f7 13 serviceEndpoints 14 - name ec2 url httpsvpce-idec2us-west-2vpceamazonawscompullSecret auths 15fips false 16sshKey ssh-ed25519 AAAA 17

IMPORTANT

If you provide your own VPC specify subnets for each availability zone that your cluster uses

1463 Configuring the cluster-wide proxy during installation

Production environments can deny direct access to the Internet and instead have an HTTP or HTTPSproxy available You can configure a new OpenShift Container Platform cluster to use a proxy byconfiguring the proxy settings in the install-configyaml file

Prerequisites

You have an existing install-configyaml file

You reviewed the sites that your cluster requires access to and determined whether any ofthem need to bypass the proxy By default all cluster egress traffic is proxied including calls tohosting cloud provider APIs You added sites to the Proxy objectrsquos specnoProxy field tobypass the proxy if necessary

The Proxy object statusnoProxy field is populated with the values of the networkingmachineNetwork[]cidr networkingclusterNetwork[]cidr and networkingserviceNetwork[] fields from your installation configuration

For installations on Amazon Web Services (AWS) Google Cloud Platform (GCP)Microsoft Azure and Red Hat OpenStack Platform (RHOSP) the Proxy object statusnoProxy field is also populated with the instance metadata endpoint(169254169254)

Procedure

1 Edit your install-configyaml file and add the proxy settings For example

A proxy URL to use for creating HTTP connections outside the cluster The URL schememust be http If you use an MITM transparent proxy network that does not requireadditional proxy configuration but requires additional CAs you must not specify an httpProxy value

A proxy URL to use for creating HTTPS connections outside the cluster If this field is notspecified then httpProxy is used for both HTTP and HTTPS connections If you use anMITM transparent proxy network that does not require additional proxy configuration butrequires additional CAs you must not specify an httpsProxy value

A comma-separated list of destination domain names domains IP addresses or othernetwork CIDRs to exclude proxying Preface a domain with to match subdomains only Forexample ycom matches xycom but not ycom Use to bypass proxy for alldestinations

If provided the installation program generates a config map that is named user-ca-bundlein the openshift-config namespace that contains one or more additional CA certificatesthat are required for proxying HTTPS connections The Cluster Network Operator thencreates a trusted-ca-bundle config map that merges these contents with the Red HatEnterprise Linux CoreOS (RHCOS) trust bundle and this config map is referenced in the Proxy objectrsquos trustedCA field The additionalTrustBundle field is required unless the

apiVersion v1baseDomain mydomaincomproxy httpProxy httpltusernamegtltpswdgtltipgtltportgt 1 httpsProxy httpltusernamegtltpswdgtltipgtltportgt 2 noProxy examplecom 3additionalTrustBundle | 4 -----BEGIN CERTIFICATE----- ltMY_TRUSTED_CA_CERTgt -----END CERTIFICATE-----

proxyrsquos identity certificate is signed by an authority from the RHCOS trust bundle If youuse an MITM transparent proxy network that does not require additional proxyconfiguration but requires additional CAs you must provide the MITM CA certificate

The installation program does not support the proxy readinessEndpoints field

2 Save the file and reference it when installing OpenShift Container Platform

The installation program creates a cluster-wide proxy that is named cluster that uses the proxy settingsin the provided install-configyaml file If no proxy settings are provided a cluster Proxy object is stillcreated but it will have a nil spec

Only the Proxy object named cluster is supported and no additional proxies can becreated

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

You can install the OpenShift CLI (oc) in order to interact with OpenShift Container Platform from a

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Procedure

Prerequisites

Procedure

Cgt oc ltcommandgt

$ echo $PATH

$ oc ltcommandgt

Example output

Prerequisites

Procedure

Example output

$ oc whoami

systemadmin

1411 Next steps

15 INSTALLING A PRIVATE CLUSTER ON AWS

In OpenShift Container Platform version 46 you can install a private cluster into an existing VPC onAmazon Web Services (AWS) The installation program provisions the rest of the requiredinfrastructure which you can further customize To customize the installation you modify parameters inthe install-configyaml file before you install the cluster

151 Prerequisites

IMPORTANT

152 Private clusters

You can deploy a private OpenShift Container Platform cluster that does not expose externalendpoints Private clusters are accessible from only an internal network and are not visible to theInternet

By default OpenShift Container Platform is provisioned to use publicly-accessible DNS and endpointsA private cluster sets the DNS Ingress Controller and API server to private when you deploy yourcluster This means that the cluster resources are only accessible from your internal network and are notvisible to the internet

To deploy a private cluster you must use existing networking that meets your requirements Your clusterresources might be shared between other clusters on the network

Additionally you must deploy a private cluster from a machine that has access the API services for thecloud you provision to the hosts on the network that you provision and to the internet to obtaininstallation media You can use any machine that meets these access requirements and follows your

companyrsquos guidelines For example this machine can be a bastion host on your cloud network or amachine that has access to the network through a VPN

1521 Private clusters in AWS

To create a private cluster on Amazon Web Services (AWS) you must provide an existing private VPCand subnets to host the cluster The installation program must also be able to resolve the DNS recordsthat the cluster requires The installation program configures the Ingress Operator and API server foraccess from only the private network

The cluster still requires access to Internet to access the AWS APIs

The following items are not required or created when you install a private cluster

Public subnets

Public load balancers which support public ingress

A public Route 53 zone that matches the baseDomain for the cluster

The installation program does use the baseDomain that you specify to create a private Route 53 zoneand the required records for the cluster The cluster is configured so that the Operators do not createpublic records for the cluster and all cluster machines are placed in the private subnets that you specify

15211 Limitations

The ability to add public functionality to a private cluster is limited

You cannot make the Kubernetes API endpoints public after installation without takingadditional actions including creating public subnets in the VPC for each availability zone in usecreating a public load balancer and configuring the control plane security groups to allow trafficfrom Internet on 6443 (Kubernetes API port)

If you use a public Service type load balancer you must tag a public subnet in each availabilityzone with kubernetesioclusterltcluster-infra-idgt shared so that AWS can use them tocreate public load balancers

Internet gateways

NAT gateways

Subnets

Route tables

VPC DHCP options

VPC endpoints

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

1532 VPC validation

In OpenShift Container Platform 46 you require access to the Internet to install your cluster The

Telemetry service which runs by default to provide metrics about cluster health and the success ofupdates also requires Internet access If your cluster is connected to the Internet Telemetry runsautomatically and your cluster is registered to the Red Hat OpenShift Cluster Manager (OCM)

IMPORTANT

Procedure

1 If you do not have an SSH key that is configured for password-less authentication on your

Example output

Next steps

Prerequisites

Procedure

3 Navigate to the page for your installation type download the installation program for your

Agent pid 31874

IMPORTANT

157 Manually creating the installation configuration file

For installations of a private OpenShift Container Platform cluster that are only accessible from aninternal network and are not visible to the Internet you must manually generate your installationconfiguration file

Prerequisites

Obtain the OpenShift Container Platform installation program and the access token for yourcluster

Procedure

1 Create an installation directory to store your required installation assets in

IMPORTANT

You must create a directory Some installation assets like bootstrap X509certificates have short expiration intervals so you must not reuse an installationdirectory If you want to reuse individual files from another cluster installationyou can copy them into your directory However the file names for theinstallation assets might change between releases Use caution when copyinginstallation files from an earlier OpenShift Container Platform version

$ mkdir ltinstallation_directorygt

2 Customize the following install-configyaml file template and save it in the ltinstallation_directorygt

You must name this configuration file install-configyaml

IMPORTANT

The install-configyaml file is consumed during the next step of the installationprocess You must back it up now

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 11 15

Optional Add this parameter to force the Cloud Credential Operator (CCO) to use the specifiedmode instead of having the CCO dynamically try to determine the capabilities of the credentialsFor details about CCO modes see the Cloud Credential Operator entry in the Red Hat Operators

rootVolume iops 4000 size 500 type io1 6 type m5xlarge replicas 3compute 7- hyperthreading Enabled 8 name worker platform aws rootVolume iops 2000 size 500 type io1 9 type c54xlarge zones - us-west-2c replicas 3metadata name test-cluster 10networking clusterNetwork - cidr 101280014 hostPrefix 23 machineNetwork - cidr 1000016 networkType OpenShiftSDN serviceNetwork - 172300016platform aws region us-west-2 11 userTags adminContact jdoe costCenter 7536 subnets 12 - subnet-1 - subnet-2 - subnet-3 amiID ami-96c6f8f7 13 serviceEndpoints 14 - name ec2 url httpsvpce-idec2us-west-2vpceamazonawscompullSecret auths 15fips false 16sshKey ssh-ed25519 AAAA 17publish Internal 18

reference content

IMPORTANT

How to publish the user-facing endpoints of your cluster Set publish to Internal to deploy aprivate cluster which cannot be accessed from the Internet The default value is External

Production environments can deny direct access to the Internet and instead have an HTTP or HTTPS

Prerequisites

Procedure

If provided the installation program generates a config map that is named user-ca-bundlein the openshift-config namespace that contains one or more additional CA certificates

IMPORTANT

Prerequisites

Procedure

For ltinstallation_directorygt specify the

Example output

IMPORTANT

Procedure

3 In the Command-line interface section select MacOS from the drop-down menu and click

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

Prerequisites

Procedure

Example output

Prerequisites

$ echo $PATH

$ oc ltcommandgt

$ oc whoami

systemadmin

Procedure

Example output

1512 Next steps

16 INSTALLING A CLUSTER ON AWS INTO A GOVERNMENT REGION

In OpenShift Container Platform version 46 you can install a cluster on Amazon Web Services (AWS)into a government region To configure the government region modify parameters in the install-configyaml file before you install the cluster

161 Prerequisites

IMPORTANT

162 AWS government regions

OpenShift Container Platform supports deploying a cluster to AWS GovCloud (US) regions AWSGovCloud is specifically designed for US government agencies at the federal state and local level aswell as contractors educational institutions and other US customers that must run sensitive workloadsin the cloud

These regions do not have published Red Hat Enterprise Linux CoreOS (RHCOS) Amazon MachineImages (AMI) to select so you must upload a custom AMI that belongs to that region

The following AWS GovCloud partitions are supported

us-gov-west-1

us-gov-east-1

The AWS GovCloud region and custom AMI must be manually configured in the install-configyaml filesince RHCOS AMIs are not provided by Red Hat for those regions

Public zones are not supported in Route 53 in AWS GovCloud Therefore clusters mustbe private if they are deployed to an AWS government region

Additionally you must deploy a private cluster from a machine that has access the API services for thecloud you provision to the hosts on the network that you provision and to the internet to obtaininstallation media You can use any machine that meets these access requirements and follows yourcompanyrsquos guidelines For example this machine can be a bastion host on your cloud network or amachine that has access to the network through a VPN

Public subnets

16311 Limitations

Internet gateways

NAT gateways

Subnets

Route tables

VPC DHCP options

VPC endpoints

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

1642 VPC validation

IMPORTANT

Procedure

Example output

Next steps

Prerequisites

Agent pid 31874

Procedure

IMPORTANT

When installing OpenShift Container Platform on Amazon Web Services (AWS) into a region requiring acustom Red Hat Enterprise Linux CoreOS (RHCOS) AMI you must manually generate your installationconfiguration file

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

apiVersion v1baseDomain examplecom 1credentialsMode Mint 2controlPlane 3 4 hyperthreading Enabled 5 name master platform aws zones - us-gov-west-1a - us-gov-west-1b

1 10 14

Required

rootVolume iops 4000 size 500 type io1 6 type m5xlarge replicas 3compute 7- hyperthreading Enabled 8 name worker platform aws rootVolume iops 2000 size 500 type io1 9 type c54xlarge zones - us-gov-west-1c replicas 3metadata name test-cluster 10networking clusterNetwork - cidr 101280014 hostPrefix 23 machineNetwork - cidr 1000016 networkType OpenShiftSDN serviceNetwork - 172300016platform aws region us-gov-west-1 userTags adminContact jdoe costCenter 7536 subnets 11 - subnet-1 - subnet-2 - subnet-3 amiID ami-96c6f8f7 12 serviceEndpoints 13 - name ec2 url httpsvpce-idec2us-west-2vpceamazonawscompullSecret auths 14fips false 15sshKey ssh-ed25519 AAAA 16publish Internal 17

reference content

IMPORTANT

1683 AWS regions without a published RHCOS AMI

You can deploy an OpenShift Container Platform cluster to Amazon Web Services (AWS) regions

without native support for a Red Hat Enterprise Linux CoreOS (RHCOS) Amazon Machine Image (AMI)or the AWS software development kit (SDK) If a published AMI is not available for an AWS region youcan upload a custom AMI prior to installing the cluster This is required if you are deploying your clusterto an AWS government region

If you are deploying to a non-government region that does not have a published RHCOS AMI and youdo not specify a custom AMI the installation program copies the us-east-1 AMI to the user accountautomatically Then the installation program creates the control plane machines with encrypted EBSvolumes using the default or user-specified Key Management Service (KMS) key This allows the AMI tofollow the same process workflow as published RHCOS AMIs

A region without native support for an RHCOS AMI is not available to select from the terminal duringcluster creation because it is not published However you can install to this region by configuring thecustom AMI in the install-configyaml file

1684 Uploading a custom RHCOS AMI in AWS

If you are deploying to a custom Amazon Web Services (AWS) region you must upload a custom RedHat Enterprise Linux CoreOS (RHCOS) Amazon Machine Image (AMI) that belongs to that region

Prerequisites

You configured an AWS account

You created an Amazon S3 bucket with the required IAM service role

You uploaded your RHCOS VMDK file to Amazon S3 The RHCOS VMDK file must be thehighest version that is less than or equal to the OpenShift Container Platform version you areinstalling

You downloaded the AWS CLI and installed it on your computer See Install the AWS CLI Usingthe Bundled Installer

Procedure

1 Export your AWS profile as an environment variable

The AWS profile name that holds your AWS credentials like govcloud

2 Export the region to associate with your custom AMI as an environment variable

The AWS region like us-gov-east-1

3 Export the version of RHCOS you uploaded to Amazon S3 as an environment variable

The RHCOS VMDK version like 460

$ export AWS_PROFILE=ltaws_profilegt 1

$ export AWS_DEFAULT_REGION=ltaws_regiongt 1

$ export RHCOS_VERSION=ltversiongt 1

4 Export the Amazon S3 bucket name as an environment variable

5 Create the containersjson file and define your RHCOS VMDK file

6 Import the RHCOS disk as an Amazon EBS snapshot

The description of your RHCOS disk being imported like rhcos-$RHCOS_VERSION-x86_64-awsx86_64

The file path to the JSON file describing your RHCOS disk The JSON file should containyour Amazon S3 bucket name and key

7 Check the status of the image import

Example output

$ export VMIMPORT_BUCKET_NAME=lts3_bucket_namegt

$ cat ltltEOF gt containersjson Description rhcos-$RHCOS_VERSION-x86_64-awsx86_64 Format vmdk UserBucket S3Bucket $VMIMPORT_BUCKET_NAME S3Key rhcos-$RHCOS_VERSION-x86_64-awsx86_64vmdk EOF

$ aws ec2 import-snapshot --region $AWS_DEFAULT_REGION --description ltdescriptiongt 1 --disk-container ltfile_pathgtcontainersjson 2

$ watch -n 5 aws ec2 describe-import-snapshot-tasks --region $AWS_DEFAULT_REGION

ImportSnapshotTasks [ Description rhcos-460-x86_64-awsx86_64 ImportTaskId import-snap-fh6i8uil SnapshotTaskDetail Description rhcos-460-x86_64-awsx86_64 DiskImageSize 8190566400 Format VMDK SnapshotId snap-06331325870076318 Status completed UserBucket S3Bucket external-images S3Key rhcos-460-x86_64-awsx86_64vmdk

Copy the SnapshotId to register the image

8 Create a custom RHCOS AMI from the RHCOS snapshot

The RHCOS VMDK architecture type like x86_64 s390x or ppc64le

The Description from the imported snapshot

The name of the RHCOS AMI

The SnapshotID from the imported snapshot

To learn more about these APIs see the AWS documentation for importing snapshots and creatingEBS-backed AMIs

Prerequisites

$ aws ec2 register-image --region $AWS_DEFAULT_REGION --architecture x86_64 1 --description rhcos-$RHCOS_VERSION-x86_64-awsx86_64 2 --ena-support --name rhcos-$RHCOS_VERSION-x86_64-awsx86_64 3 --virtualization-type hvm --root-device-name devxvda --block-device-mappings DeviceName=devxvdaEbs=DeleteOnTermination=trueSnapshotId=ltsnapshot_IDgt 4

Procedure

If provided the installation program generates a config map that is named user-ca-bundlein the openshift-config namespace that contains one or more additional CA certificatesthat are required for proxying HTTPS connections The Cluster Network Operator thencreates a trusted-ca-bundle config map that merges these contents with the Red HatEnterprise Linux CoreOS (RHCOS) trust bundle and this config map is referenced in the Proxy objectrsquos trustedCA field The additionalTrustBundle field is required unless theproxyrsquos identity certificate is signed by an authority from the RHCOS trust bundle If youuse an MITM transparent proxy network that does not require additional proxyconfiguration but requires additional CAs you must provide the MITM CA certificate

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

Procedure

Prerequisites

Procedure

Example output

$ echo $PATH

$ oc ltcommandgt

$ oc whoami

systemadmin

Prerequisites

Procedure

Example output

1613 Next steps

17 INSTALLING A CLUSTER ON USER-PROVISIONED

17 INSTALLING A CLUSTER ON USER-PROVISIONEDINFRASTRUCTURE IN AWS BY USING CLOUDFORMATIONTEMPLATES

In OpenShift Container Platform version 46 you can install a cluster on Amazon Web Services (AWS)that uses infrastructure that you provide

One way to create this infrastructure is to use the provided CloudFormation templates You can modifythe templates to customize your infrastructure or use the information that they contain to create AWSobjects according to your companyrsquos policies

IMPORTANT

The steps for performing a user-provisioned infrastructure installation are provided as anexample only Installing a cluster with infrastructure you provide requires knowledge ofthe cloud provider and the installation process of OpenShift Container Platform SeveralCloudFormation templates are provided to assist in completing these steps or to helpmodel your own You are also free to create the required resources through othermethods the templates are just an example

171 Prerequisites

You reviewed details about the OpenShift Container Platform installation and updateprocesses

You configured an AWS account to host the cluster

IMPORTANT

You downloaded the AWS CLI and installed it on your computer See Install the AWS CLI Usingthe Bundled Installer (Linux macOS or Unix) in the AWS documentation

If you use a firewall you configured it to allow the sites that your cluster requires access to

Be sure to also review this site list if you are configuring a proxy

IMPORTANT

173 Required AWS infrastructure components

To install OpenShift Container Platform on user-provisioned infrastructure in Amazon Web Services(AWS) you must manually create both the machines and their supporting infrastructure

For more information about the integration testing for different platforms see the OpenShift ContainerPlatform 4x Tested Integrations page

By using the provided CloudFormation templates you can create stacks of AWS resources thatrepresent the following components

An AWS Virtual Private Cloud (VPC)

Networking and load balancing components

Security groups and roles

An OpenShift Container Platform bootstrap node

OpenShift Container Platform control plane nodes

An OpenShift Container Platform compute node

Alternatively you can manually create the components or you can reuse existing infrastructure that

Alternatively you can manually create the components or you can reuse existing infrastructure thatmeets the cluster requirements Review the CloudFormation templates for more details about how thecomponents interrelate

1731 Cluster machines

You need AWSEC2Instance objects for the following machines

A bootstrap machine This machine is required during installation but you can remove it afteryour cluster deploys

Three control plane machines The control plane machines are not governed by a machine set

Compute machines You must create at least two compute machines which are also known asworker machines during installation These machines are not governed by a machine set

You can use the following instance types for the cluster machines with the provided CloudFormationtemplates

IMPORTANT

If m4 instance types are not available in your region such as with eu-west-3 use m5types instead

Table 117 Instance types for machines

Instance type Bootstrap Control plane Compute

i3large x

m4large x

m4xlarge x x

m42xlarge x x

m44xlarge x x

m48xlarge x x

m410xlarge x x

m416xlarge x x

m5large x

m5xlarge x x

m52xlarge x x

m54xlarge x x

m58xlarge x x

m510xlarge x x

m516xlarge x x

c4large x

c4xlarge x

c42xlarge x x

c44xlarge x x

c48xlarge x x

r4large x

r4xlarge x x

r42xlarge x x

r44xlarge x x

r48xlarge x x

r416xlarge x x

You might be able to use other instance types that meet the specifications of these instance types

1732 Certificate signing requests management

Because your cluster has limited access to automatic machine management when you use infrastructurethat you provision you must provide a mechanism for approving cluster certificate signing requests(CSRs) after installation The kube-controller-manager only approves the kubelet client CSRs The machine-approver cannot guarantee the validity of a serving certificate that is requested by usingkubelet credentials because it cannot confirm that the correct machine issued the request You mustdetermine and implement a method of verifying the validity of the kubelet serving certificate requestsand approving them

1733 Other infrastructure components

DNS entries

Load balancers (classic or network) and listeners

A public and a private Route 53 zone

Security groups

IAM roles

S3 buckets

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

Required DNS and load balancing components

Your DNS and load balancer configuration needs to use a public hosted zone and can use a privatehosted zone similar to the one that the installation program uses if it provisions the clusterrsquosinfrastructure You must create a DNS entry that resolves to your load balancer An entry for apiltcluster_namegtltdomaingt must point to the external load balancer and an entry for api-intltcluster_namegtltdomaingt must point to the internal load balancer

The cluster also requires load balancers and listeners for port 6443 which are required for theKubernetes API and its extensions and port 22623 which are required for the Ignition config files fornew machines The targets will be the master nodes Port 6443 must be accessible to both clientsexternal to the cluster and nodes within the cluster Port 22623 must be accessible to nodes within thecluster

Component AWS type Description

DNS AWSRoute53HostedZone

The hosted zone for your internal DNS

etcd recordsets

AWSRoute53RecordSet

The registration records for etcd for your control plane machines

Public loadbalancer

AWSElasticLoadBalancingV2LoadBalancer

The load balancer for your public subnets

External APIserver record

AWSRoute53RecordSetGroup

Alias records for the external API server

Externallistener

AWSElasticLoadBalancingV2Listener

A listener on port 6443 for the external load balancer

External targetgroup

AWSElasticLoadBalancingV2TargetGroup

The target group for the external load balancer

Private loadbalancer

The load balancer for your private subnets

Internal APIserver record

Alias records for the internal API server

Internal listener AWSElasticLoadBalancingV2Listener

A listener on port 22623 for the internal load balancer

Internal targetgroup

The target group for the internal load balancer

Security groups

The control plane and worker machines require access to the following ports

Group Type IP Protocol Port range

MasterSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

tcp 6443

tcp 22623

WorkerSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

BootstrapSecurityGroup

AWSEC2SecurityGroup

tcp 22

tcp 19531

Control plane Ingress

The control plane machines require the following Ingress groups Each Ingress group is a AWSEC2SecurityGroupIngress resource

Ingress group Description IP protocol Port range

MasterIngressEtcd

etcd tcp 2379- 2380

MasterIngressVxlan

Vxlan packets udp 4789

MasterIngressWorkerVxlan

MasterIngressInternal

Internal cluster communication and Kubernetesproxy metrics

tcp 9000 - 9999

MasterIngressWorkerInternal

Internal cluster communication tcp 9000 - 9999

MasterIngressKube

Kubernetes kubelet scheduler and controllermanager

tcp 10250 - 10259

MasterIngressWorkerKube

tcp 10250 - 10259

MasterIngressIngressServices

Kubernetes Ingress services tcp 30000 - 32767

MasterIngressWorkerIngressServices

Worker Ingress

The worker machines require the following Ingress groups Each Ingress group is a AWSEC2SecurityGroupIngress resource

WorkerIngressVxlan

WorkerIngressWorkerVxlan

WorkerIngressInternal

WorkerIngressWorkerInternal

WorkerIngressKube

tcp 10250

WorkerIngressWorkerKube

tcp 10250

WorkerIngressIngressServices

WorkerIngressWorkerIngressServices

Roles and instance profiles

You must grant the machines permissions in AWS The provided CloudFormation templates grant themachines Allow permissions for the following AWSIAMRole objects and provide a AWSIAMInstanceProfile for each set of roles If you do not use the templates you can grant themachines the following broad permissions or the following individual permissions

Role Effect Action Resource

Master Allow ec2

Allow elasticloadbalancing

Allow iamPassRole

Allow s3GetObject

Worker Allow ec2Describe

Bootstrap Allow ec2Describe

Allow ec2AttachVolume

Allow ec2DetachVolume

tagTagResources

tagUntagResources

ec2AllocateAddress

ec2AssociateAddress

ec2CopyImage

ec2CreateTags

ec2CreateVolume

ec2DeleteSnapshot

ec2DeregisterImage

ec2DescribeImages

ec2DescribeKeyPairs

ec2DescribeRegions

ec2DescribeSubnets

ec2DescribeTags

ec2DescribeVolumes

ec2DescribeVpcs

ec2ReleaseAddress

ec2RunInstances

ec2CreateNatGateway

ec2CreateRoute

ec2CreateRouteTable

ec2CreateSubnet

ec2CreateVpc

iamCreateRole

iamDeleteRole

iamDeleteRolePolicy

iamGetRole

iamGetRolePolicy

iamGetUser

iamListRoles

iamListUsers

iamPassRole

iamPutRolePolicy

iamTagRole

route53GetChange

s3CreateBucket

s3DeleteBucket

s3GetBucketAcl

s3GetBucketCors

s3GetBucketLocation

s3GetBucketLogging

s3GetBucketTagging

s3GetBucketWebsite

s3ListBucket

s3PutBucketAcl

s3PutBucketTagging

s3DeleteObject

s3GetObject

s3GetObjectAcl

s3GetObjectTagging

s3GetObjectVersion

s3PutObject

s3PutObjectAcl

s3PutObjectTagging

ec2DeleteVolume

iamDeleteAccessKey

iamDeleteUser

iamListRolePolicies

iamListUserPolicies

s3DeleteObject

tagGetResources

ec2DeleteNatGateway

ec2DeleteRoute

ec2DeleteRouteTable

ec2DeleteSubnet

ec2DeleteVpc

iamCreateAccessKey

iamCreateUser

iamDeleteAccessKey

iamDeleteUser

iamDeleteUserPolicy

iamGetUserPolicy

iamListAccessKeys

iamPutUserPolicy

iamTagUser

iamGetUserPolicy

iamListAccessKeys

s3HeadBucket

Prerequisites

Procedure

IMPORTANT

You can use this key to SSH into the master nodes as the user core When you deploy the cluster the

Procedure

Example output

Next steps

When you install OpenShift Container Platform provide the SSH public key to the installationprogram If you install a cluster on infrastructure that you provision you must provide this key toyour clusterrsquos machines

176 Creating the installation files for AWS

To install OpenShift Container Platform on Amazon Web Services (AWS) using user-provisionedinfrastructure you must generate the files that the installation program needs to deploy your clusterand modify them so that the cluster creates only the machines that it will use You generate and

Agent pid 31874

customize the install-configyaml file Kubernetes manifests and Ignition config files You also have theoption to first set up a separate var partition during the preparation phases of installation

1761 Optional Creating a separate var partition

It is recommended that disk partitioning for OpenShift Container Platform be left to the installerHowever there are cases where you might want to create separate partitions in a part of the filesystemthat you expect to grow

OpenShift Container Platform supports the addition of a single partition to attach storage to either the var partition or a subdirectory of var For example

varlibcontainers Holds container-related content that can grow as more images andcontainers are added to a system

varlibetcd Holds data that you might want to keep separate for purposes such asperformance optimization of etcd storage

var Holds data that you might want to keep separate for purposes such as auditing

Storing the contents of a var directory separately makes it easier to grow storage for those areas asneeded and reinstall OpenShift Container Platform at a later date and keep that data intact With thismethod you will not have to pull all your containers again nor will you have to copy massive log fileswhen you update systems

Because var must be in place before a fresh installation of Red Hat Enterprise Linux CoreOS (RHCOS)the following procedure sets up the separate var partition by creating a machine config that is insertedduring the openshift-install preparation phases of an OpenShift Container Platform installation

IMPORTANT

If you follow the steps to create a separate var partition in this procedure it is notnecessary to create the Kubernetes manifest and Ignition config files again as describedlater in this section

Procedure

1 Create a directory to hold the OpenShift Container Platform installation files

2 Run openshift-install to create a set of files in the manifest and openshift subdirectoriesAnswer the system questions as you are prompted

3 Create a MachineConfig object and add it to a file in the openshift directory For examplename the file 98-var-partitionyaml change the disk device name to the name of the storage

$ mkdir $HOMEclusterconfig

$ openshift-install create manifests --dir $HOMEclusterconfig SSH Public Key $ ls $HOMEclusterconfigopenshift99_kubeadmin-password-secretyaml99_openshift-cluster-api_master-machines-0yaml99_openshift-cluster-api_master-machines-1yaml99_openshift-cluster-api_master-machines-2yaml

device on the worker systems and set the storage size as appropriate This attaches storage toa separate var directory

The storage device name of the disk that you want to partition

When adding a data partition to the boot disk a minimum value of 25000 MiB (Mebibytes)is recommended The root file system is automatically resized to fill all available space upto the specified offset If no value is specified or if the specified value is smaller than therecommended minimum the resulting root file system will be too small and futurereinstalls of RHCOS might overwrite the beginning of the data partition

4 Run openshift-install again to create Ignition configs from a set of files in the manifest and openshift subdirectories

Now you can use the Ignition config files as input to the installation procedures to install Red HatEnterprise Linux CoreOS (RHCOS) systems

apiVersion machineconfigurationopenshiftiov1kind MachineConfigmetadata labels machineconfigurationopenshiftiorole worker name 98-var-partitionspec config ignition version 310 storage disks - device devltdevice_namegt 1 partitions - sizeMiB ltpartition_sizegt startMiB ltpartition_start_offsetgt 2 label var filesystems - path var device devdiskby-partlabelvar format xfs systemd units - name varmount enabled true contents | [Unit] Before=local-fstarget [Mount] Where=var What=devdiskby-partlabelvar [Install] WantedBy=local-fstarget

$ openshift-install create ignition-configs --dir $HOMEclusterconfig$ ls $HOMEclusterconfigauth bootstrapign masterign metadatajson workerign

Generate and customize the installation configuration file that the installation program needs to deployyour cluster

Prerequisites

You obtained the OpenShift Container Platform installation program for user-provisionedinfrastructure and the pull secret for your cluster

You checked that you are deploying your cluster to a region with an accompanying Red HatEnterprise Linux CoreOS (RHCOS) AMI published by Red Hat If you are deploying to a regionthat requires a custom AMI such as an AWS GovCloud region you must create the install-configyaml file manually

Procedure

IMPORTANT

ii Select aws as the platform to target

iii If you do not have an AWS profile stored on your computer enter the AWS access keyID and secret access key for the user that you configured to run the installationprogram

The AWS access key ID and secret access key are stored in ~awscredentials in the home directory of the current user on theinstallation host You are prompted for the credentials by the installationprogram if the credentials for the exported profile are not present in thefile Any credentials that you provide to the installation program arestored in the file

2 Optional Back up the install-configyaml file

IMPORTANT

See Configuration and credential file settings in the AWS documentation for more informationabout AWS profile and credential configuration

Prerequisites

Procedure

1764 Creating the Kubernetes manifest and Ignition config files

Because you must modify some cluster definition files and manually start the cluster machines you mustgenerate the Kubernetes manifest and Ignition config files that the cluster needs to make its machines

The installation configuration file transforms into the Kubernetes manifests The manifests wrap into theIgnition configuration files which are later used to create the cluster

IMPORTANT

The Ignition config files that the installation program generates contain certificates thatexpire after 24 hours which are then renewed at that time If the cluster is shut downbefore renewing the certificates and the cluster is later restarted after the 24 hours haveelapsed the cluster automatically recovers the expired certificates The exception is thatyou must manually approve the pending node-bootstrapper certificate signing requests(CSRs) to recover kubelet certificates See the documentation for Recovering fromexpired control plane certificates for more information

Prerequisites

You obtained the OpenShift Container Platform installation program

You created the install-configyaml installation configuration file

Procedure

1 Change to the directory that contains the installation program and generate the Kubernetesmanifests for the cluster

Example output

For ltinstallation_directorygt specify the installation directory that contains the install-configyaml file you created

2 Remove the Kubernetes manifest files that define the control plane machines

By removing these files you prevent the cluster from automatically generating control planemachines

3 Remove the Kubernetes manifest files that define the worker machines

INFO Credentials loaded from the myprofile profile in file homemyuserawscredentialsINFO Consuming Install Config from target directoryINFO Manifests created in install_dirmanifests and install_diropenshift

$ rm -f ltinstallation_directorygtopenshift99_openshift-cluster-api_master-machines-yaml

Because you create and manage the worker machines yourself you do not need to initializethese machines

4 Check that the mastersSchedulable parameter in the ltinstallation_directorygtmanifestscluster-scheduler-02-configyml Kubernetes manifestfile is set to false This setting prevents pods from being scheduled on the control planemachines

a Open the ltinstallation_directorygtmanifestscluster-scheduler-02-configyml file

b Locate the mastersSchedulable parameter and ensure that it is set to false

c Save and exit the file

5 Optional If you do not want the Ingress Operator to create DNS records on your behalf removethe privateZone and publicZone sections from the ltinstallation_directorygtmanifestscluster-dns-02-configyml DNS configuration file

Remove this section completely

If you do so you must add ingress DNS records manually in a later step

6 To create the Ignition configuration files run the following command from the directory thatcontains the installation program

For ltinstallation_directorygt specify the same installation directory

The following files are generated in the directory

auth kubeadmin-password kubeconfig bootstrapign masterign metadatajson workerign

$ rm -f ltinstallation_directorygtopenshift99_openshift-cluster-api_worker-machineset-yaml

apiVersion configopenshiftiov1kind DNSmetadata creationTimestamp null name clusterspec baseDomain exampleopenshiftcom privateZone 1 id mycluster-100419-private-zone publicZone 2 id exampleopenshiftcomstatus

$ openshift-install create ignition-configs --dir=ltinstallation_directorygt 1

177 Extracting the infrastructure name

The Ignition config files contain a unique cluster identifier that you can use to uniquely identify yourcluster in Amazon Web Services (AWS) The infrastructure name is also used to locate the appropriateAWS resources during an OpenShift Container Platform installation The provided CloudFormationtemplates contain references to this infrastructure name so you must extract it

Prerequisites

You obtained the OpenShift Container Platform installation program and the pull secret foryour cluster

You generated the Ignition config files for your cluster

You installed the jq package

Procedure

To extract and view the infrastructure name from the Ignition config file metadata run thefollowing command

Example output

The output of this command is your cluster name and a random string

178 Creating a VPC in AWS

You must create a Virtual Private Cloud (VPC) in Amazon Web Services (AWS) for your OpenShiftContainer Platform cluster to use You can customize the VPC to meet your requirements includingVPN and route tables

You can use the provided CloudFormation template and a custom parameter file to create a stack ofAWS resources that represent the VPC

If you do not use the provided CloudFormation template to create your AWSinfrastructure you must review the provided information and manually create theinfrastructure If your cluster does not initialize correctly you might have to contact RedHat support with your installation logs

Prerequisites

$ jq -r infraID ltinstallation_directorygtmetadatajson 1

openshift-vw9j6 1

You added your AWS keys and region to your local AWS profile by running aws configure

Procedure

1 Create a JSON file that contains the parameter values that the template requires

The CIDR block for the VPC

Specify a CIDR block in the format xxxx16-24

The number of availability zones to deploy the VPC in

Specify an integer between 1 and 3

The size of each subnet in each availability zone

Specify an integer between 5 and 13 where 5 is 27 and 13 is 19

2 Copy the template from the CloudFormation template for the VPC section of this topic andsave it as a YAML file on your computer This template describes the VPC that your clusterrequires

3 Launch the CloudFormation template to create a stack of AWS resources that represent theVPC

IMPORTANT

You must enter the command on a single line

[ ParameterKey VpcCidr 1 ParameterValue 1000016 2 ParameterKey AvailabilityZoneCount 3 ParameterValue 1 4 ParameterKey SubnetBits 5 ParameterValue 12 6 ]

$ aws cloudformation create-stack --stack-name ltnamegt 1 --template-body filelttemplategtyaml 2 --parameters fileltparametersgtjson 3

ltnamegt is the name for the CloudFormation stack such as cluster-vpc You need thename of this stack if you remove the cluster

lttemplategt is the relative path to and name of the CloudFormation template YAML filethat you saved

ltparametersgt is the relative path to and name of the CloudFormation parameters JSONfile

Example output

4 Confirm that the template components exist

After the StackStatus displays CREATE_COMPLETE the output displays values for thefollowing parameters You must provide these parameter values to the other CloudFormationtemplates that you run to create your cluster

VpcId The ID of your VPC

PublicSubnetIds

The IDs of the new public subnets

PrivateSubnetIds

The IDs of the new private subnets

1781 CloudFormation template for the VPC

You can use the following CloudFormation template to deploy the VPC that you need for yourOpenShift Container Platform cluster

Example 123 CloudFormation template for the VPC

arnawscloudformationus-east-1269333783861stackcluster-vpcdbedae40-2fd3-11eb-820e-12a48460849f

$ aws cloudformation describe-stacks --stack-name ltnamegt

AWSTemplateFormatVersion 2010-09-09Description Template for Best Practice VPC with 1-3 AZs

Parameters VpcCidr AllowedPattern ^(([0-9]|[1-9][0-9]|1[0-9]2|2[0-4][0-9]|25[0-5]))3([0-9]|[1-9][0-9]|1[0-9]2|2[0-4][0-9]|25[0-5])((1[6-9]|2[0-4]))$ ConstraintDescription CIDR block parameter must be in the form xxxx16-24 Default 1000016 Description CIDR block for VPC Type String AvailabilityZoneCount ConstraintDescription The number of availability zones (Min 1 Max 3) MinValue 1 MaxValue 3

Default 1 Description How many AZs to create VPC subnets for (Min 1 Max 3) Type Number SubnetBits ConstraintDescription CIDR block parameter must be in the form xxxx19-27 MinValue 5 MaxValue 13 Default 12 Description Size of each subnet to create within the availability zones (Min 5 = 27 Max 13 = 19) Type Number

Metadata AWSCloudFormationInterface ParameterGroups - Label default Network Configuration Parameters - VpcCidr - SubnetBits - Label default Availability Zones Parameters - AvailabilityZoneCount ParameterLabels AvailabilityZoneCount default Availability Zone Count VpcCidr default VPC CIDR SubnetBits default Bits Per Subnet

Conditions DoAz3 Equals [3 Ref AvailabilityZoneCount] DoAz2 Or [Equals [2 Ref AvailabilityZoneCount] Condition DoAz3]

Resources VPC Type AWSEC2VPC Properties EnableDnsSupport true EnableDnsHostnames true CidrBlock Ref VpcCidr PublicSubnet Type AWSEC2Subnet Properties VpcId Ref VPC CidrBlock Select [0 Cidr [Ref VpcCidr 6 Ref SubnetBits]] AvailabilityZone Select - 0 - FnGetAZs Ref AWSRegion PublicSubnet2 Type AWSEC2Subnet Condition DoAz2 Properties VpcId Ref VPC

CidrBlock Select [1 Cidr [Ref VpcCidr 6 Ref SubnetBits]] AvailabilityZone Select - 1 - FnGetAZs Ref AWSRegion PublicSubnet3 Type AWSEC2Subnet Condition DoAz3 Properties VpcId Ref VPC CidrBlock Select [2 Cidr [Ref VpcCidr 6 Ref SubnetBits]] AvailabilityZone Select - 2 - FnGetAZs Ref AWSRegion InternetGateway Type AWSEC2InternetGateway GatewayToInternet Type AWSEC2VPCGatewayAttachment Properties VpcId Ref VPC InternetGatewayId Ref InternetGateway PublicRouteTable Type AWSEC2RouteTable Properties VpcId Ref VPC PublicRoute Type AWSEC2Route DependsOn GatewayToInternet Properties RouteTableId Ref PublicRouteTable DestinationCidrBlock 00000 GatewayId Ref InternetGateway PublicSubnetRouteTableAssociation Type AWSEC2SubnetRouteTableAssociation Properties SubnetId Ref PublicSubnet RouteTableId Ref PublicRouteTable PublicSubnetRouteTableAssociation2 Type AWSEC2SubnetRouteTableAssociation Condition DoAz2 Properties SubnetId Ref PublicSubnet2 RouteTableId Ref PublicRouteTable PublicSubnetRouteTableAssociation3 Condition DoAz3 Type AWSEC2SubnetRouteTableAssociation Properties SubnetId Ref PublicSubnet3 RouteTableId Ref PublicRouteTable PrivateSubnet Type AWSEC2Subnet Properties VpcId Ref VPC CidrBlock Select [3 Cidr [Ref VpcCidr 6 Ref SubnetBits]] AvailabilityZone Select - 0 - FnGetAZs Ref AWSRegion

PrivateRouteTable Type AWSEC2RouteTable Properties VpcId Ref VPC PrivateSubnetRouteTableAssociation Type AWSEC2SubnetRouteTableAssociation Properties SubnetId Ref PrivateSubnet RouteTableId Ref PrivateRouteTable NAT DependsOn - GatewayToInternet Type AWSEC2NatGateway Properties AllocationId FnGetAtt - EIP - AllocationId SubnetId Ref PublicSubnet EIP Type AWSEC2EIP Properties Domain vpc Route Type AWSEC2Route Properties RouteTableId Ref PrivateRouteTable DestinationCidrBlock 00000 NatGatewayId Ref NAT PrivateSubnet2 Type AWSEC2Subnet Condition DoAz2 Properties VpcId Ref VPC CidrBlock Select [4 Cidr [Ref VpcCidr 6 Ref SubnetBits]] AvailabilityZone Select - 1 - FnGetAZs Ref AWSRegion PrivateRouteTable2 Type AWSEC2RouteTable Condition DoAz2 Properties VpcId Ref VPC PrivateSubnetRouteTableAssociation2 Type AWSEC2SubnetRouteTableAssociation Condition DoAz2 Properties SubnetId Ref PrivateSubnet2 RouteTableId Ref PrivateRouteTable2 NAT2 DependsOn - GatewayToInternet Type AWSEC2NatGateway Condition DoAz2

Properties AllocationId FnGetAtt - EIP2 - AllocationId SubnetId Ref PublicSubnet2 EIP2 Type AWSEC2EIP Condition DoAz2 Properties Domain vpc Route2 Type AWSEC2Route Condition DoAz2 Properties RouteTableId Ref PrivateRouteTable2 DestinationCidrBlock 00000 NatGatewayId Ref NAT2 PrivateSubnet3 Type AWSEC2Subnet Condition DoAz3 Properties VpcId Ref VPC CidrBlock Select [5 Cidr [Ref VpcCidr 6 Ref SubnetBits]] AvailabilityZone Select - 2 - FnGetAZs Ref AWSRegion PrivateRouteTable3 Type AWSEC2RouteTable Condition DoAz3 Properties VpcId Ref VPC PrivateSubnetRouteTableAssociation3 Type AWSEC2SubnetRouteTableAssociation Condition DoAz3 Properties SubnetId Ref PrivateSubnet3 RouteTableId Ref PrivateRouteTable3 NAT3 DependsOn - GatewayToInternet Type AWSEC2NatGateway Condition DoAz3 Properties AllocationId FnGetAtt - EIP3 - AllocationId SubnetId Ref PublicSubnet3 EIP3 Type AWSEC2EIP Condition DoAz3 Properties Domain vpc

Route3 Type AWSEC2Route Condition DoAz3 Properties RouteTableId Ref PrivateRouteTable3 DestinationCidrBlock 00000 NatGatewayId Ref NAT3 S3Endpoint Type AWSEC2VPCEndpoint Properties PolicyDocument Version 2012-10-17 Statement - Effect Allow Principal Action - Resource - RouteTableIds - Ref PublicRouteTable - Ref PrivateRouteTable - If [DoAz2 Ref PrivateRouteTable2 Ref AWSNoValue] - If [DoAz3 Ref PrivateRouteTable3 Ref AWSNoValue] ServiceName Join - - - comamazonaws - Ref AWSRegion - s3 VpcId Ref VPC

Outputs VpcId Description ID of the new VPC Value Ref VPC PublicSubnetIds Description Subnet IDs of the public subnets Value Join [ [Ref PublicSubnet If [DoAz2 Ref PublicSubnet2 Ref AWSNoValue] If [DoAz3 Ref PublicSubnet3 Ref AWSNoValue]] ] PrivateSubnetIds Description Subnet IDs of the private subnets Value Join [ [Ref PrivateSubnet If [DoAz2 Ref PrivateSubnet2 Ref AWSNoValue] If [DoAz3 Ref PrivateSubnet3 Ref AWSNoValue]] ]

You can view details about the CloudFormation stacks that you create by navigating to the AWSCloudFormation console

179 Creating networking and load balancing components in AWS

You must configure networking and classic or network load balancing in Amazon Web Services (AWS)that your OpenShift Container Platform cluster can use

You can use the provided CloudFormation template and a custom parameter file to create a stack ofAWS resources The stack represents the networking and load balancing components that yourOpenShift Container Platform cluster requires The template also creates a hosted zone and subnettags

You can run the template multiple times within a single Virtual Private Cloud (VPC)

Prerequisites

You created and configured a VPC and associated subnets in AWS

Procedure

1 Obtain the hosted zone ID for the Route 53 base domain that you specified in the install-configyaml file for your cluster You can obtain details about your hosted zone by running thefollowing command

For the ltroute53_domaingt specify the Route 53 base domain that you used when yougenerated the install-configyaml file for the cluster

Example output

In the example output the hosted zone ID is Z21IXYZABCZ2A4

$ aws route53 list-hosted-zones-by-name --dns-name ltroute53_domaingt 1

myclusterexamplecom False 100HOSTEDZONES 65F8F38E-2268-B835-E15C-AB55336FCBFA hostedzoneZ21IXYZABCZ2A4 myclusterexamplecom 10

A short representative cluster name to use for host names etc

Specify the cluster name that you used when you generated the install-configyaml filefor the cluster

The name for your cluster infrastructure that is encoded in your Ignition config files for thecluster

Specify the infrastructure name that you extracted from the Ignition config file metadatawhich has the format ltcluster-namegt-ltrandom-stringgt

The Route 53 public zone ID to register the targets with

Specify the Route 53 public zone ID which as a format similar to Z21IXYZABCZ2A4 Youcan obtain this value from the AWS console

The Route 53 zone to register the targets with

Specify the Route 53 base domain that you used when you generated the install-configyaml file for the cluster Do not include the trailing period () that is displayed in theAWS console

The public subnets that you created for your VPC

[ ParameterKey ClusterName 1 ParameterValue mycluster 2 ParameterKey InfrastructureName 3 ParameterValue mycluster-ltrandom_stringgt 4 ParameterKey HostedZoneId 5 ParameterValue ltrandom_stringgt 6 ParameterKey HostedZoneName 7 ParameterValue examplecom 8 ParameterKey PublicSubnets 9 ParameterValue subnet-ltrandom_stringgt 10 ParameterKey PrivateSubnets 11 ParameterValue subnet-ltrandom_stringgt 12 ParameterKey VpcId 13 ParameterValue vpc-ltrandom_stringgt 14 ]

Specify the PublicSubnetIds value from the output of the CloudFormation template forthe VPC

The private subnets that you created for your VPC

Specify the PrivateSubnetIds value from the output of the CloudFormation template forthe VPC

The VPC that you created for the cluster

Specify the VpcId value from the output of the CloudFormation template for the VPC

3 Copy the template from the CloudFormation template for the network and load balancerssection of this topic and save it as a YAML file on your computer This template describes thenetworking and load balancing objects that your cluster requires

IMPORTANT

If you are deploying your cluster to an AWS government region you must updatethe InternalApiServerRecord in the CloudFormation template to use CNAMErecords Records of type ALIAS are not supported for AWS government regions

4 Launch the CloudFormation template to create a stack of AWS resources that provide thenetworking and load balancing components

IMPORTANT

ltnamegt is the name for the CloudFormation stack such as cluster-dns You need thename of this stack if you remove the cluster

You must explicitly declare the CAPABILITY_NAMED_IAM capability because theprovided template creates some AWSIAMRole resources

Example output

$ aws cloudformation create-stack --stack-name ltnamegt 1 --template-body filelttemplategtyaml 2 --parameters fileltparametersgtjson 3 --capabilities CAPABILITY_NAMED_IAM 4

arnawscloudformationus-east-1269333783861stackcluster-dnscd3e5de0-2fd4-11eb-5cf0-12be5c33a183

PrivateHostedZoneId

Hosted zone ID for the private DNS

ExternalApiLoadBalancerName

Full name of the external API load balancer

InternalApiLoadBalancerName

Full name of the internal API load balancer

ApiServerDnsName

Full host name of the API server

RegisterNlbIpTargetsLambda

Lambda ARN useful to help registerderegister IP targets for these load balancers

ExternalApiTargetGroupArn

ARN of external API target group

InternalApiTargetGroupArn

ARN of internal API target group

InternalServiceTargetGroupArn

ARN of internal service target group

1791 CloudFormation template for the network and load balancers

You can use the following CloudFormation template to deploy the networking objects and loadbalancers that you need for your OpenShift Container Platform cluster

Example 124 CloudFormation template for the network and load balancers

AWSTemplateFormatVersion 2010-09-09Description Template for OpenShift Cluster Network Elements (Route53 amp LBs)

Parameters ClusterName

AllowedPattern ^([a-zA-Z][a-zA-Z0-9-]026)$ MaxLength 27 MinLength 1 ConstraintDescription Cluster name must be alphanumeric start with a letter and have a maximum of 27 characters Description A short representative cluster name to use for host names and other identifying names Type String InfrastructureName AllowedPattern ^([a-zA-Z][a-zA-Z0-9-]026)$ MaxLength 27 MinLength 1 ConstraintDescription Infrastructure name must be alphanumeric start with a letter and have a maximum of 27 characters Description A short unique cluster ID used to tag cloud resources and identify items owned or used by the cluster Type String HostedZoneId Description The Route53 public zone ID to register the targets with such as Z21IXYZABCZ2A4 Type String HostedZoneName Description The Route53 zone to register the targets with such as examplecom Omit the trailing period Type String Default examplecom PublicSubnets Description The internet-facing subnets Type ListltAWSEC2SubnetIdgt PrivateSubnets Description The internal subnets Type ListltAWSEC2SubnetIdgt VpcId Description The VPC-scoped resources will belong to this VPC Type AWSEC2VPCId

Metadata AWSCloudFormationInterface ParameterGroups - Label default Cluster Information Parameters - ClusterName - InfrastructureName - Label default Network Configuration Parameters - VpcId - PublicSubnets - PrivateSubnets - Label default DNS Parameters - HostedZoneName - HostedZoneId ParameterLabels

ClusterName default Cluster Name InfrastructureName default Infrastructure Name VpcId default VPC ID PublicSubnets default Public Subnets PrivateSubnets default Private Subnets HostedZoneName default Public Hosted Zone Name HostedZoneId default Public Hosted Zone ID

Resources ExtApiElb Type AWSElasticLoadBalancingV2LoadBalancer Properties Name Join [- [Ref InfrastructureName ext]] IpAddressType ipv4 Subnets Ref PublicSubnets Type network

IntApiElb Type AWSElasticLoadBalancingV2LoadBalancer Properties Name Join [- [Ref InfrastructureName int]] Scheme internal IpAddressType ipv4 Subnets Ref PrivateSubnets Type network

IntDns Type AWSRoute53HostedZone Properties HostedZoneConfig Comment Managed by CloudFormation Name Join [ [Ref ClusterName Ref HostedZoneName]] HostedZoneTags - Key Name Value Join [- [Ref InfrastructureName int]] - Key Join [ [kubernetesiocluster Ref InfrastructureName]] Value owned VPCs - VPCId Ref VpcId VPCRegion Ref AWSRegion

ExternalApiServerRecord Type AWSRoute53RecordSetGroup Properties Comment Alias record for the API server HostedZoneId Ref HostedZoneId RecordSets - Name Join [

[api Ref ClusterName Join [ [Ref HostedZoneName ]]] ] Type A AliasTarget HostedZoneId GetAtt ExtApiElbCanonicalHostedZoneID DNSName GetAtt ExtApiElbDNSName

InternalApiServerRecord Type AWSRoute53RecordSetGroup Properties Comment Alias record for the API server HostedZoneId Ref IntDns RecordSets - Name Join [ [api Ref ClusterName Join [ [Ref HostedZoneName ]]] ] Type A AliasTarget HostedZoneId GetAtt IntApiElbCanonicalHostedZoneID DNSName GetAtt IntApiElbDNSName - Name Join [ [api-int Ref ClusterName Join [ [Ref HostedZoneName ]]] ] Type A AliasTarget HostedZoneId GetAtt IntApiElbCanonicalHostedZoneID DNSName GetAtt IntApiElbDNSName

ExternalApiListener Type AWSElasticLoadBalancingV2Listener Properties DefaultActions - Type forward TargetGroupArn Ref ExternalApiTargetGroup LoadBalancerArn Ref ExtApiElb Port 6443 Protocol TCP

ExternalApiTargetGroup Type AWSElasticLoadBalancingV2TargetGroup Properties HealthCheckIntervalSeconds 10 HealthCheckPath readyz HealthCheckPort 6443 HealthCheckProtocol HTTPS HealthyThresholdCount 2 UnhealthyThresholdCount 2 Port 6443 Protocol TCP

TargetType ip VpcId Ref VpcId TargetGroupAttributes - Key deregistration_delaytimeout_seconds Value 60

InternalApiListener Type AWSElasticLoadBalancingV2Listener Properties DefaultActions - Type forward TargetGroupArn Ref InternalApiTargetGroup LoadBalancerArn Ref IntApiElb Port 6443 Protocol TCP

InternalApiTargetGroup Type AWSElasticLoadBalancingV2TargetGroup Properties HealthCheckIntervalSeconds 10 HealthCheckPath readyz HealthCheckPort 6443 HealthCheckProtocol HTTPS HealthyThresholdCount 2 UnhealthyThresholdCount 2 Port 6443 Protocol TCP TargetType ip VpcId Ref VpcId TargetGroupAttributes - Key deregistration_delaytimeout_seconds Value 60

InternalServiceInternalListener Type AWSElasticLoadBalancingV2Listener Properties DefaultActions - Type forward TargetGroupArn Ref InternalServiceTargetGroup LoadBalancerArn Ref IntApiElb Port 22623 Protocol TCP

InternalServiceTargetGroup Type AWSElasticLoadBalancingV2TargetGroup Properties HealthCheckIntervalSeconds 10 HealthCheckPath healthz HealthCheckPort 22623 HealthCheckProtocol HTTPS

HealthyThresholdCount 2 UnhealthyThresholdCount 2 Port 22623 Protocol TCP TargetType ip VpcId Ref VpcId TargetGroupAttributes - Key deregistration_delaytimeout_seconds Value 60

RegisterTargetLambdaIamRole Type AWSIAMRole Properties RoleName Join [- [Ref InfrastructureName nlb lambda role]] AssumeRolePolicyDocument Version 2012-10-17 Statement - Effect Allow Principal Service - lambdaamazonawscom Action - stsAssumeRole Path Policies - PolicyName Join [- [Ref InfrastructureName master policy]] PolicyDocument Version 2012-10-17 Statement - Effect Allow Action [ elasticloadbalancingRegisterTargets elasticloadbalancingDeregisterTargets ] Resource Ref InternalApiTargetGroup - Effect Allow Action [ elasticloadbalancingRegisterTargets elasticloadbalancingDeregisterTargets ] Resource Ref InternalServiceTargetGroup - Effect Allow Action [ elasticloadbalancingRegisterTargets elasticloadbalancingDeregisterTargets ] Resource Ref ExternalApiTargetGroup

RegisterNlbIpTargets Type AWSLambdaFunction Properties Handler indexhandler

Role FnGetAtt - RegisterTargetLambdaIamRole - Arn Code ZipFile | import json import boto3 import cfnresponse def handler(event context) elb = boto3client(elbv2) if event[RequestType] == Delete elbderegister_targets(TargetGroupArn=event[ResourceProperties][TargetArn]Targets=[Id event[ResourceProperties][TargetIp]]) elif event[RequestType] == Create elbregister_targets(TargetGroupArn=event[ResourceProperties][TargetArn]Targets=[Id event[ResourceProperties][TargetIp]]) responseData = cfnresponsesend(event context cfnresponseSUCCESS responseData event[ResourceProperties][TargetArn]+event[ResourceProperties][TargetIp]) Runtime python37 Timeout 120

RegisterSubnetTagsLambdaIamRole Type AWSIAMRole Properties RoleName Join [- [Ref InfrastructureName subnet-tags-lambda-role]] AssumeRolePolicyDocument Version 2012-10-17 Statement - Effect Allow Principal Service - lambdaamazonawscom Action - stsAssumeRole Path Policies - PolicyName Join [- [Ref InfrastructureName subnet-tagging-policy]] PolicyDocument Version 2012-10-17 Statement - Effect Allow Action [ ec2DeleteTags ec2CreateTags ] Resource arnawsec2subnet - Effect Allow Action [ ec2DescribeSubnets ec2DescribeTags ] Resource

RegisterSubnetTags Type AWSLambdaFunction Properties Handler indexhandler Role FnGetAtt - RegisterSubnetTagsLambdaIamRole - Arn Code ZipFile | import json import boto3 import cfnresponse def handler(event context) ec2_client = boto3client(ec2) if event[RequestType] == Delete for subnet_id in event[ResourceProperties][Subnets] ec2_clientdelete_tags(Resources=[subnet_id] Tags=[Key kubernetesiocluster + event[ResourceProperties][InfrastructureName]]) elif event[RequestType] == Create for subnet_id in event[ResourceProperties][Subnets] ec2_clientcreate_tags(Resources=[subnet_id] Tags=[Key kubernetesiocluster + event[ResourceProperties][InfrastructureName] Value shared]) responseData = cfnresponsesend(event context cfnresponseSUCCESS responseData event[ResourceProperties][InfrastructureName]+event[ResourceProperties][Subnets][0]) Runtime python37 Timeout 120

RegisterPublicSubnetTags Type CustomSubnetRegister Properties ServiceToken GetAtt RegisterSubnetTagsArn InfrastructureName Ref InfrastructureName Subnets Ref PublicSubnets

RegisterPrivateSubnetTags Type CustomSubnetRegister Properties ServiceToken GetAtt RegisterSubnetTagsArn InfrastructureName Ref InfrastructureName Subnets Ref PrivateSubnets

Outputs PrivateHostedZoneId Description Hosted zone ID for the private DNS which is required for private records Value Ref IntDns ExternalApiLoadBalancerName Description Full name of the external API load balancer Value GetAtt ExtApiElbLoadBalancerFullName InternalApiLoadBalancerName Description Full name of the internal API load balancer Value GetAtt IntApiElbLoadBalancerFullName ApiServerDnsName Description Full hostname of the API server which is required for the Ignition config files

IMPORTANT

If you are deploying your cluster to an AWS government region you must update the InternalApiServerRecord to use CNAME records Records of type ALIAS are notsupported for AWS government regions For example

You can view details about your hosted zones by navigating to the AWS Route 53 console

See Listing public hosted zones in the AWS documentation for more information about listingpublic hosted zones

1710 Creating security group and roles in AWS

You must create security groups and roles in Amazon Web Services (AWS) for your OpenShift ContainerPlatform cluster to use

You can use the provided CloudFormation template and a custom parameter file to create a stack ofAWS resources The stack represents the security groups and roles that your OpenShift ContainerPlatform cluster requires

Value Join [ [api-int Ref ClusterName Ref HostedZoneName]] RegisterNlbIpTargetsLambda Description Lambda ARN useful to help register or deregister IP targets for these load balancers Value GetAtt RegisterNlbIpTargetsArn ExternalApiTargetGroupArn Description ARN of the external API target group Value Ref ExternalApiTargetGroup InternalApiTargetGroupArn Description ARN of the internal API target group Value Ref InternalApiTargetGroup InternalServiceTargetGroupArn Description ARN of the internal service target group Value Ref InternalServiceTargetGroup

Type CNAMETTL 10ResourceRecords- GetAtt IntApiElbDNSName

Prerequisites

Procedure

Specify the CIDR block parameter that you used for the VPC that you defined in the form xxxx16-24

2 Copy the template from the CloudFormation template for security objects section of this

[ ParameterKey InfrastructureName 1 ParameterValue mycluster-ltrandom_stringgt 2 ParameterKey VpcCidr 3 ParameterValue 1000016 4 ParameterKey PrivateSubnets 5 ParameterValue subnet-ltrandom_stringgt 6 ParameterKey VpcId 7 ParameterValue vpc-ltrandom_stringgt 8 ]

2 Copy the template from the CloudFormation template for security objects section of thistopic and save it as a YAML file on your computer This template describes the security groupsand roles that your cluster requires

3 Launch the CloudFormation template to create a stack of AWS resources that represent thesecurity groups and roles

IMPORTANT

ltnamegt is the name for the CloudFormation stack such as cluster-sec You need thename of this stack if you remove the cluster

You must explicitly declare the CAPABILITY_NAMED_IAM capability because theprovided template creates some AWSIAMRole and AWSIAMInstanceProfileresources

Example output

MasterSecurityGroupId

Master Security Group ID

WorkerSecurityGroupId

Worker Security Group ID

arnawscloudformationus-east-1269333783861stackcluster-sec03bd4210-2ed7-11eb-6d7a-13fc0b61e9db

MasterInstanceProfile

Master IAM Instance Profile

WorkerInstanceProfile

Worker IAM Instance Profile

17101 CloudFormation template for security objects

You can use the following CloudFormation template to deploy the security objects that you need foryour OpenShift Container Platform cluster

Example 125 CloudFormation template for security objects

AWSTemplateFormatVersion 2010-09-09Description Template for OpenShift Cluster Security Elements (Security Groups amp IAM)

Parameters InfrastructureName AllowedPattern ^([a-zA-Z][a-zA-Z0-9-]026)$ MaxLength 27 MinLength 1 ConstraintDescription Infrastructure name must be alphanumeric start with a letter and have a maximum of 27 characters Description A short unique cluster ID used to tag cloud resources and identify items owned or used by the cluster Type String VpcCidr AllowedPattern ^(([0-9]|[1-9][0-9]|1[0-9]2|2[0-4][0-9]|25[0-5]))3([0-9]|[1-9][0-9]|1[0-9]2|2[0-4][0-9]|25[0-5])((1[6-9]|2[0-4]))$ ConstraintDescription CIDR block parameter must be in the form xxxx16-24 Default 1000016 Description CIDR block for VPC Type String VpcId Description The VPC-scoped resources will belong to this VPC Type AWSEC2VPCId PrivateSubnets Description The internal subnets Type ListltAWSEC2SubnetIdgt

Metadata AWSCloudFormationInterface ParameterGroups - Label default Cluster Information Parameters - InfrastructureName - Label default Network Configuration Parameters

- VpcId - VpcCidr - PrivateSubnets ParameterLabels InfrastructureName default Infrastructure Name VpcId default VPC ID VpcCidr default VPC CIDR PrivateSubnets default Private Subnets

Resources MasterSecurityGroup Type AWSEC2SecurityGroup Properties GroupDescription Cluster Master Security Group SecurityGroupIngress - IpProtocol icmp FromPort 0 ToPort 0 CidrIp Ref VpcCidr - IpProtocol tcp FromPort 22 ToPort 22 CidrIp Ref VpcCidr - IpProtocol tcp ToPort 6443 FromPort 6443 CidrIp Ref VpcCidr - IpProtocol tcp FromPort 22623 ToPort 22623 CidrIp Ref VpcCidr VpcId Ref VpcId

WorkerSecurityGroup Type AWSEC2SecurityGroup Properties GroupDescription Cluster Worker Security Group SecurityGroupIngress - IpProtocol icmp FromPort 0 ToPort 0 CidrIp Ref VpcCidr - IpProtocol tcp FromPort 22 ToPort 22 CidrIp Ref VpcCidr VpcId Ref VpcId

MasterIngressEtcd Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId

SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description etcd FromPort 2379 ToPort 2380 IpProtocol tcp

MasterIngressVxlan Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Vxlan packets FromPort 4789 ToPort 4789 IpProtocol udp

MasterIngressWorkerVxlan Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Vxlan packets FromPort 4789 ToPort 4789 IpProtocol udp

MasterIngressGeneve Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Geneve packets FromPort 6081 ToPort 6081 IpProtocol udp

MasterIngressWorkerGeneve Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Geneve packets FromPort 6081 ToPort 6081 IpProtocol udp

MasterIngressInternal Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Internal cluster communication FromPort 9000 ToPort 9999 IpProtocol tcp

MasterIngressWorkerInternal Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Internal cluster communication FromPort 9000 ToPort 9999 IpProtocol tcp

MasterIngressInternalUDP Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Internal cluster communication FromPort 9000 ToPort 9999 IpProtocol udp

MasterIngressWorkerInternalUDP Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Internal cluster communication FromPort 9000 ToPort 9999 IpProtocol udp

MasterIngressKube Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Kubernetes kubelet scheduler and controller manager FromPort 10250 ToPort 10259 IpProtocol tcp

MasterIngressWorkerKube Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Kubernetes kubelet scheduler and controller manager FromPort 10250 ToPort 10259 IpProtocol tcp

MasterIngressIngressServices Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Kubernetes ingress services

FromPort 30000 ToPort 32767 IpProtocol tcp

MasterIngressWorkerIngressServices Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Kubernetes ingress services FromPort 30000 ToPort 32767 IpProtocol tcp

MasterIngressIngressServicesUDP Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Kubernetes ingress services FromPort 30000 ToPort 32767 IpProtocol udp

MasterIngressWorkerIngressServicesUDP Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Kubernetes ingress services FromPort 30000 ToPort 32767 IpProtocol udp

WorkerIngressVxlan Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Vxlan packets FromPort 4789 ToPort 4789 IpProtocol udp

WorkerIngressMasterVxlan Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Vxlan packets FromPort 4789 ToPort 4789 IpProtocol udp

WorkerIngressGeneve Type AWSEC2SecurityGroupIngress

Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Geneve packets FromPort 6081 ToPort 6081 IpProtocol udp

WorkerIngressMasterGeneve Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Geneve packets FromPort 6081 ToPort 6081 IpProtocol udp

WorkerIngressInternal Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Internal cluster communication FromPort 9000 ToPort 9999 IpProtocol tcp

WorkerIngressMasterInternal Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Internal cluster communication FromPort 9000 ToPort 9999 IpProtocol tcp

WorkerIngressInternalUDP Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Internal cluster communication FromPort 9000 ToPort 9999 IpProtocol udp

WorkerIngressMasterInternalUDP Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Internal cluster communication FromPort 9000 ToPort 9999

IpProtocol udp

WorkerIngressKube Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Kubernetes secure kubelet port FromPort 10250 ToPort 10250 IpProtocol tcp

WorkerIngressWorkerKube Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Internal Kubernetes communication FromPort 10250 ToPort 10250 IpProtocol tcp

WorkerIngressIngressServices Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Kubernetes ingress services FromPort 30000 ToPort 32767 IpProtocol tcp

WorkerIngressMasterIngressServices Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Kubernetes ingress services FromPort 30000 ToPort 32767 IpProtocol tcp

WorkerIngressIngressServicesUDP Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Kubernetes ingress services FromPort 30000 ToPort 32767 IpProtocol udp

WorkerIngressMasterIngressServicesUDP Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId

SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Kubernetes ingress services FromPort 30000 ToPort 32767 IpProtocol udp

MasterIamRole Type AWSIAMRole Properties AssumeRolePolicyDocument Version 2012-10-17 Statement - Effect Allow Principal Service - ec2amazonawscom Action - stsAssumeRole Policies - PolicyName Join [- [Ref InfrastructureName master policy]] PolicyDocument Version 2012-10-17 Statement - Effect Allow Action - ec2AttachVolume - ec2AuthorizeSecurityGroupIngress - ec2CreateSecurityGroup - ec2CreateTags - ec2CreateVolume - ec2DeleteSecurityGroup - ec2DeleteVolume - ec2Describe - ec2DetachVolume - ec2ModifyInstanceAttribute - ec2ModifyVolume - ec2RevokeSecurityGroupIngress - elasticloadbalancingAddTags - elasticloadbalancingAttachLoadBalancerToSubnets - elasticloadbalancingApplySecurityGroupsToLoadBalancer - elasticloadbalancingCreateListener - elasticloadbalancingCreateLoadBalancer - elasticloadbalancingCreateLoadBalancerPolicy - elasticloadbalancingCreateLoadBalancerListeners - elasticloadbalancingCreateTargetGroup - elasticloadbalancingConfigureHealthCheck - elasticloadbalancingDeleteListener - elasticloadbalancingDeleteLoadBalancer - elasticloadbalancingDeleteLoadBalancerListeners - elasticloadbalancingDeleteTargetGroup - elasticloadbalancingDeregisterInstancesFromLoadBalancer - elasticloadbalancingDeregisterTargets - elasticloadbalancingDescribe - elasticloadbalancingDetachLoadBalancerFromSubnets - elasticloadbalancingModifyListener - elasticloadbalancingModifyLoadBalancerAttributes

- elasticloadbalancingModifyTargetGroup - elasticloadbalancingModifyTargetGroupAttributes - elasticloadbalancingRegisterInstancesWithLoadBalancer - elasticloadbalancingRegisterTargets - elasticloadbalancingSetLoadBalancerPoliciesForBackendServer - elasticloadbalancingSetLoadBalancerPoliciesOfListener - kmsDescribeKey Resource

MasterInstanceProfile Type AWSIAMInstanceProfile Properties Roles - Ref MasterIamRole

WorkerIamRole Type AWSIAMRole Properties AssumeRolePolicyDocument Version 2012-10-17 Statement - Effect Allow Principal Service - ec2amazonawscom Action - stsAssumeRole Policies - PolicyName Join [- [Ref InfrastructureName worker policy]] PolicyDocument Version 2012-10-17 Statement - Effect Allow Action - ec2DescribeInstances - ec2DescribeRegions Resource

WorkerInstanceProfile Type AWSIAMInstanceProfile Properties Roles - Ref WorkerIamRole

Outputs MasterSecurityGroupId Description Master Security Group ID Value GetAtt MasterSecurityGroupGroupId

WorkerSecurityGroupId Description Worker Security Group ID Value GetAtt WorkerSecurityGroupGroupId

MasterInstanceProfile Description Master IAM Instance Profile Value Ref MasterInstanceProfile

1711 RHCOS AMIs for the AWS infrastructure

Red Hat provides Red Hat Enterprise Linux CoreOS (RHCOS) AMIs valid for the various Amazon WebServices (AWS) zones you can specify for your OpenShift Container Platform nodes

You can also install to regions that do not have a RHCOS AMI published by importing yourown AMI

Table 118 RHCOS AMIs

AWS zone AWS AMI

af-south-1 ami-09921c9c1c36e695c

ap-east-1 ami-01ee8446e9af6b197

ap-northeast-1 ami-04e5b5722a55846ea

ap-northeast-2 ami-0fdc25c8a0273a742

ap-south-1 ami-09e3deb397cc526a8

ap-southeast-1 ami-0630e03f75e02eec4

ap-southeast-2 ami-069450613262ba03c

ca-central-1 ami-012518cdbd3057dfd

eu-central-1 ami-0bd7175ff5b1aef0c

eu-north-1 ami-06c9ec42d0a839ad2

eu-south-1 ami-0614d7440a0363d71

eu-west-1 ami-01b89df58b5d4d5fa

WorkerInstanceProfile Description Worker IAM Instance Profile Value Ref WorkerInstanceProfile

eu-west-2 ami-06f6e31ddd554f89d

eu-west-3 ami-0dc82e2517ded15a1

me-south-1 ami-07d181e3aa0f76067

sa-east-1 ami-0cd44e6dd20e6c7fa

us-east-1 ami-04a16d506e5b0e246

us-east-2 ami-0a1f868ad58ea59a7

us-west-1 ami-0a65d76e3a6f6622f

us-west-2 ami-0dd9008abadc519f1

AWS zone AWS AMI

You can deploy an OpenShift Container Platform cluster to Amazon Web Services (AWS) regionswithout native support for a Red Hat Enterprise Linux CoreOS (RHCOS) Amazon Machine Image (AMI)or the AWS software development kit (SDK) If a published AMI is not available for an AWS region youcan upload a custom AMI prior to installing the cluster This is required if you are deploying your clusterto an AWS government region

Prerequisites

You downloaded the AWS CLI and installed it on your computer See Install the AWS CLI Using

Procedure

Example output

1712 Creating the bootstrap node in AWS

ImportSnapshotTasks [ Description rhcos-460-x86_64-awsx86_64 ImportTaskId import-snap-fh6i8uil SnapshotTaskDetail Description rhcos-460-x86_64-awsx86_64 DiskImageSize 8190566400 Format VMDK SnapshotId snap-06331325870076318 Status completed UserBucket S3Bucket external-images S3Key rhcos-460-x86_64-awsx86_64vmdk ]

You must create the bootstrap node in Amazon Web Services (AWS) to use during OpenShift ContainerPlatform cluster initialization

You can use the provided CloudFormation template and a custom parameter file to create a stack ofAWS resources The stack represents the bootstrap node that your OpenShift Container Platforminstallation requires

If you do not use the provided CloudFormation template to create your bootstrap nodeyou must review the provided information and manually create the infrastructure If yourcluster does not initialize correctly you might have to contact Red Hat support with yourinstallation logs

Prerequisites

You created and configured DNS load balancers and listeners in AWS

You created the security groups and roles required for your cluster in AWS

Procedure

1 Provide a location to serve the bootstrapign Ignition config file to your cluster This file islocated in your installation directory One way to do this is to create an S3 bucket in yourclusterrsquos region and upload the Ignition config file to it

IMPORTANT

The provided CloudFormation Template assumes that the Ignition config filesfor your cluster are served from an S3 bucket If you choose to serve the filesfrom another location you must modify the templates

IMPORTANT

If you are deploying to a region that has endpoints that differ from the AWS SDKor you are providing your own custom endpoints you must use a presigned URLfor your S3 bucket instead of the s3 schema

The bootstrap Ignition config file does contain secrets like X509 keys Thefollowing steps provide basic security for the S3 bucket To provide additionalsecurity you can enable an S3 bucket policy to allow only certain users such asthe OpenShift IAM user to access objects that the bucket contains You canavoid S3 entirely and serve your bootstrap Ignition config file from any addressthat the bootstrap machine can reach

a Create the bucket

ltcluster-namegt-infra is the bucket name When creating the install-configyaml filereplace ltcluster-namegt with the name specified for the cluster

b Upload the bootstrapign Ignition config file to the bucket

c Verify that the file uploaded

Example output

$ aws s3 mb s3ltcluster-namegt-infra 1

$ aws s3 cp ltinstallation_directorygtbootstrapign s3ltcluster-namegt-infrabootstrapign 1

$ aws s3 ls s3ltcluster-namegt-infra

2019-04-03 161516 314878 bootstrapign

[ ParameterKey InfrastructureName 1 ParameterValue mycluster-ltrandom_stringgt 2 ParameterKey RhcosAmi 3 ParameterValue ami-ltrandom_stringgt 4 ParameterKey AllowedBootstrapSshCidr 5 ParameterValue 00000 6 ParameterKey PublicSubnet 7 ParameterValue subnet-ltrandom_stringgt 8 ParameterKey MasterSecurityGroupId 9 ParameterValue sg-ltrandom_stringgt 10 ParameterKey VpcId 11 ParameterValue vpc-ltrandom_stringgt 12

Current Red Hat Enterprise Linux CoreOS (RHCOS) AMI to use for the bootstrap node

Specify a valid AWSEC2ImageId value

CIDR block to allow SSH access to the bootstrap node

The public subnet that is associated with your VPC to launch the bootstrap node into

The master security group ID (for registering temporary rules)

Specify the MasterSecurityGroupId value from the output of the CloudFormationtemplate for the security group and roles

The VPC created resources will belong to

ParameterKey BootstrapIgnitionLocation 13 ParameterValue s3ltbucket_namegtbootstrapign 14 ParameterKey AutoRegisterELB 15 ParameterValue yes 16 ParameterKey RegisterNlbIpTargetsLambdaArn 17 ParameterValue arnawslambdaltregiongtltaccount_numbergtfunctionltdns_stack_namegt-RegisterNlbIpTargets-ltrandom_stringgt 18 ParameterKey ExternalApiTargetGroupArn 19 ParameterValue arnawselasticloadbalancingltregiongtltaccount_numbergttargetgroupltdns_stack_namegt-Exter-ltrandom_stringgt 20 ParameterKey InternalApiTargetGroupArn 21 ParameterValue arnawselasticloadbalancingltregiongtltaccount_numbergttargetgroupltdns_stack_namegt-Inter-ltrandom_stringgt 22 ParameterKey InternalServiceTargetGroupArn 23 ParameterValue arnawselasticloadbalancingltregiongtltaccount_numbergttargetgroupltdns_stack_namegt-Inter-ltrandom_stringgt 24 ]

Location to fetch bootstrap Ignition config file from

Specify the S3 bucket and file name in the form s3ltbucket_namegtbootstrapign

Whether or not to register a network load balancer (NLB)

Specify yes or no If you specify yes you must provide a Lambda Amazon Resource Name(ARN) value

The ARN for NLB IP target registration lambda group

Specify the RegisterNlbIpTargetsLambda value from the output of the CloudFormationtemplate for DNS and load balancing Use arnaws-us-gov if deploying the cluster to anAWS GovCloud region

The ARN for external API load balancer target group

Specify the ExternalApiTargetGroupArn value from the output of the CloudFormationtemplate for DNS and load balancing Use arnaws-us-gov if deploying the cluster to anAWS GovCloud region

The ARN for internal API load balancer target group

Specify the InternalApiTargetGroupArn value from the output of the CloudFormationtemplate for DNS and load balancing Use arnaws-us-gov if deploying the cluster to anAWS GovCloud region

The ARN for internal service load balancer target group

Specify the InternalServiceTargetGroupArn value from the output of theCloudFormation template for DNS and load balancing Use arnaws-us-gov if deployingthe cluster to an AWS GovCloud region

3 Copy the template from the CloudFormation template for the bootstrap machine section ofthis topic and save it as a YAML file on your computer This template describes the bootstrapmachine that your cluster requires

4 Launch the CloudFormation template to create a stack of AWS resources that represent thebootstrap node

IMPORTANT

ltnamegt is the name for the CloudFormation stack such as cluster-bootstrap You needthe name of this stack if you remove the cluster

Example output

BootstrapInstanceId

The bootstrap Instance ID

BootstrapPublicIp

The bootstrap node public IP address

BootstrapPrivateIp

The bootstrap node private IP address

17121 CloudFormation template for the bootstrap machine

You can use the following CloudFormation template to deploy the bootstrap machine that you need foryour OpenShift Container Platform cluster

Example 126 CloudFormation template for the bootstrap machine

arnawscloudformationus-east-1269333783861stackcluster-bootstrap12944486-2add-11eb-9dee-12dace8e3a83

AWSTemplateFormatVersion 2010-09-09Description Template for OpenShift Cluster Bootstrap (EC2 Instance Security Groups and IAM)

Parameters InfrastructureName AllowedPattern ^([a-zA-Z][a-zA-Z0-9-]026)$ MaxLength 27 MinLength 1 ConstraintDescription Infrastructure name must be alphanumeric start with a letter and have a maximum of 27 characters Description A short unique cluster ID used to tag cloud resources and identify items owned or used by the cluster Type String

RhcosAmi Description Current Red Hat Enterprise Linux CoreOS AMI to use for bootstrap Type AWSEC2ImageId AllowedBootstrapSshCidr AllowedPattern ^(([0-9]|[1-9][0-9]|1[0-9]2|2[0-4][0-9]|25[0-5]))3([0-9]|[1-9][0-9]|1[0-9]2|2[0-4][0-9]|25[0-5])(([0-9]|1[0-9]|2[0-9]|3[0-2]))$ ConstraintDescription CIDR block parameter must be in the form xxxx0-32 Default 00000 Description CIDR block to allow SSH access to the bootstrap node Type String PublicSubnet Description The public subnet to launch the bootstrap node into Type AWSEC2SubnetId MasterSecurityGroupId Description The master security group ID for registering temporary rules Type AWSEC2SecurityGroupId VpcId Description The VPC-scoped resources will belong to this VPC Type AWSEC2VPCId BootstrapIgnitionLocation Default s3my-s3-bucketbootstrapign Description Ignition config file location Type String AutoRegisterELB Default yes AllowedValues - yes - no Description Do you want to invoke NLB registration which requires a Lambda ARN parameter Type String RegisterNlbIpTargetsLambdaArn Description ARN for NLB IP target registration lambda Type String ExternalApiTargetGroupArn Description ARN for external API load balancer target group Type String InternalApiTargetGroupArn Description ARN for internal API load balancer target group Type String InternalServiceTargetGroupArn Description ARN for internal service load balancer target group Type String

Metadata AWSCloudFormationInterface ParameterGroups - Label default Cluster Information Parameters - InfrastructureName - Label default Host Information Parameters - RhcosAmi - BootstrapIgnitionLocation - MasterSecurityGroupId

- Label default Network Configuration Parameters - VpcId - AllowedBootstrapSshCidr - PublicSubnet - Label default Load Balancer Automation Parameters - AutoRegisterELB - RegisterNlbIpTargetsLambdaArn - ExternalApiTargetGroupArn - InternalApiTargetGroupArn - InternalServiceTargetGroupArn ParameterLabels InfrastructureName default Infrastructure Name VpcId default VPC ID AllowedBootstrapSshCidr default Allowed SSH Source PublicSubnet default Public Subnet RhcosAmi default Red Hat Enterprise Linux CoreOS AMI ID BootstrapIgnitionLocation default Bootstrap Ignition Source MasterSecurityGroupId default Master Security Group ID AutoRegisterELB default Use Provided ELB Automation

Conditions DoRegistration Equals [yes Ref AutoRegisterELB]

Resources BootstrapIamRole Type AWSIAMRole Properties AssumeRolePolicyDocument Version 2012-10-17 Statement - Effect Allow Principal Service - ec2amazonawscom Action - stsAssumeRole Path Policies - PolicyName Join [- [Ref InfrastructureName bootstrap policy]] PolicyDocument Version 2012-10-17 Statement - Effect Allow Action ec2Describe

Resource - Effect Allow Action ec2AttachVolume Resource - Effect Allow Action ec2DetachVolume Resource - Effect Allow Action s3GetObject Resource

BootstrapInstanceProfile Type AWSIAMInstanceProfile Properties Path Roles - Ref BootstrapIamRole

BootstrapSecurityGroup Type AWSEC2SecurityGroup Properties GroupDescription Cluster Bootstrap Security Group SecurityGroupIngress - IpProtocol tcp FromPort 22 ToPort 22 CidrIp Ref AllowedBootstrapSshCidr - IpProtocol tcp ToPort 19531 FromPort 19531 CidrIp 00000 VpcId Ref VpcId

BootstrapInstance Type AWSEC2Instance Properties ImageId Ref RhcosAmi IamInstanceProfile Ref BootstrapInstanceProfile InstanceType i3large NetworkInterfaces - AssociatePublicIpAddress true DeviceIndex 0 GroupSet - Ref BootstrapSecurityGroup - Ref MasterSecurityGroupId SubnetId Ref PublicSubnet UserData FnBase64 Sub - ignitionconfigreplacesource$S3Locversion310 - S3Loc Ref BootstrapIgnitionLocation

RegisterBootstrapApiTarget Condition DoRegistration Type CustomNLBRegister

See RHCOS AMIs for the AWS infrastructure for details about the Red Hat Enterprise LinuxCoreOS (RHCOS) AMIs for the AWS zones

1713 Creating the control plane machines in AWS

You must create the control plane machines in Amazon Web Services (AWS) that your cluster will use

You can use the provided CloudFormation template and a custom parameter file to create a stack ofAWS resources that represent the control plane nodes

IMPORTANT

The CloudFormation template creates a stack that represents three control plane nodes

Properties ServiceToken Ref RegisterNlbIpTargetsLambdaArn TargetArn Ref ExternalApiTargetGroupArn TargetIp GetAtt BootstrapInstancePrivateIp

RegisterBootstrapInternalApiTarget Condition DoRegistration Type CustomNLBRegister Properties ServiceToken Ref RegisterNlbIpTargetsLambdaArn TargetArn Ref InternalApiTargetGroupArn TargetIp GetAtt BootstrapInstancePrivateIp

RegisterBootstrapInternalServiceTarget Condition DoRegistration Type CustomNLBRegister Properties ServiceToken Ref RegisterNlbIpTargetsLambdaArn TargetArn Ref InternalServiceTargetGroupArn TargetIp GetAtt BootstrapInstancePrivateIp

Outputs BootstrapInstanceId Description Bootstrap Instance ID Value Ref BootstrapInstance

BootstrapPublicIp Description The bootstrap node public IP address Value GetAtt BootstrapInstancePublicIp

BootstrapPrivateIp Description The bootstrap node private IP address Value GetAtt BootstrapInstancePrivateIp

If you do not use the provided CloudFormation template to create your control planenodes you must review the provided information and manually create the infrastructureIf your cluster does not initialize correctly you might have to contact Red Hat supportwith your installation logs

Prerequisites

You created the bootstrap machine

Procedure

[ ParameterKey InfrastructureName 1 ParameterValue mycluster-ltrandom_stringgt 2 ParameterKey RhcosAmi 3 ParameterValue ami-ltrandom_stringgt 4 ParameterKey AutoRegisterDNS 5 ParameterValue yes 6 ParameterKey PrivateHostedZoneId 7 ParameterValue ltrandom_stringgt 8 ParameterKey PrivateHostedZoneName 9 ParameterValue myclusterexamplecom 10 ParameterKey Master0Subnet 11 ParameterValue subnet-ltrandom_stringgt 12 ParameterKey Master1Subnet 13

ParameterValue subnet-ltrandom_stringgt 14 ParameterKey Master2Subnet 15 ParameterValue subnet-ltrandom_stringgt 16 ParameterKey MasterSecurityGroupId 17 ParameterValue sg-ltrandom_stringgt 18 ParameterKey IgnitionLocation 19 ParameterValue httpsapi-intltcluster_namegtltdomain_namegt22623configmaster 20 ParameterKey CertificateAuthorities 21 ParameterValue datatextplaincharset=utf-8base64ABCxYz== 22 ParameterKey MasterInstanceProfileName 23 ParameterValue ltroles_stackgt-MasterInstanceProfile-ltrandom_stringgt 24 ParameterKey MasterInstanceType 25 ParameterValue m4xlarge 26 ParameterKey AutoRegisterELB 27 ParameterValue yes 28 ParameterKey RegisterNlbIpTargetsLambdaArn 29 ParameterValue arnawslambdaltregiongtltaccount_numbergtfunctionltdns_stack_namegt-RegisterNlbIpTargets-ltrandom_stringgt 30 ParameterKey ExternalApiTargetGroupArn 31 ParameterValue arnawselasticloadbalancingltregiongtltaccount_numbergttargetgroupltdns_stack_namegt-Exter-ltrandom_stringgt 32 ParameterKey InternalApiTargetGroupArn 33 ParameterValue arnawselasticloadbalancingltregiongtltaccount_numbergttargetgroupltdns_stack_namegt-Inter-ltrandom_stringgt 34 ParameterKey InternalServiceTargetGroupArn 35 ParameterValue arnawselasticloadbalancingltregiongtltaccount_numbergttargetgroupltdns_stack_namegt-Inter-ltrandom_stringgt 36 ]

11 13 15

12 14 16

CurrentRed Hat Enterprise Linux CoreOS (RHCOS) AMI to use for the control planemachines

Specify an AWSEC2ImageId value

Whether or not to perform DNS etcd registration

Specify yes or no If you specify yes you must provide hosted zone information

The Route 53 private zone ID to register the etcd targets with

Specify the PrivateHostedZoneId value from the output of the CloudFormation templatefor DNS and load balancing

Specify ltcluster_namegtltdomain_namegt where ltdomain_namegt is the Route 53 basedomain that you used when you generated install-configyaml file for the cluster Do notinclude the trailing period () that is displayed in the AWS console

A subnet preferably private to launch the control plane machines on

Specify a subnet from the PrivateSubnets value from the output of theCloudFormation template for DNS and load balancing

The master security group ID to associate with master nodes

The location to fetch control plane Ignition config file from

Specify the generated Ignition config file location httpsapi-intltcluster_namegtltdomain_namegt22623configmaster

The base64 encoded certificate authority string to use

Specify the value from the masterign file that is in the installation directory This value isthe long string with the format datatextplaincharset=utf-8base64ABChellip xYz==

The IAM profile to associate with master nodes

Specify the MasterInstanceProfile parameter value from the output of theCloudFormation template for the security group and roles

The type of AWS instance to use for the control plane machines

Allowed values

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c42xlarge

c44xlarge

c48xlarge

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

If m4 instance types are not available in your region such as with eu-west-3 specify an m5 type such as m5xlarge instead

2 Copy the template from the CloudFormation template for control plane machines section ofthis topic and save it as a YAML file on your computer This template describes the control planemachines that your cluster requires

3 If you specified an m5 instance type as the value for MasterInstanceType add that instancetype to the MasterInstanceTypeAllowedValues parameter in the CloudFormation template

4 Launch the CloudFormation template to create a stack of AWS resources that represent thecontrol plane nodes

IMPORTANT

ltnamegt is the name for the CloudFormation stack such as cluster-control-plane Youneed the name of this stack if you remove the cluster

Example output

The CloudFormation template creates a stack that represents three control planenodes

arnawscloudformationus-east-1269333783861stackcluster-control-plane21c7e2b0-2ee2-11eb-c6f6-0aa34627df4b

17131 CloudFormation template for control plane machines

You can use the following CloudFormation template to deploy the control plane machines that you needfor your OpenShift Container Platform cluster

Example 127 CloudFormation template for control plane machines

AWSTemplateFormatVersion 2010-09-09Description Template for OpenShift Cluster Node Launch (EC2 master instances)

Parameters InfrastructureName AllowedPattern ^([a-zA-Z][a-zA-Z0-9-]026)$ MaxLength 27 MinLength 1 ConstraintDescription Infrastructure name must be alphanumeric start with a letter and have a maximum of 27 characters Description A short unique cluster ID used to tag nodes for the kubelet cloud provider Type String RhcosAmi Description Current Red Hat Enterprise Linux CoreOS AMI to use for bootstrap Type AWSEC2ImageId AutoRegisterDNS Default yes AllowedValues - yes - no Description Do you want to invoke DNS etcd registration which requires Hosted Zone information Type String PrivateHostedZoneId Description The Route53 private zone ID to register the etcd targets with such as Z21IXYZABCZ2A4 Type String PrivateHostedZoneName Description The Route53 zone to register the targets with such as clusterexamplecom Omit the trailing period Type String Master0Subnet Description The subnets recommend private to launch the master nodes into Type AWSEC2SubnetId Master1Subnet Description The subnets recommend private to launch the master nodes into Type AWSEC2SubnetId Master2Subnet Description The subnets recommend private to launch the master nodes into Type AWSEC2SubnetId MasterSecurityGroupId Description The master security group ID to associate with master nodes Type AWSEC2SecurityGroupId IgnitionLocation Default httpsapi-int$CLUSTER_NAME$DOMAIN22623configmaster Description Ignition config file location Type String

CertificateAuthorities Default datatextplaincharset=utf-8base64ABCxYz== Description Base64 encoded certificate authority string to use Type String MasterInstanceProfileName Description IAM profile to associate with master nodes Type String MasterInstanceType Default m5xlarge Type String AllowedValues - m4xlarge - m42xlarge - m44xlarge - m48xlarge - m410xlarge - m416xlarge - m5xlarge - m52xlarge - m54xlarge - m58xlarge - m510xlarge - m516xlarge - c42xlarge - c44xlarge - c48xlarge - r4xlarge - r42xlarge - r44xlarge - r48xlarge - r416xlarge AutoRegisterELB Default yes AllowedValues - yes - no Description Do you want to invoke NLB registration which requires a Lambda ARN parameter Type String RegisterNlbIpTargetsLambdaArn Description ARN for NLB IP target registration lambda Supply the value from the cluster infrastructure or select no for AutoRegisterELB Type String ExternalApiTargetGroupArn Description ARN for external API load balancer target group Supply the value from the cluster infrastructure or select no for AutoRegisterELB Type String InternalApiTargetGroupArn Description ARN for internal API load balancer target group Supply the value from the cluster infrastructure or select no for AutoRegisterELB Type String InternalServiceTargetGroupArn Description ARN for internal service load balancer target group Supply the value from the cluster infrastructure or select no for AutoRegisterELB Type String

Metadata

AWSCloudFormationInterface ParameterGroups - Label default Cluster Information Parameters - InfrastructureName - Label default Host Information Parameters - MasterInstanceType - RhcosAmi - IgnitionLocation - CertificateAuthorities - MasterSecurityGroupId - MasterInstanceProfileName - Label default Network Configuration Parameters - VpcId - AllowedBootstrapSshCidr - Master0Subnet - Master1Subnet - Master2Subnet - Label default DNS Parameters - AutoRegisterDNS - PrivateHostedZoneName - PrivateHostedZoneId - Label default Load Balancer Automation Parameters - AutoRegisterELB - RegisterNlbIpTargetsLambdaArn - ExternalApiTargetGroupArn - InternalApiTargetGroupArn - InternalServiceTargetGroupArn ParameterLabels InfrastructureName default Infrastructure Name VpcId default VPC ID Master0Subnet default Master-0 Subnet Master1Subnet default Master-1 Subnet Master2Subnet default Master-2 Subnet MasterInstanceType default Master Instance Type MasterInstanceProfileName default Master Instance Profile Name RhcosAmi default Red Hat Enterprise Linux CoreOS AMI ID BootstrapIgnitionLocation default Master Ignition Source

CertificateAuthorities default Ignition CA String MasterSecurityGroupId default Master Security Group ID AutoRegisterDNS default Use Provided DNS Automation AutoRegisterELB default Use Provided ELB Automation PrivateHostedZoneName default Private Hosted Zone Name PrivateHostedZoneId default Private Hosted Zone ID

Conditions DoRegistration Equals [yes Ref AutoRegisterELB] DoDns Equals [yes Ref AutoRegisterDNS]

Resources Master0 Type AWSEC2Instance Properties ImageId Ref RhcosAmi BlockDeviceMappings - DeviceName devxvda Ebs VolumeSize 120 VolumeType gp2 IamInstanceProfile Ref MasterInstanceProfileName InstanceType Ref MasterInstanceType NetworkInterfaces - AssociatePublicIpAddress false DeviceIndex 0 GroupSet - Ref MasterSecurityGroupId SubnetId Ref Master0Subnet UserData FnBase64 Sub - ignitionconfigmerge[source$SOURCE]securitytlscertificateAuthorities[source$CA_BUNDLE]version310 - SOURCE Ref IgnitionLocation CA_BUNDLE Ref CertificateAuthorities Tags - Key Join [ [kubernetesiocluster Ref InfrastructureName]] Value shared

RegisterMaster0 Condition DoRegistration Type CustomNLBRegister Properties ServiceToken Ref RegisterNlbIpTargetsLambdaArn TargetArn Ref ExternalApiTargetGroupArn TargetIp GetAtt Master0PrivateIp

RegisterMaster0InternalApiTarget

Condition DoRegistration Type CustomNLBRegister Properties ServiceToken Ref RegisterNlbIpTargetsLambdaArn TargetArn Ref InternalApiTargetGroupArn TargetIp GetAtt Master0PrivateIp

RegisterMaster0InternalServiceTarget Condition DoRegistration Type CustomNLBRegister Properties ServiceToken Ref RegisterNlbIpTargetsLambdaArn TargetArn Ref InternalServiceTargetGroupArn TargetIp GetAtt Master0PrivateIp

Master1 Type AWSEC2Instance Properties ImageId Ref RhcosAmi BlockDeviceMappings - DeviceName devxvda Ebs VolumeSize 120 VolumeType gp2 IamInstanceProfile Ref MasterInstanceProfileName InstanceType Ref MasterInstanceType NetworkInterfaces - AssociatePublicIpAddress false DeviceIndex 0 GroupSet - Ref MasterSecurityGroupId SubnetId Ref Master1Subnet UserData FnBase64 Sub - ignitionconfigmerge[source$SOURCE]securitytlscertificateAuthorities[source$CA_BUNDLE]version310 - SOURCE Ref IgnitionLocation CA_BUNDLE Ref CertificateAuthorities Tags - Key Join [ [kubernetesiocluster Ref InfrastructureName]] Value shared

RegisterMaster1InternalApiTarget Condition DoRegistration Type CustomNLBRegister Properties

ServiceToken Ref RegisterNlbIpTargetsLambdaArn TargetArn Ref InternalApiTargetGroupArn TargetIp GetAtt Master1PrivateIp

RegisterMaster2InternalApiTarget Condition DoRegistration Type CustomNLBRegister Properties ServiceToken Ref RegisterNlbIpTargetsLambdaArn TargetArn Ref InternalApiTargetGroupArn TargetIp GetAtt Master2PrivateIp

EtcdSrvRecords Condition DoDns Type AWSRoute53RecordSet Properties HostedZoneId Ref PrivateHostedZoneId Name Join [ [_etcd-server-ssl_tcp Ref PrivateHostedZoneName]] ResourceRecords - Join [ [0 10 2380 Join [ [etcd-0 Ref PrivateHostedZoneName]]] ] - Join [ [0 10 2380 Join [ [etcd-1 Ref PrivateHostedZoneName]]] ] - Join [ [0 10 2380 Join [ [etcd-2 Ref PrivateHostedZoneName]]] ] TTL 60 Type SRV

Etcd0Record Condition DoDns Type AWSRoute53RecordSet Properties HostedZoneId Ref PrivateHostedZoneId Name Join [ [etcd-0 Ref PrivateHostedZoneName]] ResourceRecords - GetAtt Master0PrivateIp TTL 60 Type A

Etcd2Record Condition DoDns Type AWSRoute53RecordSet

1714 Creating the worker nodes in AWS

You can create worker nodes in Amazon Web Services (AWS) for your cluster to use

You can use the provided CloudFormation template and a custom parameter file to create a stack ofAWS resources that represent a worker node

IMPORTANT

The CloudFormation template creates a stack that represents one worker node Youmust create a stack for each worker node

If you do not use the provided CloudFormation template to create your worker nodesyou must review the provided information and manually create the infrastructure If yourcluster does not initialize correctly you might have to contact Red Hat support with yourinstallation logs

Prerequisites

Properties HostedZoneId Ref PrivateHostedZoneId Name Join [ [etcd-2 Ref PrivateHostedZoneName]] ResourceRecords - GetAtt Master2PrivateIp TTL 60 Type A

Outputs PrivateIPs Description The control-plane node private IP addresses Value Join [ [GetAtt Master0PrivateIp GetAtt Master1PrivateIp GetAtt Master2PrivateIp] ]

You created the control plane machines

Procedure

1 Create a JSON file that contains the parameter values that the CloudFormation templaterequires

[ ParameterKey InfrastructureName 1 ParameterValue mycluster-ltrandom_stringgt 2 ParameterKey RhcosAmi 3 ParameterValue ami-ltrandom_stringgt 4 ParameterKey Subnet 5 ParameterValue subnet-ltrandom_stringgt 6 ParameterKey WorkerSecurityGroupId 7 ParameterValue sg-ltrandom_stringgt 8 ParameterKey IgnitionLocation 9 ParameterValue httpsapi-intltcluster_namegtltdomain_namegt22623configworker 10 ParameterKey CertificateAuthorities 11 ParameterValue 12 ParameterKey WorkerInstanceProfileName 13 ParameterValue 14 ParameterKey WorkerInstanceType 15 ParameterValue m4large 16 ]

Current Red Hat Enterprise Linux CoreOS (RHCOS) AMI to use for the worker nodes

A subnet preferably private to launch the worker nodes on

Specify a subnet from the PrivateSubnets value from the output of the CloudFormationtemplate for DNS and load balancing

The worker security group ID to associate with worker nodes

Specify the WorkerSecurityGroupId value from the output of the CloudFormationtemplate for the security group and roles

The location to fetch bootstrap Ignition config file from

Specify the generated Ignition config location httpsapi-intltcluster_namegtltdomain_namegt22623configworker

Base64 encoded certificate authority string to use

Specify the value from the workerign file that is in the installation directory This value isthe long string with the format datatextplaincharset=utf-8base64ABChellip xYz==

The IAM profile to associate with worker nodes

Specify the WorkerInstanceProfile parameter value from the output of theCloudFormation template for the security group and roles

Allowed values

m4large

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5large

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c4large

c4xlarge

c42xlarge

c44xlarge

c48xlarge

r4large

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

If m4 instance types are not available in your region such as with eu-west-3 use m5 types instead

2 Copy the template from the CloudFormation template for worker machines section of thistopic and save it as a YAML file on your computer This template describes the networkingobjects and load balancers that your cluster requires

3 If you specified an m5 instance type as the value for WorkerInstanceType add that instancetype to the WorkerInstanceTypeAllowedValues parameter in the CloudFormation template

4 Launch the CloudFormation template to create a stack of AWS resources that represent aworker node

IMPORTANT

ltnamegt is the name for the CloudFormation stack such as cluster-worker-1 You needthe name of this stack if you remove the cluster

Example output

The CloudFormation template creates a stack that represents one worker node

6 Continue to create worker stacks until you have created enough worker machines for yourcluster You can create additional worker stacks by referencing the same template andparameter files and specifying a different stack name

IMPORTANT

You must create at least two worker machines so you must create at least twostacks that use this CloudFormation template

17141 CloudFormation template for worker machines

You can use the following CloudFormation template to deploy the worker machines that you need foryour OpenShift Container Platform cluster

Example 128 CloudFormation template for worker machines

arnawscloudformationus-east-1269333783861stackcluster-worker-1729ee301-1c2a-11eb-348f-sd9888c65b59

AWSTemplateFormatVersion 2010-09-09Description Template for OpenShift Cluster Node Launch (EC2 worker instance)

Parameters InfrastructureName AllowedPattern ^([a-zA-Z][a-zA-Z0-9-]026)$ MaxLength 27 MinLength 1 ConstraintDescription Infrastructure name must be alphanumeric start with a letter and have a maximum of 27 characters Description A short unique cluster ID used to tag nodes for the kubelet cloud provider Type String RhcosAmi Description Current Red Hat Enterprise Linux CoreOS AMI to use for bootstrap Type AWSEC2ImageId Subnet Description The subnets recommend private to launch the master nodes into Type AWSEC2SubnetId WorkerSecurityGroupId Description The master security group ID to associate with master nodes Type AWSEC2SecurityGroupId IgnitionLocation

Default httpsapi-int$CLUSTER_NAME$DOMAIN22623configworker Description Ignition config file location Type String CertificateAuthorities Default datatextplaincharset=utf-8base64ABCxYz== Description Base64 encoded certificate authority string to use Type String WorkerInstanceProfileName Description IAM profile to associate with master nodes Type String WorkerInstanceType Default m5large Type String AllowedValues - m4large - m4xlarge - m42xlarge - m44xlarge - m48xlarge - m410xlarge - m416xlarge - m5large - m5xlarge - m52xlarge - m54xlarge - m58xlarge - m510xlarge - m516xlarge - c4large - c4xlarge - c42xlarge - c44xlarge - c48xlarge - r4large - r4xlarge - r42xlarge - r44xlarge - r48xlarge - r416xlarge

Metadata AWSCloudFormationInterface ParameterGroups - Label default Cluster Information Parameters - InfrastructureName - Label default Host Information Parameters - WorkerInstanceType - RhcosAmi - IgnitionLocation - CertificateAuthorities - WorkerSecurityGroupId - WorkerInstanceProfileName

- Label default Network Configuration Parameters - Subnet ParameterLabels Subnet default Subnet InfrastructureName default Infrastructure Name WorkerInstanceType default Worker Instance Type WorkerInstanceProfileName default Worker Instance Profile Name RhcosAmi default Red Hat Enterprise Linux CoreOS AMI ID IgnitionLocation default Worker Ignition Source CertificateAuthorities default Ignition CA String WorkerSecurityGroupId default Worker Security Group ID

Resources Worker0 Type AWSEC2Instance Properties ImageId Ref RhcosAmi BlockDeviceMappings - DeviceName devxvda Ebs VolumeSize 120 VolumeType gp2 IamInstanceProfile Ref WorkerInstanceProfileName InstanceType Ref WorkerInstanceType NetworkInterfaces - AssociatePublicIpAddress false DeviceIndex 0 GroupSet - Ref WorkerSecurityGroupId SubnetId Ref Subnet UserData FnBase64 Sub - ignitionconfigmerge[source$SOURCE]securitytlscertificateAuthorities[source$CA_BUNDLE]version310 - SOURCE Ref IgnitionLocation CA_BUNDLE Ref CertificateAuthorities Tags - Key Join [ [kubernetesiocluster Ref InfrastructureName]] Value shared

Outputs PrivateIP Description The compute node private IP address Value GetAtt Worker0PrivateIp

1715 Initializing the bootstrap sequence on AWS with user-provisionedinfrastructure

After you create all of the required infrastructure in Amazon Web Services (AWS) you can start thebootstrap sequence that initializes the OpenShift Container Platform control plane

Prerequisites

You created the worker nodes

Procedure

1 Change to the directory that contains the installation program and start the bootstrap processthat initializes the OpenShift Container Platform control plane

Example output

$ openshift-install wait-for bootstrap-complete --dir=ltinstallation_directorygt 1 --log-level=info 2

INFO Waiting up to 20m0s for the Kubernetes API at httpsapimyclusterexamplecom6443INFO API v1190+9f84db3 upINFO Waiting up to 30m0s for bootstrapping to completeINFO It is now safe to remove the bootstrap resourcesINFO Time elapsed 1s

If the command exits without a FATAL warning your OpenShift Container Platform controlplane has initialized

After the control plane initializes it sets up the compute nodes and installsadditional services in the form of Operators

See Monitoring installation progress for details about monitoring the installation bootstrap andcontrol plane logs as an OpenShift Container Platform installation progresses

See Gathering bootstrap node diagnostic data for information about troubleshooting issuesrelated to the bootstrap process

You can view details about the running instances that are created by using the AWS EC2console

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

Procedure

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

$ echo $PATH

$ oc ltcommandgt

Prerequisites

Procedure

Example output

1718 Approving the certificate signing requests for your machines

When you add machines to a cluster two pending certificate signing requests (CSRs) are generated foreach machine that you added You must confirm that these CSRs are approved or if necessary approvethem yourself The client requests must be approved first followed by the server requests

Prerequisites

You added machines to your cluster

Procedure

1 Confirm that the cluster recognizes the machines

Example output

$ oc whoami

systemadmin

$ oc get nodes

NAME STATUS ROLES AGE VERSIONmaster-0 Ready master 63m v1190master-1 Ready master 63m v1190master-2 Ready master 64m v1190worker-0 NotReady worker 76s v1190worker-1 NotReady worker 70s v1190

The output lists all of the machines that you created

The preceding output might not include the compute nodes also known asworker nodes until some CSRs are approved

2 Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster

Example output

In this example two machines are joining the cluster You might see more approved CSRs in thelist

3 If the CSRs were not approved after all of the pending CSRs for the machines you added are in Pending status approve the CSRs for your cluster machines

Because the CSRs rotate automatically approve your CSRs within an hour ofadding the machines to the cluster If you do not approve them within an hour thecertificates will rotate and more than two certificates will be present for eachnode You must approve all of these certificates After you approve the initialCSRs the subsequent node client CSRs are automatically approved by thecluster kube-controller-manager

For clusters running on platforms that are not machine API enabled such as baremetal and other user-provisioned infrastructure you must implement a methodof automatically approving the kubelet serving certificate requests (CSRs) If arequest is not approved then the oc exec oc rsh and oc logs commandscannot succeed because a serving certificate is required when the API serverconnects to the kubelet Any operation that contacts the Kubelet endpointrequires this certificate approval to be in place The method must watch for newCSRs confirm that the CSR was submitted by the node-bootstrapper serviceaccount in the systemnode or systemadmin groups and confirm the identityof the node

To approve them individually run the following command for each valid CSR

$ oc get csr

NAME AGE REQUESTOR CONDITIONcsr-8b2br 15m systemserviceaccountopenshift-machine-config-operatornode-bootstrapper Pendingcsr-8vnps 15m systemserviceaccountopenshift-machine-config-operatornode-bootstrapper Pending

$ oc adm certificate approve ltcsr_namegt 1

ltcsr_namegt is the name of a CSR from the list of current CSRs

To approve all pending CSRs run the following command

Some Operators might not become available until some CSRs are approved

4 Now that your client requests are approved you must review the server requests for eachmachine that you added to the cluster

Example output

5 If the remaining CSRs are not approved and are in the Pending status approve the CSRs foryour cluster machines

6 After all client and server CSRs have been approved the machines have the Ready statusVerify this by running the following command

Example output

$ oc get csr -o go-template=range itemsif not statusmetadatanamenendend | xargs --no-run-if-empty oc adm certificate approve

$ oc get csr

NAME AGE REQUESTOR CONDITIONcsr-bfd72 5m26s systemnodeip-10-0-50-126us-east-2computeinternal Pendingcsr-c57lv 5m26s systemnodeip-10-0-95-157us-east-2computeinternal Pending

$ oc get csr -o go-template=range itemsif not statusmetadatanamenendend | xargs oc adm certificate approve

$ oc get nodes

NAME STATUS ROLES AGE VERSIONmaster-0 Ready master 73m v1200master-1 Ready master 73m v1200

It can take a few minutes after approval of the server CSRs for the machines totransition to the Ready status

Additional information

For more information on CSRs see Certificate Signing Requests

1719 Initial Operator configuration

After the control plane initializes you must immediately configure some Operators so that they allbecome available

Prerequisites

Your control plane has initialized

Procedure

1 Watch the cluster components come online

Example output

master-2 Ready master 74m v1200worker-0 Ready worker 11m v1200worker-1 Ready worker 11m v1200

$ watch -n5 oc get clusteroperators

NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCEauthentication 460 True False False 3h56mcloud-credential 460 True False False 29hcluster-autoscaler 460 True False False 29hconfig-operator 460 True False False 6h39mconsole 460 True False False 3h59mcsi-snapshot-controller 460 True False False 4h12mdns 460 True False False 4h15metcd 460 True False False 29himage-registry 460 True False False 3h59mingress 460 True False False 4h30minsights 460 True False False 29hkube-apiserver 460 True False False 29hkube-controller-manager 460 True False False 29hkube-scheduler 460 True False False 29hkube-storage-version-migrator 460 True False False 4h2mmachine-api 460 True False False 29hmachine-approver 460 True False False 6h34mmachine-config 460 True False False 3h56mmarketplace 460 True False False 4h2mmonitoring 460 True False False 6h31mnetwork 460 True False False 29hnode-tuning 460 True False False 4h30m

2 Configure the Operators that are not available

17191 Image registry storage configuration

Amazon Web Services provides default storage which means the Image Registry Operator is availableafter installation However if the Registry Operator cannot create an S3 bucket and automaticallyconfigure storage you must manually configure registry storage

Instructions are shown for configuring a persistent volume which is required for production clustersWhere applicable instructions are shown for configuring an empty directory as the storage locationwhich is available for only non-production clusters

Additional instructions are provided for allowing the image registry to use block storage types by usingthe Recreate rollout strategy during upgrades

You can configure registry storage for user-provisioned infrastructure in AWS to deploy OpenShiftContainer Platform to hidden regions See Configuring the registry for AWS user-provisionedinfrastructure for more information

171911 Configuring registry storage for AWS with user-provisioned infrastructure

During installation your cloud credentials are sufficient to create an Amazon S3 bucket and the RegistryOperator will automatically configure storage

If the Registry Operator cannot create an S3 bucket and automatically configure storage you cancreate an S3 bucket and configure storage with the following procedure

Prerequisites

You have a cluster on AWS with user-provisioned infrastructure

For Amazon S3 storage the secret is expected to contain two keys

REGISTRY_STORAGE_S3_ACCESSKEY

REGISTRY_STORAGE_S3_SECRETKEY

Procedure

Use the following procedure if the Registry Operator cannot create an S3 bucket and automaticallyconfigure storage

1 Set up a Bucket Lifecycle Policy to abort incomplete multipart uploads that are one day old

2 Fill in the storage configuration in configsimageregistryoperatoropenshiftiocluster

openshift-apiserver 460 True False False 3h56mopenshift-controller-manager 460 True False False 4h36mopenshift-samples 460 True False False 4h30moperator-lifecycle-manager 460 True False False 29hoperator-lifecycle-manager-catalog 460 True False False 29hoperator-lifecycle-manager-packageserver 460 True False False 3h59mservice-ca 460 True False False 29hstorage 460 True False False 4h30m

$ oc edit configsimageregistryoperatoropenshiftiocluster

Example configuration

WARNING

To secure your registry images in AWS block public access to the S3 bucket

171912 Configuring storage for the image registry in non-production clusters

You must configure storage for the Image Registry Operator For non-production clusters you can setthe image registry to an empty directory If you do so all images are lost if you restart the registry

Procedure

To set the image registry storage to an empty directory

WARNING

Configure this option for only non-production clusters

If you run this command before the Image Registry Operator initializes its components the oc patch command fails with the following error

Wait a few minutes and run the command again

1720 Deleting the bootstrap resources

After you complete the initial Operator configuration for the cluster remove the bootstrap resourcesfrom Amazon Web Services (AWS)

Prerequisites

You completed the initial Operator configuration for your cluster

storage s3 bucket ltbucket-namegt region ltregion-namegt

$ oc patch configsimageregistryoperatoropenshiftio cluster --type merge --patch specstorageemptyDir

Error from server (NotFound) configsimageregistryoperatoropenshiftio cluster not found

Procedure

1 Delete the bootstrap resources If you used the CloudFormation template delete its stack

Delete the stack by using the AWS CLI

ltnamegt is the name of your bootstrap stack

Delete the stack by using the AWS CloudFormation console

1721 Creating the Ingress DNS Records

If you removed the DNS Zone configuration manually create DNS records that point to the Ingress loadbalancer You can create either a wildcard record or specific records While the following procedure usesA records you can use other record types that you require such as CNAME or alias

Prerequisites

You deployed an OpenShift Container Platform cluster on Amazon Web Services (AWS) thatuses infrastructure that you provisioned

You installed the OpenShift CLI (oc)

You downloaded the AWS CLI and installed it on your computer See Install the AWS CLI Usingthe Bundled Installer (Linux macOS or Unix)

Procedure

1 Determine the routes to create

To create a wildcard record use appsltcluster_namegtltdomain_namegt where ltcluster_namegt is your cluster name and ltdomain_namegt is the Route 53 base domainfor your OpenShift Container Platform cluster

To create specific records you must create a record for each route that your cluster uses asshown in the output of the following command

Example output

2 Retrieve the Ingress Operator load balancer status and note the value of the external IP address

$ aws cloudformation delete-stack --stack-name ltnamegt 1

$ oc get --all-namespaces -o jsonpath=range items[]range statusingress[]hostnendend routes

oauth-openshiftappsltcluster_namegtltdomain_namegtconsole-openshift-consoleappsltcluster_namegtltdomain_namegtdownloads-openshift-consoleappsltcluster_namegtltdomain_namegtalertmanager-main-openshift-monitoringappsltcluster_namegtltdomain_namegtgrafana-openshift-monitoringappsltcluster_namegtltdomain_namegtprometheus-k8s-openshift-monitoringappsltcluster_namegtltdomain_namegt

2 Retrieve the Ingress Operator load balancer status and note the value of the external IP addressthat it uses which is shown in the EXTERNAL-IP column

Example output

3 Locate the hosted zone ID for the load balancer

For ltexternal_ipgt specify the value of the external IP address of the Ingress Operatorload balancer that you obtained

Example output

The output of this command is the load balancer hosted zone ID

4 Obtain the public hosted zone ID for your clusterrsquos domain

For ltdomain_namegt specify the Route 53 base domain for your OpenShift ContainerPlatform cluster

Example output

The public hosted zone ID for your domain is shown in the command output In this example it is Z3URY6TWQ91KVV

5 Add the alias records to your private zone

$ oc -n openshift-ingress get service router-default

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGErouter-default LoadBalancer 1723062215 ab328us-east-2elbamazonawscom 8031499TCP44330693TCP 5m

$ aws elb describe-load-balancers | jq -r LoadBalancerDescriptions[] | select(DNSName == ltexternal_ipgt)CanonicalHostedZoneNameID 1

Z3AADJGX6KTTL2

$ aws route53 list-hosted-zones-by-name --dns-name ltdomain_namegt 1 --query HostedZones[ ConfigPrivateZone = `true` ampamp Name == `ltdomain_namegt`]Id 2 --output text

hostedzoneZ3URY6TWQ91KVV

$ aws route53 change-resource-record-sets --hosted-zone-id ltprivate_hosted_zone_idgt --change-batch 1gt Changes [gt gt Action CREATE

For ltprivate_hosted_zone_idgt specify the value from the output of the CloudFormationtemplate for DNS and load balancing

For ltcluster_domaingt specify the domain or subdomain that you use with yourOpenShift Container Platform cluster

For lthosted_zone_idgt specify the public hosted zone ID for the load balancer that youobtained

For ltexternal_ipgt specify the value of the external IP address of the Ingress Operatorload balancer Ensure that you include the trailing period () in this parameter value

6 Add the records to your public zone

For ltpublic_hosted_zone_idgt specify the public hosted zone for your domain

For ltexternal_ipgt specify the value of the external IP address of the Ingress Operator

gt ResourceRecordSet gt Name 052appsltcluster_domaingt 2gt Type Agt AliasTargetgt HostedZoneId lthosted_zone_idgt 3gt DNSName ltexternal_ipgt 4gt EvaluateTargetHealth falsegt gt gt gt ]gt

$ aws route53 change-resource-record-sets --hosted-zone-id ltpublic_hosted_zone_idgt --change-batch 1gt Changes [gt gt Action CREATEgt ResourceRecordSet gt Name 052appsltcluster_domaingt 2gt Type Agt AliasTargetgt HostedZoneId lthosted_zone_idgt 3gt DNSName ltexternal_ipgt 4gt EvaluateTargetHealth falsegt gt gt gt ]gt

1722 Completing an AWS installation on user-provisioned infrastructure

After you start the OpenShift Container Platform installation on Amazon Web Service (AWS) user-provisioned infrastructure monitor the deployment to completion

Prerequisites

You removed the bootstrap node for an OpenShift Container Platform cluster on user-provisioned AWS infrastructure

Procedure

From the directory that contains the installation program complete the cluster installation

Example output

IMPORTANT

The kubeadmin user exists by default after an OpenShift Container Platform installation You can log

$ openshift-install --dir=ltinstallation_directorygt wait-for install-complete 1

INFO Waiting up to 40m0s for the cluster at httpsapimyclusterexamplecom6443 to initializeINFO Waiting up to 10m0s for the openshift-console route to be createdINFO Install completeINFO To access the cluster as the systemadmin user when using oc run export KUBECONFIG=homemyuserinstall_dirauthkubeconfigINFO Access the OpenShift web-console here httpsconsole-openshift-consoleappsmyclusterexamplecomINFO Login to the console with user kubeadmin and password 4vYBz-Fe5en-ymBEc-Wt6NLINFO Time elapsed 1s

Prerequisites

Procedure

Example output

1724 Additional resources

See Working with stacks in the AWS documentation for more information about AWSCloudFormation stacks

1725 Next steps

18 INSTALLING A CLUSTER ON AWS THAT USES MIRROREDINSTALLATION CONTENT

In OpenShift Container Platform version 46 you can install a cluster on Amazon Web Services (AWS)using infrastructure that you provide and an internal mirror of the installation release content

IMPORTANT

While you can install an OpenShift Container Platform cluster by using mirroredinstallation release content your cluster still requires Internet access to use the AWSAPIs

IMPORTANT

181 Prerequisites

You created a mirror registry on your mirror host and obtained the imageContentSources datafor your version of OpenShift Container Platform

IMPORTANT

Because the installation media is on the mirror host you can use that computerto complete all installation steps

IMPORTANT

If you use a firewall and plan to use the Telemetry service you configured the firewall to allowthe sites that your cluster requires access to

182 About installations in restricted networks

In OpenShift Container Platform 46 you can perform an installation that does not require an activeconnection to the Internet to obtain software components Restricted network installations can becompleted using installer-provisioned infrastructure or user-provisioned infrastructure depending onthe cloud platform to which you are installing the cluster

If you choose to perform a restricted network installation on a cloud platform you still require access toits cloud APIs Some cloud functions like Amazon Web Servicersquos IAM service require Internet access soyou might still require Internet access Depending on your network you might require less Internetaccess for an installation on bare metal hardware or on VMware vSphere

To complete a restricted network installation you must create a registry that mirrors the contents of theOpenShift Container Platform registry and contains the installation media You can create this registryon a mirror host which can access both the Internet and your closed network or by using other methodsthat meet your restrictions

IMPORTANT

Because of the complexity of the configuration for user-provisioned installationsconsider completing a standard user-provisioned infrastructure installation before youattempt a restricted network installation using user-provisioned infrastructureCompleting this test installation might make it easier to isolate and troubleshoot anyissues that might arise during your installation in a restricted network

1821 Additional limits

Clusters in restricted networks have the following additional limitations and restrictions

The ClusterVersion status includes an Unable to retrieve available updates error

By default you cannot use the contents of the Developer Catalog because you cannot accessthe required image stream tags

IMPORTANT

i3large x

m4large x

m4xlarge x x

m42xlarge x x

m44xlarge x x

m48xlarge x x

m410xlarge x x

m416xlarge x x

m5large x

m5xlarge x x

m52xlarge x x

m54xlarge x x

m58xlarge x x

m510xlarge x x

m516xlarge x x

c4large x

c4xlarge x

c42xlarge x x

c44xlarge x x

c48xlarge x x

r4large x

r4xlarge x x

r42xlarge x x

r44xlarge x x

r48xlarge x x

r416xlarge x x

DNS entries

Security groups

IAM roles

S3 buckets

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

etcd recordsets

AWSRoute53RecordSet

Public loadbalancer

Externallistener

Security groups

MasterSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

tcp 6443

tcp 22623

WorkerSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

AWSEC2SecurityGroup

tcp 22

tcp 19531

MasterIngressEtcd

etcd tcp 2379- 2380

MasterIngressVxlan

tcp 9000 - 9999

MasterIngressKube

tcp 10250 - 10259

Worker Ingress

WorkerIngressVxlan

WorkerIngressKube

tcp 10250

Master Allow ec2

Allow iamPassRole

Allow s3GetObject

tagTagResources

tagUntagResources

ec2AllocateAddress

ec2AssociateAddress

ec2CopyImage

ec2CreateTags

ec2CreateVolume

ec2DeleteSnapshot

ec2DeregisterImage

ec2DescribeImages

ec2DescribeKeyPairs

ec2DescribeRegions

ec2DescribeSubnets

ec2DescribeTags

ec2DescribeVolumes

ec2DescribeVpcs

ec2ReleaseAddress

ec2RunInstances

ec2CreateNatGateway

ec2CreateRoute

ec2CreateRouteTable

ec2CreateSubnet

ec2CreateVpc

iamCreateRole

iamDeleteRole

iamDeleteRolePolicy

iamGetRole

iamGetRolePolicy

iamGetUser

iamListRoles

iamListUsers

iamPassRole

iamPutRolePolicy

iamTagRole

route53GetChange

s3CreateBucket

s3DeleteBucket

s3GetBucketAcl

s3GetBucketCors

s3GetBucketLocation

s3GetBucketLogging

s3GetBucketTagging

s3GetBucketWebsite

s3ListBucket

s3PutBucketAcl

s3PutBucketTagging

s3DeleteObject

s3GetObject

s3GetObjectAcl

s3GetObjectTagging

s3GetObjectVersion

s3PutObject

s3PutObjectAcl

s3PutObjectTagging

ec2DeleteVolume

iamDeleteAccessKey

iamDeleteUser

iamListRolePolicies

iamListUserPolicies

s3DeleteObject

tagGetResources

ec2DeleteNatGateway

ec2DeleteRoute

ec2DeleteRouteTable

ec2DeleteSubnet

ec2DeleteVpc

iamCreateAccessKey

iamCreateUser

iamDeleteAccessKey

iamDeleteUser

iamDeleteUserPolicy

iamGetUserPolicy

iamListAccessKeys

iamPutUserPolicy

iamTagUser

iamGetUserPolicy

iamListAccessKeys

s3HeadBucket

Procedure

Example output

Next steps

Agent pid 31874

IMPORTANT

Procedure

Prerequisites

You obtained the OpenShift Container Platform installation program for user-provisionedinfrastructure and the pull secret for your cluster For a restricted network installation thesefiles are on your mirror host

Procedure

IMPORTANT

2 Edit the install-configyaml file to provide the additional information that is required for aninstallation in a restricted network

a Update the pullSecret value to contain the authentication information for your registry

For ltlocal_registrygt specify the registry domain name and optionally the port that yourmirror registry uses to serve content For example registryexamplecom or registryexamplecom5000 For ltcredentialsgt specify the base64-encoded user nameand password for your mirror registry

b Add the additionalTrustBundle parameter and value The value must be the contents ofthe certificate file that you used for your mirror registry which can be an exiting trustedcertificate authority or the self-signed certificate that you generated for the mirror registry

c Add the image content resources

Use the imageContentSources section from the output of the command to mirror therepository or the values that you used when you mirrored the content from the media thatyou brought into your restricted network

pullSecret authsltlocal_registrygt auth ltcredentialsgtemail youexamplecom

additionalTrustBundle | -----BEGIN CERTIFICATE----- ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ -----END CERTIFICATE-----

imageContentSources- mirrors - ltlocal_registrygtltlocal_repository_namegtrelease source quayioopenshift-release-devocp-release- mirrors - ltlocal_registrygtltlocal_repository_namegtrelease source registrysvcciopenshiftorgocprelease

d Optional Set the publishing strategy to Internal

By setting this option you create an internal Ingress Controller and a private load balancer

IMPORTANT

Prerequisites

Procedure

publish Internal

apiVersion v1baseDomain mydomaincomproxy httpProxy httpltusernamegtltpswdgtltipgtltportgt 1 httpsProxy httpltusernamegtltpswdgtltipgtltportgt 2 noProxy examplecom 3

additionalTrustBundle | 4 -----BEGIN CERTIFICATE----- ltMY_TRUSTED_CA_CERTgt -----END CERTIFICATE-----

IMPORTANT

Prerequisites

You obtained the OpenShift Container Platform installation program For a restricted networkinstallation these files are on your mirror host

Procedure

Example output

Prerequisites

You obtained the OpenShift Container Platform installation program and the pull secret for

Procedure

Example output

Prerequisites

Procedure

openshift-vw9j6 1

IMPORTANT

Example output

ParameterKey VpcCidr 1 ParameterValue 1000016 2 ParameterKey AvailabilityZoneCount 3 ParameterValue 1 4 ParameterKey SubnetBits 5 ParameterValue 12 6 ]

PublicSubnetIds

PrivateSubnetIds

Parameters VpcCidr AllowedPattern ^(([0-9]|[1-9][0-9]|1[0-9]2|2[0-4][0-9]|25[0-5]))3([0-9]|[1-9][0-9]|1[0-9]2|2[0-4][0-9]|25[0-5])((1[6-9]|2[0-4]))$ ConstraintDescription CIDR block parameter must be in the form xxxx16-24 Default 1000016 Description CIDR block for VPC Type String AvailabilityZoneCount ConstraintDescription The number of availability zones (Min 1 Max 3) MinValue 1 MaxValue 3 Default 1 Description How many AZs to create VPC subnets for (Min 1 Max 3) Type Number SubnetBits ConstraintDescription CIDR block parameter must be in the form xxxx19-27 MinValue 5 MaxValue 13 Default 12 Description Size of each subnet to create within the availability zones (Min 5 = 27 Max 13 = 19) Type Number

Resources VPC Type AWSEC2VPC Properties EnableDnsSupport true EnableDnsHostnames true CidrBlock Ref VpcCidr PublicSubnet Type AWSEC2Subnet Properties VpcId Ref VPC CidrBlock Select [0 Cidr [Ref VpcCidr 6 Ref SubnetBits]] AvailabilityZone Select - 0 - FnGetAZs Ref AWSRegion PublicSubnet2 Type AWSEC2Subnet Condition DoAz2 Properties VpcId Ref VPC CidrBlock Select [1 Cidr [Ref VpcCidr 6 Ref SubnetBits]] AvailabilityZone Select - 1 - FnGetAZs Ref AWSRegion PublicSubnet3 Type AWSEC2Subnet Condition DoAz3 Properties VpcId Ref VPC CidrBlock Select [2 Cidr [Ref VpcCidr 6 Ref SubnetBits]] AvailabilityZone Select

- 2 - FnGetAZs Ref AWSRegion InternetGateway Type AWSEC2InternetGateway GatewayToInternet Type AWSEC2VPCGatewayAttachment Properties VpcId Ref VPC InternetGatewayId Ref InternetGateway PublicRouteTable Type AWSEC2RouteTable Properties VpcId Ref VPC PublicRoute Type AWSEC2Route DependsOn GatewayToInternet Properties RouteTableId Ref PublicRouteTable DestinationCidrBlock 00000 GatewayId Ref InternetGateway PublicSubnetRouteTableAssociation Type AWSEC2SubnetRouteTableAssociation Properties SubnetId Ref PublicSubnet RouteTableId Ref PublicRouteTable PublicSubnetRouteTableAssociation2 Type AWSEC2SubnetRouteTableAssociation Condition DoAz2 Properties SubnetId Ref PublicSubnet2 RouteTableId Ref PublicRouteTable PublicSubnetRouteTableAssociation3 Condition DoAz3 Type AWSEC2SubnetRouteTableAssociation Properties SubnetId Ref PublicSubnet3 RouteTableId Ref PublicRouteTable PrivateSubnet Type AWSEC2Subnet Properties VpcId Ref VPC CidrBlock Select [3 Cidr [Ref VpcCidr 6 Ref SubnetBits]] AvailabilityZone Select - 0 - FnGetAZs Ref AWSRegion PrivateRouteTable Type AWSEC2RouteTable Properties VpcId Ref VPC PrivateSubnetRouteTableAssociation Type AWSEC2SubnetRouteTableAssociation Properties SubnetId Ref PrivateSubnet RouteTableId Ref PrivateRouteTable NAT DependsOn

- GatewayToInternet Type AWSEC2NatGateway Properties AllocationId FnGetAtt - EIP - AllocationId SubnetId Ref PublicSubnet EIP Type AWSEC2EIP Properties Domain vpc Route Type AWSEC2Route Properties RouteTableId Ref PrivateRouteTable DestinationCidrBlock 00000 NatGatewayId Ref NAT PrivateSubnet2 Type AWSEC2Subnet Condition DoAz2 Properties VpcId Ref VPC CidrBlock Select [4 Cidr [Ref VpcCidr 6 Ref SubnetBits]] AvailabilityZone Select - 1 - FnGetAZs Ref AWSRegion PrivateRouteTable2 Type AWSEC2RouteTable Condition DoAz2 Properties VpcId Ref VPC PrivateSubnetRouteTableAssociation2 Type AWSEC2SubnetRouteTableAssociation Condition DoAz2 Properties SubnetId Ref PrivateSubnet2 RouteTableId Ref PrivateRouteTable2 NAT2 DependsOn - GatewayToInternet Type AWSEC2NatGateway Condition DoAz2 Properties AllocationId FnGetAtt - EIP2 - AllocationId SubnetId Ref PublicSubnet2 EIP2 Type AWSEC2EIP Condition DoAz2 Properties Domain vpc

Route2 Type AWSEC2Route Condition DoAz2 Properties RouteTableId Ref PrivateRouteTable2 DestinationCidrBlock 00000 NatGatewayId Ref NAT2 PrivateSubnet3 Type AWSEC2Subnet Condition DoAz3 Properties VpcId Ref VPC CidrBlock Select [5 Cidr [Ref VpcCidr 6 Ref SubnetBits]] AvailabilityZone Select - 2 - FnGetAZs Ref AWSRegion PrivateRouteTable3 Type AWSEC2RouteTable Condition DoAz3 Properties VpcId Ref VPC PrivateSubnetRouteTableAssociation3 Type AWSEC2SubnetRouteTableAssociation Condition DoAz3 Properties SubnetId Ref PrivateSubnet3 RouteTableId Ref PrivateRouteTable3 NAT3 DependsOn - GatewayToInternet Type AWSEC2NatGateway Condition DoAz3 Properties AllocationId FnGetAtt - EIP3 - AllocationId SubnetId Ref PublicSubnet3 EIP3 Type AWSEC2EIP Condition DoAz3 Properties Domain vpc Route3 Type AWSEC2Route Condition DoAz3 Properties RouteTableId Ref PrivateRouteTable3 DestinationCidrBlock 00000 NatGatewayId Ref NAT3 S3Endpoint Type AWSEC2VPCEndpoint

Properties PolicyDocument Version 2012-10-17 Statement - Effect Allow Principal Action - Resource - RouteTableIds - Ref PublicRouteTable - Ref PrivateRouteTable - If [DoAz2 Ref PrivateRouteTable2 Ref AWSNoValue] - If [DoAz3 Ref PrivateRouteTable3 Ref AWSNoValue] ServiceName Join - - - comamazonaws - Ref AWSRegion - s3 VpcId Ref VPC

Prerequisites

Procedure

Example output

[ ParameterKey ClusterName 1 ParameterValue mycluster 2 ParameterKey InfrastructureName 3 ParameterValue mycluster-ltrandom_stringgt 4 ParameterKey HostedZoneId 5 ParameterValue ltrandom_stringgt 6 ParameterKey HostedZoneName 7

ParameterValue examplecom 8 ParameterKey PublicSubnets 9 ParameterValue subnet-ltrandom_stringgt 10 ParameterKey PrivateSubnets 11 ParameterValue subnet-ltrandom_stringgt 12 ParameterKey VpcId 13 ParameterValue vpc-ltrandom_stringgt 14 ]

IMPORTANT

Example output

PrivateHostedZoneId

ApiServerDnsName

Parameters ClusterName AllowedPattern ^([a-zA-Z][a-zA-Z0-9-]026)$ MaxLength 27 MinLength 1 ConstraintDescription Cluster name must be alphanumeric start with a letter and have a maximum of 27 characters Description A short representative cluster name to use for host names and other identifying names Type String InfrastructureName AllowedPattern ^([a-zA-Z][a-zA-Z0-9-]026)$ MaxLength 27 MinLength 1 ConstraintDescription Infrastructure name must be alphanumeric start with a letter and have a maximum of 27 characters Description A short unique cluster ID used to tag cloud resources and identify items owned or used by the cluster

Type String HostedZoneId Description The Route53 public zone ID to register the targets with such as Z21IXYZABCZ2A4 Type String HostedZoneName Description The Route53 zone to register the targets with such as examplecom Omit the trailing period Type String Default examplecom PublicSubnets Description The internet-facing subnets Type ListltAWSEC2SubnetIdgt PrivateSubnets Description The internal subnets Type ListltAWSEC2SubnetIdgt VpcId Description The VPC-scoped resources will belong to this VPC Type AWSEC2VPCId

Metadata AWSCloudFormationInterface ParameterGroups - Label default Cluster Information Parameters - ClusterName - InfrastructureName - Label default Network Configuration Parameters - VpcId - PublicSubnets - PrivateSubnets - Label default DNS Parameters - HostedZoneName - HostedZoneId ParameterLabels ClusterName default Cluster Name InfrastructureName default Infrastructure Name VpcId default VPC ID PublicSubnets default Public Subnets PrivateSubnets default Private Subnets HostedZoneName default Public Hosted Zone Name HostedZoneId default Public Hosted Zone ID

Resources

ExtApiElb Type AWSElasticLoadBalancingV2LoadBalancer Properties Name Join [- [Ref InfrastructureName ext]] IpAddressType ipv4 Subnets Ref PublicSubnets Type network

ExternalApiServerRecord Type AWSRoute53RecordSetGroup Properties Comment Alias record for the API server HostedZoneId Ref HostedZoneId RecordSets - Name Join [ [api Ref ClusterName Join [ [Ref HostedZoneName ]]] ] Type A AliasTarget HostedZoneId GetAtt ExtApiElbCanonicalHostedZoneID DNSName GetAtt ExtApiElbDNSName

InternalApiServerRecord Type AWSRoute53RecordSetGroup Properties Comment Alias record for the API server HostedZoneId Ref IntDns RecordSets - Name Join [

[api Ref ClusterName Join [ [Ref HostedZoneName ]]] ] Type A AliasTarget HostedZoneId GetAtt IntApiElbCanonicalHostedZoneID DNSName GetAtt IntApiElbDNSName - Name Join [ [api-int Ref ClusterName Join [ [Ref HostedZoneName ]]] ] Type A AliasTarget HostedZoneId GetAtt IntApiElbCanonicalHostedZoneID DNSName GetAtt IntApiElbDNSName

ExternalApiTargetGroup Type AWSElasticLoadBalancingV2TargetGroup Properties HealthCheckIntervalSeconds 10 HealthCheckPath readyz HealthCheckPort 6443 HealthCheckProtocol HTTPS HealthyThresholdCount 2 UnhealthyThresholdCount 2 Port 6443 Protocol TCP TargetType ip VpcId Ref VpcId TargetGroupAttributes - Key deregistration_delaytimeout_seconds Value 60

InternalApiListener Type AWSElasticLoadBalancingV2Listener Properties DefaultActions - Type forward TargetGroupArn Ref InternalApiTargetGroup LoadBalancerArn Ref IntApiElb

Port 6443 Protocol TCP

InternalServiceTargetGroup Type AWSElasticLoadBalancingV2TargetGroup Properties HealthCheckIntervalSeconds 10 HealthCheckPath healthz HealthCheckPort 22623 HealthCheckProtocol HTTPS HealthyThresholdCount 2 UnhealthyThresholdCount 2 Port 22623 Protocol TCP TargetType ip VpcId Ref VpcId TargetGroupAttributes - Key deregistration_delaytimeout_seconds Value 60

RegisterTargetLambdaIamRole Type AWSIAMRole Properties RoleName Join [- [Ref InfrastructureName nlb lambda role]] AssumeRolePolicyDocument

Version 2012-10-17 Statement - Effect Allow Principal Service - lambdaamazonawscom Action - stsAssumeRole Path Policies - PolicyName Join [- [Ref InfrastructureName master policy]] PolicyDocument Version 2012-10-17 Statement - Effect Allow Action [ elasticloadbalancingRegisterTargets elasticloadbalancingDeregisterTargets ] Resource Ref InternalApiTargetGroup - Effect Allow Action [ elasticloadbalancingRegisterTargets elasticloadbalancingDeregisterTargets ] Resource Ref InternalServiceTargetGroup - Effect Allow Action [ elasticloadbalancingRegisterTargets elasticloadbalancingDeregisterTargets ] Resource Ref ExternalApiTargetGroup

RegisterNlbIpTargets Type AWSLambdaFunction Properties Handler indexhandler Role FnGetAtt - RegisterTargetLambdaIamRole - Arn Code ZipFile | import json import boto3 import cfnresponse def handler(event context) elb = boto3client(elbv2) if event[RequestType] == Delete elbderegister_targets(TargetGroupArn=event[ResourceProperties][TargetArn]Targets=[Id event[ResourceProperties][TargetIp]]) elif event[RequestType] == Create elbregister_targets(TargetGroupArn=event[ResourceProperties][TargetArn]Targets=

[Id event[ResourceProperties][TargetIp]]) responseData = cfnresponsesend(event context cfnresponseSUCCESS responseData event[ResourceProperties][TargetArn]+event[ResourceProperties][TargetIp]) Runtime python37 Timeout 120

RegisterSubnetTags Type AWSLambdaFunction Properties Handler indexhandler Role FnGetAtt - RegisterSubnetTagsLambdaIamRole - Arn Code ZipFile | import json import boto3 import cfnresponse def handler(event context) ec2_client = boto3client(ec2)

if event[RequestType] == Delete for subnet_id in event[ResourceProperties][Subnets] ec2_clientdelete_tags(Resources=[subnet_id] Tags=[Key kubernetesiocluster + event[ResourceProperties][InfrastructureName]]) elif event[RequestType] == Create for subnet_id in event[ResourceProperties][Subnets] ec2_clientcreate_tags(Resources=[subnet_id] Tags=[Key kubernetesiocluster + event[ResourceProperties][InfrastructureName] Value shared]) responseData = cfnresponsesend(event context cfnresponseSUCCESS responseData event[ResourceProperties][InfrastructureName]+event[ResourceProperties][Subnets][0]) Runtime python37 Timeout 120

Outputs PrivateHostedZoneId Description Hosted zone ID for the private DNS which is required for private records Value Ref IntDns ExternalApiLoadBalancerName Description Full name of the external API load balancer Value GetAtt ExtApiElbLoadBalancerFullName InternalApiLoadBalancerName Description Full name of the internal API load balancer Value GetAtt IntApiElbLoadBalancerFullName ApiServerDnsName Description Full hostname of the API server which is required for the Ignition config files Value Join [ [api-int Ref ClusterName Ref HostedZoneName]] RegisterNlbIpTargetsLambda Description Lambda ARN useful to help register or deregister IP targets for these load balancers Value GetAtt RegisterNlbIpTargetsArn ExternalApiTargetGroupArn Description ARN of the external API target group Value Ref ExternalApiTargetGroup InternalApiTargetGroupArn Description ARN of the internal API target group Value Ref InternalApiTargetGroup InternalServiceTargetGroupArn Description ARN of the internal service target group Value Ref InternalServiceTargetGroup

IMPORTANT

Prerequisites

Procedure

[ ParameterKey InfrastructureName 1 ParameterValue mycluster-ltrandom_stringgt 2 ParameterKey VpcCidr 3 ParameterValue 1000016 4

IMPORTANT

ParameterKey PrivateSubnets 5 ParameterValue subnet-ltrandom_stringgt 6 ParameterKey VpcId 7 ParameterValue vpc-ltrandom_stringgt 8 ]

Example output

Parameters InfrastructureName AllowedPattern ^([a-zA-Z][a-zA-Z0-9-]026)$ MaxLength 27 MinLength 1

ConstraintDescription Infrastructure name must be alphanumeric start with a letter and have a maximum of 27 characters Description A short unique cluster ID used to tag cloud resources and identify items owned or used by the cluster Type String VpcCidr AllowedPattern ^(([0-9]|[1-9][0-9]|1[0-9]2|2[0-4][0-9]|25[0-5]))3([0-9]|[1-9][0-9]|1[0-9]2|2[0-4][0-9]|25[0-5])((1[6-9]|2[0-4]))$ ConstraintDescription CIDR block parameter must be in the form xxxx16-24 Default 1000016 Description CIDR block for VPC Type String VpcId Description The VPC-scoped resources will belong to this VPC Type AWSEC2VPCId PrivateSubnets Description The internal subnets Type ListltAWSEC2SubnetIdgt

Metadata AWSCloudFormationInterface ParameterGroups - Label default Cluster Information Parameters - InfrastructureName - Label default Network Configuration Parameters - VpcId - VpcCidr - PrivateSubnets ParameterLabels InfrastructureName default Infrastructure Name VpcId default VPC ID VpcCidr default VPC CIDR PrivateSubnets default Private Subnets

Resources MasterSecurityGroup Type AWSEC2SecurityGroup Properties GroupDescription Cluster Master Security Group SecurityGroupIngress - IpProtocol icmp FromPort 0 ToPort 0 CidrIp Ref VpcCidr - IpProtocol tcp FromPort 22 ToPort 22 CidrIp Ref VpcCidr

- IpProtocol tcp ToPort 6443 FromPort 6443 CidrIp Ref VpcCidr - IpProtocol tcp FromPort 22623 ToPort 22623 CidrIp Ref VpcCidr VpcId Ref VpcId

MasterIngressEtcd Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description etcd FromPort 2379 ToPort 2380 IpProtocol tcp

MasterIngressGeneve

Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Geneve packets FromPort 6081 ToPort 6081 IpProtocol udp

MasterIngressWorkerInternalUDP Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Internal cluster communication FromPort 9000

ToPort 9999 IpProtocol udp

MasterIngressIngressServices Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Kubernetes ingress services FromPort 30000 ToPort 32767 IpProtocol tcp

MasterIngressWorkerIngressServicesUDP Type AWSEC2SecurityGroupIngress Properties

GroupId GetAtt MasterSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Kubernetes ingress services FromPort 30000 ToPort 32767 IpProtocol udp

WorkerIngressGeneve Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId Description Geneve packets FromPort 6081 ToPort 6081 IpProtocol udp

WorkerIngressMasterInternalUDP Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Internal cluster communication FromPort 9000 ToPort 9999 IpProtocol udp

WorkerIngressIngressServices Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt WorkerSecurityGroupGroupId

Description Kubernetes ingress services FromPort 30000 ToPort 32767 IpProtocol tcp

WorkerIngressMasterIngressServicesUDP Type AWSEC2SecurityGroupIngress Properties GroupId GetAtt WorkerSecurityGroupGroupId SourceSecurityGroupId GetAtt MasterSecurityGroupGroupId Description Kubernetes ingress services FromPort 30000 ToPort 32767 IpProtocol udp

MasterIamRole Type AWSIAMRole Properties AssumeRolePolicyDocument Version 2012-10-17 Statement - Effect Allow Principal Service - ec2amazonawscom Action - stsAssumeRole Policies - PolicyName Join [- [Ref InfrastructureName master policy]] PolicyDocument Version 2012-10-17 Statement - Effect Allow Action - ec2AttachVolume - ec2AuthorizeSecurityGroupIngress

- ec2CreateSecurityGroup - ec2CreateTags - ec2CreateVolume - ec2DeleteSecurityGroup - ec2DeleteVolume - ec2Describe - ec2DetachVolume - ec2ModifyInstanceAttribute - ec2ModifyVolume - ec2RevokeSecurityGroupIngress - elasticloadbalancingAddTags - elasticloadbalancingAttachLoadBalancerToSubnets - elasticloadbalancingApplySecurityGroupsToLoadBalancer - elasticloadbalancingCreateListener - elasticloadbalancingCreateLoadBalancer - elasticloadbalancingCreateLoadBalancerPolicy - elasticloadbalancingCreateLoadBalancerListeners - elasticloadbalancingCreateTargetGroup - elasticloadbalancingConfigureHealthCheck - elasticloadbalancingDeleteListener - elasticloadbalancingDeleteLoadBalancer - elasticloadbalancingDeleteLoadBalancerListeners - elasticloadbalancingDeleteTargetGroup - elasticloadbalancingDeregisterInstancesFromLoadBalancer - elasticloadbalancingDeregisterTargets - elasticloadbalancingDescribe - elasticloadbalancingDetachLoadBalancerFromSubnets - elasticloadbalancingModifyListener - elasticloadbalancingModifyLoadBalancerAttributes - elasticloadbalancingModifyTargetGroup - elasticloadbalancingModifyTargetGroupAttributes - elasticloadbalancingRegisterInstancesWithLoadBalancer - elasticloadbalancingRegisterTargets - elasticloadbalancingSetLoadBalancerPoliciesForBackendServer - elasticloadbalancingSetLoadBalancerPoliciesOfListener - kmsDescribeKey Resource

WorkerIamRole Type AWSIAMRole Properties AssumeRolePolicyDocument Version 2012-10-17 Statement - Effect Allow Principal Service - ec2amazonawscom Action - stsAssumeRole

AWS zone AWS AMI

Policies - PolicyName Join [- [Ref InfrastructureName worker policy]] PolicyDocument Version 2012-10-17 Statement - Effect Allow Action - ec2DescribeInstances - ec2DescribeRegions Resource

AWS zone AWS AMI

Prerequisites

Procedure

IMPORTANT

a Create the bucket

Example output

2019-04-03 161516 314878 bootstrapign

[ ParameterKey InfrastructureName 1 ParameterValue mycluster-ltrandom_stringgt 2 ParameterKey RhcosAmi 3 ParameterValue ami-ltrandom_stringgt 4 ParameterKey AllowedBootstrapSshCidr 5 ParameterValue 00000 6 ParameterKey PublicSubnet 7 ParameterValue subnet-ltrandom_stringgt 8 ParameterKey MasterSecurityGroupId 9 ParameterValue sg-ltrandom_stringgt 10 ParameterKey VpcId 11 ParameterValue vpc-ltrandom_stringgt 12 ParameterKey BootstrapIgnitionLocation 13 ParameterValue s3ltbucket_namegtbootstrapign 14 ParameterKey AutoRegisterELB 15 ParameterValue yes 16

ParameterKey RegisterNlbIpTargetsLambdaArn 17 ParameterValue arnawslambdaltregiongtltaccount_numbergtfunctionltdns_stack_namegt-RegisterNlbIpTargets-ltrandom_stringgt 18 ParameterKey ExternalApiTargetGroupArn 19 ParameterValue arnawselasticloadbalancingltregiongtltaccount_numbergttargetgroupltdns_stack_namegt-Exter-ltrandom_stringgt 20 ParameterKey InternalApiTargetGroupArn 21 ParameterValue arnawselasticloadbalancingltregiongtltaccount_numbergttargetgroupltdns_stack_namegt-Inter-ltrandom_stringgt 22 ParameterKey InternalServiceTargetGroupArn 23 ParameterValue arnawselasticloadbalancingltregiongtltaccount_numbergttargetgroupltdns_stack_namegt-Inter-ltrandom_stringgt 24 ]

Specify the InternalApiTargetGroupArn value from the output of the CloudFormationtemplate for DNS and load balancing Use arnaws-us-gov if deploying the cluster to an

IMPORTANT

Example output

BootstrapInstanceId

BootstrapPublicIp

BootstrapPrivateIp

Parameters InfrastructureName AllowedPattern ^([a-zA-Z][a-zA-Z0-9-]026)$ MaxLength 27 MinLength 1 ConstraintDescription Infrastructure name must be alphanumeric start with a letter and have a maximum of 27 characters Description A short unique cluster ID used to tag cloud resources and identify items owned or used by the cluster Type String RhcosAmi Description Current Red Hat Enterprise Linux CoreOS AMI to use for bootstrap Type AWSEC2ImageId AllowedBootstrapSshCidr AllowedPattern ^(([0-9]|[1-9][0-9]|1[0-9]2|2[0-4][0-9]|25[0-5]))3([0-9]|[1-9][0-9]|1[0-9]2|2[0-4][0-9]|25[0-5])(([0-9]|1[0-9]|2[0-9]|3[0-2]))$ ConstraintDescription CIDR block parameter must be in the form xxxx0-32 Default 00000 Description CIDR block to allow SSH access to the bootstrap node

Type String PublicSubnet Description The public subnet to launch the bootstrap node into Type AWSEC2SubnetId MasterSecurityGroupId Description The master security group ID for registering temporary rules Type AWSEC2SecurityGroupId VpcId Description The VPC-scoped resources will belong to this VPC Type AWSEC2VPCId BootstrapIgnitionLocation Default s3my-s3-bucketbootstrapign Description Ignition config file location Type String AutoRegisterELB Default yes AllowedValues - yes - no Description Do you want to invoke NLB registration which requires a Lambda ARN parameter Type String RegisterNlbIpTargetsLambdaArn Description ARN for NLB IP target registration lambda Type String ExternalApiTargetGroupArn Description ARN for external API load balancer target group Type String InternalApiTargetGroupArn Description ARN for internal API load balancer target group Type String InternalServiceTargetGroupArn Description ARN for internal service load balancer target group Type String

Metadata AWSCloudFormationInterface ParameterGroups - Label default Cluster Information Parameters - InfrastructureName - Label default Host Information Parameters - RhcosAmi - BootstrapIgnitionLocation - MasterSecurityGroupId - Label default Network Configuration Parameters - VpcId - AllowedBootstrapSshCidr - PublicSubnet - Label default Load Balancer Automation Parameters

- AutoRegisterELB - RegisterNlbIpTargetsLambdaArn - ExternalApiTargetGroupArn - InternalApiTargetGroupArn - InternalServiceTargetGroupArn ParameterLabels InfrastructureName default Infrastructure Name VpcId default VPC ID AllowedBootstrapSshCidr default Allowed SSH Source PublicSubnet default Public Subnet RhcosAmi default Red Hat Enterprise Linux CoreOS AMI ID BootstrapIgnitionLocation default Bootstrap Ignition Source MasterSecurityGroupId default Master Security Group ID AutoRegisterELB default Use Provided ELB Automation

Resources BootstrapIamRole Type AWSIAMRole Properties AssumeRolePolicyDocument Version 2012-10-17 Statement - Effect Allow Principal Service - ec2amazonawscom Action - stsAssumeRole Path Policies - PolicyName Join [- [Ref InfrastructureName bootstrap policy]] PolicyDocument Version 2012-10-17 Statement - Effect Allow Action ec2Describe Resource - Effect Allow Action ec2AttachVolume Resource - Effect Allow Action ec2DetachVolume Resource - Effect Allow Action s3GetObject

Resource

RegisterBootstrapApiTarget Condition DoRegistration Type CustomNLBRegister Properties ServiceToken Ref RegisterNlbIpTargetsLambdaArn TargetArn Ref ExternalApiTargetGroupArn TargetIp GetAtt BootstrapInstancePrivateIp

RegisterBootstrapInternalApiTarget Condition DoRegistration Type CustomNLBRegister Properties

IMPORTANT

Prerequisites

ServiceToken Ref RegisterNlbIpTargetsLambdaArn TargetArn Ref InternalApiTargetGroupArn TargetIp GetAtt BootstrapInstancePrivateIp

Procedure

[ ParameterKey InfrastructureName 1 ParameterValue mycluster-ltrandom_stringgt 2 ParameterKey RhcosAmi 3 ParameterValue ami-ltrandom_stringgt 4 ParameterKey AutoRegisterDNS 5 ParameterValue yes 6 ParameterKey PrivateHostedZoneId 7 ParameterValue ltrandom_stringgt 8 ParameterKey PrivateHostedZoneName 9 ParameterValue myclusterexamplecom 10 ParameterKey Master0Subnet 11 ParameterValue subnet-ltrandom_stringgt 12 ParameterKey Master1Subnet 13 ParameterValue subnet-ltrandom_stringgt 14 ParameterKey Master2Subnet 15 ParameterValue subnet-ltrandom_stringgt 16 ParameterKey MasterSecurityGroupId 17 ParameterValue sg-ltrandom_stringgt 18 ParameterKey IgnitionLocation 19

ParameterValue httpsapi-intltcluster_namegtltdomain_namegt22623configmaster 20 ParameterKey CertificateAuthorities 21 ParameterValue datatextplaincharset=utf-8base64ABCxYz== 22 ParameterKey MasterInstanceProfileName 23 ParameterValue ltroles_stackgt-MasterInstanceProfile-ltrandom_stringgt 24 ParameterKey MasterInstanceType 25 ParameterValue m4xlarge 26 ParameterKey AutoRegisterELB 27 ParameterValue yes 28 ParameterKey RegisterNlbIpTargetsLambdaArn 29 ParameterValue arnawslambdaltregiongtltaccount_numbergtfunctionltdns_stack_namegt-RegisterNlbIpTargets-ltrandom_stringgt 30 ParameterKey ExternalApiTargetGroupArn 31 ParameterValue arnawselasticloadbalancingltregiongtltaccount_numbergttargetgroupltdns_stack_namegt-Exter-ltrandom_stringgt 32 ParameterKey InternalApiTargetGroupArn 33 ParameterValue arnawselasticloadbalancingltregiongtltaccount_numbergttargetgroupltdns_stack_namegt-Inter-ltrandom_stringgt 34 ParameterKey InternalServiceTargetGroupArn 35 ParameterValue arnawselasticloadbalancingltregiongtltaccount_numbergttargetgroupltdns_stack_namegt-Inter-ltrandom_stringgt 36 ]

11 13 15

12 14 16

Allowed values

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c42xlarge

c44xlarge

c48xlarge

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

Example output

Parameters InfrastructureName AllowedPattern ^([a-zA-Z][a-zA-Z0-9-]026)$ MaxLength 27 MinLength 1 ConstraintDescription Infrastructure name must be alphanumeric start with a letter and have a maximum of 27 characters Description A short unique cluster ID used to tag nodes for the kubelet cloud provider Type String RhcosAmi Description Current Red Hat Enterprise Linux CoreOS AMI to use for bootstrap Type AWSEC2ImageId AutoRegisterDNS Default yes AllowedValues - yes - no Description Do you want to invoke DNS etcd registration which requires Hosted Zone information Type String PrivateHostedZoneId Description The Route53 private zone ID to register the etcd targets with such as Z21IXYZABCZ2A4 Type String PrivateHostedZoneName Description The Route53 zone to register the targets with such as clusterexamplecom Omit the trailing period Type String Master0Subnet Description The subnets recommend private to launch the master nodes into Type AWSEC2SubnetId Master1Subnet Description The subnets recommend private to launch the master nodes into Type AWSEC2SubnetId Master2Subnet Description The subnets recommend private to launch the master nodes into Type AWSEC2SubnetId MasterSecurityGroupId Description The master security group ID to associate with master nodes Type AWSEC2SecurityGroupId IgnitionLocation Default httpsapi-int$CLUSTER_NAME$DOMAIN22623configmaster Description Ignition config file location Type String CertificateAuthorities Default datatextplaincharset=utf-8base64ABCxYz== Description Base64 encoded certificate authority string to use Type String MasterInstanceProfileName Description IAM profile to associate with master nodes Type String MasterInstanceType Default m5xlarge Type String AllowedValues - m4xlarge

- m42xlarge - m44xlarge - m48xlarge - m410xlarge - m416xlarge - m5xlarge - m52xlarge - m54xlarge - m58xlarge - m510xlarge - m516xlarge - c42xlarge - c44xlarge - c48xlarge - r4xlarge - r42xlarge - r44xlarge - r48xlarge - r416xlarge AutoRegisterELB Default yes AllowedValues - yes - no Description Do you want to invoke NLB registration which requires a Lambda ARN parameter Type String RegisterNlbIpTargetsLambdaArn Description ARN for NLB IP target registration lambda Supply the value from the cluster infrastructure or select no for AutoRegisterELB Type String ExternalApiTargetGroupArn Description ARN for external API load balancer target group Supply the value from the cluster infrastructure or select no for AutoRegisterELB Type String InternalApiTargetGroupArn Description ARN for internal API load balancer target group Supply the value from the cluster infrastructure or select no for AutoRegisterELB Type String InternalServiceTargetGroupArn Description ARN for internal service load balancer target group Supply the value from the cluster infrastructure or select no for AutoRegisterELB Type String

Metadata AWSCloudFormationInterface ParameterGroups - Label default Cluster Information Parameters - InfrastructureName - Label default Host Information Parameters - MasterInstanceType - RhcosAmi - IgnitionLocation

- CertificateAuthorities - MasterSecurityGroupId - MasterInstanceProfileName - Label default Network Configuration Parameters - VpcId - AllowedBootstrapSshCidr - Master0Subnet - Master1Subnet - Master2Subnet - Label default DNS Parameters - AutoRegisterDNS - PrivateHostedZoneName - PrivateHostedZoneId - Label default Load Balancer Automation Parameters - AutoRegisterELB - RegisterNlbIpTargetsLambdaArn - ExternalApiTargetGroupArn - InternalApiTargetGroupArn - InternalServiceTargetGroupArn ParameterLabels InfrastructureName default Infrastructure Name VpcId default VPC ID Master0Subnet default Master-0 Subnet Master1Subnet default Master-1 Subnet Master2Subnet default Master-2 Subnet MasterInstanceType default Master Instance Type MasterInstanceProfileName default Master Instance Profile Name RhcosAmi default Red Hat Enterprise Linux CoreOS AMI ID BootstrapIgnitionLocation default Master Ignition Source CertificateAuthorities default Ignition CA String MasterSecurityGroupId default Master Security Group ID AutoRegisterDNS default Use Provided DNS Automation AutoRegisterELB default Use Provided ELB Automation PrivateHostedZoneName default Private Hosted Zone Name PrivateHostedZoneId default Private Hosted Zone ID

RegisterMaster0InternalServiceTarget Condition DoRegistration Type CustomNLBRegister Properties ServiceToken Ref RegisterNlbIpTargetsLambdaArn

TargetArn Ref InternalServiceTargetGroupArn TargetIp GetAtt Master0PrivateIp

EtcdSrvRecords Condition DoDns Type AWSRoute53RecordSet

Properties HostedZoneId Ref PrivateHostedZoneId Name Join [ [_etcd-server-ssl_tcp Ref PrivateHostedZoneName]] ResourceRecords - Join [ [0 10 2380 Join [ [etcd-0 Ref PrivateHostedZoneName]]] ] - Join [ [0 10 2380 Join [ [etcd-1 Ref PrivateHostedZoneName]]] ] - Join [ [0 10 2380 Join [ [etcd-2 Ref PrivateHostedZoneName]]] ] TTL 60 Type SRV

Outputs PrivateIPs Description The control-plane node private IP addresses Value

IMPORTANT

Prerequisites

Procedure

Join [ [GetAtt Master0PrivateIp GetAtt Master1PrivateIp GetAtt Master2PrivateIp] ]

[ ParameterKey InfrastructureName 1 ParameterValue mycluster-ltrandom_stringgt 2

ParameterKey RhcosAmi 3 ParameterValue ami-ltrandom_stringgt 4 ParameterKey Subnet 5 ParameterValue subnet-ltrandom_stringgt 6 ParameterKey WorkerSecurityGroupId 7 ParameterValue sg-ltrandom_stringgt 8 ParameterKey IgnitionLocation 9 ParameterValue httpsapi-intltcluster_namegtltdomain_namegt22623configworker 10 ParameterKey CertificateAuthorities 11 ParameterValue 12 ParameterKey WorkerInstanceProfileName 13 ParameterValue 14 ParameterKey WorkerInstanceType 15 ParameterValue m4large 16 ]

Allowed values

m4large

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5large

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c4large

c4xlarge

c42xlarge

c44xlarge

c48xlarge

r4large

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

Example output

IMPORTANT

Parameters InfrastructureName AllowedPattern ^([a-zA-Z][a-zA-Z0-9-]026)$ MaxLength 27 MinLength 1 ConstraintDescription Infrastructure name must be alphanumeric start with a letter and have a maximum of 27 characters Description A short unique cluster ID used to tag nodes for the kubelet cloud provider Type String RhcosAmi Description Current Red Hat Enterprise Linux CoreOS AMI to use for bootstrap Type AWSEC2ImageId Subnet Description The subnets recommend private to launch the master nodes into Type AWSEC2SubnetId WorkerSecurityGroupId Description The master security group ID to associate with master nodes Type AWSEC2SecurityGroupId IgnitionLocation Default httpsapi-int$CLUSTER_NAME$DOMAIN22623configworker Description Ignition config file location Type String CertificateAuthorities Default datatextplaincharset=utf-8base64ABCxYz== Description Base64 encoded certificate authority string to use Type String WorkerInstanceProfileName Description IAM profile to associate with master nodes Type String WorkerInstanceType Default m5large Type String AllowedValues - m4large - m4xlarge - m42xlarge

- m44xlarge - m48xlarge - m410xlarge - m416xlarge - m5large - m5xlarge - m52xlarge - m54xlarge - m58xlarge - m510xlarge - m516xlarge - c4large - c4xlarge - c42xlarge - c44xlarge - c48xlarge - r4large - r4xlarge - r42xlarge - r44xlarge - r48xlarge - r416xlarge

Metadata AWSCloudFormationInterface ParameterGroups - Label default Cluster Information Parameters - InfrastructureName - Label default Host Information Parameters - WorkerInstanceType - RhcosAmi - IgnitionLocation - CertificateAuthorities - WorkerSecurityGroupId - WorkerInstanceProfileName - Label default Network Configuration Parameters - Subnet ParameterLabels Subnet default Subnet InfrastructureName default Infrastructure Name WorkerInstanceType default Worker Instance Type WorkerInstanceProfileName default Worker Instance Profile Name RhcosAmi default Red Hat Enterprise Linux CoreOS AMI ID IgnitionLocation default Worker Ignition Source

Prerequisites

CertificateAuthorities default Ignition CA String WorkerSecurityGroupId default Worker Security Group ID

Procedure

Example output

You can log in to your cluster as a default system user by exporting the cluster kubeconfig file The

kubeconfig file contains information about the cluster that is used by the CLI to connect a client to thecorrect cluster and API server The file is specific to a cluster and is created during OpenShift ContainerPlatform installation

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

$ oc whoami

systemadmin

$ oc get nodes

Example output

$ oc get csr

Example output

$ oc get csr

$ oc get nodes

Prerequisites

Procedure

Example output

Prerequisites

Procedure

storage

WARNING

Procedure

WARNING

Prerequisites

Procedure

s3 bucket ltbucket-namegt region ltregion-namegt

Prerequisites

Procedure

Example output

Z3AADJGX6KTTL2

$ aws route53 change-resource-record-sets --hosted-zone-id ltprivate_hosted_zone_idgt --change-batch 1gt Changes [gt gt Action CREATEgt ResourceRecordSet gt Name 052appsltcluster_domaingt 2gt Type Agt AliasTargetgt HostedZoneId lthosted_zone_idgt 3

gt DNSName ltexternal_ipgt 4gt EvaluateTargetHealth falsegt gt gt gt ]gt

Prerequisites

Procedure

1 From the directory that contains the installation program complete the cluster installation

Example output

IMPORTANT

2 Register your cluster on the Cluster registration page

Prerequisites

Procedure

Example output

1824 Next steps

Validate an installation

If the mirror registry that you used to install your cluster has a trusted CA add it to the cluster byconfiguring additional trust stores

19 UNINSTALLING A CLUSTER ON AWS

You can remove a cluster that you deployed to Amazon Web Services (AWS)

191 Removing a cluster that uses installer-provisioned infrastructure

You can remove a cluster that uses installer-provisioned infrastructure from your cloud

Prerequisites

Have a copy of the installation program that you used to deploy the cluster

Have the files that the installation program generated when you created your cluster

Procedure

1 From the directory that contains the installation program on the computer that you used toinstall the cluster run the following command

To view different details specify warn debug or error instead of info

You must specify the directory that contains the cluster definition files for yourcluster The installation program requires the metadatajson file in this directoryto delete the cluster

2 Optional Delete the ltinstallation_directorygt directory and the OpenShift Container Platforminstallation program

$ openshift-install destroy cluster --dir=ltinstallation_directorygt --log-level=info 1 2

Table of Contents








116 Next steps


121 Prerequisites














1210 Next steps

13 INSTALLING A CLUSTER ON AWS WITH NETWORK CUSTOMIZATIONS

131 Prerequisites









137 Configuring an Ingress Controller Network Load Balancer on a new AWS cluster













1314 Next steps


141 Prerequisites



1422 VPC validation

















1411 Next steps


151 Prerequisites





1532 VPC validation

















1512 Next steps


161 Prerequisites






1642 VPC validation



















1613 Next steps

17 INSTALLING A CLUSTER ON USER-PROVISIONED INFRASTRUCTURE IN AWS BY USING CLOUDFORMATION TEMPLATES

171 Prerequisites












































1725 Next steps

18 INSTALLING A CLUSTER ON AWS THAT USES MIRRORED INSTALLATION CONTENT

181 Prerequisites







































1824 Next steps



Legal Notice

Abstract

Table of Contents

10181819

2020202122232432353636373738383939404041

424244525355575859606061

62646465656666676767

Table of Contents

686870717171727374758486888990909191

929393939394949494979797979899

100101

109111

113114114115115116116117117117118118119119119

122122122123123124125126134136137139141

142142143143144144145

146146146147148149149155163163164165167168170172172174180183191

194203204204206

211215221

228232235

Table of Contents

236236237237237238241242242243243244247247248248249249250250250251251

253253259266267268270272273275276278283287295297306307312316321

329333335336337340341341

342343345346347347348348

Table of Contents

Procedure

IMPORTANT

Component

Default AWSlimit

Description

InstanceLimits

Three master nodes

Three worker nodes

Elastic IPs(EIPs)

IMPORTANT

5 5 VPCs perregion

Component

Default AWSlimit

Description

NATGateways

VPCGateway

SecurityGroups

250 2500 peraccount

Component

Default AWSlimit

Description

tagTagResources

tagUntagResources

ec2AllocateAddress

ec2AssociateAddress

ec2CopyImage

ec2CreateTags

ec2CreateVolume

ec2DeleteSnapshot

ec2DeregisterImage

ec2DescribeImages

ec2DescribeKeyPairs

ec2DescribeRegions

ec2DescribeSubnets

ec2DescribeTags

ec2DescribeVolumes

ec2DescribeVpcs

ec2ReleaseAddress

ec2RunInstances

ec2CreateNatGateway

ec2CreateRoute

ec2CreateRouteTable

ec2CreateSubnet

ec2CreateVpc

iamCreateRole

iamDeleteRole

iamDeleteRolePolicy

iamGetRole

iamGetRolePolicy

iamGetUser

iamListRoles

iamListUsers

iamPassRole

iamPutRolePolicy

iamTagRole

route53GetChange

s3CreateBucket

s3DeleteBucket

s3GetBucketAcl

s3GetBucketCors

s3GetBucketLocation

s3GetBucketLogging

s3GetBucketTagging

s3GetBucketWebsite

s3ListBucket

s3PutBucketAcl

s3PutBucketTagging

s3DeleteObject

s3GetObject

s3GetObjectAcl

s3GetObjectTagging

s3GetObjectVersion

s3PutObject

s3PutObjectAcl

s3PutObjectTagging

ec2DeleteVolume

iamDeleteAccessKey

iamDeleteUser

iamListRolePolicies

iamListUserPolicies

s3DeleteObject

tagGetResources

ec2DeleteNatGateway

ec2DeleteRoute

ec2DeleteRouteTable

ec2DeleteSubnet

ec2DeleteVpc

iamCreateAccessKey

iamCreateUser

iamDeleteAccessKey

iamDeleteUser

iamDeleteUserPolicy

iamGetUserPolicy

iamListAccessKeys

iamPutUserPolicy

iamTagUser

iamGetUserPolicy

iamListAccessKeys

s3HeadBucket

Procedure

IMPORTANT

ap-south-1 (Mumbai)

eu-south-1 (Milan)

eu-west-1 (Ireland)

eu-west-2 (London)

eu-west-3 (Paris)

us-east-2 (Ohio)

us-west-2 (Oregon)

us-gov-west-1

us-gov-east-1

116 Next steps

121 Prerequisites

IMPORTANT

Procedure

Example output

Next steps

Prerequisites

Procedure

IMPORTANT

Agent pid 31874

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 11 14

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

Prerequisites

Procedure

Example output

Prerequisites

$ echo $PATH

$ oc ltcommandgt

$ oc whoami

systemadmin

Procedure

Example output

1210 Next steps

131 Prerequisites

IMPORTANT

Procedure

Example output

Agent pid 31874

Next steps

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 12 15

3 7 11

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

Procedure

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

$ echo $PATH

Prerequisites

Procedure

Example output

Prerequisites

Procedure

$ oc ltcommandgt

$ oc whoami

systemadmin

Example output

1314 Next steps

141 Prerequisites

IMPORTANT

Internet gateways

NAT gateways

Subnets

Route tables

VPC DHCP options

VPC endpoints

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

1422 VPC validation

IMPORTANT

Procedure

Example output

Next steps

Prerequisites

Procedure

IMPORTANT

Agent pid 31874

IMPORTANT

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 11 15

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Procedure

Prerequisites

Procedure

Cgt oc ltcommandgt

$ echo $PATH

$ oc ltcommandgt

Example output

Prerequisites

Procedure

Example output

$ oc whoami

systemadmin

1411 Next steps

151 Prerequisites

IMPORTANT

Public subnets

15211 Limitations

Internet gateways

NAT gateways

Subnets

Route tables

VPC DHCP options

VPC endpoints

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

1532 VPC validation

IMPORTANT

Procedure

Example output

Next steps

Prerequisites

Procedure

Agent pid 31874

IMPORTANT

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 11 15

reference content

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

Prerequisites

Procedure

Example output

Prerequisites

$ echo $PATH

$ oc ltcommandgt

$ oc whoami

systemadmin

Procedure

Example output

1512 Next steps

161 Prerequisites

IMPORTANT

us-gov-west-1

us-gov-east-1

Public subnets

16311 Limitations

Internet gateways

NAT gateways

Subnets

Route tables

VPC DHCP options

VPC endpoints

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

1642 VPC validation

IMPORTANT

Procedure

Example output

Next steps

Prerequisites

Agent pid 31874

Procedure

IMPORTANT

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 14

Required

reference content

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

Procedure

Prerequisites

Procedure

Example output

$ echo $PATH

$ oc ltcommandgt

$ oc whoami

systemadmin

Prerequisites

Procedure

Example output

1613 Next steps

IMPORTANT

171 Prerequisites

IMPORTANT

i3large x

m4large x

m4xlarge x x

m42xlarge x x

m44xlarge x x

m48xlarge x x

m410xlarge x x

m416xlarge x x

m5large x

m5xlarge x x

m52xlarge x x

m54xlarge x x

m58xlarge x x

m510xlarge x x

m516xlarge x x

c4large x

c4xlarge x

c42xlarge x x

c44xlarge x x

c48xlarge x x

r4large x

r4xlarge x x

r42xlarge x x

r44xlarge x x

r48xlarge x x

r416xlarge x x

DNS entries

Security groups

IAM roles

S3 buckets

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

etcd recordsets

AWSRoute53RecordSet

Public loadbalancer

Externallistener

Security groups

MasterSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

tcp 6443

tcp 22623

WorkerSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

AWSEC2SecurityGroup

tcp 22

tcp 19531

MasterIngressEtcd

etcd tcp 2379- 2380

MasterIngressVxlan

tcp 9000 - 9999

MasterIngressKube

tcp 10250 - 10259

Worker Ingress

WorkerIngressVxlan

WorkerIngressKube

tcp 10250

Master Allow ec2

Allow iamPassRole

Allow s3GetObject

tagTagResources

tagUntagResources

ec2AllocateAddress

ec2AssociateAddress

ec2CopyImage

ec2CreateTags

ec2CreateVolume

ec2DeleteSnapshot

ec2DeregisterImage

ec2DescribeImages

ec2DescribeKeyPairs

ec2DescribeRegions

ec2DescribeSubnets

ec2DescribeTags

ec2DescribeVolumes

ec2DescribeVpcs

ec2ReleaseAddress

ec2RunInstances

ec2CreateNatGateway

ec2CreateRoute

ec2CreateRouteTable

ec2CreateSubnet

ec2CreateVpc

iamCreateRole

iamDeleteRole

iamDeleteRolePolicy

iamGetRole

iamGetRolePolicy

iamGetUser

iamListRoles

iamListUsers

iamPassRole

iamPutRolePolicy

iamTagRole

route53GetChange

s3CreateBucket

s3DeleteBucket

s3GetBucketAcl

s3GetBucketCors

s3GetBucketLocation

s3GetBucketLogging

s3GetBucketTagging

s3GetBucketWebsite

s3ListBucket

s3PutBucketAcl

s3PutBucketTagging

s3DeleteObject

s3GetObject

s3GetObjectAcl

s3GetObjectTagging

s3GetObjectVersion

s3PutObject

s3PutObjectAcl

s3PutObjectTagging

ec2DeleteVolume

iamDeleteAccessKey

iamDeleteUser

iamListRolePolicies

iamListUserPolicies

s3DeleteObject

tagGetResources

ec2DeleteNatGateway

ec2DeleteRoute

ec2DeleteRouteTable

ec2DeleteSubnet

ec2DeleteVpc

iamCreateAccessKey

iamCreateUser

iamDeleteAccessKey

iamDeleteUser

iamDeleteUserPolicy

iamGetUserPolicy

iamListAccessKeys

iamPutUserPolicy

iamTagUser

iamGetUserPolicy

iamListAccessKeys

s3HeadBucket

Prerequisites

Procedure

IMPORTANT

Procedure

Example output

Next steps

Agent pid 31874

IMPORTANT

Procedure

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

Prerequisites

openshift-vw9j6 1

Procedure

IMPORTANT

Example output

PublicSubnetIds

PrivateSubnetIds

Prerequisites

Procedure

Example output

IMPORTANT

Example output

PrivateHostedZoneId

ApiServerDnsName

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Example output

IpProtocol udp

AWS zone AWS AMI

Prerequisites

Procedure

Example output

Prerequisites

Procedure

IMPORTANT

a Create the bucket

Example output

2019-04-03 161516 314878 bootstrapign

IMPORTANT

Example output

BootstrapInstanceId

BootstrapPublicIp

BootstrapPrivateIp

IMPORTANT

Prerequisites

Procedure

11 13 15

12 14 16

Allowed values

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c42xlarge

c44xlarge

c48xlarge

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

Example output

Metadata

IMPORTANT

Prerequisites

Procedure

Allowed values

m4large

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5large

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c4large

c4xlarge

c42xlarge

c44xlarge

c48xlarge

r4large

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

Example output

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

Procedure

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

$ echo $PATH

$ oc ltcommandgt

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

$ oc whoami

systemadmin

$ oc get nodes

Example output

$ oc get csr

Example output

$ oc get csr

$ oc get nodes

Prerequisites

Procedure

Example output

Prerequisites

Procedure

WARNING

Procedure

WARNING

Prerequisites

Procedure

Prerequisites

Procedure

Example output

Z3AADJGX6KTTL2

Prerequisites

Procedure

Example output

IMPORTANT

Prerequisites

Procedure

Example output

1725 Next steps

IMPORTANT

181 Prerequisites

IMPORTANT

i3large x

m4large x

m4xlarge x x

m42xlarge x x

m44xlarge x x

m48xlarge x x

m410xlarge x x

m416xlarge x x

m5large x

m5xlarge x x

m52xlarge x x

m54xlarge x x

m58xlarge x x

m510xlarge x x

m516xlarge x x

c4large x

c4xlarge x

c42xlarge x x

c44xlarge x x

c48xlarge x x

r4large x

r4xlarge x x

r42xlarge x x

r44xlarge x x

r48xlarge x x

r416xlarge x x

DNS entries

Security groups

IAM roles

S3 buckets

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

etcd recordsets

AWSRoute53RecordSet

Public loadbalancer

Externallistener

Security groups

MasterSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

tcp 6443

tcp 22623

WorkerSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

AWSEC2SecurityGroup

tcp 22

tcp 19531

MasterIngressEtcd

etcd tcp 2379- 2380

MasterIngressVxlan

tcp 9000 - 9999

MasterIngressKube

tcp 10250 - 10259

Worker Ingress

WorkerIngressVxlan

WorkerIngressKube

tcp 10250

Master Allow ec2

Allow iamPassRole

Allow s3GetObject

tagTagResources

tagUntagResources

ec2AllocateAddress

ec2AssociateAddress

ec2CopyImage

ec2CreateTags

ec2CreateVolume

ec2DeleteSnapshot

ec2DeregisterImage

ec2DescribeImages

ec2DescribeKeyPairs

ec2DescribeRegions

ec2DescribeSubnets

ec2DescribeTags

ec2DescribeVolumes

ec2DescribeVpcs

ec2ReleaseAddress

ec2RunInstances

ec2CreateNatGateway

ec2CreateRoute

ec2CreateRouteTable

ec2CreateSubnet

ec2CreateVpc

iamCreateRole

iamDeleteRole

iamDeleteRolePolicy

iamGetRole

iamGetRolePolicy

iamGetUser

iamListRoles

iamListUsers

iamPassRole

iamPutRolePolicy

iamTagRole

route53GetChange

s3CreateBucket

s3DeleteBucket

s3GetBucketAcl

s3GetBucketCors

s3GetBucketLocation

s3GetBucketLogging

s3GetBucketTagging

s3GetBucketWebsite

s3ListBucket

s3PutBucketAcl

s3PutBucketTagging

s3DeleteObject

s3GetObject

s3GetObjectAcl

s3GetObjectTagging

s3GetObjectVersion

s3PutObject

s3PutObjectAcl

s3PutObjectTagging

ec2DeleteVolume

iamDeleteAccessKey

iamDeleteUser

iamListRolePolicies

iamListUserPolicies

s3DeleteObject

tagGetResources

ec2DeleteNatGateway

ec2DeleteRoute

ec2DeleteRouteTable

ec2DeleteSubnet

ec2DeleteVpc

iamCreateAccessKey

iamCreateUser

iamDeleteAccessKey

iamDeleteUser

iamDeleteUserPolicy

iamGetUserPolicy

iamListAccessKeys

iamPutUserPolicy

iamTagUser

iamGetUserPolicy

iamListAccessKeys

s3HeadBucket

Procedure

Example output

Next steps

Agent pid 31874

IMPORTANT

Procedure

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

publish Internal

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

Prerequisites

Procedure

openshift-vw9j6 1

IMPORTANT

Example output

PublicSubnetIds

PrivateSubnetIds

Prerequisites

Procedure

Example output

IMPORTANT

Example output

PrivateHostedZoneId

ApiServerDnsName

Resources

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Example output

MasterIngressGeneve

AWS zone AWS AMI

Prerequisites

Procedure

IMPORTANT

a Create the bucket

Example output

2019-04-03 161516 314878 bootstrapign

IMPORTANT

Example output

BootstrapInstanceId

BootstrapPublicIp

BootstrapPrivateIp

Resource

IMPORTANT

Prerequisites

Procedure

11 13 15

12 14 16

Allowed values

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c42xlarge

c44xlarge

c48xlarge

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

Example output

IMPORTANT

Prerequisites

Procedure

Allowed values

m4large

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5large

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c4large

c4xlarge

c42xlarge

c44xlarge

c48xlarge

r4large

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

Example output

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

$ oc whoami

systemadmin

$ oc get nodes

Example output

$ oc get csr

Example output

$ oc get csr

$ oc get nodes

Prerequisites

Procedure

Example output

Prerequisites

Procedure

storage

WARNING

Procedure

WARNING

Prerequisites

Procedure

Prerequisites

Procedure

Example output

Z3AADJGX6KTTL2

Prerequisites

Procedure

Example output

IMPORTANT

Prerequisites

Procedure

Example output

1824 Next steps

Prerequisites

Procedure

Table of Contents








116 Next steps


121 Prerequisites














1210 Next steps


131 Prerequisites






















1314 Next steps


141 Prerequisites



1422 VPC validation

















1411 Next steps


151 Prerequisites





1532 VPC validation

















1512 Next steps


161 Prerequisites






1642 VPC validation



















1613 Next steps


171 Prerequisites












































1725 Next steps


181 Prerequisites







































1824 Next steps



Legal Notice

Abstract

Table of Contents

10181819

2020202122232432353636373738383939404041

424244525355575859606061

62646465656666676767

Table of Contents

686870717171727374758486888990909191

929393939394949494979797979899

100101

109111

113114114115115116116117117117118118119119119

122122122123123124125126134136137139141

142142143143144144145

146146146147148149149155163163164165167168170172172174180183191

194203204204206

211215221

228232235

Table of Contents

236236237237237238241242242243243244247247248248249249250250250251251

253253259266267268270272273275276278283287295297306307312316321

329333335336337340341341

342343345346347347348348

Table of Contents

Procedure

IMPORTANT

Component

Default AWSlimit

Description

InstanceLimits

Three master nodes

Three worker nodes

Elastic IPs(EIPs)

IMPORTANT

5 5 VPCs perregion

Component

Default AWSlimit

Description

NATGateways

VPCGateway

SecurityGroups

250 2500 peraccount

Component

Default AWSlimit

Description

tagTagResources

tagUntagResources

ec2AllocateAddress

ec2AssociateAddress

ec2CopyImage

ec2CreateTags

ec2CreateVolume

ec2DeleteSnapshot

ec2DeregisterImage

ec2DescribeImages

ec2DescribeKeyPairs

ec2DescribeRegions

ec2DescribeSubnets

ec2DescribeTags

ec2DescribeVolumes

ec2DescribeVpcs

ec2ReleaseAddress

ec2RunInstances

ec2CreateNatGateway

ec2CreateRoute

ec2CreateRouteTable

ec2CreateSubnet

ec2CreateVpc

iamCreateRole

iamDeleteRole

iamDeleteRolePolicy

iamGetRole

iamGetRolePolicy

iamGetUser

iamListRoles

iamListUsers

iamPassRole

iamPutRolePolicy

iamTagRole

route53GetChange

s3CreateBucket

s3DeleteBucket

s3GetBucketAcl

s3GetBucketCors

s3GetBucketLocation

s3GetBucketLogging

s3GetBucketTagging

s3GetBucketWebsite

s3ListBucket

s3PutBucketAcl

s3PutBucketTagging

s3DeleteObject

s3GetObject

s3GetObjectAcl

s3GetObjectTagging

s3GetObjectVersion

s3PutObject

s3PutObjectAcl

s3PutObjectTagging

ec2DeleteVolume

iamDeleteAccessKey

iamDeleteUser

iamListRolePolicies

iamListUserPolicies

s3DeleteObject

tagGetResources

ec2DeleteNatGateway

ec2DeleteRoute

ec2DeleteRouteTable

ec2DeleteSubnet

ec2DeleteVpc

iamCreateAccessKey

iamCreateUser

iamDeleteAccessKey

iamDeleteUser

iamDeleteUserPolicy

iamGetUserPolicy

iamListAccessKeys

iamPutUserPolicy

iamTagUser

iamGetUserPolicy

iamListAccessKeys

s3HeadBucket

Procedure

IMPORTANT

ap-south-1 (Mumbai)

eu-south-1 (Milan)

eu-west-1 (Ireland)

eu-west-2 (London)

eu-west-3 (Paris)

us-east-2 (Ohio)

us-west-2 (Oregon)

us-gov-west-1

us-gov-east-1

116 Next steps

121 Prerequisites

IMPORTANT

Procedure

Example output

Next steps

Prerequisites

Procedure

IMPORTANT

Agent pid 31874

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 11 14

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

Prerequisites

Procedure

Example output

Prerequisites

$ echo $PATH

$ oc ltcommandgt

$ oc whoami

systemadmin

Procedure

Example output

1210 Next steps

131 Prerequisites

IMPORTANT

Procedure

Example output

Agent pid 31874

Next steps

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 12 15

3 7 11

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

Procedure

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

$ echo $PATH

Prerequisites

Procedure

Example output

Prerequisites

Procedure

$ oc ltcommandgt

$ oc whoami

systemadmin

Example output

1314 Next steps

141 Prerequisites

IMPORTANT

Internet gateways

NAT gateways

Subnets

Route tables

VPC DHCP options

VPC endpoints

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

1422 VPC validation

IMPORTANT

Procedure

Example output

Next steps

Prerequisites

Procedure

IMPORTANT

Agent pid 31874

IMPORTANT

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 11 15

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Procedure

Prerequisites

Procedure

Cgt oc ltcommandgt

$ echo $PATH

$ oc ltcommandgt

Example output

Prerequisites

Procedure

Example output

$ oc whoami

systemadmin

1411 Next steps

151 Prerequisites

IMPORTANT

Public subnets

15211 Limitations

Internet gateways

NAT gateways

Subnets

Route tables

VPC DHCP options

VPC endpoints

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

1532 VPC validation

IMPORTANT

Procedure

Example output

Next steps

Prerequisites

Procedure

Agent pid 31874

IMPORTANT

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 11 15

reference content

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

Prerequisites

Procedure

Example output

Prerequisites

$ echo $PATH

$ oc ltcommandgt

$ oc whoami

systemadmin

Procedure

Example output

1512 Next steps

161 Prerequisites

IMPORTANT

us-gov-west-1

us-gov-east-1

Public subnets

16311 Limitations

Internet gateways

NAT gateways

Subnets

Route tables

VPC DHCP options

VPC endpoints

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

1642 VPC validation

IMPORTANT

Procedure

Example output

Next steps

Prerequisites

Agent pid 31874

Procedure

IMPORTANT

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 14

Required

reference content

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

Procedure

Prerequisites

Procedure

Example output

$ echo $PATH

$ oc ltcommandgt

$ oc whoami

systemadmin

Prerequisites

Procedure

Example output

1613 Next steps

IMPORTANT

171 Prerequisites

IMPORTANT

i3large x

m4large x

m4xlarge x x

m42xlarge x x

m44xlarge x x

m48xlarge x x

m410xlarge x x

m416xlarge x x

m5large x

m5xlarge x x

m52xlarge x x

m54xlarge x x

m58xlarge x x

m510xlarge x x

m516xlarge x x

c4large x

c4xlarge x

c42xlarge x x

c44xlarge x x

c48xlarge x x

r4large x

r4xlarge x x

r42xlarge x x

r44xlarge x x

r48xlarge x x

r416xlarge x x

DNS entries

Security groups

IAM roles

S3 buckets

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

etcd recordsets

AWSRoute53RecordSet

Public loadbalancer

Externallistener

Security groups

MasterSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

tcp 6443

tcp 22623

WorkerSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

AWSEC2SecurityGroup

tcp 22

tcp 19531

MasterIngressEtcd

etcd tcp 2379- 2380

MasterIngressVxlan

tcp 9000 - 9999

MasterIngressKube

tcp 10250 - 10259

Worker Ingress

WorkerIngressVxlan

WorkerIngressKube

tcp 10250

Master Allow ec2

Allow iamPassRole

Allow s3GetObject

tagTagResources

tagUntagResources

ec2AllocateAddress

ec2AssociateAddress

ec2CopyImage

ec2CreateTags

ec2CreateVolume

ec2DeleteSnapshot

ec2DeregisterImage

ec2DescribeImages

ec2DescribeKeyPairs

ec2DescribeRegions

ec2DescribeSubnets

ec2DescribeTags

ec2DescribeVolumes

ec2DescribeVpcs

ec2ReleaseAddress

ec2RunInstances

ec2CreateNatGateway

ec2CreateRoute

ec2CreateRouteTable

ec2CreateSubnet

ec2CreateVpc

iamCreateRole

iamDeleteRole

iamDeleteRolePolicy

iamGetRole

iamGetRolePolicy

iamGetUser

iamListRoles

iamListUsers

iamPassRole

iamPutRolePolicy

iamTagRole

route53GetChange

s3CreateBucket

s3DeleteBucket

s3GetBucketAcl

s3GetBucketCors

s3GetBucketLocation

s3GetBucketLogging

s3GetBucketTagging

s3GetBucketWebsite

s3ListBucket

s3PutBucketAcl

s3PutBucketTagging

s3DeleteObject

s3GetObject

s3GetObjectAcl

s3GetObjectTagging

s3GetObjectVersion

s3PutObject

s3PutObjectAcl

s3PutObjectTagging

ec2DeleteVolume

iamDeleteAccessKey

iamDeleteUser

iamListRolePolicies

iamListUserPolicies

s3DeleteObject

tagGetResources

ec2DeleteNatGateway

ec2DeleteRoute

ec2DeleteRouteTable

ec2DeleteSubnet

ec2DeleteVpc

iamCreateAccessKey

iamCreateUser

iamDeleteAccessKey

iamDeleteUser

iamDeleteUserPolicy

iamGetUserPolicy

iamListAccessKeys

iamPutUserPolicy

iamTagUser

iamGetUserPolicy

iamListAccessKeys

s3HeadBucket

Prerequisites

Procedure

IMPORTANT

Procedure

Example output

Next steps

Agent pid 31874

IMPORTANT

Procedure

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

Prerequisites

openshift-vw9j6 1

Procedure

IMPORTANT

Example output

PublicSubnetIds

PrivateSubnetIds

Prerequisites

Procedure

Example output

IMPORTANT

Example output

PrivateHostedZoneId

ApiServerDnsName

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Example output

IpProtocol udp

AWS zone AWS AMI

Prerequisites

Procedure

Example output

Prerequisites

Procedure

IMPORTANT

a Create the bucket

Example output

2019-04-03 161516 314878 bootstrapign

IMPORTANT

Example output

BootstrapInstanceId

BootstrapPublicIp

BootstrapPrivateIp

IMPORTANT

Prerequisites

Procedure

11 13 15

12 14 16

Allowed values

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c42xlarge

c44xlarge

c48xlarge

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

Example output

Metadata

IMPORTANT

Prerequisites

Procedure

Allowed values

m4large

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5large

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c4large

c4xlarge

c42xlarge

c44xlarge

c48xlarge

r4large

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

Example output

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

Procedure

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

$ echo $PATH

$ oc ltcommandgt

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

$ oc whoami

systemadmin

$ oc get nodes

Example output

$ oc get csr

Example output

$ oc get csr

$ oc get nodes

Prerequisites

Procedure

Example output

Prerequisites

Procedure

WARNING

Procedure

WARNING

Prerequisites

Procedure

Prerequisites

Procedure

Example output

Z3AADJGX6KTTL2

Prerequisites

Procedure

Example output

IMPORTANT

Prerequisites

Procedure

Example output

1725 Next steps

IMPORTANT

181 Prerequisites

IMPORTANT

i3large x

m4large x

m4xlarge x x

m42xlarge x x

m44xlarge x x

m48xlarge x x

m410xlarge x x

m416xlarge x x

m5large x

m5xlarge x x

m52xlarge x x

m54xlarge x x

m58xlarge x x

m510xlarge x x

m516xlarge x x

c4large x

c4xlarge x

c42xlarge x x

c44xlarge x x

c48xlarge x x

r4large x

r4xlarge x x

r42xlarge x x

r44xlarge x x

r48xlarge x x

r416xlarge x x

DNS entries

Security groups

IAM roles

S3 buckets

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

etcd recordsets

AWSRoute53RecordSet

Public loadbalancer

Externallistener

Security groups

MasterSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

tcp 6443

tcp 22623

WorkerSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

AWSEC2SecurityGroup

tcp 22

tcp 19531

MasterIngressEtcd

etcd tcp 2379- 2380

MasterIngressVxlan

tcp 9000 - 9999

MasterIngressKube

tcp 10250 - 10259

Worker Ingress

WorkerIngressVxlan

WorkerIngressKube

tcp 10250

Master Allow ec2

Allow iamPassRole

Allow s3GetObject

tagTagResources

tagUntagResources

ec2AllocateAddress

ec2AssociateAddress

ec2CopyImage

ec2CreateTags

ec2CreateVolume

ec2DeleteSnapshot

ec2DeregisterImage

ec2DescribeImages

ec2DescribeKeyPairs

ec2DescribeRegions

ec2DescribeSubnets

ec2DescribeTags

ec2DescribeVolumes

ec2DescribeVpcs

ec2ReleaseAddress

ec2RunInstances

ec2CreateNatGateway

ec2CreateRoute

ec2CreateRouteTable

ec2CreateSubnet

ec2CreateVpc

iamCreateRole

iamDeleteRole

iamDeleteRolePolicy

iamGetRole

iamGetRolePolicy

iamGetUser

iamListRoles

iamListUsers

iamPassRole

iamPutRolePolicy

iamTagRole

route53GetChange

s3CreateBucket

s3DeleteBucket

s3GetBucketAcl

s3GetBucketCors

s3GetBucketLocation

s3GetBucketLogging

s3GetBucketTagging

s3GetBucketWebsite

s3ListBucket

s3PutBucketAcl

s3PutBucketTagging

s3DeleteObject

s3GetObject

s3GetObjectAcl

s3GetObjectTagging

s3GetObjectVersion

s3PutObject

s3PutObjectAcl

s3PutObjectTagging

ec2DeleteVolume

iamDeleteAccessKey

iamDeleteUser

iamListRolePolicies

iamListUserPolicies

s3DeleteObject

tagGetResources

ec2DeleteNatGateway

ec2DeleteRoute

ec2DeleteRouteTable

ec2DeleteSubnet

ec2DeleteVpc

iamCreateAccessKey

iamCreateUser

iamDeleteAccessKey

iamDeleteUser

iamDeleteUserPolicy

iamGetUserPolicy

iamListAccessKeys

iamPutUserPolicy

iamTagUser

iamGetUserPolicy

iamListAccessKeys

s3HeadBucket

Procedure

Example output

Next steps

Agent pid 31874

IMPORTANT

Procedure

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

publish Internal

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

Prerequisites

Procedure

openshift-vw9j6 1

IMPORTANT

Example output

PublicSubnetIds

PrivateSubnetIds

Prerequisites

Procedure

Example output

IMPORTANT

Example output

PrivateHostedZoneId

ApiServerDnsName

Resources

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Example output

MasterIngressGeneve

AWS zone AWS AMI

Prerequisites

Procedure

IMPORTANT

a Create the bucket

Example output

2019-04-03 161516 314878 bootstrapign

IMPORTANT

Example output

BootstrapInstanceId

BootstrapPublicIp

BootstrapPrivateIp

Resource

IMPORTANT

Prerequisites

Procedure

11 13 15

12 14 16

Allowed values

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c42xlarge

c44xlarge

c48xlarge

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

Example output

IMPORTANT

Prerequisites

Procedure

Allowed values

m4large

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5large

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c4large

c4xlarge

c42xlarge

c44xlarge

c48xlarge

r4large

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

Example output

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

$ oc whoami

systemadmin

$ oc get nodes

Example output

$ oc get csr

Example output

$ oc get csr

$ oc get nodes

Prerequisites

Procedure

Example output

Prerequisites

Procedure

storage

WARNING

Procedure

WARNING

Prerequisites

Procedure

Prerequisites

Procedure

Example output

Z3AADJGX6KTTL2

Prerequisites

Procedure

Example output

IMPORTANT

Prerequisites

Procedure

Example output

1824 Next steps

Prerequisites

Procedure

Table of Contents








116 Next steps


121 Prerequisites














1210 Next steps


131 Prerequisites






















1314 Next steps


141 Prerequisites



1422 VPC validation

















1411 Next steps


151 Prerequisites





1532 VPC validation

















1512 Next steps


161 Prerequisites






1642 VPC validation



















1613 Next steps


171 Prerequisites












































1725 Next steps


181 Prerequisites







































1824 Next steps



Table of Contents

10181819

2020202122232432353636373738383939404041

424244525355575859606061

62646465656666676767

Table of Contents

686870717171727374758486888990909191

929393939394949494979797979899

100101

109111

113114114115115116116117117117118118119119119

122122122123123124125126134136137139141

142142143143144144145

146146146147148149149155163163164165167168170172172174180183191

194203204204206

211215221

228232235

Table of Contents

236236237237237238241242242243243244247247248248249249250250250251251

253253259266267268270272273275276278283287295297306307312316321

329333335336337340341341

342343345346347347348348

Table of Contents

Procedure

IMPORTANT

Component

Default AWSlimit

Description

InstanceLimits

Three master nodes

Three worker nodes

Elastic IPs(EIPs)

IMPORTANT

5 5 VPCs perregion

Component

Default AWSlimit

Description

NATGateways

VPCGateway

SecurityGroups

250 2500 peraccount

Component

Default AWSlimit

Description

tagTagResources

tagUntagResources

ec2AllocateAddress

ec2AssociateAddress

ec2CopyImage

ec2CreateTags

ec2CreateVolume

ec2DeleteSnapshot

ec2DeregisterImage

ec2DescribeImages

ec2DescribeKeyPairs

ec2DescribeRegions

ec2DescribeSubnets

ec2DescribeTags

ec2DescribeVolumes

ec2DescribeVpcs

ec2ReleaseAddress

ec2RunInstances

ec2CreateNatGateway

ec2CreateRoute

ec2CreateRouteTable

ec2CreateSubnet

ec2CreateVpc

iamCreateRole

iamDeleteRole

iamDeleteRolePolicy

iamGetRole

iamGetRolePolicy

iamGetUser

iamListRoles

iamListUsers

iamPassRole

iamPutRolePolicy

iamTagRole

route53GetChange

s3CreateBucket

s3DeleteBucket

s3GetBucketAcl

s3GetBucketCors

s3GetBucketLocation

s3GetBucketLogging

s3GetBucketTagging

s3GetBucketWebsite

s3ListBucket

s3PutBucketAcl

s3PutBucketTagging

s3DeleteObject

s3GetObject

s3GetObjectAcl

s3GetObjectTagging

s3GetObjectVersion

s3PutObject

s3PutObjectAcl

s3PutObjectTagging

ec2DeleteVolume

iamDeleteAccessKey

iamDeleteUser

iamListRolePolicies

iamListUserPolicies

s3DeleteObject

tagGetResources

ec2DeleteNatGateway

ec2DeleteRoute

ec2DeleteRouteTable

ec2DeleteSubnet

ec2DeleteVpc

iamCreateAccessKey

iamCreateUser

iamDeleteAccessKey

iamDeleteUser

iamDeleteUserPolicy

iamGetUserPolicy

iamListAccessKeys

iamPutUserPolicy

iamTagUser

iamGetUserPolicy

iamListAccessKeys

s3HeadBucket

Procedure

IMPORTANT

ap-south-1 (Mumbai)

eu-south-1 (Milan)

eu-west-1 (Ireland)

eu-west-2 (London)

eu-west-3 (Paris)

us-east-2 (Ohio)

us-west-2 (Oregon)

us-gov-west-1

us-gov-east-1

116 Next steps

121 Prerequisites

IMPORTANT

Procedure

Example output

Next steps

Prerequisites

Procedure

IMPORTANT

Agent pid 31874

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 11 14

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

Prerequisites

Procedure

Example output

Prerequisites

$ echo $PATH

$ oc ltcommandgt

$ oc whoami

systemadmin

Procedure

Example output

1210 Next steps

131 Prerequisites

IMPORTANT

Procedure

Example output

Agent pid 31874

Next steps

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 12 15

3 7 11

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

Procedure

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

$ echo $PATH

Prerequisites

Procedure

Example output

Prerequisites

Procedure

$ oc ltcommandgt

$ oc whoami

systemadmin

Example output

1314 Next steps

141 Prerequisites

IMPORTANT

Internet gateways

NAT gateways

Subnets

Route tables

VPC DHCP options

VPC endpoints

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

1422 VPC validation

IMPORTANT

Procedure

Example output

Next steps

Prerequisites

Procedure

IMPORTANT

Agent pid 31874

IMPORTANT

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 11 15

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Procedure

Prerequisites

Procedure

Cgt oc ltcommandgt

$ echo $PATH

$ oc ltcommandgt

Example output

Prerequisites

Procedure

Example output

$ oc whoami

systemadmin

1411 Next steps

151 Prerequisites

IMPORTANT

Public subnets

15211 Limitations

Internet gateways

NAT gateways

Subnets

Route tables

VPC DHCP options

VPC endpoints

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

1532 VPC validation

IMPORTANT

Procedure

Example output

Next steps

Prerequisites

Procedure

Agent pid 31874

IMPORTANT

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 11 15

reference content

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

Prerequisites

Procedure

Example output

Prerequisites

$ echo $PATH

$ oc ltcommandgt

$ oc whoami

systemadmin

Procedure

Example output

1512 Next steps

161 Prerequisites

IMPORTANT

us-gov-west-1

us-gov-east-1

Public subnets

16311 Limitations

Internet gateways

NAT gateways

Subnets

Route tables

VPC DHCP options

VPC endpoints

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

1642 VPC validation

IMPORTANT

Procedure

Example output

Next steps

Prerequisites

Agent pid 31874

Procedure

IMPORTANT

Prerequisites

Procedure

IMPORTANT

String

Object

String

computearchitecture

String

IMPORTANT

Enabled or Disabled

worker

String

IMPORTANT

Enabled or Disabled

master

false or true

imageContentSources

String

Array of strings

Object

Array of objects

Integer

String

computeawsregion

platformawsamiID

platformawsuserTags

platformawssubnets

Valid subnet IDs

IMPORTANT

1 10 14

Required

reference content

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

Procedure

Prerequisites

Procedure

Example output

$ echo $PATH

$ oc ltcommandgt

$ oc whoami

systemadmin

Prerequisites

Procedure

Example output

1613 Next steps

IMPORTANT

171 Prerequisites

IMPORTANT

i3large x

m4large x

m4xlarge x x

m42xlarge x x

m44xlarge x x

m48xlarge x x

m410xlarge x x

m416xlarge x x

m5large x

m5xlarge x x

m52xlarge x x

m54xlarge x x

m58xlarge x x

m510xlarge x x

m516xlarge x x

c4large x

c4xlarge x

c42xlarge x x

c44xlarge x x

c48xlarge x x

r4large x

r4xlarge x x

r42xlarge x x

r44xlarge x x

r48xlarge x x

r416xlarge x x

DNS entries

Security groups

IAM roles

S3 buckets

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

etcd recordsets

AWSRoute53RecordSet

Public loadbalancer

Externallistener

Security groups

MasterSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

tcp 6443

tcp 22623

WorkerSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

AWSEC2SecurityGroup

tcp 22

tcp 19531

MasterIngressEtcd

etcd tcp 2379- 2380

MasterIngressVxlan

tcp 9000 - 9999

MasterIngressKube

tcp 10250 - 10259

Worker Ingress

WorkerIngressVxlan

WorkerIngressKube

tcp 10250

Master Allow ec2

Allow iamPassRole

Allow s3GetObject

tagTagResources

tagUntagResources

ec2AllocateAddress

ec2AssociateAddress

ec2CopyImage

ec2CreateTags

ec2CreateVolume

ec2DeleteSnapshot

ec2DeregisterImage

ec2DescribeImages

ec2DescribeKeyPairs

ec2DescribeRegions

ec2DescribeSubnets

ec2DescribeTags

ec2DescribeVolumes

ec2DescribeVpcs

ec2ReleaseAddress

ec2RunInstances

ec2CreateNatGateway

ec2CreateRoute

ec2CreateRouteTable

ec2CreateSubnet

ec2CreateVpc

iamCreateRole

iamDeleteRole

iamDeleteRolePolicy

iamGetRole

iamGetRolePolicy

iamGetUser

iamListRoles

iamListUsers

iamPassRole

iamPutRolePolicy

iamTagRole

route53GetChange

s3CreateBucket

s3DeleteBucket

s3GetBucketAcl

s3GetBucketCors

s3GetBucketLocation

s3GetBucketLogging

s3GetBucketTagging

s3GetBucketWebsite

s3ListBucket

s3PutBucketAcl

s3PutBucketTagging

s3DeleteObject

s3GetObject

s3GetObjectAcl

s3GetObjectTagging

s3GetObjectVersion

s3PutObject

s3PutObjectAcl

s3PutObjectTagging

ec2DeleteVolume

iamDeleteAccessKey

iamDeleteUser

iamListRolePolicies

iamListUserPolicies

s3DeleteObject

tagGetResources

ec2DeleteNatGateway

ec2DeleteRoute

ec2DeleteRouteTable

ec2DeleteSubnet

ec2DeleteVpc

iamCreateAccessKey

iamCreateUser

iamDeleteAccessKey

iamDeleteUser

iamDeleteUserPolicy

iamGetUserPolicy

iamListAccessKeys

iamPutUserPolicy

iamTagUser

iamGetUserPolicy

iamListAccessKeys

s3HeadBucket

Prerequisites

Procedure

IMPORTANT

Procedure

Example output

Next steps

Agent pid 31874

IMPORTANT

Procedure

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

Prerequisites

openshift-vw9j6 1

Procedure

IMPORTANT

Example output

PublicSubnetIds

PrivateSubnetIds

Prerequisites

Procedure

Example output

IMPORTANT

Example output

PrivateHostedZoneId

ApiServerDnsName

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Example output

IpProtocol udp

AWS zone AWS AMI

Prerequisites

Procedure

Example output

Prerequisites

Procedure

IMPORTANT

a Create the bucket

Example output

2019-04-03 161516 314878 bootstrapign

IMPORTANT

Example output

BootstrapInstanceId

BootstrapPublicIp

BootstrapPrivateIp

IMPORTANT

Prerequisites

Procedure

11 13 15

12 14 16

Allowed values

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c42xlarge

c44xlarge

c48xlarge

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

Example output

Metadata

IMPORTANT

Prerequisites

Procedure

Allowed values

m4large

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5large

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c4large

c4xlarge

c42xlarge

c44xlarge

c48xlarge

r4large

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

Example output

IMPORTANT

Prerequisites

Procedure

Example output

IMPORTANT

Procedure

$ tar xvzf ltfilegt

$ echo $PATH

Procedure

$ oc ltcommandgt

Cgt path

Cgt oc ltcommandgt

$ echo $PATH

$ oc ltcommandgt

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

$ oc whoami

systemadmin

$ oc get nodes

Example output

$ oc get csr

Example output

$ oc get csr

$ oc get nodes

Prerequisites

Procedure

Example output

Prerequisites

Procedure

WARNING

Procedure

WARNING

Prerequisites

Procedure

Prerequisites

Procedure

Example output

Z3AADJGX6KTTL2

Prerequisites

Procedure

Example output

IMPORTANT

Prerequisites

Procedure

Example output

1725 Next steps

IMPORTANT

181 Prerequisites

IMPORTANT

i3large x

m4large x

m4xlarge x x

m42xlarge x x

m44xlarge x x

m48xlarge x x

m410xlarge x x

m416xlarge x x

m5large x

m5xlarge x x

m52xlarge x x

m54xlarge x x

m58xlarge x x

m510xlarge x x

m516xlarge x x

c4large x

c4xlarge x

c42xlarge x x

c44xlarge x x

c48xlarge x x

r4large x

r4xlarge x x

r42xlarge x x

r44xlarge x x

r48xlarge x x

r416xlarge x x

DNS entries

Security groups

IAM roles

S3 buckets

Component

VPCAWSEC2VPC

AWSEC2VPCEndpoint

AWSEC2RouteTable

AWSEC2Route

AWSEC2NatGateway

AWSEC2EIP

AWSEC2NetworkAcl

Port Reason

AWSEC2RouteTable

Component

etcd recordsets

AWSRoute53RecordSet

Public loadbalancer

Externallistener

Security groups

MasterSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

tcp 6443

tcp 22623

WorkerSecurityGroup

AWSEC2SecurityGroup

icmp 0

tcp 22

AWSEC2SecurityGroup

tcp 22

tcp 19531

MasterIngressEtcd

etcd tcp 2379- 2380

MasterIngressVxlan

tcp 9000 - 9999

MasterIngressKube

tcp 10250 - 10259

Worker Ingress

WorkerIngressVxlan

WorkerIngressKube

tcp 10250

Master Allow ec2

Allow iamPassRole

Allow s3GetObject

tagTagResources

tagUntagResources

ec2AllocateAddress

ec2AssociateAddress

ec2CopyImage

ec2CreateTags

ec2CreateVolume

ec2DeleteSnapshot

ec2DeregisterImage

ec2DescribeImages

ec2DescribeKeyPairs

ec2DescribeRegions

ec2DescribeSubnets

ec2DescribeTags

ec2DescribeVolumes

ec2DescribeVpcs

ec2ReleaseAddress

ec2RunInstances

ec2CreateNatGateway

ec2CreateRoute

ec2CreateRouteTable

ec2CreateSubnet

ec2CreateVpc

iamCreateRole

iamDeleteRole

iamDeleteRolePolicy

iamGetRole

iamGetRolePolicy

iamGetUser

iamListRoles

iamListUsers

iamPassRole

iamPutRolePolicy

iamTagRole

route53GetChange

s3CreateBucket

s3DeleteBucket

s3GetBucketAcl

s3GetBucketCors

s3GetBucketLocation

s3GetBucketLogging

s3GetBucketTagging

s3GetBucketWebsite

s3ListBucket

s3PutBucketAcl

s3PutBucketTagging

s3DeleteObject

s3GetObject

s3GetObjectAcl

s3GetObjectTagging

s3GetObjectVersion

s3PutObject

s3PutObjectAcl

s3PutObjectTagging

ec2DeleteVolume

iamDeleteAccessKey

iamDeleteUser

iamListRolePolicies

iamListUserPolicies

s3DeleteObject

tagGetResources

ec2DeleteNatGateway

ec2DeleteRoute

ec2DeleteRouteTable

ec2DeleteSubnet

ec2DeleteVpc

iamCreateAccessKey

iamCreateUser

iamDeleteAccessKey

iamDeleteUser

iamDeleteUserPolicy

iamGetUserPolicy

iamListAccessKeys

iamPutUserPolicy

iamTagUser

iamGetUserPolicy

iamListAccessKeys

s3HeadBucket

Procedure

Example output

Next steps

Agent pid 31874

IMPORTANT

Procedure

Prerequisites

Procedure

IMPORTANT

Prerequisites

Procedure

publish Internal

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

Prerequisites

Procedure

openshift-vw9j6 1

IMPORTANT

Example output

PublicSubnetIds

PrivateSubnetIds

Prerequisites

Procedure

Example output

IMPORTANT

Example output

PrivateHostedZoneId

ApiServerDnsName

Resources

IMPORTANT

Prerequisites

Procedure

IMPORTANT

Example output

MasterIngressGeneve

AWS zone AWS AMI

Prerequisites

Procedure

IMPORTANT

a Create the bucket

Example output

2019-04-03 161516 314878 bootstrapign

IMPORTANT

Example output

BootstrapInstanceId

BootstrapPublicIp

BootstrapPrivateIp

Resource

IMPORTANT

Prerequisites

Procedure

11 13 15

12 14 16

Allowed values

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c42xlarge

c44xlarge

c48xlarge

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

Example output

IMPORTANT

Prerequisites

Procedure

Allowed values

m4large

m4xlarge

m42xlarge

m44xlarge

m48xlarge

m410xlarge

m416xlarge

m5large

m5xlarge

m52xlarge

m54xlarge

m58xlarge

m510xlarge

m516xlarge

c4large

c4xlarge

c42xlarge

c44xlarge

c48xlarge

r4large

r4xlarge

r42xlarge

r44xlarge

r48xlarge

r416xlarge

IMPORTANT

Example output

IMPORTANT

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

Prerequisites

Procedure

Example output

$ oc whoami

systemadmin

$ oc get nodes

Example output

$ oc get csr

Example output

$ oc get csr

$ oc get nodes

Prerequisites

Procedure

Example output

Prerequisites

Procedure

storage

WARNING

Procedure

WARNING

Prerequisites

Procedure

Prerequisites

Procedure

Example output

Z3AADJGX6KTTL2

Prerequisites

Procedure

Example output

IMPORTANT

Prerequisites

Procedure

Example output

1824 Next steps

Prerequisites

Procedure

Table of Contents








116 Next steps


121 Prerequisites














1210 Next steps


131 Prerequisites






















1314 Next steps


141 Prerequisites



1422 VPC validation

















1411 Next steps


151 Prerequisites





1532 VPC validation

















1512 Next steps


161 Prerequisites






1642 VPC validation



















1613 Next steps


171 Prerequisites












































1725 Next steps


181 Prerequisites







































1824 Next steps



Documents

OpenShift Container Platform 4...1.8.14. Creating the worker nodes in AWS OpenShift Container Platform 4.6 Installing on AWS 4. OpenShift Container Platform 4.6 Installing on AWS 6