How to Configure Apache Kafka 2.5 on AKS...3 Azure AKS Setup 3.1 Create a New Azure Kubernetes Service (AKS) A new Kubernetes cluster must be created in AKS. Use the following to build

TIBCO Software Inc. Global Headquarters

3307 Hillview Avenue

Palo Alto, CA 94304

Tel: +1 650-846-1000

Toll Free: 1 800-420-8450

Fax: +1 650-846-1005

www.tibco.com

TIBCO fuels digital business by enabling better decisions and faster, smarter actions through the TIBCO Connected Intelligence Cloud. From APIs and systems to devices and people, we interconnect everything, capture data in real time wherever it is, and augment the intelligence of your business through analytical insights. Thousands of customers around the globe rely on us to build compelling experiences, energize operations, and propel innovation. Learn how TIBCO makes digital smarter at www.tibco.com.

How to Configure Apache Kafka in an Azure

Kubernetes Environment This document describes how to configure Apache Kafka in an Azure Kubernetes environment (AKS).

Version 2.5 July 2020 Document updated for AKD 2.5

©2020 TIBCO Software Inc. All Rights Reserved. 2

Copyright Notice COPYRIGHT© 2020 TIBCO Software Inc. All rights reserved.

Trademarks TIBCO, and the TIBCO logo, are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries. All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only.

Content Warranty The information in this document is subject to change without notice. THIS DOCUMENT IS PROVIDED "AS IS" AND TIBCO MAKES NO WARRANTY, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING BUT NOT LIMITED TO ALL WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. TIBCO Software Inc. shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance or use of this material.

For more information, please contact:

TIBCO Software Inc. 3303 Hillview Avenue Palo Alto, CA 94304 USA


Table of Contents

1 Overview ........................................................................................................................................ 5 1.1 Supported Versions ............................................................................................................................. 5 1.2 Prerequisites ....................................................................................................................................... 5 1.3 Prepare Local Environment ................................................................................................................. 5 1.4 Prepare Preliminary Azure Account and Kubernetes Configuration ................................................... 6 1.5 Apache Kafka Architecture .................................................................................................................. 6

2 Preparing the Zookeeper and Kafka Docker Images ..................................................................... 7 2.1 Build Docker images ............................................................................................................................ 7 2.2 Configure the Azure Container Registry .............................................................................................. 7

2.2.1 Create the Azure Container Registry ............................................................................................... 7 2.2.2 Push the Docker Images to ACR ...................................................................................................... 8

3 Azure AKS Setup ............................................................................................................................. 9 3.1 Create a New Azure Kubernetes Service (AKS) ................................................................................... 9 3.2 Configuring Kubectl to connect to Azure Kubernetes Service .......................................................... 10

3.2.1 Configure Kubectl to connect to AKS ............................................................................................. 10 3.3 Update the newly created AKS to access the ACR(s) ........................................................................ 11

4 Deploy Zookeeper and Kafka in AKS ............................................................................................ 12 4.1 Deploying the AKD Core on AKS ........................................................................................................ 12

4.1.1 Zookeeper ..................................................................................................................................... 12 4.1.2 Kafka ............................................................................................................................................. 12

4.2 Deploy Zookeeper and Kafka ............................................................................................................ 13 4.3 Stopping, Deleting, or Accessing the AKD Servers ............................................................................ 14

5 Testing the Kafka Environment ................................................................................................... 16


Table of Figures FIGURE 1 - CREATE ACR REGISTRY ............................................................................................................................................. 8 FIGURE 2 - LOGIN INTO THE ACR ............................................................................................................................................... 8 FIGURE 3 - TAG THE DOCKER IMAGE .......................................................................................................................................... 8 FIGURE 4 - PUSH THE DOCKER IMAGE TO ACR ............................................................................................................................. 8 FIGURE 5 - KUBERNETES CLUSTER CREATION .............................................................................................................................. 10 FIGURE 6 - CONFIGURE KUBECTL ............................................................................................................................................. 10 FIGURE 7 - VERIFY CONNECTING TO THE KUBERNETES CLUSTER ...................................................................................................... 11 FIGURE 8 - UPDATE THE AKS FOR THE ACRS ............................................................................................................................. 11 FIGURE 9 - ZOOKEEPER.YAML MODIFICATION ............................................................................................................................. 12 FIGURE 10 - KAFKA.YAML MODIFICATION .................................................................................................................................. 12 FIGURE 11 - KAFKA-SERVICE.YAML EXAMPLE .............................................................................................................................. 13 FIGURE 12 - GET PODS EXAMPLE ............................................................................................................................................. 14 FIGURE 13 - TO STOP AND START THE ZOOKEEPER AND KAFKA STATEFULSETS .................................................................................. 15


1 Overview

This document will outline how to configure the Apache Kafka in a Kubernetes cluster on Azure. The Kubernetes cluster will be built using the Azure Kubernetes Service (AKS). Running the TIBCO Messaging components on Azure involves:

• Configuring the Azure Kubernetes Service (AKS) for Apache Kafka • Configuring the Azure Container Registry (ACR) for the Docker® image registry • Creating multiple Docker images for the Apache Kafka Core, where the containers will be

hosted in ACR • Creating persisted volumes for Kafka and Zookeeper • Configuring and creating Kubernetes containers based on the Docker images for the

individual components • Configuring Load Balancer(s) in Azure to access Apache Kafka

1.1 Supported Versions

The steps described in this document are supported for the following versions of the products and components involved:

• Apache Kafka 2.5.0 is required Apache Kafka can be download from either edelivery.tibco.com , or https://www.tibco.com/products/tibco-messaging/downloads.

• Docker Community/Enterprise Edition should be most recent version. Docker Community version V19.03.8 was used in conjunction with this document

• Kubernetes 1.15 or newer

1.2 Prerequisites

The reader of this document must be familiar with:

• Docker concepts • Azure console and the Azure CLI (az) • Kubernetes concepts, installation, and administration • Kubernetes CLI, kubectl • Apache Kafka configuration

1.3 Prepare Local Environment

General:

The following infrastructure should already be in place or created:

• A Linux or MacOS machine equipped for building Docker images


• The following software must already be downloaded to the Linux or macOS machine equipped for building Docker images.

Note: All software must be for Linux!

• TIBCO Apache Kafka 2.5.0 has been downloaded • The tibakd_aks_files_2.5.zip (Docker and Kubernetes build files) has been downloaded

from https://community.tibco.com/wiki/tibcor-messaging-article-links-quick-access • Create a directory, place tibakd_aks_files_25.zip in the directory. Unzip

tibakd_aks_files_2.5.zip. • Unzip the TIB_msg-akd-core-2.5.0_linux_x86_64.zip, and place TIB_msg-akd-core-

2.5.0_linux_x86_64.tar.gz in the newly created tibakd_aks_files/docker/zookeeper/bin/tar and the tibakd_aks_files/docker/kafka/bin/tar directories. These are required to build the Docker images. The deb and rpm files are not required, and can be discarded.

1.4 Prepare Preliminary Azure Account and Kubernetes Configuration

Use the following to prepare the preliminary environment to install the TIBCO messaging components on AKS.

• An active Azure account is required. If necessary, create an account at http://portal.azure.com and follow the on-screen instructions.

• Install the Azure CLI on the workstation used. • Install Docker on the workstation to build the TIBCO EMS images. • Install the kubectl command-line tool to manage and deploy applications to Kubernetes in

Azure from a workstation.

1.5 Apache Kafka Architecture

This document will outline the creation of the Kafka architecture in AKS. Using this guide, along with the accompanying software, will create:

• Kubernetes cluster that spans multiple availability zones. • Three Zookeeper instances that guarantee high availability of cluster coordinator, each

deployed in a different availability zone. • Six Kafka brokers, two in each availability zone. • Kafka cluster has been configured with replication factor of three and rack-awareness,

which guarantees that each record will be replicated across all three availability zones. Outage of single zone will not interrupt the Kafka service, nor cause data unavailability.

• (Classic - Kubernetes) Load Balancer to access the Kafka cluster. • Kafka and Zookeeper are configured to use Azure managed disks for persisted storage. The

broker does not need to copy data from all assigned partitions, because it is already locally available.


2 Preparing the Zookeeper and Kafka Docker Images

This section will outline how to build the Docker images for Zookeeper and Kafka, and how upload the images to the Azure Container Registry (ACR).

2.1 Build Docker images

• Ensure Docker is running on your workstation. Use docker images to verify Docker is available.

• Open a terminal shell and navigate to the directory where the tibakd_aks_files directory is located. The docker and kubernetes directories will be underneath, with several subdirectories.

• The tib_msg-akd-core-2.5.0_linux_x86_64.tar.gz should already be in the docker/zookeeper/bin/tar and the docker/kafka/bin/tar directory.

• Navigate to the docker/zookeeper directory. • Execute the make build command to build the Zookeeper - Docker image. Alternatively, if

you do not have make utility installed, use the run_build.sh script. The Docker image, tibco/zookeeper with the 2.5.0 tag will be created.

• Navigate to the docker/kafka directory. • Execute make build command to build the Kafka – Docker image. Alternatively, if you do

not have make utility installed, use the run_build.sh script. The Docker image, tibco/kafka with the 2.5.0 tag will be created.

• To test the Docker images, open a second terminal shell, and navigate the docker/zookeeper directory. Execute make run command to start the Zookeeper – Docker image. In the first terminal shell, and navigate the docker/kafka directory. Execute make run command to start the Kafka – Docker image. Both should start successfully, and the Kafka broker should connect to Zookeeper.

2.2 Configure the Azure Container Registry

New ACR registry must be created to host the Zookeeper and Kafka Docker images. Use this section to create the necessary ACR registry. One is required, but two separate registries can be used.

2.2.1 Create the Azure Container Registry • Create a new ACR registry, such as tibakd (if only using one registry for all

components). The registry can be created via the Azure CLI or via the console. Please note the name and loginserver of your ACR registry.


Figure 1 - Create ACR Registry

• Login into the newly created Azure ACR from the Azure CLI. > az acr login --name tibakd

Figure 2 - Login into the ACR

2.2.2 Push the Docker Images to ACR Once the Docker image is ready, the image can be tagged and pushed to ACR. Tag the image and push the Docker image to the ACR registry using the URL of the appropriate registry. Use the same steps for both the Zookeeper and Kafka images.

> docker tag tibco/zookeeper:2.5.0 /tibco/zookeeper:latest

Figure 3 - Tag the Docker Image

> docker push /tibco/zookeeper:latest

Figure 4 - Push the Docker image to ACR


3 Azure AKS Setup

3.1 Create a New Azure Kubernetes Service (AKS)

A new Kubernetes cluster must be created in AKS. Use the following to build a new Kubernetes Service in Azure. This can be created via the Azure Portal of the Azure CLI. This document will outline building the cluster via the Azure portal.

• Sign into the Azure portal at https://portal.azure.com/ • In the top left-hand corner of the Azure portal, select Create a resource > Kubernetes

Service. • Select a Subscription and Resource group. These should be the same subscription and

Resource group. Note: The same Resource group should be used throughout. • Provide a new Kubernetes Cluster Name, Region, (Kubernetes version (must be at least

1.15.11), and a DNS name prefix, such as tibakd. • For Scale, select the node size. Recommend a D8s_v3 (8 core / 32 GB RAM). The instance

size can be adjusted to meet the requirements. Must be a minimum of 6 vCPU / 20 GB RAM. If lowered, the resource limits in the zookeeper and kafka yaml files must also be decreased.

• Select a node count of 6. • Leave virtual nodes disabled and VM scale sets disabled, if desired. • Click on Next: Authentication • Select to create a new service principal • Click on Yes to Enable RBAC • Click on Next: Networking • Choose either Yes or No for application routing, depending on requirements • Choose either Basic or Advanced for Network configuration. Recommend using Basic. • Use the defaults for monitoring • Wait for the Running the Validation to complete, with validation passed. Fix any issues

before continuing! • Click on Create. It will take several minutes to complete.


Figure 5 - Kubernetes Cluster creation

3.2 Configuring Kubectl to connect to Azure Kubernetes Service

With AKS, the Kubernetes command line tool, kubectl, is used to configure the Kubernetes cluster for EMS on AKS.

3.2.1 Configure Kubectl to connect to AKS After the Kubernetes cluster has been built, kubectl must be configured to connect to the cluster on AKS. Use the following example to set the credentials for kubectl. $ az aks get-credentials \ > --resource-group \ > --name Merged "" as current context in /Users//.kube/config

Figure 6 - Configure Kubectl

Use kubectl get nodes as shown in the following example to verify connecting to the cluster. $ kubectl get nodes NAME STATUS ROLES AGE VERSION aks-agentpool-12235351-vmss000000 Ready agent 6m50s v1.15.11 aks-agentpool-12235351-vmss000001 Ready agent 6m49s v1.15.11


aks-agentpool-12235351-vmss000002 Ready agent 6m49s v1.15.11 aks-agentpool-12235351-vmss000003 Ready agent 6m45s v1.15.11 aks-agentpool-12235351-vmss000004 Ready agent 6m50s v1.15.11 aks-agentpool-12235351-vmss000005 Ready agent 6m44s v1.15.11

Figure 7 - Verify connecting to the Kubernetes Cluster

3.3 Update the newly created AKS to access the ACR(s)

In Azure, the Azure Kubernetes Service (AKS) must be updated to access the ACR(s) containing the Docker containers. Note: a newer version of the Azure-cli must be installed. Version 2.7.0 is recommended. Use az –version to determine which version if the azure-cli is installed. Use the following to update your AKS. Do this for each ACR. Note: this is the ACR name, and not the LoginServer name. az aks update -n tibakd -g --attach-acr

Figure 8 - Update the AKS for the ACRs


4 Deploy Zookeeper and Kafka in AKS

The Zookeeper and Kafka containers can now be deployed in AKS. With AKS, the Kubernetes command line tool, kubectl, is used to configure the Kubernetes cluster for Apache Kafka on AKS.

4.1 Deploying the AKD Core on AKS

The zookeeper.yaml, kafka.yaml, and the kafka-service.yaml files will need a modification for the environment. This section will outline the modifications.

4.1.1 Zookeeper The zookeeper.yaml must be modified for the appropriate ACR repository defined in section 2.2. Find and modify the following line in zookeeper.yaml from to the ACR used for the Zookeeper container similar as the examples below: image: /zookeeper:latest image: tibakd.azurecr.io/tibco/zookeeper:latest

Figure 9 - Zookeeper.yaml modification

No other modifications are required or recommended, unless familiar with Kubernetes configurations.

4.1.2 Kafka

4.1.2.1 Kafka.yaml The kafka.yaml must be modified for the appropriate ACR repository defined in section 2.2. Find and modify the following line in kafka.yaml from to the ACR used for the Zookeeper container similar as the examples below: image: /kafka:latest image: tibakd.azurecr.io/tibco/kafka:latest

Figure 10 - Kafka.yaml modification

No other modifications are required or recommended, unless familiar with Kubernetes configurations.


4.1.2.2 Kafka-Service.yaml The kafka-service.yaml file provides the service ports for internally and externally accessing Kafka. The following example shows just the services for kafka-0 broker. Brokers 1-5 will be similar. apiVersion: v1 kind: Service metadata: name: broker namespace: kafka annotations: service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" spec: ports: - port: 9092 name: client clusterIP: None selector: app: kafka --- apiVersion: v1 kind: Service metadata: name: kafka-0 namespace: kafka labels: app: kafka-0 spec: type: LoadBalancer ports: - port: 32400 (1) name: client-external targetPort: 9093 nodePort: 32400 (1) protocol: TCP selector: statefulset.kubernetes.io/pod-name: kafka-0 loadBalancerSourceRanges: - (2)

Figure 11 - Kafka-service.yaml example

1) External port to the Kafka-0 Broker. This can be changed to another port number if desired. 2) Set the trusted IP Range to the IP range that can access the Kafka Broker externally.

0.0.0.0/0 is open to the world, and is NOT recommended.

4.2 Deploy Zookeeper and Kafka

After the configuration files are updated, Zookeeper and Kafka can be deployed into Azure AKS. Use the following to deploy Zookeeper and Kafka:


• Open command shell and navigate to the tibakd_aks_files/kubernetes directory. • Execute make build command to deploy Zookeeper and Kafka. Alternatively, if you do

not have make utility installed, issue below set of commands manually. Scripts will create new Kubernetes (kafka) namespace, services and dedicated storage class for Zookeeper and Kafka pods. > kubectl apply -f ./namespace.json > kubectl apply -f ./zookeeper/zookeeper-storage.yaml,./zookeeper/zookeeper-service.yaml,./zookeeper/zookeeper.yaml > kubectl apply -f ./kafka/kafka-storage.yaml,./kafka/kafka-service.yaml,./kafka/kafka.yaml Note – It will take several minutes for this step to complete!

• Execute the command below to review state of the cluster. Wait until all Zookeeper and Kafka instances are up and running. Again, this will take some time to complete. It can take ~30 minutes to complete. There should be no restarts.

> kubectl get pods -n kafka

• The output from get pods will look similar to the following example

NAME READY STATUS RESTARTS AGE kafka-0 1/1 Running 0 172m kafka-1 1/1 Running 0 166m kafka-2 1/1 Running 0 160m kafka-3 1/1 Running 0 155m kafka-4 1/1 Running 0 149m kafka-5 1/1 Running 0 143m zookeeper-0 1/1 Running 0 172m zookeeper-1 1/1 Running 0 172m zookeeper-2 1/1 Running 0 172m

Figure 12 - Get Pods example

4.3 Stopping, Deleting, or Accessing the AKD Servers

To stop the Zookeeper or Kafka servers without deleting the servers, use the kubectl scale operation to set its number of replicas to 0. For example: > kubectl scale --replicas=0 statefulset kafka –n kafka

> kubectl scale –replicas=0 statefulset zookeeper –n kafka

To start the Zookeeper cluster again, set its number of replicas back to 3: > kubectl scale --replicas=3 statefulset zookeeper –n kafka


To start the Kafka cluster again, set its number of replicas back to 6: > kubectl scale --replicas=6 statefulset kafka –n kafka

Figure 13 - To Stop and Start the Zookeeper and Kafka Statefulsets

To delete the AKD statefulset and service entirely, use the kubectl delete operation, or make clean from the tibakd_aks_files/ftl/kubernetes directory.

The corresponding pods will also be deleted. The PVC and PV will not be deleted, nor will the corresponding Zookeeper/Kafka data. The PV and PVC must be manually deleted to remove the data. Sometimes it is necessary to login into either a Zookeeper or Kafka container running in AKS. To access any of the pods, use the following: > kubectl –n kafka exec –it - -- “/bin/bash”


5 Testing the Kafka Environment

In the previous steps Zookeeper and Kafka were deployed to a Kubernetes cluster running in the AKS environment. Zookeeper provides only internal access limited to the Kubernetes cluster, so connecting to one of the Kafka pods is necessary to create a new topic to test with. On the other hand, Kafka brokers expose local, as well as the external service via the LB.

• Connect to one of Kafka pods and create a new sample topic. > kubectl exec -it kafka-0 -n kafka -- /bin/bash # kafka-topics.sh --create --zookeeper ${_KAFKA_ZOOKEEPER_CONNECT} --replication-factor 3 --partitions 100 --topic my-topic --config min.insync.replicas=2

• While still connected to the Kafka pod, try to send and receive messages. Please note the

use of the broker Kubernetes service, which is available only internally inside the cluster. # kafka-console-producer.sh --broker-list kafka-0.broker:9092,kafka-1.broker:9092,kafka-2.broker:9092 --topic my-topic --request-required-acks all # kafka-console-consumer.sh --bootstrap-server kafka-0.broker:9092 --from-beginning --topic my-topic

• To access Kafka directly from your local workstation, we have to use the LB endpoint of at

least one Kafka broker. Use the following command to get the external IP and port number assigned to each Kafka broker. Note the external IP and port.

> kubectl get services -n kafka -o wide

• Send and receive messages from your local workstation. Please note that you do not need to

provide the endpoint of every broker, the Kafka client will discover topology of the cluster once it establishes connectivity with any of the nodes. In the following example, 34.10.235.120:32400 is the external IP and port. Note: this is assumed that the service port is set to 32400. > kafka-console-producer.sh --broker-list 34.10.235.120:32400 --topic my-topic --request-required-acks all > kafka-console-consumer.sh --bootstrap-server 34.10.235.120:32400 --from-beginning --topic my-topic

Documents

How to Configure Apache Kafka 2.5 on AKS...3 Azure AKS Setup 3.1 Create a New Azure Kubernetes Service (AKS) A new Kubernetes cluster must be created in AKS. Use the following to build