A Guide to Azure Arc-Enabled Kubernetes

What is it all about?

We needed a way to manage already existing and future Kubernetes deployments in one central tooling. Red Hat OpenShift Container Platform, Azure AKS and Exoscale SKS (managed Kubernetes) were in focus.

Therefore we had a look at Azure Arc-enabled Kubernetes in combination with Exoscale SKS.

What is Azure Arc-enabled Kubernetes?

Azure Arc-enabled Kubernetes enables you to attach Kubernetes clusters running anywhere so that you can manage and configure them in Azure. The main benefit is to have a central control plane to manage all of your Kubernetes resources which further allows you to have a consistent development and operation experience to run cloud-native applications across all of your on-premises or cloud infrastructure and on any Kubernetes platform.

Supported Kubernetes distributions

Azure Arc-enabled Kubernetes works with any Cloud Native Computing Foundation (CNCF) certified Kubernetes clusters. This includes clusters running on other public cloud providers (such as GCP or AWS) and clusters running on your on-premises data center (such as VMware vSphere or Azure Stack HCI).

Validated distributions from Microsoft (including but not limited to):

• RedHat OpenShift Container Platform
• VMware Tanzu
• SUSE Rancher (RKE & K3s)
• AWS EKS
• GCP GKE
• Detailed information can be found here

Pre-requisites

• Azure subscription (sandbox)
• Identity (user or Service principal) used to connect your cluster to Azure Arc
• Latest Azure CLI + connectedk8s cli extension
• An up–and running Kubernetes cluster (Exoscale SKS in our case)
  • with at least 850MB free for the Arc agents
  • and the corresponding kubeconfig file
• Registered providers for Azure Arc-enabled Kubernetes
  • az provider register --namespace Microsoft.Kubernetes
  • az provider register --namespace Microsoft.KubernetesConfiguration
  • az provider register --namespace Microsoft.ExtendedLocation
• To check if the providers are ready
  • az provider show -n Microsoft.Kubernetes -o table
  • az provider show -n Microsoft.KubernetesConfiguration -o table
  • az provider show -n Microsoft.ExtendedLocation -o table

Detailed information can be found here:

• Quickstart connect cluster
• Azure Arc jumpstart

Preparation

In our case we prepared an Exoscale SKS cluster with 3 medium nodes and a corresponding kubeconfig. For Windows you have to create a “.kube” folder in your user's profile and copy the kubeconfig in a file named “config”.

Also see the Exoscale SKS Quickstart on how to setup an Exoscale SKS cluster.

Steps to be taken in Azure:

• Login to your Azure environment
  • az login --tenant “<Your tenant id>"
  • az account set --subscription "<Your subscription id>"
• Create a resource group
  • az group create -l "West Europe" -n "ArcEnabledK8s-Test"
• Prepare environment variables for later steps
  • $env:tenantId="<your tenant id>"
  • $env:resourceGroup="ArcEnabledK8s-Test"
  • $env:arcClusterName="SKS-Cluster-Test"
• Install the Azure Arc for Kubernetes CLI extensions
  • az extension add --name connectedk8s
  • az extension add --name k8s-configuration
• Connect the Kubernetes cluster to Azure Arc
  • az connectedk8s connect --name $env:arcClusterName --resource-group $env:resourceGroup
• Upon completion, you will have your Kubernetes cluster connected as a new Azure Arc-enabled Kubernetes resource inside your resource group.
  • Check the started POD's in the "azure-arc" namespace
  • kubectl get pods -o wide -n azure-arc

First overview

Let’s check the Azure UI:

To also get information about the namespaces and the running workloads in Azure and more, a bearer token is needed.

Therefore we need to create a service account token in our Kubernetes cluster and generate the token for it. (How to create a service account token)

• kubectl create serviceaccount azure-test -n default
• kubectl create clusterrolebinding azure-test-binding --clusterrole cluster-admin --serviceaccount default:azure-test
• kubectl apply -f azure-test-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: azure-test-secret
  annotations:
    kubernetes.io/service-account.name: azure-test
type: kubernetes.io/service-account-token

• [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String((kubectl get secret azure-test-secret -o jsonpath='{$.data.token}')))

Test deployment

We activated the quickstart demo application from Azure (azure-vote) to test the deployment via Azure and to see a running workload by using the Azure UI.

Navigate to Workloads \ Add with YAML \ Deploy a quickstart application… and follow the steps.

The link to the application will be posted directly after the deployment. Just wait some seconds for the service to be ready.

You can now check the running POD’s in your Kubernetes environment for the azure-vote demo application.

• kubectl get pods -o wide -n azure-vote

Monitoring & Logging

Monitoring via managed Prometheus can be enabled using an Azure monitor workspace and logging via Azure log analytics workspace. (Managed Grafana could be an additional option on top)

Agents for metric and log collection are already running on your Kubernetes nodes.

Click on Monitoring \ Insights and follow the prompts to create a log analytics & Azure Monitor workspace and to activate the managed Prometheus service.

After some seconds you’ll have the possibility to check for the metrics and logs in Azure.

To search for POD/container specific logs, you can do a query like the following:

let SearchValue =  "vote";
ContainerLogV2
| where * contains tostring(SearchValue)
| take 1000

What else?

There are for sure additional things that you can now check upon the first basic setup and tests, for example:

• Try to deploy additional applications using GitOps-based configuration management.
• Enforce threat protection using Microsoft Defender for Kubernetes.
• Applying policies with Azure Policy for Kubernetes.
• Deploy other workloads/services like machine learning / data services / service mesh / …

You can find more information on scenarios and enhanced functionality here.

I hope you got a first overview of the possibilities and we would be more than happy to help you in your journey!