GitOps for AKS with Bicep

In this post, I'm going to show how you can deploy and configure GitOps (Flux) in an AKS cluster with Bicep. I wanted to get this out there as soon as I could, as it's not very well documented (yet), which I've found quite frustrating for something that (as of this blog post) is in public preview 😐. Hopefully, this'll help save others from the pain that I went through trying to get this working!

A lot of the current documentation is very Azure CLI-specific, so there was a lot of trial and error trying to get this configured using the limited REST API specs.

Prerequisites

Firstly, make sure you have registered the below feature flags and providers in your subscription:

# Register the following feature flags
az feature register --namespace Microsoft.ContainerService -n AKS-ExtensionManager 
az feature register --namespace Microsoft.KubernetesConfiguration -n extensions 
az feature register --namespace Microsoft.KubernetesConfiguration -n fluxConfigurations 

# Keep running the following until "Registered" (may take up to 20 minutes)
az feature list -o table --query "[?name=='Microsoft.ContainerService/AKS-ExtensionManager' || name=='Microsoft.KubernetesConfiguration/extensions' || name=='Microsoft.KubernetesConfiguration/fluxConfigurations'].{Name:name,State:properties.state}" 

# When all say "Registered" then re-register the AKS and related resource providers
az provider register --namespace Microsoft.ContainerService 
az provider register --namespace Microsoft.Kubernetes 
az provider register --namespace Microsoft.KubernetesConfiguration 

# Monitor the registration process
az provider show -n Microsoft.ContainerService -o table 
az provider show -n Microsoft.Kubernetes -o table 
az provider show -n Microsoft.KubernetesConfiguration -o table

Create Azure AD Pod Identity Exception

If you have Azure AD Pod Identity enabled - In the podIdentityProfile of your cluster resource, create an exception for the flux extension:

podIdentityProfile: {
  enabled: true
  userAssignedIdentities: []
  userAssignedIdentityExceptions: [
    {
      name: 'flux-extension-exception'
      namespace: 'flux-system'
      podLabels: {
        'app.kubernetes.io/name': 'flux-extension'
      }
    }
  ]
}

Install Flux Extension

With the above exception in place, install Flux by creating a Microsoft.KubernetesConfiguration/extensions resource:

resource fluxExtension 'Microsoft.KubernetesConfiguration/extensions@2021-09-01' = {
  name: 'flux'
  scope: aks
  properties: {
    autoUpgradeMinorVersion: true
    configurationProtectedSettings: {}
    configurationSettings: {
      'helm-controller.enabled': 'true'
      'source-controller.enabled': 'true'
      'kustomize-controller.enabled': 'true'
      'notification-controller.enabled': 'true'
      'image-automation-controller.enabled': 'true'
      'image-reflector-controller.enabled': 'true'
    }
    extensionType: 'microsoft.flux'
    releaseTrain: 'Stable'
    scope: {
      cluster: {
        releaseNamespace: 'flux-system'
      }
    }
  }
}

Apply Flux Configuration

With the Flux controllers now deployed, create a Microsoft.KubernetesConfiguration/fluxConfigurations resource. This is used to connect to your source repo and apply the manifests that define the desired state of your cluster.

In this example, I'm bootstrapping a staging cluster, so will configure flux to synchronise the staging Kustomize overlays:

resource fluxConfiguration 'Microsoft.KubernetesConfiguration/fluxConfigurations@2022-01-01-preview' = {
  name: 'flux2-kustomize-helm-example'
  scope: aks
  properties: {
    configurationProtectedSettings: {}
    gitRepository: {
      repositoryRef: {
        branch: 'gitops-aks'
      }
      syncIntervalInSeconds: 600
      timeoutInSeconds: 600
      url: 'https://github.com/thepaulmacca/flux2-kustomize-helm-example'
    }
    kustomizations: {
      staging: {
        path: 'clusters/staging'
        prune: true
      }
    }
    namespace: 'flux-system'
    scope: 'cluster'
    sourceKind: 'GitRepository'
  }
  dependsOn: [
    fluxExtension
  ]
}

You'll notice that the repo is connecting to a gitops-aks branch. For the configuration to apply successfully, I had to change the name in the sourceRef for each overlay, to match the name of my configuration (the name of my repo):

sourceRef:
  kind: GitRepository
  name: flux2-kustomize-helm-example

After a short while, you should see the configuration applied to your cluster:

You'll also see the deployments:

Summary

So, there we have it - GitOps deployed and configured on an AKS cluster with Bicep! I really wish there were more documentation around authenticating to private Azure/GitHub repos, but hopefully it's coming soon.

To try this out for yourself, you can find a link to the source code that deploys a cluster and config here.

Thanks for reading!