Developing Kubernetes Admission Controller with Kotlin — Fixing AKS add-on issue in UDR
I often get asked by developers to compare AKS with different managed Kubernetes offerings on the cloud, and I always point out one thing when it comes to the strength of AKS, which is AKS add-ons. Having components like Istio, Nginx, Flux, etc. as an add-on that their lifecycle being fully managed by Azure is very unique value proposition compared to other platforms. It’s worth noting though, the concept of add-ons is not specific only to AKS.
Ironically, I encountered an issue last week with AKS add-ons, particularly Flux and Nginx. And I ended up writing admission controller to fix it. Before I delve into the issue in detail, I want you understand one thing which is described in this documentation — “Required outbound network rules and FQDNs for AKS clusters”
“AKS uses an admission controller to inject the FQDN as an environment variable to all deployments under kube-system and gatekeeper-system. This ensures all system communication between nodes and API server uses the API server FQDN and not the API server IP.”
The crux of the issue lies in the fact that Flux and Nginx (precisely, the Application Routing add-on) operate within the flux-system and app-routing-system respectively where there is no admission controller injecting FQDN into those add-ons, so it ends up failing to run since it can’t reach API server. Addressing this problem is straightforward by injecting kubernetes.azure.com/set-kube-service-host-fqdn annotation to the pods in each namespaces manually, but I chose not to do it, instead do a little bit of coding with admission controller. I bet this article would be useful for developers who wants to create their own controller for Kubernetes. I choose Kotlin as it is the most productive language at my disposal at this moment. In case you aren’t a Java/Kotlin developer, you can still leverage this article to gain a comprehensive understanding of the underlying concept.
Kubernetes Admission Controllers
According to the Kubernetes official doc, here is the brief introduction of Kubernetes Admission Controllers
“In a nutshell, Kubernetes admission controllers are plugins that govern and enforce how the cluster is used. They can be thought of as a gatekeeper that intercept (authenticated) API requests and may change the request object or deny the request altogether. The admission control process has two phases: the mutating phase is executed first, followed by the validating phase. Consequently, admission controllers can act as mutating or validating controllers or as a combination of both. For example, the LimitRanger admission controller can augment pods with default resource requests and limits (mutating phase), as well as verify that pods with explicitly set resource requirements do not exceed the per-namespace limits specified in the LimitRange object (validating phase).”
Looking at the AKS cluster that I’m running currently, there are so many Mutating Webhooks and Admission Webhooks as shown below.
$ kubectl get mutatingwebhookconfigurations -A
NAME WEBHOOKS AGE
aks-node-mutating-webhook 1 203d
aks-webhook-admission-controller 1 203d
azure-wi-webhook-mutating-webhook-configuration 1 71d
capi-kubeadm-bootstrap-mutating-webhook-configuration 2 153d
capi-kubeadm-control-plane-mutating-webhook-configuration 2 153d
capi-mutating-webhook-configuration 9 153d
capz-mutating-webhook-configuration 7 153d
cert-manager-webhook 1 190d
defaults.webhook.kpack.io 1 201d
istio-revision-tag-default 4 141d
istio-sidecar-injector-1-19-0 2 141d
vpa-webhook-config 1 203d
$ kubectl get ValidatingWebhookConfiguration
NAME WEBHOOKS AGE
aks-node-validating-webhook 1 203d
azure-policy-validating-webhook-configuration 1 195d
capi-kubeadm-bootstrap-validating-webhook-configuration 2 153d
capi-kubeadm-control-plane-validating-webhook-configuration 3 153d
capi-validating-webhook-configuration 12 153d
capz-validating-webhook-configuration 10 153d
cert-manager-webhook 1 190d
gatekeeper-validating-webhook-configuration 1 195d
istio-validator-1-19-0-istio-system 1 141d
istiod-default-validator 1 141d
validation.webhook.kpack.io 1 201dThese controllers are the critical components of AKS for various features. For example, Azure Policy for AKS(azure-policy-validating-webhook-configuration) is enforcing policies using OPA by validating API request. Workload Identity add-on(azure-wi-webhook-mutating-webhook-configuration) is Mutating Admission Webhooks which takes serviceaccountname to issue the token for workload identity and inject AZURE_FEDERATED_TOKEN_FILE into the pods. With Istio add-on enabled, Istio sidecar is injected by istio-sidecar-injector, another Mutating Admission Webhooks.
Before delving into writing the admission hooks, it’s essential to highlight a few key points:
- Admission controller is just simple RESTful application which exposes ‘POST’ endpoint.
- Kubernetes mandates TLS enforcement, as it doesn’t permit plain HTTP endpoints for admission controllers.
In this article, I will demonstrate the process of developing Mutating Webhooks with Kotlin and Spring on Kubernetes. I’ll start by addressing the second point first.
Configure cert manager and certificate
There are so many different ways out there to generate the TLS certificate like openssl, cfssl, keytool, etc. but I decided to use cert manager which I think is the most natural since we’re developing Kubernetes admission controller. Not only that, cert-manager supports JKS keystore out of the box which could save us from a lot of certificate related operations.
Cert-manager provides multiple installation methods for Kubernetes, and I’ve chosen Helm — https://cert-manager.io/docs/installation/
$ helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.14.2 \
--set installCRDs=trueNext, I’ll create a JKS keystore using cert-manager. The crucial aspect here is the definition of dnsNames, which must be mapped to the service of the admission controller that we’ll be creating later on.
kind: Secret
apiVersion: v1
metadata:
name: jks-password-secret
data:
password: Y2hhbmdlaXQ=
type: Opaque
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: aks-fqdn-controller-ca-issuer
namespace: default
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: aks-fqdn-controller-cert
namespace: default
spec:
keystores:
jks:
passwordSecretRef:
name: jks-password-secret
key: password
create: true
dnsNames:
- aks-fqdn-controller
- aks-fqdn-controller.default
- aks-fqdn-controller.default.svc
- aks-fqdn-controller.default.svc.cluster.local
secretName: aks-fqdn-controller-cert-tls-secret
commonName: aks-fqdn-controller
issuerRef:
name: aks-fqdn-controller-ca-issuer
kind: IssuerWrite Mutating Admission Controller in Kotlin
Let’s tackle the first point in the previous chapter. For the code reference, I took some of the code from Baeldung, specifically domain class— Creating a Kubernetes Admission Controller in Java. I won’t cover the sample code line by line, you could find it on my GitHub repository.
I start with POST REST endpoint.
@RestController
class AdmissionReviewController(private val admissionService: AdmissionService) {
@PostMapping(path = ["/mutate"])
fun processAdmissionReviewRequest(@RequestBody request: ObjectNode): AdmissionReviewResponse =
admissionService.processAdmission(request)
}It returns the value of AdmissionReviewResponse which looks like below.
{
“apiVersion”: “admission.k8s.io/v1”,
“kind”: “AdmissionReview”,
“response”: {
“uid”: “<value from request.uid>”,
“allowed”: true,
“patchType”: “JSONPatch”,
“patch”: “eyJvcCI6ImFkZCIsInBhdGgiOiIvc3BlYy9jb250YWluZXJzLzAvZW52IiwidmFsdWUiOlt7Im5hbWUiOiJLVUJFIiwidmFsdWUiOiJ0cnVlIn1dfQ==”
}
}In the service classAdmissionService, I have a function addAnnotations which create the patch request and encode it with Base64. It essentially adds the annotation kubernetes.azure.com/set-kube-service-host-fqdn: true to the pods.
private fun addAnnotations(body: ObjectNode): AdmissionReviewData {
// Create a PATCH object
val patch: String = """
[
{
"op": "add",
"path": "/metadata/annotations",
"value": {"kubernetes.azure.com/set-kube-service-host-fqdn":"true"}
}
]
"""
return AdmissionReviewData(
allowed = true,
uid = body.path("request").required("uid").asText(),
patch = Base64.getEncoder().encodeToString(patch.toByteArray()),
patchType = "JSONPatch"
)
}One last piece is setting up SSL. We will let our Kotlin app to pickup the keystore and trust store from Kubernetes secrets that is created by cert-manager in the previous chapter. In the application.yml,
server.port: 8443
server.ssl:
enabled: true
key-store: ${CERT_PATH}/keystore.jks
key-store-password: ${PASSWORD}
trust-store: ${CERT_PATH}/truststore.jks
trust-store-password: ${PASSWORD}
logging:
level:
root: debugThen here is the Kubernetes yaml file. As I stressed before, you should be catious to match the name of the service to dnsNames for cert-manager.
apiVersion: v1
kind: Service
metadata:
name: aks-fqdn-controller
labels:
app: aks-fqdn-controller
spec:
publishNotReadyAddresses: true
ports:
- port: 8443
targetPort: 8443
selector:
app: aks-fqdn-controller
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: aks-fqdn-controller
labels:
app: aks-fqdn-controller
spec:
replicas: 2
selector:
matchLabels:
app: aks-fqdn-controller
template:
metadata:
name: aks-fqdn-controller
labels:
app: aks-fqdn-controller
spec:
containers:
- name: mutator
image: eggboy/aks-fqdn-mutating-webhook:0.0.1
ports:
- containerPort: 8443
name: https
imagePullPolicy: Always
env:
- name: PASSWORD
valueFrom:
secretKeyRef:
key: password
name: jks-password-secret
- name: CERT_PATH
value: /opt/secret
volumeMounts:
- mountPath: /opt/secret
name: cert
volumes:
- name: cert
secret:
secretName: aks-fqdn-controller-cert-tls-secretFinal step is to create a MutatingWebhookConfiguration. One critical thing not to miss is the annotation cert-manager.io/inject-ca-from which injects the CA, so that API server can connect to the admission controller with TLS. Look at the namespaceSelector, I explicitly select two namespaces, app-routing-system and flux-system.
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: aks-fqdn-controller
labels:
app: aks-fqdn-controller
annotations:
cert-manager.io/inject-ca-from: default/aks-fqdn-controller-cert
webhooks:
- admissionReviewVersions:
- v1
name: aks-fqdn-controller.default.svc.cluster.local
clientConfig:
service:
name: aks-fqdn-controller
namespace: default
path: /mutate
port: 8443
rules:
- operations: ["CREATE"]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
sideEffects: NoneOnDryRun
timeoutSeconds: 5
reinvocationPolicy: Never
failurePolicy: Fail
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: In
values: [ "app-routing-system", "flux-system" ]Wrapping Up
By leveraging cert-manager to delegate the certificate generation and JKS key store parts, I could quickly build and set up Mutating Admission Hooks with Kotlin on AKS. This approach of clean separation between application and certificate generation would help me to achieve faster iteration for future developments. For those of you reading this article, I hope this article proves to be a helpful guide. Feel free to leave any comments should you have questions or feedbacks.
Before closing this out, I wish all of you and family healthy and prosperous year 2024 as we just passed the lunar new year in this part of the world.
If you enjoyed my article, I’d appreciate a few claps or a follow. Get notified for the new articles by subscribing, and let’s stay connected on Linkedin. Thank you for your time and happy reading!
