Deploying a Kafka Cluster with mTLS on Kubernetes using Strimzi and Helm¶

Note

This guide is automated in this GitHub repo’s deploy.sh script:

https://github.com/jay-johnson/rust-with-strimzi-kafka-and-tls/blob/main/kubernetes/deploy.sh

Kafka Operator Architecture

The CNCF Landscape is a great resource for anything you want to run in kubernetes. There’s many streaming and messaging options, and today I want to deploy a tls-secured Kafka cluster using Strimzi and Let’s Encrypt for tls encryption in transit.

Prerequisites¶

Choosing Kubernetes means your organization is signing up for an ever-changing ecosystem, and this guide will likely be dated in the coming years.

Minikube¶

I’m running a minikube cluster on kubernetes 1.24.3 with 48 CPUs and 128 GB ram.

minikube profile list -o json | jq -r '.valid[0].Config | "cpus: " + (.CPUs | tostring) + " memory: " + (.Memory | tostring)'
cpus: 48 memory: 128000

kubectl get nodes | grep minikube
minikube   Ready    control-plane   13d   v1.24.3

Helm¶

I’m using helm 3.9.3:

helm version --short
v3.9.3+g414ff28

Getting Started¶

Deploy the Strimzi Operator¶

helm upgrade --install \
    -n st \
    --create-namespace \
    --set watchAnyNamespace=true \
    --set logLevel=INFO,fullReconciliationIntervalMs=20000 \
    st \
    strimzi/strimzi-kafka-operator

Wait for the Strimzi Operator¶

kubectl wait -n st --for=condition=Ready pod -l name=strimzi-cluster-operator --timeout=240s

Deploy the Cluster’s Certificate Authority (CA)¶

For more information, please refer to the github repo on: How to build and deploy your own CA for a Strimzi Kafka cluster

ca_file="./tls/ca.pem"
ca_key_file="./tls/ca-key.pem"

kubectl delete secret -n dev dev-cluster-ca-cert --ignore-not-found
kubectl create secret -n dev generic dev-cluster-ca-cert --from-file=ca.crt="${ca_file}"
kubectl label secret -n dev dev-cluster-ca-cert strimzi.io/kind=Kafka strimzi.io/cluster=dev
kubectl annotate secret -n dev dev-cluster-ca-cert strimzi.io/ca-cert-generation=0

kubectl delete secret -n dev dev-cluster-ca --ignore-not-found
kubectl create secret -n dev generic dev-cluster-ca --from-file=ca.key="${ca_key_file}"
kubectl label secret -n dev dev-cluster-ca strimzi.io/kind=Kafka strimzi.io/cluster=dev
kubectl annotate secret -n dev dev-cluster-ca strimzi.io/ca-key-generation=0

kubectl delete secret -n dev dev-clients-ca-cert --ignore-not-found
kubectl create secret -n dev generic dev-clients-ca-cert --from-file=ca.crt="${ca_file}"
kubectl label secret -n dev dev-clients-ca-cert strimzi.io/kind=Kafka strimzi.io/cluster=dev
kubectl annotate secret -n dev dev-clients-ca-cert strimzi.io/ca-cert-generation=0

# load the client CA key
kubectl delete secret -n dev dev-clients-ca --ignore-not-found
kubectl create secret -n dev generic dev-clients-ca --from-file=ca.key="${ca_key_file}"
kubectl label secret -n dev dev-clients-ca strimzi.io/kind=Kafka strimzi.io/cluster=dev
kubectl annotate secret -n dev dev-clients-ca strimzi.io/ca-key-generation=0

Deploy the Cluster’s Listener TLS Assets as a Kubernetes Secret¶

tls_chain_file="./tls/server-cert-chain.pem"
tls_key_file="./tls/server-key.pem"

kubectl create secret generic tls-kafka-cluster-0-server -n dev --from-file=server-cert-chain.pem=${tls_chain_file} --from-file=tls.key=${tls_key_file}

Deploy the Kafka Cluster¶

With the CA and Listener TLS assets deployed, you can start the Kafka cluster with:

kubectl apply -n dev -f ./kubernetes/dev.yaml

Wait for the Cluster¶

kubectl wait kafka/dev -n dev --for=condition=Ready --timeout=300s

Verify Client mTLS¶

Note

For local testing you will need to add cluster-0-broker-0.redten.io to /etc/hosts so dns resolution works:

echo "ssl test" | openssl s_client -connect \
    cluster-0-broker-0.redten.io:32151 \
    -key ./kubernetes/tls/client-key.pem \
    -cert ./kubernetes/tls/client.pem \
    -CAfile ./kubernetes/tls/ca.pem \
    -verify_return_error \
    && echo "strimzi kafka cluster is working with self-signed tls assets!"

Note

Clients must provide the tls key, cert and CAfile for establishing a valid mutual tls connection

Create Kafka Topic for Rust Messaging¶

cat <<EOL | kubectl apply -n dev -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: testing
  labels:
    strimzi.io/cluster: "dev"
spec:
  partitions: 3
  replicas: 3
EOL

Monitoring¶

Cruise Control¶

Koperator Managing a Kafka Cluster and Monitored with CruiseControl

Kubectl Port-Forward¶

Note

This is not a secure way to host for production but it works for demonstration purposes:

kubectl port-forward -n dev svc/dev-cruise-control --address 0.0.0.0 9090:9090

Troubleshooting¶

Check for errors¶

What do I do if the strimzi operator is stuck in a reconciliation loop after trying to recreate a kafka cluster?

Restart the strimzi operator pod.
```
kubectl delete -n st po $(kubectl get po -n st | grep strimzi | awk '{print $1}')
```

Uninstall¶

Uninstall Kafka Cluster¶

kubectl delete -n dev -f ./kubernetes/dev.yaml

Uninstall Strimzi¶

Note

This will hang if there are any kafka resources still in use

helm delete -n st st

Delete Secrets¶

kubectl delete secret -n dev tls-kafka-cluster-0-server dev-cluster-ca-cert dev-cluster-ca dev-clients-ca-cert dev-clients-ca --ignore-not-found

Sources¶

For those that want to refer to the official docs, I followed the Strimzi with Let’s Encrypt guide.
How to build and deploy your own CA for a Strimzi Kafka cluster
Rust producer and consumer examples from: rust-rdkafka