# Deploying a Kafka Cluster with mTLS on Kubernetes using Strimzi and Helm

```{eval-rst}
.. note::

   This guide is automated in this GitHub repo's deploy.sh script:

   https://github.com/jay-johnson/rust-with-strimzi-kafka-and-tls/blob/main/kubernetes/deploy.sh
```

![Kafka Operator Architecture](https://strimzi.io/docs/operators/latest/images/overview/kafka-concepts-supporting-components.png)

The [CNCF Landscape](https://landscape.cncf.io/) is a great resource for anything you want to run in kubernetes. There's many streaming and messaging options, and today I want to deploy a tls-secured Kafka cluster using [Strimzi](https://banzaicloud.com/docs/supertubes/kafka-operator/install-kafka-operator/) and Let's Encrypt for tls encryption in transit.

## Prerequisites

```{eval-rst}
Choosing Kubernetes means your organization is signing up for an ever-changing ecosystem, and this guide will likely be dated in the coming years.
```

### Minikube

I'm running a minikube cluster on kubernetes `1.24.3` with `48` CPUs and `128` GB ram.

```bash
minikube profile list -o json | jq -r '.valid[0].Config | "cpus: " + (.CPUs | tostring) + " memory: " + (.Memory | tostring)'
cpus: 48 memory: 128000
```

```bash
kubectl get nodes | grep minikube
minikube   Ready    control-plane   13d   v1.24.3
```

### Helm

I'm using helm ``3.9.3``:

```bash
helm version --short
v3.9.3+g414ff28
```

## Getting Started

### Deploy the Strimzi Operator

```bash
helm upgrade --install \
    -n st \
    --create-namespace \
    --set watchAnyNamespace=true \
    --set logLevel=INFO,fullReconciliationIntervalMs=20000 \
    st \
    strimzi/strimzi-kafka-operator
```

#### Wait for the Strimzi Operator

```bash
kubectl wait -n st --for=condition=Ready pod -l name=strimzi-cluster-operator --timeout=240s
```

### Deploy the Cluster's Certificate Authority (CA)

For more information, please refer to the github repo on: [How to build and deploy your own CA for a Strimzi Kafka cluster](https://github.com/scholzj/strimzi-custom-ca-test)

```bash
ca_file="./tls/ca.pem"
ca_key_file="./tls/ca-key.pem"

kubectl delete secret -n dev dev-cluster-ca-cert --ignore-not-found
kubectl create secret -n dev generic dev-cluster-ca-cert --from-file=ca.crt="${ca_file}"
kubectl label secret -n dev dev-cluster-ca-cert strimzi.io/kind=Kafka strimzi.io/cluster=dev
kubectl annotate secret -n dev dev-cluster-ca-cert strimzi.io/ca-cert-generation=0

kubectl delete secret -n dev dev-cluster-ca --ignore-not-found
kubectl create secret -n dev generic dev-cluster-ca --from-file=ca.key="${ca_key_file}"
kubectl label secret -n dev dev-cluster-ca strimzi.io/kind=Kafka strimzi.io/cluster=dev
kubectl annotate secret -n dev dev-cluster-ca strimzi.io/ca-key-generation=0

kubectl delete secret -n dev dev-clients-ca-cert --ignore-not-found
kubectl create secret -n dev generic dev-clients-ca-cert --from-file=ca.crt="${ca_file}"
kubectl label secret -n dev dev-clients-ca-cert strimzi.io/kind=Kafka strimzi.io/cluster=dev
kubectl annotate secret -n dev dev-clients-ca-cert strimzi.io/ca-cert-generation=0

# load the client CA key
kubectl delete secret -n dev dev-clients-ca --ignore-not-found
kubectl create secret -n dev generic dev-clients-ca --from-file=ca.key="${ca_key_file}"
kubectl label secret -n dev dev-clients-ca strimzi.io/kind=Kafka strimzi.io/cluster=dev
kubectl annotate secret -n dev dev-clients-ca strimzi.io/ca-key-generation=0
```

### Deploy the Cluster's Listener TLS Assets as a Kubernetes Secret

```bash
tls_chain_file="./tls/server-cert-chain.pem"
tls_key_file="./tls/server-key.pem"

kubectl create secret generic tls-kafka-cluster-0-server -n dev --from-file=server-cert-chain.pem=${tls_chain_file} --from-file=tls.key=${tls_key_file}
```

### Deploy the Kafka Cluster

With the CA and Listener TLS assets deployed, you can start the Kafka cluster with:

```bash
kubectl apply -n dev -f ./kubernetes/dev.yaml
```

#### Wait for the Cluster

```bash
kubectl wait kafka/dev -n dev --for=condition=Ready --timeout=300s
```

### Verify Client mTLS

```{eval-rst}
.. note:: For local testing you will need to add ``cluster-0-broker-0.redten.io`` to ``/etc/hosts`` so dns resolution works:
```

```bash
echo "ssl test" | openssl s_client -connect \
    cluster-0-broker-0.redten.io:32151 \
    -key ./kubernetes/tls/client-key.pem \
    -cert ./kubernetes/tls/client.pem \
    -CAfile ./kubernetes/tls/ca.pem \
    -verify_return_error \
    && echo "strimzi kafka cluster is working with self-signed tls assets!"
```

```{eval-rst}
.. note:: Clients must provide the tls key, cert and CAfile for establishing a valid mutual tls connection
```

### Create Kafka Topic for Rust Messaging

```bash
cat <<EOL | kubectl apply -n dev -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: testing
  labels:
    strimzi.io/cluster: "dev"
spec:
  partitions: 3
  replicas: 3
EOL
```

## Monitoring

### Cruise Control

![Koperator Managing a Kafka Cluster and Monitored with CruiseControl](/_images/Kafka-Cruise-Control-with-NodePort.png)

#### Kubectl Port-Forward

```{eval-rst}
.. note::

   This is not a secure way to host for production but it works for demonstration purposes:

   ::

        kubectl port-forward -n dev svc/dev-cruise-control --address 0.0.0.0 9090:9090
```

## Troubleshooting

### Check for errors

1.  What do I do if the strimzi operator is stuck in a reconciliation loop after trying to recreate a kafka cluster? 

    Restart the strimzi operator pod.

    ```bash
    kubectl delete -n st po $(kubectl get po -n st | grep strimzi | awk '{print $1}')
    ```

## Uninstall

### Uninstall Kafka Cluster

```bash
kubectl delete -n dev -f ./kubernetes/dev.yaml
```

### Uninstall Strimzi

```{eval-rst}
.. note:: This will hang if there are any ``kafka`` resources still in use
```

```bash
helm delete -n st st
```

### Delete Secrets

```bash
kubectl delete secret -n dev tls-kafka-cluster-0-server dev-cluster-ca-cert dev-cluster-ca dev-clients-ca-cert dev-clients-ca --ignore-not-found
```

## Sources

1.  For those that want to refer to the official docs, I followed the [Strimzi with Let's Encrypt guide](https://strimzi.io/blog/2021/05/07/deploying-kafka-with-lets-encrypt-certificates/).
1.  [How to build and deploy your own CA for a Strimzi Kafka cluster](https://github.com/scholzj/strimzi-custom-ca-test)
1.  Rust producer and consumer examples from: [rust-rdkafka](https://github.com/fede1024/rust-rdkafka/tree/master/examples)