Cloudera Docs

Deploying Kafka Connect clusters

You can deploy a Kafka Connect cluster by creating a KafkaConnect resource. The Kafka Connect workers are automatically configured to run in distributed mode. You can configure the number of workers. Each worker is a separate pod.

  • Ensure that the Strimzi Cluster Operator is installed and running. See Installation.
  • Ensure that you have a working Kafka cluster. The Kafka cluster does not need to be managed by Strimzi, and it does not need to run on Kubernetes.

  • Ensure that a namespace is available where you can deploy your cluster. If not, create one. 

    kubectl create namespace [***NAMESPACE***]
  • Ensure that the Secret containing credentials for the Docker registry where Cloudera Streams Messaging - Kubernetes Operator artifacts are hosted is available in the namespace where you plan on deploying your cluster. If the Secret is not available, create it.
    kubectl create secret docker-registry [***REGISTRY CREDENTIALS SECRET***] \
  --namespace [***NAMESPACE***] \
  --docker-server [****YOUR REGISTRY***] \
  --docker-username [***USERNAME***] \
  --docker-password "$(echo -n 'Enter your password: ' >&2; read -s password; echo >&2; echo $password)"

    • [***REGISTRY CREDENTIALS SECRET***] must be the same as the name of the Secret containing registry credentials that you created during Strimzi installation.

    • Replace [****YOUR REGISTRY***] with the server location of the Docker registry where Cloudera Streams Messaging - Kubernetes Operator artifacts are hosted. If your Kubernetes cluster has internet access, use container.repository.cloudera.com. Otherwise, enter the server location of your self-hosted registry.

    • Replace [***USERNAME***] with a username that provides access to the registry, and enter the corresponding password when prompted. If you are using container.repository.cloudera.com, enter your Cloudera credentials. Otherwise, enter credentials providing access to your self-hosted registry.

  • The following steps walk you through a basic cluster deployment example. If you want to deploy a Kafka Connect cluster that has third-party connectors or other types of plugins installed, see Installing Kafka Connect connector plugins.
  1. Create a YAML configuration that contains your KafkaConnect resource. 
    apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.9.0.1.4
  replicas: 3
  bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092
    • The spec.version property specifies the Kafka version to use. The property must specify a Cloudera Kafka version supported by Cloudera Streams Messaging - Kubernetes Operator. For example, 3.9.0.1.4. Do not add Apache Kafka versions, they are not supported. You can find a list of supported Kafka versions in the Release Notes.
    • The bootstrapServers property specifies the Kafka brokers to which to connect. Cloudera recommends providing multiple brokers to handle broker failures and enable connecting to another instance.

      The my-cluster-kafka-bootstrap.kafka:9092 value is the bootstrap of a Kafka cluster that is deployed on Kubernetes with Cloudera Streams Messaging - Kubernetes Operator. my-cluster is the name of the cluster specified in metadata.name of the Kafka resource. The kafka-bootstrap string is fixed. The string kafka after the dot is the namespace where the cluster is deployed.

  2. Deploy the resource. 
    kubectl apply --filename [***YAML CONFIG***] --namespace [***NAMESPACE***]

    The namespace where you deploy Kafka Connect must be watched by the Strimzi Cluster Operator.

  3. Verify that the KafkaConnect resource is ready. 
    kubectl get kafkaconnect [***CONNECT CLUSTER NAME***] --namespace [***NAMESPACE***] --watch
If cluster deployment is successful, you should see an output similar to the following.
NAME                 DESIRED REPLICAS   READY
my-connect-cluster   3                  True