Cloudera Docs

Kafka Connect plugins

Learn about the third-party connectors that Cloudera certified for use and the various default connectors, transforms, and converters that are included in Cloudera Streams Messaging Operator for Kubernetes.

Certified third-party connectors

Certified connectors are validated to work with Cloudera Streams Messaging Operator for Kubernetes but do not come packaged with it or are installed to Kafka Connect by default.

Table 1. Certified connectors in Cloudera Streams Messaging Operator for Kubernetes
Connector Certified Version Download location Documentation
Apache Iceberg Sink Connector for Kafka Connect 1.10 Maven Central Kafka Connect | Apache Iceberg

Plugins included in Cloudera Streams Messaging Operator for Kubernetes

Connectors

Cloudera Streams Messaging Operator for Kubernetes ships and supports all Kafka Connect connectors included in Apache Kafka.

The full list is as follows.

  • org.apache.kafka.connect.mirror.MirrorCheckpointConnector
  • org.apache.kafka.connect.mirror.MirrorSourceConnector
  • org.apache.kafka.connect.mirror.MirrorHeartBeatConnector
  • org.apache.kafka.connect.file.FileStreamSourceConnector
  • org.apache.kafka.connect.file.FileStreamSinkConnector
Single Message Transforms plugins (transformations and predicates)

Single Message Transforms (SMT) plugins (transformations and predicates) are deployed on top of Kafka Connect connectors. They enable you to apply message transformations and filtering on a single message basis. Cloudera Streams Messaging Operator for Kubernetes ships and supports all transformation and predicates plugins included in Apache Kafka as well as the ConvertFromBytes and ConvertToBytes plugins, which are Cloudera specific plugins.

The full list is as follows.

Transformations
  • com.cloudera.dim.kafka.connect.transforms.ConvertFromBytes
  • com.cloudera.dim.kafka.connect.transforms.ConvertToBytes
  • org.apache.kafka.connect.transforms.Cast
  • org.apache.kafka.connect.transforms.DropHeaders
  • org.apache.kafka.connect.transforms.ExtractField
  • org.apache.kafka.connect.transforms.Filter
  • org.apache.kafka.connect.transforms.Flatten
  • org.apache.kafka.connect.transforms.HeaderFrom
  • org.apache.kafka.connect.transforms.HoistField
  • org.apache.kafka.connect.transforms.InsertField
  • org.apache.kafka.connect.transforms.InsertHeader
  • org.apache.kafka.connect.transforms.MaskField
  • org.apache.kafka.connect.transforms.RegexRouter
  • org.apache.kafka.connect.transforms.ReplaceField
  • org.apache.kafka.connect.transforms.SetSchemaMetadata
  • org.apache.kafka.connect.transforms.TimestampConverter
  • org.apache.kafka.connect.transforms.TimestampRouter
  • org.apache.kafka.connect.transforms.ValueToKey
Predicates
  • org.apache.kafka.connect.transforms.predicates.HasHeaderKey
  • org.apache.kafka.connect.transforms.predicates.RecordIsTombstone
  • org.apache.kafka.connect.transforms.predicates.TopicNameMatches
Converters

Converters can be used to transform Kafka record keys and values between bytes and a specific format. In addition to the JsonConverter, there are converters for most often used primitive types as well.

The full list is as follows.

  • org.apache.kafka.connect.json.JsonConverter
  • org.apache.kafka.connect.converters.ByteArrayConverter
  • org.apache.kafka.connect.converters.BooleanConverter
  • org.apache.kafka.connect.converters.DoubleConverter
  • org.apache.kafka.connect.converters.FloatConverter
  • org.apache.kafka.connect.converters.IntegerConverter
  • org.apache.kafka.connect.converters.LongConverter
  • org.apache.kafka.connect.converters.ShortConverter
  • org.apache.kafka.connect.storage.StringConverter
Header converters

Header converters can be used to transform Kafka record headers between bytes and a specific format. Cloudera Streams Messaging Operator for Kubernetes and Kafka includes a single dedicated header converter, which is the org.apache.kafka.connect.storage.SimpleHeaderConverter.

The SimpleHeaderConverter is the default header converter and is adequate for the majority of use cases. In case your headers are of a specific format, like JSON, you can use any other converter listed in the Converters section as a header converter as well.

Replication policies

A replication policy defines the basic rules of how topics are replicated from source to target clusters when using Kafka Connect-based replication to replicate Kafka data between Kafka clusters.

The full list is as follows.

  • org.apache.kafka.connect.mirror.DefaultReplicationPolicy
  • org.apache.kafka.connect.mirror.IdentityReplicationPolicy