Skip to main content

Kafka Connect Reference Guide

Introduction

The Kafka integration for the Privitar Data Privacy Platform can be used to apply a Data Flow Job consuming the records from a Kafka topic and streaming the results to another Kafka topic.

The Confluent “Gold Verified” integration is built on top of the Kafka Connect ecosystem. It comes as a Connector you will be able to install on an existing cluster or a dedicated cluster. You will be able to reuse the components you have already built for Kafka Connect such as the Converters and Transformations.

By default, Kafka makes a connection to the Privacy Platform using Basic authentication, but the platform also supports Mutual TLS authentication.

The integration is tested to support the JsonConverter and AvroConverter provided by the Confluent Platform and can be integrated with the schema registry.

Compatibility

The platform connector is built to run on Kafka Connect 2.0 or later and is tested for the following combinations with the platform v3.8 (or later):

Kafka Connect version

Kafka version / Confluent version

Kafka Connect 2.0

Kafka 2.0 / Confluent 5.0

Kafka Connect 2.0.x

Kafka 2.3.x / Confluent 5.3

Deployment

The platform connector can be deployed to an existing cluster. However, because of the nature of the work performed by the connector, it can be more CPU intensive than other connectors. It is advised to deploy it to a cluster which will be dedicated to run Privitar Data Flow jobs.