Introduction
The Kafka integration for the Privitar Data Privacy Platform can be used to apply a Data Flow Job consuming the records from a Kafka topic and streaming the results to another Kafka topic.
The Confluent “Gold Verified” integration is built on top of the Kafka Connect ecosystem. It comes as a Connector you will be able to install on an existing cluster or a dedicated cluster. You will be able to reuse the components you have already built for Kafka Connect such as the Converters and Transformations.
By default, Kafka makes a connection to the Privacy Platform using Basic authentication, but the platform also supports Mutual TLS authentication.
The integration is tested to support the JsonConverter and AvroConverter provided by the Confluent Platform and can be integrated with the schema registry.
Compatibility
The platform connector is built to run on Kafka Connect 2.0 or later and is tested for the following combinations with the platform v3.8 (or later):
Kafka Connect version | Kafka version / Confluent version |
---|---|
Kafka Connect 2.0 | Kafka 2.0 / Confluent 5.0 |
Kafka Connect 2.0.x | Kafka 2.3.x / Confluent 5.3 |
Deployment
The platform connector can be deployed to an existing cluster. However, because of the nature of the work performed by the connector, it can be more CPU intensive than other connectors. It is advised to deploy it to a cluster which will be dedicated to run Privitar Data Flow jobs.