Skip to main content

Kafka Connect Reference Guide

Supported Data Types

This section describes the Converters used by the Privitar Connector plug-in together with the supported data types used by the Connector.

Converters

Converters are used to serialise and deserialise the data from a transportable data format to a Kafka Connect data structure.

This plugin is tested to work with the following converters:

Converter

Path

Avro (Confluent platform)

io.confluent.connect.avro.AvroConverter

Json

org.apache.kafka.connect.json.JsonConverter

String

org.apache.kafka.connect.storage.StringConverter

Long

org.apache.kafka.connect.storage.LongConverter

However, it is possible to plug any converter as long it is implementing the Kafka Connect "Converter" interface.

As described in Configuration using the REST API, you will have to specify some converters to:

  • Deserialise the key of consumed records (key.converter).

  • Deserialise the value of consumed records (value.converter).

  • Serialise the key of the anonymised records we want to send out (dest.key.converter).

  • Serialise the value of the anonymised records we want to send out (dest.value.converter).

Data Types

The Privitar Data Flow supports the following data types:

Privitar Platform type

Java type accepted in input

Java type written in output

Boolean

Boolean

Boolean

Float

Float

Float

Double

Double

Double

Byte

Byte

Byte

Short

Short

Short

Integer

Integer

Integer

Long

Long

Long

Text

String

String

Date

java.util.Date

java.util.Date

Timestamp

java.util.Date

java.util.Date

Other

Any Java type

The same Java type that was input. This type is used when data is loaded from a source format that contains types that the platform does not support. This enables data to be passed through unchanged or dropped by specifying Retain or Drop rules on the data in the platform policy. Downstream systems consuming the platformoutput can still use the data if it is retained by the platform policy.

Depending of which converter you use, you might have to add few transformations to fit your Kafka record data model with the platform schema.

For example, if you have a date/timestamp formatted as a string, you will have to add a transformation to transform it to a Java Date with properties such as (for timestamps, the type field will be Timestamp):

transforms=DateOfBirth
transforms.DateOfBirth.type=org.apache.kafka.connect.transforms.TimestampConverter$Value
transforms.DateOfBirth.field=dateOfBirth
transforms.DateOfBirth.target.type=Date
transforms.DateOfBirth.format=yyyy-MM-dd

And transform this Java Date back to a formatted date string once the Data Flow Job has been applied:

dest.transforms=DateOfBirth
dest.transforms.DateOfBirth.type=org.apache.kafka.connect.transforms.TimestampConverter$Value
dest.transforms.DateOfBirth.field=dateOfBirth
dest.transforms.DateOfBirth.target.type=string
dest.transforms.DateOfBirth.format=yyyy-MM-dd

Note

org.apache.kafka.connect.transforms.TimestampConverter does not support NULL values.

For more information about how to use the Confluent Timestamp converter, see the TimeStampConverter section in the Confluent Technical Documentation.