Supported Data Types
This section describes the Converters used by the Privitar Connector plug-in together with the supported data types used by the Connector.
Converters
Converters are used to serialise and deserialise the data from a transportable data format to a Kafka Connect data structure.
This plugin is tested to work with the following converters:
Converter | Path |
---|---|
Avro (Confluent platform) |
|
Json |
|
String |
|
Long |
|
However, it is possible to plug any converter as long it is implementing the Kafka Connect "Converter" interface.
As described in Configuration using the REST API, you will have to specify some converters to:
Deserialise the key of consumed records (
key.converter
).Deserialise the value of consumed records (
value.converter
).Serialise the key of the anonymised records we want to send out (
dest.key.converter
).Serialise the value of the anonymised records we want to send out (
dest.value.converter
).
Data Types
The Privitar Data Flow supports the following data types:
Privitar Platform type | Java type accepted in input | Java type written in output |
---|---|---|
Boolean | Boolean | Boolean |
Float | Float | Float |
Double | Double | Double |
Byte | Byte | Byte |
Short | Short | Short |
Integer | Integer | Integer |
Long | Long | Long |
Text | String | String |
Date | java.util.Date | java.util.Date |
Timestamp | java.util.Date | java.util.Date |
Other | Any Java type | The same Java type that was input. This type is used when data is loaded from a source format that contains types that the platform does not support. This enables data to be passed through unchanged or dropped by specifying Retain or Drop rules on the data in the platform policy. Downstream systems consuming the platformoutput can still use the data if it is retained by the platform policy. |
Depending of which converter you use, you might have to add few transformations to fit your Kafka record data model with the platform schema.
For example, if you have a date/timestamp formatted as a string, you will have to add a transformation to transform it to a Java Date with properties such as (for timestamps, the type field will be Timestamp):
transforms=DateOfBirth transforms.DateOfBirth.type=org.apache.kafka.connect.transforms.TimestampConverter$Value transforms.DateOfBirth.field=dateOfBirth transforms.DateOfBirth.target.type=Date transforms.DateOfBirth.format=yyyy-MM-dd
And transform this Java Date back to a formatted date string once the Data Flow Job has been applied:
dest.transforms=DateOfBirth dest.transforms.DateOfBirth.type=org.apache.kafka.connect.transforms.TimestampConverter$Value dest.transforms.DateOfBirth.field=dateOfBirth dest.transforms.DateOfBirth.target.type=string dest.transforms.DateOfBirth.format=yyyy-MM-dd
Note
org.apache.kafka.connect.transforms.TimestampConverter does not support NULL values.
For more information about how to use the Confluent Timestamp converter, see the TimeStampConverter section in the Confluent Technical Documentation.