Is your feature request related to a problem? Please describe.
Today the only supported compression algorithm is lz4. The feature request is to also support zstd with a dictionary.
Describe the solution you'd like
Allow choosing whether the compress data with lz4 or zstd with a dictionary. Also allow choosing the compression level.
Use a discovery job during the first run to generate a binary dictionary, which then will be uploaded to S3. Once
uploaded, replicators will leverage it for compression. Additionally, the customer can reuse the binary dictionary prepared by discovery for their application.
json-mapping:
"compressionConfig": {
"enabled": true,
"compression": zstd-dict-based/lz4,
"compressionLevel": 3,
"compressAllNonPrimaryColumns": false,
"compressNonPrimaryColumns": ["column-to-be-compressed"],
"targetNameColumn": "column-to-be-compressed"
},
The dictionary file will be ks_name_table_name_dictionary.json
Describe alternatives you've considered
The customer could provide the pretrained dictionary, but the ability to provide sample data instead should enable them to be more flexible and try out different compressions.
Additional context
Using a dictionary based compression is useful for columns with repeating values, for example low cardinality column, or a json column with similar field names in all/many rows.
Is your feature request related to a problem? Please describe.
Today the only supported compression algorithm is lz4. The feature request is to also support zstd with a dictionary.
Describe the solution you'd like
Allow choosing whether the compress data with lz4 or zstd with a dictionary. Also allow choosing the compression level.
Use a discovery job during the first run to generate a binary dictionary, which then will be uploaded to S3. Once
uploaded, replicators will leverage it for compression. Additionally, the customer can reuse the binary dictionary prepared by discovery for their application.
json-mapping:
"compressionConfig": {
"enabled": true,
"compression": zstd-dict-based/lz4,
"compressionLevel": 3,
"compressAllNonPrimaryColumns": false,
"compressNonPrimaryColumns": ["column-to-be-compressed"],
"targetNameColumn": "column-to-be-compressed"
},
The dictionary file will be ks_name_table_name_dictionary.json
Describe alternatives you've considered
The customer could provide the pretrained dictionary, but the ability to provide sample data instead should enable them to be more flexible and try out different compressions.
Additional context
Using a dictionary based compression is useful for columns with repeating values, for example low cardinality column, or a json column with similar field names in all/many rows.