Skip to content

CAMEL-24961: Catalog body types for the binary data formats - #28302

Merged
davsclaus merged 55 commits into
mainfrom
feature/CAMEL-24961-catalog-body-types-dataformats-binary
Oct 24, 2026
Merged

davsclaus merged 55 commits into
mainfrom
feature/CAMEL-24961-catalog-body-types-dataformats-binary

Conversation

@davsclaus

Copy link
Copy Markdown
Contributor

Stacked on #28298. This group of the catalog body types (CAMEL-24961) takes the last 20 data formats, the binary ones. With it, all 56 data formats of the catalog have body types.

What the catalog says now

Data format Unmarshal gives Marshal takes
avro an object of the instanceClassName class an Avro record: a generated class, or a GenericRecord of the schema of instanceClassName
avroJackson a Map, or a List for an array schema a Map, a List or a POJO, with the schema of the schemaResolver
protobuf the generated Message of instanceClass, from binary, or from JSON with contentTypeFormat=json a Message, or a Map of its fields
protobufJackson a Map of the fields a Map or a POJO
thrift the generated TBase of instanceClass; sjson is for marshal only a TBase
cbor an object of unmarshalType: unmarshal fails without it today a Map, a List or a POJO
fory unknown: the class in the data a Map, a List or a POJO
parquetAvro a List of the records, or an iterator with lazyLoad a List of GenericRecords or of POJOs
zipFile, tarFile the content of the one entry as a stream, or an iterator with usingIterator text, bytes, a stream or a file
gzipDeflater, zipDeflater, lzf the decompressed data as a stream text, bytes, a stream or a file
crypto, pgp the decrypted data as a stream text, bytes, a stream or a file
pqc the decrypted data as a byte array text, bytes, a stream or a file
xmlSecurity the decrypted XML as a byte array XML as text, bytes, a stream, a file or a DOM
barcode the text of the barcode the text to encode
mimeMultipart the first part as a stream, with the other parts as attachments the body, with the attachments as the other parts
smooks unknown: what the exports of the config give a Java object graph

After marshal, the body is the stream cache (a byte array when stream caching is off), as for every data format.

Also:

  • jackson and jacksonXml: the body after unmarshal now depends on allowJmsType as well, since the JMSType header picks the class when it is set.
  • The validator: it reads the body after unmarshal, and avro with its default library, Jackson, now gives parsed data.

Checked at runtime. About 120 cases run with camel run logged the class of the body after each step. They include file bodies for every data format that reads a stream, looking for the base64 problem of CAMEL-25995:

  • Read eagerly, so files work: the archive, compression and encryption formats.
  • Fails with files: mimeMultipart (below).

Bugs found:

  • CAMEL-26004 (Major): tarFile with usingIterator gives empty entries in a split that is not streaming.
  • CAMEL-26005 (Major): protobufJackson writes JSON text as it is, labelled application/protobuf (since 4.23.0).
  • CAMEL-26006: cbor unmarshal fails without unmarshalType, which is the first example of its docs.
  • CAMEL-26007: mimeMultipart gives a closed stream for a file that is not a multipart.
  • CAMEL-26008: a zip or tar with a folder entry fails as more than one entry.
  • CAMEL-26009: tarFile marshal fails after the body changed, as it trusts CamelFileLength.
  • CAMEL-26010: the catalog defaults of xmlSecurity differ from the code, and secureTag reads the body twice.
  • CAMEL-26011: barcodeFormat lists four formats that cannot be encoded.

Docs fixed:

  • gzipDeflater: a note said it is not for gzip files; it writes and reads exactly the .gz format.
  • zipDeflater: unmarshal reads any compression level.
  • tarFile:
    • the tar is not compressed;
    • the split must be streaming.
  • zipFile and tarFile: the outgoing CamelFileName is the file name without its path.
  • protobuf and thrift:
    • unmarshal needs instanceClass in every format;
    • protobuf gives a Map after a json unmarshal too.
  • parquetAvro: unmarshal gives a List or an Iterator, and marshal takes a List.
  • mimeMultipart: the content type is multipart/ with the multipartSubType, mixed by default.
  • pqc: the payload is held in memory as byte arrays; stream caching does not change that.
  • avroJackson and protobufJackson, Jackson 2 and 3: the XML and YAML examples set unmarshalType to JsonNode, as the Java example does.

Measured.

Tests

  • TextBodyFlowTest.testTheBodyOfBinaryDataFormats:
    • avro with its default library gives a Map, which the file producer cannot write;
    • zipFile, cbor with an unmarshalType, and fory say nothing.
  • These pass:
    • the 288 tests of camel-yaml-dsl-validator;
    • the 265 validator and catalog doc tests of camel-jbang-core;
    • the builds of camel-yaml-dsl-maven-plugin and camel-yaml-dsl-deserializers.
  • The answers of camel_catalog_doc stay under 18 000 characters: pgp, the largest of this group, is 8 649.

Claude Code on behalf of davsclaus

🤖 Generated with Claude Code

davsclaus and others added 30 commits October 21, 2026 13:58
…s, and the body it leaves after a step

A @BodyTypes annotation on the endpoint, data format or language class says the
body per side (consumerOut, consumerReply, producerIn, producerOut, marshal,
unmarshal and expression in and out): its carriers (null, text, bytes, stream,
file, value, map, list, pojo, unchanged, any, unknown), the Java type when it is
always the same, the format, the options it depends on, the body for other
option values, and a description. The package plugin writes it as a bodyTypes
section of the catalog JSON and checks that each side fits the kind and that
the options it names exist. A subclass declares its own.

The first slice: file, direct, seda, timer, cron, kamelet, http, https,
rest-openapi, xslt, sql and the json data format (jackson), read from the code
that sets the body and checked against the body types the message history
records when routes run. camel_catalog_doc returns them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…tp, jms, kafka, aws2-sqs, the csv, jacksonXml and yaml data formats, and the languages

The second slice of the catalog body types:
- components: platform-http, rest (and the rest-openapi consumer), ftp, ftps,
  sftp, jms, activemq, activemq6, amqp, kafka, aws2-sqs, log, mock, scheduler,
  quartz;
- data formats: csv, jacksonXml, snakeYaml;
- languages: simple, constant, jsonpath, xpath, groovy, jq.

Each value is read from the code that sets the body, and checked against the
body types the message history recorded when the tests of the modules and
reference routes ran (the ftp integration tests included).

camel_catalog_doc gives the body types a line per side, and its answer is cut
at 18000 characters instead of 16000, so the four sides of the jms components
leave the options a limit asks for room.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
paho-mqtt5: the consumer gives the payload as a byte array, the producer sends
what converts to one and leaves the body unchanged.

mongodb: the consumer gives a Document (or null for a change without its full
document with changeStreams); the producer takes a document or a query, and
leaves a body by operation: a Document or null for a find of one, a List for
findAll, findDistinct and aggregate, a Long for count, the result of the driver
for the other writes.

Read from the code that sets the body, and checked against a mosquitto and a
mongodb container started with camel infra.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…ts an option value without carriers

From the review of PR 28241:

- An iterator is read once and has no size and no index, so tooling that
  reads list as a List accepts ${body.size()} or ${body[0]} on it, which fail
  at runtime. The new iterator carrier names it: sql with outputType=StreamList
  gives an iterator, not a list.
- A @BodyType.When with no carriers (carrier = {}) wrote "carriers": [] into
  the catalog. The build now rejects it, as it does a side without carriers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…MEL-24961-catalog-body-types-2

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
… camel_catalog_doc says unknown for no carriers

From the review of PR 28242:

- The csv data format with lazyLoad=true gives an iterator, with the iterator
  carrier of PR 28241, not a list.
- camel_catalog_doc says unknown for a body without carriers, instead of
  failing. The build rejects it now, but a catalog read from elsewhere may
  have one.
- Sqs2Endpoint imports org.apache.camel.spi.* already: only the import of the
  nested BodyType.Carrier stays.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…types of the catalog instead of their own tables

The YAML validator's body walks read what a component or a data format does
with the body from the catalog, instead of from tables in their source:

- the consumer a route starts with (consumerOut): the file as read, text,
  bytes or data, instead of the file consumer only;
- the body after a to: (producerOut): unchanged for log and mock and now for a
  file write, the response of the http producer as bytes or a stream;
- the producers that need text (producerIn), instead of file, ftp, ftps and
  sftp only;
- the body after an unmarshal (unmarshalOut), instead of the json, jacksonXml,
  yaml and csv table and the options that make something else;
- the consumers that reply with text (consumerReply), instead of platform-http;
- the consumers that give no body, instead of timer, quartz, scheduler and cron.

A side holds for a step only when the step decides every option it depends
on: an option the component can also be configured with (the valueDeserializer
of kafka) counts only when the endpoint sets it, and a placeholder value is not
known. The catalog of an older Camel version has no body types, so the
validator's own catalog stands in for it. After a step that fails at runtime
the walk stops, so a file write that keeps the body is reported once.

The rules about HTTP methods, REST bindings and EIPs stay in the validator; the
EIPs get body types with CAMEL-25928.

The same reports as before on 1204 human-written route files (the Camel
repository, the examples, kamelets, camel-k and camel-kit) and on 1209
model-written ones (the r17 benchmark series), and the doc examples of the
validator tests still validate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…docling, milvus, qdrant, ai-tool and ibm-watson-language

The AI components declare the body they need and the body they leave, per
operation where the operation decides it. The answer of openai and
langchain4j-chat is text, also when it is asked for JSON, so the YAML
validator now reports a field read on it that has no unmarshal before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
… endpoint count for its body types

The operation of openai is in the path (openai:embeddings), so the validator
reads the path options with the syntax of the component, as the catalog
parses an endpoint. A path the catalog cannot read, such as a toD with an
expression, says nothing about its options.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…pe for a value its option cannot have

A byOption value must be one of the enum of its option, or true or false for
a boolean, ignoring case as tooling compares it with the endpoint: a
misspelled operation such as embedding for embeddings would never hold.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
The attribute holds an array, and the catalog JSON and BodyTypeModel call
the same field carriers. The Javadoc of BodyTypeModel.isUnknown() says that
a side with no carriers is unknown too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…AMEL-24961-catalog-body-types-ai

# Conflicts:
#	tooling/maven/camel-package-maven-plugin/src/main/java/org/apache/camel/maven/packaging/BodyTypesHelper.java
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
The body types of aws2-s3, azure-storage-blob, azure-files, dropbox,
google-storage, ibm-cos and minio, read from the code and checked at
runtime against LocalStack S3 (aws2-s3, and ibm-cos and minio through
their S3 API) and Azurite (azure-storage-blob).

The pages of aws2-s3, google-storage and minio said getObject returns an
S3Object, a Blob or a MinioObject: it returns the content of the object.

The validator calls a body the file as read (a GenericFile) only when the
file is its only carrier, not for a blob that is a stream or a file.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…e data format and language schemas

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…n read as a list, as the carriers do

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
The body types of aws2-sns, aws2-kinesis, aws2-kinesis-firehose,
google-pubsub, azure-servicebus, azure-eventhubs and azure-storage-queue,
read from the code and checked at runtime against LocalStack (SNS and
Kinesis), the Pub/Sub emulator and Azurite (queues).

aws2-sqs sends bytes, a stream or a file as text as well, and
sendBatchMessage takes a SendMessageBatchRequest too, which its producer
side now says.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
The body types of bean, class, browse, controlbus, dataformat, dataset,
dataset-test, dynamic-router, dynamic-router-control, event, exec,
knative, language, once, ref, rest-api, saga, stream, stub, validator,
wasm and xslt-saxon, and of the bean, header, exchangeProperty, variable,
ref, file, tokenize and xtokenize languages, read from the code and
checked at runtime with camel run (knative over a local environment
file).

The xpath and jq languages read the source option instead of the body
when it is set, which their expressionIn now says, as tokenize and
xtokenize do.

The component docs of stream, exec, language, once, browse and dataset
said things about the body that the code does not do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
… and infinispan

The body types of jira, github2, pqc, dns, caffeine-cache,
caffeine-loadcache, infinispan and infinispan-embedded, the most used
components in the route files that had none yet. pqc, dns, caffeine and
infinispan-embedded were checked at runtime with camel run; jira and
github2 were read from the code.

The validator now takes an option a body depends on as unknown also when
another option of the step names a body, such as xslt with output=bytes
and a resultHandlerFactory, as the BodyType javadoc says.

The component docs of jira, github2, dns and infinispan said things about
the body that the code does not do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
Declares the body before and after marshal and unmarshal for the last 20 data
formats, so all 56 data formats of the catalog have body types: avro,
avroJackson, protobuf, protobufJackson, thrift, cbor, fory, parquetAvro,
zipFile, tarFile, gzipDeflater, zipDeflater, lzf, crypto, pgp, pqc,
xmlSecurity, barcode, mimeMultipart and smooks. The body after a jackson or
jacksonXml unmarshal depends on allowJmsType too.

Fixes the doc prose about the body: gzipDeflater writes and reads .gz files,
zipDeflater reads any compression level, tarFile is not compressed and its
split must be streaming, the CamelFileName of zipFile and tarFile, protobuf
and thrift unmarshal need the instanceClass, parquetAvro takes and gives
Lists, the mimeMultipart content type, the memory pqc needs, and the XML and
YAML examples of the Jackson avro and protobuf data formats give a JsonNode,
as the Java ones do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
@davsclaus davsclaus added this to the 4.24.0 milestone Oct 22, 2026

@gnodet-bot gnodet-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Annotation-only PR that correctly documents body types for the last 20 binary data formats, completing CAMEL-24961. Traced the test assertion through the validator logic: singleElement() holds because zipFile (stream carrier), cbor with unmarshalType (pojo carrier, data()=false), and fory (unknown carrier) all return null from unmarshalled() — only avro (defaults to avroJackson, produces a Map) triggers a warning to file:out. JSON consistency verified across 6 catalog/component pairs (identical). Doc fixes are accurate against source code (gzipDeflater does produce .gz, pqc does hold data in-memory, parquetAvro marshal does require a List). No issues found.

This review was generated by an AI agent, Hermès on behalf of @gnodet.

@github-actions

Copy link
Copy Markdown
Contributor

🌟 Thank you for your contribution to the Apache Camel project! 🌟
🤖 CI automation will test this PR automatically.

🐫 Apache Camel Committers, please review the following items:

  • First-time contributors require MANUAL approval for the GitHub Actions to run
  • You can use the command /component-test (camel-)component-name1 (camel-)component-name2.. to request a test from the test bot although they are normally detected and executed by CI.
  • You can label PRs using skip-tests and test-dependents to fine-tune the checks executed by this PR.
  • Build and test logs are available in the summary page. Only Apache Camel committers have access to the summary.

⚠️ Be careful when sharing logs. Review their contents before sharing them publicly.

…atalog-body-types-3

Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…atalog-body-types-ai

# Conflicts:
#	docs/user-manual/modules/ROOT/pages/camel-jbang-validation.adoc
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/main/java/org/apache/camel/dsl/yaml/validator/CatalogBodyTypes.java
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-storage

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-messaging

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-core

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-transformation

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-dataformats-text

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
@davsclaus
davsclaus changed the base branch from feature/CAMEL-24961-catalog-body-types-dataformats-text to main October 24, 2026 07:30
…' into feature/CAMEL-24961-catalog-body-types-dataformats-binary
…atalog-body-types-dataformats-binary

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
@davsclaus davsclaus added the enhancement New feature or request label Oct 24, 2026
@davsclaus
davsclaus merged commit 601618a into main Oct 24, 2026
2 of 7 checks passed
@davsclaus
davsclaus deleted the feature/CAMEL-24961-catalog-body-types-dataformats-binary branch October 24, 2026 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants