Skip to content

CAMEL-24961: Catalog body types for the text data formats - #28298

Merged
davsclaus merged 52 commits into
mainfrom
feature/CAMEL-24961-catalog-body-types-dataformats-text
Oct 24, 2026
Merged

davsclaus merged 52 commits into
mainfrom
feature/CAMEL-24961-catalog-body-types-dataformats-text

Conversation

@davsclaus

Copy link
Copy Markdown
Contributor

Stacked on #28291. This group of the catalog body types (CAMEL-24961) takes the data formats for text and structured messages: 32 of the 56 data formats. The binary ones (avro, protobuf, the archives, crypto...) come next.

What the catalog says now

Data format Unmarshal gives Marshal takes
bindyCsv, bindyFixed, bindyKvp an object of the classType class for one record, a List of them for more (always a List with unwrapSingleInstance=false) an object of the classType class, or a List of them
univocityCsv, univocityFixed, univocityTsv a List of the rows, each a List of values, or a Map with asMap; an iterator with lazyLoad a Map, or a List of Maps
beanio a List of the beans of the mapping (the one bean with unmarshalSingleObject) a bean, or a List of them
flatpack a DataSetList: a List of Maps a List of Maps
gson, jsonb, fastjson a Map for a JSON object, a List for an array any, as jackson
groovyJson a Map (LazyMap) or a List a Map, a Node or a JsonNode; text as it is
groovyXml a groovy.util.Node a Node, a GPathResult or a Map; text as it is
jsonApi the resource object, or no body when the data is null one resource object
toon a Map or a List a Map, a List, a POJO, a value, or JSON text
ocsf an OcsfEvent, or a List with useList a Map, a List or a POJO
base64 the decoded bytes, as a stream text, bytes, a stream or a file
jaxb the JAXB object a JAXB object; text, bytes, a stream or a file as it is
soap the JAXB object of the SOAP body, or no body for an empty body or a fault a JAXB object
ubl the ph-ubl document, such as an InvoiceType the ph-ubl document
fhirJson, fhirXml the HAPI FHIR resource (IBaseResource) a HAPI FHIR resource
hl7 a HAPI Message, or a DOM Document with targetFormat=XML a Message, or HL7 as text or bytes
swiftMt, swiftMx the Prowide MT or MX class, or no body for an unknown type the Prowide object
ical an ical4j Calendar a Calendar
rss a ROME SyndFeed a SyndFeed, or the XML text of a feed
syslog a SyslogMessage a SyslogMessage, or the text of a message
grok a Map for one match, a List for several, no body for none (marshal is not supported)
iso8583 a j8583 IsoMessage an IsoMessage
asn1 the records in one byte array, or an iterator with usingIterator BER bytes, a stream or a file
dfdl the XML infoset as a DOM Document the XML infoset, as text or a DOM

After marshal, the body is the stream cache (a byte array when stream caching is off), as for the data formats that have body types already.

The validator reads the body after unmarshal. gson, jsonb, fastjson, groovyJson, toon, univocity, beanio and flatpack give parsed data, so the steps after them are checked as after jackson: for example, a file producer after unmarshal: {json: {library: Gson}} is reported. TextBodyFlowTest.testTheOptionsTheBodyDependsOn used gson as a data format without body types; it now uses gson with an unmarshalType, which makes the body unknown.

Checked at runtime. About 190 cases run with camel run logged the class of the body after each unmarshal and marshal. Among them:

  • bindy: one record gives the object and two give a List.
  • univocity: the default gives Lists of Strings, with the header line as a row.
  • gson: numbers come back as Double.
  • toon: a String body is read as JSON.
  • jaxb: XML text is written as it is.
  • hl7: text and bytes are parsed; a stream or a file fails.
  • SWIFT: the Prowide objects only; text fails with a ClassCastException.
  • grok: gives a Map, a List or no body.

Bugs found:

  • CAMEL-25995 (Major): base64 unmarshal of a file fails with "Stream closed", since 4.9.0.
  • CAMEL-25996: univocity asMap fails for one data row without a final newline.
  • CAMEL-25997: bindy marshal writes empty lines for a Map of values or a String.
  • CAMEL-25998: groovyJson and groovyXml cannot marshal a List.
  • CAMEL-25999: toon writes bytes as numbers, a stream as nothing and a file as its path.
  • CAMEL-26000 (improvement): jsonApi has no collection documents.
  • CAMEL-26001: ical marshal of text gives a NullPointerException.
  • CAMEL-26002: iso8583 errors for a missing ISO-Type and a configFile outside the classpath.
  • CAMEL-26003: flatpack marshal of a single Map, and its name "flatback".

Docs fixed:

  • bindy: the unmarshal section said the body is a List of Maps of class names, which has not been the case since 2011: it is the classType objects. Its examples would throw a ClassCastException.
  • jaxb: mustBeJAXBElement is false by default, so XML text is written as it is.
  • gson, jsonb, fastjson, jackson2, jackson3: "all the JSON data formats do the same" now names the four that do, and a file is one from a consumer.
  • base64: unmarshal gives a stream, not a byte[].
  • rss: marshal gives XML, not a String.
  • grok: the body is a Map for one match, and the examples' %{COMBINEDAPACHELOG lacked its }.
  • asn1: unmarshal gives a byte array, not an OutputStream.
  • dfdl: the marshal section said unmarshal.
  • swiftMt, swiftMx: marshal needs the Prowide object.

Measured.

Tests

  • TextBodyFlowTest.testTheBodyOfTextDataFormats: gson and univocityCsv give parsed data that the file producer cannot write, while univocityCsv with lazyLoad and jaxb say nothing.
  • These pass:
    • the 287 tests of camel-yaml-dsl-validator;
    • the 265 validator and catalog doc tests of camel-jbang-core;
    • the builds of camel-yaml-dsl-maven-plugin and camel-yaml-dsl-deserializers.
  • The answers of camel_catalog_doc stay under 18 000 characters: jaxb, the largest of this group, is 7 629.

Claude Code on behalf of davsclaus

🤖 Generated with Claude Code

davsclaus and others added 30 commits October 21, 2026 13:58
…s, and the body it leaves after a step

A @BodyTypes annotation on the endpoint, data format or language class says the
body per side (consumerOut, consumerReply, producerIn, producerOut, marshal,
unmarshal and expression in and out): its carriers (null, text, bytes, stream,
file, value, map, list, pojo, unchanged, any, unknown), the Java type when it is
always the same, the format, the options it depends on, the body for other
option values, and a description. The package plugin writes it as a bodyTypes
section of the catalog JSON and checks that each side fits the kind and that
the options it names exist. A subclass declares its own.

The first slice: file, direct, seda, timer, cron, kamelet, http, https,
rest-openapi, xslt, sql and the json data format (jackson), read from the code
that sets the body and checked against the body types the message history
records when routes run. camel_catalog_doc returns them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…tp, jms, kafka, aws2-sqs, the csv, jacksonXml and yaml data formats, and the languages

The second slice of the catalog body types:
- components: platform-http, rest (and the rest-openapi consumer), ftp, ftps,
  sftp, jms, activemq, activemq6, amqp, kafka, aws2-sqs, log, mock, scheduler,
  quartz;
- data formats: csv, jacksonXml, snakeYaml;
- languages: simple, constant, jsonpath, xpath, groovy, jq.

Each value is read from the code that sets the body, and checked against the
body types the message history recorded when the tests of the modules and
reference routes ran (the ftp integration tests included).

camel_catalog_doc gives the body types a line per side, and its answer is cut
at 18000 characters instead of 16000, so the four sides of the jms components
leave the options a limit asks for room.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
paho-mqtt5: the consumer gives the payload as a byte array, the producer sends
what converts to one and leaves the body unchanged.

mongodb: the consumer gives a Document (or null for a change without its full
document with changeStreams); the producer takes a document or a query, and
leaves a body by operation: a Document or null for a find of one, a List for
findAll, findDistinct and aggregate, a Long for count, the result of the driver
for the other writes.

Read from the code that sets the body, and checked against a mosquitto and a
mongodb container started with camel infra.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…ts an option value without carriers

From the review of PR 28241:

- An iterator is read once and has no size and no index, so tooling that
  reads list as a List accepts ${body.size()} or ${body[0]} on it, which fail
  at runtime. The new iterator carrier names it: sql with outputType=StreamList
  gives an iterator, not a list.
- A @BodyType.When with no carriers (carrier = {}) wrote "carriers": [] into
  the catalog. The build now rejects it, as it does a side without carriers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…MEL-24961-catalog-body-types-2

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
… camel_catalog_doc says unknown for no carriers

From the review of PR 28242:

- The csv data format with lazyLoad=true gives an iterator, with the iterator
  carrier of PR 28241, not a list.
- camel_catalog_doc says unknown for a body without carriers, instead of
  failing. The build rejects it now, but a catalog read from elsewhere may
  have one.
- Sqs2Endpoint imports org.apache.camel.spi.* already: only the import of the
  nested BodyType.Carrier stays.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…types of the catalog instead of their own tables

The YAML validator's body walks read what a component or a data format does
with the body from the catalog, instead of from tables in their source:

- the consumer a route starts with (consumerOut): the file as read, text,
  bytes or data, instead of the file consumer only;
- the body after a to: (producerOut): unchanged for log and mock and now for a
  file write, the response of the http producer as bytes or a stream;
- the producers that need text (producerIn), instead of file, ftp, ftps and
  sftp only;
- the body after an unmarshal (unmarshalOut), instead of the json, jacksonXml,
  yaml and csv table and the options that make something else;
- the consumers that reply with text (consumerReply), instead of platform-http;
- the consumers that give no body, instead of timer, quartz, scheduler and cron.

A side holds for a step only when the step decides every option it depends
on: an option the component can also be configured with (the valueDeserializer
of kafka) counts only when the endpoint sets it, and a placeholder value is not
known. The catalog of an older Camel version has no body types, so the
validator's own catalog stands in for it. After a step that fails at runtime
the walk stops, so a file write that keeps the body is reported once.

The rules about HTTP methods, REST bindings and EIPs stay in the validator; the
EIPs get body types with CAMEL-25928.

The same reports as before on 1204 human-written route files (the Camel
repository, the examples, kamelets, camel-k and camel-kit) and on 1209
model-written ones (the r17 benchmark series), and the doc examples of the
validator tests still validate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…docling, milvus, qdrant, ai-tool and ibm-watson-language

The AI components declare the body they need and the body they leave, per
operation where the operation decides it. The answer of openai and
langchain4j-chat is text, also when it is asked for JSON, so the YAML
validator now reports a field read on it that has no unmarshal before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
… endpoint count for its body types

The operation of openai is in the path (openai:embeddings), so the validator
reads the path options with the syntax of the component, as the catalog
parses an endpoint. A path the catalog cannot read, such as a toD with an
expression, says nothing about its options.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…pe for a value its option cannot have

A byOption value must be one of the enum of its option, or true or false for
a boolean, ignoring case as tooling compares it with the endpoint: a
misspelled operation such as embedding for embeddings would never hold.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
The attribute holds an array, and the catalog JSON and BodyTypeModel call
the same field carriers. The Javadoc of BodyTypeModel.isUnknown() says that
a side with no carriers is unknown too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…AMEL-24961-catalog-body-types-ai

# Conflicts:
#	tooling/maven/camel-package-maven-plugin/src/main/java/org/apache/camel/maven/packaging/BodyTypesHelper.java
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
The body types of aws2-s3, azure-storage-blob, azure-files, dropbox,
google-storage, ibm-cos and minio, read from the code and checked at
runtime against LocalStack S3 (aws2-s3, and ibm-cos and minio through
their S3 API) and Azurite (azure-storage-blob).

The pages of aws2-s3, google-storage and minio said getObject returns an
S3Object, a Blob or a MinioObject: it returns the content of the object.

The validator calls a body the file as read (a GenericFile) only when the
file is its only carrier, not for a blob that is a stream or a file.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…e data format and language schemas

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…n read as a list, as the carriers do

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
The body types of aws2-sns, aws2-kinesis, aws2-kinesis-firehose,
google-pubsub, azure-servicebus, azure-eventhubs and azure-storage-queue,
read from the code and checked at runtime against LocalStack (SNS and
Kinesis), the Pub/Sub emulator and Azurite (queues).

aws2-sqs sends bytes, a stream or a file as text as well, and
sendBatchMessage takes a SendMessageBatchRequest too, which its producer
side now says.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
The body types of bean, class, browse, controlbus, dataformat, dataset,
dataset-test, dynamic-router, dynamic-router-control, event, exec,
knative, language, once, ref, rest-api, saga, stream, stub, validator,
wasm and xslt-saxon, and of the bean, header, exchangeProperty, variable,
ref, file, tokenize and xtokenize languages, read from the code and
checked at runtime with camel run (knative over a local environment
file).

The xpath and jq languages read the source option instead of the body
when it is set, which their expressionIn now says, as tokenize and
xtokenize do.

The component docs of stream, exec, language, once, browse and dataset
said things about the body that the code does not do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
… and infinispan

The body types of jira, github2, pqc, dns, caffeine-cache,
caffeine-loadcache, infinispan and infinispan-embedded, the most used
components in the route files that had none yet. pqc, dns, caffeine and
infinispan-embedded were checked at runtime with camel run; jira and
github2 were read from the code.

The validator now takes an option a body depends on as unknown also when
another option of the step names a body, such as xslt with output=bytes
and a resultHandlerFactory, as the BodyType javadoc says.

The component docs of jira, github2, dns and infinispan said things about
the body that the code does not do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
davsclaus and others added 3 commits October 22, 2026 12:32
…sform, and validator components

The body types of velocity, freemarker, mustache, string-template, chunk,
jte, thymeleaf, mvel, jolt, jslt, jsonata, xj, xquery, json-validator,
schematron and bean-validator, and of the mvel and xquery languages, read
from the code and checked at runtime with camel run.

The docs of jolt and jsonata showed JSON text sent to the default
options, which take parsed JSON; the examples now set inputType and
outputType to JsonString. The template docs now say which context entries
need allowContextMapAll and that the template headers need
allowTemplateFromHeader, and a few other pages said things about the body
or the template that the code does not do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
Declares the body before and after marshal and unmarshal for 32 data formats:
bindyCsv, bindyFixed, bindyKvp, univocityCsv, univocityFixed, univocityTsv,
beanio, flatpack, gson, jsonb, fastjson, groovyJson, groovyXml, jsonApi, toon,
ocsf, base64, jaxb, soap, ubl, fhirJson, fhirXml, hl7, swiftMt, swiftMx, ical,
rss, syslog, grok, iso8583, asn1 and dfdl.

The validator now sees the parsed data that gson, jsonb, fastjson, groovyJson,
toon, univocity, beanio and flatpack give, as it does for jackson; the test that
used gson as a data format without body types uses gson with an unmarshalType.

Fixes the doc prose about the body: bindy unmarshal gives the classType objects
(not a List of Maps), jaxb mustBeJAXBElement is false by default, the JSON data
formats that write JSON text as it is, base64 unmarshal gives a stream, rss,
grok (and its COMBINEDAPACHELOG examples), asn1, dfdl and the SWIFT marshal.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
@davsclaus davsclaus added this to the 4.24.0 milestone Oct 22, 2026

@gnodet-bot gnodet-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No issues found.

Thorough, well-crafted metadata PR. Triple-consistency check passes across all 32 data formats (Java @BodyTypes annotation → component-local JSON → catalog JSON) — every format is in sync. The testTheBodyOfTextDataFormats test is correctly structured: warns for gson/univocityCsv → file (incompatible body), silent for univocityCsv+lazyLoad (iterator, unknown) and jaxb (class-specific, unknown). Documentation fixes are accurate (especially the 13-year-old bindy docs correction and the grok pattern examples). The base branch (#28291) dependency is correctly managed.

This review was generated by an AI agent, Hermès on behalf of @gnodet.

@github-actions

Copy link
Copy Markdown
Contributor

🌟 Thank you for your contribution to the Apache Camel project! 🌟
🤖 CI automation will test this PR automatically.

🐫 Apache Camel Committers, please review the following items:

  • First-time contributors require MANUAL approval for the GitHub Actions to run
  • You can use the command /component-test (camel-)component-name1 (camel-)component-name2.. to request a test from the test bot although they are normally detected and executed by CI.
  • You can label PRs using skip-tests and test-dependents to fine-tune the checks executed by this PR.
  • Build and test logs are available in the summary page. Only Apache Camel committers have access to the summary.

⚠️ Be careful when sharing logs. Review their contents before sharing them publicly.

…atalog-body-types-3

Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…atalog-body-types-ai

# Conflicts:
#	docs/user-manual/modules/ROOT/pages/camel-jbang-validation.adoc
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/main/java/org/apache/camel/dsl/yaml/validator/CatalogBodyTypes.java
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-storage

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-messaging

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-core

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-transformation

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
@davsclaus
davsclaus changed the base branch from feature/CAMEL-24961-catalog-body-types-transformation to main October 24, 2026 07:23
…atalog-body-types-dataformats-text

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
@davsclaus davsclaus added the enhancement New feature or request label Oct 24, 2026
@davsclaus
davsclaus merged commit 071382f into main Oct 24, 2026
3 of 7 checks passed
@davsclaus
davsclaus deleted the feature/CAMEL-24961-catalog-body-types-dataformats-text branch October 24, 2026 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants