Skip to content

CAMEL-24961: Catalog body types for as2, jdbc, smooks, vertx-http, tika, snmp, pdf, file-watch, scp, graphql and micrometer - #28327

Merged
davsclaus merged 76 commits into
mainfrom
feature/CAMEL-24961-catalog-body-types-most-used-3
Oct 24, 2026
Merged

davsclaus merged 76 commits into
mainfrom
feature/CAMEL-24961-catalog-body-types-most-used-3

Conversation

@davsclaus

@davsclaus davsclaus commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #28314. This group covers the components most used in the example routes that still had no catalog body types (CAMEL-24961): as2, jdbc (and spring-jdbc, whose endpoint extends it), smooks, vertx-http, tika, snmp, pdf, file-watch, scp, graphql and micrometer.

What the catalog says now

Component Consumer Producer input Producer output
as2 server/listen: the EDI message as text, or a stream with the base64 or binary transfer encoding; receipt/receive: the receipt (MDN) entity any, not known when inBody is set: with inBody=ediMessage the EDI message, text or a stream the receipt entity when dispositionNotificationTo asks for one, else no body
jdbc, spring-jdbc the SQL statement as text, bytes, a stream or a file a select: a List of Maps; SelectOne: the Map, the value of its one column, or no body; StreamList: an iterator; unchanged for a statement that is not a select
smooks text, bytes, a stream or a file unknown: what the exports of the Smooks config give
vertx-http no body, text, bytes, a stream or a file bytes, or no body
tika text, bytes, a stream or a file parse: bytes, the content in a ByteArrayOutputStream; detect: the media type as text
snmp the PDU as XML text any; an SNMP4J PDU for TRAP POLL: an SnmpMessage; GET_NEXT: a List of them; TRAP: unchanged
pdf create and append: text, bytes, a stream or a file; extractText: the PDF as bytes, a stream, a file or a PDDocument; merge: any create, append and merge: bytes, the PDF in a ByteArrayOutputStream; extractText: text
file-watch the java.io.File of the event
scp text, bytes, a stream or a file; also no body with allowNullBody unchanged
graphql the query as text, bytes, a stream or a file, when none of query, queryFile and queryHeader is set the response as text, or no body
micrometer any unchanged

Checked at runtime. Routes run with camel run logged the class of the body after each step, with text, bytes, streams, files, Maps, Lists, POJOs and no body. Local servers where needed: an H2 data source, a fake HTTP server, an SSH server with scp (Apache MINA SSHD), an SNMP agent (SNMP4J), and the as2 server and receipt consumer of the component itself.

  • as2:
    • client/send takes text or a stream; bytes, a file and a Map fail with "Failed to create EDI message entity";
    • the body after the send is the receipt with a synchronous MDN, and no body without one or with an asynchronous one;
    • server/listen gives the EDI message as text, or a stream with the base64 or binary transfer encoding;
    • receipt/receive gives a DispositionNotificationMultipartReportEntity.
  • jdbc:
    • text, bytes, a stream and a file all run as SQL; a Map and a List are sent as their toString(), which fails;
    • a select gives a List of LinkedHashMap rows;
    • SelectOne gives the Map, the value of a one-column row (a String, a Long, a byte[] or a Blob), or null, and fails for two rows;
    • StreamList gives a StreamListIterator; an update, an insert and DDL keep the body.
  • smooks: text, bytes, a stream and a file are read; a POJO (not wrapped in a JavaSource), a Map, a List and no body fail. The output is the extracted bean of a JavaSink, the bean Map without extract, the text of a StringSink, and a stream of the input without exports.
  • vertx-http:
    • text, bytes, a stream, a file and no body are sent; a Map, a List, a POJO, a number and a Path fail;
    • the response is a byte[] whatever its size, and null for a 204.
  • tika:
    • text, bytes, a stream and a file are read; a Map, a List, a POJO and no body fail;
    • parse gives a ByteArrayOutputStream of XHTML, HTML or text; detect gives the media type as a String.
  • snmp:
    • the poll and trap consumers give the XML text of the PDU;
    • the trap producer sends a PDU body and keeps it, and a text body fails;
    • the POLL producer gives an SnmpMessage, GET_NEXT a List of them.
  • pdf:
    • create and append write the body as text;
    • extractText reads bytes, a stream or a file, but not the ByteArrayOutputStream that create leaves (convert it to bytes first) and not text;
    • merge takes its files from a header.
  • file-watch: the body is a java.io.File for CREATE, MODIFY and DELETE events, and for directories too.
  • scp: text, bytes, a stream and a file are stored; a Map, a List and a POJO fail; no body fails unless allowNullBody=true.
  • graphql:
    • with the query option the body is not sent, except a camel-util-json JsonObject, which is the variables;
    • without it the body is the query;
    • the response is a String, with GraphQL errors inside a 200 not failing the exchange.
  • micrometer: counters, summaries and timers keep the body, whatever it is.

Choices

  • The ByteArrayOutputStream that tika parse and pdf create, append and merge leave is bytes, with that class as its javaType (second commit, after a review question): it holds the bytes of the PDF or of the parsed content and converts to a byte array, so a field read after it is reported as after any bytes.
  • as2 producerIn is any, and not known when inBody is set, as for fhir (third commit, after a review question): inBody can name any of the 23 arguments of client/send, and the EDI message is text or a stream only because of CAMEL-26052, which is a bug rather than a rule to check.
  • graphql producerIn depends on query, queryFile and queryHeader: the body is checked only when it is the query.
  • vertx-http responsePayloadAsByteArray is also a component option, so it is only in the description: naming it would make the output side unknown for every step that leaves it out.
  • spring-jdbc has the same body types as jdbc: its endpoint extends JdbcEndpoint, and body types are not inherited.

Bugs found and filed, one per component:

  • CAMEL-26052 (Major), as2: client/send sends only text or a stream, so the file of a file consumer fails; an HTTP error of the partner does not fail the exchange; the default of requestUri does not apply; ediMessageType is not read; quoted-printable sends base64; the server logs an ERROR for every unencrypted message.
  • CAMEL-26053 (Major), graphql: the Content-Type and Accept headers of the message replace application/json on the request, the query header is sent as an HTTP header, and CamelHttpMethod=post stays on the message after the call (since CAMEL-22562); a Map body is sent as its toString() as the query.
  • CAMEL-26054 (Major), pdf: the pageSize values of the catalog are ignored (A4 gives a Letter page); pdf:create followed by pdf:extractText fails; a CR or a tab fails with the default lineTermination.
  • CAMEL-26055, snmp: the producer without type sends no OIDs; the trap producer fails with a NullPointerException for a body that is not a PDU; GET_NEXT fails on a consumer; the SnmpMessage body of the producer does not convert to text.
  • CAMEL-26056, jdbc: an outputClass that does not resolve fails with a NullPointerException; generated keys are mapped to outputClass; a Blob cannot be read after the producer; a null body fails with a NullPointerException.
  • CAMEL-26057, tika: a body that does not convert to a stream fails with a NullPointerException; the output encoding is lost when the output converts to text.
  • CAMEL-26058, vertx-http: a Path body fails; a urlencoded form keeps its escapes.
  • CAMEL-26059, scp: a NullPointerException without a known hosts file; the remote directory must exist.
  • CAMEL-26060, file-watch: CamelFileLastModified is the time of the event, not of the file.
  • CAMEL-26061, micrometer: the docs say a warning is logged where the exchange fails.
  • CAMEL-26062, smooks: SAXSource, remote file and File bodies fail with a reader in the config (read in the code).

Docs fixed (the prose about the body only):

  • jdbc: the rows are LinkedHashMaps of a select; what SelectOne and StreamList give; another statement keeps the body.
  • graphql: the query comes from the body when no query option is set, and only a camel-util-json JsonObject is the variables.
  • tika: what the body is before and after detect and parse, and that other types need their Tika parser module.
  • pdf: convert the output of create to bytes before extractText.
  • smooks: a POJO is read from a JavaSource body.
  • snmp: what the POLL, GET_NEXT and TRAP producers give, and the XML example without the declaration the code does not write.

Measured.

Tests

  • TextBodyFlowTest.testTheBodyOfQueriesDocumentsAndFiles:
    • reported: a Map sent to jdbc, scp, vertx-http, tika parse, pdf create, and graphql without a query option; a field read after graphql, tika detect, pdf extractText, the as2 server and the snmp trap consumer;
    • not reported: a Map sent to graphql with a query, and a field read after jdbc, micrometer (the body is kept) and smooks.
  • CatalogDocsTest uses ldif instead of jdbc as the component without body types.
  • These pass:
    • the 293 tests of camel-yaml-dsl-validator;
    • the 265 validator and catalog doc tests of camel-jbang-core;
    • the builds of camel-yaml-dsl-maven-plugin and camel-yaml-dsl-deserializers.
  • The answers of camel_catalog_doc stay under 18 000 characters: vertx-http, the largest, is 13 148.

Claude Code on behalf of davsclaus

🤖 Generated with Claude Code

davsclaus and others added 30 commits October 21, 2026 13:58
…s, and the body it leaves after a step

A @BodyTypes annotation on the endpoint, data format or language class says the
body per side (consumerOut, consumerReply, producerIn, producerOut, marshal,
unmarshal and expression in and out): its carriers (null, text, bytes, stream,
file, value, map, list, pojo, unchanged, any, unknown), the Java type when it is
always the same, the format, the options it depends on, the body for other
option values, and a description. The package plugin writes it as a bodyTypes
section of the catalog JSON and checks that each side fits the kind and that
the options it names exist. A subclass declares its own.

The first slice: file, direct, seda, timer, cron, kamelet, http, https,
rest-openapi, xslt, sql and the json data format (jackson), read from the code
that sets the body and checked against the body types the message history
records when routes run. camel_catalog_doc returns them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…tp, jms, kafka, aws2-sqs, the csv, jacksonXml and yaml data formats, and the languages

The second slice of the catalog body types:
- components: platform-http, rest (and the rest-openapi consumer), ftp, ftps,
  sftp, jms, activemq, activemq6, amqp, kafka, aws2-sqs, log, mock, scheduler,
  quartz;
- data formats: csv, jacksonXml, snakeYaml;
- languages: simple, constant, jsonpath, xpath, groovy, jq.

Each value is read from the code that sets the body, and checked against the
body types the message history recorded when the tests of the modules and
reference routes ran (the ftp integration tests included).

camel_catalog_doc gives the body types a line per side, and its answer is cut
at 18000 characters instead of 16000, so the four sides of the jms components
leave the options a limit asks for room.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
paho-mqtt5: the consumer gives the payload as a byte array, the producer sends
what converts to one and leaves the body unchanged.

mongodb: the consumer gives a Document (or null for a change without its full
document with changeStreams); the producer takes a document or a query, and
leaves a body by operation: a Document or null for a find of one, a List for
findAll, findDistinct and aggregate, a Long for count, the result of the driver
for the other writes.

Read from the code that sets the body, and checked against a mosquitto and a
mongodb container started with camel infra.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…ts an option value without carriers

From the review of PR 28241:

- An iterator is read once and has no size and no index, so tooling that
  reads list as a List accepts ${body.size()} or ${body[0]} on it, which fail
  at runtime. The new iterator carrier names it: sql with outputType=StreamList
  gives an iterator, not a list.
- A @BodyType.When with no carriers (carrier = {}) wrote "carriers": [] into
  the catalog. The build now rejects it, as it does a side without carriers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…MEL-24961-catalog-body-types-2

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
… camel_catalog_doc says unknown for no carriers

From the review of PR 28242:

- The csv data format with lazyLoad=true gives an iterator, with the iterator
  carrier of PR 28241, not a list.
- camel_catalog_doc says unknown for a body without carriers, instead of
  failing. The build rejects it now, but a catalog read from elsewhere may
  have one.
- Sqs2Endpoint imports org.apache.camel.spi.* already: only the import of the
  nested BodyType.Carrier stays.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…types of the catalog instead of their own tables

The YAML validator's body walks read what a component or a data format does
with the body from the catalog, instead of from tables in their source:

- the consumer a route starts with (consumerOut): the file as read, text,
  bytes or data, instead of the file consumer only;
- the body after a to: (producerOut): unchanged for log and mock and now for a
  file write, the response of the http producer as bytes or a stream;
- the producers that need text (producerIn), instead of file, ftp, ftps and
  sftp only;
- the body after an unmarshal (unmarshalOut), instead of the json, jacksonXml,
  yaml and csv table and the options that make something else;
- the consumers that reply with text (consumerReply), instead of platform-http;
- the consumers that give no body, instead of timer, quartz, scheduler and cron.

A side holds for a step only when the step decides every option it depends
on: an option the component can also be configured with (the valueDeserializer
of kafka) counts only when the endpoint sets it, and a placeholder value is not
known. The catalog of an older Camel version has no body types, so the
validator's own catalog stands in for it. After a step that fails at runtime
the walk stops, so a file write that keeps the body is reported once.

The rules about HTTP methods, REST bindings and EIPs stay in the validator; the
EIPs get body types with CAMEL-25928.

The same reports as before on 1204 human-written route files (the Camel
repository, the examples, kamelets, camel-k and camel-kit) and on 1209
model-written ones (the r17 benchmark series), and the doc examples of the
validator tests still validate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…docling, milvus, qdrant, ai-tool and ibm-watson-language

The AI components declare the body they need and the body they leave, per
operation where the operation decides it. The answer of openai and
langchain4j-chat is text, also when it is asked for JSON, so the YAML
validator now reports a field read on it that has no unmarshal before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
… endpoint count for its body types

The operation of openai is in the path (openai:embeddings), so the validator
reads the path options with the syntax of the component, as the catalog
parses an endpoint. A path the catalog cannot read, such as a toD with an
expression, says nothing about its options.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…pe for a value its option cannot have

A byOption value must be one of the enum of its option, or true or false for
a boolean, ignoring case as tooling compares it with the endpoint: a
misspelled operation such as embedding for embeddings would never hold.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
The attribute holds an array, and the catalog JSON and BodyTypeModel call
the same field carriers. The Javadoc of BodyTypeModel.isUnknown() says that
a side with no carriers is unknown too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…AMEL-24961-catalog-body-types-ai

# Conflicts:
#	tooling/maven/camel-package-maven-plugin/src/main/java/org/apache/camel/maven/packaging/BodyTypesHelper.java
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
The body types of aws2-s3, azure-storage-blob, azure-files, dropbox,
google-storage, ibm-cos and minio, read from the code and checked at
runtime against LocalStack S3 (aws2-s3, and ibm-cos and minio through
their S3 API) and Azurite (azure-storage-blob).

The pages of aws2-s3, google-storage and minio said getObject returns an
S3Object, a Blob or a MinioObject: it returns the content of the object.

The validator calls a body the file as read (a GenericFile) only when the
file is its only carrier, not for a blob that is a stream or a file.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…e data format and language schemas

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…n read as a list, as the carriers do

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
The body types of aws2-sns, aws2-kinesis, aws2-kinesis-firehose,
google-pubsub, azure-servicebus, azure-eventhubs and azure-storage-queue,
read from the code and checked at runtime against LocalStack (SNS and
Kinesis), the Pub/Sub emulator and Azurite (queues).

aws2-sqs sends bytes, a stream or a file as text as well, and
sendBatchMessage takes a SendMessageBatchRequest too, which its producer
side now says.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
The body types of bean, class, browse, controlbus, dataformat, dataset,
dataset-test, dynamic-router, dynamic-router-control, event, exec,
knative, language, once, ref, rest-api, saga, stream, stub, validator,
wasm and xslt-saxon, and of the bean, header, exchangeProperty, variable,
ref, file, tokenize and xtokenize languages, read from the code and
checked at runtime with camel run (knative over a local environment
file).

The xpath and jq languages read the source option instead of the body
when it is set, which their expressionIn now says, as tokenize and
xtokenize do.

The component docs of stream, exec, language, once, browse and dataset
said things about the body that the code does not do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
… and infinispan

The body types of jira, github2, pqc, dns, caffeine-cache,
caffeine-loadcache, infinispan and infinispan-embedded, the most used
components in the route files that had none yet. pqc, dns, caffeine and
infinispan-embedded were checked at runtime with camel run; jira and
github2 were read from the code.

The validator now takes an option a body depends on as unknown also when
another option of the step names a body, such as xslt with output=bytes
and a resultHandlerFactory, as the BodyType javadoc says.

The component docs of jira, github2, dns and infinispan said things about
the body that the code does not do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claus Ibsen <claus.ibsen@gmail.com>
…atalog-body-types-ai

# Conflicts:
#	docs/user-manual/modules/ROOT/pages/camel-jbang-validation.adoc
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/main/java/org/apache/camel/dsl/yaml/validator/CatalogBodyTypes.java
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-storage

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-messaging

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-core

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-transformation

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-dataformats-text

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…' into feature/CAMEL-24961-catalog-body-types-dataformats-binary
…atalog-body-types-dataformats-binary

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-most-used-2

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
…atalog-body-types-aws-2

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
@davsclaus
davsclaus changed the base branch from feature/CAMEL-24961-catalog-body-types-aws-2 to main October 24, 2026 08:07
…atalog-body-types-most-used-3

# Conflicts:
#	dsl/camel-yaml-dsl/camel-yaml-dsl-validator/src/test/java/org/apache/camel/dsl/yaml/validator/TextBodyFlowTest.java
@davsclaus davsclaus added the enhancement New feature or request label Oct 24, 2026
@davsclaus
davsclaus merged commit 6183f07 into main Oct 24, 2026
2 of 7 checks passed
@davsclaus
davsclaus deleted the feature/CAMEL-24961-catalog-body-types-most-used-3 branch October 24, 2026 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants