Skip to content

Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381 - #40648

Open
nitinware wants to merge 11 commits into
apache:masterfrom
nitinware:couchbaseio
Open

nitinware wants to merge 11 commits into
apache:masterfrom
nitinware:couchbaseio

Conversation

@nitinware

@nitinware nitinware commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381.

Design discussion: https://lists.apache.org/thread/cvzpfnw717zbnf7z6slfkfbp6zg6604g

Design doc: https://docs.google.com/document/d/1ALlXaiVSa_Dn9jJxM9MsmhEhBLPW_FP8RswN8HG33iU

Implementation

The commits are organized by component to make the changes easier to review:

  1. Write: Concurrent, batched UPSERT operations using withIdFn and withContentFn, with bounded in-flight concurrency and flushing in @FinishBundle.
  2. JsonObjectCoder: UTF-8 JSON encoding for Couchbase SDK JsonObject values.
  3. Read: Bounded SQL++ query execution as a single unit of work, with an optional parsing function (defaulting to JsonObject), named/positional parameters, scan consistency, and opt-in withRedistribute() applied after parsing.
  4. Schema-aware reads: readRows(Schema) converts query results into Beam Row values using RowJson, respecting the supplied schema's nullability constraints.
  5. Documentation: Class Javadocs, CHANGES.md, and the connector documentation table.
  6. Review improvements: Display data, a build configuration comment, and an upsert idempotency test.

Testing

  • 39 unit tests across CouchbaseIOTest and JsonObjectCoderTest.
  • 15 Testcontainers integration tests across CouchbaseIOReadTest and CouchbaseIOWriteTest, using couchbase/server:enterprise-8.0.3.
  • Tests cover authentication failures (the password must not appear in errors, toString, or display data), type mismatches, missing required fields, and repeated UPSERT operations.
  • Local validation passes: ./gradlew :sdks:java:io:couchbase:build (Checkstyle, Javadoc, Checker Framework nullness validation, and dependency analysis).
  • checkJavaLinkage could not run locally because of a corporate repository proxy restriction; CI validation is pending.
  • TLS (couchbases://) is delegated to the Couchbase SDK but is not covered by an integration test.

Known limitations / follow-ups

  • Connection lifecycle: One Couchbase Cluster per DoFn instance. JVM-wide sharing may be considered after benchmarking.
  • Source parallelism: SQL++ reads execute as a single unit of work. Partitioned reads are deferred.
  • SchemaTransform / Managed I/O: Deferred pending further community feedback (design doc §12, question 2).
  • Additional capabilities: Advanced write modes, durability configuration, dead-letter handling, and DCP-based CDC are future work.

R: @kennknowles

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • If this contribution is large, please file an Apache Individual Contributor License Agreement.
  • Replace these instructions with a description of your change and why you have made it.
  • Mention the appropriate issue in your description (for example: addresses #123 or fixes #456).
  • Update CHANGES.md with noteworthy changes.

See CONTRIBUTING.md for more details.

Adds the sdks/java/io/couchbase module with ConnectionConfiguration and
CouchbaseIO.write(), which upserts each element as a JSON document in
batches, with bounded concurrency through the reactive API. Includes unit
tests, Testcontainers integration tests against Couchbase Server 8.0.3,
and a PreCommit workflow for the module.
Encodes a JsonObject as its JSON string, which is 2 to 3 times smaller
than the SerializableCoder Beam would otherwise infer. It will be the
default output coder for CouchbaseIO.read().
Adds CouchbaseIO.read(), which runs a SQL++ query through the reactive
API and streams the rows. Supports named and positional parameters, scan
consistency, a parsing function applied inside the read, an output coder,
and an optional redistribute after parsing. The default output is
JsonObject, encoded with JsonObjectCoder.
@nitinware nitinware changed the title Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381. Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381 [DonNotMerge] Oct 18, 2026
@nitinware nitinware changed the title Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381 [DonNotMerge] [DoNotMerge] Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381 [DonNotMerge] Oct 18, 2026
@kennknowles
kennknowles self-requested a review October 19, 2026 16:03
@github-actions github-actions Bot added the size/xl XL: >=1000 lines changed label Oct 24, 2026
@nitinware nitinware changed the title [DoNotMerge] Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381 [DonNotMerge] Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381 Oct 25, 2026
@nitinware
nitinware marked this pull request as ready for review October 25, 2026 22:48
@codecov

codecov Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (12bbb17) to head (2db33de).
⚠️ Report is 26 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master   #40648       +/-   ##
===========================================
- Coverage   79.83%        0   -79.84%     
===========================================
  Files         494        0      -494     
  Lines       83575        0    -83575     
===========================================
- Hits        66722        0    -66722     
+ Misses      16853        0    -16853     
Flag Coverage Δ
python ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@nitinware

Copy link
Copy Markdown
Contributor Author

@kennknowles just checking in on this PR, would appreciate your review whenever you have a chance. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add CouchbaseIO

2 participants