[prgs-rsingh]

Summary

MLE-28075 - Configure incremental write options via Flux API

Incremental write could be configured from the CLI (--incremental-write* options) but not from the Flux API. This PR adds IncrementalWriteOptions, which hangs off WriteDocumentsOptions:

Flux.importGenericFiles()
    .from("path/to/files")
    .to(options -> options
        .incrementalWrite(iw -> iw
            .enable()
            .schemaAndView("example", "incrementalWrite")
            .hashName("myHash")
            .timestampName("myTimestamp")
            .jsonExclusions("/lastModified")
            .xmlExclusions("/root/lastModified")
            .disableJsonCanonicalization()
            .logSkipped(500)))
    .execute();

Changes

• New API interface IncrementalWriteOptions (@since 2.2.0), with one method per existing CLI option:

  CLI option	API method
  --incremental-write	enable()
  --incremental-write-schema + --incremental-write-view	schemaAndView(schema, view)
  --incremental-write-hash-name	hashName(name)
  --incremental-write-timestamp-name	timestampName(name)
  --incremental-write-disable-json-canonicalization	disableJsonCanonicalization()
  --incremental-write-json-exclusion	jsonExclusions(String...)
  --incremental-write-xml-exclusion	xmlExclusions(String...)
  --log-skipped	logSkipped(int)

• WriteDocumentsOptions#incrementalWrite(Consumer<IncrementalWriteOptions>): the API entry point. As in the CLI, incremental write is only turned on by enable(); the other options have no effect without it.

• IncrementalWriteParams now implements IncrementalWriteOptions. Its picocli annotations and makeOptions() are unchanged, so existing CLI behavior is unaffected.

• WriteDocumentParams implements incrementalWrite(...) by delegating to the existing IncrementalWriteParams. TDE generation reads the hash name from the same object, so a custom hashName set through the API carries over to includeIncrementalWriteColumns().

• copy command: DocumentCopier.CopyWriteDocumentsOptions extends WriteDocumentsOptions, so CopyCommand.CopyWriteDocumentsParams now implements incrementalWrite(...). To keep the copy API and CLI consistent, IncrementalWriteParams is also added as a mixin. copy therefore now accepts the same unprefixed --incremental-write* / --log-skipped options as the import commands, as it already does with the splitter, embedder and classifier mixins. (See follow-up comment for confirmation on this.)

• Docs: noted in docs/import/incremental-write.md that copy supports incremental write. The doc covers CLI options only, like the other feature pages.

Notes

• --incremental-write-source-uri-name (PDP-2114 Added --incremental-write-source-uri-name #677) is not included here. It depends on the unreleased connector change in PDP-2114 Incremental write source URI support marklogic-spark-connector#697. The matching sourceUriName(...) API method can be added once both are merged.
• logSkipped(0) behaves the same as --log-skipped 0. The option is omitted so the connector's default of 0, which disables logging, applies.

Testing

• New IncrementalWriteOptionsTest: covers all options, defaults, and incremental write staying off unless requested.
• CopyOptionsTest: new cases cover the copy CLI options, the copy API, and incremental write staying off unless requested.
• GenericFilesImporterTest#incrementalWrite: integration test that imports the same files twice and verifies the hash and timestamp metadata are written and that unchanged documents are skipped on the second run (timestamp unchanged; logs Skipped count: 4). To support it, the test-app content database now has an incrementalWriteHash metadata field with an unsignedLong range index.
• Ran locally: IncrementalWriteOptionsTest, CopyOptionsTest, ImportFilesOptionsTest, SplitterOptionsTest (26 passing), plus GenericFilesImporterTest (8 passing) against a freshly deployed test-app on MarkLogic 12. :flux-cli:javadoc builds cleanly.

[prgs-rsingh]

Adding incrementalWrite(...) to WriteDocumentsOptions also adds it to the copy API (DocumentCopier.CopyWriteDocumentsOptions), but the copy CLI has never had --incremental-write* options. For consistency I've added incremental write to the copy CLI too, reusing the same unprefixed option names the import commands use, as the splitter, embedder and classifier options already do on copy. Is that OK, or should copy support it through the API only, or not at all?

[Copilot]

🟡 Changes recommended

The new test uses an apiHash lookup without provisioning the required metadata field and range index, causing it to fail on a fresh test-app deployment before reaching the skipped-document assertions.

1 open issue

• Provision apiHash range index before hash lookup test · New

What changed in this PR

This PR exposes incremental-write configuration through the Flux Java API, reusing the existing CLI parameters, and adds incremental-write support to copy.

Changes:

• Adds IncrementalWriteOptions and enables configuration through WriteDocumentsOptions.
• Connects import and copy options to the shared incremental-write implementation.
• Adds option-mapping tests, an integration test, and API usage documentation.

File	Description
flux-cli/​src/​test/​java/​com/​marklogic/​flux/​impl/​copy/​CopyOptionsTest.java	Tests copy CLI and API options and default behavior.
flux-cli/​src/​test/​java/​com/​marklogic/​flux/​api/​IncrementalWriteOptionsTest.java	Tests API option mappings, defaults, and opt-in behavior.
flux-cli/​src/​test/​java/​com/​marklogic/​flux/​api/​GenericFilesImporterTest.java	Adds repeat-import verification of hashes, timestamps, and skipping.
flux-cli/​src/​main/​java/​com/​marklogic/​flux/​impl/​importdata/​WriteDocumentParams.java	Enables incremental writing through import API options.
flux-cli/​src/​main/​java/​com/​marklogic/​flux/​impl/​importdata/​IncrementalWriteParams.java	Implements the new API using existing CLI parameters.
flux-cli/​src/​main/​java/​com/​marklogic/​flux/​impl/​copy/​CopyCommand.java	Wires incremental-write options into copy CLI and API configuration.
flux-cli/​src/​main/​java/​com/​marklogic/​flux/​api/​WriteDocumentsOptions.java	Adds the incremental-write configuration entry point.
flux-cli/​src/​main/​java/​com/​marklogic/​flux/​api/​IncrementalWriteOptions.java	Defines the public incremental-write configuration interface.
docs/​import/​incremental-write.md	Documents API usage, CLI mappings, and copy support.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues remain for the incremental-write API and copy-command changes.

No open issues 🎉

1 resolved since last review

• Provision apiHash range index before hash lookup test

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

There are no active review issues for the incremental-write API and copy-command changes.

No open issues 🎉

_{Review effort: Balanced}