Skip to content

fix(ingest): correct --write-mode to the real load-mode vocabulary - #279

Open
eddietejeda wants to merge 1 commit into
mainfrom
fix/write-mode-consistency-278
Open

fix(ingest): correct --write-mode to the real load-mode vocabulary#279
eddietejeda wants to merge 1 commit into
mainfrom
fix/write-mode-consistency-278

Conversation

@eddietejeda

@eddietejeda eddietejeda commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Closes #278.

Problem

ingest create --write-mode hardcoded value_parser = ["replace", "upsert"], which contradicted the rest of the CLI and the service:

  • ingest sources fields <family> renders the server-reported per-family write_modes (e.g. the filesystem family: replace, append). So one CLI surface advertised append while ingest create --write-mode append failed with invalid value 'append' [possible values: replace, upsert].
  • Confirmed against the live OpenAPI spec and the dlt worker: the real service-wide load-mode vocabulary is replace/append (keyless) plus the key-based upsert/update/delete (the worker maps upsert → dlt {disposition: merge, strategy: upsert}). The old pair both omitted valid modes (append, update, delete) and offered upsert as if universal, though it is key-gated.

Fix

  • Correct the enum to the full service-wide set [replace, append, upsert, update, delete]. clap still catches typos up front, and every mode a family may advertise is now accepted; per-family/per-table support (e.g. a key-based mode on a keyless table) is enforced by the service.
  • Rewrite the flag help to describe keyless vs key-based modes and point at ingest sources fields as the authoritative per-family source.
  • Fix skills/hotdata/SKILL.md: the write_mode note listed only replace | upsert and wrongly said upsert needs a continuous bucket ingest — it needs a declared key on the destination table.

Before / after

# before:  hotdata ingest create ... --write-mode append
error: invalid value 'append' for '--write-mode <WRITE_MODE>'  [possible values: replace, upsert]

# after:   --write-mode append  -> forwarded to the service (proceeds)
# after:   --write-mode replce  -> invalid value 'replce'  [possible values: replace, append, upsert, update, delete]

Testing

  • cargo build clean; cargo test --bin hotdata ingest → 82 passed, 0 failed.
  • Verified on the built binary: append now parses and reaches the service; typos are still caught locally against the full enum.

cc @anoop-narang

@eddietejeda
eddietejeda requested a review from a team as a code owner August 25, 2026 21:05
@eddietejeda
eddietejeda requested review from anoop-narang and removed request for a team August 25, 2026 21:05
Comment thread src/commands/ingest.rs Outdated
Comment on lines +245 to +249
/// `hotdata ingest sources fields <family>` — the authoritative source. The
/// flag forwards any value and lets the service validate, rather than
/// hardcoding a list that drifts from what families accept (that drift is
/// exactly what showed a family advertising `append` while this flag
/// refused it: see issue #278).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super nit: this doc comment is clap long-help, so hotdata ingest create --help will print "the flag forwards any value and lets the service validate, rather than hardcoding a list that drifts... see issue #278" to end users. The other issue-number references in this repo (indexes.rs:91, sdk.rs:1717, query.rs:314) are all in internal function docs and // comments, not in user-facing help strings. Consider keeping the user-relevant half and dropping the implementation rationale + tracker ref (or moving them to a // comment above the field, which is where the same argument in --record-shape effectively lives). (not blocking)

Comment thread src/commands/ingest.rs Outdated
/// hardcoding a list that drifts from what families accept (that drift is
/// exactly what showed a family advertising `append` while this flag
/// refused it: see issue #278).
#[arg(long = "write-mode")]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: nothing pins the new behavior. The existing write_mode tests (destination_flags_default_schema_and_write_mode, ~line 2396) call build_destination directly and never go through clap, so re-adding a value_parser here would not fail any test — the exact regression #278 describes. src/commands/results.rs:165-173 already has the pattern for a clap-level parse test (Wrapper + try_parse_from); a one-liner asserting ingest create ... --write-mode append parses would guard it. (not blocking)

claude[bot]
claude Bot previously approved these changes Aug 25, 2026
@codecov

codecov Bot commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@eddietejeda eddietejeda changed the title fix(ingest): stop --write-mode from rejecting modes families advertise fix(ingest): correct --write-mode to the real load-mode vocabulary Aug 25, 2026
@eddietejeda
eddietejeda force-pushed the fix/write-mode-consistency-278 branch from 1cd5b53 to 9b05476 Compare August 25, 2026 21:14
claude[bot]
claude Bot previously approved these changes Aug 25, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness of the change looks fine: the enum now covers the modes the server-reported write_modes can advertise (src/client/ingest.rs:1136,1197,1234 show replace/append), build_destination still defaults to replace (src/commands/ingest.rs:938), and the --destination JSON path is unaffected. SKILL.md text matches.

The two prior non-blocking threads (issue ref in the clap long-help, no clap-level parse test pinning --write-mode append) are still open; both remain non-blocking.

The --write-mode flag hardcoded value_parser = ["replace", "upsert"], which
contradicted what the CLI itself shows and what the service accepts:

- `ingest sources fields <family>` renders the server-reported per-family
  write_modes (e.g. filesystem: replace, append), so the CLI advertised
  `append` while the create flag rejected it with 'invalid value append'.
- The real service-wide load-mode vocabulary is replace/append (keyless) plus
  the key-based upsert/update/delete (confirmed against the OpenAPI spec and
  the dlt worker, which maps upsert -> dlt merge/upsert). The old pair both
  omitted valid modes (append, update, delete) and offered `upsert` as if
  universal, though it is key-gated.

Correct the enum to the full set so clap still catches typos up front while
accepting every mode a family may advertise; per-family/per-table support is
still enforced by the service. Fix the SKILL.md note likewise (it wrongly said
upsert needs a continuous bucket ingest; it needs a declared key).

Closes #278

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff (the branch was force-pushed, so no since-last-review diff was available). The corrected value_parser list matches the server-reported write_modes vocabulary used elsewhere in the CLI (src/client/ingest.rs:1136,1197,1222,1234), the replace default in build_destination (src/commands/ingest.rs:938) is unchanged, and the SKILL.md text is consistent with the new flag help. No blocking issues.

The two nits from the prior cycle (issue-number reference inside the user-facing clap long-help, and no clap-level test pinning the accepted values) are still open but remain non-blocking.

@eddietejeda

Copy link
Copy Markdown
Contributor Author

Live parse verification

  • --write-mode append now clears clap and reaches the service (fails with datasource_not_found for a fake datasource id — i.e., the request was actually sent; before this PR it died locally with invalid value 'append').
  • --write-mode replce → local clap error listing the full vocabulary [replace, append, upsert, update, delete].
  • update and delete accepted by clap and forwarded.

No conflicts with main (GitHub reports CLEAN); CI green. LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

deleted

1 participant