Skip to content

sink/periodic-jobs: deliver and report one item at a time (DI-3015) - #323

Merged
jbndbc merged 1 commit into
DBCPR-328-dataio-chunk-scheduling-redesignfrom
DI-3015-periodic-jobs-sink-migration
Sep 15, 2026
Merged

jbndbc merged 1 commit into
DBCPR-328-dataio-chunk-scheduling-redesignfrom
DI-3015-periodic-jobs-sink-migration

Conversation

@jbndbc

@jbndbc jbndbc commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Migrate the sink from MessageConsumerAdapter to SinkMessageConsumerAdapter. job-store now dispatches one JMS message per item, and the sink implements deliverItem(ConsumedMessage, ChunkItem) instead of assembling a DELIVERED chunk. Header reading, result reporting and the tracking-id scope move to the framework.

The sink aggregates a whole job before delivering anything, so it overrides usesDeliveryWatermark() to false. It still reports every item individually, since the DELIVERING counters and the per-job gate are driven by those reports. The job termination item arrives as an ordinary item message carrying ChunkItem.Type.JOB_END.

Drop the five second sleep that preceded finalization. It guessed at a window the chunk protocol opened by reporting the result before committing the sink's own transaction. deliverItem commits first and the framework reports afterwards, so a reported item is an item whose datablocks are durable and the termination chunk is released only against complete data.

InvalidMessageException from finalization is now reported as a FAILED termination item rather than discarding the message, which completes the job and sets its fatal error flag instead of leaving it forever incomplete.

Migrate the sink from MessageConsumerAdapter to SinkMessageConsumerAdapter.
job-store now dispatches one JMS message per item, and the sink implements
deliverItem(ConsumedMessage, ChunkItem) instead of assembling a DELIVERED
chunk. Header reading, result reporting and the tracking-id scope move to
the framework.

The sink aggregates a whole job before delivering anything, so it overrides
usesDeliveryWatermark() to false. It still reports every item individually,
since the DELIVERING counters and the per-job gate are driven by those
reports. The job termination item arrives as an ordinary item message
carrying ChunkItem.Type.JOB_END.

Drop the five second sleep that preceded finalization. It guessed at a
window the chunk protocol opened by reporting the result before committing
the sink's own transaction. deliverItem commits first and the framework
reports afterwards, so a reported item is an item whose datablocks are
durable and the termination chunk is released only against complete data.

InvalidMessageException from finalization is now reported as a FAILED
termination item rather than discarding the message, which completes the
job and sets its fatal error flag instead of leaving it forever incomplete.

@ja7 ja7 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hvordan i forhold til
it rolls the JMS session back and the item is redelivered until the broker gives up
Jeg er ikke sikker på hvordan vi håndtere det i dag. ( er det JobStore der arbonere på DeadMessageQueue ? )

@jbndbc

jbndbc commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Hvordan i forhold til it rolls the JMS session back and the item is redelivered until the broker gives up Jeg er ikke sikker på hvordan vi håndtere det i dag. ( er det JobStore der arbonere på DeadMessageQueue ? )

Ja, det er samme mekanisme som i det nuværende, bare lavet om til at fungere på item niveau i stedet for chunk niveau. Den funktionalitet lå i et tidligere PR.

@jbndbc jbndbc closed this Sep 14, 2026
@jbndbc jbndbc reopened this Sep 14, 2026
@jbndbc
jbndbc merged commit b584a39 into DBCPR-328-dataio-chunk-scheduling-redesign Sep 15, 2026
1 of 2 checks passed
@jbndbc
jbndbc deleted the DI-3015-periodic-jobs-sink-migration branch September 15, 2026 11:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants