DBCDK · jbndbc · Sep 16, 2026 · Sep 7, 2026
@@ -285,10 +285,13 @@ what covers the ordinary case where they finish after. Both apply the cross-job
 both are no-ops for a job that has no termination chunk at all.
 
 Site A has to know whether the chunk it just removed was the termination chunk, so that the
-job's barrier does not count itself. That falls out of the removal token if
+job's barrier does not count itself. That falls out of the removal token, because
 `DependencyTrackingService.remove(TrackingKey)` returns the removed `DependencyTracking`
-rather than a boolean: the winning caller gets the proof it won *and* the `is_termination`
-flag of the row it removed, in one call and with no extra read.
+rather than void: the winning caller gets the proof it won *and* the `is_termination`
+flag of the entry it removed, in one call and with no extra read. The flag is a field on that
+object as well as a column on the row, which is sound for a value decided when the row is created
+and never changed afterwards. `gate_open` has no such field, deliberately, see [Who writes the gate
+columns before Phase 9](#barrier-chunks--per-job-gate).
 
 **Who writes the gate columns before Phase 9.** Everything above is written as if
 `dependencytracking` were already a plain PostgreSQL table. It becomes one at [Phase 9](
@@ -422,13 +425,14 @@ why the increment must be a single atomic statement
 (`SET data_chunks_delivered = data_chunks_delivered + 1`) rather than a read followed by
 a write.
 
-The removal of the chunk's `dependencytracking` row is the natural token, since exactly
-one concurrent caller can perform it. The increment must therefore be conditioned on
-having won that removal, not on a preceding read of the row - the current
-`get`-then-`remove` sequence in `chunkDeliveringDone` is a non-atomic check-then-act that
-two callers can both pass. Harmless for the idempotent work that follows it today, not
-harmless for a counter. See [Phase 1](#phase-1--gate-and-ordered-dispatch-job-store-service)
-for the required signature change. The `>=` in the pseudocode above is defensive only:
+The removal of the chunk's `dependencytracking` row is the token, since exactly one
+concurrent caller can perform it, and the increment is conditioned on having won that
+removal rather than on a preceding read of the row. `chunkDeliveringDone` still reads the
+entry first, to answer "is this chunk in `QUEUED_FOR_DELIVERY`" and to turn an unknown chunk
+away, but that read is a filter and not the token: `get`-then-`remove` is a non-atomic
+check-then-act that two callers can both pass. Harmless for the idempotent work that follows
+it, not harmless for a counter. See [Phase 1](#phase-1--gate-and-ordered-dispatch-job-store-service)
+for the signature. The `>=` in the pseudocode above is defensive only:
 it does not mitigate either hazard above, since double counting reaches the total exactly
 and a lost update leaves the counter below it forever. It guards only against the counter
 being pushed above `data_chunks_expected` by something other than the increment, such as a
@@ -2033,7 +2037,11 @@ Two ordering constraints shape the sequence:
   transaction
 - **DI-3049** Change `DependencyTrackingService.remove(TrackingKey)` to return the removed
   `DependencyTracking`, so the caller learns whether *this* call
-  performed the removal, and condition the `data_chunks_delivered` increment on it.
+  performed the removal, and condition the `data_chunks_delivered` increment on it. Carry
+  `is_termination` on `DependencyTracking` as well, so the caller that won the removal reads the
+  branch off the entry it was handed rather than querying the row it has just removed. That is the
+  shape Phase 9 arrives at anyway, once the row is deleted synchronously and a read after the
+  removal would return nothing.
   It already computes this and discards it: `dependencyTracker.remove(key)` returns the
   previous value atomically per key and the method checks `removed == null`, but returns
   `void`, so no caller can learn it won the race. Without this the increment sits behind

@@ -117,7 +117,16 @@ depend on a row existing at a particular moment.
 Job-store owns the **gate column values**. `DependencyTrackingStore`'s upsert names five columns in
 its `on conflict ... do update set` clause, and cannot clobber a column it does not name.
 `is_termination` and `gate_open` are outside that list, which is what lets job-store write them in
-plain synchronous SQL with no lag. **Never add those two columns to that clause.**
+plain synchronous SQL with no lag. **Never add those two columns to that clause**, which
+`JobGateIT.mapStoreDoesNotClobberGateColumns` is there to enforce.
+
+`is_termination` is also a field on `DependencyTracking`, and that is not a contradiction. The clause
+still does not name it, so nothing about who writes the column changes, and a copy on the map value
+cannot go stale for a value decided when the row is created and never changed. A delivery reads the
+branch off the entry it removed rather than querying a row it has just removed, see
+[Where the gate is decided](#where-the-gate-is-decided). **`gate_open` gets no such field.** Four
+sites write it over a chunk's life, so a copy would be a second answer to a question with one, and a
+stale open gate dispatches a job's end-of-job work ahead of the data it summarises.
 
 Three consequences follow:
 
@@ -161,6 +170,16 @@ Five things about those sites are easy to get wrong:
   having a termination chunk, it would lose every chunk delivered before that chunk existed. As a
   read-then-write, it could lose an update and leave the counter permanently one short. Either way
   the gate never opens.
+- **Site A runs once per chunk, and what makes that true is the removal.** `chunkDeliveringDone` is
+  called again by every redelivery, and the broker's failure detection can produce two genuinely
+  concurrent calls for one chunk, so the count hangs off the one thing only one caller can do:
+  `DependencyTrackingService.remove` hands the removed entry to whichever caller removed it and null
+  to every other. The read of the entry before it is a filter on status, not the token, since
+  `get`-then-`remove` is a check-then-act two callers can both pass. Counted twice, the counter still
+  lands on `data_chunks_expected` exactly, only while a data chunk is still in flight, so the failure
+  is a job that reads as complete rather than one that stalls. The rest of `chunkDeliveringDone` runs
+  for every caller: `removeFromWaitingOn` reports only the entries it changed, so the caller that
+  lost the removal finds nothing left to unblock.
 - **`data_chunks_expected = 0` means two opposite things**: the migration default on jobs that
   predate the gate, which must be ignored, and a genuine job with no data chunks, whose gate must be
   decided at once. `is_termination` is what tells them apart, so the gate keys on that and uses the
@@ -275,6 +294,14 @@ Two ways a gate outlives its reason to be shut, each of them a job that never co
   loss on the termination branch never lifts the barrier, stalling every later job from that
   submitter.
 
+That second one is inherent while `dependencytracking` is a Hazelcast map, and it is accepted rather
+than overlooked. The removal is what both the count's once-only property and a redelivery's
+early return hang off, so no choice of marker closes the window: it closes when the row is deleted in
+the same transaction as the count, which is where the map goes away. Accepting it buys the far worse
+failure being gone. A count that is lost leaves a job visibly stuck and swept within the hour, while
+a count taken twice reaches the total early and lets end-of-job work run on data that never arrived,
+with nothing in the job state saying so.
+
 `AdminBean.recheckBlocks` sweeps for both hourly, which bounds the damage to one sweep interval. It
 lifts the barrier of any job left holding one with no termination row, then opens any gate closed
 with no earlier unlifted barrier, requiring additionally for a termination chunk that its own job's
@@ -302,10 +329,10 @@ Delivery order and `gate_open` are read from PostgreSQL, while `status` is read
 the table's copy of it is written write-behind and lags. The bulk sweep therefore takes an ordered
 candidate list from SQL and re-checks each candidate against the map before dispatching it.
 
-The query cannot be a Hazelcast predicate, and the reason is the ownership split above:
-`gate_open` and `is_termination` are columns on the table and deliberately not fields on
-`DependencyTracking`, so no predicate can see them. Putting them on the map value to make one
-possible is the very thing that would let the MapStore clobber them.
+The query cannot be a Hazelcast predicate, and the reason is the ownership split above: `gate_open`
+is a column on the table and deliberately not a field on `DependencyTracking`, so no predicate can
+see it. Putting it on the map value to make one possible is what would give a chunk two answers to
+whether it may be dispatched, one of them written behind the other's back.
 
 ## Delivery watermark
 
@@ -377,6 +404,12 @@ Each entry holds:
 - **`matchKeys`** — string keys derived from sequence analysis data plus an optional barrier key; used to find chunks this one must sequence after
 - **`waitFor`** — indexed form of matchKeys as `WaitFor(sinkId, submitter, key)` tuples, used for Hazelcast predicate queries
 - **`waitingOn`** — set of `TrackingKey`s this chunk is currently blocked by
+- **`termination`** - whether this chunk is its job's termination chunk, mirroring the
+  `is_termination` column. The column is the authority and this is read back from it whenever an
+  entry is loaded, which is sound because the value is decided when the row is created and never
+  changes. It is here so that the caller who removes an entry on delivery can tell the two branches
+  of the gate apart from the entry it was handed, see [Where the gate is decided](#where-the-gate-is-decided).
+  **`gate_open` has no counterpart here, and must not get one**, see below
 - **`priority`**, **`lastModified`**, **`retries`**
 
 ## Chunk lifecycle

@@ -36,6 +36,7 @@ public class DependencyTracking implements DependencyTrackingRO, Serializable, C
     private final int submitter;
     private Instant lastModified = Instant.now();
     private int retries = 0;
+    private boolean termination = false;
 
     public DependencyTracking(TrackingKey key, int sinkId, int submitter, Set<String> sequenceData) {
         this.key = key;
@@ -63,6 +64,7 @@ public DependencyTracking(ResultSet rs) throws SQLException {
         submitter = rs.getInt("submitter");
         lastModified = rs.getTimestamp("lastmodified").toInstant();
         retries = rs.getInt("retries");
+        termination = rs.getBoolean("is_termination");
         waitFor = toWaitForIndexSet(sinkId, submitter, matchKeys);
     }
 
@@ -121,6 +123,28 @@ public int getSubmitter() {
         return submitter;
     }
 
+    /**
+     * Says whether this chunk is its job's termination chunk.
+     * <p>
+     * The {@code is_termination} column is the authority and this is a copy of it, which is sound
+     * because the value is decided when the row is created and never changes afterwards. It is set
+     * on the entry the termination chunk is scheduled with, and read back from the column whenever
+     * an entry is loaded from the table.
+     * <p>
+     * {@code gate_open} deliberately has no counterpart here. It is written by four sites over a
+     * chunk's life, so a copy on this object could be stale, and a stale open gate dispatches a
+     * job's end-of-job work ahead of the data it summarises.
+     */
+    @Override
+    public boolean isTermination() {
+        return termination;
+    }
+
+    public DependencyTracking setTermination(boolean termination) {
+        this.termination = termination;
+        return this;
+    }
+
     @Override
     public int getPriority() {
         return priority;

@@ -18,6 +18,8 @@ public interface DependencyTrackingRO {
 
     int getSubmitter();
 
+    boolean isTermination();
+
     int getPriority();
 
     Instant getLastModified();

@@ -43,6 +43,11 @@ public class DependencyTrackingStore implements MapStore<TrackingKey, Dependency
      * update list.</b> Adding them would reset every termination chunk's gate on its next status
      * transition, dispatching job-end work ahead of the data it summarises.
      * <p>
+     * {@link DependencyTracking#isTermination()} carries {@code is_termination} on the value object
+     * and does not change that. The column is written where the termination row is created and read
+     * back by {@link DependencyTracking#DependencyTracking(java.sql.ResultSet)}, so this store has
+     * no reason to name it in either list.
+     * <p>
      * See docs/chunk-scheduling-redesign.md, "Who writes the gate columns before Phase 9", and
      * {@code JobGateRepository} in the war module.
      */

@@ -235,15 +235,28 @@ private StatusChangeEvent setStatus(TrackingKey key, ChunkSchedulingStatus newSt
         return statusChangeEvent;
     }
 
-    public void remove(TrackingKey key) {
+    /**
+     * Removes a chunk's entry and hands the removed entry to the caller that removed it.
+     * <p>
+     * The return value is the once-only token for this removal. {@code IMap.remove} is atomic per
+     * key across the cluster, so however many callers ask to remove one chunk, exactly one is given
+     * the entry and every other is given null. A caller that has to act once per chunk asks that
+     * question here rather than by reading the entry first, which two callers can both pass.
+     * {@code JobSchedulerBean.chunkDeliveringDone} counts a delivered data chunk on that basis.
+     *
+     * @param key chunk to remove
+     * @return the removed entry, or null if this call did not remove it
+     */
+    public DependencyTracking remove(TrackingKey key) {
         DependencyTracking removed = dependencyTracker.remove(key);
-        if(removed == null) return;
+        if(removed == null) return null;
         countersMap.executeOnKey(removed.getSinkId(), new UpdateCounter(removed.getStatus(), -1));
         if(enableWaitForTracking) {
             PredicateBuilder.EntryObject o = Predicates.newPredicateBuilder().getEntryObject();
             lastTracker.removeAll(o.get("jobId").equal(key.getJobId()).and(o.get("chunkId").equal(key.getChunkId())));
         }
         LOGGER.info("Removed tracking key {} from dependency tracker", key.toChunkIdentifier());
+        return removed;
     }
 
     public void remove(Predicate<TrackingKey, DependencyTracking> predicate) {

@@ -2,6 +2,7 @@
 
 import dk.dbc.dataio.commons.types.interceptor.Stopwatch;
 import dk.dbc.dataio.jobstore.distributed.ChunkSchedulingStatus;
+import dk.dbc.dataio.jobstore.distributed.DependencyTracking;
 import dk.dbc.dataio.jobstore.distributed.TrackingKey;
 import jakarta.annotation.Resource;
 import jakarta.ejb.EJB;
@@ -121,21 +122,21 @@ private JobGateBean self() {
      * transaction commits before JAX-RS writes the response, so no sink can acknowledge a message
      * whose increment has not committed.
      * <p>
-     * Not idempotent, and nothing here detects a repeat. The caller arrives after a {@code get}, a
-     * status check and a {@code remove} on the chunk's dependency tracking entry, and those three
-     * are not atomic, so two concurrent acknowledgements of one chunk can both pass them and both
-     * be counted.
+     * Counts once per chunk, because it is reached only by the caller that removed the chunk's
+     * dependency tracking entry. That removal is atomic per key, so of two concurrent
+     * acknowledgements of one chunk only one arrives here and the other is told it did not remove
+     * anything. Every fact this method works from comes off the removed entry, including whether the
+     * chunk was its job's termination chunk.
      *
-     * @param key       delivered chunk
-     * @param sinkId    sink the chunk was delivered to
-     * @param submitter submitter the chunk's job belongs to
+     * @param removed the delivered chunk's removed dependency tracking entry
      */
     @Stopwatch
-    public void advanceGateState(TrackingKey key, int sinkId, int submitter) {
-        if (jobGateRepository.isTerminationChunk(key)) {
-            liftBarrierAndRetrigger(key.getJobId(), sinkId, submitter);
+    public void advanceGateState(DependencyTracking removed) {
+        int jobId = removed.getKey().getJobId();
+        if (removed.isTermination()) {
+            liftBarrierAndRetrigger(jobId, removed.getSinkId(), removed.getSubmitter());
         } else {
-            countDataChunk(key.getJobId(), sinkId, submitter);
+            countDataChunk(jobId, removed.getSinkId(), removed.getSubmitter());
         }
     }
 

@@ -172,22 +172,6 @@ public void upsertGateRow(TrackingKey key, int sinkId, int submitter, ChunkSched
                 .executeUpdate();
     }
 
-    /**
-     * @param key tracking key to ask about
-     * @return true if the chunk is its job's termination chunk
-     * <p>
-     * A missing row answers false, which is correct either way round: a chunk with no row is not a
-     * termination chunk, and a data chunk has to be counted whether or not its row is there.
-     */
-    public boolean isTerminationChunk(TrackingKey key) {
-        return !entityManager.createNativeQuery(
-                        "SELECT 1 FROM dependencytracking WHERE jobid = ?1 AND chunkid = ?2 AND is_termination")
-                .setParameter(1, key.getJobId())
-                .setParameter(2, key.getChunkId())
-                .getResultList()
-                .isEmpty();
-    }
-
     /**
      * @param sinkId    sink the job delivers to
      * @param submitter the job's submitter
@@ -299,8 +283,8 @@ public List<TrackingKey> laterClosedGates(int sinkId, int submitter, int jobId)
      * <p>
      * Leaving the width out buys two things. The set is read in one place, the scheduler, where the
      * job and its cached sink are already in hand, rather than at every call site that lifts a
-     * barrier, one of which holds only a {@code DependencyTrackingRO} and would need an extra job
-     * load per delivered termination chunk. And a row stays reopenable after the width that closed
+     * barrier, one of which holds only the removed dependency tracking entry and would need an extra
+     * job load per delivered termination chunk. And a row stays reopenable after the width that closed
      * it has gone: one closed by an earlier deployment, or by a sink whose type has since changed,
      * would otherwise be stranded closed with only the sweep to rescue it.
      * <p>