Skip to content

re-creation of deleted/expired collection jobs - #4915

Merged
tgeoghegan merged 3 commits into
mainfrom
timg/collection-job-delete-tests
Oct 24, 2026
Merged

tgeoghegan merged 3 commits into
mainfrom
timg/collection-job-delete-tests

Conversation

@tgeoghegan

@tgeoghegan tgeoghegan commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Add tests to exercise various actions on deleted or expired colletion jobs, and then make changes as needed to make them pass.

GET or POST on a deleted job now yields HTTP 408 Gone plus a problem document with a custom error type (following the precedent for aggregation jobs with Error::DeletedAggregationJob).

POST to a deleted job but with a mutated query yields HTTP 400 Bad Request plus a batchOverlap problem document.

We treat data that is expired but not yet garbage collected as effectively deleted. Thus GET on an expired collection job yields HTTP 404 Not Found (same as if we'd never seen the job at all). POST on such a job yields HTTP 400 Bad Request plus a custom error type (following the precedent for aggregate shares with Error::AggregateShareRequestRejected).

To achieve this, we modify CollectableBatchMode:: collection_jobs_intersecting_query. Insead of, having implementations ignore expired collection job rows, the method now returns an iterator of (CollectionJob, bool), with the boolean indicating whether the job is expired, allowing application level code in aggregator/src/aggregator.rs to make decisions about what response to serve.

And there's extensive refactoring of the collection job test setup code to make the test code less verbose and repetitive.

At the datastore layer, Transaction::get_collection_job_by_query is now specialized to only be usable for LeaderSelected batch mode. The function was only ever used for such tasks anyway, so taking away the generic parameter B: BatchMode simplifies the method. The equivalent TimeInterval method is get_collection_jobs_intersecting_interval.

@tgeoghegan
tgeoghegan added this pull request to stack #4914 October 19, 2026 18:30
@tgeoghegan

Copy link
Copy Markdown
Contributor Author

Draft because these tests are failing at the moment.

@tgeoghegan
tgeoghegan force-pushed the timg/collection-job-delete-tests branch 2 times, most recently from c361322 to 602eae9 Compare October 19, 2026 19:15
@tgeoghegan

Copy link
Copy Markdown
Contributor Author

Now we have the tests we want so that we can assure that Janus behaves correctly. Next, we have to decide how we want Janus to behave. There's eight cases of interest:

  • Leader selected, deleted job, re-create job with identical request
  • Leader selected, deleted job, re-create job with same idempotency key but mutated request
  • Leader selected, expired job, re-create job with identical request
  • Leader selected, expired job, re-create job with same idempotency key but mutated request
  • Time interval, deleted job, re-create job with identical request
  • Time interval, deleted job, re-create job with same time interval but mutated request
  • Time interval, expired job, re-create job with identical request
  • Time interval, expired job, re-create job with same time interval but mutated request

I argue that:

Regardless of whether a job was deleted or expired, trying to re-create it with the same query (idempotency key or time interval, resp.) but varying other parameters (extensions, agg param) should fail with batchOverlap.

If a job was deleted and we try to re-create it with an identical request, we should serve HTTP 204 No Content, the same response we'd give if you tried to GET a deleted collection job. Note that's 204 No Content, not 404 Not Found! We are acknowledging that the collection job ever existed.

Finally, if a job expires and we try to re-create it with identical parameters, then we could serve HTTP 200 and the CollectionJobResp. It turns out that even if the data is expired, we still have a collection_jobs row in state Finished, with aggregate shares and a checksum and everything. But I think it would be better to serve 204 No Content, on the premise that the data has expired and thus is effectively deleted, especially since the garbage collector will eventually catch up and delete it for real.

To sum up, I am proposing:

  • Leader selected, deleted job, re-create job with identical request -> 204 No Content
  • Leader selected, deleted job, re-create job with same idempotency key but mutated request -> 400 Bad Request + batchOverlap problem document
  • Leader selected, expired job, re-create job with identical request -> 204 No Content
  • Leader selected, expired job, re-create job with same idempotency key but mutated request -> 400 Bad Request + batchOverlap problem document
  • Time interval, deleted job, re-create job with identical request -> 204 No Content
  • Time interval, deleted job, re-create job with same time interval but mutated request -> 400 Bad Request + batchOverlap problem document
  • Time interval, expired job, re-create job with identical request -> 204 No Content
  • Time interval, expired job, re-create job with same time interval but mutated request -> 400 Bad Request + batchOverlap problem document

Base automatically changed from timg/clarify-batch-aggregation-scrubbing to main October 20, 2026 16:57
@tgeoghegan

Copy link
Copy Markdown
Contributor Author

I think we also have work to do on the collection job GET endpoint here. I feel that if I DELETE or POST a job, and I get back 204 No Content, then subsequent GETs on that same job should yield the same representation. Currently our GET endpoint yields 404 Not Found in such cases.

@divergentdave

Copy link
Copy Markdown
Collaborator

I think a 404 for the GET after deletion/expiry is fine. MDN's page on 204 No Content suggests that 204 is mainly used in response to DELETE and PUT requests.

@tgeoghegan
tgeoghegan force-pushed the timg/collection-job-delete-tests branch from 602eae9 to cf91dfc Compare October 21, 2026 00:52
@tgeoghegan

Copy link
Copy Markdown
Contributor Author

Per discussion today:

I'll add tests as needed and make them pass in this PR.

@tgeoghegan
tgeoghegan force-pushed the timg/collection-job-delete-tests branch from cf91dfc to 9ac8374 Compare October 22, 2026 23:50
@tgeoghegan tgeoghegan changed the title Tests for deleted/expired collection job creation re-creation of deleted/expired collection jobs Oct 22, 2026
@tgeoghegan
tgeoghegan marked this pull request as ready for review October 22, 2026 23:51
@tgeoghegan
tgeoghegan requested a review from a team as a code owner October 22, 2026 23:51
@tgeoghegan
tgeoghegan force-pushed the timg/collection-job-delete-tests branch from 9ac8374 to 0a97d69 Compare October 22, 2026 23:57
assert_eq!(response.status(), StatusCode::NO_CONTENT);

let mut response = test_case.get_collection_job(&collection_job_id).await;
test_case.check_gone_response(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to highlight this point: what we discussed this past Tuesday was serving HTTP 404 in this case (as well as in the corresponding leader selected test case). But while doing it, I noticed that we have something very similar in place for deleted aggregation jobs, and it makes sense to me to harmonize with that. Plus, no matter what, you get a 4XX status and a problem document. Since DAP implementations are meant to not get picky about status anyway, 404 vs. 408 doesn't seem like a huge deal.

@tgeoghegan
tgeoghegan force-pushed the timg/collection-job-delete-tests branch from 0a97d69 to 40298c0 Compare October 23, 2026 15:08
Comment thread db/00000000000001_initial_schema.up.sql Resolved
Comment thread aggregator/src/aggregator.rs Resolved
Add tests to exercise various actions on deleted or expired colletion
jobs, and then make changes as needed to make them pass.

GET or POST on a deleted job now yields HTTP 408 Gone plus a problem
document with a custom error type (following the precedent for
aggregation jobs with `Error::DeletedAggregationJob`).

POST to a deleted job but with a mutated query yields HTTP 400 Bad
Request plus a batchOverlap problem document.

We treat data that is expired but not yet garbage collected as
effectively deleted. Thus GET on an expired collection job yields HTTP
404 Not Found (same as if we'd never seen the job at all). POST on such
a job yields HTTP 400 Bad Request plus a custom error type (following
the precedent for aggregate shares with
`Error::AggregateShareRequestRejected`).

To achieve this, we modify
`CollectableBatchMode:: collection_jobs_intersecting_query`. Insead of,
having implementations ignore expired collection job rows, the method
now returns an iterator of `(CollectionJob, bool)`, with the boolean
indicating whether the job is expired, allowing application level code
in `aggregator/src/aggregator.rs` to make decisions about what response
to serve.

And there's extensive refactoring of the collection job test setup code
to make the test code less verbose and repetitive.

At the datastore layer, `Transaction::get_collection_job_by_query` is
now specialized to only be usable for LeaderSelected batch mode. The
function was only ever used for such tasks anyway, so taking away the
generic parameter `B: BatchMode` simplifies the method. The equivalent
TimeInterval method is `get_collection_jobs_intersecting_interval`.
@tgeoghegan
tgeoghegan force-pushed the timg/collection-job-delete-tests branch from 40298c0 to fe85cdf Compare October 23, 2026 16:06
Comment thread aggregator_core/src/batch_mode.rs Outdated
Comment on lines 65 to 77
tx.get_collection_jobs_intersecting_interval(vdaf, task_id, batch_identifier)
.await
.map(|v| v.into_iter().map(|(job, _)| job).collect())
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going to ask whether we should be filtering on the expired boolean here, but actually this method is no longer used anywhere, so we could just delete it.

vdaf: Arc<A>,
req_bytes: &[u8],
) -> Result<(CollectionJobId, Option<Vec<u8>>), Error> {
let req =

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For posterity:

Suggested change
// Parse the collection job request and call an inner method, so we can decorate it with a
// tracing span that includes request fields.
let req =

// complete.
let batch_id = test_case
.datastore
// Update the collection job with the aggregate shares and some aggregation jobs. collection

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: This transaction isn't actually touching any aggregation jobs.

@tgeoghegan
tgeoghegan merged commit 4b07919 into main Oct 24, 2026
7 of 8 checks passed
@tgeoghegan
tgeoghegan deleted the timg/collection-job-delete-tests branch October 24, 2026 19:12
@tgeoghegan

Copy link
Copy Markdown
Contributor Author

Public documentation for the errors added here is in divviup/public-docs#204.

tgeoghegan added a commit to divviup/public-docs that referenced this pull request Oct 29, 2026
0.9.x version of Janus can use new custom error types defined by Divvi
Up, introduced in divviup/janus#4915. This
commit adds documentation for those errors corresponding to the problem
documents Janus constructs, and does a little bit of wordsmithing on
existing error docs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants