Repository navigation
re-creation of deleted/expired collection jobs - #4915
Conversation
|
Draft because these tests are failing at the moment. |
c361322 to
602eae9
Compare
|
Now we have the tests we want so that we can assure that Janus behaves correctly. Next, we have to decide how we want Janus to behave. There's eight cases of interest:
I argue that: Regardless of whether a job was deleted or expired, trying to re-create it with the same query (idempotency key or time interval, resp.) but varying other parameters (extensions, agg param) should fail with batchOverlap. If a job was deleted and we try to re-create it with an identical request, we should serve HTTP 204 No Content, the same response we'd give if you tried to GET a deleted collection job. Note that's 204 No Content, not 404 Not Found! We are acknowledging that the collection job ever existed. Finally, if a job expires and we try to re-create it with identical parameters, then we could serve HTTP 200 and the CollectionJobResp. It turns out that even if the data is expired, we still have a To sum up, I am proposing:
|
|
I think we also have work to do on the collection job GET endpoint here. I feel that if I DELETE or POST a job, and I get back 204 No Content, then subsequent GETs on that same job should yield the same representation. Currently our GET endpoint yields 404 Not Found in such cases. |
|
I think a 404 for the GET after deletion/expiry is fine. MDN's page on 204 No Content suggests that 204 is mainly used in response to DELETE and PUT requests. |
602eae9 to
cf91dfc
Compare
|
Per discussion today:
I'll add tests as needed and make them pass in this PR. |
cf91dfc to
9ac8374
Compare
9ac8374 to
0a97d69
Compare
| assert_eq!(response.status(), StatusCode::NO_CONTENT); | ||
|
|
||
| let mut response = test_case.get_collection_job(&collection_job_id).await; | ||
| test_case.check_gone_response( |
There was a problem hiding this comment.
I want to highlight this point: what we discussed this past Tuesday was serving HTTP 404 in this case (as well as in the corresponding leader selected test case). But while doing it, I noticed that we have something very similar in place for deleted aggregation jobs, and it makes sense to me to harmonize with that. Plus, no matter what, you get a 4XX status and a problem document. Since DAP implementations are meant to not get picky about status anyway, 404 vs. 408 doesn't seem like a huge deal.
0a97d69 to
40298c0
Compare
Add tests to exercise various actions on deleted or expired colletion jobs, and then make changes as needed to make them pass. GET or POST on a deleted job now yields HTTP 408 Gone plus a problem document with a custom error type (following the precedent for aggregation jobs with `Error::DeletedAggregationJob`). POST to a deleted job but with a mutated query yields HTTP 400 Bad Request plus a batchOverlap problem document. We treat data that is expired but not yet garbage collected as effectively deleted. Thus GET on an expired collection job yields HTTP 404 Not Found (same as if we'd never seen the job at all). POST on such a job yields HTTP 400 Bad Request plus a custom error type (following the precedent for aggregate shares with `Error::AggregateShareRequestRejected`). To achieve this, we modify `CollectableBatchMode:: collection_jobs_intersecting_query`. Insead of, having implementations ignore expired collection job rows, the method now returns an iterator of `(CollectionJob, bool)`, with the boolean indicating whether the job is expired, allowing application level code in `aggregator/src/aggregator.rs` to make decisions about what response to serve. And there's extensive refactoring of the collection job test setup code to make the test code less verbose and repetitive. At the datastore layer, `Transaction::get_collection_job_by_query` is now specialized to only be usable for LeaderSelected batch mode. The function was only ever used for such tasks anyway, so taking away the generic parameter `B: BatchMode` simplifies the method. The equivalent TimeInterval method is `get_collection_jobs_intersecting_interval`.
40298c0 to
fe85cdf
Compare
| tx.get_collection_jobs_intersecting_interval(vdaf, task_id, batch_identifier) | ||
| .await | ||
| .map(|v| v.into_iter().map(|(job, _)| job).collect()) | ||
| } |
There was a problem hiding this comment.
I was going to ask whether we should be filtering on the expired boolean here, but actually this method is no longer used anywhere, so we could just delete it.
| vdaf: Arc<A>, | ||
| req_bytes: &[u8], | ||
| ) -> Result<(CollectionJobId, Option<Vec<u8>>), Error> { | ||
| let req = |
There was a problem hiding this comment.
For posterity:
| // Parse the collection job request and call an inner method, so we can decorate it with a | |
| // tracing span that includes request fields. | |
| let req = |
| // complete. | ||
| let batch_id = test_case | ||
| .datastore | ||
| // Update the collection job with the aggregate shares and some aggregation jobs. collection |
There was a problem hiding this comment.
nit: This transaction isn't actually touching any aggregation jobs.
|
Public documentation for the errors added here is in divviup/public-docs#204. |
0.9.x version of Janus can use new custom error types defined by Divvi Up, introduced in divviup/janus#4915. This commit adds documentation for those errors corresponding to the problem documents Janus constructs, and does a little bit of wordsmithing on existing error docs.
Add tests to exercise various actions on deleted or expired colletion jobs, and then make changes as needed to make them pass.
GET or POST on a deleted job now yields HTTP 408 Gone plus a problem document with a custom error type (following the precedent for aggregation jobs with
Error::DeletedAggregationJob).POST to a deleted job but with a mutated query yields HTTP 400 Bad Request plus a batchOverlap problem document.
We treat data that is expired but not yet garbage collected as effectively deleted. Thus GET on an expired collection job yields HTTP 404 Not Found (same as if we'd never seen the job at all). POST on such a job yields HTTP 400 Bad Request plus a custom error type (following the precedent for aggregate shares with
Error::AggregateShareRequestRejected).To achieve this, we modify
CollectableBatchMode:: collection_jobs_intersecting_query. Insead of, having implementations ignore expired collection job rows, the method now returns an iterator of(CollectionJob, bool), with the boolean indicating whether the job is expired, allowing application level code inaggregator/src/aggregator.rsto make decisions about what response to serve.And there's extensive refactoring of the collection job test setup code to make the test code less verbose and repetitive.
At the datastore layer,
Transaction::get_collection_job_by_queryis now specialized to only be usable for LeaderSelected batch mode. The function was only ever used for such tasks anyway, so taking away the generic parameterB: BatchModesimplifies the method. The equivalent TimeInterval method isget_collection_jobs_intersecting_interval.