[earakely-scale]

Stacked on #149 (which is on #147): merge those first, then this retargets to main.

Summary

A grant-issuing store can now take an object a remote party uploads in parallel parts. The issuer stays the one that makes, checks or discards the object.

ObjectStore.begin_write(object_url, *, media_type, max_bytes, kinds, expires_in=None) returns a PendingWrite:

• grant: the WriteObject to hand over, of a kind in kinds, preferring http-put-parts;
• complete(uploaded): makes the object, or raises UploadFailedError and aborts the write. It refuses a reported size over max_bytes, then checks the stored bytes are the reported ones, so nothing over the limit becomes an object;
• abort(): idempotent. Leaving its with block without completing aborts. A staged upload (S3 multipart) is discarded. A one-PUT write's object lands as the PUT finishes and stays, because object stores do not delete. So the caller uses object_url only after complete, and begins each write at a fresh URL.

The default wraps issue_write_grant in one URL:

• a one-part http-put-parts grant for a receiver that takes parts, else an http-put;
• bounded by max_single_upload_bytes;
• complete checks the object landed at the reported size.

Every store that issues write grants therefore supports begin_write with no change, including out-of-tree ones.

S3ObjectStore parts:

• Start: a multipart upload created with the media type, each part presigned. Parts are 64 MiB, or larger so a grant never names more than MAX_PARTS.
• Complete: lists the stored parts and finishes the upload only when they are parts 1..n, each part_bytes but the last, totalling the reported size. Otherwise it aborts.
• Errors:
  • a ClientError, or a botocore connection or timeout error, starting the upload becomes GrantUnavailableError, which callers already read as "offer today's form";
  • either one listing or completing becomes UploadFailedError, a ValueError, so a caller that fails one item on bad input fails only that write;
  • abort logs instead of raising, including on botocore connection and timeout errors, so it never hides the error that triggered it.
  • a one-PUT write whose store cannot read back the object (a denied HEAD, a provider error) fails as UploadFailedError too.

Wrappers:

• LocalRunObjectStore.begin_write goes through the same @local write fence as issue_write_grant.
• StagedObjectStore refuses begin_write instead of letting __getattr__ hand the agent a grant on the store it wraps. A staged form comes later, with the agent issuers.

Regional signing (affects every S3 grant on AWS). When its client sends to one of AWS's own S3 hosts with no addressing style configured, S3ObjectStore.from_config sets addressing_style="virtual" (path-style for dotted bucket names), so presigned URLs name the bucket's region. botocore's default signs against the global endpoint, and a bucket less than a day old answers that with 307 TemporaryRedirect; grant transfers deliberately don't follow redirects. Found running parts uploads against a fresh bucket.

• An endpoint configured in AWS's place (an S3-compatible store through AWS_ENDPOINT_URL_S3, a PrivateLink endpoint), and an AWS profile's s3.addressing_style, are left as boto3 resolves them.
• API requests already went to the regional host; only signed URLs move, for buckets outside us-east-1, from <bucket>.s3.amazonaws.com to <bucket>.s3.<region>.amazonaws.com. A sandbox egress allowlist written for the global host must name the regional one.

The S3 write cuts and checks its parts with the protocol's part_ranges, the rule the uploader sends by.

Accepted limit. A one-PUT write's grant stays good until it expires, so the receiver, which chose the bytes anyway, could replace an accepted object until then. The S3 parts path cannot: completing it ends the upload, and a re-PUT of a part URL gets NoSuchUpload (checked on a real bucket).

issue_write_grant is unchanged, and nothing calls begin_write yet; snapshot_env adopts it next.

Tests

• pending_write_test.py, the default:
  • a one-URL parts grant vs a put grant vs neither;
  • the upload cap;
  • completion checking the landed size;
  • a reported size over the limit refused, at the boundary;
  • an abandoned one-URL write being harmless.
• aws/s3_parts_write_test.py, on moto:
  • one presigned URL per part with the right part numbers;
  • completion making the object with its content type;
  • missing, over-reported and under-reported parts refused and the upload discarded;
  • leaving the block aborting the upload;
  • receivers without parts getting one bounded PUT;
  • start and finish refusals translated, and S3 out of reach at the start, the listing and the completion;
  • an upload over the grant's limit refused and discarded;
  • an abort that cannot reach S3 leaving the original UploadFailedError raised;
  • regional and dotted-name grant hosts, and path-style addressing kept for configured endpoints (S3-compatible and PrivateLink) and for a profile that sets it.
• Conformance: GRANT_CASES gains a completed object write, a misreported one refused, and an abandoned one leaving nothing. They run against the local grant server here; the S3 parts path was also checked by hand on a real bucket.
• Wrappers: the routing wrapper routes begin_write and fences it in an @local run, and the staged store refuses it. The @local write-fence test lists begin_write among the writes a run must not make to the configured store.

Mutants, all killed:

• no part check;
• no abort on a failed complete;
• the default ignoring parts;
• the default skipping the size check;
• no content type on the upload;
• the global endpoint;
• an untranslated start failure;
• routing to the configured store;
• the fence bypassed;
• the staged store forwarding;
• no limit check, and the limit off by one;
• an abort that raises botocore errors;
• each of the start, listing and completion catches narrowed back to ClientError.

The unit tier passes.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge under its documented single-PUT limits.

Summary

Adds PendingWrite, a default single-PUT implementation, and S3 multipart uploads with checks before completion. The wrappers route or refuse the new operation.

• Callers can accept a remote object write only after checking its reported size.
• S3 writes can arrive in parts and become objects only after the parts are checked.
• AWS S3 grants use URLs addressed to the bucket’s region.
• New pending writes follow the existing store routing and staging boundaries.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[begin_write] --> B[Return upload grant]
    B --> C[Receiver uploads bytes]
    C --> D[complete]
    D --> E{Reported size within limit?}
    E -->|No| F[Abort write]
    E -->|Yes| G[Check stored size or parts]
    G -->|Mismatch or error| F
    G -->|Match| H[Accept object]
    H --> I[S3 completes multipart upload]

Loading

_{Reviews (8) · Last reviewed commit: "fix(store): keep a configured endpoint's..." · Reviewed by Greptile}

[earakely-scale]

Merged the updated #149 in; no conflicts of its own.

Two fixes in the follow-up commits:

• Regional addressing now applies only on AWS's own S3 hosts with no addressing style configured. Applied to every endpoint, it sent an S3-compatible store's requests to bucket.minio:9000 instead of minio:9000/bucket. It also overrode a PrivateLink endpoint and an AWS profile's s3.addressing_style.
• The @local write-fence test lists begin_write.