feat: add Pod-scoped S3 gateway commands

Cargo.toml

@@ -10,6 +10,10 @@ path = "src/main.rs"
 [dependencies]
 anyhow = "1.0.102"
 axum = "0.8.8"
+aws-sdk-s3 = { version = "=1.132.0", default-features = false, features = ["rt-tokio"] }
+aws-smithy-http = "0.63.6"
+aws-smithy-runtime-api = { version = "1.12.0", features = ["client", "http-1x"] }
+aws-smithy-types = "1.4.7"
 clap = { version = "4.6.1", features = ["derive"] }
 dirs = "6.0.0"
 flate2 = "1.1.2"

README.md

@@ -5,7 +5,7 @@
 
 # Brainpod CLI
 
-A non-interactive CLI for managing Brainpod pods, images, blueprints, revisions, resources, deployments, database tunnels, and events. Its default output is deterministic line-oriented text suitable for LLMs and shell tools. Add `--json` to receive machine-readable JSON; login, database tunnels, and event watches use NDJSON.
+A non-interactive CLI for managing Brainpod pods, images, blueprints, revisions, resources, deployments, object storage, database tunnels, and events. Its default output is deterministic line-oriented text suitable for LLMs and shell tools. Add `--json` to receive machine-readable JSON; login, database tunnels, and event watches use NDJSON.
 
 The CLI builds application images locally from an existing Dockerfile or with Railpack, then pushes them directly to the selected pod's private Brainpod registry namespace. Image builds probe the API's cluster architectures, prefer amd64 and then arm64, and store the selected default architecture in the configuration. Use `--platform linux/arm64` for a one-off override.
 
@@ -373,3 +373,59 @@ Create and deploy:
 brainpod resource create --file resources.json --json
 brainpod deploy --summary "Configure application resources" --json
 ```
+
+## S3 gateway
+
+Use the normal Brainpod API token and selected Pod (`--pod`, `BRAINPOD_POD`, or configured default). The key needs explicit `s3:list`, `s3:read`, and/or `s3:write` policies for that Pod; write includes replacement and deletion. Existing keys are not expanded automatically. Buckets must already be provisioned through the resource workflow.
+
+```sh
+brainpod --pod my-pod s3 ls
+brainpod --pod my-pod s3 ls s3://uploads/images/
+brainpod --pod my-pod s3 cp ./photo.jpg s3://uploads/images/photo.jpg
+brainpod --pod my-pod s3 upload ./photo.jpg s3://uploads/photo.jpg
+brainpod --pod my-pod s3 cp s3://uploads/photo.jpg ./downloaded.jpg
+brainpod --pod my-pod s3 cp s3://uploads/photo.jpg s3://archive/photo.jpg
+brainpod --pod my-pod s3 mv s3://uploads/photo.jpg s3://uploads/renamed.jpg
+brainpod --pod my-pod s3 rm s3://uploads/renamed.jpg
+```
+
+Remote paths are `s3://<logical-bucket>/<literal-key>` within the selected Pod, not provider bucket names or URLs. Keys are literal UTF-8: quote shell metacharacters, and do not pre-percent-encode spaces, plus signs, or percent signs. Repeated and trailing slashes are preserved; dot-segment object keys are rejected to prevent URL normalization. Listings show one level of objects/prefixes and automatically fetch every page. `--json` returns an `items` array or a transfer/deletion result.
+
+Transfers require explicit file/key destinations (no implicit basename or directory expansion). Local destinations must not already exist; downloads are saved atomically after success. Remote destinations replace existing keys. All `cp`/`mv` transfers, including downloads and gateway-to-gateway copies, and file-based `upload` are limited to **25 MiB** and buffered in bounded memory. Explicit streaming upload/download modes are described below. Gateway copies download then upload, not native CopyObject. `mv` deletes its source only after a successful copy; a deletion failure reports that both copies may remain. Moves are not atomic: avoid concurrent changes to the source. `rm` deletes one exact key, including a trailing-slash marker, never prefix contents.
+
+The default gateway origin is `https://s3.brainpod.io`; the CLI prepends the selected Pod name to its hostname. Override the base origin with `--s3-endpoint`, `BRAINPOD_S3_ENDPOINT`, or `brainpod config set s3-endpoint https://s3.example.test` (in that precedence order). For local development, use `http://s3.brainpod.localhost` with working wildcard DNS. HTTP is unencrypted and should only be used locally. The API endpoint does not change the S3 origin.
+
+Bucket/object listings, PUT/delete, and multipart initiation, parts, completion, and abort use typed `aws-sdk-s3` operations and SDK XML serialization/deserialization. A gateway transport keeps logical buckets path-style beneath the Pod hostname, uses unsigned SDK auth with Bearer authentication added only at the gateway, removes the SDK's `x-id` query parameter and nonsemantic `x-amz-user-agent` telemetry header, disables optional checksums, and requires buffered, non-`aws-chunked` request bodies. Requests are not automatically retried. Response byte limits remain in place; XML parsing and service-error detection are delegated to the SDK.
+
+Downloads explicitly accept the gateway's 307 redirect and use a separate unauthenticated HTTPS request to the short-lived provider URL. Both download requests remain implemented with `reqwest`, not SDK GetObject, to enforce this redirect contract. Brainpod Bearer credentials are never sent to the provider; further redirects are rejected. Presigned URLs are not printed. There is no AWS SigV4 client authentication, public HEAD, recursive/sync transfer or deletion, bucket administration, ranged reads, or metadata preservation for `cp`/`mv`. Multipart is reserved for explicit stdin streaming uploads. Transfer errors return a nonzero exit status; interrupted uploads can have an uncertain remote outcome, so inspect the key before retrying.
+
+### Streaming transfers
+
+`upload -` uses sequential **8 MiB** multipart parts (up to 10,000 parts, about 78 GiB), rather than buffering the entire object. It requires a multipart-capable endpoint and `s3:write`.
+
+EOF alone cannot prove that an upstream producer succeeded, so stdin uploads require `--require-success-file PATH`. After EOF, completion is allowed only when that regular file contains exactly `commit` followed by one newline. Use a fresh private directory for each attempt and write the marker only after the entire producer pipeline succeeds. Keep the producer subshell's stdout open until after writing the marker, so EOF cannot race its creation.
+
+```bash
+set -euo pipefail
+run_dir=$(mktemp -d)
+trap 'rm -rf "$run_dir"' EXIT
+(
+  set -euo pipefail
+  cat ./large-object.bin
+  printf 'commit\n' > "$run_dir/success"
+) | brainpod s3 upload - s3://uploads/large-object.bin \
+      --require-success-file "$run_dir/success" \
+      --metadata project=example
+```
+
+Generic object metadata is accepted on multipart initiation using repeated `--metadata KEY=VALUE`. Keys must be nonempty valid HTTP header names; values must be nonempty printable ASCII. Metadata names and values together may total at most 2048 bytes, and duplicate keys are rejected case-insensitively. Endpoint-specific metadata policies are enforced by the server.
+
+Missing or invalid markers, input errors, rejected parts, completion errors reported by the SDK (including HTTP 200 `<Error>`), SIGINT, and SIGTERM fail the upload and attempt to abort it. Completion/network failures can have an uncertain outcome; inspect the exact key before retrying. Abrupt process termination or failed aborts can leave incomplete uploads requiring endpoint cleanup.
+
+Stream an object to stdout without the small-copy limit:
+
+```sh
+brainpod s3 download s3://uploads/large-object.bin - > ./large-object.bin
+```
+
+`download` currently requires destination `-`. Stdout contains only raw object bytes, even with `--json`; diagnostics stay on stderr. Provider requests use unauthenticated HTTPS with redirects disabled. An interrupted transfer can emit a partial object, so check the complete pipeline's exit status; early consumer closure follows normal Unix broken-pipe behavior. Network inactivity is bounded, but streaming downloads have no whole-object timeout or size cap. Input/output use bounded worker handoffs so cancellation does not wait for an unresponsive pipe reader/writer.

src/cmd/mod.rs

@@ -33,6 +33,8 @@ pub enum Command {
     Blueprint(BlueprintArgs),
     /// Manage container images
     Image(ImageArgs),
+    /// Transfer and list objects through the Pod-scoped S3 gateway
+    S3(crate::s3::S3Args),
     /// Inspect pod revisions
     Revision(RevisionArgs),
     /// Create a local TCP tunnel to a deployed database
@@ -95,6 +97,7 @@ enum ConfigKey {
     Endpoint,
     ControlPlaneEndpoint,
     RegistryEndpoint,
+    S3Endpoint,
     ApiToken,
     Pod,
     Architecture,
@@ -497,7 +500,11 @@ pub fn needs_api_token(command: &Command) -> bool {
 pub fn needs_client(command: &Command) -> bool {
     !matches!(
         command,
-        Command::Describe(_) | Command::Agent(_) | Command::Login(_) | Command::Config(_)
+        Command::S3(_)
+            | Command::Describe(_)
+            | Command::Agent(_)
+            | Command::Login(_)
+            | Command::Config(_)
     )
 }
 
@@ -509,6 +516,7 @@ pub struct CommandContext<'a> {
     pub pod: Option<&'a str>,
     pub api_token: Option<&'a str>,
     pub registry_endpoint: &'a str,
+    pub s3_endpoint: &'a str,
     pub config: &'a mut Config,
     pub config_path: &'a Path,
     pub show_progress: bool,
@@ -524,6 +532,7 @@ pub async fn handle(command: Command, context: CommandContext<'_>) -> Result<Com
         pod,
         api_token,
         registry_endpoint,
+        s3_endpoint,
         config,
         config_path,
         show_progress,
@@ -550,6 +559,15 @@ pub async fn handle(command: Command, context: CommandContext<'_>) -> Result<Com
             View::Login,
         )),
         Command::Config(args) => handle_config(args, config, config_path),
+        Command::S3(args) => {
+            crate::s3::handle(
+                args,
+                s3_endpoint,
+                pod_required(pod)?,
+                api_token.ok_or_else(|| anyhow!("API token is required"))?,
+            )
+            .await
+        }
         Command::Whoami => Ok(CommandOutput::new(
             client_required(client)?.get(&["v1", "me"], &[]).await?,
             View::Whoami,
@@ -652,6 +670,7 @@ fn handle_config(args: ConfigArgs, config: &mut Config, path: &Path) -> Result<C
                 "endpoint": config.endpoint,
                 "controlPlaneEndpoint": config.control_plane_endpoint,
                 "registryEndpoint": config.registry_endpoint,
+                "s3Endpoint": config.s3_endpoint,
                 "apiTokenConfigured": config.api_token.is_some(),
                 "pod": config.pod,
                 "architecture": config.architecture,
@@ -675,6 +694,10 @@ fn handle_config(args: ConfigArgs, config: &mut Config, path: &Path) -> Result<C
                     config.control_plane_endpoint = Some(value);
                     "controlPlaneEndpoint"
                 }
+                ConfigKey::S3Endpoint => {
+                    config.s3_endpoint = Some(value);
+                    "s3Endpoint"
+                }
                 ConfigKey::RegistryEndpoint => {
                     config.registry_endpoint = Some(value);
                     "registryEndpoint"
@@ -708,6 +731,10 @@ fn handle_config(args: ConfigArgs, config: &mut Config, path: &Path) -> Result<C
                     config.control_plane_endpoint = None;
                     "controlPlaneEndpoint"
                 }
+                ConfigKey::S3Endpoint => {
+                    config.s3_endpoint = None;
+                    "s3Endpoint"
+                }
                 ConfigKey::RegistryEndpoint => {
                     config.registry_endpoint = None;
                     "registryEndpoint"

src/config.rs

@@ -7,13 +7,15 @@ use serde::{Deserialize, Serialize};
 pub const DEFAULT_ENDPOINT: &str = "https://api.brainpod.io";
 pub const DEFAULT_CONTROL_PLANE_ENDPOINT: &str = "https://control-plane.brainpod.io";
 pub const DEFAULT_DASHBOARD_ENDPOINT: &str = "https://console.brainpod.io";
+pub const DEFAULT_S3_ENDPOINT: &str = "https://s3.brainpod.io";
 pub const DEFAULT_REGISTRY_ENDPOINT: &str = "https://registry.brainpod.io";
 
 #[derive(Clone, Debug, Default, Deserialize, Serialize)]
 #[serde(deny_unknown_fields)]
 pub struct Config {
     pub endpoint: Option<String>,
     pub control_plane_endpoint: Option<String>,
+    pub s3_endpoint: Option<String>,
     pub registry_endpoint: Option<String>,
     pub api_token: Option<String>,
     pub pod: Option<String>,

src/describe.rs

@@ -38,6 +38,7 @@ pub fn generate(mut root: Command, path: &[String]) -> Result<Value> {
             "eventWatchJson": "Event watches emit one JSON value per line as NDJSON.",
             "tunnelJson": "Tunnels emit listening, optional credentials, and closed events as NDJSON.",
             "waitProgress": "Interactive waits report unhealthy-to-healthy transitions on stderr; progress is suppressed when stderr is redirected or --json is used.",
+            "s3Download": "s3 download <source> - writes raw bytes only, even with --json; errors remain on stderr.",
             "errors": "Errors are written to stderr and return a non-zero exit code; --json also makes errors JSON."
         },
         "guidance": [
@@ -251,6 +252,7 @@ fn requirements(path: &[&str]) -> (Option<bool>, Option<bool>) {
         ["blueprint", _] | ["whoami"] | ["cluster"] | ["cluster", _] | ["pod"] | ["pod", _] => {
             (Some(true), Some(false))
         }
+        ["s3"] | ["s3", _] => (Some(true), Some(true)),
         ["image"]
         | ["image", _]
         | ["revision"]
@@ -288,6 +290,18 @@ fn effect(path: &[&str]) -> (&'static str, &'static str) {
         | ["resource", "list"]
         | ["resource", "get"]
         | ["resource", "variables"] => ("read", "Reads remote state without changing it."),
+        ["s3", "ls"] => (
+            "read",
+            "Lists gateway buckets or one level of object prefixes.",
+        ),
+        ["s3", "download"] => (
+            "read-stream",
+            "Streams provider object bytes to stdout without a text or JSON summary.",
+        ),
+        ["s3", _] => (
+            "local-and-remote-write",
+            "Transfers one bounded object or deletes one exact key; moves delete only after a successful copy.",
+        ),
         ["events"] => (
             "read-or-stream",
             "Reads or continuously streams remote events.",

src/main.rs

@@ -15,13 +15,14 @@ mod describe;
 mod image;
 mod openapi;
 mod output;
+mod s3;
 mod tunnel;
 
 use client::{ApiError, Client};
 use cmd::Command;
 use config::{
     Config, DEFAULT_CONTROL_PLANE_ENDPOINT, DEFAULT_DASHBOARD_ENDPOINT, DEFAULT_ENDPOINT,
-    DEFAULT_REGISTRY_ENDPOINT,
+    DEFAULT_REGISTRY_ENDPOINT, DEFAULT_S3_ENDPOINT,
 };
 
 const UPGRADE_URL: &str = "https://console.brainpod.io/onboarding?upgrade=1";
@@ -30,7 +31,7 @@ const UPGRADE_URL: &str = "https://console.brainpod.io/onboarding?upgrade=1";
 #[command(
     name = "brainpod",
     version,
-    about = "Manage Brainpod deployments, images, resources, and database tunnels",
+    about = "Manage Brainpod deployments, images, resources, object storage, and database tunnels",
     after_help = "For machine-readable command metadata, run `brainpod describe --json`."
 )]
 pub(crate) struct Opts {
@@ -54,6 +55,10 @@ pub(crate) struct Opts {
     #[arg(long, global = true)]
     registry_endpoint: Option<String>,
 
+    /// S3 gateway base origin; the selected Pod is prepended to its hostname
+    #[arg(long, global = true)]
+    s3_endpoint: Option<String>,
+
     /// Default pod for pod-scoped commands
     #[arg(long, global = true)]
     pod: Option<String>,
@@ -152,6 +157,11 @@ async fn run(opts: Opts) -> Result<output::CommandOutput> {
         .or_else(|| environment("BRAINPOD_REGISTRY_ENDPOINT"))
         .or_else(|| config.registry_endpoint.clone())
         .unwrap_or_else(|| DEFAULT_REGISTRY_ENDPOINT.to_owned());
+    let s3_endpoint = opts
+        .s3_endpoint
+        .or_else(|| environment("BRAINPOD_S3_ENDPOINT"))
+        .or_else(|| config.s3_endpoint.clone())
+        .unwrap_or_else(|| DEFAULT_S3_ENDPOINT.to_owned());
     let pod = opts
         .pod
         .or_else(|| environment("BRAINPOD_POD"))
@@ -189,6 +199,7 @@ async fn run(opts: Opts) -> Result<output::CommandOutput> {
             pod: pod.as_deref(),
             api_token: api_token.as_deref(),
             registry_endpoint: &registry_endpoint,
+            s3_endpoint: &s3_endpoint,
             config: &mut config,
             config_path: &config_path,
             show_progress,

src/output.rs

@@ -6,6 +6,7 @@ use serde_json::{Value, json};
 use crate::client::{EventStreamMessage, EventWatch};
 
 pub enum CommandOutput {
+    Silent,
     Buffered { value: Value, view: View },
     EventWatch(Box<EventWatch>),
 }
@@ -22,6 +23,7 @@ impl CommandOutput {
 
 #[derive(Clone, Copy)]
 pub enum View {
+    S3,
     Describe,
     ResourceSchema,
     AgentStart,
@@ -60,6 +62,7 @@ pub enum View {
 
 pub async fn write(output: CommandOutput, json: bool) -> Result<()> {
     match output {
+        CommandOutput::Silent => Ok(()),
         CommandOutput::Buffered { value, view } => write_buffered(&value, view, json),
         CommandOutput::EventWatch(watch) => write_event_watch(*watch, json).await,
     }
@@ -159,6 +162,7 @@ fn stream_error(data: &str) -> anyhow::Error {
 
 fn render(value: &Value, view: View, color: bool) -> Vec<String> {
     match view {
+        View::S3 => crate::s3::render(value),
         View::Describe => render_describe(value),
         View::ResourceSchema => render_resource_schema(value),
         View::AgentStart => crate::agent::render_start(value),
@@ -514,6 +518,7 @@ fn render_config_show(value: &Value) -> Vec<String> {
             field(value, "controlPlaneEndpoint")
         ),
         format!("Registry endpoint: {}", field(value, "registryEndpoint")),
+        format!("S3 endpoint: {}", field(value, "s3Endpoint")),
         format!(
             "API token configured: {}",
             yes_no(value_at(value, "apiTokenConfigured"))