S3 Manager

A Web GUI written in Go to manage S3 buckets from any provider.

Features

• Manage several S3 accounts side by side and switch between them
• List, create and delete buckets
• View and edit a bucket's policy, including making a bucket publicly readable
• List a bucket's objects with search, sorting and pagination
• Upload single objects or whole folders to a bucket
• Download an object, or several selected ones as a ZIP archive
• Open an object in the browser
• Delete a single object or several selected ones
• Create a time-limited download link for an object
• Check whether an object is publicly accessible and copy its public link
• Show object metadata (including user metadata) and object versions

Usage

Configuration

The application is configured with environment variables.

S3 instances

Every S3 account the app should manage is configured with a numbered set of variables, starting at S3_1_. The app stops looking at the first number that has no NAME, so the numbering must not have gaps. Each instance appears in the app under its NAME and is reachable under /<NAME>/buckets (or /<NUMBER>/buckets), so pick names that work in a URL:

S3_1_NAME=production
S3_1_ENDPOINT=s3.amazonaws.com
S3_1_ACCESS_KEY_ID=XXX
S3_1_SECRET_ACCESS_KEY=xxx

S3_2_NAME=backups
S3_2_ENDPOINT=minio.example.com:9000
S3_2_ACCESS_KEY_ID=YYY
S3_2_SECRET_ACCESS_KEY=yyy

A single instance may also be configured without a number (it is then named Default), which is how earlier versions of the app were configured:

ENDPOINT=s3.amazonaws.com
ACCESS_KEY_ID=XXX
SECRET_ACCESS_KEY=xxx

At least one instance must be configured. Each instance reads these variables:

• NAME: The name the instance is shown and addressed under (required in the numbered form)
• ENDPOINT: The endpoint of your S3 server (defaults to s3.amazonaws.com)
• REGION: The region of your S3 server (defaults to unset)
• ACCESS_KEY_ID, SECRET_ACCESS_KEY: Your S3 key pair (required unless USE_IAM is true or SIGNATURE_TYPE is Anonymous)
• USE_IAM: Use IAM role instead of key pair (defaults to false)
• IAM_ENDPOINT: Endpoint for IAM role retrieving (defaults to unset, which works for AWS)
• USE_SSL: Whether your S3 server uses SSL or not (defaults to true)
• SKIP_SSL_VERIFICATION: Whether the HTTP client should skip SSL verification (defaults to false)
• SIGNATURE_TYPE: The signature type to be used (defaults to V4; valid values are V2, V4, V4Streaming, Anonymous). Anonymous sends unsigned requests, which is how public buckets are browsed
• BUCKET_LOOKUP: How buckets are addressed in requests (defaults to Auto; valid values are Auto, DNS, Path). DNS uses bucket.endpoint, Path uses endpoint/bucket and Auto picks per provider. Set it to DNS if your provider answers with Virtual host domain is required while accessing a specific bucket
• PUBLIC_URL: A template for the public links of objects, for when they are served from a CDN or a custom domain (defaults to unset, which links to the object on ENDPOINT following BUCKET_LOOKUP). {key} is replaced by the object key and {bucket} by the bucket name, for example https://files.example.com/{bucket}/{key}; {key} is required
• BUCKETS: A comma-separated list of bucket names to show in addition to the ones the instance lists itself (defaults to unset), for example public buckets owned by someone else. They are shown even if the instance refuses to list its buckets, as it does for anonymous access

Application

These variables apply to the whole app and are never prefixed:

• PORT: The port the app should listen on (defaults to 8080)
• ALLOW_DELETE: Enable deleting objects, folders and empty buckets (defaults to true)
• LIST_RECURSIVE: List all objects in buckets recursively (defaults to false)
• SHOW_VERSIONS: Show all object versions in bucket view and enable version-specific downloads (defaults to false; bucket must have versioning enabled)
• SHOW_METADATA: Show the object metadata action and enable the metadata endpoint (defaults to true)
• TZ: IANA timezone used when displaying object Last Modified times (defaults to UTC; for example Europe/Berlin)
• BUCKET_NAME: Show only this bucket, or these comma-separated buckets, in the bucket list of every instance (defaults to unset, showing all buckets). It only filters the list: other buckets can still be opened by name or URL, so it is not an access restriction
• SSE_TYPE: The server side encryption applied to uploaded objects (defaults to unset, which leaves it off; valid values are SSE, KMS and SSE-C)
• SSE_KEY: The key for KMS (a key ID) or SSE-C (exactly 32 bytes). With SSE-C the key is also sent when reading objects, since they can't be read without it; download links and public links don't carry it and so don't work for such objects
• TIMEOUT: The read and write timeout in seconds (defaults to 600)
• ROOT_URL: A root URL prefix if running behind a reverse proxy (defaults to unset)

Browsing large buckets

The default bucket view (sorted by name ascending, unsearched) asks S3 for one page of objects at a time, so it is fast for any bucket size. Since S3 never reports how much it did not return, that view has no total object count and no last page to jump to. Any other view (another sort, a search, All items per page or SHOW_VERSIONS) loads the whole location instead and stops after 10,000 objects, saying so on the page. Search or open a folder to reach the rest.

Browsing public buckets

Public buckets, such as the datasets of the Registry of Open Data on AWS, can be browsed without an account by configuring an instance for anonymous access and naming the buckets to show:

SIGNATURE_TYPE=Anonymous
REGION=us-east-1
BUCKETS=noaa-ghcn-pds

Anonymous requests cannot look up the region of a bucket, so REGION has to match it; a bucket in another region reports the region it is in. Any other bucket can be opened by name with the folder button of the bucket list, even if it isn't configured. Public buckets usually only allow reading, so uploading and deleting fails on them.

Deployment

The container image is available on Docker Hub:

podman run -p 8080:8080 -e 'ENDPOINT=s3.amazonaws.com' -e 'ACCESS_KEY_ID=XXX' -e 'SECRET_ACCESS_KEY=xxx' docker.io/cloudlena/s3manager

For Kubernetes, there is a Helm chart.

Running behind a reverse proxy

Everyone who can reach an S3 Manager can use all of its S3 instances, so to give each team only its own accounts, run one S3 Manager per team behind a reverse proxy that authenticates each location, and set each one's ROOT_URL to the location it is served under.

If the nginx configuration block looks like:

location /teamx/ {
    proxy_pass http://s3manager-teamx:8080/;
    auth_basic "teamx";
    auth_basic_user_file /conf/teamx-htpasswd;
}
location /teamy/ {
    proxy_pass http://s3manager-teamy:8080/;
    <other nginx settings>
}

Then the instance behind the s3manager-teamx service has ROOT_URL=teamx and the instance behind s3manager-teamy has ROOT_URL=teamy.

Development

podman compose up  # two RustFS instances and an S3 Manager for both at http://localhost:8080
make run
make test
make lint
make build-image

Star History