FileForge

A pluggable cloud-storage bridge built with Django REST Framework. External apps register through cryptographic API keys and route file uploads to any registered provider (Google Drive, Cloudinary, or a custom adapter) via a unified REST API.

Key Features

Feature	Description
Two-surface auth	JWT for developer management (Apps, API keys); API keys (`ffk_...`) for server-to-server storage calls
Pluggable providers	Google Drive and Cloudinary built-in; add any backend by subclassing `BaseStorageProvider`
Async & sync uploads	Queue small files for background upload (Django-Q2) or block and get the result in one round-trip
Direct upload	FileForge issues a signed URL so large files go straight from the client to the provider — never touching your server
Range-aware streaming	`GET /api/files/{id}/stream/` proxies files in 5 MB chunks and honours `Range` headers for audio/video seeking
Folder support	Organise files into hierarchical folders (Cloudinary and Google Drive)
Collection support	Group related assets into collections (Cloudinary)
Secret masking	Credential API responses mask sensitive fields; PATCH uses a `"***"` sentinel so partial updates never wipe stored secrets
No Redis needed	Django-Q2 uses the ORM as its broker — works out of the box with SQLite or Postgres

Quick Start

# 1. Clone and install dependencies
git clone https://github.com/your-org/fileforge.git
cd fileforge
pip install -r requirements.txt

# 2. Run migrations
python manage.py migrate

# 3. Start the service (migrates, starts Q2 worker, dev server on :5000)
./run.sh

# 4. Verify
curl http://localhost:5000/api/health/

{ "status": "ok", "providers": ["cloudinary", "google_drive"] }

First integration (4 steps)

# Register a developer account
curl -X POST http://localhost:5000/auth/register/ \
  -H "Content-Type: application/json" \
  -d '{"email":"you@example.com","password":"StrongPass1!","password_confirm":"StrongPass1!"}'

# Get a JWT token
TOKEN=$(curl -s -X POST http://localhost:5000/auth/token/ \
  -H "Content-Type: application/json" \
  -d '{"email":"you@example.com","password":"StrongPass1!"}' | python -c "import sys,json; print(json.load(sys.stdin)['access'])")

# Create an App and get its ID
APP_ID=$(curl -s -X POST http://localhost:5000/auth/apps/ \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"My App"}' | python -c "import sys,json; print(json.load(sys.stdin)['id'])")

# Create an API key — save raw_key immediately, it is shown only once
curl -X POST http://localhost:5000/auth/apps/$APP_ID/keys/ \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"production"}'

Use the returned raw_key (ffk_...) as Authorization: Bearer <raw_key> on all /api/ storage calls.

Project Structure

fileforge/
├── fileforge/                  # Django project (settings, URLs, WSGI/ASGI)
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── fileforge_auth/             # Management / control-plane app
│   ├── models.py               # DeveloperUser, App, ApiKey
│   ├── authentication.py       # ApiKeyAuthentication backend
│   ├── permissions.py          # IsAuthenticatedApp, IsAuthenticatedDeveloper
│   ├── serializers/            # Registration, profile, app, key serializers
│   ├── views.py                # Register, me, apps, keys endpoints
│   └── urls.py                 # Mounted at /auth/
├── storage/                    # Storage / data-plane app
│   ├── models.py               # File, StorageCredential
│   ├── providers/
│   │   ├── base.py             # BaseStorageProvider contract
│   │   ├── registry.py         # Name → class mapping (plugin-ready)
│   │   ├── google_drive.py     # Google Drive adapter
│   │   └── cloudinary_provider.py
│   ├── services/
│   │   └── storage_manager.py  # Single orchestration entry point
│   ├── tasks/
│   │   └── file_tasks.py       # process_file_upload, cleanup_temp_files
│   ├── utils/                  # Temp storage, upload-strategy helper
│   ├── serializers.py
│   ├── views.py                # DRF views mounted at /api/
│   └── urls.py
├── manage.py
├── run.sh                      # Dev launcher: migrate → qcluster → runserver
└── requirements.txt

Authentication

FileForge exposes two independent API surfaces:

Surface	Base path	Auth method	Used by
Management API	`/auth/`	JWT (`Authorization: Bearer <access_token>`)	Developers — creating Apps, issuing API keys
Storage API	`/api/`	API Key (`Authorization: Bearer ffk_...`)	Your backend servers — uploading and managing files

Important: Never expose API keys to browsers or end users. All /api/ calls should be made server-side.

Getting a JWT token (management)

POST /auth/token/ HTTP/1.1
Content-Type: application/json

{"email": "you@example.com", "password": "..."}

Using an API key (storage)

GET /api/files/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...

Architecture

Layer	Path	Notes
Auth models	`fileforge_auth/models.py`	`DeveloperUser`, `App`, `ApiKey` — identity and ownership
Auth backend	`fileforge_auth/authentication.py`	Validates `Bearer ffk_...` tokens
Permissions	`fileforge_auth/permissions.py`	`IsAuthenticatedApp` (API key), `IsAuthenticatedDeveloper` (JWT + `DeveloperUser` instance check)
Auth views	`fileforge_auth/views.py`	Register, token, me, app CRUD, key management
Provider interface	`storage/providers/base.py`	`BaseStorageProvider` — strict contract all providers must satisfy
Provider registry	`storage/providers/registry.py`	Name → class mapping; plugin-ready
Built-in providers	`storage/providers/`	Google Drive (OAuth2 + service account) and Cloudinary
Service layer	`storage/services/storage_manager.py`	Single entry point; views never call providers directly
Async tasks	`storage/tasks/file_tasks.py`	`process_file_upload`, `cleanup_temp_files`
Utilities	`storage/utils/`	Disk temp storage; upload-strategy threshold helper
Storage models	`storage/models.py`	`File`, `StorageCredential`
Storage API	`storage/views.py`, `storage/urls.py`	DRF views mounted at `/api/` — require API key auth

Endpoint Overview

Management API (`/auth/`) — JWT required

Method	Endpoint	Description
`POST`	`/auth/register/`	Create developer account (public)
`POST`	`/auth/token/`	Obtain JWT pair (public)
`POST`	`/auth/token/refresh/`	Refresh JWT pair (public)
`GET/PATCH`	`/auth/me/`	Developer profile
`POST`	`/auth/me/change-password/`	Change password
`GET/POST`	`/auth/apps/`	List / create Apps
`GET/PATCH/DELETE`	`/auth/apps/{id}/`	App detail
`GET/POST`	`/auth/apps/{id}/keys/`	List / create API keys
`POST`	`/auth/apps/{id}/keys/{key_id}/revoke/`	Revoke an API key
`GET/POST`	`/auth/apps/{id}/providers/`	List / upsert provider credentials via JWT
`GET/PATCH/DELETE`	`/auth/apps/{id}/providers/{provider}/`	Provider credential detail

Storage API (`/api/`) — API key required

Method	Endpoint	Description
`GET`	`/api/health/`	Liveness probe (public)
`GET`	`/api/providers/`	Registered providers and capabilities
`GET/POST`	`/api/credentials/`	List / upsert per-app credentials
`GET/PATCH/DELETE`	`/api/credentials/{id}/`	Credential detail
`GET/POST`	`/api/files/`	List files / upload (async or sync mode)
`GET/PATCH/DELETE`	`/api/files/{id}/`	File detail, rename, delete
`GET`	`/api/files/{id}/stream/`	Stream file bytes (Range-aware)
`POST`	`/api/files/direct-upload/`	Initiate direct upload (returns signed URL)
`POST`	`/api/files/direct-upload/complete/`	Finalize direct upload
`GET/POST`	`/api/folders/`	List / create folders
`DELETE`	`/api/folders/{path}/`	Delete a folder
`GET/POST`	`/api/collections/`	List / create collections
`GET`	`/api/collections/{id}/`	Collection details
`POST/DELETE`	`/api/collections/{id}/assets/`	Add / remove assets from collection

Sample Requests & Responses

`GET /` — Service description

GET / HTTP/1.1
Host: localhost:5000

{
  "service": "FileForge",
  "description": "Pluggable cloud storage bridge",
  "endpoints": {
    "register": "/auth/register/",
    "token_obtain": "/auth/token/",
    "token_refresh": "/auth/token/refresh/",
    "me": "/auth/me/",
    "apps": "/auth/apps/",
    "files_list_create": "/api/files/",
    "file_detail": "/api/files/{id}/",
    "direct_upload_init": "/api/files/direct-upload/",
    "direct_upload_complete": "/api/files/direct-upload/complete/",
    "providers": "/api/providers/",
    "credentials": "/api/credentials/",
    "health": "/api/health/"
  }
}

`GET /api/health/` — Liveness probe

GET /api/health/ HTTP/1.1
Host: localhost:5000

{ "status": "ok", "providers": ["cloudinary", "google_drive"] }

`GET /api/providers/` — List providers and capabilities

GET /api/providers/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...

{
  "providers": [
    { "name": "cloudinary", "supports_direct_upload": true, "supports_streaming": false, "supports_folders": true, "supports_collections": true },
    { "name": "google_drive", "supports_direct_upload": true, "supports_streaming": true, "supports_folders": true, "supports_collections": false }
  ]
}

`GET /api/credentials/` — List credentials

GET /api/credentials/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...

[
  {
    "id": 1,
    "owner": "app_xk3m9pq7rz1c",
    "provider": "cloudinary",
    "credentials": {
      "cloud_name": "my-cloud",
      "api_key": "123456789012345",
      "api_secret": "••••••••••••••••••••••••"
    },
    "is_default": true,
    "created_at": "2026-04-01T10:00:00Z",
    "updated_at": "2026-04-01T10:00:00Z"
  }
]

`POST /api/credentials/` — Create or update credentials

POST /api/credentials/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...
Content-Type: application/json

{
  "provider": "cloudinary",
  "credentials": {
    "cloud_name": "my-cloud",
    "api_key": "123456789012345",
    "api_secret": "my-api-secret"
  },
  "is_default": true
}

HTTP/1.1 201 Created

{
  "id": 1,
  "owner": "app_xk3m9pq7rz1c",
  "provider": "cloudinary",
  "credentials": {
    "cloud_name": "my-cloud",
    "api_key": "123456789012345",
    "api_secret": "••••••••••••••••••••••••"
  },
  "is_default": true,
  "created_at": "2026-04-01T10:00:00Z",
  "updated_at": "2026-04-01T10:00:00Z"
}

Sensitive credential fields are masked in all responses. Send "***" for any field you want to leave unchanged on PATCH.

`POST /api/files/` — Upload a file

Files at or below the provider's sync threshold (default 5 MB) can use mode: "async" (default — returns 202 immediately, poll for result) or mode: "sync" (blocks until the upload completes, returns 200).

Basic upload:

POST /api/files/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...
Content-Type: multipart/form-data; boundary=----Boundary

------Boundary
Content-Disposition: form-data; name="file"; filename="report.pdf"
Content-Type: application/pdf

<binary file data>
------Boundary
Content-Disposition: form-data; name="provider"

cloudinary
------Boundary--

Upload to a specific folder:

POST /api/files/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...
Content-Type: multipart/form-data; boundary=----Boundary

------Boundary
Content-Disposition: form-data; name="file"; filename="shoe-photo.jpg"
Content-Type: image/jpeg

<binary file data>
------Boundary
Content-Disposition: form-data; name="provider"

cloudinary
------Boundary
Content-Disposition: form-data; name="folder"

products/shoes
------Boundary--

Upload and add to a collection:

POST /api/files/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...
Content-Type: multipart/form-data; boundary=----Boundary

------Boundary
Content-Disposition: form-data; name="file"; filename="banner.png"
Content-Type: image/png

<binary file data>
------Boundary
Content-Disposition: form-data; name="provider"

cloudinary
------Boundary
Content-Disposition: form-data; name="collection_id"

12345
------Boundary--

HTTP/1.1 202 Accepted

{
  "id": 43,
  "name": "report.pdf",
  "size": 512000,
  "content_type": "application/pdf",
  "provider": "cloudinary",
  "provider_file_id": null,
  "url": null,
  "status": "pending",
  "owner": "app_xk3m9pq7rz1c",
  "upload_strategy": "async",
  "folder": "products/shoes",
  "collection_id": "12345",
  "collection_name": "",
  "created_at": "2026-04-27T09:00:00Z",
  "updated_at": "2026-04-27T09:00:00Z"
}

Poll GET /api/files/43/ until status is "completed" or "failed".

Errors:

HTTP/1.1 413  { "detail": "File exceeds maximum upload size of 104857600 bytes." }
HTTP/1.1 413  { "detail": "File is too large for sync upload on this provider; use POST /api/files/direct-upload/ instead.", "provider": "cloudinary", "size": 12582912 }

`GET /api/files/{id}/` — Retrieve a file

GET /api/files/43/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...

{
  "id": 43,
  "name": "report.pdf",
  "size": 512000,
  "status": "completed",
  "provider": "cloudinary",
  "provider_file_id": "report",
  "url": "https://res.cloudinary.com/my-cloud/raw/upload/report.pdf",
  "owner": "app_xk3m9pq7rz1c",
  "upload_strategy": "async",
  "created_at": "2026-04-27T09:00:00Z",
  "updated_at": "2026-04-27T09:00:06Z"
}

`PATCH /api/files/{id}/` — Rename a file

PATCH /api/files/43/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...
Content-Type: application/json

{ "name": "q1-report-final.pdf" }

`DELETE /api/files/{id}/` — Delete a file

Removes the record and deletes the underlying object from the provider.

DELETE /api/files/43/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...

HTTP/1.1 204 No Content

`GET /api/files/{id}/stream/` — Stream a file (Range-aware)

GET /api/files/43/stream/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...
Range: bytes=0-1048575

HTTP/1.1 206 Partial Content
Content-Type: audio/mpeg
Content-Range: bytes 0-1048575/8388608
Accept-Ranges: bytes

Only providers with supports_streaming: true (e.g. Google Drive) use native chunked streaming. Others load the full file via download().

`POST /api/files/direct-upload/` — Initiate a direct upload

Use for files above the provider's sync threshold. The file bytes never pass through FileForge.

Basic direct upload:

POST /api/files/direct-upload/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...
Content-Type: application/json

{ "name": "large-video.mp4", "provider": "cloudinary", "size": 52428800, "content_type": "video/mp4" }

Direct upload to a specific folder:

POST /api/files/direct-upload/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...
Content-Type: application/json

{
  "name": "product-demo.mp4",
  "provider": "cloudinary",
  "size": 104857600,
  "content_type": "video/mp4",
  "folder": "products/videos"
}

HTTP/1.1 201 Created

{
  "file_id": 44,
  "upload_url": "https://api.cloudinary.com/v1_1/my-cloud/video/upload",
  "method": "POST",
  "fields": { "timestamp": "...", "public_id": "large-video", "api_key": "...", "signature": "..." },
  "headers": {},
  "expires_in": null,
  "provider_ref": { "public_id": "large-video", "resource_type": "video" }
}

Upload directly to upload_url using the returned method and fields, then call /complete/.

`POST /api/files/direct-upload/complete/` — Finalize a direct upload

POST /api/files/direct-upload/complete/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...
Content-Type: application/json

{
  "file_id": 44,
  "provider_file_id": "large-video",
  "url": "https://res.cloudinary.com/my-cloud/video/upload/large-video.mp4",
  "provider_response": { "public_id": "large-video", "resource_type": "video", "bytes": 52428800 }
}

HTTP/1.1 200 OK

{ "id": 44, "status": "completed", "url": "https://res.cloudinary.com/my-cloud/video/upload/large-video.mp4", ... }

`GET /api/folders/` — List folders

GET /api/folders/?provider=cloudinary HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...

HTTP/1.1 200 OK

{
  "folders": [
    { "name": "products", "path": "products", "created_at": "2026-04-01T10:00:00Z" },
    { "name": "avatars", "path": "avatars", "created_at": "2026-04-01T10:00:00Z" }
  ]
}

`POST /api/folders/` — Create a folder

POST /api/folders/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...
Content-Type: application/json

{
  "provider": "cloudinary",
  "path": "products/shoes"
}

HTTP/1.1 201 Created

{
  "name": "shoes",
  "path": "products/shoes",
  "created_at": "2026-04-27T09:00:00Z"
}

`DELETE /api/folders/{path}/` — Delete a folder

DELETE /api/folders/products/shoes/?provider=cloudinary HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...

HTTP/1.1 204 No Content

Note: Cloudinary only allows deleting empty folders.

`GET /api/collections/` — List collections

GET /api/collections/?provider=cloudinary HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...

HTTP/1.1 200 OK

{
  "collections": [
    {
      "id": "12345",
      "name": "product-photos",
      "description": "High-quality product images",
      "asset_count": 42,
      "created_at": "2026-04-01T10:00:00Z"
    }
  ]
}

`POST /api/collections/` — Create a collection

POST /api/collections/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...
Content-Type: application/json

{
  "provider": "cloudinary",
  "name": "marketing-assets",
  "description": "Assets for marketing campaigns"
}

HTTP/1.1 201 Created

{
  "id": "12346",
  "name": "marketing-assets",
  "description": "Assets for marketing campaigns",
  "asset_count": 0,
  "created_at": "2026-04-27T09:00:00Z"
}

`POST /api/collections/{id}/assets/` — Add files to collection

POST /api/collections/12345/assets/ HTTP/1.1
Authorization: Bearer ffk_xK3mAbc123...
Content-Type: application/json

{
  "provider": "cloudinary",
  "file_ids": ["products/shoe-photo", "products/shoe-angle"]
}

HTTP/1.1 200 OK

{
  "id": "12345",
  "name": "product-photos",
  "asset_count": 44
}

Adding a Custom Provider

Create storage/providers/<name>.py subclassing BaseStorageProvider:

from .base import BaseStorageProvider, UploadResult

class MyProvider(BaseStorageProvider):
    name = "my_provider"
    supports_direct_upload = False
    supports_streaming = False
    supports_folders = False
    supports_collections = False

    def upload(self, file, path, *, content_type=None, size=None, folder=None, **kwargs):
        return UploadResult(provider_file_id="...", url="...")

    def download(self, file_id, **kwargs): ...
    def delete(self, file_id, **kwargs): ...
    def update(self, file_id, **kwargs): ...
    def get_url(self, file_id, **kwargs): ...
    
    # Optional: Override these if your provider supports folders
    # def list_folders(self, path="", **kwargs): ...
    # def create_folder(self, path, **kwargs): ...
    # def delete_folder(self, path, **kwargs): ...
    
    # Optional: Override these if your provider supports collections
    # def list_collections(self, **kwargs): ...
    # def create_collection(self, name, **kwargs): ...
    # def add_to_collection(self, collection_id, file_ids, **kwargs): ...
    # def remove_from_collection(self, collection_id, file_ids, **kwargs): ...

Register it in storage/providers/registry.py inside register_default_providers().

No views, serializers, or services need to change.

Configuration

Variable	Default	Description
`SECRET_KEY`	(insecure dev key)	Django secret key. Required in production — server refuses to start with `RuntimeError` if `DJANGO_DEBUG=0` and this is not set.
`DJANGO_DEBUG`	`1`	Set to `0` in production.
`FILEFORGE_MAX_UPLOAD_SIZE`	`104857600` (100 MB)	Hard limit on accepted file size.
`FILEFORGE_DEFAULT_MAX_SYNC_SIZE`	`5242880` (5 MB)	Files larger than this are redirected to the direct-upload flow.
`FILEFORGE_GOOGLE_DRIVE_MAX_SYNC_SIZE`	`5242880` (5 MB)	Per-provider sync threshold override.
`FILEFORGE_CLOUDINARY_MAX_SYNC_SIZE`	`10485760` (10 MB)	Per-provider sync threshold override.
`FILEFORGE_TEMP_DIR`	`{BASE_DIR}/tmp_uploads`	Buffer directory for incoming files. Created on first startup.
`FILEFORGE_Q_WORKERS`	`2`	Django-Q2 background worker threads.
`FILEFORGE_Q_SYNC`	`0`	Set to `1` to run background tasks synchronously (useful in tests).
`JWT_ACCESS_MINUTES`	`30`	JWT access token lifetime.
`JWT_REFRESH_DAYS`	`7`	JWT refresh token lifetime (rotation enabled).

Google Drive (pick one auth mode):

Variable	Mode
`GOOGLE_CLIENT_ID` + `GOOGLE_CLIENT_SECRET` + `GOOGLE_REFRESH_TOKEN`	OAuth2 (personal Drive)
`GOOGLE_SERVICE_ACCOUNT_JSON` or `GOOGLE_SERVICE_ACCOUNT_FILE`	Service account (Workspace / shared drives)
`GOOGLE_DRIVE_FOLDER_ID`	Both — optional parent folder

Cloudinary — pick one:

CLOUDINARY_URL=cloudinary://api_key:api_secret@cloud_name
# or individually:
CLOUDINARY_CLOUD_NAME=...  CLOUDINARY_API_KEY=...  CLOUDINARY_API_SECRET=...

Per-owner credential overrides stored via POST /api/credentials/ always win over environment defaults.

Production Deployment

python manage.py migrate --noinput
python manage.py qcluster &
exec gunicorn --bind 0.0.0.0:5000 --workers 2 fileforge.wsgi:application

Set DJANGO_DEBUG=0 and a strong SECRET_KEY before deploying. The server will refuse to start if SECRET_KEY is the insecure default when DEBUG is off.

Contributing

Fork the repository and create a feature branch.
Follow the existing code style — DRF views with explicit authentication_classes and permission_classes on every view.
Add or update tests in fileforge_auth/tests.py / storage/tests.py.
Open a pull request describing what changed and why.

See DOCUMENTATION.md for the full API reference, provider guide, and architecture details.

Name		Name	Last commit message	Last commit date
fileforge		fileforge
fileforge_auth		fileforge_auth
storage		storage
.gitignore		.gitignore
DOCUMENTATION.md		DOCUMENTATION.md
README.md		README.md
manage.py		manage.py
replit.md		replit.md
requirements.txt		requirements.txt
run.sh		run.sh
task.md		task.md

Latest commit

History

Folders and files

Repository files navigation

FileForge

Key Features

Quick Start

First integration (4 steps)

Project Structure

Authentication

Getting a JWT token (management)

Using an API key (storage)

Architecture

Endpoint Overview

Management API (/auth/) — JWT required

Storage API (/api/) — API key required

Sample Requests & Responses

GET / — Service description

GET /api/health/ — Liveness probe

GET /api/providers/ — List providers and capabilities

GET /api/credentials/ — List credentials

POST /api/credentials/ — Create or update credentials

POST /api/files/ — Upload a file

GET /api/files/{id}/ — Retrieve a file

PATCH /api/files/{id}/ — Rename a file

DELETE /api/files/{id}/ — Delete a file

GET /api/files/{id}/stream/ — Stream a file (Range-aware)

POST /api/files/direct-upload/ — Initiate a direct upload

POST /api/files/direct-upload/complete/ — Finalize a direct upload

GET /api/folders/ — List folders

POST /api/folders/ — Create a folder

DELETE /api/folders/{path}/ — Delete a folder

GET /api/collections/ — List collections

POST /api/collections/ — Create a collection

POST /api/collections/{id}/assets/ — Add files to collection