From trained model to reviewed prediction.

MLsuite turns trained machine-learning models into reproducible, reviewable applications. Upload a model, run single or bulk predictions through a form generated from its schema, collect human reviews and corrections, and export everything for evaluation or retraining.

It is a deployed product, not a library: a React frontend, a Spring Boot API and a Python ML runtime, with PostgreSQL for application state and MinIO for artifacts, all run by Docker Compose. Try the live deployment or self-host your own.

Why MLsuite?

Testing a model with real users usually means rebuilding the same infrastructure: artifact storage, input validation, custom forms, prediction history, permissions, review workflows and exports.

MLsuite puts that loop in one open-source product. Serving a predict() endpoint is not the goal. The goal is to preserve exactly which model and schema produced a result, let people inspect and correct it, and turn those corrections into useful data. Nothing is for sale: if MLsuite goes in the wrong direction, the full stack is here to inspect, change and fork.

What it does

• Store and manage trained model artifacts.
• Analyze model inputs and generate versioned schemas.
• Render schemas as validated forms with MLForm.
• Run individual and bulk predictions.
• Preserve prediction history with exact model and schema identity.
• Collect reviews, corrections, questionnaires and explanation feedback.
• Extend reports and fields through plugins.
• Export inputs, outputs and feedback for downstream work.
• Publish a schema bookmark to a public page at /explore/<id>, listed in the public feed at /explore, where anyone, signed in or not, can fill its form (with saved runs you mark as examples) and run it. Every public run is kept as one of the organization's inferences, marked as made on the public page, and the browser that made it finds it again under Your runs, where the visitor can review its results without an account.
• Give an organization a logo, shown beside its name in the workspace and on its public pages, and give a public bookmark a publication note (the paper it was published in, a DOI, terms of use). Links shared to a public page unfurl with the bookmark's name, text and the publisher's logo.
• Let outside software run an organization's bookmarks through the client API with keys, usage metering and limits.
• Manage organizations, teams, roles, invitations and workspace permissions, and read who changed invitations, roles and API access in the organization's audit log (roles with VIEW_AUDIT_LOG).
• Let superadmins review every public bookmark on the instance and unpublish any of them, and watch every organization's API traffic.
• Monitor the local Compose stack through the operations service.

Where it fits

MLsuite is the last step of the eshkra path from a trained model to reviewed predictions. Each project also works on its own.

Step	Project	Role
Package	MLbundle	Writes the model, manifest and schema into one .mlbundle file.
Describe	MLschema	Infers the field schema MLbundle stores, from the training DataFrame.
Render	MLform	Draws the schema as the prediction form and its reports.
Run and review	MLsuite	Runs predictions, collects feedback and keeps every result traceable.

Model artifacts

MLsuite accepts .mlbundle files exported with mlbundle 0.1.0a2 or later for tabular classification and regression, and no other artifact format, no separate DataFrame and no bundle written by an earlier mlbundle version. A bundle carries a scikit-learn, XGBoost, CatBoost or ONNX model in its native pickle-free format with its manifest and schema, and MLsuite uses that schema as it comes: export with training_data= to infer the fields from the training frame; otherwise every feature is a number field.

Fields map to the model's own feature names; a model without them takes its features by position, so its fields map to 0, 1, …. One-hot columns are grouped with the separator the bundle was exported with. Every model output has its own report, mapped to the output's position. A scikit-learn bundle loads only with the scikit-learn version that exported it, which is the version pinned in backend/pyproject.toml. Crystal Tree explanations apply to supported decision-tree estimators only.

Getting started

Warning

MLsuite is early software. Expect rough edges and breaking changes.

You need Git and Docker with Docker Compose. Clone the repository and create your environment file:

git clone https://github.com/UlloaSP/mlsuite.git
cd mlsuite
cp .env.example .env

On Windows PowerShell, use Set-Location mlsuite and Copy-Item .env.example .env.

Open .env and give every blank secret its own value, especially DB_PASS, STORAGE_SECRET_KEY, OPS_AGENT_SHARED_SECRET and MLSUITE_SUPERADMIN_PASSWORD. Keep .env private; commit only sanitized defaults to .env.example.

Start the development stack:

docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build -d

Then open http://localhost:5173. Without a session it shows the public feed at /explore, whose header offers Sign in and Create account and brings you back to the page you were on. Sign in at /login with the superadmin account configured in .env, or register at /login?mode=register: a new account gets its own personal organization.

Inspect, follow and stop the stack with the same files:

docker compose -f docker-compose.yml -f docker-compose.dev.yml ps
docker compose -f docker-compose.yml -f docker-compose.dev.yml logs -f
docker compose -f docker-compose.yml -f docker-compose.dev.yml down

An old disposable development volume can be recreated with down -v. To keep an existing pre-Flyway development database, audit it against the baseline first and set FLYWAY_BASELINE_ON_MIGRATE=true for exactly one startup; return it to false immediately afterwards.

Deploying

Compose topology and releases

docker-compose.yml is the single operational topology. The development override builds local application images. docker-compose.prod.yml supplies production-only operational metadata and has no image defaults: its mandatory release override, docker-compose.release.yml, supplies every application image pinned by digest. CI compares the fully rendered models and rejects any other drift.

Successful publications from main create immutable build-<commit> releases with digest-pinned images and a Compose override; MLsuite does not update latest. Read release selection and verification before using published images. The release bundle supplies verified artifacts; deployment, migration, backup, readiness, promotion and rollback remain operator responsibilities.

Websites

landing/ and docs/ deploy to Cloudflare Workers Static Assets as mlsuite-landing (mlsuite.eshkra.dev) and mlsuite-docs (docs.mlsuite.eshkra.dev). CI checks and builds both sites and validates their Wrangler configuration without uploading. Only when every CI job passes on a push to main, or on a manual CI run on main, are the tested builds deployed without rebuilding, each through its own GitHub environment (landing, docs) limited to main, followed by an HTTPS check of the homepage title. Pull requests, forks and develop never deploy. The repository needs the Actions secrets CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID. To roll back, revert the change and merge it to main.

Production cutover

For a real cutover, run the repeatable, gated operator wizard from the repository root: ./scripts/production-cutover.sh. It creates coordinated backups, requires an isolated restore rehearsal, runs both migrations, checks readiness and records the rollback prerequisites. Review the procedure first; it changes the configured production environment only after explicit confirmations. It records the base, production and digest-pinned release files in OPS_AGENT_COMPOSE_FILES so operational restarts use the same release.

Backups and recovery

The initial supported on-premise topology is explicitly local-single-disk. The wizard creates a coordinated PostgreSQL plus MinIO recovery point, checksums it, restores it in an isolated environment, and requires an explicit acknowledgement that physical disk loss is not covered. These local backups protect against a bad deployment or logical deletion; they are not PITR or disaster recovery. The documented initial objective is a 24-hour logical-recovery RPO and an 8-hour RTO.

Install the persistent daily systemd timer once on the Linux host, using the account that can access Docker, and run or verify the same backup manually:

sudo bash ./scripts/install-backup-timer.sh .env <service-user>
systemctl list-timers mlsuite-backup.timer

ENV_FILE=.env ./scripts/create-local-backup.sh
ENV_FILE=.env bash ./scripts/verify-backup-rpo.sh

The timer runs daily even after a schedule missed during downtime. The cutover refuses to proceed unless it is enabled and the newest checksummed backup satisfies RECOVERY_RPO_MINUTES. Its isolated restore rehearsal is timed, must finish within RECOVERY_RTO_MINUTES, and records the measured duration in RECOVERY_LAST_RTO_SECONDS and the result in RECOVERY_LAST_RESTORE_RESULT. A failed rehearsal stops the cutover and stays visible instead of leaving a stale successful result.

The backup command pauses the frontend and API, dumps PostgreSQL, takes a cold archive of MinIO that preserves version IDs, verifies checksums, applies LOCAL_BACKUP_RETENTION_COUNT, checks the remaining disk space, and restarts only the application services that were running. Backups live under LOCAL_BACKUP_ROOT, outside the Docker data volumes but currently on the same physical disk. Keep at least 15% free with LOCAL_BACKUP_MIN_FREE_PERCENT. Before stopping writes, the command requires headroom for both the archive and the restore rehearsal: roughly 2.5 times the live PostgreSQL plus MinIO data, on top of that reserve.

Database and storage

Flyway is the only schema owner; Hibernate validates the result and never creates or updates production tables. The release runs the same immutable API image in a one-shot db-migrate service before the API may start.

Before the first Flyway-managed release:

1. Stop the frontend and API so PostgreSQL and MinIO are captured at one write-consistent point.
2. Create the coordinated local PostgreSQL and MinIO backup. The cold MinIO archive preserves object versions, delete markers and the version IDs that PostgreSQL references; keep the MinIO image pinned when restoring it. Until another physical device exists, record disk or host loss as an accepted, unrecoverable risk.
3. Enable and verify bucket versioning. Do not configure lifecycle expiry inside the rollback window.
4. Compare the existing schema with api/src/main/resources/db/migration/V1__baseline.sql.
5. If and only if it represents that schema, baseline and migrate it explicitly:

docker compose --env-file .env -f docker-compose.yml -f docker-compose.prod.yml \
  -f docker-compose.release.yml \
  run --rm \
  -e FLYWAY_BASELINE_ON_MIGRATE=true \
  -e FLYWAY_BASELINE_VERSION=1 \
  db-migrate

Never leave FLYWAY_BASELINE_ON_MIGRATE enabled. An empty installation needs no baseline; run db-migrate normally.

Compose provisions three PostgreSQL roles idempotently: DB_ADMIN_USER owns the service, DB_MIGRATION_USER owns DDL and Flyway objects, and DB_USER receives only the DML privileges the application needs. Use distinct credentials in production; on an externally managed database, create equivalent roles before running Compose. For a volume created by an older MLsuite Compose, set DB_LEGACY_OWNER and DB_LEGACY_PASS to its original DB_USER credentials: the provisioner transfers table and sequence ownership before Flyway runs and, since PostgreSQL cannot demote its bootstrap superuser, revokes that legacy role's login once the new administrator is verified. The migration role must be allowed to install pg_trgm, or a database administrator must install it first.

MinIO follows the same separation. MINIO_ROOT_USER and MINIO_ROOT_PASSWORD are used only by MinIO and its one-shot initializer. STORAGE_ACCESS_KEY and STORAGE_SECRET_KEY identify a separate application user whose generated policy is restricted to the configured bucket. The API and artifact migrator never receive the root credentials.

Artifact migration

After the schema migration and before artifact cutover, inspect status and run the resumable backfill, then verify it and prune orphans:

docker compose --env-file .env -f docker-compose.yml -f docker-compose.prod.yml \
  -f docker-compose.release.yml \
  --profile operations run --rm \
  -e ARTIFACT_MIGRATION_COMMAND=status artifact-migrate

docker compose --env-file .env -f docker-compose.yml -f docker-compose.prod.yml \
  -f docker-compose.release.yml \
  --profile operations run --rm \
  -e ARTIFACT_MIGRATION_COMMAND=migrate artifact-migrate

docker compose --env-file .env -f docker-compose.yml -f docker-compose.prod.yml \
  -f docker-compose.release.yml \
  --profile operations run --rm \
  -e ARTIFACT_MIGRATION_COMMAND=verify artifact-migrate

docker compose --env-file .env -f docker-compose.yml -f docker-compose.prod.yml \
  -f docker-compose.release.yml \
  --profile operations run --rm \
  -e ARTIFACT_MIGRATION_COMMAND=prune-orphans artifact-migrate

The job claims rows with FOR UPDATE SKIP LOCKED, records attempts and failures, uploads to immutable model/hash keys, reads each upload back and compares its SHA-256 before marking it verified. Normal reads and verification use the exact MinIO version ID persisted in PostgreSQL, not whichever version is latest.

Stopping the job is safe: each claimed artifact renews its lease, and expired RUNNING leases are reclaimed on the next execution. A worker that loses ownership leaves its content-addressed version in place, and the next owner verifies and attaches that exact version instead of risking deletion. A corrupt version is never attached; its exact failed version is removed and a retry writes and verifies a new version at the same key. After correcting an operational cause, reset exhausted rows with ARTIFACT_MIGRATION_COMMAND=retry-failed and run migrate again.

prune-orphans removes only unreferenced content-addressed model objects older than ARTIFACT_MIGRATION_ORPHAN_GRACE_SECONDS (24 hours by default). It rechecks the database immediately before deleting the exact current object version; plugin objects and unknown key layouts are outside its scope. Run it only while application writes and other artifact migration workers are stopped; the production cutover enforces that ordering.

The migration never clears model_file. Keep STORAGE_RETAIN_INLINE_COPY=true for this release so the previous application can still read every model during the rollback window.

Promotion and rollback

Production promotion gates are:

• db-migrate completed and Flyway validation has no drift;
• artifact status reports no FAILED, RUNNING, INLINE_ONLY or UNVERIFIED rows before the backfill is declared complete;
• artifact verify succeeds with no missing, truncated or hash-mismatched object;
• /actuator/health/readiness is healthy on the canary API;
• a PostgreSQL plus MinIO restore has been exercised in an isolated environment.

MODEL_MUTATIONS_REQUIRE_VERSION=false exists only for a controlled compatibility window before promotion, so an already-open older frontend can mutate models without the new parameter. The production cutover requires true: old cached clients must refresh, missing versions receive 428 Precondition Required, and mismatched versions receive 409 Conflict. New clients always send the version and keep stale-write protection.

Deploy backward-compatible API instances gradually. On application failure, roll back the image while the inline copies remain. On an incompatible persistent state, prefer roll-forward or restore PostgreSQL and MinIO together. Clearing model_file and dropping it are deferred to later releases, after the retention window and a successful restore drill.

Concurrency and monitoring

The API serves concurrent clients with one transaction per request, database constraints as the final authority, and optimistic versioning on models: of two clients changing the same model, the stale request receives 409 Conflict. Hikari bounds database concurrency with DB_POOL_MAX_SIZE, DB_POOL_MIN_IDLE and DB_POOL_CONNECTION_TIMEOUT_MS; size the pool from the PostgreSQL connection budget, leaving capacity for the migration job and operations, rather than matching it to the HTTP thread count.

Monitor storageDeletionQueue on /actuator/info and alert on any failed row or a pending count that grows continuously. Cleanup workers use database leases, so several API instances process disjoint tasks and an expired worker cannot complete or fail work reclaimed by another instance.

Public bookmarks

Public bookmarks run on the same runtime as every workspace, for visitors without an account.

• Model size. PUBLIC_PREDICTION_MAX_MODEL_SIZE_MB (default 50) is the largest model artifact a public bookmark may bind. Publishing, moving a public bookmark to another snapshot and each public run are refused above it, because every run ships the artifact to the runtime. The same limit applies to API client runs.
• Concurrency. PUBLIC_PREDICTION_MAX_CONCURRENT (default 2) public runs may execute at once; further ones receive 503 Service Unavailable until a slot frees. API clients have a share of their own, so neither can take the other's.
• Quota. Each caller may run one public bookmark a limited number of times in 24 hours, counted from their first run of it: PUBLIC_PREDICTION_ANONYMOUS_RUNS_PER_DAY (default 50) per client address without a session (an IPv6 address counts as its /64 network) and PUBLIC_PREDICTION_SIGNED_IN_RUNS_PER_DAY (default 500) per signed-in account, wherever it connects from. Past the limit the run receives 429 Too Many Requests with Retry-After; a run the server could not perform (502, 503) is not counted. Counts live in the API process memory: they restart with the API and are correct only with a single API instance, since each further instance would allow the full limit on its own.

A public run is stored like a workspace run, with its inputs, each model's result and any feedback, and is listed in the organization's Inferences with the origin "Public page"; it is never expired or deleted on its own. The browser that made it is named by a visitor id the API issues in the mlsuite_visitor cookie on the first run (HttpOnly, SameSite=Lax, sent only to /api/public, kept a year, marked Secure behind HTTPS). With it the page lists that browser's runs of the bookmark and lets the visitor review each one, storing the answers as that visitor's. A signed-in account running the public page keeps the browser's session and adds its name to the run. The quota is still counted by network, not by cookie, so clearing cookies gives no more runs.

A public page is drawn by the browser, so the frontend nginx sends link-unfurling crawlers (social networks, chat apps) to /api/public/bookmarks/<id>/preview, a page of Open Graph tags that redirects anyone else to the bookmark. Its image is the organization's logo, readable by anyone at /api/public/organizations/<org id>/logo, or the MLsuite mark when the organization has none. Logos are PNG or JPG uploads of up to 2 MB, cropped to a square and stored lossless as 256-pixel PNG in the database.

Client address behind a proxy

The client address comes from one place: the bundled nginx sends the peer it accepted as X-MLSuite-Client-Address, overwriting any value the client sent, and the API reads only that header, never X-Forwarded-For. Keep the API reachable only through that nginx, as the Compose files do.

When another reverse proxy or TLS terminator sits in front of nginx, every visitor arrives from that proxy's address and would share one count, so tell nginx which proxy to trust. Mount a file like this one into the frontend container as /etc/nginx/conf.d/real-ip.conf, naming the address or network your proxy connects from:

set_real_ip_from 10.0.0.5;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

The proxy must set or append to X-Forwarded-For itself. Do not list a network visitors can connect from, or they could choose their own address again.

Client API

Outside software runs an organization's bookmarks as one of its API clients. Members with the MANAGE_API_ACCESS permission manage them in the app's API section; members with VIEW_API_ACCESS can read it. They create a client named after the software, issue it keys and give it bookmarks. Keys (mls_...) are shown once, stored only as their SHA-256, never expire and are revoked by deletion; a client may hold several so a key can be replaced without stopping the software. A client runs exactly the bookmarks it was given, public or not, while their schema is active and their snapshot binds exactly one model under the public model size limit. Every change is kept in the organization's audit log.

POST /api/v1/bookmarks/<bookmark id>/predictions
X-MLSuite-Key: mls_...
Content-Type: application/json

{"features": {"age": 52, "sex__male": 1, "sex__female": 0}}

The bookmark id is the opaque id the API section shows, the same one its public page uses, assigned when the bookmark is first published or given to a client. Features go under the model's own signature feature names; each value is a number, a boolean, a text of at most 256 characters, or null, and only the runtime checks the record against the model. The answer says what ran and gives the model's outputs in output order:

{"requestId": "3f0c6a8e-…", "bookmark": {"id": "…", "name": "production"},
 "snapshot": {"version": 3, "name": "Baseline"}, "model": {"name": "risk-forest", "sha256": "…"},
 "outputs": [{"prediction": "high", "probabilities": {"low": 0.2, "high": 0.8}}, {"value": 41.5}]}

Every refusal is an RFC 9457 problem (application/problem+json) with a stable code: invalid_body and invalid_features (400), invalid_key (401), client_suspended (403), bookmark_not_found (404, also for a bookmark the client was not given), bookmark_not_runnable and bookmark_changed (409), features_rejected (422, with the runtime's reason), rate_limited and quota_exceeded (429), internal_error (500), runtime_failed (502) and runtime_busy (503). 429 and 503 carry Retry-After. The endpoint accepts any origin without credentials and exposes X-Request-Id and Retry-After. A key in browser JavaScript is readable by anyone who loads the page, so call it from a backend.

Each call made with a valid key is logged in api_request under the requestId it was answered with (also sent as X-Request-Id): client, key prefix, bookmark, status, problem code, duration and the inference it made, never the features or outputs, which stay in the run. The run is kept as the organization's inference with the origin "API" and the client's name as its author. The log is the metering: the API section shows each organization its usage, request log and endpoint docs, and superadmins see every organization's traffic under API traffic, where they suspend a client (with a reason its organization reads) or set an organization's monthly limit, both audited.

Setting	Default	Limit
API_ACCESS_REQUESTS_PER_MINUTE	60	Calls one client may make in a clock minute.
API_ACCESS_MONTHLY_CALLS	10000	Succeeded calls an organization's clients may make together in a calendar month (UTC), unless a superadmin set its own limit.
API_ACCESS_MAX_CONCURRENT	2	Client runs executing at once.
API_ACCESS_REQUEST_RETENTION_DAYS	90 (at least 31)	How long calls are kept.

The limits are counted from the log, so they hold across restarts and API instances; calls still running are counted once they end.

The client API replaced the bookmark API keys of POST /api/public/model-predictions, which no longer exists. Upgrading turns each key into a client key that keeps its secret: the keys an organization issued under one consumer name become one client that may run each of their bookmarks, now whether or not they are public. Consumers must move to the new path and body.

Architecture

flowchart LR
    UI[React + MLForm] --> API[Spring Boot API]
    API --> DB[(PostgreSQL)]
    API --> STORE[(MinIO)]
    API --> ML[Python ML runtime]
    ML --> API

Loading

The browser sends authenticated requests to Spring. Only /api/public/** answers without a session, and it serves nothing but bookmarks a member with the publish permission made public: their form, the inputs of the runs that member marked as examples, and runs of their models, routed by Spring and stored as the organization's inferences under the visitor who made them. The client API under /api/v1 answers to client keys instead of sessions. Spring enforces workspace permissions, stores durable state and delegates artifact analysis and prediction to Python. Results return through Spring, so model, schema, input, output and feedback identities remain traceable.

Persistence is abstracted by capability, not by a generic database wrapper. Spring Data repository interfaces isolate aggregate persistence, while ArtifactMigrationQueue hides the PostgreSQL-specific claiming implementation and StorageDeletionWorkQueue provides the same boundary for asynchronous cleanup. Both use per-claim fencing tokens, leases and FOR UPDATE SKIP LOCKED, so concurrent workers claim disjoint rows and late workers cannot finalize reclaimed work. Object storage remains a separate port because it is a genuinely external system with different failure and consistency modes.

Development

Read CONTRIBUTING.md and AGENTS.md before changing code. Run focused checks for the component you changed, and use the integrated Compose stack when a change crosses service boundaries.

The frontend uses Vite+ through the global vp command. Read the frontend architecture and the design system before working on it.

cd frontend
vp i
vp check
vp test

The Spring API requires Java 25 and Maven:

cd api
mvn test

The API's JSON contract lives in api/openapi.json, and the frontend types in frontend/src/shared/api/openapi.gen.ts are generated from it. After changing a controller or DTO, run mvn test -Dtest=OpenApiContractTest -Dopenapi.update=true in api/, then vp run api:types in frontend/. CI fails when either file is stale.

The Python runtime requires Python 3.14+ and uv:

cd backend
uv sync --extra dev
uv run pytest tests/test_runtime_api.py

Documentation and community

• Documentation
• Immutable releases
• CI and repository protection
• Contributing, Security policy and Code of Conduct

MLsuite is research-driven, early and currently centered on Python model runtimes; expect bugs. Its contracts are designed to support more model types without making the UI or API depend on one framework.

Report bugs through Issues, propose features in Ideas and ask usage questions in Q&A. For large features or contract changes, agree on direction before implementation.

License

MLsuite is available under the MIT License.