From 945444613110fd50742d7d64c99bf6b10c90f520 Mon Sep 17 00:00:00 2001 From: dylan Date: Sat, 26 Sep 2026 22:32:55 -0500 Subject: [PATCH] Align miner documentation with current taxonomy and scoring --- README.md | 12 ++-- docs/Discriminative-Mining.md | 64 ++++++++++++--------- docs/Generative-Mining.md | 13 +++-- docs/Incentive.md | 104 ++++++++++------------------------ docs/Mining.md | 6 +- 5 files changed, 87 insertions(+), 112 deletions(-) diff --git a/README.md b/README.md index b0a5f8a5..e7f73fa3 100644 --- a/README.md +++ b/README.md @@ -31,14 +31,14 @@ GAS runs two parallel competition tracks on Bittensor Subnet 34: | Track | What You Do | How You're Scored | |-------|-------------|-------------------| -| **Discriminative Mining** | Submit media-provenance classifiers (image, video, audio) | `sn34_score` -- geometric mean of normalized MCC and Brier performance, using the active round's binary or multiclass scoring mode | -| **Generative Mining** | Run a server that generates synthetic media on demand | Base reward for valid content × multiplier for fooling discriminators | +| **Discriminative Mining** | Submit media-provenance classifiers (image, video, audio) | `sn34_score`: classification performance and probability accuracy, with the active round's weights and robustness blend | +| **Generative Mining** | Run a server that generates synthetic media on demand | Verified content, volume, and model pricing in modalities that clear the fool-rate gate | **Key facts:** - **Three modalities**: Image, video, and audio detection are all scored independently - **Model format**: Safetensors only (ONNX submissions are not accepted) - **Datasets refresh weekly** with fresh GAS-Station data alongside static benchmarks -- **One model per hotkey** for discriminative miners +- **Submission allowances** are shared across modalities; see [submission limits](docs/Discriminative-Mining.md#submission-limits) for free and repeat submissions See [Incentive Mechanism](docs/Incentive.md) for full scoring details. @@ -70,10 +70,10 @@ gascli validator start # Miners: Start or restart generative miner gascli generator start -# Miners: Push one discriminator model per hotkey +# Miners: Submit a discriminator model gascli d push --image-model image_detector.zip \ --wallet-name default --wallet-hotkey default -# Video or audio: use --video-model or --audio-model on a different hotkey +# For video or audio, use --video-model or --audio-model # Miners: Check your benchmark performance (epistula-authenticated) gascli d perf --wallet-name default --wallet-hotkey default @@ -96,7 +96,7 @@ pm2 start validator.config.js # Miners: Start or restart generative miner pm2 start gen_miner.config.js -# Miners: Push one discriminator model per hotkey +# Miners: Submit a discriminator model source .venv/bin/activate python neurons/discriminator/push_model.py \ --image-model image_detector.zip \ diff --git a/docs/Discriminative-Mining.md b/docs/Discriminative-Mining.md index b9297938..c600e85e 100644 --- a/docs/Discriminative-Mining.md +++ b/docs/Discriminative-Mining.md @@ -6,20 +6,21 @@ Follow the [Installation Guide](Installation.md) to set up your environment befo ## Discriminative Mining Overview -- Miners submit media-provenance classifiers across three modalities: **image**, **video**, and **audio**. -- Image models classify `[real, synthetic, semisynthetic]`; video models classify `[real, synthetic, semisynthetic, rendered]`; audio remains `[real, synthetic]`. -- The visual taxonomy is experimental. Semisynthetic media retains materially captured visual content alongside spatially localized generated or replaced content. Fully synthesized output remains synthetic even when captured media conditions generation. -- Miners do not need to host hardware for inference. +Submit an image, video, or audio classifier. Evaluation runs on subnet +infrastructure; miners do not need to host inference hardware. Class order is part of the submission contract: | Modality | `num_classes` | Logit indices | | --- | ---: | --- | | Image | 3 | `0=real`, `1=synthetic`, `2=semisynthetic` | -| Video | 4 | `0=real`, `1=synthetic`, `2=semisynthetic`, `3=rendered` | +| Video | 3 | `0=real`, `1=synthetic`, `2=semisynthetic` | | Audio | 2 | `0=real`, `1=synthetic` | -See GASBench's [Classification Taxonomy and Scoring](https://github.com/BitMind-AI/gasbench/blob/main/docs/Classification-and-Scoring.md) for the normative class definitions, metric formulas, and binary compatibility collapse. +Non-AI rendering, such as CGI and game footage, belongs to `real` for both image +and video. Audio metadata labeled `semisynthetic` maps to `synthetic`. +See GASBench's [classification and scoring guide](https://github.com/BitMind-AI/gasbench/blob/main/docs/Classification-and-Scoring.md) +for class definitions and how probabilities contribute to the score. ## Model Preparation @@ -31,12 +32,12 @@ Discriminative miners must submit models in **safetensors format**: **📖 [Safetensors Model Specification](https://github.com/bitmind-ai/gasbench/blob/main/docs/Safetensors.md)** - Requirements for model submission -Each hotkey can submit **one** model, in one of these modalities: +Choose one modality per upload: - `image_detector.zip` - Image classification model - `video_detector.zip` - Video classification model - `audio_detector.zip` - Audio classification model -A second modality needs a second registered hotkey. +Submissions share the hotkey's allowance across modalities; see [Submission Limits](#submission-limits) for repeat submissions. ## Pushing Your Model @@ -45,7 +46,7 @@ First, activate the virtual environment: source .venv/bin/activate ``` -Push one model per hotkey using the `push` command: +Upload a model using the `push` command: ```bash gascli d push \ @@ -53,7 +54,7 @@ gascli d push \ --wallet-name your_wallet_name \ --wallet-hotkey your_hotkey_name -# Video or audio (same or another hotkey): +# Use --video-model or --audio-model for other modalities: # gascli d push --video-model video_detector.zip --wallet-hotkey video_key # gascli d push --audio-model audio_detector.zip --wallet-hotkey audio_key ``` @@ -104,16 +105,19 @@ Each registered hotkey gets **one free counted submission** (image, video, or au ### Scoring -Each model is scored per modality using `sn34_score`, a geometric mean of normalized MCC and Brier performance: +`sn34_score` is the competition score. It combines classification performance +(MCC) with probability accuracy (Brier error), using the active round's scoring +mode, dataset weights, and augmentation settings. `benchmark_score` is class +accuracy, which is used for the entrance exam. -$$sn34_{score} = \sqrt{M_{norm} \cdot B_{norm}}$$ - -The normalized terms apply exponents $1.2$ to MCC performance and $1.8$ to Brier performance. Image and video currently use multiclass Gorodkin MCC and multiclass Brier, rewarding correct distinctions between provenance classes. Audio uses the equivalent two-class calculation. Binary real-versus-not-real metrics are still reported for compatibility and diagnosis. See [Incentive Mechanism](Incentive.md) for the full formula. +See GASBench's [classification and scoring guide](https://github.com/BitMind-AI/gasbench/blob/main/docs/Classification-and-Scoring.md) +for the formula and result fields, and [Incentive Mechanism](Incentive.md#discriminator-rewards) +for how the score affects rewards. ### Model Requirements - **Format**: Safetensors only (ONNX is no longer accepted) -- **Submission cap**: one counted model per hotkey (see [Submission Limits](#submission-limits)) +- **Submission allowance**: see [Submission Limits](#submission-limits) for the free allowance and repeat submissions ### Sandbox and Import Restrictions @@ -171,9 +175,9 @@ After a successful push, your model goes through a two-stage evaluation process Before your model is ever scored on the network, it must pass an **entrance exam** — a fast sanity check run against a reduced sample of the benchmark datasets. -- Internally this runs `gasbench run --small`, which downloads one archive per dataset and evaluates roughly 100 samples per dataset -- Your model must achieve **≥ 80% accuracy** averaged across all submitted modalities to pass -- The exam has a **maximum wall-clock timeout of 1 hour 25 minutes** (5,100 seconds); models that exceed this are treated as failed +- The exam uses GASBench `small` mode on datasets selected for the submitted modality and vertical +- The submitted model must achieve **≥ 80% class accuracy** (`benchmark_score`) to pass +- The evaluator enforces a time budget; models that exhaust it fail the exam - The exam runs in an **isolated sandbox** — your code has no network access and cannot interact with the host environment - Submissions are statically analyzed and executed in an isolated sandbox; prohibited code or imports result in rejection @@ -186,7 +190,8 @@ Before your model is ever scored on the network, it must pass an **entrance exam | `exam_failed` | Accuracy below 80% — model will not be scored | | `blocked` | Cheat pattern detected — model is permanently blocked | -You can use `gasbench run --small` locally to replicate exam conditions before pushing: +Use `gasbench run --small` as a local preflight before pushing. The hosted exam +also applies its own dataset selection, sandbox, and resource limits: ```bash gasbench run --image-model ./my_image_model/ --small @@ -196,25 +201,32 @@ gasbench run --audio-model ./my_audio_model/ --small ### Stage 2: Full Benchmark (`--full` mode) -Models that pass the entrance exam are benchmarked against the **complete dataset suite**, which includes: +Models that pass the entrance exam are evaluated on a larger sample of the +datasets selected for their modality and vertical, including: -- All public benchmark datasets across image, video, and audio modalities +- Public benchmark datasets - **Private holdout datasets** — curated datasets not visible to miners, used to prevent overfitting to the public benchmark set - Refreshed weekly with new data from the GAS-Station pipeline -The full benchmark has a **maximum wall-clock timeout of 5 hours** (18,000 seconds) per modality. This `sn34_score` is what the King of the Hill competition uses: a high enough score can take or keep a lane, and emissions then follow the 85/10/5 split on the current king plus the previous two. The active round configuration selects provenance weighting, multiclass scoring, and augmentation robustness parameters; see [Incentive Mechanism](Incentive.md). +The evaluator enforces a time budget for the full run. The resulting +`sn34_score`, including any configured robustness blend, is used in the King of +the Hill competition. See [Incentive Mechanism](Incentive.md#king-of-the-hill) for +challenge margins and emission shares. -You can simulate a full benchmark run locally (without holdouts) to get a sense of your model's performance: +For a local image run with multiclass scoring: ```bash -gasbench run --image-model ./my_image_model/ --full +gasbench run --image-model ./my_image_model/ --full --multiclass-scoring ``` -> **Note**: Local full runs will not include the private holdout datasets used in the actual network evaluation. +Local runs use the public datasets available to you. Their scores are not +directly comparable to a hosted round unless the datasets, sampling, scoring +mode, weights, and augmentation settings match. ### Checking Your Performance -Once your model has been benchmarked, you can query your scores directly from the CLI. This is authenticated with your hotkey so you can only see your own results — including the active round that isn't shown on the public leaderboard. +Query your runs from the CLI. Requests are authenticated with your hotkey and +return your own results: ```bash # View all your benchmark runs diff --git a/docs/Generative-Mining.md b/docs/Generative-Mining.md index 9f98cc8e..bfc2f8bb 100644 --- a/docs/Generative-Mining.md +++ b/docs/Generative-Mining.md @@ -12,7 +12,11 @@ Generative miners create synthetic media (images and videos) according to prompt 3. **Sample volume** -- verified volume ramps through the first 10 samples, then logarithmically 4. **Model choice** -- more expensive generation models earn higher per-unit rewards (see [Model Pricing](#model-pricing-and-rewards) below) -Validators still challenge onboarding and under-bar miners (8 + 6 of 50 slots) so new UIDs can build a fool-rate sample. Those slots do not pay until the miner qualifies. A miner this validator has asked enough times with no answer is skipped for that modality instead of sitting in onboarding. See [Incentive Mechanism](Incentive.md) for the full reward formula. +By default, validators sample challenge targets uniformly from registered +generators; fool-rate qualification controls payment. The optional bucket +allocation reserves onboarding and probe slots. See +[challenge allocation](Incentive.md#challenge-slots) for both modes and +[generator rewards](Incentive.md#generator-rewards) for the payout rules. Generative miners operate as FastAPI servers that receive generation requests from validators and respond asynchronously via webhooks. @@ -101,10 +105,9 @@ All configured services must produce **C2PA-signed content**. Setting a modality ## Model Pricing and Rewards -The validator weights each submitted media sample by the **cost of the model that produced it**. -More expensive models earn proportionally higher per-sample rewards. This incentivizes miners -to invest in quality generation infrastructure instead of racing to the bottom with the cheapest -available model. +The validator applies a model-price multiplier to verified samples. Higher +reference prices increase the multiplier through square-root scaling, so a +fourfold price increase gives a twofold multiplier. ### How It Works diff --git a/docs/Incentive.md b/docs/Incentive.md index 7d17ef77..62ae2f2c 100644 --- a/docs/Incentive.md +++ b/docs/Incentive.md @@ -1,61 +1,24 @@ # Incentive Mechanism ## Benchmark Runs -Submitted discriminator miners are evaluated against a subset of the data sources listed below. Miners do not need to host hardware for inference. A portion of the evaluation data comes from generative miners, who are rewarded based on their ability to submit data that both pass validator sanity checks (prompt alignment, etc.) and fool discriminators in benchmark runs. -Each modality (image, video, audio) is scored independently using the `sn34_score` metric, which combines classification performance (MCC) with probability calibration (Brier score). The active round selects binary or multiclass scoring per modality. +Submitted classifiers are evaluated separately for image, video, and audio. +Miners do not need to host inference hardware. The competition uses each run's +final `sn34_score`, including any configured augmentation blend. -
-Evaluation Datasets +Image and video use `[real, synthetic, semisynthetic]`; audio uses +`[real, synthetic]`. Non-AI rendered media belongs to `real`. See GASBench's +[classification and scoring guide](https://github.com/BitMind-AI/gasbench/blob/main/docs/Classification-and-Scoring.md) +for the class definitions and scoring rules. -Benchmark datasets are regularly expanded. Image uses real, synthetic, and semisynthetic classes; video additionally includes rendered media; audio remains binary. The experimental visual taxonomy is defined in GASBench's [Classification Taxonomy and Scoring](https://github.com/BitMind-AI/gasbench/blob/main/docs/Classification-and-Scoring.md). Datasets include continuously updated [GAS-Station](https://huggingface.co/gasstation) data from generative miners. - -**Public datasets** (available for training via gasbench): -- **Image**: [`image_datasets.yaml`](https://github.com/BitMind-AI/gasbench/blob/main/src/gasbench/dataset/configs/image_datasets.yaml) -- **Video**: [`video_datasets.yaml`](https://github.com/BitMind-AI/gasbench/blob/main/src/gasbench/dataset/configs/video_datasets.yaml) -- **Audio**: [`audio_datasets.yaml`](https://github.com/BitMind-AI/gasbench/blob/main/src/gasbench/dataset/configs/audio_datasets.yaml) - -**Holdout datasets**: In addition to the public datasets above, each benchmark round includes holdout datasets that are not publicly available during the round. Holdout data is critical to ensure models generalize well and to mitigate overfitting. At the end of each round, many of the holdout datasets are released and added to the public gasbench datasets for future training. Some holdout datasets cannot be released publicly due to licensing or other restrictions. - -
- -
-Generative Models - -The following models are run by validators to produce a continual, fresh stream of synthetic and semisynthetic data. The outputs of these models are uploaded at regular intervals to public datasets in the [GAS-Station](https://huggingface.co/gasstation) Hugging Face org for miner training and evaluation. - -### Text-to-Image Models - -- [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) -- [SG161222/RealVisXL_V4.0](https://huggingface.co/SG161222/RealVisXL_V4.0) -- [Corcelio/mobius](https://huggingface.co/Corcelio/mobius) -- [prompthero/openjourney-v4](https://huggingface.co/prompthero/openjourney-v4) -- [cagliostrolab/animagine-xl-3.1](https://huggingface.co/cagliostrolab/animagine-xl-3.1) -- [runwayml/stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5) + [Kvikontent/midjourney-v6](https://huggingface.co/Kvikontent/midjourney-v6) LoRA -- [black-forest-labs/FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) -- [DeepFloyd/IF](https://huggingface.co/DeepFloyd/IF) -- [deepseek-ai/Janus-Pro-7B](https://huggingface.co/deepseek-ai/Janus-Pro-7B) -- [THUDM/CogView4-6B](https://huggingface.co/THUDM/CogView4-6B) - -### Image-to-Image Models - -- [diffusers/stable-diffusion-xl-1.0-inpainting-0.1](https://huggingface.co/diffusers/stable-diffusion-xl-1.0-inpainting-0.1) -- [Lykon/dreamshaper-8-inpainting](https://huggingface.co/Lykon/dreamshaper-8-inpainting) - -### Text-to-Video Models - -- [tencent/HunyuanVideo](https://huggingface.co/tencent/HunyuanVideo) -- [genmo/mochi-1-preview](https://huggingface.co/genmo/mochi-1-preview) -- [THUDM/CogVideoX-5b](https://huggingface.co/THUDM/CogVideoX-5b) -- [ByteDance/AnimateDiff-Lightning](https://huggingface.co/ByteDance/AnimateDiff-Lightning) -- [Wan-AI/Wan2.2-TI2V-5B-Diffusers](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B-Diffusers) - -### Image-to-Video Models - -- [THUDM/CogVideoX1.5-5B-I2V](https://huggingface.co/THUDM/CogVideoX1.5-5B-I2V) - -
+The [GASBench dataset registry](https://github.com/BitMind-AI/gasbench/tree/main/src/gasbench/dataset/configs) +lists public datasets. Full evaluations also use private holdouts to measure +generalization, alongside fresh [GAS-Station](https://huggingface.co/gasstation) +data from generative miners. Holdouts may be released for future training when +licensing permits. +For generation services and model choices, see the +[Generative Mining Guide](Generative-Mining.md#generation-services). ## Generator Rewards @@ -118,36 +81,29 @@ This design incentivizes generators to: ### Scoring: `sn34_score` -Each discriminator model is scored per modality using two components: - -1. **MCC** measures classification quality. Binary mode uses ordinary MCC after collapsing every non-real class into synthetic. Multiclass mode uses Gorodkin's $R_K$, the multiclass generalization of MCC. -2. **Brier score** measures calibration. Binary mode uses the mean squared error of $p_{\text{not real}}$, whose constant-guess baseline is $0.25$. Multiclass mode uses the mean of $\sum_k(p_k-y_k)^2$, whose uniform-guess baseline for $K$ classes is $B_0=(K-1)/K$. - -For the selected mode, let $M$ be MCC, $B$ be Brier score, and $B_0$ be the corresponding random baseline: - -$$M_{norm} = \operatorname{clip}\left(\frac{M+1}{2},0,1\right)^{1.2}$$ - -$$B_{norm} = \max\left(0,\frac{B_0-B}{B_0}\right)^{1.8}$$ - -$$sn34_{score} = \sqrt{M_{norm} \cdot B_{norm}}$$ - -Image and video currently use multiclass scoring. Audio uses binary scoring; with two classes, the normalized multiclass calculation is mathematically identical. Every run also reports `binary_sn34_score` and `multiclass_sn34_score` so the two views can be compared. - -### Dataset composition and augmentation robustness - -The round configuration assigns target score shares to public, private holdout, and GAS-Station samples. Those shares are converted into per-sample weights and applied consistently to accuracy, MCC, Brier, cross-entropy, and the resulting SN34 score. +Each evaluation pass combines MCC (classification performance) and Brier error +(probability accuracy). The round configuration selects binary or multiclass +scoring and target weights for public, holdout, and GAS-Station samples. -When the robustness pass is enabled, the final score is: +When an augmentation pass contributes to the score: -$$sn34_{final} = (1-w)\,sn34_{base} + w\,sn34_{aug}$$ +```text +sn34_score = (1 - aug_weight) * base_sn34_score + aug_weight * aug_sn34_score +``` -The benchmark records `base_sn34_score`, `aug_sn34_score`, and robustness diagnostics. Exact composition shares, augmentation sample counts, and $w$ are round configuration, so they may change between benchmark versions rather than being permanent protocol constants. +Use the final `sn34_score` for competition comparisons. `benchmark_score` is +accuracy, while `binary_sn34_score` and `multiclass_sn34_score` describe the base +pass. The round configuration controls the scoring mode, dataset shares, +augmentation sample count, and blend weight. -The normative implementation details and complete metric field glossary live in GASBench's [Classification Taxonomy and Scoring](https://github.com/BitMind-AI/gasbench/blob/main/docs/Classification-and-Scoring.md). +GASBench's [classification and scoring guide](https://github.com/BitMind-AI/gasbench/blob/main/docs/Classification-and-Scoring.md) +defines the score calculation and result fields. ### King of the Hill -Discriminator emission is King of the Hill. Each modality has one reigning model. Validators set that lane's weight on registered hotkeys every tempo — not on an escrow wallet. Each hotkey may land **one counted submission** for the life of that registration (any modality; exam failures do not count; a new model needs a new key). +Each modality has one reigning model. Validators assign that lane's emissions +to registered hotkeys every tempo. See [Submission Limits](Discriminative-Mining.md#submission-limits) +for the free submission allowance and the same-hotkey resubmission process. Current split: diff --git a/docs/Mining.md b/docs/Mining.md index ebd38200..a0591fd5 100644 --- a/docs/Mining.md +++ b/docs/Mining.md @@ -3,7 +3,11 @@ GAS supports two types of miners that work together in an adversarial loop: ## [Discriminative Mining](Discriminative-Mining.md) 📖 -Miners submit classifiers that detect AI-generated content across **image, video, and audio** modalities. Models are evaluated against diverse benchmark datasets and scored using the `sn34_score` metric (accuracy + calibration). Each hotkey gets one counted submission (any modality); see [Discriminative Mining](Discriminative-Mining.md#submission-limits). +Miners submit **image, video, or audio** classifiers for hosted evaluation. The +competition uses `sn34_score`, which combines classification performance and +probability accuracy with the configured robustness blend. See +[Discriminative Mining](Discriminative-Mining.md) for model preparation and +[submission limits](Discriminative-Mining.md#submission-limits). ## [Generative Mining](Generative-Mining.md) 🎨 Miners create synthetic media (images and videos) that challenges the discriminators. They generate increasingly realistic content to test and improve detection capabilities, and are rewarded for verified volume in modalities that clear the 7-day fool-rate gate.