Skip to content

feat(schema): add 'ai' challenge category - #81

Open
r3y3r53 wants to merge 1 commit into
mainfrom
feat/ai-category
Open

r3y3r53 wants to merge 1 commit into
mainfrom
feat/ai-category

Conversation

@r3y3r53

@r3y3r53 r3y3r53 commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Problem

category has no value that fits AI/LLM security labs. The enum is:

web | network | crypto | forensics | misc

So an entire class of OASIS labs — prompt injection, agent tool exploitation,
RAG poisoning, LLM SSRF, text-to-SQL agent injection, confused-deputy email
agents — has to be mislabelled as web, which is already taken by classic
appsec. This is currently live: all 15 AI labs in the benchmark set are tagged
"category": "web".

That mislabelling has real effects, not just cosmetic ones:

  • oasis challenges --category ai cannot select the AI labs, so you cannot
    list, filter, or batch-run the AI suite as a group.
  • The interactive runner (src/interactive/run-flow.ts) prints
    formatCategory(category) in its selection list, so AI labs are presented to
    users as web challenges.
  • Any reporting that groups results by category mixes AI-lab results into web
    numbers.

Change

Add ai as an allowed category, in the three places that must stay in sync:

Place What
src/commands/validate.ts validCategories — this is enforced, invalid categories fail validation
spec/challenge-schema.json the category enum
spec/CHALLENGE-SPEC.md the field table
-const validCategories = ['web', 'network', 'crypto', 'forensics', 'misc'];
+const validCategories = ['web', 'network', 'crypto', 'forensics', 'misc', 'ai'];

category is typed string throughout (types.ts, registry.ts), and
formatCategory simply renders whatever it is given, so no other code changes
are needed — no exhaustive switch over categories exists.

Verification

Positive — all 15 AI labs validate with category: "ai":

$ for lab in oasis-challenges/*/; do oasis validate "$lab"; done
valid: 15  invalid: 0

Negative — the enum is genuinely enforced, so this change does not just
widen the door to anything:

$ # challenge.json with category: "nonsense"
$ oasis validate /tmp/vtest
  Validating: vtest
  ✗ INVALID

The validator rejects unknown categories, and accepted ai only after the enum
was extended; the change is the minimum required to make the AI labs
representable.

Notes

  • This is the schema half of a two-part change. The lab-side half (setting
    category: "ai" on the 15 labs) lives in training-labs under
    oasis-challenges/, and is already verified against this validator.
  • misc remains available; ai was added rather than replacing anything, so
    existing challenges are unaffected.

The category enum had no value for AI/LLM security challenges, so AI labs
(prompt injection, agent tool exploitation, RAG poisoning, LLM SSRF) had to be
mislabelled as 'web'.

Add 'ai' to the allowed categories in three places that must stay in sync:
  - src/commands/validate.ts  (validCategories)
  - spec/challenge-schema.json (category enum)
  - spec/CHALLENGE-SPEC.md     (field table)

Verified: oasis validate accepts category 'ai', and a negative test confirms
the enum is enforced ('nonsense' is still rejected).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant