Conversation
The category enum had no value for AI/LLM security challenges, so AI labs
(prompt injection, agent tool exploitation, RAG poisoning, LLM SSRF) had to be
mislabelled as 'web'.
Add 'ai' to the allowed categories in three places that must stay in sync:
- src/commands/validate.ts (validCategories)
- spec/challenge-schema.json (category enum)
- spec/CHALLENGE-SPEC.md (field table)
Verified: oasis validate accepts category 'ai', and a negative test confirms
the enum is enforced ('nonsense' is still rejected).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
categoryhas no value that fits AI/LLM security labs. The enum is:So an entire class of OASIS labs — prompt injection, agent tool exploitation,
RAG poisoning, LLM SSRF, text-to-SQL agent injection, confused-deputy email
agents — has to be mislabelled as
web, which is already taken by classicappsec. This is currently live: all 15 AI labs in the benchmark set are tagged
"category": "web".That mislabelling has real effects, not just cosmetic ones:
oasis challenges --category aicannot select the AI labs, so you cannotlist, filter, or batch-run the AI suite as a group.
src/interactive/run-flow.ts) printsformatCategory(category)in its selection list, so AI labs are presented tousers as web challenges.
numbers.
Change
Add
aias an allowed category, in the three places that must stay in sync:src/commands/validate.tsvalidCategories— this is enforced, invalid categories fail validationspec/challenge-schema.jsoncategoryenumspec/CHALLENGE-SPEC.mdcategoryis typedstringthroughout (types.ts,registry.ts), andformatCategorysimply renders whatever it is given, so no other code changesare needed — no exhaustive switch over categories exists.
Verification
Positive — all 15 AI labs validate with
category: "ai":Negative — the enum is genuinely enforced, so this change does not just
widen the door to anything:
The validator rejects unknown categories, and accepted
aionly after the enumwas extended; the change is the minimum required to make the AI labs
representable.
Notes
category: "ai"on the 15 labs) lives intraining-labsunderoasis-challenges/, and is already verified against this validator.miscremains available;aiwas added rather than replacing anything, soexisting challenges are unaffected.