Skip to content

Run analyze for MCPChecker so configured Compass facts can be pushed - #107

Open
suriya-93 wants to merge 1 commit into
RHEcosystemAppEng:mainfrom
suriya-93:fix/mcpchecker-analyze
Open

suriya-93 wants to merge 1 commit into
RHEcosystemAppEng:mainfrom
suriya-93:fix/mcpchecker-analyze

Conversation

@suriya-93

Copy link
Copy Markdown

Summary

MCPChecker runs were skipping the analyze phase on the CI pipeline, so aggregate_scorecard never ran and submissions could not push evaluation facts to Compass via metadata.yaml gate_policy. This PR runs analyze for MCPChecker and handles mcpchecker-report.json without changing Harbor/ASE/A2A/AEH behavior.

Compass push remains opt-in per submission (gate_policy in each package’s metadata.yaml). No submission metadata is changed in this PR.

Changes

  • CI pipelines (ci-pipeline.yaml, ci-pipeline-dev.yaml): remove when that skipped analyze for eval-engine=mcpchecker.
  • analyze-and-check-degradation task: add MCPChecker branch (read mcpchecker-report.json, set Tekton results); skip A/B report.md regeneration for MCPChecker; guard report.md cat when missing.
  • evaluate task: fix MCPChecker step when gates from harbor/both to mcpchecker so eval/aggregate actually run for MCPChecker.

Test plan

  • MCPChecker PipelineRun: analyze runs after evaluate; scorecard step runs when enabled.
  • Harbor or ASE run: same outcomes as before.
  • Submission without gate_policy: no Compass POST from pipeline.
  • After merge (cluster): apply updated Tekton YAML in ab-eval-flow:
    • oc apply -f pipeline/pipelines/ci-pipeline.yaml
    • oc apply -f pipeline/tasks/phases/evaluate.yaml
    • oc apply -f pipeline/tasks/post/analyze-and-check-degradation.yaml
    • Confirm: oc get pipeline.tekton.dev/abevalflow-pipeline -n ab-eval-flow -o yaml | grep -A6 "name: analyze" — analyze should show runAfter: ["evaluate"] and should not show operator: notin.
  • Local: YAML parses; no changes outside the four pipeline files.

Related

  • Security MCP eval pilot (skill-submissions); Compass gate_policy PR planned separately after Compass confirms factRef/UI.

@suriya-93

Copy link
Copy Markdown
Author

@GuyZivRH - could you please /ok-to-test and review this? Would appreciate your feedback. Thanks!

@GuyZivRH GuyZivRH left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants