Repository navigation
CLI conformance test environment for the OpenStack API and terraform (SPEC-028) #107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
senolcolak
wants to merge
47
commits into
main
Choose a base branch
from
feature/105
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
47 commits
Select commit
Hold shift + click to select a range
19edcb6
docs(specs): add SPEC-028 plan — OpenStack CLI conformance test envir…
ef2a63e
test(cli): add the OpenStack CLI conformance harness core (SPEC-028 p…
2b4ed22
fix(glance): honor the standard v2 image list filters and persist tag…
f2b03fd
fix(nova): four conformance gaps found by the SPEC-028 CLI suite
2807f13
fix(neutron): honour the admin tenant filter on network list and the …
cfacd1b
fix(keystone): seven conformance gaps found by the SPEC-028 CLI suite
8886434
fix(cinder): seven conformance gaps found by the SPEC-028 CLI suite
6154505
test(cli): sweep volumes with plain delete before --force
1ce60ef
ci: gate the CLI conformance suite on every PR, nightly full + terraf…
9551d9b
test(terraform): apply/destroy the unmodified compat module against a…
730015e
test(cli): conformance suites for compute, glance, identity, volume a…
649ec97
ci(nightly): install the pinned CLI for the terraform conformance job
2b7a6f3
fix(git): track the CLI harness gate scripts
4c97eed
fix(neutron,keystone,cinder): three review follow-ups from the SPEC-0…
f510cd1
fix(cli): three harness defects found by the SPEC-028 review pass
a076e22
ci: install the CLI in a venv and run the nightly suite in strict mode
4cd0e41
test(terraform): preserve failure logs and tighten the residue check
893bd1e
test(cli): assert the fixed compute/volume behaviour outright
ccc4574
fix(neutron): nine CLI-conformance divergences found by the network s…
9632372
test(cli): network suite — 41 cases, all green against the fixed server
956ba5e
docs(cli): document run.sh --strict in the Running section
9d24bca
fix(keystone): two more CLI-conformance findings from the full run
9592cfe
fix: review round 2 — CI terraform job, residue gate, rollbacks, stal…
1100f7f
fix: review round 3 — last stale-doc stragglers and a test-comment claim
a1e0ea6
fix: comprehensive review round 4 — scoping, atomicity, timeouts, har…
f4b9ebb
fix(neutron): keep the admin carve-out out of the project-scoped cache
8b039d1
ci: run the cache regression test in CI; pin its cache-hit claim
a78870d
test(cli): drop a dead known-failure probe in the token-revoke neg case
391690c
test(cli): fail closed on corrupt results; keep standalone suites run…
fb29751
fix: comprehensive review round 7 — scoping symmetry, NULL scans, cac…
f2c348e
test(cli): round-8 delta polish — exit before leasing, subnet probe, …
c7e167a
fix: round-8 delta review — ownership labels, show carve-outs, panic …
f82dd16
fix: round-9 review — dead failure check, vacuous pin, unshelve resta…
9a0ab5f
fix: round-10 review — self-heal persists the revived domain, 078, pa…
725c173
fix: resolve conformance lint failures
senolcolak 1306ed9
fix(ci): repair conformance jobs and validate coverage evidence
senolcolak 127a822
fix(ci): pin compatible Helm and patched Go toolchains
senolcolak e5c34b5
fix(test): require reliable Terraform cleanup evidence
senolcolak f0a70f9
fix(ci): avoid overlapping Go cache restores
senolcolak 5708584
fix(cinder): retain project filter with all-tenants requests
senolcolak 8397cff
fix(cli): baseline known failures; fix shared-pool attribution and ti…
9674ebd
Merge current main into CLI conformance branch
senolcolak 083410e
fix: repair conformance branch after main integration
senolcolak 6afe955
merge: integrate main security fixes into CLI conformance
senolcolak c40d58e
fix(database): align server-group unique indexes across backends
senolcolak a754139
fix: preserve resources and fail closed in conformance workflows
senolcolak 54fd0b1
fix: preserve response metadata and tighten conformance evidence
senolcolak File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -20,14 +20,9 @@ jobs: | |||||
| uses: actions/setup-go@v6 | ||||||
| with: | ||||||
| go-version: 'stable' | ||||||
|
|
||||||
| - name: Cache Go modules | ||||||
| uses: actions/cache@v5 | ||||||
| with: | ||||||
| path: ~/go/pkg/mod | ||||||
| key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }} | ||||||
| restore-keys: | | ||||||
| ${{ runner.os }}-go- | ||||||
| cache-dependency-path: | | ||||||
| go.sum | ||||||
| tools/go.sum | ||||||
|
|
||||||
| - name: Build binary | ||||||
| run: make build | ||||||
|
|
@@ -46,6 +41,12 @@ jobs: | |||||
| - name: Run linters | ||||||
| run: make lint | ||||||
|
|
||||||
| - name: Test CLI harness reporting and coverage gates | ||||||
| run: python3 -m unittest discover -s test/cli -p 'test_*.py' -v | ||||||
|
|
||||||
| - name: Test Terraform harness evidence gates | ||||||
| run: python3 -m unittest discover -s test/terraform -p 'test_*.py' -v | ||||||
|
|
||||||
| - name: Upload binary artifact | ||||||
| uses: actions/upload-artifact@v7 | ||||||
| with: | ||||||
|
|
@@ -84,14 +85,9 @@ jobs: | |||||
| uses: actions/setup-go@v6 | ||||||
| with: | ||||||
| go-version: 'stable' | ||||||
|
|
||||||
| - name: Cache Go modules | ||||||
| uses: actions/cache@v5 | ||||||
| with: | ||||||
| path: ~/go/pkg/mod | ||||||
| key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }} | ||||||
| restore-keys: | | ||||||
| ${{ runner.os }}-go- | ||||||
| cache-dependency-path: | | ||||||
| go.sum | ||||||
| tools/go.sum | ||||||
|
|
||||||
| - name: Install qemu-img for image normalization tests | ||||||
| run: sudo apt-get update && sudo apt-get install -y --no-install-recommends qemu-utils | ||||||
|
|
@@ -129,14 +125,9 @@ jobs: | |||||
| uses: actions/setup-go@v6 | ||||||
| with: | ||||||
| go-version: 'stable' | ||||||
|
|
||||||
| - name: Cache Go modules | ||||||
| uses: actions/cache@v5 | ||||||
| with: | ||||||
| path: ~/go/pkg/mod | ||||||
| key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }} | ||||||
| restore-keys: | | ||||||
| ${{ runner.os }}-go- | ||||||
| cache-dependency-path: | | ||||||
| go.sum | ||||||
| tools/go.sum | ||||||
|
|
||||||
| - name: Start lightstack services | ||||||
| run: | | ||||||
|
|
@@ -250,14 +241,9 @@ jobs: | |||||
| uses: actions/setup-go@v6 | ||||||
| with: | ||||||
| go-version: 'stable' | ||||||
|
|
||||||
| - name: Cache Go modules | ||||||
| uses: actions/cache@v5 | ||||||
| with: | ||||||
| path: ~/go/pkg/mod | ||||||
| key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }} | ||||||
| restore-keys: | | ||||||
| ${{ runner.os }}-go- | ||||||
| cache-dependency-path: | | ||||||
| go.sum | ||||||
| tools/go.sum | ||||||
|
|
||||||
| - name: Build | ||||||
| run: make build | ||||||
|
|
@@ -423,6 +409,113 @@ jobs: | |||||
| # TODO: Add e2e-fast and e2e-full jobs once E2E test scenarios are implemented. | ||||||
| # Planned: VM lifecycle, volume attach/detach, network isolation, Terraform validation, Horizon UI tests. | ||||||
|
|
||||||
|
Comment on lines
409
to
411
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The same main merge brought back two things that 1ce60ef had removed:
Suggested change
|
||||||
| cli-conformance: | ||||||
| name: CLI Conformance | ||||||
| runs-on: ubuntu-latest | ||||||
| needs: build | ||||||
| timeout-minutes: 40 | ||||||
|
|
||||||
| # SPEC-028: drive the build artifact with the pinned python OpenStack CLI. | ||||||
| # The downloaded binary runs natively against a PostgreSQL service (no | ||||||
| # docker), so the suite always exercises exactly what the build job made. | ||||||
| services: | ||||||
| postgres: | ||||||
| image: postgres:18.4-alpine | ||||||
| env: | ||||||
| POSTGRES_USER: lightstack | ||||||
| POSTGRES_PASSWORD: secret | ||||||
| POSTGRES_DB: lightstack_test | ||||||
| ports: | ||||||
| - 5432:5432 | ||||||
| options: >- | ||||||
| --health-cmd "pg_isready -U lightstack -d lightstack_test" | ||||||
| --health-interval 5s | ||||||
| --health-timeout 5s | ||||||
| --health-retries 10 | ||||||
|
|
||||||
| steps: | ||||||
| - name: Checkout code | ||||||
| uses: actions/checkout@v6 | ||||||
|
|
||||||
| - name: Download lightstack binary | ||||||
| uses: actions/download-artifact@v5 | ||||||
| with: | ||||||
| name: lightstack-binary | ||||||
| path: bin/ | ||||||
|
|
||||||
| - name: Make binary executable | ||||||
| run: chmod +x bin/lightstack | ||||||
|
|
||||||
| - name: Install local storage tools | ||||||
| run: | | ||||||
| sudo apt-get update | ||||||
| sudo apt-get install -y qemu-utils | ||||||
|
|
||||||
| - name: Start lightstack server (native) | ||||||
| run: | | ||||||
| # The native server runs as the unprivileged runner. Keep its local | ||||||
| # image and volume backing in the job workspace, not /var/lib. | ||||||
| sed '/^state_path:/d' config/lightstack.yaml > "$RUNNER_TEMP/lightstack.yaml" | ||||||
| printf '\nstate_path: "%s"\n' "$RUNNER_TEMP/lightstack-state" >> "$RUNNER_TEMP/lightstack.yaml" | ||||||
| LIGHTSTACK_DB_URL="postgres://lightstack:secret@localhost:5432/lightstack_test?sslmode=disable" \ | ||||||
| LIGHTSTACK_JWT_SECRET="ci-secret" LIGHTSTACK_ENDPOINT_HOST="localhost" \ | ||||||
| nohup ./bin/lightstack --config "$RUNNER_TEMP/lightstack.yaml" --migrations migrations > /tmp/lightstack.log 2>&1 & | ||||||
|
|
||||||
| - name: Wait for lightstack to be ready | ||||||
| run: | | ||||||
| echo "Waiting for all Lightstack services to be ready..." | ||||||
|
|
||||||
| # Check with proper API version paths | ||||||
| timeout 120 bash -c ' | ||||||
| until curl -sf http://localhost:35357/v3 > /dev/null 2>&1 && \ | ||||||
| curl -sf http://localhost:8774/v2.1 > /dev/null 2>&1 && \ | ||||||
| curl -sf http://localhost:9696/v2.0 > /dev/null 2>&1 && \ | ||||||
| curl -sf http://localhost:8776/v3 > /dev/null 2>&1 && \ | ||||||
| curl -sf http://localhost:9292/v2 > /dev/null 2>&1; do | ||||||
| echo "⏳ Services not ready, retrying... ($(date +%H:%M:%S))" | ||||||
| sleep 3 | ||||||
| done | ||||||
| ' | ||||||
|
|
||||||
| echo "✅ All Lightstack services are ready" | ||||||
|
|
||||||
| - name: Install pinned OpenStack CLI | ||||||
| # A venv, not the system python: ubuntu-latest is PEP 668 | ||||||
| # externally-managed, and the harness resolves the CLI from PATH | ||||||
| # (command -v openstack), so the venv's bin dir joins GITHUB_PATH. | ||||||
| run: | | ||||||
| python3 -m venv /tmp/osc-venv | ||||||
| /tmp/osc-venv/bin/pip install -c test/cli/constraints.txt python-openstackclient | ||||||
| echo "/tmp/osc-venv/bin" >> "$GITHUB_PATH" | ||||||
|
|
||||||
| - name: Run CLI conformance suite (fast subset) | ||||||
| run: ./test/cli/bin/run.sh --fast | ||||||
| env: | ||||||
| OS_AUTH_URL: "http://localhost:35357/v3" | ||||||
| OS_USERNAME: "admin" | ||||||
| OS_PASSWORD: "secret" | ||||||
| OS_PROJECT_NAME: "default" | ||||||
| OS_USER_DOMAIN_NAME: "Default" | ||||||
| OS_PROJECT_DOMAIN_NAME: "Default" | ||||||
|
|
||||||
| - name: Check CLI conformance coverage | ||||||
| run: ./test/cli/bin/coverage.sh | ||||||
|
|
||||||
| - name: Dump server log on failure | ||||||
| if: failure() | ||||||
| run: tail -n 300 /tmp/lightstack.log | ||||||
|
|
||||||
| - name: Upload CLI conformance results | ||||||
| if: always() | ||||||
| uses: actions/upload-artifact@v7 | ||||||
| with: | ||||||
| name: cli-conformance-results | ||||||
| path: test/cli/results/ | ||||||
| retention-days: 7 | ||||||
|
|
||||||
| # The e2e surface is the CLI conformance suite: cli-conformance here (fast | ||||||
| # subset) and the nightly full + terraform runs (.github/workflows/nightly.yml). | ||||||
|
|
||||||
| helm-check: | ||||||
| name: Helm Charts | ||||||
| runs-on: ubuntu-latest | ||||||
|
|
@@ -493,7 +586,7 @@ jobs: | |||||
| status-check: | ||||||
| name: CI Status | ||||||
| runs-on: ubuntu-latest | ||||||
| needs: [build, unit-tests, contract-tests, contract-tests-sqlite, integration-tests, dco-check, rgw-integration-tests, helm-check] | ||||||
| needs: [build, unit-tests, contract-tests, contract-tests-sqlite, integration-tests, cli-conformance, dco-check, rgw-integration-tests, helm-check] | ||||||
| if: always() | ||||||
|
|
||||||
| steps: | ||||||
|
|
@@ -504,6 +597,7 @@ jobs: | |||||
| echo "Contract Tests: ${{ needs.contract-tests.result }}" | ||||||
| echo "Contract Tests (SQLite): ${{ needs.contract-tests-sqlite.result }}" | ||||||
| echo "Integration Tests: ${{ needs.integration-tests.result }}" | ||||||
| echo "CLI Conformance: ${{ needs.cli-conformance.result }}" | ||||||
| echo "SAP Sign-off: ${{ needs.dco-check.result }}" | ||||||
| echo "RGW Integration Tests: ${{ needs.rgw-integration-tests.result }}" | ||||||
| echo "Helm Charts: ${{ needs.helm-check.result }}" | ||||||
|
|
@@ -535,7 +629,12 @@ jobs: | |||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| if [ "${{ needs.dco-check.result }}" != "success" ]; then | ||||||
| if [ "${{ needs.cli-conformance.result }}" != "success" ]; then | ||||||
| echo "❌ CLI conformance suite failed" | ||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| if [ "${{ github.event_name }}" = "pull_request" ] && [ "${{ needs.dco-check.result }}" != "success" ]; then | ||||||
| echo "❌ SAP sign-off check failed — all commits need Signed-off-by: name@sap.com" | ||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
redisservice that 8b039d1 added to this job, so thatTestGetNetworkCacheDoesNotCrossRoles_DB"actually run[s] in CI", didn't survive the main merge (9674ebd). That test is the regression test for the admin carve-out's cache poisoning. It now skips on every CI run (t.Skipf("no local Redis …")), as do the PR'sTestAddImageTagEvictsCacheandTestDeleteImageTagEvictsCache. All three pass against a Redis container on PostgreSQL and SQLite, and skip without one. Re-adding the service under this job'sservices:brings them back: