Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
19edcb6
docs(specs): add SPEC-028 plan — OpenStack CLI conformance test envir…
Oct 15, 2026
ef2a63e
test(cli): add the OpenStack CLI conformance harness core (SPEC-028 p…
Oct 15, 2026
2b4ed22
fix(glance): honor the standard v2 image list filters and persist tag…
Oct 15, 2026
f2b03fd
fix(nova): four conformance gaps found by the SPEC-028 CLI suite
Oct 15, 2026
2807f13
fix(neutron): honour the admin tenant filter on network list and the …
Oct 15, 2026
cfacd1b
fix(keystone): seven conformance gaps found by the SPEC-028 CLI suite
Oct 15, 2026
8886434
fix(cinder): seven conformance gaps found by the SPEC-028 CLI suite
Oct 15, 2026
6154505
test(cli): sweep volumes with plain delete before --force
Oct 15, 2026
1ce60ef
ci: gate the CLI conformance suite on every PR, nightly full + terraf…
Oct 15, 2026
9551d9b
test(terraform): apply/destroy the unmodified compat module against a…
Oct 15, 2026
730015e
test(cli): conformance suites for compute, glance, identity, volume a…
Oct 15, 2026
649ec97
ci(nightly): install the pinned CLI for the terraform conformance job
Oct 16, 2026
2b7a6f3
fix(git): track the CLI harness gate scripts
Oct 16, 2026
4c97eed
fix(neutron,keystone,cinder): three review follow-ups from the SPEC-0…
Oct 16, 2026
f510cd1
fix(cli): three harness defects found by the SPEC-028 review pass
Oct 16, 2026
a076e22
ci: install the CLI in a venv and run the nightly suite in strict mode
Oct 16, 2026
4cd0e41
test(terraform): preserve failure logs and tighten the residue check
Oct 16, 2026
893bd1e
test(cli): assert the fixed compute/volume behaviour outright
Oct 16, 2026
ccc4574
fix(neutron): nine CLI-conformance divergences found by the network s…
Oct 16, 2026
9632372
test(cli): network suite — 41 cases, all green against the fixed server
Oct 16, 2026
956ba5e
docs(cli): document run.sh --strict in the Running section
Oct 16, 2026
9d24bca
fix(keystone): two more CLI-conformance findings from the full run
Oct 16, 2026
9592cfe
fix: review round 2 — CI terraform job, residue gate, rollbacks, stal…
Oct 16, 2026
1100f7f
fix: review round 3 — last stale-doc stragglers and a test-comment claim
Oct 16, 2026
a1e0ea6
fix: comprehensive review round 4 — scoping, atomicity, timeouts, har…
Oct 16, 2026
f4b9ebb
fix(neutron): keep the admin carve-out out of the project-scoped cache
Oct 16, 2026
8b039d1
ci: run the cache regression test in CI; pin its cache-hit claim
Oct 16, 2026
a78870d
test(cli): drop a dead known-failure probe in the token-revoke neg case
Oct 16, 2026
391690c
test(cli): fail closed on corrupt results; keep standalone suites run…
Oct 16, 2026
fb29751
fix: comprehensive review round 7 — scoping symmetry, NULL scans, cac…
Oct 16, 2026
f2c348e
test(cli): round-8 delta polish — exit before leasing, subnet probe, …
Oct 16, 2026
c7e167a
fix: round-8 delta review — ownership labels, show carve-outs, panic …
Oct 16, 2026
f82dd16
fix: round-9 review — dead failure check, vacuous pin, unshelve resta…
Oct 16, 2026
9a0ab5f
fix: round-10 review — self-heal persists the revived domain, 078, pa…
Oct 16, 2026
725c173
fix: resolve conformance lint failures
senolcolak Oct 30, 2026
1306ed9
fix(ci): repair conformance jobs and validate coverage evidence
senolcolak Oct 30, 2026
127a822
fix(ci): pin compatible Helm and patched Go toolchains
senolcolak Oct 30, 2026
e5c34b5
fix(test): require reliable Terraform cleanup evidence
senolcolak Oct 30, 2026
f0a70f9
fix(ci): avoid overlapping Go cache restores
senolcolak Oct 30, 2026
5708584
fix(cinder): retain project filter with all-tenants requests
senolcolak Oct 30, 2026
8397cff
fix(cli): baseline known failures; fix shared-pool attribution and ti…
Nov 2, 2026
9674ebd
Merge current main into CLI conformance branch
senolcolak Nov 11, 2026
083410e
fix: repair conformance branch after main integration
senolcolak Nov 11, 2026
6afe955
merge: integrate main security fixes into CLI conformance
senolcolak Nov 13, 2026
c40d58e
fix(database): align server-group unique indexes across backends
senolcolak Nov 13, 2026
a754139
fix: preserve resources and fail closed in conformance workflows
senolcolak Nov 13, 2026
54fd0b1
fix: preserve response metadata and tighten conformance evidence
senolcolak Nov 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
167 changes: 133 additions & 34 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,9 @@ jobs:
uses: actions/setup-go@v6
with:
go-version: 'stable'

- name: Cache Go modules
uses: actions/cache@v5
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-go-
cache-dependency-path: |
go.sum
tools/go.sum

- name: Build binary
run: make build
Expand All @@ -46,6 +41,12 @@ jobs:
- name: Run linters
run: make lint

- name: Test CLI harness reporting and coverage gates
run: python3 -m unittest discover -s test/cli -p 'test_*.py' -v

- name: Test Terraform harness evidence gates
run: python3 -m unittest discover -s test/terraform -p 'test_*.py' -v

- name: Upload binary artifact
uses: actions/upload-artifact@v7
with:
Expand Down Expand Up @@ -84,14 +85,9 @@ jobs:
uses: actions/setup-go@v6
with:
go-version: 'stable'

- name: Cache Go modules
uses: actions/cache@v5
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-go-
cache-dependency-path: |
go.sum
tools/go.sum
Comment on lines +88 to +90

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The redis service that 8b039d1 added to this job, so that TestGetNetworkCacheDoesNotCrossRoles_DB "actually run[s] in CI", didn't survive the main merge (9674ebd). That test is the regression test for the admin carve-out's cache poisoning. It now skips on every CI run (t.Skipf("no local Redis …")), as do the PR's TestAddImageTagEvictsCache and TestDeleteImageTagEvictsCache. All three pass against a Redis container on PostgreSQL and SQLite, and skip without one. Re-adding the service under this job's services: brings them back:

      redis:
        image: redis:7-alpine
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s
          --health-timeout 5s
          --health-retries 10


- name: Install qemu-img for image normalization tests
run: sudo apt-get update && sudo apt-get install -y --no-install-recommends qemu-utils
Expand Down Expand Up @@ -129,14 +125,9 @@ jobs:
uses: actions/setup-go@v6
with:
go-version: 'stable'

- name: Cache Go modules
uses: actions/cache@v5
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-go-
cache-dependency-path: |
go.sum
tools/go.sum

- name: Start lightstack services
run: |
Expand Down Expand Up @@ -250,14 +241,9 @@ jobs:
uses: actions/setup-go@v6
with:
go-version: 'stable'

- name: Cache Go modules
uses: actions/cache@v5
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-go-
cache-dependency-path: |
go.sum
tools/go.sum

- name: Build
run: make build
Expand Down Expand Up @@ -423,6 +409,113 @@ jobs:
# TODO: Add e2e-fast and e2e-full jobs once E2E test scenarios are implemented.
# Planned: VM lifecycle, volume attach/detach, network isolation, Terraform validation, Horizon UI tests.

Comment on lines 409 to 411

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same main merge brought back two things that 1ce60ef had removed:

  • This TODO, which now contradicts the comment at line 516 ("The e2e surface is the CLI conformance suite").
  • The unpinned python3 -m pip install python-openstackclient in integration-tests (line 328). Nothing in that job uses it: quick_test.sh makes no openstack call.
Suggested change
# TODO: Add e2e-fast and e2e-full jobs once E2E test scenarios are implemented.
# Planned: VM lifecycle, volume attach/detach, network isolation, Terraform validation, Horizon UI tests.

cli-conformance:
name: CLI Conformance
runs-on: ubuntu-latest
needs: build
timeout-minutes: 40

# SPEC-028: drive the build artifact with the pinned python OpenStack CLI.
# The downloaded binary runs natively against a PostgreSQL service (no
# docker), so the suite always exercises exactly what the build job made.
services:
postgres:
image: postgres:18.4-alpine
env:
POSTGRES_USER: lightstack
POSTGRES_PASSWORD: secret
POSTGRES_DB: lightstack_test
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U lightstack -d lightstack_test"
--health-interval 5s
--health-timeout 5s
--health-retries 10

steps:
- name: Checkout code
uses: actions/checkout@v6

- name: Download lightstack binary
uses: actions/download-artifact@v5
with:
name: lightstack-binary
path: bin/

- name: Make binary executable
run: chmod +x bin/lightstack

- name: Install local storage tools
run: |
sudo apt-get update
sudo apt-get install -y qemu-utils

- name: Start lightstack server (native)
run: |
# The native server runs as the unprivileged runner. Keep its local
# image and volume backing in the job workspace, not /var/lib.
sed '/^state_path:/d' config/lightstack.yaml > "$RUNNER_TEMP/lightstack.yaml"
printf '\nstate_path: "%s"\n' "$RUNNER_TEMP/lightstack-state" >> "$RUNNER_TEMP/lightstack.yaml"
LIGHTSTACK_DB_URL="postgres://lightstack:secret@localhost:5432/lightstack_test?sslmode=disable" \
LIGHTSTACK_JWT_SECRET="ci-secret" LIGHTSTACK_ENDPOINT_HOST="localhost" \
nohup ./bin/lightstack --config "$RUNNER_TEMP/lightstack.yaml" --migrations migrations > /tmp/lightstack.log 2>&1 &

- name: Wait for lightstack to be ready
run: |
echo "Waiting for all Lightstack services to be ready..."

# Check with proper API version paths
timeout 120 bash -c '
until curl -sf http://localhost:35357/v3 > /dev/null 2>&1 && \
curl -sf http://localhost:8774/v2.1 > /dev/null 2>&1 && \
curl -sf http://localhost:9696/v2.0 > /dev/null 2>&1 && \
curl -sf http://localhost:8776/v3 > /dev/null 2>&1 && \
curl -sf http://localhost:9292/v2 > /dev/null 2>&1; do
echo "⏳ Services not ready, retrying... ($(date +%H:%M:%S))"
sleep 3
done
'

echo "✅ All Lightstack services are ready"

- name: Install pinned OpenStack CLI
# A venv, not the system python: ubuntu-latest is PEP 668
# externally-managed, and the harness resolves the CLI from PATH
# (command -v openstack), so the venv's bin dir joins GITHUB_PATH.
run: |
python3 -m venv /tmp/osc-venv
/tmp/osc-venv/bin/pip install -c test/cli/constraints.txt python-openstackclient
echo "/tmp/osc-venv/bin" >> "$GITHUB_PATH"

- name: Run CLI conformance suite (fast subset)
run: ./test/cli/bin/run.sh --fast
env:
OS_AUTH_URL: "http://localhost:35357/v3"
OS_USERNAME: "admin"
OS_PASSWORD: "secret"
OS_PROJECT_NAME: "default"
OS_USER_DOMAIN_NAME: "Default"
OS_PROJECT_DOMAIN_NAME: "Default"

- name: Check CLI conformance coverage
run: ./test/cli/bin/coverage.sh

- name: Dump server log on failure
if: failure()
run: tail -n 300 /tmp/lightstack.log

- name: Upload CLI conformance results
if: always()
uses: actions/upload-artifact@v7
with:
name: cli-conformance-results
path: test/cli/results/
retention-days: 7

# The e2e surface is the CLI conformance suite: cli-conformance here (fast
# subset) and the nightly full + terraform runs (.github/workflows/nightly.yml).

helm-check:
name: Helm Charts
runs-on: ubuntu-latest
Expand Down Expand Up @@ -493,7 +586,7 @@ jobs:
status-check:
name: CI Status
runs-on: ubuntu-latest
needs: [build, unit-tests, contract-tests, contract-tests-sqlite, integration-tests, dco-check, rgw-integration-tests, helm-check]
needs: [build, unit-tests, contract-tests, contract-tests-sqlite, integration-tests, cli-conformance, dco-check, rgw-integration-tests, helm-check]
if: always()

steps:
Expand All @@ -504,6 +597,7 @@ jobs:
echo "Contract Tests: ${{ needs.contract-tests.result }}"
echo "Contract Tests (SQLite): ${{ needs.contract-tests-sqlite.result }}"
echo "Integration Tests: ${{ needs.integration-tests.result }}"
echo "CLI Conformance: ${{ needs.cli-conformance.result }}"
echo "SAP Sign-off: ${{ needs.dco-check.result }}"
echo "RGW Integration Tests: ${{ needs.rgw-integration-tests.result }}"
echo "Helm Charts: ${{ needs.helm-check.result }}"
Expand Down Expand Up @@ -535,7 +629,12 @@ jobs:
exit 1
fi

if [ "${{ needs.dco-check.result }}" != "success" ]; then
if [ "${{ needs.cli-conformance.result }}" != "success" ]; then
echo "❌ CLI conformance suite failed"
exit 1
fi

if [ "${{ github.event_name }}" = "pull_request" ] && [ "${{ needs.dco-check.result }}" != "success" ]; then
echo "❌ SAP sign-off check failed — all commits need Signed-off-by: name@sap.com"
exit 1
fi
Expand Down
Loading
Loading