diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a5c7c839..78e9e3dc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -93,6 +93,9 @@ jobs: restore-keys: | ${{ runner.os }}-go- + - name: Install qemu-img for image normalization tests + run: sudo apt-get update && sudo apt-get install -y --no-install-recommends qemu-utils + - name: Run unit tests with race detector run: go test -race -coverprofile=coverage.txt -covermode=atomic $(go list ./... | grep -v '/test/contract') env: diff --git a/CHANGELOG.md b/CHANGELOG.md index b26f2fbf..f61084ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 --- +## [Unreleased] + +### 🔒 Security — identity & service authorization +- **Keystone management surface authorized**: the identity API was authentication-only, so any member could reset any user's password (including the admin's) via `PATCH /users/:id`, self-grant the admin role via role assignment, dump every tenant's recoverable EC2/S3 secret via the credentials API, mint application credentials for other users, and enumerate users/projects/roles/groups/domains/catalog. User/project/role/group/domain/service/endpoint management is now admin-only; user-scoped subresources are self-or-admin; PATCH of one's own password/enabled stays admin-only so ChangePassword keeps its original-password contract. Role assignment also 404s on nonexistent project/user/role ids instead of silently recording garbage. +- **Glance image upload ownership**: uploading bytes to a public image was allowed for any project that could *see* it — image poisoning of an operator-staged base every tenant boots. Upload now follows the same owner-or-admin rule as delete (403 visible-but-foreign, 404 invisible). +- **Glance tasks & cache**: tasks are owner-scoped and no longer echo internal errors; cache management (prefetch/evict/clear) is admin-only. +- **Nova flavor catalog**: flavor create/delete, extra-specs and tenant-access grants are admin-only; flavor limits match Flamingo's signed 32-bit range, including zero root disk for boot-from-volume. +- **Nova deployment views**: `os-hypervisors` (index/detail/statistics), `os-services` and `os-console-auth-tokens` require the admin role because they expose deployment-wide state. +- **Cinder operator surfaces**: volume types (incl. project-access grants), QoS specs and manage/unmanage are admin-only; volume-type reads stay open for tenants. +- **Neutron shared networks**: `shared=true` on create is admin-only, like `router:external`. +- **S3 credential scope and revocation**: editable credential blobs cannot override the recorded project, and S3/EC2 authentication refuses missing or revoked project roles and failed role lookups. +- **Identity self-service UUIDs**: uppercase and compact UUIDs identify the same user for authorization, including credentials and profile updates. +- **Volume-transfer auth keys** are compared in constant time. + +### 🐛 Correctness +- **Glance checksums describe the stored bytes**: normalizing backends rewrite a qcow2 upload to a raw base, but the recorded checksum hashed the upload stream, so `EnsureLocalBase` failed on every node without the local base and boot-from-image errored. The checksum is now taken from the stored object; unreadable stored data or failed state writes fail the upload instead of returning success. Concurrent uploads atomically claim the queued image, and failed uploads reset it for retry. +- **Neutron IP allocation is collision-free**: allocation drew purely at random, so ports in one subnet could land on the same address; allocation and insertion share a subnet lock across port creation, server boot and interface attach. Canonical IPv6 comparisons prevent duplicate spellings, a complete fallback finds the last free address, and tiny subnets (≤ /29) no longer panic `crypto/rand.Int`. +- **Metadata `network_data.json` works**: the query selected a nonexistent `fixed_ip` column and answered 500 to every guest; it now resolves each port's JSONB addresses with their own subnets: one link per port, unique network IDs, correct IPv4/IPv6 masks, routes and DNS services, and empty arrays instead of null. +- **Nova volume attach/detach are atomic**: single transactions keep attachments and volume state consistent. Attach serializes device assignment per server and uses the existing transaction connection, avoiding pool deadlock and device collisions. +- **Nova boot port lifecycle**: a failed multi-network boot releases the ports it already allocated; server delete deletes only the ports Nova created and *detaches* tenant-created ports instead of destroying them. +- **Keypair name races** map the unique-constraint violation to the same 409 as the pre-check. +- **Request bodies bounded**: glance upload/stage enforce the 10 GiB cap with bounded memory; staging accepts OpenStack's PUT method; imported SSH public keys are capped at 64 KiB. + +--- + ## [0.6.1] - 2026-04-22 ### 📋 v0.7.0 Planning & Specifications @@ -59,11 +84,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### 📊 Observability - **Zerolog migration**: Replace remaining `log.Printf` calls with structured `zerolog` logging (M-1) -- **Scan error logging**: Add warnings for previously silent `rows.Scan` failures across all list handlers (M-4) +- **Scan error logging**: Add warnings for previously silent `rows.Scan` failures across all list handlers - **Swallowed error propagation**: Log or propagate 15 instances of silently discarded errors (M-6) ### 🛠️ Utilities & Fixes -- **Shared pagination helper**: `common.ParsePagination()` extracts duplicated limit/offset/marker parsing (M-3) +- **Shared pagination helper**: `common.ParsePagination()` extracts duplicated limit/offset/marker parsing - **Version consistency**: Fix Placement (1.39→1.40) and Nova (2.79→2.90) version mismatches with constants (H-10, L-4) - **Quota queries**: `GetLimits` reads project quotas from database instead of hardcoded values (H-12) - **Flavor pagination**: Replace UUID-based cursor with `created_at` for deterministic ordering (M-10) @@ -85,7 +110,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Updated INDEX.md**: Add Testing & Quality section with links to all moved files ### Deferred to v0.7.0 -- Database dependency injection (H-3) — 665 call sites, needs dedicated design +- Database dependency injection — 665 call sites, needs dedicated design - Rate limiting (H-6) — needs design spec for per-project limits - Re-enable CI linting (H-4) — needs lint error cleanup first diff --git a/docs/review/pr-104-follow-up-2026-11-06.md b/docs/review/pr-104-follow-up-2026-11-06.md new file mode 100644 index 00000000..88d70d6c --- /dev/null +++ b/docs/review/pr-104-follow-up-2026-11-06.md @@ -0,0 +1,30 @@ +# PR #104 follow-up review — 2026-11-06 + +Reviewed the authorization and correctness changes at `e026de8`, including their failure and concurrency paths. The following additional findings are fixed on the same PR. + +| Area | Reproduced failure | Fix | +|---|---|---| +| S3 credentials | Editing one's own generic credential blob changed the project returned by S3 authentication. | Authenticate against the recorded project; use the blob's tenant only for legacy rows without a project column. EC2 credential views use the same precedence. | +| S3 role revocation | A correctly signed request still returned 200 after the user's project role was removed. | Require a current project role and fail closed on role-query errors. | +| Identity self-service | Uppercase and compact UUID spellings returned 403 for the user's own profile and application credentials. | Compare parsed UUID values while preserving authorization-before-validation ordering. | +| Volume attachment | One database connection caused a timeout; concurrent attachments to different volumes collided on the device uniqueness constraint. | Lock the instance before choosing a device and use the transaction's connection for the device query. | +| Concurrent port allocation | Twelve concurrent requests through Neutron create, Nova boot allocation and Nova interface attach created eight ports holding one address. | Hold a shared subnet row lock across address selection and insertion in all three paths. | +| Dense subnets and IPv6 | A subnet with one free address falsely reported exhaustion; expanded and compressed IPv6 addresses were treated as different claims. | Complete the random search with a bounded sequential scan and canonicalize addresses. Gateway lookup errors now fail allocation. | +| Image staging | OpenStack's PUT route returned 404; an 8 MiB body caused a growing read buffer of over 2 MiB despite the upload cap. | Register PUT, stream with bounded memory, preserve 413, and prevent staging from overwriting a concurrently changed image state. | +| Image upload finalization | Injected database write failures still returned 204, sometimes leaving the image in saving. The normalized-checksum error branch substituted the original stream hash. | Atomically claim queued images, fail unsuccessful activation, require the stored checksum and reset unsuccessful uploads using a bounded cleanup context. | +| Guest network metadata | Multiple addresses on one port shared a network ID; addressless ports returned null networks; IPv6 was labeled IPv4. Routes and DNS lacked the fields consumed by cloud-init. | Unique IDs, proper IPv4/IPv6 masks, routes and DNS services, and empty arrays. | +| Flavor validation | Valid zero-disk and large flavors were rejected by arbitrary limits. | Match Flamingo's signed 32-bit bounds, positive RAM/vCPU requirements and nonnegative disk sizes. | + +The metadata shape and flavor bounds were checked against OpenStack Nova's `stable/2025.2` implementations in [netutils.py](https://github.com/openstack/nova/blob/stable/2025.2/nova/virt/netutils.py) and [parameter_types.py](https://github.com/openstack/nova/blob/stable/2025.2/nova/api/validation/parameter_types.py). + +## Validation + +- Linux, Go 1.27.3 and isolated PostgreSQL 18.3. +- Full unit suite with the race detector and database tests enabled passed. Packages changed afterward were rerun with the race detector. +- Pinned `make lint`: zero issues. Linux binary build passed. +- Horizon create-UI API probe: 85 passed, zero failed. This is an API probe, not a browser test. +- Live contract suite: 308 passed, 12 skipped, five failed. Running the original PR head in a separate database produced exactly the same counts and failing tests. These fixtures invoke a hardcoded `docker exec lightstack-postgres`; the isolated test container does not provide that command or access to the user's Docker containers. + +The five unchanged fixture failures are `TestCinderUnmanageVolume_Contract`, `TestNeutronGetAgent_Contract`, `TestNeutronAddL3AgentToRouter_Contract`, `TestTrunkBasicCRUD`, and `TestTrunkSubports`. The Cinder fixture panics when parsing the missing command's output. This local contract run is not a complete pass; the repository's CI environment remains the full contract gate. + +No real libvirt VM, host networking or Ceph RGW deployment was exercised locally. PostgreSQL-backed handler tests cover the authorization, transaction and error paths above; existing storage tests cover the real qemu-img conversion. diff --git a/internal/cinder/manage.go b/internal/cinder/manage.go index 3bf6a33d..506119ab 100644 --- a/internal/cinder/manage.go +++ b/internal/cinder/manage.go @@ -15,6 +15,15 @@ import ( // ManageVolume handles POST /v3/os-volume-manage func (svc *Service) ManageVolume(c *gin.Context) { + // Admin-only: volume types, QoS specs and the + // manage/unmanage surface shape storage for every tenant and address + // host-level backends (upstream: volume_extension:* and + // volume_extension:volume_admin_actions:*, all admin_api). Checked before + // the body is bound so a non-admin gets 403, not a 400. + if !common.RequireAdmin(c, "cinder:ManageVolume") { + return + } + projectID := c.GetString("project_id") var req struct { @@ -74,6 +83,15 @@ func (svc *Service) ManageVolume(c *gin.Context) { // ListManageableVolumes handles GET /v3/manageable_volumes func (svc *Service) ListManageableVolumes(c *gin.Context) { + // Admin-only: volume types, QoS specs and the + // manage/unmanage surface shape storage for every tenant and address + // host-level backends (upstream: volume_extension:* and + // volume_extension:volume_admin_actions:*, all admin_api). Checked before + // the body is bound so a non-admin gets 403, not a 400. + if !common.RequireAdmin(c, "cinder:ListManageableVolumes") { + return + } + host := c.Query("host") if host == "" { @@ -90,6 +108,15 @@ func (svc *Service) ListManageableVolumes(c *gin.Context) { // UnmanageVolume handles volume action: os-unmanage func (svc *Service) UnmanageVolume(c *gin.Context, volumeID string) { + // Admin-only: volume types, QoS specs and the + // manage/unmanage surface shape storage for every tenant and address + // host-level backends (upstream: volume_extension:* and + // volume_extension:volume_admin_actions:*, all admin_api). Checked before + // the body is bound so a non-admin gets 403, not a 400. + if !common.RequireAdmin(c, "cinder:UnmanageVolume") { + return + } + projectID := c.GetString("project_id") // Verify volume exists and belongs to project @@ -136,6 +163,14 @@ func (svc *Service) UnmanageVolume(c *gin.Context, volumeID string) { // ManageSnapshot handles POST /v3/os-snapshot-manage func (svc *Service) ManageSnapshot(c *gin.Context) { + // Admin-only: like os-volume-manage, this imports + // an operator's existing backing into the API — upstream gates it + // volume_admin_actions:snapshot_manage. Checked before the body is bound + // so a non-admin gets 403, not a 400. + if !common.RequireAdmin(c, "cinder:ManageSnapshot") { + return + } + projectID := c.GetString("project_id") var req struct { diff --git a/internal/cinder/manage_authz_test.go b/internal/cinder/manage_authz_test.go new file mode 100644 index 00000000..4a7c91d1 --- /dev/null +++ b/internal/cinder/manage_authz_test.go @@ -0,0 +1,84 @@ +package cinder + +import ( + "context" + "net/http" + "strings" + "testing" + + "github.com/google/uuid" +) + +// TestOperatorSurfacesAdminOnly pins the admin gates on volume types, QoS +// specs and the manage/unmanage surface shape storage for every tenant (and +// manage addresses host-level backends). They were authentication-only — a +// plain member could create volume types every tenant then booted from, add +// its project to a private type, and import host backings into its quota. +// Reads of volume types stay open: tenants need them to pick one at create. +func TestOperatorSurfacesAdminOnly(t *testing.T) { + requireDB(t) + r := newCinderTestRouter() + + const project = "00000000-0000-0000-0000-000000000002" // seeded default project + member := map[string]string{"X-Test-Project": project, "X-Test-Roles": "member"} + admin := map[string]string{"X-Test-Project": project, "X-Test-Roles": "admin"} + + memberDenied := []struct{ name, method, path, body string }{ + {"create volume type", http.MethodPost, "/v3/types", `{"volume_type":{"name":"member-type"}}`}, + {"manage a volume", http.MethodPost, "/v3/os-volume-manage", `{"volume":{"host":"h1","ref":{"source-name":"x"}}}`}, + {"manage a snapshot", http.MethodPost, "/v3/os-snapshot-manage", `{"snapshot":{"volume_id":"00000000-0000-0000-0000-000000000000","ref":{"source-name":"x"}}}`}, + {"list manageable volumes", http.MethodGet, "/v3/manageable_volumes", ""}, + {"create a QoS spec", http.MethodPost, "/v3/qos-specs", `{"qos_specs":{"name":"member-qos"}}`}, + } + for _, tc := range memberDenied { + t.Run(tc.name, func(t *testing.T) { + if code, body := doReq(r, tc.method, tc.path, tc.body, member); code != http.StatusForbidden { + t.Errorf("%s %s as member: code=%d body=%s; want 403", tc.method, tc.path, code, body) + } + }) + } + + // The type catalog stays readable for members (they pick a type at create). + if code, body := doReq(r, http.MethodGet, "/v3/types", "", member); code != http.StatusOK { + t.Errorf("GET /v3/types as member: code=%d body=%s; want 200 (reads stay open)", code, body) + } + + // And the admin keeps the write paths. Unique name + cleanup: the volume + // types table is shared across runs of the shared test database. + typeName := "authz-admin-type-" + uuid.NewString()[:8] + if code, body := doReq(r, http.MethodPost, "/v3/types", `{"volume_type":{"name":"`+typeName+`"}}`, admin); code != http.StatusCreated && code != http.StatusOK { + t.Fatalf("create volume type as admin: code=%d body=%s; want 2xx", code, body) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM volume_types WHERE name = $1`, typeName) + }) +} + +// Round 2 of this review: a volume type created without a description has a +// NULL one, and ListVolumeTypes scanned it into a plain string — the row +// failed to scan and the type silently vanished from the catalog (and +// GetVolumeType 500'd). The list must serve both descriptions and their +// absence. +func TestListVolumeTypesServesNullDescription(t *testing.T) { + requireDB(t) + r := newCinderTestRouter() + + const project = "00000000-0000-0000-0000-000000000002" + admin := map[string]string{"X-Test-Project": project, "X-Test-Roles": "admin"} + + typeName := "authz-undescribed-" + uuid.NewString()[:8] + code, body := doReq(r, http.MethodPost, "/v3/types", `{"volume_type":{"name":"`+typeName+`"}}`, admin) + if code != http.StatusCreated && code != http.StatusOK { + t.Fatalf("create undescribed volume type: code=%d body=%s", code, body) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM volume_types WHERE name = $1`, typeName) + }) + + if code, body := doReq(r, http.MethodGet, "/v3/types", "", admin); code != http.StatusOK { + t.Fatalf("list volume types: code=%d body=%s", code, body) + } + if !strings.Contains(body, typeName) { + t.Errorf("a volume type created without a description is missing from GET /v3/types (the NULL-description scan drop): %s", body) + } +} diff --git a/internal/cinder/operator_authz_regression_test.go b/internal/cinder/operator_authz_regression_test.go new file mode 100644 index 00000000..7fa8031b --- /dev/null +++ b/internal/cinder/operator_authz_regression_test.go @@ -0,0 +1,95 @@ +package cinder + +import ( + "context" + "fmt" + "net/http" + "testing" + + "github.com/google/uuid" +) + +func TestOperatorUpdateDeleteAndUnmanageRoutesRequireAdmin(t *testing.T) { + requireDB(t) + r := newCinderTestRouter() + member := map[string]string{"X-Test-Project": seededProjectID, "X-Test-User": seededUserID, "X-Test-Roles": "member"} + admin := map[string]string{"X-Test-Project": seededProjectID, "X-Test-User": seededUserID, "X-Test-Roles": "admin"} + + typeID, qosID := uuid.NewString(), uuid.NewString() + volumeID, unmanageVolumeID, resetVolumeID, snapshotID := uuid.NewString(), uuid.NewString(), uuid.NewString(), uuid.NewString() + if _, err := testPool.Exec(t.Context(), "INSERT INTO volume_types (id, name, description, is_public) VALUES ($1, $2, '', false)", typeID, "review-type-"+typeID[:8]); err != nil { + t.Fatalf("seed volume type: %v", err) + } + if _, err := testPool.Exec(t.Context(), "INSERT INTO qos_specs (id, name, consumer, specs, project_id) VALUES ($1, $2, 'back-end', '{}', $3)", qosID, "review-qos-"+qosID[:8], seededProjectID); err != nil { + t.Fatalf("seed QoS spec: %v", err) + } + for _, volume := range []struct{ id, name string }{ + {volumeID, "review-managed-snapshot-" + volumeID[:8]}, + {unmanageVolumeID, "review-unmanage-" + unmanageVolumeID[:8]}, + {resetVolumeID, "review-reset-" + resetVolumeID[:8]}, + } { + if _, err := testPool.Exec(t.Context(), "INSERT INTO volumes (id, name, project_id, user_id, size_gb, status) VALUES ($1, $2, $3, $4, 1, 'available')", volume.id, volume.name, seededProjectID, seededUserID); err != nil { + t.Fatalf("seed volume: %v", err) + } + } + if _, err := testPool.Exec(t.Context(), "INSERT INTO snapshots (id, name, volume_id, project_id, size_gb, status) VALUES ($1, 'review-snapshot-reset', $2, $3, 1, 'creating')", snapshotID, resetVolumeID, seededProjectID); err != nil { + t.Fatalf("seed snapshot: %v", err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), "DELETE FROM snapshots WHERE id = $1 OR volume_id = $2", snapshotID, volumeID) + _, _ = testPool.Exec(context.Background(), "DELETE FROM volumes WHERE id IN ($1, $2, $3)", volumeID, unmanageVolumeID, resetVolumeID) + _, _ = testPool.Exec(context.Background(), "DELETE FROM qos_specs WHERE id = $1", qosID) + _, _ = testPool.Exec(context.Background(), "DELETE FROM volume_types WHERE id = $1", typeID) + }) + + manageSnapshotBody := fmt.Sprintf("{\"snapshot\":{\"volume_id\":%q,\"ref\":{\"source-name\":\"source\"},\"name\":\"review-import-%s\"}}", volumeID, volumeID[:8]) + memberCases := []struct{ name, method, path, body string }{ + {"type update", http.MethodPut, "/v3/types/" + typeID, "{}"}, + {"type delete", http.MethodDelete, "/v3/types/" + typeID, ""}, + {"extra specs create", http.MethodPost, "/v3/types/" + typeID + "/extra_specs", "{}"}, + {"extra spec update", http.MethodPut, "/v3/types/" + typeID + "/extra_specs/review", "{}"}, + {"extra spec delete", http.MethodDelete, "/v3/types/" + typeID + "/extra_specs/review", ""}, + {"type access grant", http.MethodPost, "/v3/types/" + typeID + "/action", "{}"}, + {"QoS create", http.MethodPost, "/v3/qos-specs", "{}"}, + {"QoS update", http.MethodPut, "/v3/qos-specs/" + qosID, "{}"}, + {"QoS delete", http.MethodDelete, "/v3/qos-specs/" + qosID, ""}, + {"manage volume", http.MethodPost, "/v3/os-volume-manage", "{}"}, + {"list manageable volumes", http.MethodGet, "/v3/manageable_volumes?host=h1", ""}, + {"manage snapshot", http.MethodPost, "/v3/os-snapshot-manage", manageSnapshotBody}, + {"unmanage volume", http.MethodPost, "/v3/volumes/" + unmanageVolumeID + "/action", `{"os-unmanage":{}}`}, + {"reset volume status", http.MethodPost, "/v3/volumes/" + resetVolumeID + "/action", `{"os-reset_status":{"status":"available"}}`}, + {"reset snapshot status", http.MethodPost, "/v3/snapshots/" + snapshotID + "/action", `{"os-reset_status":{"status":"available"}}`}, + } + for _, tc := range memberCases { + t.Run(tc.name, func(t *testing.T) { + if code, body := doReq(r, tc.method, tc.path, tc.body, member); code != http.StatusForbidden { + t.Errorf("member %s %s: code=%d body=%s; want 403", tc.method, tc.path, code, body) + } + }) + } + + checkAdmin := func(method, path, body string, want int) { + t.Helper() + if code, response := doReq(r, method, path, body, admin); code != want { + t.Errorf("admin %s %s: code=%d body=%s; want %d", method, path, code, response, want) + } + } + checkAdmin(http.MethodPut, "/v3/types/"+typeID, "{\"volume_type\":{\"description\":\"admin\"}}", http.StatusOK) + checkAdmin(http.MethodPost, "/v3/types/"+typeID+"/extra_specs", "{\"extra_specs\":{\"review\":\"admin\"}}", http.StatusOK) + checkAdmin(http.MethodPut, "/v3/types/"+typeID+"/extra_specs/review", "{\"review\":\"updated\"}", http.StatusOK) + checkAdmin(http.MethodDelete, "/v3/types/"+typeID+"/extra_specs/review", "", http.StatusNoContent) + checkAdmin(http.MethodPost, "/v3/types/"+typeID+"/action", fmt.Sprintf("{\"addProjectAccess\":{\"project\":%q}}", seededProjectID), http.StatusAccepted) + checkAdmin(http.MethodPost, "/v3/types/"+typeID+"/action", fmt.Sprintf("{\"removeProjectAccess\":{\"project\":%q}}", seededProjectID), http.StatusAccepted) + checkAdmin(http.MethodPut, "/v3/qos-specs/"+qosID, "{\"qos_specs\":{\"iops\":\"10\"}}", http.StatusOK) + checkAdmin(http.MethodDelete, "/v3/qos-specs/"+qosID, "", http.StatusAccepted) + checkAdmin(http.MethodGet, "/v3/manageable_volumes?host=h1", "", http.StatusOK) + checkAdmin(http.MethodPost, "/v3/os-snapshot-manage", manageSnapshotBody, http.StatusAccepted) + checkAdmin(http.MethodPost, "/v3/volumes/"+unmanageVolumeID+"/action", "{\"os-unmanage\":{}}", http.StatusAccepted) + checkAdmin(http.MethodPost, "/v3/volumes/"+resetVolumeID+"/action", "{\"os-reset_status\":{\"status\":\"available\"}}", http.StatusAccepted) + checkAdmin(http.MethodPost, "/v3/snapshots/"+snapshotID+"/action", "{\"os-reset_status\":{\"status\":\"available\"}}", http.StatusAccepted) + checkAdmin(http.MethodDelete, "/v3/types/"+typeID, "", http.StatusAccepted) + checkAdmin(http.MethodPost, "/v3/os-volume-manage", "{\"volume\":{\"host\":\"h1\",\"ref\":{\"source-name\":\"x\"},\"name\":\"review-managed\"}}", http.StatusAccepted) + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), "DELETE FROM volumes WHERE name = 'review-managed'") + }) +} diff --git a/internal/cinder/qos_specs.go b/internal/cinder/qos_specs.go index dd3a67bb..76ede9e9 100644 --- a/internal/cinder/qos_specs.go +++ b/internal/cinder/qos_specs.go @@ -66,6 +66,15 @@ func (svc *Service) ListQoSSpecs(c *gin.Context) { // CreateQoSSpec creates a new QoS specification func (svc *Service) CreateQoSSpec(c *gin.Context) { + // Admin-only: volume types, QoS specs and the + // manage/unmanage surface shape storage for every tenant and address + // host-level backends (upstream: volume_extension:* and + // volume_extension:volume_admin_actions:*, all admin_api). Checked before + // the body is bound so a non-admin gets 403, not a 400. + if !common.RequireAdmin(c, "cinder:CreateQoSSpec") { + return + } + var req struct { QosSpecs struct { Name string `json:"name" binding:"required"` @@ -152,6 +161,15 @@ func (svc *Service) GetQoSSpec(c *gin.Context) { // UpdateQoSSpec updates a QoS specification func (svc *Service) UpdateQoSSpec(c *gin.Context) { + // Admin-only: volume types, QoS specs and the + // manage/unmanage surface shape storage for every tenant and address + // host-level backends (upstream: volume_extension:* and + // volume_extension:volume_admin_actions:*, all admin_api). Checked before + // the body is bound so a non-admin gets 403, not a 400. + if !common.RequireAdmin(c, "cinder:UpdateQoSSpec") { + return + } + qosID := c.Param("id") projectID := c.GetString("project_id") @@ -233,6 +251,15 @@ func (svc *Service) UpdateQoSSpec(c *gin.Context) { // DeleteQoSSpec deletes a QoS specification func (svc *Service) DeleteQoSSpec(c *gin.Context) { + // Admin-only: volume types, QoS specs and the + // manage/unmanage surface shape storage for every tenant and address + // host-level backends (upstream: volume_extension:* and + // volume_extension:volume_admin_actions:*, all admin_api). Checked before + // the body is bound so a non-admin gets 403, not a 400. + if !common.RequireAdmin(c, "cinder:DeleteQoSSpec") { + return + } + qosID := c.Param("id") projectID := c.GetString("project_id") diff --git a/internal/cinder/quota_enforcement_test.go b/internal/cinder/quota_enforcement_test.go index 97fc501e..38cdbc4d 100644 --- a/internal/cinder/quota_enforcement_test.go +++ b/internal/cinder/quota_enforcement_test.go @@ -529,15 +529,25 @@ func TestQuotaConcurrentCreates(t *testing.T) { // assertion is the one that matters, since it holds even if the HTTP codes race. // concurrentQuotaReqs fires n identical requests at once and reports how many were -// accepted. +// accepted. admin marks the caller's role: os-volume-manage requires an admin +// while still consuming the target project's quota, so its test +// drives the route as an administrator. func concurrentQuotaReqs(r *gin.Engine, n int, method, path, body string) []int { + return concurrentQuotaReqsAs(r, n, method, path, body, "member") +} + +func concurrentQuotaReqsAs(r *gin.Engine, n int, method, path, body, roles string) []int { codes := make([]int, n) var wg sync.WaitGroup for i := range codes { wg.Add(1) go func(i int) { defer wg.Done() - codes[i], _ = quotaReq(r, method, path, body, quotaProject) + codes[i], _ = doReq(r, method, path, body, map[string]string{ + "X-Test-Project": quotaProject, + "X-Test-User": "00000000-0000-0000-0000-000000000001", + "X-Test-Roles": roles, + }) }(i) } wg.Wait() @@ -651,6 +661,9 @@ func TestQuotaConcurrentExtends(t *testing.T) { // and with it the old checkVolumeQuota: that helper ran the ceiling read, the count check // and the gigabytes check as three separate round trips under two separate lock // acquisitions, so its own three parts were not atomic with each other either. +// The caller is an administrator because os-volume-manage requires the +// upstream volume_admin_actions:manage policy while the import still +// consumes the target project's quota. func TestQuotaConcurrentManagedVolumes(t *testing.T) { requireDB(t) seedQuotaProject(t) @@ -659,7 +672,7 @@ func TestQuotaConcurrentManagedVolumes(t *testing.T) { r := quotaRouter(t) body := `{"volume":{"host":"h1","ref":{"source-name":"race"},"name":"race"}}` - codes := concurrentQuotaReqs(r, 4, http.MethodPost, "/v3/"+quotaProject+"/os-volume-manage", body) + codes := concurrentQuotaReqsAs(r, 4, http.MethodPost, "/v3/"+quotaProject+"/os-volume-manage", body, "admin") if n := countAccepted(codes); n != 1 { t.Errorf("concurrent managed imports against one remaining allowance: %d accepted (codes %v); want exactly 1", n, codes) diff --git a/internal/cinder/quotas.go b/internal/cinder/quotas.go index 3dc5b7a2..c4e99a4a 100644 --- a/internal/cinder/quotas.go +++ b/internal/cinder/quotas.go @@ -178,8 +178,7 @@ func (svc *Service) GetQuotaSet(c *gin.Context) { func (svc *Service) UpdateQuotaSet(c *gin.Context) { // Admin-only: quota limits are an operator concern. Checked before the body is // bound so a non-admin gets 403, not a request-validation 400 (SPEC-020 FR-020). - if !common.HasRole(c, common.RoleAdmin) { - common.SendError(c, common.NewForbiddenError("Policy doesn't allow quota updates to be performed. Admin role required.")) + if !common.RequireAdmin(c, "cinder:UpdateQuotaSet") { return } @@ -255,8 +254,7 @@ func (svc *Service) UpdateQuotaSet(c *gin.Context) { // DeleteQuotaSet resets quotas to defaults for a project func (svc *Service) DeleteQuotaSet(c *gin.Context) { // Admin-only, matching UpdateQuotaSet and upstream Cinder (SPEC-020 FR-020). - if !common.HasRole(c, common.RoleAdmin) { - common.SendError(c, common.NewForbiddenError("Policy doesn't allow quota updates to be performed. Admin role required.")) + if !common.RequireAdmin(c, "cinder:DeleteQuotaSet") { return } diff --git a/internal/cinder/transfers.go b/internal/cinder/transfers.go index 3be5c8d1..ad246749 100644 --- a/internal/cinder/transfers.go +++ b/internal/cinder/transfers.go @@ -2,6 +2,7 @@ package cinder import ( "crypto/rand" + "crypto/subtle" "encoding/hex" "errors" "fmt" @@ -317,8 +318,11 @@ func (svc *Service) AcceptVolumeTransfer(c *gin.Context) { return } - // Verify auth key - if req.Accept.AuthKey != storedAuthKey { + // Verify auth key. Constant-time: a plain string + // compare short-circuits on the first differing byte, and a transfer id is + // cheap to enumerate, so the key could be recovered byte-by-byte from + // response timing. + if subtle.ConstantTimeCompare([]byte(req.Accept.AuthKey), []byte(storedAuthKey)) != 1 { common.SendError(c, common.NewBadRequestError("invalid auth key")) return } diff --git a/internal/cinder/volume_type_access.go b/internal/cinder/volume_type_access.go index ca72e229..eeb952fb 100644 --- a/internal/cinder/volume_type_access.go +++ b/internal/cinder/volume_type_access.go @@ -14,6 +14,14 @@ import ( func (svc *Service) VolumeTypeAction(c *gin.Context) { typeID := c.Param("id") + // Admin-only: project access to a private volume + // type is a cross-tenant grant — without the check any member could add + // their project to (or strip another project from) any private type. + // Checked before the body is bound so a non-admin gets 403, not a 400. + if !common.RequireAdmin(c, "cinder:VolumeTypeAction") { + return + } + var req map[string]any if err := c.ShouldBindJSON(&req); err != nil { common.SendError(c, common.NewBadRequestError("invalid request body")) diff --git a/internal/cinder/volume_types.go b/internal/cinder/volume_types.go index afd62350..c2d5b431 100644 --- a/internal/cinder/volume_types.go +++ b/internal/cinder/volume_types.go @@ -15,6 +15,15 @@ import ( // CreateVolumeType creates a new volume type func (svc *Service) CreateVolumeType(c *gin.Context) { + // Admin-only: volume types, QoS specs and the + // manage/unmanage surface shape storage for every tenant and address + // host-level backends (upstream: volume_extension:* and + // volume_extension:volume_admin_actions:*, all admin_api). Checked before + // the body is bound so a non-admin gets 403, not a 400. + if !common.RequireAdmin(c, "cinder:CreateVolumeType") { + return + } + var req struct { VolumeType struct { Name string `json:"name" binding:"required"` @@ -58,6 +67,15 @@ func (svc *Service) CreateVolumeType(c *gin.Context) { // UpdateVolumeType updates a volume type func (svc *Service) UpdateVolumeType(c *gin.Context) { + // Admin-only: volume types, QoS specs and the + // manage/unmanage surface shape storage for every tenant and address + // host-level backends (upstream: volume_extension:* and + // volume_extension:volume_admin_actions:*, all admin_api). Checked before + // the body is bound so a non-admin gets 403, not a 400. + if !common.RequireAdmin(c, "cinder:UpdateVolumeType") { + return + } + typeID := c.Param("id") var req struct { @@ -121,6 +139,15 @@ func (svc *Service) UpdateVolumeType(c *gin.Context) { // DeleteVolumeType deletes a volume type func (svc *Service) DeleteVolumeType(c *gin.Context) { + // Admin-only: volume types, QoS specs and the + // manage/unmanage surface shape storage for every tenant and address + // host-level backends (upstream: volume_extension:* and + // volume_extension:volume_admin_actions:*, all admin_api). Checked before + // the body is bound so a non-admin gets 403, not a 400. + if !common.RequireAdmin(c, "cinder:DeleteVolumeType") { + return + } + typeID := c.Param("id") result, err := svc.pool.Exec(c.Request.Context(), @@ -172,6 +199,15 @@ func (svc *Service) ListVolumeTypeExtraSpecs(c *gin.Context) { // CreateVolumeTypeExtraSpecs handles POST /v3/types/:id/extra_specs func (svc *Service) CreateVolumeTypeExtraSpecs(c *gin.Context) { + // Admin-only: volume types, QoS specs and the + // manage/unmanage surface shape storage for every tenant and address + // host-level backends (upstream: volume_extension:* and + // volume_extension:volume_admin_actions:*, all admin_api). Checked before + // the body is bound so a non-admin gets 403, not a 400. + if !common.RequireAdmin(c, "cinder:CreateVolumeTypeExtraSpecs") { + return + } + typeID := c.Param("id") var req struct { @@ -235,6 +271,15 @@ func (svc *Service) GetVolumeTypeExtraSpecKey(c *gin.Context) { // UpdateVolumeTypeExtraSpecKey handles PUT /v3/types/:id/extra_specs/:key func (svc *Service) UpdateVolumeTypeExtraSpecKey(c *gin.Context) { + // Admin-only: volume types, QoS specs and the + // manage/unmanage surface shape storage for every tenant and address + // host-level backends (upstream: volume_extension:* and + // volume_extension:volume_admin_actions:*, all admin_api). Checked before + // the body is bound so a non-admin gets 403, not a 400. + if !common.RequireAdmin(c, "cinder:UpdateVolumeTypeExtraSpecKey") { + return + } + typeID := c.Param("id") key := c.Param("key") @@ -297,6 +342,15 @@ func (svc *Service) UpdateVolumeTypeExtraSpecKey(c *gin.Context) { // DeleteVolumeTypeExtraSpecKey handles DELETE /v3/types/:id/extra_specs/:key func (svc *Service) DeleteVolumeTypeExtraSpecKey(c *gin.Context) { + // Admin-only: volume types, QoS specs and the + // manage/unmanage surface shape storage for every tenant and address + // host-level backends (upstream: volume_extension:* and + // volume_extension:volume_admin_actions:*, all admin_api). Checked before + // the body is bound so a non-admin gets 403, not a 400. + if !common.RequireAdmin(c, "cinder:DeleteVolumeTypeExtraSpecKey") { + return + } + typeID := c.Param("id") key := c.Param("key") diff --git a/internal/cinder/volumes.go b/internal/cinder/volumes.go index a8f1d2a6..2bfef1ba 100644 --- a/internal/cinder/volumes.go +++ b/internal/cinder/volumes.go @@ -1588,8 +1588,7 @@ func (svc *Service) VolumeAction(c *gin.Context) { // Handle reset status action (admin operation) if resetStatusData, ok := req["os-reset_status"]; ok { // Verify admin role - if !common.HasRole(c, common.RoleAdmin) { - common.SendError(c, common.NewForbiddenError("Policy doesn't allow os-reset_status to be performed.")) + if !common.RequireAdmin(c, "cinder:VolumeAction") { return } @@ -1993,8 +1992,7 @@ func (svc *Service) SnapshotAction(c *gin.Context) { common.SendError(c, common.NewBadRequestError("unknown action")) return } - if !common.HasRole(c, common.RoleAdmin) { - common.SendError(c, common.NewForbiddenError("Policy doesn't allow os-reset_status to be performed.")) + if !common.RequireAdmin(c, "cinder:SnapshotAction") { return } resetStatusMap, ok := resetStatusData.(map[string]any) @@ -2028,7 +2026,7 @@ func (svc *Service) SnapshotAction(c *gin.Context) { // ListVolumeTypes lists all volume types func (svc *Service) ListVolumeTypes(c *gin.Context) { rows, err := svc.pool.Query(c.Request.Context(), ` - SELECT id, name, description, is_public, created_at + SELECT id, name, COALESCE(description, ''), is_public, created_at FROM volume_types ORDER BY name `) diff --git a/internal/common/authz.go b/internal/common/authz.go new file mode 100644 index 00000000..faca15b9 --- /dev/null +++ b/internal/common/authz.go @@ -0,0 +1,22 @@ +package common + +import ( + "github.com/gin-gonic/gin" + "github.com/rs/zerolog/log" +) + +// RequireAdmin denies requests that do not carry the deployment-wide admin +// role. operation identifies the API operation in the error and refusal log. +func RequireAdmin(c *gin.Context, operation string) bool { + if HasRole(c, RoleAdmin) { + return true + } + log.Warn(). + Str("caller_project", c.GetString("project_id")). + Str("caller_user", c.GetString("user_id")). + Str("operation", operation). + Msg("refused operator-only API operation") + SendError(c, NewForbiddenError( + "Policy doesn't allow "+operation+" to be performed. Admin role required.")) + return false +} diff --git a/internal/common/errors.go b/internal/common/errors.go index dfefb875..9e7c318d 100644 --- a/internal/common/errors.go +++ b/internal/common/errors.go @@ -249,6 +249,16 @@ func NewServiceUnavailableError(message string) *OpenStackError { } } +// NewRequestEntityTooLargeError creates a 413 error, upstream's answer to a +// request body over the configured size cap (e.g. Glance's image_size_cap). +func NewRequestEntityTooLargeError(message string) *OpenStackError { + return &OpenStackError{ + StatusCode: http.StatusRequestEntityTooLarge, + Code: "requestEntityTooLarge", + Message: message, + } +} + func NewInternalServerError(message string) *OpenStackError { return &OpenStackError{ StatusCode: http.StatusInternalServerError, diff --git a/internal/common/fixed_ips.go b/internal/common/fixed_ips.go index 311b6735..f8bcd8e6 100644 --- a/internal/common/fixed_ips.go +++ b/internal/common/fixed_ips.go @@ -5,6 +5,7 @@ package common // metadata service (instance-by-address lookup) read it to find which port holds an address. type FixedIPs []struct { IPAddress string `json:"ip_address"` + SubnetID string `json:"subnet_id"` } // Holds reports whether ip is one of the port's fixed addresses, compared as a whole string. diff --git a/internal/glance/cache.go b/internal/glance/cache.go index f7a5c967..45132f03 100644 --- a/internal/glance/cache.go +++ b/internal/glance/cache.go @@ -11,6 +11,13 @@ import ( // ListCachedImages lists images in the cache func (svc *Service) ListCachedImages(c *gin.Context) { + // Admin-only: the cache is a per-node operator + // surface — upstream's cache_list policy. The rest of the cache handlers + // were gated in this review's first pass; the list was missed. + if !common.RequireAdmin(c, "glance:ListCachedImages") { + return + } + // Query images that have been marked as cached rows, err := svc.pool.Query(c.Request.Context(), ` SELECT id, name, size_bytes, cached_at @@ -61,10 +68,16 @@ func (svc *Service) ListCachedImages(c *gin.Context) { c.JSON(http.StatusOK, gin.H{"cached_images": cachedImages}) } -// PrefetchImage prefetches an image into the cache +// PrefetchImage prefetches an image into the cache. Admin-only: +// cache management is an operator surface (upstream +// `admin_api`) — it moves shared disk state for every tenant. func (svc *Service) PrefetchImage(c *gin.Context) { imageID := c.Param("id") + if !common.RequireAdmin(c, "glance:PrefetchImage") { + return + } + // Verify image exists var exists bool err := svc.pool.QueryRow(c.Request.Context(), @@ -93,10 +106,14 @@ func (svc *Service) PrefetchImage(c *gin.Context) { c.Status(http.StatusAccepted) } -// DeleteCachedImage removes an image from the cache +// DeleteCachedImage removes an image from the cache. Admin-only. func (svc *Service) DeleteCachedImage(c *gin.Context) { imageID := c.Param("id") + if !common.RequireAdmin(c, "glance:DeleteCachedImage") { + return + } + result, err := svc.pool.Exec(c.Request.Context(), ` UPDATE images SET cached_at = NULL @@ -117,8 +134,14 @@ func (svc *Service) DeleteCachedImage(c *gin.Context) { c.Status(http.StatusNoContent) } -// ClearCache clears all cached images +// ClearCache clears all cached images. Admin-only: +// without the gate any member could flush the deployment-wide cache (DoS on +// boot-time caching) with one request. func (svc *Service) ClearCache(c *gin.Context) { + if !common.RequireAdmin(c, "glance:ClearCache") { + return + } + _, err := svc.pool.Exec(c.Request.Context(), ` UPDATE images SET cached_at = NULL diff --git a/internal/glance/images.go b/internal/glance/images.go index 5135ceb2..29d6d028 100644 --- a/internal/glance/images.go +++ b/internal/glance/images.go @@ -37,8 +37,11 @@ type Service struct { s3Endpoint string imageStore *storage.ImageStore cache *cache.Cache + uploadCap int64 } +const defaultMaxImageUploadBytes int64 = 10 << 30 + // NewService creates a new Glance service. // // Returns an error if the configured storage backend fails to initialize @@ -61,6 +64,7 @@ func NewService(pool database.DB, mode, cephPool, cephConf string, s3opts storag s3Endpoint: s3opts.Endpoint, imageStore: imageStore, cache: cacheInstance, + uploadCap: defaultMaxImageUploadBytes, }, nil } @@ -91,6 +95,7 @@ func (svc *Service) RegisterRoutes(r *gin.RouterGroup) { // Image import r.POST("/images/:id/stage", svc.StageImageData) + r.PUT("/images/:id/stage", svc.StageImageData) r.POST("/images/:id/import", svc.ImportImage) r.GET("/images/:id/import", svc.GetImageImportInfo) @@ -957,29 +962,65 @@ func (svc *Service) UploadImageData(c *gin.Context) { imageID := c.Param("id") projectID := c.GetString("project_id") - // Check if image exists and is in queued status - var status string + // Ownership, not visibility: seeing a public + // image must not grant write access to its bytes — otherwise anyone + // could poison an operator-staged public base image that every tenant + // boots from. Same rule as delete (F6): owner or admin, 403 for a + // visible-but-foreign image, 404 for an invisible one. + var uploadOwner sql.NullString + var uploadVisibility string err := svc.pool.QueryRow(c.Request.Context(), - "SELECT status FROM images WHERE id = $1 AND (visibility = 'public' OR project_id = $2)", - imageID, projectID, - ).Scan(&status) - + "SELECT project_id, visibility FROM images WHERE id = $1", + imageID, + ).Scan(&uploadOwner, &uploadVisibility) if errors.Is(err, database.ErrNoRows) { common.SendError(c, common.NewNotFoundError("image")) return } - - if status != "queued" { - common.SendError(c, common.NewConflictError("image data already exists")) + if err != nil { + log.Error().Err(err).Str("operation", "upload_image_lookup").Str("image_id", imageID).Msg("failed to look up image") + common.SendError(c, common.NewInternalServerError("failed to upload image")) + return + } + switch imageDeleteAuthz(common.HasRole(c, common.RoleAdmin), uploadOwner, projectID, uploadVisibility) { + case http.StatusForbidden: + common.SendError(c, common.NewForbiddenError("not authorized to upload data to this image")) + return + case http.StatusNotFound: + common.SendError(c, common.NewNotFoundError("image")) return } - // Update status to saving - if _, err := svc.pool.Exec(c.Request.Context(), - "UPDATE images SET status = $1, updated_at = $2 WHERE id = $3", - "saving", time.Now(), imageID); err != nil { - log.Warn().Err(err).Str("image_id", imageID).Msg("failed to mark image as saving") + // Bound the upload: without a cap an unbounded + // stream lands on disk (and, in local mode, is normalized) until the + // host fills. Same default as upstream Glance's image_size_cap. + c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, svc.uploadCap) + + // Claim the queued image atomically: only one request may write its bytes. + claimed, err := svc.pool.Exec(c.Request.Context(), + "UPDATE images SET status = 'saving', updated_at = $1 WHERE id = $2 AND status = 'queued'", + time.Now(), imageID) + if err != nil { + log.Error().Err(err).Str("image_id", imageID).Msg("failed to mark image as saving") + common.SendError(c, common.NewInternalServerError("failed to start image upload")) + return } + if claimed.RowsAffected() == 0 { + common.SendError(c, common.NewConflictError("image data already exists")) + return + } + active := false + defer func() { + if active { + return + } + // A disconnected client must not leave the image permanently in saving. + ctx, cancel := context.WithTimeout(context.WithoutCancel(c.Request.Context()), time.Second) + defer cancel() + if _, err := svc.pool.Exec(ctx, "UPDATE images SET status = 'queued' WHERE id = $1 AND status = 'saving'", imageID); err != nil { + log.Warn().Err(err).Str("image_id", imageID).Msg("rollback: failed to reset image upload") + } + }() // Upload to storage, hashing the uploaded bytes so the image's checksum // (MD5, Glance convention) is recorded — it lets a lazily fetched base be @@ -987,32 +1028,60 @@ func (svc *Service) UploadImageData(c *gin.Context) { hasher := md5.New() // #nosec G401 -- Glance checksum is MD5 (compatibility), not a security control size, normalizedToRaw, err := svc.imageStore.UploadImage(c.Request.Context(), imageID, io.TeeReader(c.Request.Body, hasher)) if err != nil { - // Revert status to queued (best-effort; the upload error is the - // user-facing one) - if _, revertErr := svc.pool.Exec(c.Request.Context(), - "UPDATE images SET status = $1 WHERE id = $2", - "queued", imageID); revertErr != nil { - log.Warn().Err(revertErr).Str("image_id", imageID).Msg("rollback: failed to revert image status to queued after upload failure") + // Over the cap is 413 — upstream's answer to image_size_cap — not a + // storage failure (round 2 of this review: the MaxBytesReader error + // surfaced from the storage read as a 503). + var tooLarge *http.MaxBytesError + if errors.As(err, &tooLarge) { + common.SendError(c, common.NewRequestEntityTooLargeError("Image data exceeds the maximum upload size")) + return } log.Error().Err(err).Str("operation", "upload_image").Str("image_id", imageID).Msg("failed to upload image to storage") common.SendError(c, common.NewServiceUnavailableError("failed to upload image")) return } + if err := svc.activateUploadedImage(c.Request.Context(), imageID, size, normalizedToRaw, hex.EncodeToString(hasher.Sum(nil))); err != nil { + log.Error().Err(err).Str("image_id", imageID).Msg("failed to finalize image upload") + common.SendError(c, common.NewServiceUnavailableError("failed to finalize image upload")) + return + } + active = true + c.Status(http.StatusNoContent) +} + +// activateUploadedImage publishes only a stored object whose checksum and state +// were recorded successfully. An unreadable normalized object must never fall +// back to the original stream's checksum: those bytes can be different. +func (svc *Service) activateUploadedImage(ctx context.Context, imageID string, size int64, normalizedToRaw bool, checksum string) error { // Mark active and set size + checksum. Normalizing backends (local/RBD real // mode) store a raw base, so disk_format becomes "raw" (SPEC-009 WS-0/FR-003); // the S3 backend stores the bytes as-is, so its create-time disk_format is left // untouched rather than mislabeled "raw" (SPEC-014). - checksum := hex.EncodeToString(hasher.Sum(nil)) - if _, err := svc.pool.Exec(c.Request.Context(), + if normalizedToRaw { + // The recorded checksum must describe the STORED bytes: downloads serve + // them, and EnsureLocalBase verifies a lazily fetched base against this + // value on every boot — hashing the upload stream + // instead broke boot-from-image on any node without the local base, + // because MD5(raw replica) != MD5(original qcow2)). + stored, csErr := svc.imageStore.StoredChecksum(ctx, imageID) + if csErr != nil { + return fmt.Errorf("hash stored image: %w", csErr) + } + checksum = stored + } + result, err := svc.pool.Exec(ctx, `UPDATE images SET status = $1, size_bytes = $2, disk_format = CASE WHEN $3 THEN 'raw' ELSE disk_format END, - checksum = $4, updated_at = $5 WHERE id = $6`, - "active", size, normalizedToRaw, checksum, time.Now(), imageID); err != nil { - log.Error().Err(err).Str("image_id", imageID).Msg("failed to mark image as active after successful upload") + checksum = $4, updated_at = $5 WHERE id = $6 AND status = 'saving'`, + "active", size, normalizedToRaw, checksum, time.Now(), imageID) + if err != nil { + return fmt.Errorf("record active image: %w", err) } - - c.Status(http.StatusNoContent) + if result.RowsAffected() != 1 { + return errors.New("image is no longer in saving state") + } + return nil } // DownloadImageData downloads image data diff --git a/internal/glance/images_db_test.go b/internal/glance/images_db_test.go index 669f1dab..b41cff4c 100644 --- a/internal/glance/images_db_test.go +++ b/internal/glance/images_db_test.go @@ -46,6 +46,20 @@ const ( ) func newGlanceTestRouter(t *testing.T) *gin.Engine { + t.Helper() + return newGlanceTestRouterForService(t, newGlanceTestService(t)) +} + +func newGlanceTestService(t *testing.T) *Service { + t.Helper() + svc, err := NewService(testPool, "stub", "", "", storage.S3Options{}, nil) + if err != nil { + t.Fatalf("NewService: %v", err) + } + return svc +} + +func newGlanceTestRouterForService(t *testing.T, svc *Service) *gin.Engine { t.Helper() gin.SetMode(gin.TestMode) r := gin.New() @@ -53,14 +67,13 @@ func newGlanceTestRouter(t *testing.T) *gin.Engine { if p := c.GetHeader("X-Test-Project"); p != "" { c.Set("project_id", p) } + if userID := c.GetHeader("X-Test-User"); userID != "" { + c.Set("user_id", userID) + } if roles := c.GetHeader("X-Test-Roles"); roles != "" { c.Set("roles", strings.Split(roles, ",")) } }) - svc, err := NewService(testPool, "stub", "", "", storage.S3Options{}, nil) - if err != nil { - t.Fatalf("NewService: %v", err) - } svc.RegisterRoutes(&r.RouterGroup) return r } @@ -95,6 +108,10 @@ func hdr(project, roles string) map[string]string { return map[string]string{"X-Test-Project": project, "X-Test-Roles": roles} } +func userHdr(userID, roles string) map[string]string { + return map[string]string{"X-Test-User": userID, "X-Test-Roles": roles} +} + // TestDeleteImageAuthz_DB exercises the F6 wiring against Postgres for a public image // whose owner is retained: a non-owner non-admin gets 403 (image stays), a missing id // gets 404, and the owner (no admin role) deletes it (204, row gone via rows-affected). diff --git a/internal/glance/import.go b/internal/glance/import.go index 73dd0a00..a0b6a994 100644 --- a/internal/glance/import.go +++ b/internal/glance/import.go @@ -9,6 +9,7 @@ import ( "github.com/cobaltcore-dev/lightstack/internal/database" "github.com/gin-gonic/gin" + "github.com/rs/zerolog/log" ) // StageImageData stages image data before import @@ -29,7 +30,8 @@ func (svc *Service) StageImageData(c *gin.Context) { } if err != nil { - c.JSON(http.StatusInternalServerError, gin.H{"message": err.Error()}) + log.Error().Err(err).Str("image_id", imageID).Msg("image staging database error") + c.JSON(http.StatusInternalServerError, gin.H{"message": "Failed to stage image data"}) return } @@ -39,28 +41,38 @@ func (svc *Service) StageImageData(c *gin.Context) { return } - // Read staged data (in real implementation, would save to staging area) - stagedData, err := io.ReadAll(c.Request.Body) + // Consume staging data with bounded memory. The staging backend is currently + // a stub, so only its byte count is retained; the upload cap can be 10 GiB. + c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, svc.uploadCap) + stagedSize, err := io.Copy(io.Discard, c.Request.Body) if err != nil { + // Over the cap is 413, upstream's answer to image_size_cap (round 2 + // of this review — it used to read as a 400 read failure). + var tooLarge *http.MaxBytesError + if errors.As(err, &tooLarge) { + c.JSON(http.StatusRequestEntityTooLarge, gin.H{"message": "Image data exceeds the maximum upload size"}) + return + } c.JSON(http.StatusBadRequest, gin.H{"message": "Failed to read image data"}) return } // Update image status to uploading - _, err = svc.pool.Exec(c.Request.Context(), ` + result, err := svc.pool.Exec(c.Request.Context(), ` UPDATE images SET status = $1, size_bytes = $2, updated_at = $3 - WHERE id = $4 - `, "uploading", len(stagedData), time.Now(), imageID) + WHERE id = $4 AND status = 'queued' + `, "uploading", stagedSize, time.Now(), imageID) if err != nil { - c.JSON(http.StatusInternalServerError, gin.H{"message": err.Error()}) + log.Error().Err(err).Str("image_id", imageID).Msg("image staging database error") + c.JSON(http.StatusInternalServerError, gin.H{"message": "Failed to stage image data"}) + return + } + if result.RowsAffected() == 0 { + c.JSON(http.StatusConflict, gin.H{"message": "Image is not in queued state"}) return } - - // In stub mode, data is discarded - // In real mode, would save to staging location - _ = stagedData c.Status(http.StatusNoContent) } diff --git a/internal/glance/stage_limits_test.go b/internal/glance/stage_limits_test.go new file mode 100644 index 00000000..5e0b5715 --- /dev/null +++ b/internal/glance/stage_limits_test.go @@ -0,0 +1,72 @@ +package glance + +import ( + "context" + "io" + "net/http" + "net/http/httptest" + "testing" + + "github.com/google/uuid" +) + +// Records the largest read buffer without allocating a large request body. +type stageReader struct { + remaining int64 + maxRead int +} + +func (r *stageReader) Read(p []byte) (int, error) { + r.maxRead = max(r.maxRead, len(p)) + if r.remaining == 0 { + return 0, io.EOF + } + n := int(min(int64(len(p)), r.remaining)) + clear(p[:n]) + r.remaining -= int64(n) + return n, nil +} + +func TestStageImageStreamsWithinLimit(t *testing.T) { + requireDB(t) + const size = 8 << 20 + svc := newGlanceTestService(t) + svc.uploadCap = size + router := newGlanceTestRouterForService(t, svc) + for _, extra := range []int64{0, 1} { + id := uuid.NewString() + if _, err := testPool.Exec(t.Context(), `INSERT INTO images (id, name, project_id, status, visibility) + VALUES ($1, 'stage-limit', $2, 'queued', 'private')`, id, seededProjectID); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { testPool.Exec(context.Background(), "DELETE FROM images WHERE id=$1", id) }) + reader := &stageReader{remaining: size + extra} + req := httptest.NewRequest(http.MethodPut, "/images/"+id+"/stage", reader) + for k, v := range hdr(seededProjectID, "member") { + req.Header.Set(k, v) + } + w := httptest.NewRecorder() + router.ServeHTTP(w, req) + wantCode, wantStatus := http.StatusNoContent, "uploading" + if extra != 0 { + wantCode, wantStatus = http.StatusRequestEntityTooLarge, "queued" + } + if w.Code != wantCode { + t.Errorf("stage: %d %s; want %d", w.Code, w.Body.String(), wantCode) + } + if reader.maxRead > 64<<10 { + t.Errorf("stage allocated a %d-byte read buffer; want bounded streaming", reader.maxRead) + } + var status string + var storedSize int64 + if err := testPool.QueryRow(t.Context(), "SELECT status, COALESCE(size_bytes, 0) FROM images WHERE id=$1", id).Scan(&status, &storedSize); err != nil { + t.Fatal(err) + } + if status != wantStatus { + t.Errorf("status=%s; want %s", status, wantStatus) + } + if extra == 0 && storedSize != size { + t.Errorf("size=%d; want %d", storedSize, size) + } + } +} diff --git a/internal/glance/tasks.go b/internal/glance/tasks.go index cdaff4ba..ea1dd624 100644 --- a/internal/glance/tasks.go +++ b/internal/glance/tasks.go @@ -34,7 +34,7 @@ func (svc *Service) CreateTask(c *gin.Context) { `, taskID, req.Type, "pending", req.Input, owner, now, now) if err != nil { - c.JSON(http.StatusInternalServerError, gin.H{"message": err.Error()}) + c.JSON(http.StatusInternalServerError, gin.H{"message": "failed to create task"}) return } @@ -51,16 +51,19 @@ func (svc *Service) CreateTask(c *gin.Context) { }) } -// ListTasks handles GET /v2/tasks +// ListTasks handles GET /v2/tasks. Owner-scoped: task +// input/result carry the caller's image references and endpoint URLs — a +// member must see only their own tasks; an admin sees all. func (svc *Service) ListTasks(c *gin.Context) { rows, err := svc.pool.Query(c.Request.Context(), ` SELECT id, type, status, input, result, owner, created_at, updated_at FROM image_tasks + WHERE owner = $1 OR $2 ORDER BY created_at DESC LIMIT 100 - `) + `, c.GetString("user_id"), common.HasRole(c, common.RoleAdmin)) if err != nil { - c.JSON(http.StatusInternalServerError, gin.H{"message": err.Error()}) + c.JSON(http.StatusInternalServerError, gin.H{"message": "failed to list tasks"}) return } defer rows.Close() @@ -108,7 +111,8 @@ func (svc *Service) ListTasks(c *gin.Context) { }) } -// GetTask handles GET /v2/tasks/:id +// GetTask handles GET /v2/tasks/:id. Owner-or-admin: +// input/result/message are the caller's own task payload, not public data. func (svc *Service) GetTask(c *gin.Context) { taskID := c.Param("id") @@ -123,8 +127,14 @@ func (svc *Service) GetTask(c *gin.Context) { `, taskID).Scan(&taskType, &status, &inputJSON, &resultJSON, &owner, &message, &createdAt, &updatedAt) if err != nil { - // Log the actual error for debugging - c.JSON(http.StatusNotFound, gin.H{"message": "Task not found", "debug_error": err.Error()}) + // Not found — and no internal error details in the response body + // because err.Error() could leak driver internals. + c.JSON(http.StatusNotFound, gin.H{"message": "Task not found"}) + return + } + + if owner != c.GetString("user_id") && !common.HasRole(c, common.RoleAdmin) { + c.JSON(http.StatusNotFound, gin.H{"message": "Task not found"}) return } diff --git a/internal/glance/tasks_authz_test.go b/internal/glance/tasks_authz_test.go new file mode 100644 index 00000000..31677187 --- /dev/null +++ b/internal/glance/tasks_authz_test.go @@ -0,0 +1,80 @@ +package glance + +import ( + "context" + "encoding/json" + "net/http" + "strings" + "testing" + "time" + + "github.com/google/uuid" +) + +func TestTaskListAndGetAreOwnerScoped(t *testing.T) { + requireDB(t) + r := newGlanceTestRouter(t) + ownerID, foreignID := seededUserID, uuid.NewString() + ownerTaskID, foreignTaskID := uuid.NewString(), uuid.NewString() + now := time.Now().UTC() + + for _, task := range []struct{ id, owner, secret string }{ + {ownerTaskID, ownerID, "owner-private-input"}, + {foreignTaskID, foreignID, "foreign-private-input"}, + } { + input, err := json.Marshal(map[string]string{"private": task.secret}) + if err != nil { + t.Fatal(err) + } + result, err := json.Marshal(map[string]string{"private_result": task.secret}) + if err != nil { + t.Fatal(err) + } + if _, err := testPool.Exec(t.Context(), ` + INSERT INTO image_tasks (id, type, status, input, result, message, owner, created_at, updated_at) + VALUES ($1, 'import', 'success', $2, $3, $4, $5, $6, $6) + `, task.id, input, result, task.secret+" internal message", task.owner, now); err != nil { + t.Fatalf("seed task %s: %v", task.id, err) + } + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), "DELETE FROM image_tasks WHERE id IN ($1, $2)", ownerTaskID, foreignTaskID) + }) + + member := userHdr(ownerID, "member") + code, body := doReq(r, http.MethodGet, "/tasks", member) + if code != http.StatusOK { + t.Fatalf("member list tasks: code=%d body=%s", code, body) + } + var listed struct { + Tasks []map[string]any `json:"tasks"` + } + if err := json.Unmarshal([]byte(body), &listed); err != nil { + t.Fatalf("decode task list: %v", err) + } + if len(listed.Tasks) != 1 || listed.Tasks[0]["id"] != ownerTaskID { + t.Errorf("member task list = %s; want only task %s", body, ownerTaskID) + } + if strings.Contains(body, "foreign-private-input") { + t.Errorf("member task list leaked foreign task data: %s", body) + } + + code, body = doReq(r, http.MethodGet, "/tasks/"+ownerTaskID, member) + if code != http.StatusOK || !strings.Contains(body, "owner-private-input") { + t.Errorf("owner get task: code=%d body=%s; want 200 with own task data", code, body) + } + code, body = doReq(r, http.MethodGet, "/tasks/"+foreignTaskID, member) + if code != http.StatusNotFound || strings.Contains(body, "foreign-private-input") || strings.Contains(body, "internal message") { + t.Errorf("foreign get task: code=%d body=%s; want opaque 404", code, body) + } + + admin := userHdr(uuid.NewString(), "admin") + code, body = doReq(r, http.MethodGet, "/tasks", admin) + if code != http.StatusOK || !strings.Contains(body, ownerTaskID) || !strings.Contains(body, foreignTaskID) { + t.Errorf("admin list tasks: code=%d body=%s; want both tasks", code, body) + } + code, body = doReq(r, http.MethodGet, "/tasks/"+foreignTaskID, admin) + if code != http.StatusOK || !strings.Contains(body, "foreign-private-input") { + t.Errorf("admin get foreign task: code=%d body=%s; want 200", code, body) + } +} diff --git a/internal/glance/upload_authz_test.go b/internal/glance/upload_authz_test.go new file mode 100644 index 00000000..38decb29 --- /dev/null +++ b/internal/glance/upload_authz_test.go @@ -0,0 +1,140 @@ +package glance + +import ( + "context" + "net/http" + "strings" + "testing" + + "github.com/google/uuid" +) + +// TestUploadImageDataOwnership_DB pins the rule that visibility is a +// read grant, not a write grant. A public image staged by one project must not +// accept bytes from another project's member — that was image poisoning: every +// tenant booting the public base image would run the attacker's data. The +// owner and an admin keep their paths. +func TestUploadImageDataOwnership_DB(t *testing.T) { + requireDB(t) + ctx := context.Background() + + otherProject := "00000000-0000-0000-0000-0000000000bb" + id := uuid.NewString() + if _, err := testPool.Exec(ctx, + `INSERT INTO images (id, name, project_id, status, visibility) VALUES ($1,'pub-poison',$2,'queued','public')`, + id, seededProjectID); err != nil { + t.Fatalf("insert image: %v", err) + } + t.Cleanup(func() { testPool.Exec(ctx, "DELETE FROM images WHERE id=$1", id) }) + + r := newGlanceTestRouter(t) + upload := func(hdr map[string]string) int { + code, _ := doReqBody(r, "PUT", "/images/"+id+"/file", "not really an image", hdr) + return code + } + + // A foreign member that can SEE the public image: 403 (visible, not owned). + if code := upload(hdr(otherProject, "member")); code != 403 { + t.Errorf("foreign member upload to public image: code=%d; want 403", code) + } + // A foreign member that cannot even see a private image: 404. + if _, err := testPool.Exec(ctx, "UPDATE images SET visibility='private' WHERE id=$1", id); err != nil { + t.Fatalf("set private: %v", err) + } + if code := upload(hdr(otherProject, "member")); code != 404 { + t.Errorf("foreign member upload to private image: code=%d; want 404", code) + } + if _, err := testPool.Exec(ctx, "UPDATE images SET visibility='public' WHERE id=$1", id); err != nil { + t.Fatalf("set public: %v", err) + } + // The owner may still upload (2xx; the stub store accepts anything). + if code := upload(hdr(seededProjectID, "member")); code < 200 || code >= 300 { + t.Errorf("owner upload: code=%d; want 2xx", code) + } + // An admin may upload to a fresh queued public image. + id2 := uuid.NewString() + if _, err := testPool.Exec(ctx, + `INSERT INTO images (id, name, project_id, status, visibility) VALUES ($1,'pub-admin',$2,'queued','public')`, + id2, seededProjectID); err != nil { + t.Fatalf("insert image 2: %v", err) + } + t.Cleanup(func() { testPool.Exec(ctx, "DELETE FROM images WHERE id=$1", id2) }) + if code, _ := doReqBody(r, "PUT", "/images/"+id2+"/file", "admin bytes", hdr(otherProject, "admin")); code < 200 || code >= 300 { + t.Errorf("admin upload: code=%d; want 2xx", code) + } +} + +// TestImageCacheManagementAdminOnly_DB pins the rule that cache +// mutation is an operator surface — a member flushing the deployment-wide +// cache (ClearCache) or evicting entries is a denial of service on +// boot-time caching. +func TestImageCacheManagementAdminOnly_DB(t *testing.T) { + requireDB(t) + ctx := context.Background() + + id := uuid.NewString() + if _, err := testPool.Exec(ctx, + `INSERT INTO images (id, name, project_id, status, visibility, cached_at) VALUES ($1,'cached',$2,'active','public',NOW())`, + id, seededProjectID); err != nil { + t.Fatalf("insert image: %v", err) + } + t.Cleanup(func() { testPool.Exec(ctx, "DELETE FROM images WHERE id=$1", id) }) + + r := newGlanceTestRouter(t) + + if code, _ := doReq(r, "DELETE", "/cache/images/"+id, hdr(seededProjectID, "member")); code != 403 { + t.Errorf("member cache eviction: code=%d; want 403", code) + } + if code, _ := doReq(r, "DELETE", "/cache/images", hdr(seededProjectID, "member")); code != 403 { + t.Errorf("member cache clear: code=%d; want 403", code) + } + if code, _ := doReq(r, "PUT", "/cache/images/"+id, hdr(seededProjectID, "member")); code != 403 { + t.Errorf("member cache prefetch: code=%d; want 403", code) + } + // The list is part of the same operator surface (round 2 of this review: + // the write paths were gated, the list was missed). + if code, _ := doReq(r, "GET", "/cache/images", hdr(seededProjectID, "member")); code != 403 { + t.Errorf("member cache list: code=%d; want 403", code) + } + if code, body := doReq(r, "GET", "/cache/images", hdr(seededProjectID, "admin")); code != 200 || !strings.Contains(body, id) { + t.Errorf("admin cache list: code=%d body=%s; want 200 listing the cached image", code, body) + } + // Still cached after the denied attempts. + var cached bool + if err := testPool.QueryRow(ctx, "SELECT cached_at IS NOT NULL FROM images WHERE id=$1", id).Scan(&cached); err != nil || !cached { + t.Errorf("denied cache clear still took effect (cached=%v err=%v)", cached, err) + } +} + +// TestOverCapUploadAnswers413 pins the upload cap: a body over the configured +// limit must answer 413 (upstream's image_size_cap semantics). The per-service +// cap lets the test avoid streaming 10 GiB; in stub mode the store's read +// surfaces http.MaxBytesError raw, and +// in local mode it is wrapped with %w — errors.As covers both. +func TestOverCapUploadAnswers413(t *testing.T) { + requireDB(t) + ctx := context.Background() + + id := uuid.NewString() + if _, err := testPool.Exec(ctx, + `INSERT INTO images (id, name, project_id, status, visibility) VALUES ($1,'cap-me',$2,'queued','private')`, + id, seededProjectID); err != nil { + t.Fatalf("insert image: %v", err) + } + t.Cleanup(func() { testPool.Exec(ctx, "DELETE FROM images WHERE id=$1", id) }) + + svc := newGlanceTestService(t) + svc.uploadCap = 8 + r := newGlanceTestRouterForService(t, svc) + code, body := doReqBody(r, "PUT", "/images/"+id+"/file", + "this body is longer than eight bytes", hdr(seededProjectID, "member")) + if code != http.StatusRequestEntityTooLarge { + t.Errorf("over-cap upload: code=%d body=%s; want 413", code, body) + } + + // And the image is left queued, not stuck in saving. + var status string + if err := testPool.QueryRow(ctx, "SELECT status FROM images WHERE id=$1", id).Scan(&status); err != nil || status != "queued" { + t.Errorf("image status after refused upload = %q (err=%v); want queued", status, err) + } +} diff --git a/internal/glance/upload_failure_test.go b/internal/glance/upload_failure_test.go new file mode 100644 index 00000000..f93af14f --- /dev/null +++ b/internal/glance/upload_failure_test.go @@ -0,0 +1,77 @@ +package glance + +import ( + "context" + "fmt" + "net/http" + "strings" + "testing" + + "github.com/cobaltcore-dev/lightstack/internal/database/dbtest" + "github.com/cobaltcore-dev/lightstack/pkg/storage" +) + +func TestUploadFailsWhenStateCannotBeRecorded(t *testing.T) { + requireDB(t) + for _, status := range []string{"saving", "active"} { + t.Run(status, func(t *testing.T) { + dbtest.RequirePostgres(t) + id := seedPatchImage(t) + if _, err := testPool.Exec(t.Context(), "UPDATE images SET status='queued' WHERE id=$1", id); err != nil { + t.Fatal(err) + } + // A trigger injects a real write failure for only this image and state. + name := "reject_upload_" + strings.ReplaceAll(id, "-", "") + _, err := testPool.Exec(t.Context(), fmt.Sprintf( + //sqlcheck:postgres-only + ` + CREATE FUNCTION %s() RETURNS trigger LANGUAGE plpgsql AS $$ + BEGIN + IF NEW.id = '%s'::uuid AND NEW.status = '%s' THEN + RAISE EXCEPTION 'injected image write failure'; + END IF; + RETURN NEW; + END $$; + CREATE TRIGGER %s BEFORE UPDATE ON images FOR EACH ROW EXECUTE FUNCTION %s();`, name, id, status, name, name)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + testPool.Exec(context.Background(), fmt.Sprintf("DROP TRIGGER %s ON images; DROP FUNCTION %s();", name, name)) + }) + code, body := doReqBody(newGlanceTestRouter(t), http.MethodPut, "/images/"+id+"/file", "test image", hdr(seededProjectID, "member")) + if code < 500 { + t.Errorf("failed state write returned %d %s; want 5xx", code, body) + } + if strings.Contains(body, "injected") { + t.Errorf("database details leaked: %s", body) + } + if got := imageColumn(t, id, "status"); got != "queued" { + t.Errorf("status=%s; want queued for retry", got) + } + }) + } +} + +func TestUnreadableNormalizedImageCannotBecomeActive(t *testing.T) { + requireDB(t) + id := seedPatchImage(t) + if _, err := testPool.Exec(t.Context(), "UPDATE images SET status='saving' WHERE id=$1", id); err != nil { + t.Fatal(err) + } + svc, err := NewService(testPool, "stub", "", "", storage.S3Options{}, nil) + if err != nil { + t.Fatal(err) + } + // No local object exists for this fresh UUID. StoredChecksum performs a real + // filesystem read and fails; the upload-stream hash must not be substituted. + if err := svc.activateUploadedImage(t.Context(), id, 1024, true, "original-stream-checksum"); err == nil { + t.Fatal("activated an unreadable normalized image") + } + if got := imageColumn(t, id, "status"); got != "saving" { + t.Errorf("status=%s; checksum failure must not activate the image", got) + } + if got := imageColumn(t, id, "checksum"); got != "" { + t.Errorf("recorded an unverified checksum %q", got) + } +} diff --git a/internal/keystone/application_credentials.go b/internal/keystone/application_credentials.go index 120cecd9..4b536b37 100644 --- a/internal/keystone/application_credentials.go +++ b/internal/keystone/application_credentials.go @@ -19,6 +19,12 @@ import ( func (svc *Service) ListApplicationCredentials(c *gin.Context) { userID := c.Param("id") + // Self-or-admin: credential metadata and role + // lists are footprint information. + userID, denied := denyIfNotSelfOrAdmin(c, userID) + if denied { + return + } rows, err := svc.pool.Query(c.Request.Context(), ` SELECT id, user_id, project_id, name, description, expires_at, unrestricted, created_at FROM application_credentials @@ -102,6 +108,13 @@ func (svc *Service) ListApplicationCredentials(c *gin.Context) { func (svc *Service) CreateApplicationCredential(c *gin.Context) { userID := c.Param("id") + // Self-or-admin: the 201 response hands back the + // plaintext secret, and requested roles are stored verbatim — minting one + // for another user would be both secret disclosure and impersonation. + userID, denied := denyIfNotSelfOrAdmin(c, userID) + if denied { + return + } var req struct { ApplicationCredential struct { Name string `json:"name" binding:"required"` @@ -201,6 +214,15 @@ func (svc *Service) GetApplicationCredential(c *gin.Context) { userID := c.Param("id") credID := c.Param("cred_id") + // Self-or-admin. + userID, denied := denyIfNotSelfOrAdmin(c, userID) + if denied { + return + } + if denyIfMalformedID(c, "Application Credential", credID) { + return + } + var id, userIDVal, name string var projectID, description *string var expiresAt *time.Time @@ -272,6 +294,16 @@ func (svc *Service) DeleteApplicationCredential(c *gin.Context) { userID := c.Param("id") credID := c.Param("cred_id") + // Self-or-admin: deleting another user's + // credential is an availability attack on their automation. + userID, denied := denyIfNotSelfOrAdmin(c, userID) + if denied { + return + } + if denyIfMalformedID(c, "Application Credential", credID) { + return + } + result, err := svc.pool.Exec(c.Request.Context(), "DELETE FROM application_credentials WHERE id = $1 AND user_id = $2", credID, userID) @@ -315,6 +347,13 @@ func (svc *Service) GetApplicationCredentialByID(c *gin.Context) { return } + // Owner-or-admin: a foreign credential answers + // 404 rather than 403 so the id-only lookup does not leak existence. + if userID != c.GetString("user_id") && !common.HasRole(c, common.RoleAdmin) { + common.SendError(c, common.NewIdentityNotFoundError("Application Credential", credID)) + return + } + credential := map[string]any{ "id": id, "user_id": userID, diff --git a/internal/keystone/authz.go b/internal/keystone/authz.go new file mode 100644 index 00000000..47629975 --- /dev/null +++ b/internal/keystone/authz.go @@ -0,0 +1,43 @@ +package keystone + +import ( + "github.com/cobaltcore-dev/lightstack/internal/common" + "github.com/gin-gonic/gin" + "github.com/google/uuid" +) + +// The identity management API is authentication-only by transport: every route +// under /v3 (except /auth/*) requires a valid token. What it historically did +// NOT require is authorization — the management surface (users, projects, +// roles, assignments, groups, domains, catalog, credentials) honored any +// authenticated caller, so a plain member could reset the admin's password, +// self-grant the admin role, or dump another tenant's S3 secret. These helpers +// put the same check the EC2-credentials API and ChangePassword already use +// (self-or-admin) plus the admin-only variant across that surface. +// +// Convention (cloud-storage#515, asserted by TestIdentityAuthzPrecedence): +// on a route with an authz check, the check runs in-handler BEFORE id +// validation, so a caller without permission gets 403 even for a malformed id. +// Such routes therefore sit on `mgmt` (no uuidParamMiddleware) and validate +// their path ids with denyIfMalformedID below, after the authz decision. + +// denyIfMalformedID writes the Keystone-verbatim 404 and returns true when id +// is not a well-formed UUID. Call it after the authz check (see above). +func denyIfMalformedID(c *gin.Context, kind, id string) bool { + if uuid.Validate(id) != nil { + common.SendError(c, common.NewIdentityNotFoundError(kind, id)) + return true + } + return false +} + +// sameUser compares UUID values, including the compact IDs used by RGW tokens. +// Invalid or missing caller IDs must never authorize self-service. +func sameUser(caller, target string) bool { + callerID, err := uuid.Parse(caller) + if err != nil { + return false + } + targetID, err := uuid.Parse(target) + return err == nil && callerID == targetID +} diff --git a/internal/keystone/credentials.go b/internal/keystone/credentials.go index 3b692f66..9c0d0c3e 100644 --- a/internal/keystone/credentials.go +++ b/internal/keystone/credentials.go @@ -14,11 +14,16 @@ import ( // ListCredentials returns all credentials func (svc *Service) ListCredentials(c *gin.Context) { + // The generic credentials API returns the + // recoverable EC2/S3 secret blob, so it is scoped like the EC2 API — + // a non-admin sees only their own credentials; an admin sees all. + callerID := c.GetString("user_id") rows, err := svc.pool.Query(c.Request.Context(), ` SELECT id, user_id, project_id, type, blob, created_at FROM credentials + WHERE user_id = $1 OR $2 ORDER BY created_at DESC - `) + `, callerID, common.HasRole(c, common.RoleAdmin)) if err != nil { log.Error().Err(err).Str("operation", "list_credentials").Msg("Failed to query credentials") common.SendError(c, common.NewInternalServerError("failed to query credentials")) @@ -75,6 +80,14 @@ func (svc *Service) CreateCredential(c *gin.Context) { return } + // Self-or-admin: a credential's blob is the + // recoverable secret consumed by S3/EC2 signature auth — minting one for + // another user (with a known secret) would defeat that auth. + if !sameUser(c.GetString("user_id"), req.Credential.UserID) && !common.HasRole(c, common.RoleAdmin) { + common.SendError(c, common.NewForbiddenError("cannot create credentials for another user")) + return + } + credID := uuid.New() now := time.Now() @@ -108,10 +121,38 @@ func (svc *Service) CreateCredential(c *gin.Context) { c.JSON(201, gin.H{"credential": credential}) } +// credentialOwnerOrViewer enforces the owner-or-admin rule on a single +// credential: 404 when it does not exist, 403 when it +// belongs to another user and the caller is not admin. +func (svc *Service) credentialOwnerOrViewer(c *gin.Context, credID string) bool { + var owner string + err := svc.pool.QueryRow(c.Request.Context(), + "SELECT user_id FROM credentials WHERE id = $1", credID, + ).Scan(&owner) + if errors.Is(err, database.ErrNoRows) { + common.SendError(c, common.NewIdentityNotFoundError("credential", credID)) + return false + } + if err != nil { + log.Error().Err(err).Str("operation", "credential_owner").Str("cred_id", credID).Msg("Failed to query credential") + common.SendError(c, common.NewInternalServerError("failed to query credential")) + return false + } + if !sameUser(c.GetString("user_id"), owner) && !common.HasRole(c, common.RoleAdmin) { + common.SendError(c, common.NewForbiddenError("cannot manage another user's credentials")) + return false + } + return true +} + // GetCredential returns a specific credential by ID func (svc *Service) GetCredential(c *gin.Context) { credID := c.Param("id") + if !svc.credentialOwnerOrViewer(c, credID) { + return + } + var id, userID, credType, blob string var projectID *string @@ -149,6 +190,10 @@ func (svc *Service) GetCredential(c *gin.Context) { func (svc *Service) UpdateCredential(c *gin.Context) { credID := c.Param("id") + if !svc.credentialOwnerOrViewer(c, credID) { + return + } + var req struct { Credential map[string]any `json:"credential"` } @@ -193,6 +238,10 @@ func (svc *Service) UpdateCredential(c *gin.Context) { func (svc *Service) DeleteCredential(c *gin.Context) { credID := c.Param("id") + if !svc.credentialOwnerOrViewer(c, credID) { + return + } + result, err := svc.pool.Exec(c.Request.Context(), "DELETE FROM credentials WHERE id = $1", credID) diff --git a/internal/keystone/domains.go b/internal/keystone/domains.go index f7965220..7d4c9e80 100644 --- a/internal/keystone/domains.go +++ b/internal/keystone/domains.go @@ -17,6 +17,13 @@ import ( // domain instead of the one they asked for (SPEC-022 FR-001a). Names are matched // exactly — upstream treats domain names as case-sensitive. func (svc *Service) ListDomains(c *gin.Context) { + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + query := `SELECT id, name, description, enabled FROM domains` args := []any{} if name := c.Query("name"); name != "" { @@ -69,6 +76,13 @@ func (svc *Service) ListDomains(c *gin.Context) { // CreateDomain handles POST /v3/domains func (svc *Service) CreateDomain(c *gin.Context) { + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + var req struct { Domain struct { Name string `json:"name" binding:"required"` @@ -120,6 +134,13 @@ func (svc *Service) CreateDomain(c *gin.Context) { func (svc *Service) GetDomain(c *gin.Context) { domainID := c.Param("id") + // Admin-only: domain records enumerate the + // deployment's trust structure. Checked before the alias resolution + // below, which doubles as this route's in-handler id validation (#515). + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // This route is registered on the non-validating group precisely so the `default` // alias reaches here (see the note in RegisterRoutes). Resolve it to a real id // first; the resolver also rejects any other non-identifier value without a query, @@ -168,6 +189,17 @@ func (svc *Service) GetDomain(c *gin.Context) { func (svc *Service) UpdateDomain(c *gin.Context) { domainID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "domain", domainID) { + return + } + var req struct { Domain map[string]any `json:"domain" binding:"required"` } @@ -257,6 +289,17 @@ func (svc *Service) UpdateDomain(c *gin.Context) { func (svc *Service) DeleteDomain(c *gin.Context) { domainID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "domain", domainID) { + return + } + // Check if domain is enabled var enabled bool err := svc.pool.QueryRow(c.Request.Context(), ` @@ -296,6 +339,17 @@ func (svc *Service) DeleteDomain(c *gin.Context) { func (svc *Service) GetDomainConfig(c *gin.Context) { domainID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "domain", domainID) { + return + } + // Verify domain exists var exists bool err := svc.pool.QueryRow(c.Request.Context(), ` diff --git a/internal/keystone/ec2_credentials.go b/internal/keystone/ec2_credentials.go index e6615d07..81153975 100644 --- a/internal/keystone/ec2_credentials.go +++ b/internal/keystone/ec2_credentials.go @@ -52,17 +52,10 @@ func ec2CredentialView(userID, tenantID, access, secret string) gin.H { // API returns the secret, so every handler must be owner-or-admin only — otherwise any // authenticated tenant could read or mint another user's S3 credential and reach that // tenant's RGW namespace (SPEC-014 FR-007; mirrors the self-or-admin check in -// ChangePassword). On success it returns the id in canonical form, so the caller's -// queries match on SQLite's text ids. +// ChangePassword). The management group's canonicalization middleware puts valid ids +// in canonical form so the caller's queries match on SQLite's text ids. func denyIfNotSelfOrAdmin(c *gin.Context, userID string) (string, bool) { - // Canonicalise a well-formed id up front so the self-check compares like with like - // (the token's user_id is canonical) and the caller's query matches on SQLite's text - // ids. A malformed id is left as-is: it cannot match the caller's own id, so an - // unauthorised caller still gets 403 before the 404 below (#515). - if parsed, err := uuid.Parse(userID); err == nil { - userID = parsed.String() - } - if c.GetString("user_id") != userID && !common.HasRole(c, common.RoleAdmin) { + if !sameUser(c.GetString("user_id"), userID) && !common.HasRole(c, common.RoleAdmin) { common.SendError(c, common.NewForbiddenError("cannot manage another user's EC2 credentials")) return "", true } @@ -187,7 +180,7 @@ func (svc *Service) ListEC2Credentials(c *gin.Context) { continue } tid := b.TenantID - if tid == "" && projectID != nil { + if projectID != nil { tid = *projectID } creds = append(creds, ec2CredentialView(userID, tid, b.Access, b.Secret)) @@ -228,7 +221,7 @@ func (svc *Service) GetEC2Credential(c *gin.Context) { var b ec2Blob _ = json.Unmarshal([]byte(blob), &b) tid := b.TenantID - if tid == "" && projectID != nil { + if projectID != nil { tid = *projectID } c.JSON(200, gin.H{"credential": ec2CredentialView(userID, tid, b.Access, b.Secret)}) diff --git a/internal/keystone/groups.go b/internal/keystone/groups.go index d43432aa..1684a955 100644 --- a/internal/keystone/groups.go +++ b/internal/keystone/groups.go @@ -14,6 +14,13 @@ import ( // ListGroups handles GET /v3/groups func (svc *Service) ListGroups(c *gin.Context) { + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + var f listFilter f.eq(c, "name", "name") f.eqDomain(c, "domain_id", "domain_id") @@ -61,6 +68,13 @@ func (svc *Service) ListGroups(c *gin.Context) { // CreateGroup handles POST /v3/groups func (svc *Service) CreateGroup(c *gin.Context) { + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + var req struct { Group struct { Name string `json:"name" binding:"required"` @@ -125,6 +139,17 @@ func (svc *Service) CreateGroup(c *gin.Context) { func (svc *Service) GetGroup(c *gin.Context) { groupID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "group", groupID) { + return + } + var name, domainID, description string var createdAt time.Time @@ -158,6 +183,17 @@ func (svc *Service) GetGroup(c *gin.Context) { func (svc *Service) UpdateGroup(c *gin.Context) { groupID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "group", groupID) { + return + } + var req struct { Group struct { Name *string `json:"name"` @@ -221,6 +257,17 @@ func (svc *Service) UpdateGroup(c *gin.Context) { func (svc *Service) DeleteGroup(c *gin.Context) { groupID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "group", groupID) { + return + } + result, err := svc.pool.Exec(c.Request.Context(), "DELETE FROM groups WHERE id = $1", groupID, @@ -244,6 +291,17 @@ func (svc *Service) DeleteGroup(c *gin.Context) { func (svc *Service) ListGroupUsers(c *gin.Context) { groupID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "group", groupID) { + return + } + rows, err := svc.pool.Query(c.Request.Context(), `SELECT u.id, u.name, u.domain_id, u.enabled FROM users u @@ -293,6 +351,17 @@ func (svc *Service) AddUserToGroup(c *gin.Context) { groupID := c.Param("id") userID := c.Param("user_id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "group", groupID) || denyIfMalformedID(c, "user", userID) { + return + } + _, err := svc.pool.Exec(c.Request.Context(), `INSERT INTO group_members (group_id, user_id) VALUES ($1, $2) @@ -313,6 +382,17 @@ func (svc *Service) RemoveUserFromGroup(c *gin.Context) { groupID := c.Param("id") userID := c.Param("user_id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "group", groupID) || denyIfMalformedID(c, "user", userID) { + return + } + result, err := svc.pool.Exec(c.Request.Context(), "DELETE FROM group_members WHERE group_id = $1 AND user_id = $2", groupID, userID, diff --git a/internal/keystone/handlers.go b/internal/keystone/handlers.go index 77fd72d9..b7153adf 100644 --- a/internal/keystone/handlers.go +++ b/internal/keystone/handlers.go @@ -53,7 +53,11 @@ func (svc *Service) RegisterRoutes(r *gin.RouterGroup, authMiddleware gin.Handle // All management routes require authentication mgmt := v3.Group("") - mgmt.Use(authMiddleware) + // Canonicalize well-formed UUID path params after authentication. This keeps + // uppercase and compact UUIDs usable on SQLite text columns while leaving + // malformed params untouched so in-handler authorization can still return + // 403 before its 404 validation. + mgmt.Use(authMiddleware, canonicalizeUUIDParams) // Malformed UUID path ids (params named "id" or "*_id") get a Keystone-verbatim // 404 before the handler, on every management route EXCEPT the authz-gated ones @@ -63,19 +67,22 @@ func (svc *Service) RegisterRoutes(r *gin.RouterGroup, authMiddleware gin.Handle val := mgmt.Group("") val.Use(uuidParamMiddleware) - // Users - val.GET("/users", svc.ListUsers) - val.POST("/users", svc.CreateUser) - val.GET("/users/:id", svc.GetUser) - val.PATCH("/users/:id", svc.UpdateUser) - mgmt.DELETE("/users/:id", svc.DeleteUser) // authz-gated (admin) - mgmt.POST("/users/:id/password", svc.ChangePassword) // authz-gated (self-or-admin) - val.GET("/users/:id/projects", svc.GetUserProjects) - val.GET("/users/:id/groups", svc.GetUserGroups) - val.POST("/users/:id/application_credentials", svc.CreateApplicationCredential) - val.GET("/users/:id/application_credentials", svc.ListApplicationCredentials) - val.GET("/users/:id/application_credentials/:cred_id", svc.GetApplicationCredential) - val.DELETE("/users/:id/application_credentials/:cred_id", svc.DeleteApplicationCredential) + // Users. Authz: reads and writes of the user + // collection are admin-only; user-scoped subresources are self-or-admin. + // Routes with an authz check sit on `mgmt` and validate ids in-handler so + // 403 precedes the 404 (#515, asserted by TestIdentityAuthzPrecedence). + val.GET("/users", svc.ListUsers) // admin-only (in-handler) + val.POST("/users", svc.CreateUser) // admin-only (in-handler) + mgmt.GET("/users/:id", svc.GetUser) // self-or-admin + mgmt.PATCH("/users/:id", svc.UpdateUser) // self-or-admin, admin-only fields + mgmt.DELETE("/users/:id", svc.DeleteUser) // authz-gated (admin) + mgmt.POST("/users/:id/password", svc.ChangePassword) // authz-gated (self-or-admin) + mgmt.GET("/users/:id/projects", svc.GetUserProjects) // self-or-admin + mgmt.GET("/users/:id/groups", svc.GetUserGroups) // self-or-admin + mgmt.POST("/users/:id/application_credentials", svc.CreateApplicationCredential) // self-or-admin + mgmt.GET("/users/:id/application_credentials", svc.ListApplicationCredentials) // self-or-admin + mgmt.GET("/users/:id/application_credentials/:cred_id", svc.GetApplicationCredential) // self-or-admin + mgmt.DELETE("/users/:id/application_credentials/:cred_id", svc.DeleteApplicationCredential) // self-or-admin // EC2/S3 credentials — authz-gated (self-or-admin via denyIfNotSelfOrAdmin); // the :id (user) is validated in-handler after that check (SPEC-014 FR-005). mgmt.POST("/users/:id/credentials/OS-EC2", svc.CreateEC2Credential) @@ -83,36 +90,38 @@ func (svc *Service) RegisterRoutes(r *gin.RouterGroup, authMiddleware gin.Handle mgmt.GET("/users/:id/credentials/OS-EC2/:access", svc.GetEC2Credential) mgmt.DELETE("/users/:id/credentials/OS-EC2/:access", svc.DeleteEC2Credential) - // Projects (role assignments must come before /projects/:id to avoid conflicts) - val.GET("/projects", svc.ListProjects) - val.POST("/projects", svc.CreateProject) - val.PUT("/projects/:id/users/:user_id/roles/:role_id", svc.AssignRole) - val.DELETE("/projects/:id/users/:user_id/roles/:role_id", svc.UnassignRole) - val.GET("/projects/:id/users/:user_id/roles", svc.ListUserProjectRoles) - val.GET("/projects/:id/users/:user_id/roles/:role_id", svc.CheckRoleAssignment) - val.GET("/projects/:id", svc.GetProject) - val.PATCH("/projects/:id", svc.UpdateProject) - val.DELETE("/projects/:id", svc.DeleteProject) - - // Groups - val.GET("/groups", svc.ListGroups) - val.POST("/groups", svc.CreateGroup) - val.GET("/groups/:id", svc.GetGroup) - val.PATCH("/groups/:id", svc.UpdateGroup) - val.DELETE("/groups/:id", svc.DeleteGroup) - val.GET("/groups/:id/users", svc.ListGroupUsers) - val.PUT("/groups/:id/users/:user_id", svc.AddUserToGroup) - val.DELETE("/groups/:id/users/:user_id", svc.RemoveUserFromGroup) - - // Roles - val.GET("/roles", svc.ListRoles) - val.POST("/roles", svc.CreateRole) - val.GET("/roles/:id", svc.GetRole) - val.PATCH("/roles/:id", svc.UpdateRole) - val.DELETE("/roles/:id", svc.DeleteRole) - - // Role Assignments - val.GET("/role_assignments", svc.ListRoleAssignments) + // Projects (role assignments must come before /projects/:id to avoid conflicts). + // Authz: project management and role assignment + // are admin-only — the assignment pair is the privilege-escalation primitive. + val.GET("/projects", svc.ListProjects) // admin-only (in-handler) + val.POST("/projects", svc.CreateProject) // admin-only (in-handler) + mgmt.PUT("/projects/:id/users/:user_id/roles/:role_id", svc.AssignRole) // admin-only + mgmt.DELETE("/projects/:id/users/:user_id/roles/:role_id", svc.UnassignRole) // admin-only + mgmt.GET("/projects/:id/users/:user_id/roles", svc.ListUserProjectRoles) // admin-only + mgmt.GET("/projects/:id/users/:user_id/roles/:role_id", svc.CheckRoleAssignment) // admin-only + mgmt.GET("/projects/:id", svc.GetProject) // admin-only + mgmt.PATCH("/projects/:id", svc.UpdateProject) // admin-only + mgmt.DELETE("/projects/:id", svc.DeleteProject) // admin-only + + // Groups — admin-only. + val.GET("/groups", svc.ListGroups) // admin-only (in-handler) + val.POST("/groups", svc.CreateGroup) // admin-only (in-handler) + mgmt.GET("/groups/:id", svc.GetGroup) + mgmt.PATCH("/groups/:id", svc.UpdateGroup) + mgmt.DELETE("/groups/:id", svc.DeleteGroup) + mgmt.GET("/groups/:id/users", svc.ListGroupUsers) + mgmt.PUT("/groups/:id/users/:user_id", svc.AddUserToGroup) + mgmt.DELETE("/groups/:id/users/:user_id", svc.RemoveUserFromGroup) + + // Roles — admin-only. + val.GET("/roles", svc.ListRoles) // admin-only (in-handler) + val.POST("/roles", svc.CreateRole) // admin-only (in-handler) + mgmt.GET("/roles/:id", svc.GetRole) + mgmt.PATCH("/roles/:id", svc.UpdateRole) + mgmt.DELETE("/roles/:id", svc.DeleteRole) + + // Role Assignments — admin-only. + val.GET("/role_assignments", svc.ListRoleAssignments) // admin-only (in-handler) // Domains. // @@ -125,24 +134,26 @@ func (svc *Service) RegisterRoutes(r *gin.RouterGroup, authMiddleware gin.Handle // The exception is one route, one method, one alias. Every other non-identifier // value on it still gets #515's verbatim 404, produced in-handler; and PATCH, // DELETE and /config stay on `val`, where the middleware still answers. - val.GET("/domains", svc.ListDomains) - val.POST("/domains", svc.CreateDomain) - mgmt.GET("/domains/:id", svc.GetDomain) // alias-aware (see above) - val.PATCH("/domains/:id", svc.UpdateDomain) - val.DELETE("/domains/:id", svc.DeleteDomain) - val.GET("/domains/:id/config", svc.GetDomainConfig) - - // Services (catalog management) - val.GET("/services", svc.ListServices) - val.POST("/services", svc.CreateService) - val.GET("/services/:id", svc.GetService) - val.PATCH("/services/:id", svc.UpdateService) - val.DELETE("/services/:id", svc.DeleteService) - - // Endpoints - val.GET("/endpoints", svc.ListEndpoints) - val.POST("/endpoints", svc.CreateEndpoint) - val.DELETE("/endpoints/:id", svc.DeleteEndpoint) + // + // Domain management is admin-only. + val.GET("/domains", svc.ListDomains) // admin-only (in-handler) + val.POST("/domains", svc.CreateDomain) // admin-only (in-handler) + mgmt.GET("/domains/:id", svc.GetDomain) // alias-aware (see above), admin-only + mgmt.PATCH("/domains/:id", svc.UpdateDomain) // admin-only + mgmt.DELETE("/domains/:id", svc.DeleteDomain) // admin-only + mgmt.GET("/domains/:id/config", svc.GetDomainConfig) // admin-only + + // Services (catalog management) — admin-only. + val.GET("/services", svc.ListServices) // admin-only (in-handler) + val.POST("/services", svc.CreateService) // admin-only (in-handler) + mgmt.GET("/services/:id", svc.GetService) + mgmt.PATCH("/services/:id", svc.UpdateService) + mgmt.DELETE("/services/:id", svc.DeleteService) + + // Endpoints — admin-only. + val.GET("/endpoints", svc.ListEndpoints) // admin-only (in-handler) + val.POST("/endpoints", svc.CreateEndpoint) // admin-only (in-handler) + mgmt.DELETE("/endpoints/:id", svc.DeleteEndpoint) // Credentials val.GET("/credentials", svc.ListCredentials) @@ -390,6 +401,11 @@ func (svc *Service) ListAuthProjects(c *gin.Context) { // ListUsers lists all users func (svc *Service) ListUsers(c *gin.Context) { + // Admin-only: the user collection exposes every + // account's name, enabled state and domain — identity enumeration. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } var f listFilter f.eq(c, "name", "name") f.eqDomain(c, "domain_id", "domain_id") @@ -434,6 +450,13 @@ func (svc *Service) ListUsers(c *gin.Context) { func (svc *Service) GetUser(c *gin.Context) { userID := c.Param("id") + // Self-or-admin: user records (name, email, + // enabled) are only for the user itself or an admin. + userID, denied := denyIfNotSelfOrAdmin(c, userID) + if denied { + return + } + var id, name, domainID string var enabled bool err := svc.pool.QueryRow(c.Request.Context(), @@ -457,6 +480,13 @@ func (svc *Service) GetUser(c *gin.Context) { // ListProjects lists all projects func (svc *Service) ListProjects(c *gin.Context) { + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + var f listFilter f.eq(c, "name", "name") f.eqDomain(c, "domain_id", "domain_id") @@ -531,6 +561,17 @@ func identityLink(c *gin.Context, collection, id string) gin.H { func (svc *Service) GetProject(c *gin.Context) { projectID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "project", projectID) { + return + } + var id, name string var description, domainID sql.NullString // nullable, as in ListProjects var enabled bool @@ -556,6 +597,13 @@ func (svc *Service) GetProject(c *gin.Context) { // CreateProject handles POST /v3/projects func (svc *Service) CreateProject(c *gin.Context) { + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + var req struct { Project struct { Name string `json:"name" binding:"required"` @@ -630,6 +678,17 @@ func (svc *Service) CreateProject(c *gin.Context) { func (svc *Service) UpdateProject(c *gin.Context) { projectID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "project", projectID) { + return + } + var req struct { Project struct { Name *string `json:"name"` @@ -697,6 +756,17 @@ func (svc *Service) UpdateProject(c *gin.Context) { func (svc *Service) DeleteProject(c *gin.Context) { projectID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "project", projectID) { + return + } + result, err := svc.pool.Exec(c.Request.Context(), "DELETE FROM projects WHERE id = $1", projectID, @@ -718,6 +788,13 @@ func (svc *Service) DeleteProject(c *gin.Context) { // ListRoles lists all roles func (svc *Service) ListRoles(c *gin.Context) { + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + var f listFilter f.eq(c, "name", "name") rows, err := svc.pool.Query(c.Request.Context(), @@ -757,6 +834,17 @@ func (svc *Service) ListRoles(c *gin.Context) { func (svc *Service) GetRole(c *gin.Context) { roleID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "role", roleID) { + return + } + var id, name string err := svc.pool.QueryRow(c.Request.Context(), "SELECT id, name FROM roles WHERE id = $1", @@ -777,6 +865,13 @@ func (svc *Service) GetRole(c *gin.Context) { // CreateRole handles POST /v3/roles func (svc *Service) CreateRole(c *gin.Context) { + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + var req struct { Role struct { Name string `json:"name" binding:"required"` @@ -816,6 +911,17 @@ func (svc *Service) CreateRole(c *gin.Context) { func (svc *Service) UpdateRole(c *gin.Context) { roleID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "role", roleID) { + return + } + var req struct { Role struct { Name *string `json:"name"` @@ -861,6 +967,17 @@ func (svc *Service) UpdateRole(c *gin.Context) { func (svc *Service) DeleteRole(c *gin.Context) { roleID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "role", roleID) { + return + } + result, err := svc.pool.Exec(c.Request.Context(), "DELETE FROM roles WHERE id = $1", roleID, @@ -886,8 +1003,41 @@ func (svc *Service) AssignRole(c *gin.Context) { userID := c.Param("user_id") roleID := c.Param("role_id") + // Admin-only: role assignment is the privilege + // escalation primitive — without this check any authenticated user could + // grant itself (or anyone) the admin role on any project. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "project", projectID) || denyIfMalformedID(c, "user", userID) || + denyIfMalformedID(c, "role", roleID) { + return + } + + // The foreign keys prevent dangling assignments; these lookups preserve + // Keystone's resource-specific 404 instead of leaking a database error. + var projectExists, userExists, roleExists bool + err := svc.pool.QueryRow(c.Request.Context(), ` + SELECT EXISTS(SELECT 1 FROM projects WHERE id = $1), + EXISTS(SELECT 1 FROM users WHERE id = $2), + EXISTS(SELECT 1 FROM roles WHERE id = $3) + `, projectID, userID, roleID).Scan(&projectExists, &userExists, &roleExists) + for _, tc := range []struct { + kind, id string + exists bool + }{ + {"project", projectID, projectExists}, + {"user", userID, userExists}, + {"role", roleID, roleExists}, + } { + if existsOr404(c, err, tc.exists, tc.kind, tc.id) { + return + } + } + // Insert role assignment (idempotent) - _, err := svc.pool.Exec(c.Request.Context(), + _, err = svc.pool.Exec(c.Request.Context(), `INSERT INTO role_assignments (id, user_id, project_id, role_id, created_at) VALUES (gen_random_uuid(), $1, $2, $3, NOW()) ON CONFLICT (user_id, project_id, role_id) DO NOTHING`, @@ -908,6 +1058,17 @@ func (svc *Service) UnassignRole(c *gin.Context) { userID := c.Param("user_id") roleID := c.Param("role_id") + // Admin-only: revoking roles is the inverse + // escalation — any user could strip an admin's role and lock the + // deployment out. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + if denyIfMalformedID(c, "project", projectID) || denyIfMalformedID(c, "user", userID) || + denyIfMalformedID(c, "role", roleID) { + return + } + result, err := svc.pool.Exec(c.Request.Context(), "DELETE FROM role_assignments WHERE user_id = $1 AND project_id = $2 AND role_id = $3", userID, projectID, roleID, @@ -932,6 +1093,17 @@ func (svc *Service) ListUserProjectRoles(c *gin.Context) { projectID := c.Param("id") userID := c.Param("user_id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "project", projectID) || denyIfMalformedID(c, "user", userID) { + return + } + rows, err := svc.pool.Query(c.Request.Context(), `SELECT r.id, r.name FROM roles r @@ -978,6 +1150,17 @@ func (svc *Service) CheckRoleAssignment(c *gin.Context) { userID := c.Param("user_id") roleID := c.Param("role_id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "project", projectID) || denyIfMalformedID(c, "user", userID) || denyIfMalformedID(c, "role", roleID) { + return + } + var exists bool err := svc.pool.QueryRow(c.Request.Context(), `SELECT EXISTS( @@ -1003,10 +1186,13 @@ func (svc *Service) CheckRoleAssignment(c *gin.Context) { // ListRoleAssignments handles GET /v3/role_assignments func (svc *Service) ListRoleAssignments(c *gin.Context) { - // Build the WHERE from the id filters that are present. eqID canonicalises each id, so - // a dash-free or upper-case value matches on SQLite (ids are text) as it did on - // PostgreSQL's uuid columns, and a non-UUID matches nothing rather than 500ing — the - // same behaviour every other identity collection has (list_filters.go). + // Admin-only: role assignments grant authority and expose + // the identity management surface. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + + // Canonical UUID filters preserve alternate spellings on both database backends. var f listFilter f.eqID(c, "user.id", "ra.user_id") f.eqID(c, "scope.project.id", "ra.project_id") @@ -1161,6 +1347,20 @@ func (svc *Service) CreateUser(c *gin.Context) { func (svc *Service) UpdateUser(c *gin.Context) { userID := c.Param("id") + // Self-or-admin: without this check any + // authenticated caller could reset any user's password — including the + // admin's — or disable any account, because this handler (unlike + // ChangePassword) never asked for the original password. + isAdmin := common.HasRole(c, common.RoleAdmin) + if !sameUser(c.GetString("user_id"), userID) && !isAdmin { + common.SendError(c, common.NewForbiddenError("cannot update another user")) + return + } + // Authorization passed; validate the id after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "user", userID) { + return + } + var req struct { User struct { Name *string `json:"name"` @@ -1176,6 +1376,15 @@ func (svc *Service) UpdateUser(c *gin.Context) { return } + // A non-admin may only edit their own profile fields. Setting a password + // without the original (ChangePassword's contract) or flipping `enabled` + // stays admin-only — otherwise this route would be a weaker password + // change and a self-enable. + if !isAdmin && (req.User.Password != nil || req.User.Enabled != nil) { + common.SendError(c, common.NewForbiddenError("admin role required to set password or enabled")) + return + } + // Check if user exists var exists bool err := svc.pool.QueryRow(c.Request.Context(), @@ -1267,12 +1476,6 @@ func (svc *Service) UpdateUser(c *gin.Context) { func (svc *Service) DeleteUser(c *gin.Context) { userID := c.Param("id") - // Canonicalise a well-formed id so the delete matches on SQLite's text ids; a - // malformed id is left for the uuid.Validate guard below to 404 after the authz check. - if parsed, err := uuid.Parse(userID); err == nil { - userID = parsed.String() - } - // Only admins may delete users if !common.HasRole(c, common.RoleAdmin) { common.SendError(c, common.NewForbiddenError("admin role required to delete users")) @@ -1280,8 +1483,7 @@ func (svc *Service) DeleteUser(c *gin.Context) { } // Authorization passed; validate the id after it so 403 precedes this 404 (#515). - if uuid.Validate(userID) != nil { - common.SendError(c, common.NewIdentityNotFoundError("user", userID)) + if denyIfMalformedID(c, "user", userID) { return } @@ -1311,24 +1513,15 @@ func (svc *Service) DeleteUser(c *gin.Context) { func (svc *Service) ChangePassword(c *gin.Context) { userID := c.Param("id") - // Canonicalise a well-formed id up front so the self-check compares like with like - // (the token's user_id is canonical) and the queries match on SQLite's text ids. A - // malformed id is left as-is: it can't match the caller's own id, so an unauthorised - // caller still gets 403 before the 404 below (#515). - if parsed, err := uuid.Parse(userID); err == nil { - userID = parsed.String() - } - // Users may only change their own password; admins may change any user's password callerID := c.GetString("user_id") - if callerID != userID && !common.HasRole(c, common.RoleAdmin) { + if !sameUser(callerID, userID) && !common.HasRole(c, common.RoleAdmin) { common.SendError(c, common.NewForbiddenError("cannot change another user's password")) return } // Authorization passed; validate the id after it so 403 precedes this 404 (#515). - if uuid.Validate(userID) != nil { - common.SendError(c, common.NewIdentityNotFoundError("user", userID)) + if denyIfMalformedID(c, "user", userID) { return } @@ -1389,6 +1582,13 @@ func (svc *Service) ChangePassword(c *gin.Context) { func (svc *Service) GetUserProjects(c *gin.Context) { userID := c.Param("id") + // Self-or-admin: a user's project membership is + // footprint information. + userID, denied := denyIfNotSelfOrAdmin(c, userID) + if denied { + return + } + // Check if user exists var exists bool err := svc.pool.QueryRow(c.Request.Context(), @@ -1449,6 +1649,12 @@ func (svc *Service) GetUserProjects(c *gin.Context) { func (svc *Service) GetUserGroups(c *gin.Context) { userID := c.Param("id") + // Self-or-admin. + userID, denied := denyIfNotSelfOrAdmin(c, userID) + if denied { + return + } + // Check if user exists var exists bool err := svc.pool.QueryRow(c.Request.Context(), diff --git a/internal/keystone/identity_authz_gates_test.go b/internal/keystone/identity_authz_gates_test.go new file mode 100644 index 00000000..e69f3fe5 --- /dev/null +++ b/internal/keystone/identity_authz_gates_test.go @@ -0,0 +1,194 @@ +package keystone + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +// TestIdentityManagementAuthz_Gates pins the identity management gates: user updates, role assignment, +// project/role/group/domain/catalog management — must not honor a plain +// member. Before the fix every one of these answered 2xx to any authenticated +// caller, making PATCH /users/:id a passwordless account takeover and +// PUT .../roles/:role_id a one-request privilege escalation. +func TestIdentityManagementAuthz_Gates(t *testing.T) { + requireDB(t) + r := newIdentityTestRouter() + + const adminUID = "00000000-0000-0000-0000-000000000001" // seeded admin + const projectID = "00000000-0000-0000-0000-000000000002" + const memberUID = "00000000-0000-0000-0000-0000000000aa" + member := map[string]string{"X-Test-Roles": "member", "X-Test-User": memberUID} + + // A valid target id, so these failures are the authz check and not the 404. + takeover := `{"user":{"password":"attacker-controlled"}}` + + for _, tc := range []struct { + name, method, path, body string + }{ + {"member cannot reset another user's password (C-1)", "PATCH", "/v3/users/" + adminUID, takeover}, + {"member cannot disable another user (C-1)", "PATCH", "/v3/users/" + adminUID, `{"user":{"enabled":false}}`}, + {"member cannot self-grant a role (C-2)", "PUT", "/v3/projects/" + projectID + "/users/" + memberUID + "/roles/00000000-0000-0000-0000-000000000003", ""}, + {"member cannot revoke a role (C-2)", "DELETE", "/v3/projects/" + projectID + "/users/" + adminUID + "/roles/00000000-0000-0000-0000-000000000003", ""}, + {"member cannot create users", "POST", "/v3/users", `{"user":{"name":"x"}}`}, + {"member cannot enumerate users (M-5)", "GET", "/v3/users", ""}, + {"member cannot read another user's record (M-5)", "GET", "/v3/users/" + adminUID, ""}, + {"member cannot create projects", "POST", "/v3/projects", `{"project":{"name":"x"}}`}, + {"member cannot update projects", "PATCH", "/v3/projects/" + projectID, `{"project":{"name":"x"}}`}, + {"member cannot delete projects", "DELETE", "/v3/projects/" + projectID, ""}, + {"member cannot create roles", "POST", "/v3/roles", `{"role":{"name":"x"}}`}, + {"member cannot list role assignments (M-5)", "GET", "/v3/role_assignments", ""}, + {"member cannot create groups", "POST", "/v3/groups", `{"group":{"name":"x"}}`}, + {"member cannot list domains", "GET", "/v3/domains", ""}, + {"member cannot create services", "POST", "/v3/services", `{"service":{"name":"x","type":"compute"}}`}, + {"member cannot list endpoints", "GET", "/v3/endpoints", ""}, + {"member cannot mint credentials for another user (C-3)", "POST", "/v3/credentials", `{"credential":{"user_id":"` + adminUID + `","type":"ec2","blob":"{}"}}`}, + {"member cannot mint an application credential for another user (H-4)", "POST", "/v3/users/" + adminUID + "/application_credentials", `{"application_credential":{"name":"x"}}`}, + {"member cannot list another user's application credentials (H-4)", "GET", "/v3/users/" + adminUID + "/application_credentials", ""}, + } { + t.Run(tc.name, func(t *testing.T) { + req := httptest.NewRequest(tc.method, tc.path, strings.NewReader(tc.body)) + if tc.body != "" { + req.Header.Set("Content-Type", "application/json") + } + for k, v := range member { + req.Header.Set(k, v) + } + w := httptest.NewRecorder() + r.ServeHTTP(w, req) + if w.Code != http.StatusForbidden { + t.Errorf("%s %s: code=%d body=%s; want 403", tc.method, tc.path, w.Code, w.Body.String()) + } + }) + } +} + +// TestIdentitySelfServiceStillWorks is the complement: the gates must not lock +// a member out of their own record — PATCH of one's own profile fields (name, +// email) stays allowed, while the admin-only fields on the same route +// (password, enabled) stay denied even for self. +func TestIdentitySelfServiceStillWorks(t *testing.T) { + requireDB(t) + r := newIdentityTestRouter() + + // A real member row — the authz pass must not lock a user out of their + // own record. + req := httptest.NewRequest("POST", "/v3/users", strings.NewReader( + `{"user":{"name":"authz-self","password":"Scratch-1","domain_id":"default"}}`)) + req.Header.Set("Content-Type", "application/json") + for k, v := range adminHdr { + req.Header.Set(k, v) + } + w := httptest.NewRecorder() + r.ServeHTTP(w, req) + if w.Code != http.StatusCreated { + t.Fatalf("create scratch user: code=%d body=%s", w.Code, w.Body.String()) + } + var created struct { + User struct { + ID string `json:"id"` + } `json:"user"` + } + if err := json.Unmarshal(w.Body.Bytes(), &created); err != nil || created.User.ID == "" { + t.Fatalf("create scratch user: no id in %s", w.Body.String()) + } + t.Cleanup(func() { + req := httptest.NewRequest("DELETE", "/v3/users/"+created.User.ID, nil) + for k, v := range adminHdr { + req.Header.Set(k, v) + } + w := httptest.NewRecorder() + r.ServeHTTP(w, req) + }) + member := map[string]string{"X-Test-Roles": "member", "X-Test-User": created.User.ID} + + for _, alias := range []string{strings.ToUpper(created.User.ID), strings.ReplaceAll(created.User.ID, "-", "")} { + for _, tc := range []struct{ method, suffix, body string }{ + {"GET", "", ""}, + {"GET", "/application_credentials", ""}, + {"PATCH", "", `{"user":{"email":"self@example.com"}}`}, + } { + req := httptest.NewRequest(tc.method, "/v3/users/"+alias+tc.suffix, strings.NewReader(tc.body)) + req.Header.Set("Content-Type", "application/json") + for k, v := range member { + req.Header.Set(k, v) + } + w := httptest.NewRecorder() + r.ServeHTTP(w, req) + if w.Code != http.StatusOK { + t.Errorf("self %s %s: %d %s", tc.method, alias+tc.suffix, w.Code, w.Body.String()) + } + } + } + + doPatch := func(body string) int { + req := httptest.NewRequest("PATCH", "/v3/users/"+created.User.ID, strings.NewReader(body)) + req.Header.Set("Content-Type", "application/json") + for k, v := range member { + req.Header.Set(k, v) + } + w := httptest.NewRecorder() + r.ServeHTTP(w, req) + return w.Code + } + + if code := doPatch(`{"user":{"email":"member@example.com"}}`); code != http.StatusOK { + t.Errorf("self PATCH of profile fields: code=%d; want 200", code) + } + if code := doPatch(`{"user":{"password":"new-password"}}`); code != http.StatusForbidden { + t.Errorf("self PATCH of password: code=%d; want 403 (use ChangePassword, which requires the original)", code) + } + if code := doPatch(`{"user":{"enabled":false}}`); code != http.StatusForbidden { + t.Errorf("self PATCH of enabled: code=%d; want 403", code) + } +} + +// TestUpdateUserAdminPathUnchanged pins the admin side of the C-1 fix: an +// admin can still set password/enabled through PATCH /users/:id. +func TestUpdateUserAdminPathUnchanged(t *testing.T) { + requireDB(t) + r := newIdentityTestRouter() + + do := func(method, path, body string, hdr map[string]string) (*httptest.ResponseRecorder, string) { + var reader *strings.Reader + if body != "" { + reader = strings.NewReader(body) + } else { + reader = strings.NewReader("") + } + req := httptest.NewRequest(method, path, reader) + if body != "" { + req.Header.Set("Content-Type", "application/json") + } + for k, v := range hdr { + req.Header.Set(k, v) + } + w := httptest.NewRecorder() + r.ServeHTTP(w, req) + return w, w.Body.String() + } + + // Create a scratch user as admin. + w, body := do("POST", "/v3/users", `{"user":{"name":"authz-scratch","password":"Scratch-1","domain_id":"default"}}`, adminHdr) + if w.Code != http.StatusCreated { + t.Fatalf("create scratch user: code=%d body=%s", w.Code, body) + } + var created struct { + User struct { + ID string `json:"id"` + } `json:"user"` + } + if err := json.Unmarshal([]byte(body), &created); err != nil || created.User.ID == "" { + t.Fatalf("create scratch user: no id in %s", body) + } + t.Cleanup(func() { do("DELETE", "/v3/users/"+created.User.ID, "", adminHdr) }) + + // Admin PATCH with password + enabled must still succeed. + w, body = do("PATCH", "/v3/users/"+created.User.ID, + `{"user":{"password":"Scratch-2","enabled":true}}`, adminHdr) + if w.Code != http.StatusOK { + t.Errorf("admin PATCH password/enabled: code=%d body=%s; want 200", w.Code, body) + } +} diff --git a/internal/keystone/identity_notfound_test.go b/internal/keystone/identity_notfound_test.go index b6017035..a7292158 100644 --- a/internal/keystone/identity_notfound_test.go +++ b/internal/keystone/identity_notfound_test.go @@ -13,17 +13,30 @@ import ( // TestIdentityNotFound_ReadPaths (US1) covers the ticket's core resources: a // malformed OR a valid-but-unknown id returns the identical Keystone-verbatim // 404 ("Could not find : ."), while an existing id still 200s. -// DB-backed (requireDB); routes are wired via the real RegisterRoutes. +// DB-backed (requireDB); routes are wired via the real RegisterRoutes. These +// reads are admin/self-gated, so the caller is admin — +// the 404 shape is what this test pins, not who may see it. func TestIdentityNotFound_ReadPaths(t *testing.T) { requireDB(t) gin.SetMode(gin.TestMode) r := gin.New() svc := NewService(testPool, nil, nil) - svc.RegisterRoutes(&r.RouterGroup, func(c *gin.Context) { c.Next() }) + svc.RegisterRoutes(&r.RouterGroup, func(c *gin.Context) { + if roles := c.GetHeader("X-Test-Roles"); roles != "" { + c.Set("roles", strings.Split(roles, ",")) + } + if uid := c.GetHeader("X-Test-User"); uid != "" { + c.Set("user_id", uid) + } + c.Next() + }) serve := func(path string) (int, string) { + req := httptest.NewRequest(http.MethodGet, path, nil) + req.Header.Set("X-Test-Roles", "admin") + req.Header.Set("X-Test-User", "00000000-0000-0000-0000-00000000a0a0") w := httptest.NewRecorder() - r.ServeHTTP(w, httptest.NewRequest(http.MethodGet, path, nil)) + r.ServeHTTP(w, req) return w.Code, w.Body.String() } @@ -40,7 +53,7 @@ func TestIdentityNotFound_ReadPaths(t *testing.T) { t.Fatalf("no seeded %s row to test the 200 case: %v", res.noun, err) } - // malformed id -> verbatim 404 (via middleware, before the DB) + // malformed id -> verbatim 404 (after the authz check, before the DB) if code, body := serve(res.base + "not-a-uuid"); code != http.StatusNotFound || !strings.Contains(body, "Could not find "+res.noun+": not-a-uuid.") { t.Errorf("malformed: code=%d body=%s", code, body) diff --git a/internal/keystone/review_authz_regressions_test.go b/internal/keystone/review_authz_regressions_test.go new file mode 100644 index 00000000..aec976fc --- /dev/null +++ b/internal/keystone/review_authz_regressions_test.go @@ -0,0 +1,109 @@ +package keystone + +import ( + "context" + "net/http" + "strings" + "testing" + + "github.com/google/uuid" +) + +func identityUserHeaders(userID, roles string) map[string]string { + return map[string]string{"X-Test-User": userID, "X-Test-Roles": roles} +} + +func TestGenericCredentialOwnerAndAdminAuthorization(t *testing.T) { + requireDB(t) + r := newIdentityTestRouter() + ownerID, outsiderID := "00000000-0000-0000-0000-000000000001", uuid.NewString() + ownerCredentialID, adminCredentialID := uuid.NewString(), uuid.NewString() + for _, id := range []string{ownerCredentialID, adminCredentialID} { + if _, err := testPool.Exec(t.Context(), "INSERT INTO credentials (id, user_id, type, blob, created_at, updated_at) VALUES ($1, $2, 'ec2', 'review-secret', now(), now())", id, ownerID); err != nil { + t.Fatalf("seed credential %s: %v", id, err) + } + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), "DELETE FROM credentials WHERE id IN ($1, $2)", ownerCredentialID, adminCredentialID) + }) + + owner := identityUserHeaders(ownerID, "member") + outsider := identityUserHeaders(outsiderID, "member") + admin := identityUserHeaders(outsiderID, "admin") + for _, tc := range []struct{ method, path, body string }{ + {http.MethodGet, "/v3/credentials/" + ownerCredentialID, ""}, + {http.MethodPatch, "/v3/credentials/" + ownerCredentialID, "{\"credential\":{\"blob\":\"outsider-secret\"}}"}, + {http.MethodDelete, "/v3/credentials/" + ownerCredentialID, ""}, + } { + if code, body := doJSONReq(r, tc.method, tc.path, tc.body, outsider); code != http.StatusForbidden { + t.Errorf("outsider %s %s: code=%d body=%s; want 403", tc.method, tc.path, code, body) + } + } + for _, tc := range []struct { + headers map[string]string + wantID string + }{{owner, ownerCredentialID}, {outsider, ""}, {admin, adminCredentialID}} { + code, body := doJSONReq(r, http.MethodGet, "/v3/credentials", "", tc.headers) + if code != http.StatusOK || (tc.wantID != "" && !strings.Contains(body, tc.wantID)) || (tc.wantID == "" && strings.Contains(body, ownerCredentialID)) { + t.Errorf("credential list: code=%d body=%s", code, body) + } + } + if code, body := doJSONReq(r, http.MethodGet, "/v3/credentials/"+ownerCredentialID, "", owner); code != http.StatusOK || !strings.Contains(body, "review-secret") { + t.Errorf("owner get credential: code=%d body=%s", code, body) + } + if code, body := doJSONReq(r, http.MethodPatch, "/v3/credentials/"+ownerCredentialID, "{\"credential\":{\"blob\":\"owner-updated\"}}", owner); code != http.StatusOK || !strings.Contains(body, "owner-updated") { + t.Errorf("owner update credential: code=%d body=%s", code, body) + } + if code, body := doJSONReq(r, http.MethodDelete, "/v3/credentials/"+ownerCredentialID, "", owner); code != http.StatusNoContent { + t.Errorf("owner delete credential: code=%d body=%s", code, body) + } + if code, body := doJSONReq(r, http.MethodGet, "/v3/credentials/"+adminCredentialID, "", admin); code != http.StatusOK { + t.Errorf("admin get credential: code=%d body=%s", code, body) + } + if code, body := doJSONReq(r, http.MethodPatch, "/v3/credentials/"+adminCredentialID, "{\"credential\":{\"blob\":\"admin-updated\"}}", admin); code != http.StatusOK || !strings.Contains(body, "admin-updated") { + t.Errorf("admin update credential: code=%d body=%s", code, body) + } + if code, body := doJSONReq(r, http.MethodDelete, "/v3/credentials/"+adminCredentialID, "", admin); code != http.StatusNoContent { + t.Errorf("admin delete credential: code=%d body=%s", code, body) + } +} + +func TestApplicationCredentialMalformedIDsAreValidatedAfterAuthorization(t *testing.T) { + requireDB(t) + r := newIdentityTestRouter() + memberID := uuid.NewString() + member := identityUserHeaders(memberID, "member") + foreignUserID := "00000000-0000-0000-0000-000000000001" + for _, method := range []string{http.MethodGet, http.MethodDelete} { + path := "/v3/users/" + memberID + "/application_credentials/not-a-uuid" + if code, body := doJSONReq(r, method, path, "", member); code != http.StatusNotFound || !strings.Contains(body, "Could not find Application Credential: not-a-uuid.") { + t.Errorf("self %s malformed credential id: code=%d body=%s; want Keystone 404", method, code, body) + } + foreignPath := "/v3/users/" + foreignUserID + "/application_credentials/not-a-uuid" + if code, body := doJSONReq(r, method, foreignPath, "", member); code != http.StatusForbidden { + t.Errorf("foreign %s malformed credential id: code=%d body=%s; want authz-first 403", method, code, body) + } + } +} + +func TestAssignRoleUnknownIDsReturnResourceNotFound(t *testing.T) { + requireDB(t) + const projectID = "00000000-0000-0000-0000-000000000002" + const userID = "00000000-0000-0000-0000-000000000001" + const roleID = "00000000-0000-0000-0000-000000000003" + unknownID := uuid.NewString() + admin := identityUserHeaders(uuid.NewString(), "admin") + r := newIdentityTestRouter() + for _, tc := range []struct{ kind, path string }{ + {"project", "/v3/projects/" + unknownID + "/users/" + userID + "/roles/" + roleID}, + {"user", "/v3/projects/" + projectID + "/users/" + unknownID + "/roles/" + roleID}, + {"role", "/v3/projects/" + projectID + "/users/" + userID + "/roles/" + unknownID}, + } { + t.Run(tc.kind, func(t *testing.T) { + code, body := doJSONReq(r, http.MethodPut, tc.path, "", admin) + if code != http.StatusNotFound || !strings.Contains(body, "Could not find "+tc.kind+": "+unknownID+".") { + t.Errorf("unknown %s: code=%d body=%s; want resource-specific 404", tc.kind, code, body) + } + }) + } +} diff --git a/internal/keystone/routes_wiring_test.go b/internal/keystone/routes_wiring_test.go index 5dd73746..c29882a3 100644 --- a/internal/keystone/routes_wiring_test.go +++ b/internal/keystone/routes_wiring_test.go @@ -24,15 +24,23 @@ func TestRegisterRoutesMiddlewareWiring(t *testing.T) { // cannot. Registering a management route on `v3` instead of `mgmt` is a one-token // slip that would leave the resource readable without a token. authCalls := 0 - stubAuth := func(c *gin.Context) { authCalls++; c.Next() } // authenticated, no roles + stubAuth := func(c *gin.Context) { // authenticated; admin only via header + authCalls++ + if c.GetHeader("X-Test-Admin") != "" { + c.Set("roles", []string{"admin"}) + } + c.Next() + } svc := NewService(testPool, nil, nil) svc.RegisterRoutes(&r.RouterGroup, stubAuth) // Non-gated route (on val): malformed id → middleware 404, handler never runs. + // (Credentials keep their :id on val: the id-only lookup 404s for foreign owners + // anyway, so the middleware 404 does not preempt an authz decision there.) w := httptest.NewRecorder() - r.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/v3/projects/not-a-uuid", nil)) - if w.Code != http.StatusNotFound || !strings.Contains(w.Body.String(), "Could not find project: not-a-uuid.") { - t.Fatalf("GET /projects/not-a-uuid: code=%d body=%s; want 404 verbatim", w.Code, w.Body.String()) + r.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/v3/credentials/not-a-uuid", nil)) + if w.Code != http.StatusNotFound || !strings.Contains(w.Body.String(), "Could not find credential: not-a-uuid.") { + t.Fatalf("GET /credentials/not-a-uuid: code=%d body=%s; want 404 verbatim", w.Code, w.Body.String()) } // Authz-gated route (on mgmt, NOT behind the uuid middleware): a non-admin @@ -44,39 +52,55 @@ func TestRegisterRoutesMiddlewareWiring(t *testing.T) { t.Fatalf("DELETE /users/not-a-uuid (non-admin): code=%d body=%s; want 403 (authz precedes)", w.Code, w.Body.String()) } - // GET /domains/:id also lives on mgmt, for a different reason: it must resolve - // the "default" alias, which the uuid middleware would reject before the handler - // ran (SPEC-022 FR-001a). The narrow exception is that one alias — every other - // non-identifier value still gets #515's verbatim 404, now produced in-handler. - // No database needed: this asserts the 404 shape, which precedes any query. + // GET /domains/:id lives on `mgmt` so it can resolve the `default` alias + // (SPEC-022 FR-001a), and it is admin-only like the rest of the domain + // management surface. A non-admin gets 403 before the + // resolver runs; an admin still gets the #515 verbatim 404 for any other + // non-identifier value, now produced in-handler. w = httptest.NewRecorder() authBefore := authCalls r.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/v3/domains/not-a-uuid", nil)) + if w.Code != http.StatusForbidden { + t.Fatalf("GET /domains/not-a-uuid (non-admin): code=%d body=%s; want 403 (admin-only)", w.Code, w.Body.String()) + } + w = httptest.NewRecorder() + adminReq := httptest.NewRequest(http.MethodGet, "/v3/domains/not-a-uuid", nil) + adminReq.Header.Set("X-Test-Admin", "1") + r.ServeHTTP(w, adminReq) if w.Code != http.StatusNotFound || !strings.Contains(w.Body.String(), "Could not find domain: not-a-uuid.") { - t.Fatalf("GET /domains/not-a-uuid: code=%d body=%s; want 404 verbatim from the handler", w.Code, w.Body.String()) + t.Fatalf("GET /domains/not-a-uuid (admin): code=%d body=%s; want 404 verbatim from the handler", w.Code, w.Body.String()) } // Moving the route off `val` must not have moved it off the authenticated tree as // well: registered on the public `v3` group by mistake, it would answer the same - // 404 above while leaving every domain readable without a token. The status alone - // cannot see that, so assert the middleware ran. This piggybacks on the malformed - // id deliberately — resolveDomainRef answers it without a query, so the assertion - // needs no database (TestResolveDomainRefMalformedNeedsNoDatabase pins that). + // 403/404 above while leaving every domain readable without a token. The status + // alone cannot see that, so assert the middleware ran. This piggybacks on the + // malformed id deliberately — resolveDomainRef answers it without a query, so the + // assertion needs no database (TestResolveDomainRefMalformedNeedsNoDatabase pins that). if authCalls == authBefore { t.Fatal("GET /v3/domains/:id bypassed the auth middleware — it must be on mgmt, not the public v3 group") } - // The sibling domain routes stay on val, so the middleware still answers for - // them — the exception is one route and one method, not the whole resource. + // The domain write/config routes moved to `mgmt` with the admin gate, so + // their malformed-id 404 is likewise produced in-handler, after the authz + // decision — same shape, different producer. for _, tc := range []struct{ method, path string }{ {http.MethodPatch, "/v3/domains/not-a-uuid"}, {http.MethodDelete, "/v3/domains/not-a-uuid"}, {http.MethodGet, "/v3/domains/not-a-uuid/config"}, } { w = httptest.NewRecorder() - r.ServeHTTP(w, httptest.NewRequest(tc.method, tc.path, nil)) + req := httptest.NewRequest(tc.method, tc.path, nil) + r.ServeHTTP(w, req) + if w.Code != http.StatusForbidden { + t.Fatalf("%s %s (non-admin): code=%d body=%s; want 403 (admin-only)", tc.method, tc.path, w.Code, w.Body.String()) + } + w = httptest.NewRecorder() + req = httptest.NewRequest(tc.method, tc.path, nil) + req.Header.Set("X-Test-Admin", "1") + r.ServeHTTP(w, req) if w.Code != http.StatusNotFound || !strings.Contains(w.Body.String(), "Could not find domain: not-a-uuid.") { - t.Fatalf("%s %s: code=%d body=%s; want the middleware's 404", tc.method, tc.path, w.Code, w.Body.String()) + t.Fatalf("%s %s (admin): code=%d body=%s; want the in-handler 404", tc.method, tc.path, w.Code, w.Body.String()) } } } diff --git a/internal/keystone/s3tokens.go b/internal/keystone/s3tokens.go index 8c8061f1..84139636 100644 --- a/internal/keystone/s3tokens.go +++ b/internal/keystone/s3tokens.go @@ -124,7 +124,10 @@ func (svc *Service) validateSignedRequest(c *gin.Context) { } pid := b.TenantID - if pid == "" && projectID != nil { + // The generic credential API allows owners to edit the blob. Its embedded + // tenant must not override the credential's immutable, recorded project. + // Retain the blob fallback only for legacy rows without project_id. + if projectID != nil { pid = *projectID } svc.renderS3Token(c, userID, pid) @@ -225,21 +228,27 @@ func (svc *Service) renderS3Token(c *gin.Context, userID, projectID string) { SELECT DISTINCT r.name FROM roles r JOIN role_assignments ra ON ra.role_id = r.id WHERE ra.user_id = $1 AND ra.project_id = $2`, userID, projectID) - if err == nil { - defer rows.Close() - for rows.Next() { - var name string - if err := rows.Scan(&name); err != nil { - log.Warn().Err(err).Str("operation", "renderS3Token").Msg("failed to scan role row") - } else { - roles = append(roles, gin.H{"name": name}) - } - } - // Log rather than fail the token: RGW uses the role list for tenant mapping, and - // refusing the token outright would deny S3 access entirely. - if err := rows.Err(); err != nil { - log.Error().Err(err).Str("user_id", userID).Msg("s3token role rows iteration error; role list may be truncated") + if err != nil { + log.Warn().Err(err).Msg("s3tokens: role lookup failed") + common.SendError(c, common.NewUnauthorizedError(authFailed)) + return + } + defer rows.Close() + for rows.Next() { + var name string + if err := rows.Scan(&name); err != nil { + log.Warn().Err(err).Msg("s3tokens: role scan failed") + common.SendError(c, common.NewUnauthorizedError(authFailed)) + return } + roles = append(roles, gin.H{"name": name}) + } + // Credentials survive role removal. A valid signature alone no longer + // authorizes a project after that membership has been revoked. + if err := rows.Err(); err != nil || len(roles) == 0 { + log.Warn().Err(err).Str("user_id", userID).Msg("s3tokens: no verified project role") + common.SendError(c, common.NewUnauthorizedError(authFailed)) + return } // RGW parses the response as a full Keystone token and requires expires_at diff --git a/internal/keystone/s3tokens_disabled_test.go b/internal/keystone/s3tokens_disabled_test.go index d4f05b15..76db581e 100644 --- a/internal/keystone/s3tokens_disabled_test.go +++ b/internal/keystone/s3tokens_disabled_test.go @@ -169,6 +169,50 @@ func TestS3TokensDisabledProjectDenied(t *testing.T) { assertMaskedS3Denial(t, postS3Tokens(t, signedS3TokensBody(access, secret))) } +func TestS3TokensRevokedProjectRoleDenied(t *testing.T) { + access, secret := seedS3DisabledIdentity(t) + if _, err := testPool.Exec(t.Context(), "DELETE FROM role_assignments WHERE user_id=$1 AND project_id=$2", s3DisabledUserID, s3DisabledProjectID); err != nil { + t.Fatal(err) + } + assertMaskedS3Denial(t, postS3Tokens(t, signedS3TokensBody(access, secret))) +} + +func TestCredentialBlobCannotChangeS3Project(t *testing.T) { + access, secret := seedS3DisabledIdentity(t) + var credentialID string + if err := testPool.QueryRow(t.Context(), "SELECT id FROM credentials WHERE access_key=$1", access).Scan(&credentialID); err != nil { + t.Fatal(err) + } + blob, _ := json.Marshal(ec2Blob{Access: access, Secret: secret, UserID: s3DisabledUserID, TenantID: rgwCompatProjectID}) + body, _ := json.Marshal(map[string]any{"credential": map[string]any{"blob": string(blob)}}) + req := httptest.NewRequest(http.MethodPatch, "/v3/credentials/"+credentialID, strings.NewReader(string(body))) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("X-Test-User", s3DisabledUserID) + req.Header.Set("X-Test-Roles", "member") + w := httptest.NewRecorder() + newIdentityTestRouter().ServeHTTP(w, req) + if w.Code != http.StatusOK { + t.Fatalf("update own blob: %d %s", w.Code, w.Body.String()) + } + w = postS3Tokens(t, signedS3TokensBody(access, secret)) + if w.Code != http.StatusOK { + t.Fatalf("authenticate original credential: %d %s", w.Code, w.Body.String()) + } + var result struct { + Token struct { + Project struct { + ID string `json:"id"` + } `json:"project"` + } `json:"token"` + } + if err := json.Unmarshal(w.Body.Bytes(), &result); err != nil { + t.Fatal(err) + } + if result.Token.Project.ID != canonicalID(s3DisabledProjectID) { + t.Errorf("mutable blob changed scoped project to %q", result.Token.Project.ID) + } +} + // TestRenderS3TokenMissingIdentityDenied: a stale EC2 blob tenant (project // row gone while the credential survives) or a user row that vanished under // the credential must deny — not render an envelope with empty names. diff --git a/internal/keystone/services.go b/internal/keystone/services.go index 4c80981b..d9b7bb22 100644 --- a/internal/keystone/services.go +++ b/internal/keystone/services.go @@ -15,6 +15,13 @@ import ( // ListServices returns all services in the catalog func (svc *Service) ListServices(c *gin.Context) { + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + var f listFilter f.eq(c, "type", "type") f.eq(c, "name", "name") @@ -64,6 +71,13 @@ func (svc *Service) ListServices(c *gin.Context) { // CreateService creates a new service in the catalog func (svc *Service) CreateService(c *gin.Context) { + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + var req struct { Service struct { Type string `json:"type" binding:"required"` @@ -115,6 +129,17 @@ func (svc *Service) CreateService(c *gin.Context) { func (svc *Service) GetService(c *gin.Context) { serviceID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "service", serviceID) { + return + } + var id, svcType, name string var description *string var enabled bool @@ -153,6 +178,17 @@ func (svc *Service) GetService(c *gin.Context) { func (svc *Service) DeleteService(c *gin.Context) { serviceID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "service", serviceID) { + return + } + result, err := svc.pool.Exec(c.Request.Context(), "DELETE FROM services WHERE id = $1", serviceID) @@ -175,6 +211,17 @@ func (svc *Service) DeleteService(c *gin.Context) { func (svc *Service) UpdateService(c *gin.Context) { serviceID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "service", serviceID) { + return + } + // Parse request body var req struct { Service struct { @@ -257,6 +304,13 @@ func (svc *Service) UpdateService(c *gin.Context) { // ListEndpoints returns all endpoints in the catalog func (svc *Service) ListEndpoints(c *gin.Context) { + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + var f listFilter f.eq(c, "interface", "e.interface") f.eqID(c, "service_id", "e.service_id") @@ -309,6 +363,13 @@ func (svc *Service) ListEndpoints(c *gin.Context) { // CreateEndpoint creates a new endpoint in the catalog func (svc *Service) CreateEndpoint(c *gin.Context) { + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + var req struct { Endpoint struct { ServiceID string `json:"service_id" binding:"required"` @@ -361,6 +422,17 @@ func (svc *Service) CreateEndpoint(c *gin.Context) { func (svc *Service) DeleteEndpoint(c *gin.Context) { endpointID := c.Param("id") + // Admin-only: the identity management surface + // must not honor a plain member — this is the role/assignment/catalog + // authority. + if !common.RequireAdmin(c, "keystone:identity-management") { + return + } + // Authorization passed; validate the ids after it so 403 precedes the 404 (#515). + if denyIfMalformedID(c, "endpoint", endpointID) { + return + } + result, err := svc.pool.Exec(c.Request.Context(), "DELETE FROM endpoints WHERE id = $1", endpointID) diff --git a/internal/keystone/uuid_middleware.go b/internal/keystone/uuid_middleware.go index 114ced3e..14091b03 100644 --- a/internal/keystone/uuid_middleware.go +++ b/internal/keystone/uuid_middleware.go @@ -16,10 +16,10 @@ import ( // faithful behavior, and validating up front avoids the `22P02` cast that would // otherwise surface as 500 (cloud-storage#515). // -// A well-formed but non-canonical id — dash-free (the form rgw_compat_tenant_ids -// advertises) or upper-case — is rewritten to canonical form in place, so the handler's -// id comparison matches on SQLite, where ids are text, as it did on PostgreSQL's `uuid` -// columns. uuid.Parse accepts exactly the forms uuid.Validate did. +// Well-formed UUID params have already been canonicalized by the parent management +// group's canonicalizeUUIDParams middleware, so this middleware only rejects malformed +// values. That ordering keeps the id comparison compatible with SQLite's text columns +// while preserving the same 404 behavior as PostgreSQL's `uuid` columns. // // It validates only params named "id" or "_id"; the EC2 "access" key // and any other param pass through untouched (FR-005). It is registered only on @@ -31,17 +31,31 @@ func uuidParamMiddleware(c *gin.Context) { if !isUUIDParam(p.Key) { continue } - id, err := uuid.Parse(p.Value) - if err != nil { + if uuid.Validate(p.Value) != nil { common.AbortWithError(c, common.NewIdentityNotFoundError(resourceNoun(c.FullPath(), p.Key), p.Value)) return } - c.Params[i].Value = id.String() } // No c.Next() needed: when a middleware returns without aborting, gin's engine // advances to the next handler on its own. } +// canonicalizeUUIDParams normalizes only well-formed UUID path params. It is +// installed after authentication on the management group, before uuidParamMiddleware +// on its validation-only subgroup, so authorization-gated handlers can still reject +// malformed IDs after their authorization check. +func canonicalizeUUIDParams(c *gin.Context) { + for i := range c.Params { + p := &c.Params[i] + if !isUUIDParam(p.Key) { + continue + } + if id, err := uuid.Parse(p.Value); err == nil { + p.Value = id.String() + } + } +} + // isUUIDParam reports whether a gin path-param name addresses a UUID id. func isUUIDParam(name string) bool { return name == "id" || strings.HasSuffix(name, "_id") diff --git a/internal/keystone/uuid_middleware_test.go b/internal/keystone/uuid_middleware_test.go index e0f4cbbf..22cc467f 100644 --- a/internal/keystone/uuid_middleware_test.go +++ b/internal/keystone/uuid_middleware_test.go @@ -216,13 +216,13 @@ func TestUUIDParamMiddlewareAbortsBeforeHandler(t *testing.T) { } // A well-formed but non-canonical id (dash-free, as rgw_compat_tenant_ids advertises, or -// upper-case) is rewritten to canonical form, so the handler's id comparison matches on -// SQLite, where ids are text. +// upper-case) is canonicalized by the management middleware before validation, so the +// handler's id comparison matches on SQLite, where ids are text. func TestUUIDParamMiddlewareCanonicalisesID(t *testing.T) { gin.SetMode(gin.TestMode) r := gin.New() g := r.Group("/v3") - g.Use(uuidParamMiddleware) + g.Use(canonicalizeUUIDParams, uuidParamMiddleware) var seen string g.GET("/projects/:id", func(c *gin.Context) { @@ -242,3 +242,21 @@ func TestUUIDParamMiddlewareCanonicalisesID(t *testing.T) { } } } + +func TestCanonicalizeUUIDParamsLeavesMalformedIDsForHandlers(t *testing.T) { + gin.SetMode(gin.TestMode) + c, _ := gin.CreateTestContext(httptest.NewRecorder()) + c.Params = gin.Params{ + {Key: "id", Value: "11111111111111111111111111111111"}, + {Key: "cred_id", Value: "NOT-A-UUID"}, + {Key: "access", Value: "legacy-access-key"}, + } + + canonicalizeUUIDParams(c) + want := []string{"11111111-1111-1111-1111-111111111111", "NOT-A-UUID", "legacy-access-key"} + for i, expected := range want { + if got := c.Params[i].Value; got != expected { + t.Errorf("param %s = %q, want %q", c.Params[i].Key, got, expected) + } + } +} diff --git a/internal/metadata/network_data_test.go b/internal/metadata/network_data_test.go new file mode 100644 index 00000000..ecaf4901 --- /dev/null +++ b/internal/metadata/network_data_test.go @@ -0,0 +1,211 @@ +package metadata + +import ( + "encoding/json" + "net/http" + "testing" + + "github.com/google/uuid" +) + +// GetNetworkDataJSON selected `p.fixed_ip`, a column +// that does not exist — ports store `fixed_ips` as a JSONB array — so the +// query failed and every guest fetching network_data.json got a 500, leaving +// cloud-init to fall back to DHCP or configure nothing. This pins the fixed +// shape: one link per port, one network entry per address, resolved through +// the address's own subnet (cidr, netmask, gateway, DNS). +func TestNetworkDataJSONServesPortAddresses(t *testing.T) { + requireDB(t) + ctx := t.Context() + + instanceID := seedInstance(t, nil) + + var networkID string + if err := testPool.QueryRow(ctx, + `INSERT INTO networks (name, project_id, status, admin_state_up) + VALUES ('md-net', $1, 'ACTIVE', true) RETURNING id::text`, + seededProjectID).Scan(&networkID); err != nil { + t.Fatalf("seed network: %v", err) + } + t.Cleanup(func() { testPool.Exec(ctx, "DELETE FROM networks WHERE id::text = $1", networkID) }) + + var subnetID string + if err := testPool.QueryRow(ctx, + `INSERT INTO subnets (name, network_id, project_id, cidr, gateway_ip, dns_nameservers) + VALUES ('md-sub', $1, $2, '192.168.42.0/24', '192.168.42.1', $3) RETURNING id::text`, + networkID, seededProjectID, []string{"1.1.1.1"}).Scan(&subnetID); err != nil { + t.Fatalf("seed subnet: %v", err) + } + + portID := uuid.NewString() + var subnetV6 string + if err := testPool.QueryRow(ctx, + `INSERT INTO subnets (name, network_id, project_id, cidr, gateway_ip, dns_nameservers) + VALUES ('md-sub-v6', $1, $2, '2001:db8::/64', '2001:db8::1', $3) RETURNING id::text`, + networkID, seededProjectID, []string{"2001:4860:4860::8888"}).Scan(&subnetV6); err != nil { + t.Fatal(err) + } + fixed := `[{"subnet_id":"` + subnetID + `","ip_address":"192.168.42.77"},` + + `{"subnet_id":"` + subnetID + `","ip_address":"192.168.42.78"},` + + `{"subnet_id":"` + subnetV6 + `","ip_address":"2001:db8::77"}]` + if _, err := testPool.Exec(ctx, + `INSERT INTO ports (id, name, network_id, project_id, device_id, device_owner, mac_address, status, fixed_ips) + VALUES ($1, 'md-port', $2, $3, $4, 'compute:nova', '52:54:00:0a:0b:0c', 'ACTIVE', $5::jsonb)`, + portID, networkID, seededProjectID, instanceID, fixed); err != nil { + t.Fatalf("seed port: %v", err) + } + t.Cleanup(func() { testPool.Exec(ctx, "DELETE FROM ports WHERE id::text = $1", portID) }) + + r := newMetadataTestRouter() + code, body := metaGet(r, "/openstack/latest/network_data.json", instanceID) + if code != http.StatusOK { + t.Fatalf("network_data.json: code=%d body=%s; want 200 (the N-7 defect answered 500)", code, body) + } + + var doc struct { + Links []struct { + ID string `json:"id"` + EthernetMACAddress string `json:"ethernet_mac_address"` + } `json:"links"` + Networks []struct { + ID string `json:"id"` + Type string `json:"type"` + NetworkID string `json:"network_id"` + Link string `json:"link"` + IPAddress string `json:"ip_address"` + Netmask string `json:"netmask"` + Gateway string `json:"gateway"` + DNSServers []string `json:"dns_nameservers"` + Routes []struct { + Network string `json:"network"` + Netmask string `json:"netmask"` + Gateway string `json:"gateway"` + } `json:"routes"` + Services []struct { + Type string `json:"type"` + Address string `json:"address"` + } `json:"services"` + } `json:"networks"` + Services []struct { + Type string `json:"type"` + Address string `json:"address"` + } `json:"services"` + } + if err := json.Unmarshal([]byte(body), &doc); err != nil { + t.Fatalf("network_data.json is not JSON: %v (%s)", err, body) + } + + // One dual-stack port with three addresses: one interface, three networks. + if len(doc.Links) != 1 { + t.Errorf("links = %d; want 1 (one link per port, not per address): %s", len(doc.Links), body) + } + if len(doc.Links) == 1 && doc.Links[0].EthernetMACAddress != "52:54:00:0a:0b:0c" { + t.Errorf("link mac = %q; want the port's", doc.Links[0].EthernetMACAddress) + } + if len(doc.Networks) != 3 { + t.Fatalf("networks = %d; want 3 (one per address): %s", len(doc.Networks), body) + } + ips := map[string]bool{} + networkIDs := map[string]bool{} + for _, n := range doc.Networks { + if networkIDs[n.ID] { + t.Errorf("duplicate network id %q", n.ID) + } + networkIDs[n.ID] = true + if len(doc.Links) == 1 && n.Link != doc.Links[0].ID { + t.Errorf("network link %q does not reference its port", n.Link) + } + ips[n.IPAddress] = true + zero := "0.0.0.0" + if n.Type == "ipv6" { + zero = "::" + } + if len(n.Routes) != 1 || n.Routes[0].Network != zero || n.Routes[0].Netmask != zero || n.Routes[0].Gateway != n.Gateway { + t.Errorf("invalid default route for %s: %+v", n.IPAddress, n.Routes) + } + if len(n.Services) != 1 || n.Services[0].Type != "dns" || len(n.DNSServers) != 1 || n.Services[0].Address != n.DNSServers[0] { + t.Errorf("invalid DNS services for %s: %+v", n.IPAddress, n.Services) + } + if n.NetworkID != networkID { + t.Errorf("network_id=%q; want %q", n.NetworkID, networkID) + } + if n.IPAddress == "2001:db8::77" { + if n.Type != "ipv6" || n.Netmask != "ffff:ffff:ffff:ffff::" || n.Gateway != "2001:db8::1" { + t.Errorf("invalid IPv6 metadata: %+v", n) + } + if len(n.DNSServers) != 1 || n.DNSServers[0] != "2001:4860:4860::8888" { + t.Errorf("invalid IPv6 DNS: %v", n.DNSServers) + } + continue + } + if n.Type != "ipv4" { + t.Errorf("IPv4 type=%q", n.Type) + } + if n.Netmask != "255.255.255.0" { + t.Errorf("network %s: netmask = %q; want 255.255.255.0", n.IPAddress, n.Netmask) + } + if n.Gateway != "192.168.42.1" { + t.Errorf("network %s: gateway = %q; want the subnet's", n.IPAddress, n.Gateway) + } + if len(n.DNSServers) != 1 || n.DNSServers[0] != "1.1.1.1" { + t.Errorf("network %s: dns = %v; want [1.1.1.1]", n.IPAddress, n.DNSServers) + } + } + if !ips["192.168.42.77"] || !ips["192.168.42.78"] { + t.Errorf("networks missing the port's addresses: %s", body) + } + if len(doc.Services) != 2 { + t.Errorf("services must contain both DNS servers once: %+v", doc.Services) + } +} + +// TestNetworkDataJSONToleratesNullFixedIPs pins the metadata half of the +// JSON-scalar-null defect. A port on a subnet-less network persisted JSON +// `null` in fixed_ips; COALESCE does not rescue that (JSON null is not SQL +// NULL), so jsonb_array_elements raised and network_data.json answered 500. +func TestNetworkDataJSONToleratesNullFixedIPs(t *testing.T) { + requireDB(t) + ctx := t.Context() + + instanceID := seedInstance(t, nil) + + var networkID string + if err := testPool.QueryRow(ctx, + `INSERT INTO networks (name, project_id, status, admin_state_up) + VALUES ('md-null-net', $1, 'ACTIVE', true) RETURNING id::text`, + seededProjectID).Scan(&networkID); err != nil { + t.Fatalf("seed network: %v", err) + } + t.Cleanup(func() { testPool.Exec(ctx, "DELETE FROM networks WHERE id::text = $1", networkID) }) + + portID := uuid.NewString() + if _, err := testPool.Exec(ctx, + `INSERT INTO ports (id, name, network_id, project_id, device_id, device_owner, mac_address, status, fixed_ips) + VALUES ($1, 'md-null-port', $2, $3, $4, 'compute:nova', '52:54:00:0a:0b:0d', 'ACTIVE', 'null'::jsonb)`, + portID, networkID, seededProjectID, instanceID); err != nil { + t.Fatalf("seed port: %v", err) + } + t.Cleanup(func() { testPool.Exec(ctx, "DELETE FROM ports WHERE id::text = $1", portID) }) + + code, body := metaGet(newMetadataTestRouter(), "/openstack/latest/network_data.json", instanceID) + if code != http.StatusOK { + t.Fatalf("network_data.json with a JSON-null fixed_ips: code=%d body=%s; want 200", code, body) + } + + var doc struct { + Links []json.RawMessage `json:"links"` + Networks []json.RawMessage `json:"networks"` + } + if err := json.Unmarshal([]byte(body), &doc); err != nil { + t.Fatalf("network_data.json is not JSON: %v (%s)", err, body) + } + if len(doc.Links) != 1 { + t.Errorf("links = %d; want the address-less port's one link: %s", len(doc.Links), body) + } + if len(doc.Networks) != 0 { + t.Errorf("networks = %d; an address-less port contributes none: %s", len(doc.Networks), body) + } + if doc.Networks == nil { + t.Errorf("networks must be an empty array, not null: %s", body) + } +} diff --git a/internal/metadata/service.go b/internal/metadata/service.go index cba78fe7..83cc42fe 100644 --- a/internal/metadata/service.go +++ b/internal/metadata/service.go @@ -352,78 +352,156 @@ func (svc *Service) GetNetworkDataJSON(c *gin.Context) { return } - // Query all ports for this instance - rows, err := svc.pool.Query(c.Request.Context(), ` - SELECT p.id, p.mac_address, p.fixed_ip, p.network_id, n.name, s.cidr, s.gateway_ip, s.dns_nameservers + // Read each port once, then decode fixed_ips and resolve the referenced subnets. + // Doing this in Go keeps JSON array traversal and text-array decoding portable + // between PostgreSQL and SQLite, while preserving one link per port and one + // network entry per fixed address. + portRows, err := svc.pool.Query(c.Request.Context(), ` + SELECT p.id::text, p.mac_address, p.fixed_ips, p.network_id::text FROM ports p JOIN networks n ON p.network_id = n.id - LEFT JOIN subnets s ON p.network_id = s.network_id - WHERE p.device_id = $1 - ORDER BY p.created_at + WHERE p.device_id::text = $1 + ORDER BY p.created_at, p.id `, instanceID) - if err != nil { log.Error().Err(err).Str("operation", "get_network_data_json").Str("instance_id", instanceID).Msg("failed to query instance ports") common.SendError(c, common.NewInternalServerError("failed to get network data")) return } - defer rows.Close() - - var links []gin.H - var networks []gin.H - - linkID := 0 - for rows.Next() { - var portID, mac, ip, networkID, networkName, cidr, gateway string - var dnsServers []string + type networkDataPort struct { + id, mac, networkID string + fixedIPs common.FixedIPs + } + ports := []networkDataPort{} + networkIDs := []string{} + seenNetworkIDs := map[string]bool{} + for portRows.Next() { + var port networkDataPort + if err := portRows.Scan(&port.id, &port.mac, &port.fixedIPs, &port.networkID); err != nil { + log.Error().Err(err).Str("instance_id", instanceID).Msg("failed to scan network data port") + portRows.Close() + common.SendError(c, common.NewInternalServerError("failed to get network data")) + return + } + ports = append(ports, port) + if !seenNetworkIDs[port.networkID] { + seenNetworkIDs[port.networkID] = true + networkIDs = append(networkIDs, port.networkID) + } + } + if err := portRows.Err(); err != nil { + portRows.Close() + log.Error().Err(err).Msg("network data port rows iteration error") + common.SendError(c, common.NewInternalServerError("failed to get network data")) + return + } + portRows.Close() - if err := rows.Scan(&portID, &mac, &ip, &networkID, &networkName, &cidr, &gateway, &dnsServers); err != nil { - log.Error().Err(err).Str("instance_id", instanceID).Msg("failed to scan network data row") + type networkDataSubnet struct { + cidr, gateway string + dnsServers []string + } + subnets := map[string]networkDataSubnet{} + if len(networkIDs) > 0 { + subnetRows, err := svc.pool.Query(c.Request.Context(), ` + SELECT id::text, cidr, COALESCE(gateway_ip, ''), dns_nameservers + FROM subnets WHERE network_id::text = ANY($1::text[]) + ORDER BY network_id, id + `, networkIDs) + if err != nil { + log.Error().Err(err).Msg("failed to query network data subnets") common.SendError(c, common.NewInternalServerError("failed to get network data")) return } + for subnetRows.Next() { + var id, cidr, gateway string + var dnsServers []string + if err := subnetRows.Scan(&id, &cidr, &gateway, &dnsServers); err != nil { + subnetRows.Close() + log.Error().Err(err).Msg("failed to scan network data subnet") + common.SendError(c, common.NewInternalServerError("failed to get network data")) + return + } + if dnsServers == nil { + dnsServers = []string{} + } + subnets[id] = networkDataSubnet{cidr: cidr, gateway: gateway, dnsServers: dnsServers} + } + if err := subnetRows.Err(); err != nil { + subnetRows.Close() + log.Error().Err(err).Msg("network data subnet rows iteration error") + common.SendError(c, common.NewInternalServerError("failed to get network data")) + return + } + subnetRows.Close() + } - // Create link (interface) + links := []gin.H{} + networks := []gin.H{} + services := []gin.H{} + seenDNS := map[string]bool{} + for linkID, port := range ports { + linkName := fmt.Sprintf("tap%d", linkID) links = append(links, gin.H{ - "id": fmt.Sprintf("tap%d", linkID), + "id": linkName, "type": "bridge", - "ethernet_mac_address": mac, + "ethernet_mac_address": port.mac, "mtu": 1500, }) - // Create network config - netConfig := gin.H{ - "id": fmt.Sprintf("network%d", linkID), - "link": fmt.Sprintf("tap%d", linkID), - "type": "ipv4", - "ip_address": ip, - "netmask": cidrToNetmask(cidr), // Convert /24 → "255.255.255.0" - } - - if gateway != "" { - netConfig["gateway"] = gateway - } + for _, fixedIP := range port.fixedIPs { + ip := fixedIP.IPAddress + if ip == "" { + continue + } + subnet := subnets[fixedIP.SubnetID] - if len(dnsServers) > 0 { - netConfig["dns_nameservers"] = dnsServers + // Match Nova metadata for each address family, with a distinct network ID + // even when several addresses share one link. + addressType := "ipv4" + if addr := net.ParseIP(ip); addr != nil && addr.To4() == nil { + addressType = "ipv6" + } + netConfig := gin.H{ + "id": fmt.Sprintf("network%d", len(networks)), + "link": linkName, + "type": addressType, + "network_id": port.networkID, + "ip_address": ip, + "netmask": cidrToNetmask(subnet.cidr), + "routes": []gin.H{}, + "services": []gin.H{}, + } + if subnet.gateway != "" { + netConfig["gateway"] = subnet.gateway + zero := "0.0.0.0" + if addressType == "ipv6" { + zero = "::" + } + netConfig["routes"] = []gin.H{{"network": zero, "netmask": zero, "gateway": subnet.gateway}} + } + if len(subnet.dnsServers) > 0 { + netConfig["dns_nameservers"] = subnet.dnsServers + dnsServices := make([]gin.H, 0, len(subnet.dnsServers)) + for _, dns := range subnet.dnsServers { + entry := gin.H{"type": "dns", "address": dns} + dnsServices = append(dnsServices, entry) + if !seenDNS[dns] { + services = append(services, entry) + seenDNS[dns] = true + } + } + netConfig["services"] = dnsServices + } + networks = append(networks, netConfig) } - - networks = append(networks, netConfig) - linkID++ - } - // Fail rather than serve a partial network_data.json: a missing interface would - // leave the guest misconfigured, and cloud-init retries on error. - if err := rows.Err(); err != nil { - log.Error().Err(err).Msg("network data rows iteration error") - common.SendError(c, common.NewInternalServerError("failed to get network data")) - return } // Return network_data.json structure c.JSON(http.StatusOK, gin.H{ "links": links, "networks": networks, - "services": []gin.H{}, + "services": services, }) } @@ -759,6 +837,5 @@ func cidrToNetmask(cidr string) string { } return cidr } - mask := ipNet.Mask - return fmt.Sprintf("%d.%d.%d.%d", mask[0], mask[1], mask[2], mask[3]) + return net.IP(ipNet.Mask).String() } diff --git a/internal/neutron/allocate_free_ip_test.go b/internal/neutron/allocate_free_ip_test.go new file mode 100644 index 00000000..2744079f --- /dev/null +++ b/internal/neutron/allocate_free_ip_test.go @@ -0,0 +1,301 @@ +package neutron + +import ( + "context" + "encoding/json" + "fmt" + "net" + "testing" + + "github.com/google/uuid" +) + +func TestAllocateFreeIPFindsLastAddress(t *testing.T) { + requireDB(t) + networkID, subnetID := seedN4Subnet(t, "192.168.249.0/24", "192.168.249.1") + for i := 2; i < 254; i++ { + seedN4Port(t, networkID, subnetID, fmt.Sprintf("192.168.249.%d", i)) + } + for range 10 { + ip, err := AllocateFreeIP(t.Context(), testPool, subnetID, "192.168.249.0/24") + if err != nil || ip != "192.168.249.254" { + t.Fatalf("last address: ip=%q err=%v; want 192.168.249.254", ip, err) + } + } +} + +func TestReleaseInstancePortsPreservesTenantOwnedPorts(t *testing.T) { + requireDB(t) + networkID, _ := seedN4Subnet(t, "192.168.252.0/24", "192.168.252.1") + instanceID := uuid.NewString() + novaPortID, tenantPortID := uuid.NewString(), uuid.NewString() + for _, port := range []struct{ id, owner string }{ + {novaPortID, "compute:nova"}, + {tenantPortID, ""}, + } { + if _, err := testPool.Exec(t.Context(), ` + INSERT INTO ports (id, name, network_id, project_id, device_id, device_owner, mac_address, fixed_ips) + VALUES ($1, $2, $3, $4, $5, NULLIF($6, ''), $7, '[]') + `, port.id, "release-"+port.id[:8], networkID, seededProjectID, instanceID, port.owner, "52:54:00:aa:bb:cc"); err != nil { + t.Fatalf("seed port %s: %v", port.id, err) + } + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), "DELETE FROM ports WHERE id IN ($1, $2)", novaPortID, tenantPortID) + }) + + svc := NewService(testPool, "stub", nil) + if err := svc.ReleaseInstancePorts(t.Context(), instanceID); err != nil { + t.Fatalf("ReleaseInstancePorts: %v", err) + } + for _, tc := range []struct { + id string + want int + }{{novaPortID, 0}, {tenantPortID, 1}} { + var count int + if err := testPool.QueryRow(t.Context(), "SELECT count(*) FROM ports WHERE id = $1", tc.id).Scan(&count); err != nil { + t.Fatal(err) + } + if count != tc.want { + t.Errorf("port %s remains %d times; want %d", tc.id, count, tc.want) + } + } + if err := svc.ReleaseInstancePorts(t.Context(), instanceID); err != nil { + t.Errorf("second ReleaseInstancePorts call: %v", err) + } +} + +func TestIPAvailableCanonicalIPv6(t *testing.T) { + requireDB(t) + networkID, subnetID := seedN4Subnet(t, "2001:db8::/64", "2001:0db8:0:0:0:0:0:1") + seedN4Port(t, networkID, subnetID, "2001:0db8:0:0:0:0:0:2") + for _, ip := range []string{"2001:db8::1", "2001:db8::2", "2001:DB8::2"} { + available, err := IPAvailable(t.Context(), testPool, subnetID, ip) + if err != nil || available { + t.Errorf("IPAvailable(%s)=%v, %v; want occupied", ip, available, err) + } + } +} + +// seedN4Subnet inserts a network + subnet and returns their ids. +func seedN4Subnet(t *testing.T, cidr, gateway string) (networkID, subnetID string) { + t.Helper() + ctx := context.Background() + if err := testPool.QueryRow(ctx, + `INSERT INTO networks (name, project_id, status, admin_state_up) + VALUES ($1, $2, 'ACTIVE', true) RETURNING id::text`, + "n4-net-"+uuid.NewString()[:8], seededProjectID).Scan(&networkID); err != nil { + t.Fatalf("seeding network: %v", err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM networks WHERE id::text = $1`, networkID) + }) + if err := testPool.QueryRow(ctx, + `INSERT INTO subnets (name, network_id, project_id, cidr, gateway_ip) + VALUES ('n4-sub', $1, $2, $3, NULLIF($4, '')) RETURNING id::text`, + networkID, seededProjectID, cidr, gateway).Scan(&subnetID); err != nil { + t.Fatalf("seeding subnet: %v", err) + } + return networkID, subnetID +} + +// seedN4Port inserts a port holding one fixed_ip in the subnet. +func seedN4Port(t *testing.T, networkID, subnetID, ip string) { + t.Helper() + fixed, _ := json.Marshal([]map[string]any{ + {"subnet_id": subnetID, "ip_address": ip}, + }) + if _, err := testPool.Exec(context.Background(), + `INSERT INTO ports (name, network_id, project_id, mac_address, status, fixed_ips) + VALUES ('n4-port', $1, $2, $3, 'ACTIVE', $4)`, + networkID, seededProjectID, "52:54:00:aa:bb:cc", fixed); err != nil { + t.Fatalf("seeding port: %v", err) + } +} + +// TestAllocateFreeIPAvoidsUsedAddresses pins that allocation +// used to be a purely random draw, so two ports in one subnet landed on the +// same address at birthday-problem rates — two MACs claiming one IP. On a /28 +// with 12 of 14 candidates already held, the old code duplicated an address +// ~86% of the time; the fixed allocator must never return a held one. +func TestAllocateFreeIPAvoidsUsedAddresses(t *testing.T) { + requireDB(t) + _, subnetID := seedN4Subnet(t, "192.168.250.0/28", "") + + held := map[string]bool{"192.168.250.1": true, "192.168.250.14": true} + // Hold addresses across the range the allocator may now draw from — it is + // upstream-conformant and no longer biased to offsets 10..13. + for _, ip := range []string{"192.168.250.1", "192.168.250.10", "192.168.250.11", "192.168.250.12", "192.168.250.14"} { + held[ip] = true + seedN4Port(t, mustN4NetworkID(t, subnetID), subnetID, ip) + } + + for range 20 { + ip, err := AllocateFreeIP(context.Background(), testPool, subnetID, "192.168.250.0/28") + if err != nil { + t.Fatalf("allocateFreeIP: %v", err) + } + if held[ip] { + t.Fatalf("allocated %s, which another port already holds — the N-4 collision", ip) + } + } +} + +// TestAllocateFreeIPSkipsGateway: the gateway belongs to the subnet; a port +// claiming it breaks routing for everyone in the subnet. +func TestAllocateFreeIPSkipsGateway(t *testing.T) { + requireDB(t) + _, subnetID := seedN4Subnet(t, "192.168.251.0/29", "192.168.251.1") + + for range 20 { + ip, err := AllocateFreeIP(context.Background(), testPool, subnetID, "192.168.251.0/29") + if err != nil { + t.Fatalf("allocateFreeIP: %v", err) + } + if ip == "192.168.251.1" { + t.Fatalf("allocated the subnet's own gateway address") + } + } +} + +// TestAllocateFreeIPExhaustedSubnet: a full subnet is an error, not a silent +// duplicate or an infinite loop. A /30 has two usable addresses; holding both +// must exhaust it. +func TestAllocateFreeIPExhaustedSubnet(t *testing.T) { + requireDB(t) + networkID, subnetID := seedN4Subnet(t, "192.168.252.0/30", "") + seedN4Port(t, networkID, subnetID, "192.168.252.1") + seedN4Port(t, networkID, subnetID, "192.168.252.2") + + if _, err := AllocateFreeIP(context.Background(), testPool, subnetID, "192.168.252.0/30"); err == nil { + t.Fatal("allocating from a subnet whose every usable address is held must fail") + } +} + +// TestAllocateIPTinySubnetDoesNotPanic pins the adjacent defect found while +// fixing N-4: the old candidate range [10, maxOffset) went negative for +// subnets with fewer than ~12 addresses, and crypto/rand.Int PANICS on a +// non-positive bound. +func TestAllocateIPTinySubnetDoesNotPanic(t *testing.T) { + for _, cidr := range []string{ + "192.168.253.0/30", + "192.168.253.0/29", + "192.168.253.0/28", + "192.168.253.0/31", + "192.168.253.1/32", + "fd00::/64", + "fd00::/126", + } { + if ip := allocateIP(cidr); ip == "" { + t.Errorf("allocateIP(%s) = \"\"; want a usable address", cidr) + } + } +} + +// TestAllocateFreeIPv6Subnet: an IPv6 subnet is allocated from normally. The +// old allocator shifted by hostBits, so a /64 overflowed int64 and panicked — +// IPv6 could not be used at all. +func TestAllocateFreeIPv6Subnet(t *testing.T) { + requireDB(t) + _, subnetID := seedN4Subnet(t, "fd00:1::/64", "") + for range 10 { + ip, err := AllocateFreeIP(context.Background(), testPool, subnetID, "fd00:1::/64") + if err != nil { + t.Fatalf("allocateFreeIP on IPv6: %v", err) + } + if parsed := net.ParseIP(ip); parsed == nil { + t.Fatalf("allocated %q, which is not an address", ip) + } + } +} + +// TestAllocateFreeIPToleratesLegacyNullFixedIPs is the regression for the +// tenant-triggerable deployment-wide allocation DoS. A port on a subnet-less +// network used to persist JSON scalar `null` in ports.fixed_ips. Because +// jsonb_array_elements was an unguarded FROM-clause SRF, one such row made +// every allocation in every subnet fail with SQLSTATE 22023. +func TestAllocateFreeIPToleratesLegacyNullFixedIPs(t *testing.T) { + requireDB(t) + ctx := context.Background() + + // A network with no subnet and a LEGACY row whose fixed_ips is JSON null, + // exactly as the old writer left it. + var bareNetworkID string + if err := testPool.QueryRow(ctx, + `INSERT INTO networks (name, project_id, status, admin_state_up) + VALUES ($1, $2, 'ACTIVE', true) RETURNING id::text`, + "n4-null-net-"+uuid.NewString()[:8], seededProjectID).Scan(&bareNetworkID); err != nil { + t.Fatalf("seeding subnet-less network: %v", err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM networks WHERE id::text = $1`, bareNetworkID) + }) + if _, err := testPool.Exec(ctx, + `INSERT INTO ports (name, network_id, project_id, mac_address, status, fixed_ips) + VALUES ('n4-null-port', $1, $2, '52:54:00:aa:bb:cc', 'ACTIVE', 'null'::jsonb)`, + bareNetworkID, seededProjectID); err != nil { + t.Fatalf("seeding legacy null-fixed_ips port: %v", err) + } + + // A different, healthy subnet must still allocate. Before the guard this + // failed for every subnet in the deployment. + _, subnetID := seedN4Subnet(t, "192.168.254.0/29", "") + ip, err := AllocateFreeIP(ctx, testPool, subnetID, "192.168.254.0/29") + if err != nil { + t.Fatalf("allocation on a healthy subnet failed because another port has JSON-null fixed_ips: %v", err) + } + if ip == "" { + t.Fatal("allocated an empty address") + } +} + +// TestCreatePortOnSubnetlessNetworkWritesEmptyArray: the writer side. A port on +// a network with no subnet must persist `[]`, never JSON null, so no NEW poison +// rows are created even though readers still tolerate old ones. +func TestCreatePortOnSubnetlessNetworkWritesEmptyArray(t *testing.T) { + requireDB(t) + ctx := context.Background() + r := newNeutronTestRouter() + + var networkID string + if err := testPool.QueryRow(ctx, + `INSERT INTO networks (name, project_id, status, admin_state_up) + VALUES ($1, $2, 'ACTIVE', true) RETURNING id::text`, + "n4-empty-net-"+uuid.NewString()[:8], seededProjectID).Scan(&networkID); err != nil { + t.Fatalf("seeding subnet-less network: %v", err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM ports WHERE network_id = $1`, networkID) + _, _ = testPool.Exec(context.Background(), `DELETE FROM networks WHERE id = $1`, networkID) + }) + + body := `{"port":{"network_id":"` + networkID + `","name":"n4-empty-port"}}` + code, resp := doReqBody(r, "POST", "/v2.0/ports", body, + map[string]string{"X-Test-Project": seededProjectID}) + if code != 201 { + t.Fatalf("create port on subnet-less network: code=%d body=%s; want 201", code, resp) + } + + var fixedIPsJSON []byte + if err := testPool.QueryRow(ctx, + `SELECT fixed_ips FROM ports WHERE network_id = $1`, networkID).Scan(&fixedIPsJSON); err != nil { + t.Fatalf("reading the created port's fixed_ips: %v", err) + } + var fixedIPs []json.RawMessage + if err := json.Unmarshal(fixedIPsJSON, &fixedIPs); err != nil { + t.Fatalf("decoding the created port's fixed_ips %s: %v", fixedIPsJSON, err) + } + if fixedIPs == nil || len(fixedIPs) != 0 { + t.Fatalf("fixed_ips is %s; want an empty JSON array so every reader can unpack it", fixedIPsJSON) + } +} + +func mustN4NetworkID(t *testing.T, subnetID string) string { + t.Helper() + var id string + if err := testPool.QueryRow(context.Background(), + `SELECT network_id::text FROM subnets WHERE id::text = $1`, subnetID).Scan(&id); err != nil { + t.Fatalf("resolving the subnet's network: %v", err) + } + return id +} diff --git a/internal/neutron/network.go b/internal/neutron/network.go index c99e3f69..dae2cfcd 100644 --- a/internal/neutron/network.go +++ b/internal/neutron/network.go @@ -570,8 +570,17 @@ func (svc *Service) CreateNetwork(c *gin.Context) { adminStateUp = *req.Network.AdminStateUp } + // Making a network shared exposes it to every tenant in the deployment — + // an operator decision, like router:external below. A member-created + // shared network was visible to, and attachable by, every project. Same + // shape as the external gate: only `true` is gated, matching upstream's + // explicit-set rule. shared := false if req.Network.Shared != nil { + if *req.Network.Shared && !common.HasRole(c, common.RoleAdmin) { + common.SendError(c, common.NewForbiddenError("only an administrator may create a shared network")) + return + } shared = *req.Network.Shared } diff --git a/internal/neutron/ports.go b/internal/neutron/ports.go index 47e401d0..85d07dc4 100644 --- a/internal/neutron/ports.go +++ b/internal/neutron/ports.go @@ -10,6 +10,7 @@ import ( "math/big" "net" "net/http" + "net/netip" "strconv" "time" @@ -43,7 +44,7 @@ func (svc *Service) CreatePort(c *gin.Context) { projectID := c.GetString("project_id") portID := uuid.New().String() - tapName := "tap-" + portID[:8] + tapName := tapNameFor(portID) macAddress := generateMAC() adminStateUp := true @@ -75,25 +76,18 @@ func (svc *Service) CreatePort(c *gin.Context) { } // Get subnet to allocate IP - var subnetID, cidr string + var subnetID string err = svc.pool.QueryRow(c.Request.Context(), - "SELECT id, cidr FROM subnets WHERE network_id = $1 LIMIT 1", + "SELECT id FROM subnets WHERE network_id = $1 LIMIT 1", req.Port.NetworkID, - ).Scan(&subnetID, &cidr) + ).Scan(&subnetID) - var fixedIPs []map[string]any - if err == nil { - // Allocate IP from subnet - allocatedIP := allocateIP(cidr) - fixedIPs = []map[string]any{ - { - "subnet_id": subnetID, - "ip_address": allocatedIP, - }, - } + if err != nil && !errors.Is(err, database.ErrNoRows) { + log.Error().Err(err).Msg("port create: subnet lookup failed") + common.SendError(c, common.NewInternalServerError("failed to look up subnet")) + return } - - fixedIPsJSON, _ := json.Marshal(fixedIPs) + fixedIPs := []map[string]any{} // Create the TAP where the bridge is — the namespace Lightstack runs in. CreateTAPDevice no longer takes // a namespace at all, so this placement is now a property of the API rather than of the call. @@ -133,15 +127,33 @@ func (svc *Service) CreatePort(c *gin.Context) { // Insert into database now := time.Now() - _, err = svc.pool.Exec(c.Request.Context(), ` + err = database.WithTx(c.Request.Context(), svc.pool, func(tx database.Tx) error { + if subnetID != "" { + cidr, err := LockSubnet(c.Request.Context(), tx, subnetID) + if err != nil { + return err + } + ip, err := AllocateFreeIP(c.Request.Context(), tx, subnetID, cidr) + if err != nil { + return err + } + fixedIPs = append(fixedIPs, map[string]any{"subnet_id": subnetID, "ip_address": ip}) + } + fixedIPsJSON, _ := json.Marshal(fixedIPs) + _, err := tx.Exec(c.Request.Context(), ` INSERT INTO ports (id, name, network_id, project_id, device_id, device_owner, mac_address, admin_state_up, status, fixed_ips, created_at, updated_at) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12) `, portID, req.Port.Name, req.Port.NetworkID, projectID, sql.NullString{String: req.Port.DeviceID, Valid: req.Port.DeviceID != ""}, - sql.NullString{String: req.Port.DeviceOwner, Valid: req.Port.DeviceOwner != ""}, macAddress, adminStateUp, "ACTIVE", fixedIPsJSON, now, now) - + sql.NullString{String: req.Port.DeviceOwner, Valid: req.Port.DeviceOwner != ""}, macAddress, adminStateUp, "ACTIVE", fixedIPsJSON, now, now) + return err + }) if err != nil { - log.Error().Err(err).Str("operation", "create_port").Msg("database error") _ = svc.tapManager.DeleteTAPDevice(tapName) + if errors.Is(err, ErrNoAvailableIP) { + common.SendError(c, common.NewConflictError("no IP address available in the network's subnet")) + return + } + log.Error().Err(err).Str("operation", "create_port").Msg("database error") common.SendError(c, common.NewInternalServerError("failed to create port")) return } @@ -531,48 +543,7 @@ func (svc *Service) DeletePort(c *gin.Context) { } portID = resolvedID - tapName := "tap-" + portID[:8] - - // Detach XDP program if eBPF mode - if svc.sgManager != nil && svc.mode == "ebpf" { - if err := svc.sgManager.DetachFromInterface(tapName); err != nil { - log.Warn().Err(err).Str("port_id", portID).Str("tap", tapName).Msg("failed to detach XDP program from interface") - } - - // Remove port from eBPF maps (need MAC address). A scan failure - // leaves macAddress empty, which skips the eBPF cleanup below — - // safe degradation; a leftover map entry will be reconciled on - // the next sync. - var macAddress string - if err := svc.pool.QueryRow(c.Request.Context(), - "SELECT mac_address FROM ports WHERE id = $1", - portID, - ).Scan(&macAddress); err != nil { - log.Warn().Err(err).Str("port_id", portID).Msg("failed to resolve port MAC during teardown") - } - - if macAddress != "" { - if mac, err := net.ParseMAC(macAddress); err == nil { - if err := svc.sgManager.RemoveSecurityGroupFromPort(portID, mac); err != nil { - log.Warn().Err(err).Str("port_id", portID).Str("mac", mac.String()).Msg("failed to remove security group from port") - } - } - } - } - - // Delete TAP device from the DEFAULT namespace, where CreatePort put it (F10). Best-effort: a - // leftover TAP is hygiene, not correctness — the port row is about to be deleted. - if err := svc.tapManager.DeleteTAPDevice(tapName); err != nil { - log.Warn().Err(err).Str("tap", tapName).Msg("failed to delete TAP device") - } - - // Remove FDB entry if VXLAN is enabled - if svc.vxlanCoordinator != nil { - if err := svc.vxlanCoordinator.RemoveFDBEntry(c.Request.Context(), portID); err != nil { - // Log but don't fail - log.Warn().Err(err).Str("port_id", portID).Msg("failed to remove FDB entry") - } - } + svc.teardownPortHost(c.Request.Context(), portID) // Delete from database. The row was resolved above, so this deletes by an id already known to // exist and to belong to the caller. @@ -653,22 +624,172 @@ func allocateIP(cidr string) string { return "" } - // Count usable host addresses (excluding network and broadcast) ones, bits := ipNet.Mask.Size() hostBits := bits - ones - if hostBits < 2 { + if hostBits < 0 { return "" } - maxOffset := int64(1<= 31: // /31: both addresses usable + lo, span = 0, 2 + case isV4: // exclude network and broadcast + lo, span = 1, (int64(1)<= 32: // IPv6: bounded random window inside the subnet + // The offset must fit incrementIP's uint and int64 arithmetic; a /64 + // has far more host bits than this, so sample a window rather than + // shifting by 64 (which overflows int64 and used to panic). + lo, span = 1, 1<<32 + default: // small IPv6 subnet: first+1 .. last + lo, span = 1, (int64(1)< on the host per attempt (mirrors CreatePort's rollbackPort). // Deleting the TAP detaches it from the bridge; nothing else needs undoing. - tapName := "tap-" + portID[:8] + tapName := tapNameFor(portID) releaseTAP := func(reason string) { if svc.mode == "stub" { return @@ -1579,6 +1688,19 @@ func (svc *Service) AllocatePortForInstance(ctx context.Context, networkID, proj // which keeps the API and the dataplane in agreement. now := time.Now() if err := database.WithTx(ctx, svc.pool, func(tx database.Tx) error { + cidr, err = LockSubnet(ctx, tx, subnetID) + if err != nil { + return err + } + allocatedIP, err = AllocateFreeIP(ctx, tx, subnetID, cidr) + if errors.Is(err, ErrNoAvailableIP) { + return &tenantVisibleError{msg: fmt.Sprintf("failed to allocate a free IP from subnet %s", subnetID)} + } + if err != nil { + return err + } + fixedIPsJSON, _ := json.Marshal([]map[string]any{{"subnet_id": subnetID, "ip_address": allocatedIP}}) + if _, err := tx.Exec(ctx, ` INSERT INTO ports (id, name, network_id, project_id, device_id, device_owner, mac_address, admin_state_up, status, fixed_ips, created_at, updated_at) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12) @@ -1723,3 +1845,98 @@ func derefOr[T any](v *T, fallback T) T { } return *v } + +// ReleaseInstancePorts deletes the ports Neutron allocated for an instance +// (device_owner 'compute:nova'), with the same best-effort host cleanup +// DeletePort does. It is the rollback for a boot that allocated ports and then +// failed on a later network: the loop used to +// markInstanceError and return with the earlier networks' ports still in the +// database — rows the API still listed for a server that would never exist, +// with leases still reserved, that only an operator could clean up. +// +// Scoped to 'compute:nova' on purpose: a port the tenant created and attached +// (os-interface with port_id) is theirs, not the instance's — releasing a +// failed boot must not delete it (same discriminator as server delete). +func (svc *Service) ReleaseInstancePorts(ctx context.Context, instanceID string) error { + rows, err := svc.pool.Query(ctx, ` + SELECT id::text, network_id::text FROM ports + WHERE device_id = $1 AND device_owner = 'compute:nova' + `, instanceID) + if err != nil { + return fmt.Errorf("looking up instance ports: %w", err) + } + defer rows.Close() + + type released struct{ portID, networkID string } + var toRelease []released + for rows.Next() { + var r released + if err := rows.Scan(&r.portID, &r.networkID); err != nil { + return fmt.Errorf("scanning instance port: %w", err) + } + toRelease = append(toRelease, r) + } + if err := rows.Err(); err != nil { + return fmt.Errorf("iterating instance ports: %w", err) + } + if len(toRelease) == 0 { + return nil + } + + for _, r := range toRelease { + svc.teardownPortHost(ctx, r.portID) + } + + if _, err := svc.pool.Exec(ctx, ` + DELETE FROM ports WHERE device_id = $1 AND device_owner = 'compute:nova' + `, instanceID); err != nil { + return fmt.Errorf("deleting instance ports: %w", err) + } + + // Refresh DHCP for the affected networks so the released addresses stop + // being reserved for a server that will not boot. + for _, r := range toRelease { + if _, _, err := svc.applyDHCPForNetwork(ctx, r.networkID, true); err != nil { + log.Warn().Err(err).Str("network_id", r.networkID). + Msg("release: failed to refresh DHCP after port release") + } + } + return nil +} + +func tapNameFor(portID string) string { + if len(portID) < 8 { + return "tap-" + portID + } + return "tap-" + portID[:8] +} + +// teardownPortHost removes host-side state for a port. Rollback and DeletePort +// use the same path so eBPF, TAP, and VXLAN cleanup cannot drift. +func (svc *Service) teardownPortHost(ctx context.Context, portID string) { + tapName := tapNameFor(portID) + if svc.sgManager != nil && svc.mode == "ebpf" { + if err := svc.sgManager.DetachFromInterface(tapName); err != nil { + log.Warn().Err(err).Str("port_id", portID).Str("tap", tapName).Msg("failed to detach XDP program from interface") + } + var macAddress string + if err := svc.pool.QueryRow(ctx, "SELECT mac_address FROM ports WHERE id = $1", portID).Scan(&macAddress); err != nil { + log.Warn().Err(err).Str("port_id", portID).Msg("failed to resolve port MAC during teardown") + } + if macAddress != "" { + if mac, err := net.ParseMAC(macAddress); err == nil { + if err := svc.sgManager.RemoveSecurityGroupFromPort(portID, mac); err != nil { + log.Warn().Err(err).Str("port_id", portID).Str("mac", mac.String()).Msg("failed to remove security group from port") + } + } + } + } + if err := svc.tapManager.DeleteTAPDevice(tapName); err != nil { + log.Warn().Err(err).Str("port_id", portID).Str("tap", tapName).Msg("failed to delete TAP device") + } + if svc.vxlanCoordinator != nil { + if err := svc.vxlanCoordinator.RemoveFDBEntry(ctx, portID); err != nil { + log.Warn().Err(err).Str("port_id", portID).Msg("failed to remove FDB entry") + } + } +} diff --git a/internal/neutron/shared_gate_test.go b/internal/neutron/shared_gate_test.go new file mode 100644 index 00000000..152e5cf5 --- /dev/null +++ b/internal/neutron/shared_gate_test.go @@ -0,0 +1,35 @@ +package neutron + +import ( + "net/http" + "testing" +) + +// A `shared` network is visible to every tenant in the +// deployment, but CreateNetwork accepted it from anyone — a member could open +// their network to every project (and every project's guests to it). It is an +// operator decision, gated exactly like router:external on create: only +// `true` is refused to a member, so a client that echoes the default +// (shared=false) keeps working for tenants, as upstream's explicit-set rule +// does for external. +func TestNonAdminCannotCreateSharedNetwork(t *testing.T) { + requireDB(t) + r := newNeutronTestRouter() + tenant := map[string]string{"X-Test-Project": seededProjectID, "X-Test-Roles": "member"} + admin := map[string]string{"X-Test-Project": seededProjectID, "X-Test-Roles": "admin"} + + if code, network := createNetwork(t, r, tenant, `{"network":{"name":"m4-tenant-shared","shared":true}}`); code != http.StatusForbidden { + t.Fatalf("member creating a shared network: code=%d body=%v; want 403", code, network) + } + + // shared=false is the default — a tenant echoing it is not asking for + // anything, and must not be refused (upstream's explicit-set rule). + if code, _ := createNetwork(t, r, tenant, `{"network":{"name":"m4-tenant-internal","shared":false}}`); code != http.StatusCreated { + t.Fatalf("member creating a normal network: want 201") + } + + // An administrator keeps the shared path. + if code, network := createNetwork(t, r, admin, `{"network":{"name":"m4-admin-shared","shared":true}}`); code != http.StatusCreated || network["shared"] != true { + t.Fatalf("admin creating a shared network: code=%d body=%v; want 201 with shared=true", code, network) + } +} diff --git a/internal/nova/aggregates.go b/internal/nova/aggregates.go index 5ef1169e..cd972f7a 100644 --- a/internal/nova/aggregates.go +++ b/internal/nova/aggregates.go @@ -26,7 +26,7 @@ import ( // ListAggregates lists all host aggregates func (svc *Service) ListAggregates(c *gin.Context) { - if !requireAdmin(c, "os-aggregates:index") { + if !common.RequireAdmin(c, "os-aggregates:index") { return } rows, err := svc.pool.Query(c.Request.Context(), ` @@ -92,7 +92,7 @@ func (svc *Service) ListAggregates(c *gin.Context) { // CreateAggregate creates a new host aggregate func (svc *Service) CreateAggregate(c *gin.Context) { - if !requireAdmin(c, "os-aggregates:create") { + if !common.RequireAdmin(c, "os-aggregates:create") { return } var req struct { @@ -141,7 +141,7 @@ func (svc *Service) CreateAggregate(c *gin.Context) { // GetAggregate retrieves a specific host aggregate func (svc *Service) GetAggregate(c *gin.Context) { - if !requireAdmin(c, "os-aggregates:show") { + if !common.RequireAdmin(c, "os-aggregates:show") { return } aggregateID := c.Param("id") @@ -194,7 +194,7 @@ func (svc *Service) GetAggregate(c *gin.Context) { // UpdateAggregate updates a host aggregate func (svc *Service) UpdateAggregate(c *gin.Context) { - if !requireAdmin(c, "os-aggregates:update") { + if !common.RequireAdmin(c, "os-aggregates:update") { return } aggregateID := c.Param("id") @@ -256,7 +256,7 @@ func (svc *Service) UpdateAggregate(c *gin.Context) { // DeleteAggregate deletes a host aggregate func (svc *Service) DeleteAggregate(c *gin.Context) { - if !requireAdmin(c, "os-aggregates:delete") { + if !common.RequireAdmin(c, "os-aggregates:delete") { return } aggregateID := c.Param("id") @@ -285,7 +285,7 @@ func (svc *Service) AggregateAction(c *gin.Context) { // Gated here rather than in the three sub-handlers below: a gate placed on // the recognised actions only would leave the unknown-action branch — and any // action added later — reachable by any tenant (SPEC-023 FR-008f). - if !requireAdmin(c, "os-aggregates:action") { + if !common.RequireAdmin(c, "os-aggregates:action") { return } diff --git a/internal/nova/authz.go b/internal/nova/authz.go index 62e8576c..cffd5853 100644 --- a/internal/nova/authz.go +++ b/internal/nova/authz.go @@ -101,35 +101,6 @@ func resolveServerOr404(c *gin.Context, pool database.DB, ref string) (id, owner return id, owner, true } -// requireAdmin gates a surface that belongs to the DEPLOYMENT rather than to any -// project — host aggregates and the deployment-wide migration list (SPEC-023 -// FR-008e, FR-008f). It reports whether the caller may proceed, sending the 403 -// itself so call sites stay one line. -// -// These are deliberately NOT owner-or-administrator. Their tables carry no -// project_id, so there is no owner for the rule to fall back to; access is -// administrative or nothing. That missing column is also how they came to be -// ungated at all: every other handler in this package was written against a -// table with a project_id, so a scoping predicate was the obvious thing to -// write — here there was nothing obvious to write, and nothing was written. -// -// 403 rather than 404: unlike an instance, the existence of these surfaces is -// not a secret (they are in the API's own version document), so there is nothing -// to disclose by admitting they are there. -func requireAdmin(c *gin.Context, operation string) bool { - if common.HasRole(c, common.RoleAdmin) { - return true - } - log.Warn(). - Str("caller_project", c.GetString("project_id")). - Str("caller_user", c.GetString("user_id")). - Str("operation", operation). - Msg("refused operator-only compute operation") - common.SendError(c, common.NewForbiddenError( - "Policy doesn't allow "+operation+" to be performed. Admin role required.")) - return false -} - // denyCrossTenant records a refused cross-tenant compute access, naming the // parties (SPEC-023 FR-007b). // diff --git a/internal/nova/boot_failure_test.go b/internal/nova/boot_failure_test.go index c66f64f3..2f9bf5cc 100644 --- a/internal/nova/boot_failure_test.go +++ b/internal/nova/boot_failure_test.go @@ -6,6 +6,9 @@ import ( "errors" "fmt" "net/http" + "os" + "os/exec" + "path/filepath" "strings" "sync" "testing" @@ -13,6 +16,8 @@ import ( "github.com/cobaltcore-dev/lightstack/internal/neutron" "github.com/cobaltcore-dev/lightstack/pkg/hypervisor" + "github.com/cobaltcore-dev/lightstack/pkg/paths" + "github.com/cobaltcore-dev/lightstack/pkg/storage" "github.com/gin-gonic/gin" ) @@ -39,9 +44,11 @@ func (e tenantErr) TenantMessage() string { return string(e) } // fakeNeutron is a NeutronService whose behaviour each test chooses. type fakeNeutron struct { - allocErrFor map[string]error // network id -> error to return - usableErr map[string]error // network id -> pre-flight verdict + allocErrFor map[string]error // network id -> error to return + usableErr map[string]error // network id -> pre-flight verdict + portFor map[string]*neutron.PortInfo // network id -> successful port payload allocCalls int + released []string // instance ids passed to ReleaseInstancePorts (N-6) // The admin flag each call carried. Guarded because allocation runs in the // VM-creation goroutine while the test reads. @@ -58,6 +65,9 @@ func (f *fakeNeutron) AllocatePortForInstance(_ context.Context, networkID, _, _ if err, ok := f.allocErrFor[networkID]; ok && err != nil { return nil, err } + if p, ok := f.portFor[networkID]; ok && p != nil { + return p, nil + } return nil, errors.New("fakeNeutron: no port payload configured") } @@ -71,6 +81,23 @@ func (f *fakeNeutron) NetworkUsableBy(_ context.Context, networkID, _ string, ad return nil } +// ReleaseInstancePorts records the rollback calls so a +// test can assert that a failed boot released the ports it had already +// allocated. +func (f *fakeNeutron) ReleaseInstancePorts(_ context.Context, instanceID string) error { + f.mu.Lock() + f.released = append(f.released, instanceID) + f.mu.Unlock() + return nil +} + +// releasedInstances returns the instance ids whose ports were released. +func (f *fakeNeutron) releasedInstances() []string { + f.mu.Lock() + defer f.mu.Unlock() + return append([]string(nil), f.released...) +} + // adminFlags returns the admin flag of every pre-flight and every allocation so far. func (f *fakeNeutron) adminFlags() (usable, alloc []bool) { f.mu.Lock() @@ -501,3 +528,113 @@ func TestFailedBootLeavesADeletableServer(t *testing.T) { t.Errorf("%d ports still reference the deleted server (FR-019)", orphanPorts) } } + +// A multi-network boot that fails on a later network used +// to leak the ports already allocated for the earlier ones — rows the API kept +// listing for a server that would never run, with leases still reserved. The +// failure path must release what it allocated (the rollback is scoped to the +// instance's nova-owned ports by the network service). +func TestFailedBootReleasesAllocatedPorts(t *testing.T) { + requireDB(t) + + good := seedBootNetwork(t, seededProjectID, "n6-good-net") + bad := seedBootNetwork(t, seededProjectID, "n6-bad-net") + fake := &fakeNeutron{ + portFor: map[string]*neutron.PortInfo{ + good: {ID: "11111111-1111-1111-1111-111111111111", NetworkID: good, + MAC: "52:54:00:12:34:56", IPAddress: "192.168.100.10"}, + }, + allocErrFor: map[string]error{ + bad: errors.New("subnet exhausted"), + }, + } + _, r := newBootService(t, fake) + + const name = "n6-partial-release" + cleanupServerNamed(t, name) + + body := fmt.Sprintf(`{"server":{"name":%q,"flavorRef":"m1.small","imageRef":%q,"networks":[{"uuid":%q},{"uuid":%q}]}}`, + name, seededImageID(t), good, bad) + if code, respBody := doReq(r, http.MethodPost, "/v2.1/servers", body, bootHdr()); code != http.StatusAccepted && code != http.StatusCreated { + t.Fatalf("create = %d. body: %s", code, respBody) + } + + if got := awaitInstanceStatus(t, name, "ERROR"); got != "ERROR" { + t.Fatalf("status = %q; want ERROR (the later network's failure fails the boot)", got) + } + + var instanceID string + if err := testPool.QueryRow(t.Context(), + `SELECT id::text FROM instances WHERE name = $1 AND project_id = $2`, name, seededProjectID, + ).Scan(&instanceID); err != nil { + t.Fatalf("resolving the failed instance: %v", err) + } + + released := fake.releasedInstances() + if len(released) != 1 || released[0] != instanceID { + t.Errorf("ReleaseInstancePorts calls = %v; want exactly [%s] — the ports allocated for the good network must be rolled back with the failed boot", released, instanceID) + } +} + +// The release must not fire on the success path: a booted server keeps its ports. +// One network, one working port payload; the instance reaches ACTIVE and no +// release is recorded. +func TestSuccessfulBootKeepsItsPorts(t *testing.T) { + requireDB(t) + + // A fully successful boot stages a per-instance qcow2 overlay, which + // needs qemu-img. The CI unit-test runner does not install it (the same + // guard pkg/storage's tests use); without the guard this test fails there + // with the boot in ERROR on overlay staging instead of testing port + // release. + if _, err := exec.LookPath("qemu-img"); err != nil { + t.Skip("qemu-img not installed — the success-path boot needs overlay staging") + } + + // Redirect the node-local state root into a test directory BEFORE the + // service is built: the boot runs in a goroutine the service's Shutdown + // (registered by newBootService, and so running LAST in cleanup's LIFO + // order) must stop before this restore runs, or a still-booting goroutine + // could read the paths global mid-rewrite (round 3 of this review). The + // image store resolves the root per call, so /var/lib is never touched. + origBase := paths.Base() + paths.SetBase(t.TempDir()) + t.Cleanup(func() { paths.SetBase(origBase) }) + + good := seedBootNetwork(t, seededProjectID, "n6-ok-net") + fake := &fakeNeutron{ + portFor: map[string]*neutron.PortInfo{ + good: {ID: "22222222-2222-2222-2222-222222222222", NetworkID: good, + MAC: "52:54:00:12:34:57", IPAddress: "192.168.100.11"}, + }, + } + _, r := newBootService(t, fake) + + const name = "n6-keep-ports" + cleanupServerNamed(t, name) + + // The boot path stages a per-instance overlay over the image's local raw + // base; without the base file the boot fails for reasons unrelated to + // ports. Stage a real base in the redirected root. + imgID := seededImageID(t) + basePath := storage.LocalImagePath(imgID) + if err := os.MkdirAll(filepath.Dir(basePath), 0o755); err != nil { + t.Fatalf("staging image base dir: %v", err) + } + if err := os.WriteFile(basePath, make([]byte, 1<<20), 0o644); err != nil { + t.Fatalf("staging image base file: %v", err) + } + + body := fmt.Sprintf(`{"server":{"name":%q,"flavorRef":"m1.small","imageRef":%q,"networks":[{"uuid":%q}]}}`, + name, imgID, good) + if code, respBody := doReq(r, http.MethodPost, "/v2.1/servers", body, bootHdr()); code != http.StatusAccepted && code != http.StatusCreated { + t.Fatalf("create = %d. body: %s", code, respBody) + } + + if got := awaitInstanceStatus(t, name, "ACTIVE"); got != "ACTIVE" { + t.Fatalf("status = %q; want ACTIVE (nothing on this path fails)", got) + } + if released := fake.releasedInstances(); len(released) != 0 { + t.Errorf("ReleaseInstancePorts calls = %v; want none — a booted server keeps its ports", released) + } +} diff --git a/internal/nova/console.go b/internal/nova/console.go index ca174287..fbdb8627 100644 --- a/internal/nova/console.go +++ b/internal/nova/console.go @@ -380,6 +380,9 @@ func unsupportedConsoleTypeMsg(consoleType string) string { // projection directory without asking us, so nothing about the console depends // on this endpoint working — but `openstack` and third-party tooling expect it. func (svc *Service) ShowConsoleAuthToken(c *gin.Context) { + if !common.RequireAdmin(c, "os-console-auth-tokens:show") { + return + } if svc.consoleStore == nil { common.SendError(c, common.NewNotFoundError("console token")) return diff --git a/internal/nova/flavor_bounds_test.go b/internal/nova/flavor_bounds_test.go new file mode 100644 index 00000000..02e5022b --- /dev/null +++ b/internal/nova/flavor_bounds_test.go @@ -0,0 +1,41 @@ +package nova + +import ( + "context" + "fmt" + "net/http" + "testing" + + "github.com/google/uuid" +) + +func TestFlavorBoundsMatchFlamingo(t *testing.T) { + requireDB(t) + r := newNovaTestRouter() + // Flamingo parameter_types.flavor_param_{positive,non_negative} use + // signed 32-bit limits. A zero root disk is valid for boot-from-volume. + for _, tc := range []struct { + name, fields string + want int + }{ + {"zero-disk", `"ram":512,"vcpus":1,"disk":0`, http.StatusOK}, + {"upper-bound", `"ram":2147483647,"vcpus":2147483647,"disk":2147483647`, http.StatusOK}, + {"missing-disk", `"ram":512,"vcpus":1`, http.StatusBadRequest}, + {"zero-ram", `"ram":0,"vcpus":1,"disk":1`, http.StatusBadRequest}, + {"zero-cpu", `"ram":512,"vcpus":0,"disk":1`, http.StatusBadRequest}, + {"negative-disk", `"ram":512,"vcpus":1,"disk":-1`, http.StatusBadRequest}, + {"overflow-ram", `"ram":2147483648,"vcpus":1,"disk":1`, http.StatusBadRequest}, + {"overflow-cpu", `"ram":512,"vcpus":2147483648,"disk":1`, http.StatusBadRequest}, + {"overflow-disk", `"ram":512,"vcpus":1,"disk":2147483648`, http.StatusBadRequest}, + {"overflow-ephemeral", `"ram":512,"vcpus":1,"disk":1,"OS-FLV-EXT-DATA:ephemeral":2147483648`, http.StatusBadRequest}, + } { + t.Run(tc.name, func(t *testing.T) { + name := "bounds-" + uuid.NewString() + t.Cleanup(func() { testPool.Exec(context.Background(), "DELETE FROM flavors WHERE name=$1", name) }) + code, body := doReq(r, http.MethodPost, "/v2.1/flavors", fmt.Sprintf(`{"flavor":{"name":%q,%s}}`, name, tc.fields), operatorHdr("admin")) + if code != tc.want { + t.Errorf("create flavor: %d %s; want %d", code, body, tc.want) + } + }) + } +} diff --git a/internal/nova/flavors.go b/internal/nova/flavors.go index fb6aa180..5e8ea820 100644 --- a/internal/nova/flavors.go +++ b/internal/nova/flavors.go @@ -1,6 +1,7 @@ package nova import ( + "math" "net/http" "github.com/cobaltcore-dev/lightstack/internal/common" @@ -12,12 +13,19 @@ import ( // CreateFlavor handles POST /v2.1/flavors func (svc *Service) CreateFlavor(c *gin.Context) { + // Admin-only: the flavor catalog is a shared, + // deployment-wide resource — members must not create, delete, or + // re-scope it (upstream: admin_api for all flavor management). + if !common.RequireAdmin(c, "flavor-create") { + return + } + var req struct { Flavor struct { Name string `json:"name" binding:"required"` RAM int `json:"ram" binding:"required"` VCPUs int `json:"vcpus" binding:"required"` - Disk int `json:"disk" binding:"required"` + Disk *int `json:"disk" binding:"required"` Ephemeral int `json:"OS-FLV-EXT-DATA:ephemeral"` IsPublic *bool `json:"os-flavor-access:is_public"` } `json:"flavor" binding:"required"` @@ -28,17 +36,22 @@ func (svc *Service) CreateFlavor(c *gin.Context) { return } - // Validate input ranges - if req.Flavor.RAM < 0 { - common.SendError(c, common.NewBadRequestError("RAM must be non-negative")) + // Flamingo's flavor schema uses signed 32-bit values. RAM/vCPUs must be + // positive; zero root disk is valid for a server booted from a volume. + if req.Flavor.RAM < 1 || req.Flavor.RAM > math.MaxInt32 { + common.SendError(c, common.NewBadRequestError("RAM must be between 1 and 2147483647 MB")) return } - if req.Flavor.VCPUs < 0 { - common.SendError(c, common.NewBadRequestError("VCPUs must be non-negative")) + if req.Flavor.VCPUs < 1 || req.Flavor.VCPUs > math.MaxInt32 { + common.SendError(c, common.NewBadRequestError("VCPUs must be between 1 and 2147483647")) return } - if req.Flavor.Disk < 0 { - common.SendError(c, common.NewBadRequestError("Disk must be non-negative")) + if *req.Flavor.Disk < 0 || *req.Flavor.Disk > math.MaxInt32 { + common.SendError(c, common.NewBadRequestError("Disk must be between 0 and 2147483647 GB")) + return + } + if req.Flavor.Ephemeral < 0 || req.Flavor.Ephemeral > math.MaxInt32 { + common.SendError(c, common.NewBadRequestError("Ephemeral disk must be between 0 and 2147483647 GB")) return } @@ -54,7 +67,7 @@ func (svc *Service) CreateFlavor(c *gin.Context) { _, err := svc.pool.Exec(ctx, `INSERT INTO flavors (id, name, vcpus, ram_mb, disk_gb, is_public) VALUES ($1, $2, $3, $4, $5, $6)`, - flavorID, req.Flavor.Name, req.Flavor.VCPUs, req.Flavor.RAM, req.Flavor.Disk, isPublic, + flavorID, req.Flavor.Name, req.Flavor.VCPUs, req.Flavor.RAM, *req.Flavor.Disk, isPublic, ) if err != nil { // Check for unique constraint violation (duplicate name) @@ -78,7 +91,7 @@ func (svc *Service) CreateFlavor(c *gin.Context) { "name": req.Flavor.Name, "vcpus": req.Flavor.VCPUs, "ram": req.Flavor.RAM, - "disk": req.Flavor.Disk, + "disk": *req.Flavor.Disk, "OS-FLV-EXT-DATA:ephemeral": req.Flavor.Ephemeral, "OS-FLV-DISABLED:disabled": false, "os-flavor-access:is_public": isPublic, @@ -90,6 +103,13 @@ func (svc *Service) CreateFlavor(c *gin.Context) { // DeleteFlavor handles DELETE /v2.1/flavors/:id func (svc *Service) DeleteFlavor(c *gin.Context) { + // Admin-only: the flavor catalog is a shared, + // deployment-wide resource — members must not create, delete, or + // re-scope it (upstream: admin_api for all flavor management). + if !common.RequireAdmin(c, "flavor-delete") { + return + } + flavorID := c.Param("id") ctx := c.Request.Context() @@ -153,6 +173,13 @@ func (svc *Service) GetFlavorExtraSpecs(c *gin.Context) { // CreateFlavorExtraSpecs handles POST /v2.1/flavors/:id/os-extra_specs func (svc *Service) CreateFlavorExtraSpecs(c *gin.Context) { + // Admin-only: the flavor catalog is a shared, + // deployment-wide resource — members must not create, delete, or + // re-scope it (upstream: admin_api for all flavor management). + if !common.RequireAdmin(c, "flavor-extra-specs-create") { + return + } + flavorID := c.Param("id") var req struct { @@ -203,6 +230,13 @@ func (svc *Service) GetFlavorExtraSpecKey(c *gin.Context) { // UpdateFlavorExtraSpecKey handles PUT /v2.1/flavors/:id/os-extra_specs/:key func (svc *Service) UpdateFlavorExtraSpecKey(c *gin.Context) { + // Admin-only: the flavor catalog is a shared, + // deployment-wide resource — members must not create, delete, or + // re-scope it (upstream: admin_api for all flavor management). + if !common.RequireAdmin(c, "flavor-extra-specs-update") { + return + } + flavorID := c.Param("id") key := c.Param("key") @@ -237,6 +271,13 @@ func (svc *Service) UpdateFlavorExtraSpecKey(c *gin.Context) { // DeleteFlavorExtraSpecKey handles DELETE /v2.1/flavors/:id/os-extra_specs/:key func (svc *Service) DeleteFlavorExtraSpecKey(c *gin.Context) { + // Admin-only: the flavor catalog is a shared, + // deployment-wide resource — members must not create, delete, or + // re-scope it (upstream: admin_api for all flavor management). + if !common.RequireAdmin(c, "flavor-extra-specs-delete") { + return + } + flavorID := c.Param("id") key := c.Param("key") @@ -261,6 +302,13 @@ func (svc *Service) DeleteFlavorExtraSpecKey(c *gin.Context) { // FlavorAction handles POST /v2.1/flavors/:id/action func (svc *Service) FlavorAction(c *gin.Context) { + // Admin-only: the flavor catalog is a shared, + // deployment-wide resource — members must not create, delete, or + // re-scope it (upstream: admin_api for all flavor management). + if !common.RequireAdmin(c, "flavor-access-modify") { + return + } + flavorID := c.Param("id") var req map[string]any @@ -327,6 +375,13 @@ func (svc *Service) FlavorAction(c *gin.Context) { // GetFlavorAccess handles GET /v2.1/flavors/:id/os-flavor-access func (svc *Service) GetFlavorAccess(c *gin.Context) { + // Admin-only: the flavor catalog is a shared, + // deployment-wide resource — members must not create, delete, or + // re-scope it (upstream: admin_api for all flavor management). + if !common.RequireAdmin(c, "flavor-access-list") { + return + } + flavorID := c.Param("id") rows, err := svc.pool.Query(c.Request.Context(), diff --git a/internal/nova/handlers.go b/internal/nova/handlers.go index b03a01a7..1a5d878c 100644 --- a/internal/nova/handlers.go +++ b/internal/nova/handlers.go @@ -60,6 +60,12 @@ const ( // NeutronService defines the interface for Neutron operations Nova needs. type NeutronService interface { AllocatePortForInstance(ctx context.Context, networkID, projectID, instanceID string, securityGroupIDs []string, admin bool) (any, error) + // ReleaseInstancePorts is AllocatePortForInstance's rollback: it deletes the + // ports Neutron allocated for a boot that failed after allocating them + // — the port rows, their host plumbing and their + // DHCP reservations. Scoped to nova-owned ports; a tenant-created port the + // server was told to use is the tenant's, not the boot's. + ReleaseInstancePorts(ctx context.Context, instanceID string) error // NetworkUsableBy reports whether a project may attach a guest to a network, // returning a non-nil error naming the reason if not. admin is whether the caller // holds the administrator role, which decides an unshared external network; both @@ -1115,6 +1121,23 @@ func (svc *Service) CreateServer(c *gin.Context) { // Allocate ports from Neutron for requested networks log.Debug().Str("instance_id", instanceID).Int("network_count", len(req.Server.Networks)).Msg("Allocating ports") var networks []hypervisor.NetworkConfig + portsAllocated := false + bootSucceeded := false + // Any failure from the first successful allocation onward (a later + // network, the boot volume, libvirt itself) leaves the earlier ports + // allocated for a server that will never run. Release them on the way + // out unless the boot completed: they used to leak — still listed by + // the API, leases still reserved. + defer func() { + if portsAllocated && !bootSucceeded && svc.neutronSvc != nil { + rctx, rcancel := context.WithTimeout(context.Background(), 15*time.Second) + defer rcancel() + if err := svc.neutronSvc.ReleaseInstancePorts(rctx, instanceID); err != nil { + log.Warn().Err(err).Str("instance_id", instanceID). + Msg("failed to release allocated ports after failed boot") + } + } + }() if svc.neutronSvc != nil && len(req.Server.Networks) > 0 { for _, network := range req.Server.Networks { log.Debug().Str("network_id", network.UUID).Msg("Allocating port for network") @@ -1159,6 +1182,7 @@ func (svc *Service) CreateServer(c *gin.Context) { MACAddress: portInfo.MAC, BridgeName: networking.BridgeNameForNetwork(portInfo.NetworkID), }) + portsAllocated = true } } @@ -1342,6 +1366,9 @@ func (svc *Service) CreateServer(c *gin.Context) { } log.Debug().Str("libvirt_uuid", libvirtUUID).Msg("VM created successfully") + // The boot completed: the allocated ports belong to the running + // server now, and the release defer must leave them alone. + bootSucceeded = true logger.Info(). Str("instance_id", instanceID). @@ -2123,11 +2150,29 @@ func (svc *Service) DeleteServer(c *gin.Context) { return } - // Delete the instance's auto-allocated neutron ports (Nova owns them). Without - // this the port lingers after the server is gone, and deleting its network then - // fails with "network has active ports" — which breaks `terraform destroy` - // (SPEC-009 / SC-007) and leaks ports. - if _, e := tx.Exec(ctx, "DELETE FROM ports WHERE device_id = $1", instanceUUID); e != nil { + // The instance's ports, split by who owns them: + // + // - Ports Neutron allocated for the boot (device_owner 'compute:nova') are + // Nova's to delete. Without this they linger after the server is gone and + // deleting the network then fails with "network has active ports" — which + // breaks `terraform destroy` (SPEC-009 / SC-007) and leaks ports. + // + // - A port the tenant created and attached (os-interface with port_id) is + // the tenant's resource, not the server's: upstream detaches it — clears + // the device, parks it DOWN — rather than deleting it. The blanket + // `DELETE ... WHERE device_id = $1` used to destroy it: the tenant lost a + // port they made, with its fixed IP, by deleting a server. + if _, e := tx.Exec(ctx, ` + UPDATE ports + SET device_id = NULL, device_owner = NULL, status = 'DOWN', updated_at = NOW() + WHERE device_id = $1 AND COALESCE(device_owner, '') <> 'compute:nova' + `, instanceUUID); e != nil { + logger.Error().Err(e).Str("instance_id", instanceUUID).Msg("delete: detach tenant ports failed") + common.SendError(c, common.NewInternalServerError("failed to delete server")) + return + } + if _, e := tx.Exec(ctx, + "DELETE FROM ports WHERE device_id = $1 AND device_owner = 'compute:nova'", instanceUUID); e != nil { logger.Error().Err(e).Str("instance_id", instanceUUID).Msg("delete: remove instance ports failed") common.SendError(c, common.NewInternalServerError("failed to delete server")) return @@ -2918,6 +2963,9 @@ func (svc *Service) ListImagesDetail(c *gin.Context) { // ListHypervisors lists hypervisors (mock for Horizon) func (svc *Service) ListHypervisors(c *gin.Context) { + if !common.RequireAdmin(c, "os-hypervisors:index") { + return + } c.JSON(200, gin.H{"hypervisors": []gin.H{ { "id": 1, @@ -2941,6 +2989,9 @@ func (svc *Service) ListHypervisors(c *gin.Context) { // ListHypervisorsDetail lists hypervisors with details (mock for Horizon) func (svc *Service) ListHypervisorsDetail(c *gin.Context) { + if !common.RequireAdmin(c, "os-hypervisors:detail") { + return + } cpuInfoJSON := `{"arch":"x86_64","model":"Skylake-Server-IBRS","vendor":"Intel","features":[],"topology":{"cores":8,"threads":2,"sockets":1}}` c.JSON(200, gin.H{"hypervisors": []gin.H{ @@ -2967,6 +3018,9 @@ func (svc *Service) ListHypervisorsDetail(c *gin.Context) { // GetHypervisorStatistics returns aggregated hypervisor statistics (for Horizon) func (svc *Service) GetHypervisorStatistics(c *gin.Context) { + if !common.RequireAdmin(c, "os-hypervisors:statistics") { + return + } // Count running instances. Scan failures degrade to zero (the stats // response still ships) and are logged for diagnosis. var runningVMs int @@ -3212,6 +3266,9 @@ func (svc *Service) GetLimits(c *gin.Context) { // ListServices returns list of compute services func (svc *Service) ListServices(c *gin.Context) { + if !common.RequireAdmin(c, "os-services:index") { + return + } // Format: "2006-01-02T15:04:05.000000" (without Z, microseconds) now := time.Now().Format("2006-01-02T15:04:05.000000") diff --git a/internal/nova/interface_attach.go b/internal/nova/interface_attach.go index 117d6ba2..987cc50a 100644 --- a/internal/nova/interface_attach.go +++ b/internal/nova/interface_attach.go @@ -1,16 +1,15 @@ package nova import ( - "context" "encoding/json" "errors" "fmt" + "net" "net/http" "time" - "github.com/cobaltcore-dev/lightstack/internal/database" - "github.com/cobaltcore-dev/lightstack/internal/common" + "github.com/cobaltcore-dev/lightstack/internal/database" "github.com/cobaltcore-dev/lightstack/internal/neutron" "github.com/gin-gonic/gin" "github.com/google/uuid" @@ -60,35 +59,57 @@ func (svc *Service) AttachInterface(c *gin.Context) { var networkID string var fixedIP string var macAddress string + // fixedIPEntries is what the response reports as the attachment's + // fixed_ips: the port's actual entries (port_id path) or the one just + // allocated (net_id path). Empty means none — not a fabricated one. + var fixedIPEntries common.FixedIPs + // novaCreatedPort is true when this request had Nova create the port + // (net_id path); false when it attached a port the tenant already owned + // (port_id path). It decides the device_owner stamp below (N-8). + novaCreatedPort := false // If port_id provided, use existing port if req.InterfaceAttachment.PortID != "" { portID = req.InterfaceAttachment.PortID - // Verify port exists and is available + // Verify port exists and is available. device_id is nullable — every + // free port has NULL there — so it is read COALESCE'd rather than + // scanned into a plain string (round 3 of this review): that scan + // fails on NULL, and the error used to be dropped, leaving the attach + // "correct" only because a failed Scan invalidates the lookup, even if some earlier columns were read. Any real database error on this lookup surfaced the same way: + // a bogus 200 with an empty net_id and MAC. var deviceID string - var fixedIPsJSON []byte + var fixedIPs common.FixedIPs err := svc.pool.QueryRow(c.Request.Context(), - "SELECT network_id, fixed_ips, mac_address, device_id FROM ports WHERE id = $1 AND project_id = $2", + "SELECT network_id, fixed_ips, mac_address, COALESCE(device_id::text, '') FROM ports WHERE id = $1 AND project_id = $2", portID, projectID, - ).Scan(&networkID, &fixedIPsJSON, &macAddress, &deviceID) + ).Scan(&networkID, &fixedIPs, &macAddress, &deviceID) if errors.Is(err, database.ErrNoRows) { common.SendError(c, common.NewNotFoundError("port")) return } + if err != nil { + log.Error().Err(err).Str("operation", "attach_interface").Str("port_id", portID).Msg("port lookup failed") + common.SendError(c, common.NewInternalServerError("database error")) + return + } if deviceID != "" { common.SendError(c, common.NewConflictError("port already attached to another device")) return } - // Parse fixed_ips from JSONB - // For simplicity, just extract first IP - fixedIP = "192.168.1.10" // Placeholder, would parse from fixedIPsJSON + // The port's actual fixed_ips, for the response and for nothing else: + // its addressing lives in the JSONB, and a hardcoded placeholder used + // to be reported instead (round 2 of this review). A port with no + // addresses reports an empty list, as upstream does — not a fabricated + // {"ip_address": ""}. + fixedIPEntries = fixedIPs } else if req.InterfaceAttachment.NetID != "" { // Create new port on the specified network networkID = req.InterfaceAttachment.NetID + novaCreatedPort = true // The rule booting applies (neutron.GuestMayAttach): the server's own networks and shared // ones, and an unshared external network for administrators only — upstream applies @@ -115,39 +136,109 @@ func (svc *Service) AttachInterface(c *gin.Context) { portID = uuid.New().String() macAddress = generateMACAddress() - // Get subnet info for IP allocation - var cidr string - err = svc.pool.QueryRow(c.Request.Context(), - "SELECT cidr FROM subnets WHERE network_id = $1 LIMIT 1", - networkID, - ).Scan(&cidr) - - if errors.Is(err, database.ErrNoRows) { + // Resolve the subnet containing an explicit address. Networks can have + // both IPv4 and IPv6 subnets, so an arbitrary first subnet can reject a + // valid address from the other family. + subnetRows, err := svc.pool.Query(c.Request.Context(), + "SELECT id, cidr FROM subnets WHERE network_id = $1 ORDER BY id", networkID) + if err != nil { + log.Error().Err(err).Str("operation", "allocate_ip").Msg("subnet lookup error") + common.SendError(c, common.NewInternalServerError("failed to allocate IP")) + return + } + type subnetInfo struct{ id, cidr string } + var subnets []subnetInfo + for subnetRows.Next() { + var subnet subnetInfo + if err := subnetRows.Scan(&subnet.id, &subnet.cidr); err != nil { + subnetRows.Close() + log.Error().Err(err).Str("operation", "allocate_ip").Msg("subnet scan error") + common.SendError(c, common.NewInternalServerError("failed to allocate IP")) + return + } + subnets = append(subnets, subnet) + } + rowsErr := subnetRows.Err() + subnetRows.Close() + if rowsErr != nil { + log.Error().Err(rowsErr).Str("operation", "allocate_ip").Msg("subnet iteration error") + common.SendError(c, common.NewInternalServerError("failed to allocate IP")) + return + } + if len(subnets) == 0 { common.SendError(c, common.NewBadRequestError("network has no subnet")) return } + subnetID, cidr := subnets[0].id, subnets[0].cidr // Use provided fixed_ip or allocate one if req.InterfaceAttachment.FixedIP != "" { fixedIP = req.InterfaceAttachment.FixedIP - } else { - // Allocate next available IP from subnet - fixedIP, err = allocateNextIP(c.Request.Context(), svc.pool, networkID, cidr) - if err != nil { - log.Error().Err(err).Str("operation", "allocate_ip").Msg("IP allocation error") - common.SendError(c, common.NewInternalServerError("failed to allocate IP")) + // A caller-specified address is validated, not trusted (round 3 + // of this review): it must parse and lie in the subnet (400), and + // no other port may hold it (409, upstream's "IP address already + // allocated") — the explicit side of the N-4 collision rule the + // allocator covers for generated addresses. Parsing also keeps + // arbitrary strings out of the JSONB build below. + ip := net.ParseIP(fixedIP) + if ip == nil { + common.SendError(c, common.NewBadRequestError("fixed_ip is not a valid address in the network's subnet")) return } + matchedSubnet := false + for _, subnet := range subnets { + _, ipNet, cidrErr := net.ParseCIDR(subnet.cidr) + if cidrErr == nil && ipNet.Contains(ip) { + subnetID, cidr = subnet.id, subnet.cidr + matchedSubnet = true + break + } + } + if !matchedSubnet { + common.SendError(c, common.NewBadRequestError("fixed_ip is not a valid address in the network's subnet")) + return + } + fixedIP = ip.String() } - // Create port - fixedIPsJSON := fmt.Sprintf(`[{"ip_address": "%s", "subnet_id": "%s"}]`, fixedIP, networkID) - _, err = svc.pool.Exec(c.Request.Context(), ` - INSERT INTO ports (id, network_id, project_id, name, mac_address, fixed_ips, device_id, device_owner, status, created_at, updated_at) - VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, $8, $9, $10, $11) - `, portID, networkID, projectID, fmt.Sprintf("port-%s", portID[:8]), macAddress, fixedIPsJSON, instanceID, "compute:nova", "ACTIVE", time.Now(), time.Now()) - + // Allocate/check and insert while holding the subnet lock shared with + // Neutron's port-create and server-boot paths. + err = database.WithTx(c.Request.Context(), svc.pool, func(tx database.Tx) error { + cidr, err = neutron.LockSubnet(c.Request.Context(), tx, subnetID) + if err != nil { + return err + } + if fixedIP != "" { + available, err := neutron.IPAvailable(c.Request.Context(), tx, subnetID, fixedIP) + if err != nil { + return err + } + if !available { + return neutron.ErrNoAvailableIP + } + } else { + fixedIP, err = neutron.AllocateFreeIP(c.Request.Context(), tx, subnetID, cidr) + if err != nil { + return err + } + } + fixedIPEntries = common.FixedIPs{{IPAddress: fixedIP, SubnetID: subnetID}} + fixedIPsJSON, _ := json.Marshal(fixedIPEntries) + _, err = tx.Exec(c.Request.Context(), ` + INSERT INTO ports (id, network_id, project_id, name, mac_address, fixed_ips, device_id, device_owner, status, created_at, updated_at) + VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, $8, $9, $10, $11) + `, portID, networkID, projectID, fmt.Sprintf("port-%s", portID[:8]), macAddress, fixedIPsJSON, instanceID, "compute:nova", "ACTIVE", time.Now(), time.Now()) + return err + }) if err != nil { + if errors.Is(err, neutron.ErrNoAvailableIP) { + message := "no IP address available in the network's subnet" + if req.InterfaceAttachment.FixedIP != "" { + message = "IP address already allocated in the subnet" + } + common.SendError(c, common.NewConflictError(message)) + return + } log.Error().Err(err).Str("operation", "create_port").Msg("database error") common.SendError(c, common.NewInternalServerError("failed to create port")) return @@ -157,12 +248,24 @@ func (svc *Service) AttachInterface(c *gin.Context) { return } - // Update port to attach to instance + // Attach the port to the instance. device_owner is stamped + // 'compute:nova' only on the port Nova itself just created — that stamp + // is the ownership discriminator server delete and the boot-failure + // rollback key on: a tenant's port attached by id + // keeps its own device_owner, so deleting the server DETACHES it instead + // of destroying it — upstream's deallocate_for_instance skips ports it + // did not create for exactly this reason. The old blanket stamp is also + // what made the tenant-port half of N-8 unreachable through the real + // attach flow (round 2 of this review). + var deviceOwner any + if novaCreatedPort { + deviceOwner = "compute:nova" + } _, err = svc.pool.Exec(c.Request.Context(), ` UPDATE ports - SET device_id = $1, device_owner = $2, status = $3, updated_at = $4 + SET device_id = $1, device_owner = COALESCE($2, device_owner), status = $3, updated_at = $4 WHERE id = $5 - `, instanceID, "compute:nova", "ACTIVE", time.Now(), portID) + `, instanceID, deviceOwner, "ACTIVE", time.Now(), portID) if err != nil { log.Error().Err(err).Str("operation", "update_port_attach").Msg("database error") @@ -190,12 +293,18 @@ func (svc *Service) AttachInterface(c *gin.Context) { // an administrator's change in their own history (SPEC-023 FR-007a). recordInstanceAction(c, svc.pool, instanceID, projectID, "attach_interface") + // The port's actual fixed_ips — entries with their subnet ids, or an + // empty list for a port with none (round 3 of this review: this used to + // fabricate exactly one {"ip_address": ""}). + if fixedIPEntries == nil { + fixedIPEntries = common.FixedIPs{} + } c.JSON(http.StatusOK, gin.H{ "interfaceAttachment": gin.H{ "id": attachmentID, "port_id": portID, "net_id": networkID, - "fixed_ips": []gin.H{{"ip_address": fixedIP}}, + "fixed_ips": fixedIPEntries, "mac_addr": macAddress, "port_state": "ACTIVE", }, @@ -210,10 +319,15 @@ func (svc *Service) DetachInterface(c *gin.Context) { } portID := c.Param("port_id") - // Verify port is attached to this instance + // Verify port is attached to this instance. device_id is nullable — a + // detached port has NULL there — so it is read COALESCE'd rather than + // scanned into a plain string (same class as the attach-path fix in this + // review: that scan fails on NULL, the error was dropped, and the "not + // attached" answer below was right only by accident, with real database + // errors 409-ing as "not attached" instead of 500-ing). var attachedInstanceID string err := svc.pool.QueryRow(c.Request.Context(), - "SELECT device_id FROM ports WHERE id = $1", + "SELECT COALESCE(device_id::text, '') FROM ports WHERE id = $1", portID, ).Scan(&attachedInstanceID) @@ -221,6 +335,11 @@ func (svc *Service) DetachInterface(c *gin.Context) { common.SendError(c, common.NewNotFoundError("port")) return } + if err != nil { + log.Error().Err(err).Str("operation", "detach_interface").Str("port_id", portID).Msg("port lookup failed") + common.SendError(c, common.NewInternalServerError("database error")) + return + } if attachedInstanceID != instanceID { common.SendError(c, common.NewConflictError("port not attached to this instance")) @@ -285,10 +404,10 @@ func (svc *Service) ListInterfaceAttachments(c *gin.Context) { var attachments []gin.H for rows.Next() { var id, portID, networkID, macAddress string - var fixedIPsJSON []byte + var fixedIPs common.FixedIPs var portState *string - if err := rows.Scan(&id, &portID, &networkID, &fixedIPsJSON, &macAddress, &portState); err != nil { + if err := rows.Scan(&id, &portID, &networkID, &fixedIPs, &macAddress, &portState); err != nil { log.Warn().Err(err).Str("operation", "ListInterfaceAttachments").Msg("failed to scan row") continue } @@ -298,18 +417,19 @@ func (svc *Service) ListInterfaceAttachments(c *gin.Context) { state = *portState } - // Parse fixed_ips from JSONB - var fixedIPs []map[string]any - if err := json.Unmarshal(fixedIPsJSON, &fixedIPs); err == nil && len(fixedIPs) > 0 { - attachments = append(attachments, gin.H{ - "id": id, - "port_id": portID, - "net_id": networkID, - "fixed_ips": fixedIPs, - "mac_addr": macAddress, - "port_state": state, - }) + // An attachment with no addresses remains visible and reports an empty + // fixed_ips list, matching the attach response. + if fixedIPs == nil { + fixedIPs = common.FixedIPs{} } + attachments = append(attachments, gin.H{ + "id": id, + "port_id": portID, + "net_id": networkID, + "fixed_ips": fixedIPs, + "mac_addr": macAddress, + "port_state": state, + }) } if err := rows.Err(); err != nil { log.Error().Err(err).Msg("rows iteration error") @@ -330,52 +450,3 @@ func generateMACAddress() string { return fmt.Sprintf("fa:16:3e:%02x:%02x:%02x", uuid.New()[0], uuid.New()[1], uuid.New()[2]) } - -// allocateNextIP allocates the next available IP from a subnet -func allocateNextIP(ctx context.Context, pool database.DB, networkID, cidr string) (string, error) { - // Simple allocation: get last used IP and increment - // In production, would use proper IPAM - var fixedIPsJSON []byte - err := pool.QueryRow(ctx, - "SELECT fixed_ips FROM ports WHERE network_id = $1 ORDER BY created_at DESC LIMIT 1", - networkID, - ).Scan(&fixedIPsJSON) - - if errors.Is(err, database.ErrNoRows) { - // First allocation, use .10 - baseIP := cidr[:len(cidr)-3] // Remove /24 - return baseIP + "10", nil - } - - if err != nil { - return "", err - } - - // For simplicity, parse JSON and increment - // In production, would use proper IPAM library - var ips []map[string]any - if err := json.Unmarshal(fixedIPsJSON, &ips); err != nil { - // Fallback to simple allocation - baseIP := cidr[:len(cidr)-3] - return baseIP + "11", nil - } - - if len(ips) == 0 { - baseIP := cidr[:len(cidr)-3] - return baseIP + "10", nil - } - - lastIP := ips[0]["ip_address"].(string) - - // Increment last octet - var a, b, c, d int - if _, err := fmt.Sscanf(lastIP, "%d.%d.%d.%d", &a, &b, &c, &d); err != nil { - return "", fmt.Errorf("malformed IP %q in subnet allocation table: %w", lastIP, err) - } - d++ - if d > 254 { - return "", fmt.Errorf("subnet exhausted") - } - - return fmt.Sprintf("%d.%d.%d.%d", a, b, c, d), nil -} diff --git a/internal/nova/interface_attach_portid_test.go b/internal/nova/interface_attach_portid_test.go new file mode 100644 index 00000000..a4716d63 --- /dev/null +++ b/internal/nova/interface_attach_portid_test.go @@ -0,0 +1,307 @@ +package nova + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "strings" + "testing" + + "github.com/google/uuid" +) + +// Regression coverage for F1/F2/F3: the port_id path of +// os-interface. +// +// F1 — every free port has device_id NULL, and the lookup scanned it into a +// plain string. The scan failed, the error was dropped, and the attach +// "worked" only because pgx fills earlier columns before the failed one; any +// real database error on that lookup also surfaced as a bogus 200 with an +// empty net_id and MAC. The lookup must read device_id nullably and handle +// its errors. +// +// F2 — the response reported the port's first fixed IP (a hardcoded +// placeholder before, then only partially fixed) and fabricated +// {"ip_address": ""} for a port with no addresses. It must report the port's +// actual fixed_ips entries with their subnet ids, or an empty list. +// +// F3 — an explicit fixed_ip was stored unvalidated: not checked against the +// subnet, not checked against the addresses other ports hold (the explicit +// side of the N-4 collision rule), and arbitrary text reached the JSONB +// build. It must 400 out-of-subnet and 409 taken addresses. +func TestAttachInterfacePortIDReportsRealState(t *testing.T) { + requireDB(t) + ctx := context.Background() + r := newNovaTestRouter() + + server, _ := seedAuthzFixtures(t) + member := map[string]string{"X-Test-Project": authzProjectA, "X-Test-User": seededUserID, "X-Test-Roles": "member"} + + netID := uuid.NewString() + subnetID := uuid.NewString() + if _, err := testPool.Exec(ctx, + `INSERT INTO networks (id, name, project_id, status, admin_state_up) + VALUES ($1, 'r3-iface-net', $2, 'ACTIVE', true)`, netID, authzProjectA); err != nil { + t.Fatalf("seeding network: %v", err) + } + if _, err := testPool.Exec(ctx, + `INSERT INTO subnets (id, name, network_id, project_id, cidr) + VALUES ($1, 'r3-iface-sub', $2, $3, '10.249.0.0/24')`, subnetID, netID, authzProjectA); err != nil { + t.Fatalf("seeding subnet: %v", err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM interface_attachments WHERE port_id IN (SELECT id FROM ports WHERE network_id = $1)`, netID) + _, _ = testPool.Exec(context.Background(), `DELETE FROM ports WHERE network_id = $1`, netID) + _, _ = testPool.Exec(context.Background(), `DELETE FROM subnets WHERE network_id = $1`, netID) + _, _ = testPool.Exec(context.Background(), `DELETE FROM networks WHERE id = $1`, netID) + }) + + // A free tenant port (device_id NULL — the state every detached port is + // in) holding one address. + freePort := uuid.NewString() + fixed, _ := json.Marshal([]map[string]any{ + {"subnet_id": subnetID, "ip_address": "10.249.0.55"}, + }) + if _, err := testPool.Exec(ctx, + `INSERT INTO ports (id, name, network_id, project_id, mac_address, status, fixed_ips) + VALUES ($1, 'r3-free', $2, $3, '52:54:00:aa:bb:01', 'DOWN', $4::jsonb)`, + freePort, netID, authzProjectA, fixed); err != nil { + t.Fatalf("seeding free port: %v", err) + } + + code, body := doReq(r, http.MethodPost, "/v2.1/servers/"+server+"/os-interface", + fmt.Sprintf(`{"interfaceAttachment":{"port_id":%q}}`, freePort), member) + if code != http.StatusOK { + t.Fatalf("attaching a free (device_id NULL) port: code=%d body=%s; want 200 — the F1 defect made this ride on a swallowed scan error", code, body) + } + var resp struct { + InterfaceAttachment struct { + NetID string `json:"net_id"` + MacAddr string `json:"mac_addr"` + FixedIPs []struct { + IPAddress string `json:"ip_address"` + SubnetID string `json:"subnet_id"` + } `json:"fixed_ips"` + } `json:"interfaceAttachment"` + } + if err := json.Unmarshal([]byte(body), &resp); err != nil { + t.Fatalf("response is not JSON: %v (%s)", err, body) + } + if resp.InterfaceAttachment.NetID != netID || resp.InterfaceAttachment.MacAddr != "52:54:00:aa:bb:01" { + t.Errorf("attachment reports net_id=%q mac=%q; want the port's real values (the F1 defect reported empties on any lookup error)", resp.InterfaceAttachment.NetID, resp.InterfaceAttachment.MacAddr) + } + if len(resp.InterfaceAttachment.FixedIPs) != 1 || + resp.InterfaceAttachment.FixedIPs[0].IPAddress != "10.249.0.55" || + resp.InterfaceAttachment.FixedIPs[0].SubnetID != subnetID { + t.Errorf("attachment fixed_ips = %+v; want the port's actual entry with its subnet id (F2)", resp.InterfaceAttachment.FixedIPs) + } +} + +// The empty half of F2: a port with no addresses reports an empty list, not a +// fabricated {"ip_address": ""}. +func TestAttachInterfacePortIDWithoutAddressesReportsNone(t *testing.T) { + requireDB(t) + ctx := context.Background() + r := newNovaTestRouter() + + server, _ := seedAuthzFixtures(t) + member := map[string]string{"X-Test-Project": authzProjectA, "X-Test-User": seededUserID, "X-Test-Roles": "member"} + + netID := uuid.NewString() + if _, err := testPool.Exec(ctx, + `INSERT INTO networks (id, name, project_id, status, admin_state_up) + VALUES ($1, 'r3-iface-empty', $2, 'ACTIVE', true)`, netID, authzProjectA); err != nil { + t.Fatalf("seeding network: %v", err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM interface_attachments WHERE port_id IN (SELECT id FROM ports WHERE network_id = $1)`, netID) + _, _ = testPool.Exec(context.Background(), `DELETE FROM ports WHERE network_id = $1`, netID) + _, _ = testPool.Exec(context.Background(), `DELETE FROM networks WHERE id = $1`, netID) + }) + + addrlessPort := uuid.NewString() + if _, err := testPool.Exec(ctx, + `INSERT INTO ports (id, name, network_id, project_id, mac_address, status, fixed_ips) + VALUES ($1, 'r3-addrless', $2, $3, '52:54:00:aa:bb:02', 'DOWN', '[]'::jsonb)`, + addrlessPort, netID, authzProjectA); err != nil { + t.Fatalf("seeding address-less port: %v", err) + } + + code, body := doReq(r, http.MethodPost, "/v2.1/servers/"+server+"/os-interface", + fmt.Sprintf(`{"interfaceAttachment":{"port_id":%q}}`, addrlessPort), member) + if code != http.StatusOK { + t.Fatalf("attach: code=%d body=%s", code, body) + } + if want := `"fixed_ips":[]`; !strings.Contains(body, want) { + t.Errorf("a port with no addresses must report fixed_ips: [], not a fabricated entry: %s", body) + } +} + +// F3: a caller-specified fixed_ip is validated — 400 outside the subnet, 409 +// when another port holds it — instead of being stored blind. +func TestAttachInterfaceExplicitFixedIPIsValidated(t *testing.T) { + requireDB(t) + ctx := context.Background() + r := newNovaTestRouter() + + server, _ := seedAuthzFixtures(t) + member := map[string]string{"X-Test-Project": authzProjectA, "X-Test-User": seededUserID, "X-Test-Roles": "member"} + + netID := uuid.NewString() + subnetID := uuid.NewString() + if _, err := testPool.Exec(ctx, + `INSERT INTO networks (id, name, project_id, status, admin_state_up) + VALUES ($1, 'r3-iface-explicit', $2, 'ACTIVE', true)`, netID, authzProjectA); err != nil { + t.Fatalf("seeding network: %v", err) + } + if _, err := testPool.Exec(ctx, + `INSERT INTO subnets (id, name, network_id, project_id, cidr, gateway_ip) + VALUES ($1, 'r3-iface-explicit-sub', $2, $3, '10.250.0.0/24', '10.250.0.1')`, + subnetID, netID, authzProjectA); err != nil { + t.Fatalf("seeding subnet: %v", err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM ports WHERE network_id = $1`, netID) + _, _ = testPool.Exec(context.Background(), `DELETE FROM subnets WHERE network_id = $1`, netID) + _, _ = testPool.Exec(context.Background(), `DELETE FROM networks WHERE id = $1`, netID) + }) + + // A port already holding the address the caller will ask for. + holder := uuid.NewString() + fixed, _ := json.Marshal([]map[string]any{ + {"subnet_id": subnetID, "ip_address": "10.250.0.77"}, + }) + if _, err := testPool.Exec(ctx, + `INSERT INTO ports (id, name, network_id, project_id, mac_address, status, fixed_ips) + VALUES ($1, 'r3-holder', $2, $3, '52:54:00:aa:bb:03', 'DOWN', $4::jsonb)`, + holder, netID, authzProjectA, fixed); err != nil { + t.Fatalf("seeding holder port: %v", err) + } + + attach := func(body string) (int, string) { + return doReq(r, http.MethodPost, "/v2.1/servers/"+server+"/os-interface", body, member) + } + + if code, body := attach(fmt.Sprintf(`{"interfaceAttachment":{"net_id":%q,"fixed_ip":"10.250.0.77"}}`, netID)); code != http.StatusConflict { + t.Errorf("explicit fixed_ip held by another port: code=%d body=%s; want 409 (the explicit side of N-4)", code, body) + } + if code, body := attach(fmt.Sprintf(`{"interfaceAttachment":{"net_id":%q,"fixed_ip":"192.0.2.1"}}`, netID)); code != http.StatusBadRequest { + t.Errorf("explicit fixed_ip outside the subnet: code=%d body=%s; want 400", code, body) + } + if code, body := attach(fmt.Sprintf(`{"interfaceAttachment":{"net_id":%q,"fixed_ip":"not-an-ip"}}`, netID)); code != http.StatusBadRequest { + t.Errorf("non-IP fixed_ip: code=%d body=%s; want 400 (not a 500 from the JSONB cast)", code, body) + } + + // A free, in-subnet address still works. + code, body := attach(fmt.Sprintf(`{"interfaceAttachment":{"net_id":%q,"fixed_ip":"10.250.0.99"}}`, netID)) + if code != http.StatusOK { + t.Fatalf("valid explicit fixed_ip: code=%d body=%s; want 200", code, body) + } + if !strings.Contains(body, `"ip_address": "10.250.0.99"`) && !strings.Contains(body, `"ip_address":"10.250.0.99"`) { + t.Errorf("response must report the requested address: %s", body) + } +} + +func TestAttachInterfaceSelectsMatchingSubnetOnDualStackNetwork(t *testing.T) { + requireDB(t) + ctx := context.Background() + r := newNovaTestRouter() + server, _ := seedAuthzFixtures(t) + member := map[string]string{"X-Test-Project": authzProjectA, "X-Test-User": seededUserID, "X-Test-Roles": "member"} + netID := uuid.NewString() + v4ID, v6ID := uuid.NewString(), uuid.NewString() + if v6ID < v4ID { + v4ID, v6ID = v6ID, v4ID + } + if _, err := testPool.Exec(ctx, "INSERT INTO networks (id, name, project_id, status, admin_state_up) VALUES ($1, 'review-dual-stack', $2, 'ACTIVE', true)", netID, authzProjectA); err != nil { + t.Fatalf("seed network: %v", err) + } + for _, subnet := range []struct{ id, name, cidr, gateway string }{ + {v4ID, "review-dual-v4", "10.251.0.0/24", "10.251.0.1"}, + {v6ID, "review-dual-v6", "2001:db8:251::/64", "2001:db8:251::1"}, + } { + if _, err := testPool.Exec(ctx, "INSERT INTO subnets (id, name, network_id, project_id, cidr, gateway_ip) VALUES ($1, $2, $3, $4, $5, $6)", subnet.id, subnet.name, netID, authzProjectA, subnet.cidr, subnet.gateway); err != nil { + t.Fatalf("seed subnet %s: %v", subnet.id, err) + } + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), "DELETE FROM interface_attachments WHERE port_id IN (SELECT id FROM ports WHERE network_id = $1)", netID) + _, _ = testPool.Exec(context.Background(), "DELETE FROM ports WHERE network_id = $1", netID) + _, _ = testPool.Exec(context.Background(), "DELETE FROM subnets WHERE network_id = $1", netID) + _, _ = testPool.Exec(context.Background(), "DELETE FROM networks WHERE id = $1", netID) + }) + + const requestedIP = "2001:db8:251::42" + bodyJSON, err := json.Marshal(map[string]any{"interfaceAttachment": map[string]string{"net_id": netID, "fixed_ip": requestedIP}}) + if err != nil { + t.Fatalf("marshal interface attach request: %v", err) + } + body := string(bodyJSON) + if code, response := doReq(r, http.MethodPost, "/v2.1/servers/"+server+"/os-interface", body, member); code != http.StatusOK { + t.Fatalf("attach IPv6 address on dual-stack network: code=%d body=%s", code, response) + } + var fixedIPsJSON []byte + if err := testPool.QueryRow(ctx, "SELECT fixed_ips FROM ports WHERE network_id = $1 AND device_id = $2", netID, server).Scan(&fixedIPsJSON); err != nil { + t.Fatalf("read attached port addresses: %v", err) + } + var entries []struct { + SubnetID string `json:"subnet_id"` + IPAddress string `json:"ip_address"` + } + if err := json.Unmarshal(fixedIPsJSON, &entries); err != nil { + t.Fatalf("decode fixed_ips %s: %v", fixedIPsJSON, err) + } + if len(entries) != 1 || entries[0].SubnetID != v6ID || entries[0].IPAddress != requestedIP { + t.Errorf("attached fixed_ips = %s; want subnet %s with address %s", fixedIPsJSON, v6ID, requestedIP) + } +} + +// Round 3, L-1: an attachment whose port has no addresses must appear in the +// os-interface list with an empty fixed_ips — the list used to skip it, so an +// attachment created a moment earlier was invisible, disagreeing with the +// attach response that (since F2) reports it. +func TestListInterfacesIncludesAddresslessAttachments(t *testing.T) { + requireDB(t) + ctx := context.Background() + r := newNovaTestRouter() + + server, _ := seedAuthzFixtures(t) + member := map[string]string{"X-Test-Project": authzProjectA, "X-Test-User": seededUserID, "X-Test-Roles": "member"} + + netID := uuid.NewString() + if _, err := testPool.Exec(ctx, + `INSERT INTO networks (id, name, project_id, status, admin_state_up) + VALUES ($1, 'r3-list-empty', $2, 'ACTIVE', true)`, netID, authzProjectA); err != nil { + t.Fatalf("seeding network: %v", err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM interface_attachments WHERE port_id IN (SELECT id FROM ports WHERE network_id = $1)`, netID) + _, _ = testPool.Exec(context.Background(), `DELETE FROM ports WHERE network_id = $1`, netID) + _, _ = testPool.Exec(context.Background(), `DELETE FROM networks WHERE id = $1`, netID) + }) + + addrlessPort := uuid.NewString() + if _, err := testPool.Exec(ctx, + `INSERT INTO ports (id, name, network_id, project_id, mac_address, status, fixed_ips) + VALUES ($1, 'r3-list-addrless', $2, $3, '52:54:00:aa:bb:04', 'ACTIVE', '[]'::jsonb)`, + addrlessPort, netID, authzProjectA); err != nil { + t.Fatalf("seeding address-less port: %v", err) + } + + if code, body := doReq(r, http.MethodPost, "/v2.1/servers/"+server+"/os-interface", + fmt.Sprintf(`{"interfaceAttachment":{"port_id":%q}}`, addrlessPort), member); code != http.StatusOK { + t.Fatalf("attach: code=%d body=%s", code, body) + } + + code, body := doReq(r, http.MethodGet, "/v2.1/servers/"+server+"/os-interface", "", member) + if code != http.StatusOK { + t.Fatalf("list interfaces: code=%d body=%s", code, body) + } + if !strings.Contains(body, addrlessPort) { + t.Errorf("an attachment whose port has no addresses is missing from the os-interface list: %s", body) + } + if !strings.Contains(body, `"fixed_ips":[]`) { + t.Errorf("the address-less attachment must list fixed_ips as [], matching the attach response: %s", body) + } +} diff --git a/internal/nova/ip_allocation_concurrency_test.go b/internal/nova/ip_allocation_concurrency_test.go new file mode 100644 index 00000000..faaa671a --- /dev/null +++ b/internal/nova/ip_allocation_concurrency_test.go @@ -0,0 +1,103 @@ +package nova + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/cobaltcore-dev/lightstack/internal/neutron" + "github.com/gin-gonic/gin" + "github.com/google/uuid" +) + +// All three port writers share one last address. An exhausted request must +// fail without creating a second port holding it. +func TestConcurrentPortWritersReserveOneAddress(t *testing.T) { + requireDB(t) + serverID := seedConsoleInstance(t, seededProjectID) + networkID, subnetID := uuid.NewString(), uuid.NewString() + if _, err := testPool.Exec(t.Context(), `INSERT INTO networks (id, name, project_id, status) + VALUES ($1, 'concurrent-ip', $2, 'ACTIVE')`, networkID, seededProjectID); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM interface_attachments WHERE port_id IN (SELECT id FROM ports WHERE network_id=$1)`, networkID) + _, _ = testPool.Exec(context.Background(), "DELETE FROM networks WHERE id=$1", networkID) + }) + if _, err := testPool.Exec(t.Context(), `INSERT INTO subnets (id, name, network_id, project_id, cidr) + VALUES ($1, 'one-address', $2, $3, '192.0.2.7/32')`, subnetID, networkID, seededProjectID); err != nil { + t.Fatal(err) + } + if _, err := neutron.EnsureDefaultSecurityGroup(t.Context(), testPool, seededProjectID); err != nil { + t.Fatal(err) + } + novaRouter := newNovaTestRouter() + networkSvc := neutron.NewService(testPool, "stub", nil) + portRouter := gin.New() + portRouter.Use(func(c *gin.Context) { + c.Set("project_id", seededProjectID) + c.Set("roles", []string{"member"}) + }) + networkSvc.RegisterRoutes(&portRouter.RouterGroup) + ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second) + defer cancel() + const count = 12 + start := make(chan struct{}) + type result struct { + code int + body string + } + results := make(chan result, count) + for i := range count { + go func() { + <-start + if i%3 == 0 { + _, err := networkSvc.AllocatePortForInstance(ctx, networkID, seededProjectID, serverID, nil, false) + if err == nil { + results <- result{http.StatusCreated, ""} + return + } + code := http.StatusInternalServerError + if strings.Contains(err.Error(), "failed to allocate a free IP") { + code = http.StatusConflict + } + results <- result{code, err.Error()} + return + } + router, path, body := portRouter, "/v2.0/ports", fmt.Sprintf(`{"port":{"network_id":%q}}`, networkID) + if i%3 == 1 { + router, path = novaRouter, "/v2.1/servers/"+serverID+"/os-interface" + body = fmt.Sprintf(`{"interfaceAttachment":{"net_id":%q,"fixed_ip":"192.0.2.7"}}`, networkID) + } + req := httptest.NewRequest(http.MethodPost, path, strings.NewReader(body)).WithContext(ctx) + req.Header.Set("Content-Type", "application/json") + for k, v := range testAuthHdr(seededProjectID, seededUserID) { + req.Header.Set(k, v) + } + w := httptest.NewRecorder() + router.ServeHTTP(w, req) + results <- result{w.Code, w.Body.String()} + }() + } + close(start) + successes := 0 + for range count { + r := <-results + if r.code == http.StatusOK || r.code == http.StatusCreated { + successes++ + } else if r.code != http.StatusConflict { + t.Errorf("unexpected response %d %s", r.code, r.body) + } + } + var ports int + if err := testPool.QueryRow(t.Context(), "SELECT count(*) FROM ports WHERE network_id=$1", networkID).Scan(&ports); err != nil { + t.Fatal(err) + } + if successes != 1 || ports != 1 { + t.Errorf("successes=%d ports=%d; want one address claimed exactly once", successes, ports) + } +} diff --git a/internal/nova/keypairs.go b/internal/nova/keypairs.go index f4e6c1a3..90a4a025 100644 --- a/internal/nova/keypairs.go +++ b/internal/nova/keypairs.go @@ -127,6 +127,14 @@ func (svc *Service) CreateKeypair(c *gin.Context) { var privateKey *string var fingerprint string + // Bound the import: an imported public key is + // parsed and stored verbatim — an unbounded string is an unbounded row. + // Authorized-key blobs are a few hundred bytes; 64 KiB is generous. + if len(req.Keypair.PublicKey) > 64<<10 { + common.SendError(c, common.NewBadRequestError("public key too large")) + return + } + if req.Keypair.PublicKey != "" { // Import existing public key publicKey = req.Keypair.PublicKey @@ -164,7 +172,10 @@ func (svc *Service) CreateKeypair(c *gin.Context) { return } - // Insert into database + // Insert into database. The name pre-check above is only a friendly fast + // path — under concurrency the UNIQUE(user_id, name) constraint is the + // real guard, so its violation is the same 409, not a 500 when concurrent + // check-then-insert requests collide. now := time.Now() _, err = svc.pool.Exec(c.Request.Context(), ` INSERT INTO keypairs (user_id, name, public_key, fingerprint, created_at) @@ -172,6 +183,10 @@ func (svc *Service) CreateKeypair(c *gin.Context) { `, userID, req.Keypair.Name, publicKey, fingerprint, now) if err != nil { + if database.IsUniqueViolation(err) { + common.SendError(c, common.NewConflictError("Keypair with this name already exists")) + return + } log.Error().Err(err).Str("operation", "create_keypair").Msg("database error") common.SendError(c, common.NewInternalServerError("failed to create keypair")) return diff --git a/internal/nova/migrations.go b/internal/nova/migrations.go index 2a1affa5..88720027 100644 --- a/internal/nova/migrations.go +++ b/internal/nova/migrations.go @@ -18,7 +18,7 @@ import ( // doing. The comment below already said as much; the check was simply missing. func (svc *Service) ListMigrations(c *gin.Context) { // List all migrations (admin endpoint in real OpenStack) - if !requireAdmin(c, "os-migrations:index") { + if !common.RequireAdmin(c, "os-migrations:index") { return } rows, err := svc.pool.Query(c.Request.Context(), ` diff --git a/internal/nova/operator_surfaces_test.go b/internal/nova/operator_surfaces_test.go index 3199f121..8987049d 100644 --- a/internal/nova/operator_surfaces_test.go +++ b/internal/nova/operator_surfaces_test.go @@ -130,6 +130,63 @@ func TestAggregateAdminUnchanged(t *testing.T) { _, _ = testPool.Exec(context.Background(), `DELETE FROM host_aggregates WHERE name = 'spec023-agg-2'`) } +// TestDeploymentReadSurfacesRequireAdmin: the hypervisor and service views +// describe the whole deployment, not a project, so they are administrator-only +// like the migration list and host aggregates. /os-hypervisors/statistics is +// the sharp one: it reports a real deployment-wide running-VM count. +func TestDeploymentReadSurfacesRequireAdmin(t *testing.T) { + requireDB(t) + seedAuthzFixtures(t) + r := newNovaTestRouter() + + paths := []string{ + "/v2.1/os-hypervisors", + "/v2.1/os-hypervisors/detail", + "/v2.1/os-hypervisors/statistics", + "/v2.1/os-services", + } + for _, path := range paths { + t.Run("member "+path, func(t *testing.T) { + code, body := doReq(r, http.MethodGet, path, "", operatorHdr("member")) + if code != http.StatusForbidden { + t.Errorf("member GET %s: code=%d, want 403. body: %s", path, code, body) + } + }) + t.Run("admin "+path, func(t *testing.T) { + code, body := doReq(r, http.MethodGet, path, "", operatorHdr("admin")) + if code != http.StatusOK { + t.Errorf("admin GET %s: code=%d, want 200. body: %s", path, code, body) + } + }) + } +} + +// TestConsoleAuthTokenRequiresAdmin: the compatibility token view resolves an +// arbitrary console token, which names another user's instance and its +// internal access path. It is an operator surface, gated before the store is +// even consulted (a member gets 403, not a store-dependent 404). +func TestConsoleAuthTokenRequiresAdmin(t *testing.T) { + requireDB(t) + r, store := newConsoleTestRouter(t, "stub") + id := seedConsoleInstance(t, seededProjectID) + tok, err := store.Issue(t.Context(), id, seededProjectID, testConsoleSocket, testDomainID) + if err != nil { + t.Fatalf("issuing a console token: %v", err) + } + + member := operatorHdr("member") + code, body := doReq(r, http.MethodGet, "/v2.1/os-console-auth-tokens/"+tok, "", member) + if code != http.StatusForbidden { + t.Errorf("member reading a console token: code=%d, want 403. body: %s", code, body) + } + + admin := testAuthHdr(seededProjectID, seededUserID) + code, body = doReq(r, http.MethodGet, "/v2.1/os-console-auth-tokens/"+tok, "", admin) + if code != http.StatusOK { + t.Errorf("admin reading a console token: code=%d, want 200. body: %s", code, body) + } +} + // TestMigrationListRequiresAdmin: O1. The deployment-wide list reports every // project's instance movements — server uuid, source and destination node, // flavour changes — and was readable by any authenticated tenant. The handler's diff --git a/internal/nova/port_lifecycle_test.go b/internal/nova/port_lifecycle_test.go new file mode 100644 index 00000000..540d10b3 --- /dev/null +++ b/internal/nova/port_lifecycle_test.go @@ -0,0 +1,184 @@ +package nova + +import ( + "context" + "fmt" + "net/http" + "testing" + + "github.com/cobaltcore-dev/lightstack/pkg/paths" +) + +// DeleteServer used a blanket +// `DELETE FROM ports WHERE device_id = $1`, which destroyed ports the TENANT +// created and attached (os-interface with port_id) along with the ones Nova +// allocated for the boot. Upstream deletes only the ports it created and +// detaches the rest — the tenant's port survives the server, parked DOWN with +// no device, reusable or deletable by its owner. +// +// The tenant port is attached through the real os-interface flow: the fix +// keys on device_owner='compute:nova' marking Nova's own +// ports, and AttachInterface used to stamp that owner on a tenant's port too — +// which would have made the discriminator, and this behaviour, unreachable. +func TestDeleteServerDetachesTenantPortsButDeletesItsOwn(t *testing.T) { + requireDB(t) + ctx := context.Background() + + netID := seedBootNetwork(t, seededProjectID, "n8-net") + + var instanceID string + if err := testPool.QueryRow(ctx, + `INSERT INTO instances (name, project_id, user_id, flavor_id, status) + VALUES ('n8-server', $1, $2, (SELECT id FROM flavors LIMIT 1), 'ACTIVE') + RETURNING id::text`, seededProjectID, seededUserID).Scan(&instanceID); err != nil { + t.Fatalf("seeding instance: %v", err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM ports WHERE network_id::text = $1`, netID) + _, _ = testPool.Exec(context.Background(), `DELETE FROM instances WHERE id::text = $1`, instanceID) + }) + + seedPort := func(owner string) string { + var id string + if err := testPool.QueryRow(ctx, ` + INSERT INTO ports (name, network_id, project_id, device_id, device_owner, mac_address, status, fixed_ips) + VALUES ($1, $2, $3, $4, NULLIF($5, ''), $6, 'ACTIVE', '[]'::jsonb) RETURNING id::text`, + "n8-"+owner, netID, seededProjectID, instanceID, owner, fmt.Sprintf("52:54:00:00:00:%02x", len(owner))).Scan(&id); err != nil { + t.Fatalf("seeding %s port: %v", owner, err) + } + return id + } + novaPort := seedPort("compute:nova") + tenantPort := seedPort("") // tenant-created, attached via os-interface + + r := newNovaTestRouter() + owner := map[string]string{ + "X-Test-Project": seededProjectID, "X-Test-User": seededUserID, "X-Test-Roles": "member", + } + + // Detach the tenant port from the instance first, then re-attach it + // through the real os-interface flow, so the test exercises the handler + // that used to stamp it compute:nova. + if _, err := testPool.Exec(ctx, + `UPDATE ports SET device_id = NULL WHERE id::text = $1`, tenantPort); err != nil { + t.Fatalf("freeing the tenant port for attach: %v", err) + } + attachBody := fmt.Sprintf(`{"interfaceAttachment":{"port_id":%q}}`, tenantPort) + if code, body := doReq(r, http.MethodPost, "/v2.1/servers/"+instanceID+"/os-interface", attachBody, owner); code != http.StatusOK { + t.Fatalf("attaching the tenant port via os-interface: code=%d body=%s", code, body) + } + + // The attach must NOT have claimed the port as Nova's: device_owner is the + // ownership discriminator server delete keys on. + var deviceOwner string + if err := testPool.QueryRow(ctx, + `SELECT COALESCE(device_owner, '') FROM ports WHERE id::text = $1`, tenantPort).Scan(&deviceOwner); err != nil { + t.Fatalf("reading the attached port's owner: %v", err) + } + if deviceOwner == "compute:nova" { + t.Fatal("os-interface stamped a tenant-created port as compute:nova; it would be deleted with the server instead of detached") + } + + if code, body := doReq(r, http.MethodDelete, "/v2.1/servers/"+instanceID, "", owner); code != http.StatusNoContent { + t.Fatalf("deleting the server: code=%d body=%s", code, body) + } + + // Nova's own port is gone with the server. + var novaPortGone bool + if err := testPool.QueryRow(ctx, + `SELECT NOT EXISTS(SELECT 1 FROM ports WHERE id::text = $1)`, novaPort).Scan(&novaPortGone); err != nil { + t.Fatalf("checking the nova-owned port: %v", err) + } + if !novaPortGone { + t.Error("the boot-allocated port (device_owner compute:nova) survived the server delete; it must be deleted with it") + } + + // The tenant's port survives, detached: no device, parked DOWN. + var deviceID, status string + var exists bool + if err := testPool.QueryRow(ctx, + `SELECT EXISTS(SELECT 1 FROM ports WHERE id::text = $1)`, tenantPort).Scan(&exists); err != nil || !exists { + t.Fatalf("the tenant-created port must survive the server delete (exists=%v err=%v)", exists, err) + } + if err := testPool.QueryRow(ctx, ` + SELECT COALESCE(device_id::text, ''), status + FROM ports WHERE id::text = $1`, tenantPort).Scan(&deviceID, &status); err != nil { + t.Fatalf("reading the surviving port: %v", err) + } + if deviceID != "" { + t.Errorf("the surviving port is still attached (device_id=%q); want it detached", deviceID) + } + if status != "DOWN" { + t.Errorf("the surviving port's status = %q; want DOWN (detached, as upstream parks it)", status) + } +} + +// The net_id half of the same discriminator: a port Nova creates for an +// os-interface attach IS Nova's, and must be deleted with the server. +func TestDeleteServerDeletesInterfaceAttachCreatedPort(t *testing.T) { + requireDB(t) + ctx := context.Background() + + netID := seedBootNetwork(t, seededProjectID, "n8b-net") + // The net_id path needs a subnet to allocate from. + if _, err := testPool.Exec(ctx, + `INSERT INTO subnets (name, network_id, project_id, cidr) + VALUES ('n8b-sub', $1, $2, '192.168.77.0/24')`, netID, seededProjectID); err != nil { + t.Fatalf("seeding subnet: %v", err) + } + + // Redirect the node-local state root (image bases, overlays) into a test + // directory so nothing touches /var/lib. + origBase := paths.Base() + paths.SetBase(t.TempDir()) + t.Cleanup(func() { paths.SetBase(origBase) }) + + var instanceID string + if err := testPool.QueryRow(ctx, + `INSERT INTO instances (name, project_id, user_id, flavor_id, status) + VALUES ('n8b-server', $1, $2, (SELECT id FROM flavors LIMIT 1), 'ACTIVE') + RETURNING id::text`, seededProjectID, seededUserID).Scan(&instanceID); err != nil { + t.Fatalf("seeding instance: %v", err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), `DELETE FROM ports WHERE network_id::text = $1`, netID) + _, _ = testPool.Exec(context.Background(), `DELETE FROM instances WHERE id::text = $1`, instanceID) + }) + + r := newNovaTestRouter() + owner := map[string]string{ + "X-Test-Project": seededProjectID, "X-Test-User": seededUserID, "X-Test-Roles": "member", + } + + attachBody := fmt.Sprintf(`{"interfaceAttachment":{"net_id":%q}}`, netID) + code, body := doReq(r, http.MethodPost, "/v2.1/servers/"+instanceID+"/os-interface", attachBody, owner) + if code != http.StatusOK { + t.Fatalf("attaching via net_id: code=%d body=%s", code, body) + } + var createdPort string + if err := testPool.QueryRow(ctx, + `SELECT id::text FROM ports WHERE network_id::text = $1 AND device_id::text = $2`, + netID, instanceID).Scan(&createdPort); err != nil { + t.Fatalf("resolving the attached port: %v", err) + } + var deviceOwner string + if err := testPool.QueryRow(ctx, + `SELECT COALESCE(device_owner, '') FROM ports WHERE id::text = $1`, createdPort).Scan(&deviceOwner); err != nil { + t.Fatalf("reading the created port's owner: %v", err) + } + if deviceOwner != "compute:nova" { + t.Fatalf("a port Nova created for os-interface must be stamped compute:nova (got %q); it is Nova's to clean up", deviceOwner) + } + + if code, body := doReq(r, http.MethodDelete, "/v2.1/servers/"+instanceID, "", owner); code != http.StatusNoContent { + t.Fatalf("deleting the server: code=%d body=%s", code, body) + } + var exists bool + if err := testPool.QueryRow(ctx, + `SELECT EXISTS(SELECT 1 FROM ports WHERE id::text = $1)`, createdPort).Scan(&exists); err != nil { + t.Fatalf("checking the created port: %v", err) + } + if exists { + t.Error("the port Nova created for os-interface survived the server delete; it must be deleted with it") + } +} diff --git a/internal/nova/review_regressions_test.go b/internal/nova/review_regressions_test.go new file mode 100644 index 00000000..deb12de0 --- /dev/null +++ b/internal/nova/review_regressions_test.go @@ -0,0 +1,90 @@ +package nova + +import ( + "context" + "fmt" + "net/http" + "strings" + "testing" + + "github.com/google/uuid" +) + +func TestFlavorOperatorHandlersRequireAdmin(t *testing.T) { + requireDB(t) + r := newNovaTestRouter() + flavorID := uuid.NewString() + flavorName := "review-flavor-" + flavorID[:8] + if _, err := testPool.Exec(t.Context(), ` + INSERT INTO flavors (id, name, vcpus, ram_mb, disk_gb, is_public) + VALUES ($1, $2, 1, 512, 1, false) + `, flavorID, flavorName); err != nil { + t.Fatalf("seed flavor: %v", err) + } + t.Cleanup(func() { _, _ = testPool.Exec(context.Background(), "DELETE FROM flavors WHERE id = $1", flavorID) }) + + member := map[string]string{"X-Test-Project": seededProjectID, "X-Test-User": seededUserID, "X-Test-Roles": "member"} + admin := map[string]string{"X-Test-Project": seededProjectID, "X-Test-User": seededUserID, "X-Test-Roles": "admin"} + paths := []struct{ method, path, body string }{ + {http.MethodPost, "/v2.1/flavors", `{"flavor":{"name":"member-flavor","ram":512,"vcpus":1,"disk":1}}`}, + {http.MethodDelete, "/v2.1/flavors/" + flavorID, ""}, + {http.MethodPost, "/v2.1/flavors/" + flavorID + "/os-extra_specs", `{"extra_specs":{"review":"member"}}`}, + {http.MethodPut, "/v2.1/flavors/" + flavorID + "/os-extra_specs/review", `{"review":"member"}`}, + {http.MethodDelete, "/v2.1/flavors/" + flavorID + "/os-extra_specs/review", ""}, + {http.MethodPost, "/v2.1/flavors/" + flavorID + "/action", `{"addTenantAccess":{"tenant":"` + seededProjectID + `"}}`}, + {http.MethodGet, "/v2.1/flavors/" + flavorID + "/os-flavor-access", ""}, + } + for _, tc := range paths { + if code, body := doReq(r, tc.method, tc.path, tc.body, member); code != http.StatusForbidden { + t.Errorf("member %s %s: code=%d body=%s; want 403", tc.method, tc.path, code, body) + } + } + + createdName := "review-created-" + uuid.NewString()[:8] + if code, body := doReq(r, http.MethodPost, "/v2.1/flavors", + fmt.Sprintf(`{"flavor":{"name":%q,"ram":512,"vcpus":1,"disk":1}}`, createdName), admin); code != http.StatusOK { + t.Fatalf("admin create flavor: code=%d body=%s; want 200", code, body) + } + t.Cleanup(func() { _, _ = testPool.Exec(context.Background(), "DELETE FROM flavors WHERE name = $1", createdName) }) + checkAdmin := func(method, path, body string, want int) { + t.Helper() + if code, response := doReq(r, method, path, body, admin); code != want { + t.Errorf("admin %s %s: code=%d body=%s; want %d", method, path, code, response, want) + } + } + checkAdmin(http.MethodPost, "/v2.1/flavors/"+flavorID+"/os-extra_specs", `{"extra_specs":{"review":"admin"}}`, http.StatusOK) + checkAdmin(http.MethodPut, "/v2.1/flavors/"+flavorID+"/os-extra_specs/review", `{"review":"updated"}`, http.StatusOK) + checkAdmin(http.MethodDelete, "/v2.1/flavors/"+flavorID+"/os-extra_specs/review", "", http.StatusNoContent) + checkAdmin(http.MethodPost, "/v2.1/flavors/"+flavorID+"/action", `{"addTenantAccess":{"tenant":"`+seededProjectID+`"}}`, http.StatusOK) + checkAdmin(http.MethodGet, "/v2.1/flavors/"+flavorID+"/os-flavor-access", "", http.StatusOK) + checkAdmin(http.MethodDelete, "/v2.1/flavors/"+flavorID, "", http.StatusNoContent) +} + +func TestKeypairImportLimitAndDuplicateConflict(t *testing.T) { + requireDB(t) + r := newNovaTestRouter() + hdr := map[string]string{"X-Test-Project": seededProjectID, "X-Test-User": seededUserID, "X-Test-Roles": "member"} + oversized := fmt.Sprintf(`{"keypair":{"name":"too-large","public_key":%q}}`, strings.Repeat("x", (64<<10)+1)) + if code, body := doReq(r, http.MethodPost, "/v2.1/os-keypairs", oversized, hdr); code != http.StatusBadRequest { + t.Errorf("oversized keypair import: code=%d body=%s; want 400", code, body) + } + + publicKey, _, fingerprint, err := generateSSHKeyPair() + if err != nil { + t.Fatalf("generate test SSH key: %v", err) + } + name := "review-duplicate-" + uuid.NewString()[:8] + if _, err := testPool.Exec(t.Context(), ` + INSERT INTO keypairs (user_id, name, public_key, fingerprint) + VALUES ($1, $2, $3, $4) + `, seededUserID, name, publicKey, fingerprint); err != nil { + t.Fatalf("seed existing keypair: %v", err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), "DELETE FROM keypairs WHERE user_id = $1 AND name = $2", seededUserID, name) + }) + body := fmt.Sprintf(`{"keypair":{"name":%q,"public_key":%q}}`, name, publicKey) + if code, response := doReq(r, http.MethodPost, "/v2.1/os-keypairs", body, hdr); code != http.StatusConflict { + t.Errorf("duplicate keypair import: code=%d body=%s; want 409", code, response) + } +} diff --git a/internal/nova/tenant_usage.go b/internal/nova/tenant_usage.go index b2c60354..2aeabed6 100644 --- a/internal/nova/tenant_usage.go +++ b/internal/nova/tenant_usage.go @@ -19,7 +19,7 @@ import ( // by grouping instances and has no owner row of its own, which is how it came to // have no ownership check at all. func (svc *Service) ListTenantUsage(c *gin.Context) { - if !requireAdmin(c, "os-simple-tenant-usage:index") { + if !common.RequireAdmin(c, "os-simple-tenant-usage:index") { return } diff --git a/internal/nova/volume_attachment.go b/internal/nova/volume_attachment.go index 53fcd3f4..df0a32a1 100644 --- a/internal/nova/volume_attachment.go +++ b/internal/nova/volume_attachment.go @@ -8,9 +8,8 @@ import ( "os" "time" - "github.com/cobaltcore-dev/lightstack/internal/database" - "github.com/cobaltcore-dev/lightstack/internal/common" + "github.com/cobaltcore-dev/lightstack/internal/database" "github.com/cobaltcore-dev/lightstack/pkg/hypervisor" "github.com/cobaltcore-dev/lightstack/pkg/storage" "github.com/gin-gonic/gin" @@ -67,6 +66,51 @@ type VolumeAttachment struct { AttachedAt time.Time `json:"attachedAt"` } +// novaAttachRefusal is a client-facing attach refusal (wrong volume state, +// already attached) carried out of the attach transaction so the handler can +// answer with the right status instead of a generic 500. Mirrors cinder's +// attachRefusal. +type novaAttachRefusal struct { + message string +} + +func (e *novaAttachRefusal) Error() string { return e.message } + +var errNovaVolumeNotFound = errors.New("volume not found") +var errNovaAttachmentNotFound = errors.New("volume attachment not found") +var errNovaVolumeStateChanged = errors.New("volume attachment state changed during rollback") + +// rollbackVolumeAttachment removes an attachment and restores the volume in a +// single transaction. A half rollback would leave an in-use volume with no +// attachment row, or an available volume whose attachment row still exists. +func rollbackVolumeAttachment(ctx context.Context, pool database.DB, attachmentID, volumeID, instanceID string) error { + cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second) + defer cancel() + return database.WithTx(cleanupCtx, pool, func(tx database.Tx) error { + tag, err := tx.Exec(cleanupCtx, + "DELETE FROM volume_attachments WHERE id = $1 AND volume_id = $2 AND instance_id = $3", + attachmentID, volumeID, instanceID) + if err != nil { + return err + } + if tag.RowsAffected() != 1 { + return errNovaAttachmentNotFound + } + tag, err = tx.Exec(cleanupCtx, ` + UPDATE volumes + SET attached_to_instance_id = NULL, status = 'available', updated_at = $1 + WHERE id = $2 AND attached_to_instance_id = $3 + `, time.Now(), volumeID, instanceID) + if err != nil { + return err + } + if tag.RowsAffected() != 1 { + return errNovaVolumeStateChanged + } + return nil + }) +} + // AttachVolumeRequest represents a volume attachment request type AttachVolumeRequest struct { VolumeAttachment struct { @@ -105,70 +149,82 @@ func (svc *Service) AttachVolume(c *gin.Context) { return } - // Verify volume exists and is available - var volumeStatus string - var attachedToInstance any - err = svc.pool.QueryRow(c.Request.Context(), - "SELECT status, attached_to_instance_id FROM volumes WHERE id = $1 AND project_id = $2", - req.VolumeAttachment.VolumeID, projectID, - ).Scan(&volumeStatus, &attachedToInstance) - - if errors.Is(err, database.ErrNoRows) { - common.SendError(c, common.NewNotFoundError("volume")) - return - } - - if volumeStatus != "available" { - common.SendError(c, common.NewBadRequestError(fmt.Sprintf("volume status is %s, must be available", volumeStatus))) - return - } - - if attachedToInstance != nil { - common.SendError(c, common.NewBadRequestError("volume is already attached to an instance")) - return - } - - // Auto-assign device if not provided - device := req.VolumeAttachment.Device - if device == "" { - device, err = svc.getNextAvailableDevice(c.Request.Context(), instanceID) - if err != nil { - log.Error().Err(err).Str("operation", "get_next_device").Msg("device assignment error") - common.SendError(c, common.NewInternalServerError("failed to assign device")) - return + // Verify volume exists and is available, create the attachment and flip + // the volume to in-use — all in one transaction with the volume row + // locked: the old check-then-act let the loser of + // a concurrent double-attach surface as a 500 from the UNIQUE constraint, + // and a failure between the INSERT and the UPDATE left the volume row + // inconsistent with its attachment. Mirrors cinder's attach CAS. + var device string + attachmentID := uuid.NewString() + var now time.Time + err = database.WithTx(c.Request.Context(), svc.pool, func(tx database.Tx) error { + // Different volumes attached to one server must reserve distinct devices. + // Lock the instance before the volume, and use this transaction for every + // query: acquiring another pool connection here can deadlock a full pool. + if err := tx.QueryRow(c.Request.Context(), + "SELECT COALESCE(libvirt_domain_id, '') FROM instances WHERE id = $1 FOR UPDATE", + instanceID).Scan(&libvirtDomainID); err != nil { + return err + } + var volumeStatus string + var attachedToInstance any + if err := tx.QueryRow(c.Request.Context(), + "SELECT status, attached_to_instance_id FROM volumes WHERE id = $1 AND project_id = $2 FOR UPDATE", + req.VolumeAttachment.VolumeID, projectID, + ).Scan(&volumeStatus, &attachedToInstance); err != nil { + if errors.Is(err, database.ErrNoRows) { + return errNovaVolumeNotFound + } + return err + } + if volumeStatus != "available" { + return &novaAttachRefusal{message: fmt.Sprintf("volume status is %s, must be available", volumeStatus)} + } + if attachedToInstance != nil { + // Upstream answers a re-attach with 400 (Invalid input / + // volume already attached), not 409 — round 2 of this review. + return &novaAttachRefusal{message: "volume is already attached to an instance"} } - } - - // Create attachment record - attachmentID := uuid.New().String() - now := time.Now() - - _, err = svc.pool.Exec(c.Request.Context(), ` - INSERT INTO volume_attachments (id, volume_id, instance_id, device, attached_at) - VALUES ($1, $2, $3, $4, $5) - `, attachmentID, req.VolumeAttachment.VolumeID, instanceID, device, now) - - if err != nil { - log.Error().Err(err).Str("operation", "create_volume_attachment").Msg("database error") - common.SendError(c, common.NewInternalServerError("failed to create volume attachment")) - return - } - // Update volume status - _, err = svc.pool.Exec(c.Request.Context(), ` - UPDATE volumes - SET attached_to_instance_id = $1, status = $2, updated_at = $3 - WHERE id = $4 - `, instanceID, "in-use", now, req.VolumeAttachment.VolumeID) + // Auto-assign device if not provided. The instance lock above serializes + // assignments across different volumes attached concurrently. + device = req.VolumeAttachment.Device + if device == "" { + device, err = getNextAvailableDevice(c.Request.Context(), tx, instanceID) + if err != nil { + return fmt.Errorf("device assignment: %w", err) + } + } + now = time.Now() + if _, err := tx.Exec(c.Request.Context(), ` + INSERT INTO volume_attachments (id, volume_id, instance_id, device, attached_at) + VALUES ($1, $2, $3, $4, $5) + `, attachmentID, req.VolumeAttachment.VolumeID, instanceID, device, now); err != nil { + return err + } + if _, err := tx.Exec(c.Request.Context(), ` + UPDATE volumes + SET attached_to_instance_id = $1, status = $2, updated_at = $3 + WHERE id = $4 + `, instanceID, "in-use", now, req.VolumeAttachment.VolumeID); err != nil { + return err + } + return nil + }) if err != nil { - // Rollback attachment (best-effort; the update error is the - // user-facing one) - if _, rbErr := svc.pool.Exec(c.Request.Context(), "DELETE FROM volume_attachments WHERE id = $1", attachmentID); rbErr != nil { - log.Warn().Err(rbErr).Str("attachment_id", attachmentID).Msg("rollback: failed to delete volume attachment after volume update failure") + var refusal *novaAttachRefusal + if errors.As(err, &refusal) { + common.SendError(c, common.NewBadRequestError(refusal.message)) + return + } + if errors.Is(err, errNovaVolumeNotFound) { + common.SendError(c, common.NewNotFoundError("volume")) + return } - log.Error().Err(err).Str("operation", "update_volume_status").Msg("database error") - common.SendError(c, common.NewInternalServerError("failed to update volume status")) + log.Error().Err(err).Str("operation", "attach_volume_tx").Msg("database error") + common.SendError(c, common.NewInternalServerError("failed to attach volume")) return } @@ -180,19 +236,21 @@ func (svc *Service) AttachVolume(c *gin.Context) { ctx, cancel := context.WithTimeout(c.Request.Context(), 10*time.Second) defer cancel() var rbdPool, rbdImage string - _ = svc.pool.QueryRow(ctx, + if err := svc.pool.QueryRow(ctx, "SELECT rbd_pool, rbd_image FROM volumes WHERE id = $1", - req.VolumeAttachment.VolumeID).Scan(&rbdPool, &rbdImage) + req.VolumeAttachment.VolumeID).Scan(&rbdPool, &rbdImage); err != nil { + log.Error().Err(err).Str("volume_id", req.VolumeAttachment.VolumeID).Msg("failed to read volume backing before attaching") + if rollbackErr := rollbackVolumeAttachment(c.Request.Context(), svc.pool, attachmentID, req.VolumeAttachment.VolumeID, instanceID); rollbackErr != nil { + log.Error().Err(rollbackErr).Str("attachment_id", attachmentID).Msg("failed to roll back volume attachment after backing lookup failure") + } + common.SendError(c, common.NewInternalServerError("failed to attach volume to the instance")) + return + } diskXML := volumeDiskXML(req.VolumeAttachment.VolumeID, device, rbdPool, rbdImage) if err := svc.vmManager.AttachDevice(ctx, libvirtDomainID, diskXML); err != nil { log.Error().Err(err).Str("volume_id", req.VolumeAttachment.VolumeID).Str("libvirt_domain", libvirtDomainID).Msg("failed to attach volume to VM; reverting") - if _, e := svc.pool.Exec(c.Request.Context(), "DELETE FROM volume_attachments WHERE id = $1", attachmentID); e != nil { - log.Warn().Err(e).Str("attachment_id", attachmentID).Msg("revert: delete attachment") - } - if _, e := svc.pool.Exec(c.Request.Context(), - "UPDATE volumes SET attached_to_instance_id = NULL, status = 'available', updated_at = $1 WHERE id = $2", - time.Now(), req.VolumeAttachment.VolumeID); e != nil { - log.Warn().Err(e).Str("volume_id", req.VolumeAttachment.VolumeID).Msg("revert: reset volume status") + if rollbackErr := rollbackVolumeAttachment(c.Request.Context(), svc.pool, attachmentID, req.VolumeAttachment.VolumeID, instanceID); rollbackErr != nil { + log.Error().Err(rollbackErr).Str("attachment_id", attachmentID).Msg("failed to roll back volume attachment after hypervisor failure") } common.SendError(c, common.NewInternalServerError("failed to attach volume to the instance")) return @@ -290,6 +348,11 @@ func (svc *Service) DetachVolume(c *gin.Context) { common.SendError(c, common.NewNotFoundError("volume attachment")) return } + if err != nil { + log.Error().Err(err).Str("volume_id", volumeID).Str("instance_id", instanceID).Msg("failed to look up volume attachment") + common.SendError(c, common.NewInternalServerError("failed to get volume attachment")) + return + } // Detach from hypervisor first if svc.vmManager != nil && libvirtDomainID != "" { @@ -303,40 +366,49 @@ func (svc *Service) DetachVolume(c *gin.Context) { volumeID, ).Scan(&rbdPool, &rbdImage) - if err == nil { - diskXML := volumeDiskXML(volumeID, device, rbdPool, rbdImage) - - if err := svc.vmManager.DetachDevice(ctx, libvirtDomainID, diskXML); err != nil { - log.Error().Err(err).Str("operation", "detach_disk_libvirt").Msg("libvirt error") - common.SendError(c, common.NewInternalServerError("failed to detach disk")) - return - } + if err != nil { + log.Error().Err(err).Str("volume_id", volumeID).Msg("failed to read volume backing before detaching") + common.SendError(c, common.NewInternalServerError("failed to detach disk")) + return } - } + diskXML := volumeDiskXML(volumeID, device, rbdPool, rbdImage) - // Delete attachment record - _, err = svc.pool.Exec(c.Request.Context(), - "DELETE FROM volume_attachments WHERE id = $1", - attachmentID, - ) - - if err != nil { - log.Error().Err(err).Str("operation", "delete_volume_attachment").Msg("database error") - common.SendError(c, common.NewInternalServerError("failed to delete volume attachment")) - return + if err := svc.vmManager.DetachDevice(ctx, libvirtDomainID, diskXML); err != nil { + log.Error().Err(err).Str("operation", "detach_disk_libvirt").Msg("libvirt error") + common.SendError(c, common.NewInternalServerError("failed to detach disk")) + return + } } - // Update volume status + // Delete the attachment and release the volume in one transaction. As two + // statements, a failure after the DELETE left the + // volume stranded in-use with no attachment row — unfixable through the + // API, since detach could no longer find the attachment to remove. now := time.Now() - _, err = svc.pool.Exec(c.Request.Context(), ` - UPDATE volumes - SET attached_to_instance_id = NULL, status = $1, updated_at = $2 - WHERE id = $3 - `, "available", now, volumeID) - + err = database.WithTx(c.Request.Context(), svc.pool, func(tx database.Tx) error { + tag, err := tx.Exec(c.Request.Context(), + "DELETE FROM volume_attachments WHERE id = $1", + attachmentID) + if err != nil { + return err + } + if tag.RowsAffected() == 0 { + return errNovaAttachmentNotFound + } + _, err = tx.Exec(c.Request.Context(), ` + UPDATE volumes + SET attached_to_instance_id = NULL, status = $1, updated_at = $2 + WHERE id = $3 + `, "available", now, volumeID) + return err + }) if err != nil { - log.Error().Err(err).Str("operation", "update_volume_status_detach").Msg("database error") - common.SendError(c, common.NewInternalServerError("failed to update volume status")) + if errors.Is(err, errNovaAttachmentNotFound) { + common.SendError(c, common.NewNotFoundError("volume attachment")) + return + } + log.Error().Err(err).Str("operation", "detach_volume_tx").Msg("database error") + common.SendError(c, common.NewInternalServerError("failed to detach volume")) return } @@ -442,9 +514,9 @@ func attachedVolumes(vols []gin.H) []gin.H { } // getNextAvailableDevice finds the next available device name (vdb, vdc, etc.) -func (svc *Service) getNextAvailableDevice(ctx context.Context, instanceID string) (string, error) { +func getNextAvailableDevice(ctx context.Context, tx database.Tx, instanceID string) (string, error) { // Query existing devices - rows, err := svc.pool.Query(ctx, + rows, err := tx.Query(ctx, "SELECT device FROM volume_attachments WHERE instance_id = $1", instanceID, ) @@ -457,8 +529,7 @@ func (svc *Service) getNextAvailableDevice(ctx context.Context, instanceID strin for rows.Next() { var device string if err := rows.Scan(&device); err != nil { - log.Warn().Err(err).Str("operation", "getNextAvailableDevice").Msg("failed to scan row") - continue + return "", fmt.Errorf("scan volume device: %w", err) } usedDevices[device] = true } diff --git a/internal/nova/volume_attachment_concurrency_test.go b/internal/nova/volume_attachment_concurrency_test.go new file mode 100644 index 00000000..cf8fcb2d --- /dev/null +++ b/internal/nova/volume_attachment_concurrency_test.go @@ -0,0 +1,99 @@ +package nova + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/cobaltcore-dev/lightstack/internal/database" + "github.com/cobaltcore-dev/lightstack/internal/database/dbtest" + "github.com/gin-gonic/gin" +) + +func seedAttachVolume(t *testing.T) string { + t.Helper() + var id string + if err := testPool.QueryRow(t.Context(), `INSERT INTO volumes (name, project_id, user_id, size_gb, status) + VALUES ('attach-review', $1, $2, 1, 'available') RETURNING id`, seededProjectID, seededUserID).Scan(&id); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), "DELETE FROM volume_attachments WHERE volume_id = $1", id) + _, _ = testPool.Exec(context.Background(), "DELETE FROM volumes WHERE id = $1", id) + }) + return id +} + +func attachWithPool(ctx context.Context, pool database.DB, instanceID, volumeID string) *httptest.ResponseRecorder { + w := httptest.NewRecorder() + c, _ := gin.CreateTestContext(w) + c.Request = httptest.NewRequest(http.MethodPost, "/", strings.NewReader( + fmt.Sprintf(`{"volumeAttachment":{"volumeId":%q}}`, volumeID))).WithContext(ctx) + c.Request.Header.Set("Content-Type", "application/json") + c.Params = gin.Params{{Key: "id", Value: instanceID}} + c.Set("project_id", seededProjectID) + c.Set("user_id", seededUserID) + c.Set("roles", []string{"member"}) + svc := &Service{pool: pool} + svc.AttachVolume(c) + return w +} + +func TestAttachVolumeWithOneConnection(t *testing.T) { + requireDB(t) + instanceID, volumeID := seedConsoleInstance(t, seededProjectID), seedAttachVolume(t) + poolConfig := *database.DefaultPoolConfig() + poolConfig.MaxConns = 1 + pool, err := database.Open(t.Context(), dbtest.URL(), database.Options{ + Pool: poolConfig, + }) + if err != nil { + t.Fatal(err) + } + defer pool.Close() + ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second) + defer cancel() + w := attachWithPool(ctx, pool, instanceID, volumeID) + if w.Code != http.StatusOK { + t.Fatalf("attach with one connection: %d %s", w.Code, w.Body.String()) + } +} + +func TestConcurrentVolumeAttachmentsUseDistinctDevices(t *testing.T) { + requireDB(t) + instanceID := seedConsoleInstance(t, seededProjectID) + const count = 8 + volumes := make([]string, count) + for i := range volumes { + volumes[i] = seedAttachVolume(t) + } + start := make(chan struct{}) + results := make(chan *httptest.ResponseRecorder, count) + ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second) + defer cancel() + for _, id := range volumes { + go func() { + <-start + results <- attachWithPool(ctx, testPool, instanceID, id) + }() + } + close(start) + for range count { + w := <-results + if w.Code != http.StatusOK { + t.Errorf("concurrent attach: %d %s", w.Code, w.Body.String()) + } + } + var total, distinct int + if err := testPool.QueryRow(t.Context(), `SELECT count(*), count(DISTINCT device) + FROM volume_attachments WHERE instance_id = $1`, instanceID).Scan(&total, &distinct); err != nil { + t.Fatal(err) + } + if total != count || distinct != count { + t.Errorf("attachments=%d distinct devices=%d; want %d each", total, distinct, count) + } +} diff --git a/internal/nova/volume_attachment_refusal_test.go b/internal/nova/volume_attachment_refusal_test.go new file mode 100644 index 00000000..2bd83ae6 --- /dev/null +++ b/internal/nova/volume_attachment_refusal_test.go @@ -0,0 +1,71 @@ +package nova + +import ( + "context" + "encoding/json" + "net/http" + "testing" + + "github.com/google/uuid" +) + +func TestVolumeAttachAndDetachRefusals(t *testing.T) { + requireDB(t) + r := newNovaTestRouter() + instanceID := seedConsoleInstance(t, seededProjectID) + inUseID, alreadyAttachedID := uuid.NewString(), uuid.NewString() + for _, volume := range []struct { + id, name, status string + attached bool + }{ + {inUseID, "review-in-use-" + inUseID[:8], "in-use", false}, + {alreadyAttachedID, "review-attached-" + alreadyAttachedID[:8], "available", true}, + } { + query := "INSERT INTO volumes (id, name, project_id, user_id, size_gb, status) VALUES ($1, $2, $3, $4, 1, $5)" + args := []any{volume.id, volume.name, seededProjectID, seededUserID, volume.status} + if volume.attached { + query = "INSERT INTO volumes (id, name, project_id, user_id, size_gb, status, attached_to_instance_id) VALUES ($1, $2, $3, $4, 1, $5, $6)" + args = append(args, instanceID) + } + if _, err := testPool.Exec(t.Context(), query, args...); err != nil { + t.Fatalf("seed volume %s: %v", volume.id, err) + } + } + t.Cleanup(func() { + _, _ = testPool.Exec(context.Background(), "DELETE FROM volume_attachments WHERE volume_id IN ($1, $2)", inUseID, alreadyAttachedID) + _, _ = testPool.Exec(context.Background(), "DELETE FROM volumes WHERE id IN ($1, $2)", inUseID, alreadyAttachedID) + }) + + headers := testAuthHdr(seededProjectID, seededUserID) + attach := func(volumeID string) (int, string) { + body, err := json.Marshal(map[string]any{"volumeAttachment": map[string]string{"volumeId": volumeID}}) + if err != nil { + t.Fatalf("marshal attach request: %v", err) + } + return doReq(r, http.MethodPost, "/v2.1/servers/"+instanceID+"/os-volume_attachments", string(body), headers) + } + for _, tc := range []struct { + name, volumeID string + want int + }{ + {"unknown volume", uuid.NewString(), http.StatusNotFound}, + {"wrong status", inUseID, http.StatusBadRequest}, + {"already attached", alreadyAttachedID, http.StatusBadRequest}, + } { + t.Run(tc.name, func(t *testing.T) { + if code, body := attach(tc.volumeID); code != tc.want { + t.Errorf("attach %s: code=%d body=%s; want %d", tc.name, code, body, tc.want) + } + }) + } + if code, body := doReq(r, http.MethodDelete, "/v2.1/servers/"+instanceID+"/os-volume_attachments/"+uuid.NewString(), "", headers); code != http.StatusNotFound { + t.Errorf("detach without an attachment: code=%d body=%s; want 404", code, body) + } + var count int + if err := testPool.QueryRow(t.Context(), "SELECT count(*) FROM volume_attachments WHERE volume_id IN ($1, $2)", inUseID, alreadyAttachedID).Scan(&count); err != nil { + t.Fatal(err) + } + if count != 0 { + t.Errorf("refused requests created %d volume attachments", count) + } +} diff --git a/pkg/storage/image_store.go b/pkg/storage/image_store.go index fc57a43e..dc65bc12 100644 --- a/pkg/storage/image_store.go +++ b/pkg/storage/image_store.go @@ -344,14 +344,18 @@ func (s *ImageStore) uploadImageStub(imageID string, reader io.Reader) (int64, e // this raw file. The upload streams to a temp file, then qemu-img converts it // to raw (or renames it if already raw / qemu-img is unavailable). func (s *ImageStore) uploadImageLocal(ctx context.Context, imageID string, reader io.Reader) (int64, error) { - ctx, cancel := context.WithTimeout(ctx, s.timeout) - defer cancel() - + // No deadline here, deliberately: s.timeout is for metadata ops + // (head/delete, see its field comment); the body transfer streams to + // completion under the CALLER's context — the request's, cancelled on + // client disconnect — exactly as the S3 path's connect-only timeout does + // not bound transfers (FR-014). The 5 s cap here used to fire long before + // glance's 10 GiB upload cap could, turning oversized uploads into 503s + // instead of 413s (round 3 of this review). imagePath := filepath.Join(s.localPath, imageFileName(imageID)) tmpPath := imagePath + ".upload" - // Stream the upload to a temp file. Wrap the source so the timeout fires — - // os.File doesn't natively respect context. + // Stream the upload to a temp file. Wrap the source so cancellation fires + // — os.File doesn't natively respect context. tmp, err := os.Create(tmpPath) if err != nil { return 0, fmt.Errorf("failed to create image file: %w", err) @@ -605,6 +609,30 @@ func (s *ImageStore) uploadLocalToS3(ctx context.Context, imageID string) error return nil } +// StoredChecksum returns the MD5 hex digest of the image's stored local bytes. +// Normalizing backends (local, local,rbd, local,s3) rewrite the upload — a +// qcow2 becomes a raw base — so the checksum Glance advertises, downloads +// serve and EnsureLocalBase verifies against must be taken from the stored +// object, not from the client's upload stream: the stream hash made +// boot-from-image fail on any node without the local base, because +// MD5(raw replica) never equals MD5(original qcow2). For S3-only and +// stub modes the stored bytes are the upload bytes, so the stream hash the +// caller already computed is the stored checksum and this method is not +// consulted. A future RBD-only backend should hash its own object instead. +func (s *ImageStore) StoredChecksum(ctx context.Context, imageID string) (string, error) { + imagePath := filepath.Join(s.localPath, imageFileName(imageID)) + f, err := os.Open(imagePath) + if err != nil { + return "", err + } + defer f.Close() + h := md5.New() // #nosec G401 -- Glance checksum is MD5 (compatibility), not a security control + if _, err := io.Copy(h, &ctxReader{ctx: ctx, r: f}); err != nil { + return "", err + } + return hex.EncodeToString(h.Sum(nil)), nil +} + // DownloadImage downloads an image from storage func (s *ImageStore) DownloadImage(ctx context.Context, imageID string, writer io.Writer) error { switch s.mode { diff --git a/pkg/storage/image_store_test.go b/pkg/storage/image_store_test.go index cd2dd9f4..0ac58fec 100644 --- a/pkg/storage/image_store_test.go +++ b/pkg/storage/image_store_test.go @@ -274,3 +274,62 @@ func makeQcow2(t *testing.T, dir string) string { _ = os.Remove(raw) return qcow } + +// TestStoredChecksumMatchesStoredBytesNotUploadBytes pins the stored-object +// checksum behavior: normalizing backends rewrite the upload (qcow2 → raw base), so the +// checksum recorded for an image must describe the STORED bytes — downloads +// serve them and EnsureLocalBase verifies a lazily fetched base against the +// recorded value on every boot. Hashing the upload stream instead made +// MD5(raw replica) != MD5(original qcow2), so boot-from-image failed on any +// node without the local base. +func TestStoredChecksumMatchesStoredBytesNotUploadBytes(t *testing.T) { + if _, err := exec.LookPath("qemu-img"); err != nil { + t.Skip("qemu-img not available") + } + s, err := NewImageStore("local", "", "", S3Options{}) + if err != nil { + t.Fatalf("NewImageStore: %v", err) + } + s.localPath = t.TempDir() // same-package override: keep the test off /var/lib + + // A real qcow2: not raw bytes, so normalizeToRaw must convert it. + qcow2Path := filepath.Join(t.TempDir(), "disk.qcow2") + if out, err := exec.Command("qemu-img", "create", "-f", "qcow2", qcow2Path, "1M").CombinedOutput(); err != nil { + t.Fatalf("qemu-img create: %v: %s", err, out) + } + qcow2Bytes, err := os.ReadFile(qcow2Path) + if err != nil { + t.Fatal(err) + } + + imageID := "11111111-2222-3333-4444-555555555555" + size, normalized, err := s.UploadImage(context.Background(), imageID, bytes.NewReader(qcow2Bytes)) + if err != nil { + t.Fatalf("UploadImage: %v", err) + } + if !normalized { + t.Fatal("local mode must report normalization") + } + // The stored size is the raw virtual size, not the qcow2 file size. + if size != 1<<20 { + t.Errorf("stored size = %d; want the raw size %d", size, 1<<20) + } + + streamHash := fmt.Sprintf("%x", md5.Sum(qcow2Bytes)) + stored, err := s.StoredChecksum(context.Background(), imageID) + if err != nil { + t.Fatalf("StoredChecksum: %v", err) + } + storedBytes, err := os.ReadFile(filepath.Join(s.localPath, imageFileName(imageID))) + if err != nil { + t.Fatalf("read stored image: %v", err) + } + wantHash := fmt.Sprintf("%x", md5.Sum(storedBytes)) + + if stored != wantHash { + t.Errorf("StoredChecksum = %s; want MD5 of the stored bytes %s", stored, wantHash) + } + if stored == streamHash { + t.Error("StoredChecksum equals the upload-stream hash; the qcow2 bytes were normalized, so the hashes must differ — this is the N3 regression") + } +}