fix: retry provisioning and add IPv6, bucket policy, hostUsers

[loverustfs]

Type of Change

• New Feature
• Bug Fix
• Documentation
• Performance Improvement
• Test/CI
• Refactor
• Other:

Related Issues

• Provisioning doesn’t retry tenant S3 admin API failures — policies/users/buckets not created #232 (valid — fixed)
• ClusterRole permission issue with Admission plugin #206 (partially already fixed; remaining hostUsers work — fixed)
• IPV6 support #182 (valid — fixed)
• Support configuring Bucket Access Policy via Tenant CRD #200 (valid — fixed)
• exited 1: local erasure endpoints must use distinct physical disks #162 (invalid for this operator — documented, no disk-layout change)
• Create a data plane operator for RustFS #218 (out of scope — documented; Tenant provisioning already covers policies/users/buckets)

Summary of Changes

I triaged every open issue on this repository and only implemented the ones that are real operator bugs or in-scope CRD features.

#232 Provisioning retry. Transient RustFS admin/S3 and Kubernetes failures (network errors, 408/429/425/5xx, TLS-not-ready) now mark items Pending and requeue after 10s instead of writing Failed and waiting for an unrelated spec change. Permanent 4xx and configuration errors still fail and await_change(). Users that reference a still-pending spec policy stay Pending rather than UserPolicyNotFound. Adversarial review tightened “resource absent” detection so a 5xx/429 proxy page containing “Not Found” is not treated as a missing user, policy, or bucket policy.

#182 IPv6. Operator/STS/Console sockets bind :: then fall back to 0.0.0.0, with OPERATOR_BIND_ADDRESS / CONSOLE_BIND_ADDRESS overrides. Tenant spec.network sets Service ipFamilyPolicy/ipFamilies and switches RUSTFS_ADDRESS to [::]:port when IPv6 or dual-stack is requested. Helm network.* and *.bindAddress cover chart-managed Services and process binds.

#200 Bucket access policy. spec.buckets[].anonymous canned policies (Private|Download|Upload|Public) and policy.configMapKeyRef custom JSON. CEL plus reconcile reject mixing a custom policy with non-Private anonymous access. ConfigMap references are indexed so Tenant reconcile wakes on policy ConfigMap changes.

#206 OpenShift restricted-v3. ClusterRole finalizer RBAC was already present. This change adds Tenant spec.hostUsers and sets hostUsers: false on chart Deployments when openshift.enabled=true, plus the OpenShift empty security-context default for generated Tenant pods.

#162 / #218. Not operator bugs. Distinct physical disks are a RustFS node topology requirement; a separate data-plane operator is out of scope because this controller already provisions policies, users, and buckets onto the Tenant cluster. User-guide notes added; issues already labeled invalid.

Checklist

• I have read and followed the CONTRIBUTING.md guidelines
• Passed make pre-commit (fmt-check + clippy + test + console-lint + console-fmt-check)
• Added/updated necessary tests
• Documentation updated (if needed)
• CHANGELOG.md updated under [Unreleased] (if user-visible change)
• CI/CD passed (if applicable)

Impact

• Breaking change (CRD/API compatibility)
• Requires doc/config/deployment update
• Other impact:

CRD additive fields: spec.hostUsers, spec.network, spec.buckets[].anonymous, spec.buckets[].policy. Existing Tenants keep IPv4 listen addresses and private buckets. Helm adds optional network and bind-address values.

Verification

make pre-commit

Local result: pre-commit: all checks passed.

GitHub CI on 750602a: Test and Lint succeeded; Skip Duplicate Actions succeeded; Docker Build skipped (workflow condition).

Targeted coverage added for:

• create-bucket 503 requeues vs 400 fails
• finish() prefers Retry over existing Failed / pending items
• user stays Pending when the referenced policy is Pending
• canned anonymous Download/Upload/Public documents
• GET bucket policy NoSuchBucketPolicy as absent; 503 + “Not Found” body stays transient
• kube API 408/429/5xx transient vs 404/400/403/409 permanent
• IPv6 / dual-stack listen addresses and Service IP families
• OpenShift chart hostUsers: false vs default omit

Additional Notes

Adversarial pass after the first implementation found that body_mentions_not_found was applied to every non-success status. A 503 HTML page containing “Not Found” would have been treated as a missing bucket policy or user. Absence is now limited to 404 and non-retryable 4xx semantic codes.

---

Thank you for your contribution! Please ensure your PR follows the community standards (CODE_OF_CONDUCT.md) and sign the CLA if this is your first contribution.