docs: add credential and API key rotation guide

aigw/product/enterprise-offering/org-management/api-key-rotation.mdx

@@ -4,7 +4,7 @@ description: "Periodically replace API key secrets without downtime using manual
 ---
 
 <Note>
-**Availability:** API key rotation is available on Prisma AIRS AI Gateway Cloud.
+**Availability:** API key rotation is available on **all accounts** on Prisma AIRS AI Gateway Cloud.
 
 For hybrid deployments, it requires Gateway `v2.5.0+`.
 </Note>

aigw/support/rotate-credentials.mdx

@@ -0,0 +1,252 @@
+---
+title: "Rotate Credentials and API Keys"
+description: "Step-by-step guide to rotating every credential stored in or issued by the AI Gateway: API keys, provider credentials, MCP credentials, SSO and SCIM secrets, and self-hosted gateway secrets."
+---
+
+This guide lists every credential that the AI Gateway stores or issues for an organisation, and how to rotate each one. Work through the sections in order.
+
+<Warning>
+The AI Gateway does not support grace periods when you replace credentials stored in integrations, providers, MCP servers, or other settings. Saving a new value does not take effect on every gateway at once. Hybrid and self-hosted gateways clear cached keys, integrations, and configs on a sync that runs about once a minute, and values read from a vault are cached for up to 5 minutes. For each credential: create the new value at the source, update the AI Gateway, wait for the change to propagate, confirm requests succeed with the new value on every gateway, then revoke the old value at the source.
+</Warning>
+
+## Credentials at a glance
+
+| # | Credential | Where it lives | Rotate at |
+|---|---|---|---|
+| 1 | [AI Gateway API keys](#1-ai-gateway-api-keys) (admin, workspace service, workspace user) and [Admin API service accounts](#admin-api-service-accounts) | AI Gateway, Strata Cloud Manager | AI Gateway, Strata Cloud Manager |
+| 2 | [AI provider credentials](#2-ai-provider-credentials) (integrations, providers, virtual keys) | AI Gateway | Your AI provider, then the AI Gateway |
+| 3 | [MCP credentials](#3-mcp-credentials) | AI Gateway | The MCP server or IdP, then the AI Gateway |
+| 4 | [Secret references](#4-secret-references-and-vault-credentials) | Your vault | Your vault |
+| 5 | [SSO, SCIM, and JWT settings](#5-sso-scim-and-jwt-settings) | AI Gateway | Your IdP, then the AI Gateway |
+| 6 | [Credentials embedded in configs and guardrails](#6-credentials-embedded-in-configs-and-guardrails) | AI Gateway | The upstream service, then the AI Gateway |
+| 7 | [Observability and log export credentials](#7-observability-and-log-export-credentials) | AI Gateway or your gateway | Your destination, then the AI Gateway |
+| 8 | [Self-hosted gateway secrets](#8-self-hosted-gateway-secrets) | Your deployment | Your deployment |
+
+## Before you start
+
+- **Use a rotation account you trust.** Sign in to the AI Gateway management console with SSO or your own credentials. Don't depend on an existing API key to run the rotation.
+- **Plan for downtime.** Replacing a credential breaks any application still using the old value. Roll out new values to your applications before revoking the old ones.
+- **Keep a list.** Record each credential you rotate, so you can confirm none was missed. Use the [checklist](#rotation-checklist) at the end of this page.
+- **Know your permissions.** Rotating organisation-level resources requires an organisation owner or admin. Workspace-level resources require a workspace admin or manager.
+
+## 1. AI Gateway API keys
+
+The AI Gateway issues three kinds of API keys. They authenticate requests to the inference API and are **not** accepted on the Admin API. See [API Keys (AuthN and AuthZ)](/aigw/product/enterprise-offering/org-management/api-keys-authn-and-authz) for how they differ.
+
+- **Admin API keys:** organisation-level keys that work across workspaces
+- **Workspace service API keys:** workspace-level keys for backends and automation
+- **Workspace user API keys:** workspace-level keys owned by individual users
+
+Admin API requests use a different credential: a Strata Cloud Manager service account. Rotate those too. See [Admin API service accounts](#admin-api-service-accounts).
+
+### Choose a rotation method
+
+| Method | Key ID | Old secret | Use when |
+|---|---|---|---|
+| **Create a new key, then delete the old key** | Changes | Revoked when you delete the key. Gateways can keep accepting it until their cache refreshes (about a minute). | The old key must stop working as soon as possible |
+| **[Rotate the key](/aigw/product/enterprise-offering/org-management/api-key-rotation)** | Unchanged | Stays valid for the transition period (minimum 30 minutes) | Usage history, budgets, and attribution must stay attached to the same key |
+
+<Note>
+The rotate endpoint always keeps the previous secret valid for the transition period, with a minimum of 30 minutes. To end access as soon as possible, create a new key and delete the old one. Gateways cache API keys and clear changed ones on a sync that runs about once a minute, so confirm the old key returns `401` on every gateway, including self-hosted ones, before treating it as revoked.
+</Note>
+
+### Rotate with the console
+
+<Steps>
+<Step title="Open API Keys">
+Go to **API Keys**.
+</Step>
+<Step title="Create a replacement key">
+Click **Create**. Select the same type, workspace, scopes, and rate or budget limits as the key it replaces. Also copy its default config and its **Config Override** setting. Without them the new key loses the routing and guardrails attached to the old one, or lets requests override the config.
+</Step>
+<Step title="Deploy the new key">
+Update your applications, CI pipelines, secret stores, and `Authorization: Bearer` headers with the new key.
+</Step>
+<Step title="Delete the old key">
+After traffic moves to the new key, delete the old key from the list.
+</Step>
+</Steps>
+
+### Rotate with the API
+
+These endpoints are on the Admin API and require a Strata Cloud Manager access token, not an AI Gateway API key. See [Admin API authentication](/aigw/api-reference/admin-api/authentication).
+
+1. [Create an API key](/aigw/api-reference/api-keys/post-api-keys-by-sub-type) with `POST /v2/api-keys/{sub-type}`. Set the same default config and config-override setting as the old key.
+2. Deploy the new key.
+3. Delete the old key with `DELETE /v2/api-keys/{id}`.
+
+To keep the same key ID instead, call [`POST /v2/api-keys/{id}/rotate`](/aigw/api-reference/api-keys/post-api-keys-by-id-rotate). The response contains the new secret and the time the previous secret expires.
+
+### Order of rotation
+
+1. Rotate **admin API keys** first. They have the broadest scope on the inference path.
+2. Rotate **workspace service API keys** next. They can manage other workspace keys.
+3. Rotate **workspace user API keys** last. Ask each user to replace theirs, or rotate them as an admin.
+
+List existing keys with `GET /v2/api-keys` to find every key to replace. Delete any key you don't recognise or no longer use.
+
+### Admin API service accounts
+
+Admin API calls are authorised with a short-lived token issued to a service account in Strata Cloud Manager, using the service account's Client ID and Client Secret. Rotate these as well:
+
+1. In Strata Cloud Manager, go to **System Settings > Identity & Access** and issue a new credential for each service account used with the Admin API. If you can't issue a new credential, create a replacement service account in the same tenant with the same role assignments.
+2. Update every job, pipeline, and secret store that holds the old Client Secret.
+3. Delete the old service account or credential.
+
+Access tokens expire after 15 minutes, so tokens issued with the old credential stop working within that time. See [Admin API authentication](/aigw/api-reference/admin-api/authentication).
+
+## 2. AI provider credentials
+
+Integrations, providers, and virtual keys hold credentials for your AI providers: API keys, cloud access keys, service account JSON, and similar. Rotating the key only in the AI Gateway does not invalidate the old one. **Rotate the credential at the provider.**
+
+For each integration:
+
+<Steps>
+<Step title="Create a new credential at the provider">
+Generate a new API key, access key, or service account key in your provider's console.
+</Step>
+<Step title="Update the credential in the AI Gateway">
+In the management console, open **Model Catalog > Integrations**, select the integration, and replace the stored credential. To use the API, call `PUT /integrations/{slug}` or `PUT /providers/{slug}`.
+</Step>
+<Step title="Test a request">
+Send a test request through a model on the integration to confirm the new credential works.
+</Step>
+<Step title="Revoke the old credential at the provider">
+Delete or disable the old key in your provider's console.
+</Step>
+</Steps>
+
+### Credentials by provider
+
+| Provider | Credential to rotate |
+|---|---|
+| OpenAI | API key. Rotate the optional organisation and project IDs only if your provider treats them as secret. |
+| Anthropic, Gemini, and other API-key providers | API key |
+| AWS Bedrock | AWS access key ID and secret access key. For assumed-role setups, also review the role's trust policy and rotate the external ID if one is set. See [Connect Bedrock with Amazon Assumed Role](/aigw/product/model-catalog/connect-bedrock-with-amazon-assumed-role). |
+| Azure OpenAI | API key, or the Entra ID client secret for service principal authentication |
+| Google Vertex AI | Service account key. Create a new key, update the AI Gateway, then delete the old key. |
+| Custom or self-hosted model endpoints | Any bearer token or custom header value you configured |
+
+<Note>
+Integrations that use [workload identity federation](/aigw/product/model-catalog/workload-identity-federation) do not store a static provider key, but they do use an identity provider client secret. Create a replacement client secret at the IdP, update it in the integration (or the vault entry it references), send a test request, then revoke the old secret. Also review the trust relationship and allowed subjects on the cloud side.
+</Note>
+
+### Virtual keys (deprecated)
+
+Existing virtual keys hold provider credentials too. Rotate them the same way using the management console. Alternatively, migrate to [integrations and providers](/aigw/product/model-catalog/integrations) and delete the virtual key.
+
+### Credentials used in requests
+
+If your applications send provider credentials per request, such as an `Authorization` header or `x-portkey-*` provider headers, rotate those in your application secret stores as well.
+
+## 3. MCP credentials
+
+[MCP integrations and servers](/aigw/product/mcp-gateway/mcp-registry) store credentials that the AI Gateway uses to reach upstream MCP servers. Rotate each one at the source, then update the AI Gateway.
+
+| Authentication type | What to rotate | Steps |
+|---|---|---|
+| **Static headers** ([custom auth](/aigw/product/mcp-gateway/authentication/custom-auth)) | Bearer tokens, API keys, or custom header values | Generate new credentials on the MCP server. Update the headers on the MCP integration or server in the AI Gateway. Revoke the old credentials. |
+| **Client credentials** | OAuth client secret | Create a new secret at your IdP. Update **Client Secret** in the integration, or in the vault entry it references. Wait up to 5 minutes for vault-cached values to refresh. Revoke the old secret. |
+| **OAuth Auto** ([OAuth](/aigw/product/mcp-gateway/authentication/oauth)) | OAuth client secret, if you supplied your own OAuth app. Per-user access tokens. | Create a new client secret in the upstream OAuth app and update the integration. To force users to re-authorise, remove their stored connections with `DELETE /mcp-servers/{mcpServerId}/connections`. |
+| **External OAuth or JWT validation** | Introspection client secret, if configured | Rotate the introspection credentials at your IdP and update the integration. JWKS URIs and public keys are not secret. |
+
+Also rotate any tokens the MCP server itself holds for downstream systems. Those live outside the AI Gateway.
+
+Use `GET /mcp-servers` and `GET /mcp-integrations` to list every server and integration that needs review.
+
+## 4. Secret references and vault credentials
+
+[Secret references](/aigw/product/enterprise-offering/secret-references) point to secrets in AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault. The AI Gateway stores the reference configuration, including the auth config it uses to reach your vault. It does not store the secret value itself.
+
+Rotate in this order:
+
+1. **Vault access credentials.** Rotate the credentials in each secret reference's `auth_config` (for example an IAM access key, an Azure client secret, or a HashiCorp Vault token or AppRole secret). Update the reference with [`PUT /secret-references/{secretReferenceId}`](/aigw/api-reference/secret-references/update-secret-reference) on `https://api.apps.paloaltonetworks.com/ai_gw/admin/v2`. Updates to `auth_config` are merged, so send only the changed fields.
+2. **The secrets themselves.** Rotate the secret values in your vault. Gateways cache fetched values for 5 minutes, so the new value takes effect within that window with no change in the AI Gateway.
+3. **Revoke the old vault credentials.**
+
+## 5. SSO, SCIM, and JWT settings
+
+<AccordionGroup>
+<Accordion title="SSO (OIDC and SAML)">
+The OIDC client secret is stored in the AI Gateway. Create a new client secret at your IdP, then update it under **Admin Settings > Authentication Settings > OIDC** (or **SAML**). Revoke the old secret at the IdP. If the OIDC app uses `private_key_jwt`, there is no client secret to rotate. The signing credential is the private key set in `jwks` (or the `OIDC_CLIENT_JWKS` environment variable on self-hosted single-organisation deployments). Generate a new key pair, register the new public key at the IdP alongside the old one, replace the private `jwks` in the AI Gateway SSO settings, confirm sign-in works, then remove the old public key from the IdP. For SAML, rotate the IdP signing certificate if your policies require it, and upload the new metadata. See [SSO](/aigw/product/enterprise-offering/org-management/sso).
+</Accordion>
+<Accordion title="SCIM provisioning token">
+Generate a new SCIM token under **Admin Settings > Authentication Settings > SCIM Provisioning**. Update the bearer token in your IdP's provisioning configuration (for example [Okta](/aigw/product/enterprise-offering/org-management/scim/okta) or [Azure AD](/aigw/product/enterprise-offering/org-management/scim/azure-ad)). Confirm provisioning still works, then revoke the old token. See [SCIM](/aigw/product/enterprise-offering/org-management/scim/scim).
+</Accordion>
+<Accordion title="JWT authentication">
+The AI Gateway validates JWTs using your IdP's public keys, which are not secret. Rotate signing keys at your IdP if you suspect they were exposed, and confirm the JWKS configuration in the AI Gateway is current. See [JWT](/aigw/product/enterprise-offering/org-management/jwt).
+</Accordion>
+<Accordion title="Active sessions and users">
+Ask users to sign out and back in. Review [organisation users and pending invites](/aigw/product/enterprise-offering/org-management/organizations), and remove any user or invite you don't recognise.
+</Accordion>
+</AccordionGroup>
+
+## 6. Credentials embedded in configs and guardrails
+
+Credentials sometimes end up inside saved resources instead of integrations. Review each of the following for static secrets:
+
+- **[Configs](/aigw/product/ai-gateway/configs):** provider keys, `Authorization` headers, or custom headers inside targets or `override_params`
+- **[Guardrails](/aigw/product/guardrails):** API keys or headers for third-party guardrail providers, and webhook URLs, headers, or signing secrets for custom guardrails
+- **Custom hosts and custom headers:** bearer tokens for private endpoints
+
+For each secret found, rotate it at the upstream service, update the saved resource, and revoke the old value. Prefer [secret references](/aigw/product/enterprise-offering/secret-references) over inline values so future rotation happens in your vault.
+
+## 7. Observability and log export credentials
+
+Rotate any credential the AI Gateway uses to write to your systems:
+
+- **OpenTelemetry exports:** authentication headers or tokens for your OTel collector. See [OpenTelemetry](/aigw/product/enterprise-offering/otel/otel).
+- **Log storage for self-hosted gateways:** object storage access keys, HMAC secrets, or service account keys.
+- **Third-party observability integrations** that you configured with a token.
+
+Create the new credential at the destination, update the AI Gateway configuration, confirm data is arriving, then revoke the old credential.
+
+## 8. Self-hosted gateway secrets
+
+For [hybrid deployments](/aigw/self-hosting/hybrid-deployments/architecture), rotate every secret in your deployment configuration:
+
+- **`PORTKEY_CLIENT_AUTH`**: the credential your gateway uses to sync with the AI Gateway control plane. Contact Palo Alto Networks support to have it reissued, then update your deployment and restart the gateway.
+- **Log store credentials:** for example `LOG_STORE_ACCESS_KEY` and `LOG_STORE_SECRET_KEY` (S3, GCS HMAC, or Azure storage).
+- **Cache and database credentials:** Redis, and any database your deployment uses.
+- **Provider keys and other values in your environment files, Helm values, or Kubernetes secrets.**
+- **Test keys:** replace the `PORTKEY_API_KEY` and `OPENAI_API_KEY` values from any smoke-test scripts.
+
+Redeploy the gateway after updating secrets and confirm it registers and serves requests.
+
+## After rotating
+
+1. **Confirm the old credentials fail.** Wait at least a few minutes for gateway sync and caches to refresh, then send a request with each old AI Gateway API key through every gateway, including self-hosted ones. Expect a `401`.
+2. **Test end-to-end.** Send a request through every integration, MCP server, and key path your applications use.
+3. **Review audit logs.** Check [audit logs](/aigw/product/enterprise-offering/audit-logs) under **Admin Settings > Audit Logs** for unexpected creation of API keys, integrations, users, or configuration changes.
+4. **Review request logs and provider usage.** Check request logs and your AI providers' usage dashboards for requests you didn't make.
+5. **Review users and workspaces.** Remove any user, workspace member, or workspace you don't recognise.
+
+## Rotation checklist
+
+Copy this list to track progress.
+
+- [ ] Admin API keys replaced; old keys deleted
+- [ ] Admin API service account credentials replaced; old credentials deleted
+- [ ] Workspace service API keys replaced; old keys deleted
+- [ ] Workspace user API keys replaced; old keys deleted
+- [ ] Application secret stores updated with new AI Gateway keys
+- [ ] Every integration and provider credential rotated at the provider, updated in the AI Gateway, and old credentials revoked
+- [ ] Virtual key credentials rotated, or virtual keys migrated and deleted
+- [ ] MCP integration and server credentials rotated; user connections reset where needed
+- [ ] Secret reference vault credentials rotated; secret values rotated
+- [ ] SSO client secret (or `private_key_jwt` key pair) rotated
+- [ ] SCIM token regenerated; IdP updated
+- [ ] Configs, guardrails, and custom hosts reviewed for embedded secrets
+- [ ] OTel and log export credentials rotated
+- [ ] Self-hosted gateway secrets rotated; gateway redeployed
+- [ ] Old AI Gateway API keys confirmed to return `401`
+- [ ] Audit logs and provider usage reviewed
+
+## Related
+
+<CardGroup cols={2}>
+  <Card title="API Key Rotation" icon="rotate" href="/aigw/product/enterprise-offering/org-management/api-key-rotation" />
+  <Card title="API Keys (AuthN and AuthZ)" icon="key" href="/aigw/product/enterprise-offering/org-management/api-keys-authn-and-authz" />
+  <Card title="Secret References" icon="vault" href="/aigw/product/enterprise-offering/secret-references" />
+  <Card title="Audit Logs" icon="clipboard-list" href="/aigw/product/enterprise-offering/audit-logs" />
+</CardGroup>