diff --git a/aigw/api-reference/inference-api/config-object.mdx b/aigw/api-reference/inference-api/config-object.mdx index 3c280e2ab..90ecd7e13 100644 --- a/aigw/api-reference/inference-api/config-object.mdx +++ b/aigw/api-reference/inference-api/config-object.mdx @@ -668,6 +668,7 @@ You can find more examples of schemas [below](/aigw/api-reference/inference-api/ | `override_params` | Pass model name and other hyper parameters | object | No | "model", "temperature", "frequency\_penalty", "logit\_bias", "logprobs", "top\_logprobs", "max\_tokens", "n", "presence\_penalty", "response\_format", "seed", "stop", "top_p", etc. | Pass everything that's typically part of the payload | | `default_params` | Inject parameters into the request body only when not already present (respects client-sent values) | object | No | Any payload field | See [Default and drop params](/aigw/product/ai-gateway/configs#default-and-drop-params) | | `drop_params` | Remove parameters from the final request body using bracket-notation paths (nested keys, array indices, and `[*]` wildcards) | array of strings | No | Any payload field path | See [Default and drop params](/aigw/product/ai-gateway/configs#default-and-drop-params) | +| `auto_caching` | Add Anthropic's top-level `cache_control: {type: "ephemeral"}` to requests that don't already set one | boolean | No | `true`, `false` | Inherited by nested targets. See [Automatic Caching](/aigw/integrations/llms/anthropic/prompt-caching#automatic-caching) | ### Strategy Object Details diff --git a/aigw/integrations/libraries/claude-code-bedrock.mdx b/aigw/integrations/libraries/claude-code-bedrock.mdx index 7d2be14f2..8d3cbc20c 100644 --- a/aigw/integrations/libraries/claude-code-bedrock.mdx +++ b/aigw/integrations/libraries/claude-code-bedrock.mdx @@ -27,6 +27,10 @@ Enter your AWS credentials: Create a slug like `bedrock-prod`. + +**IAM permissions:** the credentials (or assumed role) you use here need `bedrock:InvokeModel`, `bedrock:InvokeModelWithResponseStream` **and** `bedrock:CountTokens`. Claude Code calls the count-tokens API on every turn, so a role without it fails with `is not authorized to perform: bedrock:CountTokens`. Add `bedrock:GetInferenceProfile` if your model ID is an inference profile ARN. See [Connect Bedrock with Amazon Assumed Role](/aigw/product/model-catalog/connect-bedrock-with-amazon-assumed-role) for the full policy. + + diff --git a/aigw/integrations/llms/anthropic/prompt-caching.mdx b/aigw/integrations/llms/anthropic/prompt-caching.mdx index 6570b3f04..0c7d3111f 100644 --- a/aigw/integrations/llms/anthropic/prompt-caching.mdx +++ b/aigw/integrations/llms/anthropic/prompt-caching.mdx @@ -100,7 +100,104 @@ Instead of marking individual messages, you can have the gateway turn on Anthrop } ``` -The gateway adds a top-level `cache_control: { "type": "ephemeral" }` to the request. A `cache_control` you set on the request yourself is kept as is. `auto_caching` is inherited by nested targets and can be overridden on a target. It applies to Claude models on Anthropic, Google Vertex AI, Azure AI Foundry, and Claude Platform on AWS for `/chat/completions` and `/messages`, including requests to the Responses API (`/v1/responses`) that are sent to Claude. It also applies to `/messages` on Amazon Bedrock (InvokeModel) and Bedrock Mantle (`anthropic.*` models only). Bedrock Converse is not covered. +The gateway adds a top-level `cache_control: { "type": "ephemeral" }` to the request, and Anthropic places the cache breakpoint on the last cacheable block. A `cache_control` you set on the request yourself is kept as is. `auto_caching` is inherited by nested targets and can be overridden on a target. + +### When to use it + +`auto_caching` is for requests where the **client never sends `cache_control` markers of its own**: + +- **Clients you don't control the request body for** — OpenAI-compatible SDKs, internal applications, and anything that has no way to express Anthropic's cache markers. +- **Codex and other Responses API clients.** Codex calls `/v1/responses`, which the gateway converts internally before sending it to Claude. The Responses API has no `cache_control` field, so the client cannot request caching at all — `auto_caching` injects the marker on that converted request for you. +- **Clients where you also want to control the TTL.** Combine `auto_caching` with `default_params` to set a 1-hour cache without touching the client. See [Controlling the TTL](#controlling-the-ttl). + +If your client already sends its own block-level markers — Claude Code is the common case — read [Clients that send their own cache markers](#clients-that-send-their-own-cache-markers) before enabling this. + +### Supported providers and endpoints + +The endpoint column is the route **your client calls on the gateway**. + +| Provider | `/v1/messages` | `/v1/chat/completions` | `/v1/responses` | +| --- | --- | --- | --- | +| Anthropic | Yes | Yes | Yes | +| Claude Platform on AWS | Yes | Yes | Yes | +| Google Vertex AI | Yes | Yes | Yes | +| Azure AI Foundry | Yes | Yes 1 | Yes 1 | +| Amazon Bedrock | Yes (InvokeModel) 2 | No (Converse) | No | + +1 Anthropic deployments only — the gateway uses Claude's request format when the Foundry deployment URL identifies an Anthropic model. + +2 Some older Claude deployments on Bedrock do not support a top-level `cache_control`. The gateway does not check the model before adding it, so the request is sent as is and Bedrock may reject it. If you hit this on an older model, turn `auto_caching` off for that target (and remove any `cache_control` set through `default_params`), and use block-level `cache_control` markers instead. + +Where a cell says No, the request still succeeds — it is simply sent without a top-level `cache_control`. + + +Sending the `x-portkey-beta: use-responses-api-2026-07-30` header with a `/v1/responses` request keeps it on the provider's native Responses API instead of converting it. No `cache_control` is injected on that path for any provider, and providers without a native Responses endpoint return an error. Leave the header off if you want automatic caching. + + + +`auto_caching` is opt-in per config, not an organisation-wide default. Before turning it on broadly, consider: + +- **One-shot traffic can cost more.** The cache marker goes after the last block of the request. If your prompt ends in content that is unique to each request, every request pays the 1.25x cache write price and never gets a cache hit. It saves money on multi-turn chats and agent loops. +- **Anthropic can return a 400** if the client already uses all 4 explicit cache breakpoints, or has a `cache_control` on the last block with a different TTL than the automatic one. Requests without cache markers are not affected, so the risk is clients that mix their own markers with `auto_caching`. + + +### Controlling the TTL + +`auto_caching` on its own uses Anthropic's default 5-minute TTL. To cache with a 1-hour TTL, set the top-level `cache_control` through `default_params`: + +```json +{ + "provider": "@anthropic-provider", + "auto_caching": true, + "default_params": { + "cache_control": { "type": "ephemeral", "ttl": "1h" } + } +} +``` + +A 1-hour cache write costs 2x the base input token price, against 1.25x for 5 minutes. Cache reads cost 0.1x at either TTL. + +- Pass `cache_control` as an object. The gateway forwards it unchanged, so the string `"ephemeral"` would reach Anthropic as a plain string. +- Use `default_params` rather than `override_params`. `default_params` only fills in `cache_control` when the request doesn't already have one, while `override_params` always overwrites a `cache_control` the client sent deliberately. See [Default and drop params](/aigw/product/ai-gateway/configs#default-and-drop-params). +- `default_params` works on `/v1/messages` for every provider in the table above, including Bedrock Mantle's non-Claude models. On `/v1/chat/completions` and `/v1/responses` it is carried only by the providers marked Yes in that column; elsewhere the value is dropped and the request is sent uncached. + +### Clients that send their own cache markers + +Some clients place `cache_control` markers on individual blocks themselves. **Claude Code** does this on every request — it marks `system` blocks, tool definitions, and `messages` entries, including the `role: "system"` entries it appends mid-conversation. It never sends a top-level `cache_control`, so `auto_caching` will add one *on top of* its markers. + + +Anthropic allows a maximum of **4 cache breakpoints per request**, and the automatic marker consumes one of them. If Claude Code is already using all four, the gateway's injected marker makes five and Anthropic returns a **400**. A marker on the last block with a TTL different from the injected one also returns a 400. + +Claude Code recovers from this by retrying once with the marker moved onto the last conversation message and keeping it there until it exits, so sessions are not broken — but the affected block is billed as uncached input for the rest of the session. + + +You have two options. + +**Option 1 — Keep Claude Code's own caching (recommended).** Leave `auto_caching` off for configs serving Claude Code. Its markers pass through unchanged, and the TTL is configured in the client with the [`promptCacheTtl`](https://code.claude.com/docs/en/settings-reference#promptcachettl) and [`subagentPromptCacheTtl`](https://code.claude.com/docs/en/settings-reference#subagentprompt-cache-ttl) settings, or the `ENABLE_PROMPT_CACHING_1H` environment variable. Claude Code places its breakpoints at stable prefix boundaries, which is usually better tuned than a single automatic breakpoint. + +**Option 2 — Switch to top-level caching controlled by the gateway.** If you want the TTL decided centrally rather than per developer machine, strip the client's block-level markers with `drop_params` so the injected top-level marker is the only one left: + +```json +{ + "provider": "@anthropic-provider", + "auto_caching": true, + "default_params": { + "cache_control": { "type": "ephemeral", "ttl": "1h" } + }, + "drop_params": [ + "system[*].cache_control", + "messages[*].cache_control", + "messages[*].content[*].cache_control", + "tools[*].cache_control" + ] +} +``` + +All four paths are required — omitting `messages[*].cache_control` leaves the marker on mid-conversation `role: "system"` entries. With no block markers left, neither 400 case can occur. Drop `default_params` from the example to use the 5-minute TTL instead. + + +With `drop_params` in place, the client's own TTL settings no longer have any effect, because the TTL lives inside the markers being removed. A developer who sets `ENABLE_PROMPT_CACHING_1H` will still get whatever the config specifies. The gateway becomes the only place the TTL is set, so use separate configs or targets if different teams need different TTLs. + ## Seeing Cache Results in Prisma AIRS AI Gateway diff --git a/aigw/product/ai-gateway/configs.mdx b/aigw/product/ai-gateway/configs.mdx index f6b7625f5..cb9d0c13d 100644 --- a/aigw/product/ai-gateway/configs.mdx +++ b/aigw/product/ai-gateway/configs.mdx @@ -135,6 +135,8 @@ Each config target can shape the request body before it reaches the upstream pro Execution order: `default_params` → `override_params` → `drop_params`. All three fields inherit through nested target configs the same way `override_params` does — values defined on a parent target apply to its children unless overridden. +To set Anthropic's top-level `cache_control` for requests that don't send one, see [Controlling the TTL](/aigw/integrations/llms/anthropic/prompt-caching#controlling-the-ttl). For a 5-minute cache, the [`auto_caching`](/aigw/integrations/llms/anthropic/prompt-caching#automatic-caching) flag does the same without any params. If your client sends its own block-level cache markers, see [Clients that send their own cache markers](/aigw/integrations/llms/anthropic/prompt-caching#clients-that-send-their-own-cache-markers). + ### `drop_params` paths Each entry in `drop_params` is a path into the request body. The gateway deletes the matching property before forwarding the request. Paths that do not exist on the request are silently ignored. diff --git a/aigw/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx b/aigw/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx index a9222907a..73325c018 100644 --- a/aigw/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx +++ b/aigw/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx @@ -15,6 +15,8 @@ This role you create will be used by the AI Gateway to execute InvokeModel comma ### Create a permission policy in your AWS account using the following JSON +These actions cover standard inference — chat completions, the Messages API, embeddings, image generation, and token counting: + ```json { "Version": "2012-10-17", @@ -24,7 +26,8 @@ This role you create will be used by the AI Gateway to execute InvokeModel comma "Effect": "Allow", "Action": [ "bedrock:InvokeModel", - "bedrock:InvokeModelWithResponseStream" + "bedrock:InvokeModelWithResponseStream", + "bedrock:CountTokens" ], "Resource": "*" } @@ -32,6 +35,98 @@ This role you create will be used by the AI Gateway to execute InvokeModel comma } ``` + +`bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` also authorize the Converse and ConverseStream APIs, which the AI Gateway uses for most models. + +`bedrock:CountTokens` backs the `/v1/messages/count_tokens` endpoint. Clients such as Claude Code call it on every turn, so omitting it shows up as `is not authorized to perform: bedrock:CountTokens`. It is a recent addition to Bedrock — roles created before it existed need it added explicitly. + + +### Additional permissions by feature + +Add these only if you use the corresponding feature — everything else works with the three actions above. + +| Feature | Additional actions | +| :-- | :-- | +| [Inference profiles](/aigw/integrations/llms/bedrock/aws-bedrock#inference-profiles) (the model ID is an inference profile ARN) | `bedrock:GetInferenceProfile` | +| Model discovery (listing Bedrock models through the AI Gateway) | `bedrock:ListFoundationModels` | +| [Bedrock Guardrails](/aigw/integrations/guardrails/bedrock-guardrails) | `bedrock:ApplyGuardrail` | +| [Reranking](/aigw/integrations/llms/bedrock/rerank) | `bedrock:Rerank` | +| [Batch inference](/aigw/integrations/llms/bedrock/batches) | `bedrock:CreateModelInvocationJob`, `bedrock:GetModelInvocationJob`, `bedrock:ListModelInvocationJobs`, `bedrock:StopModelInvocationJob`, plus S3 access to the input/output bucket and `iam:PassRole` for the Bedrock service role — see the [full batch policy](/aigw/integrations/llms/bedrock/batches#information-about-permissions-and-iam-roles) | +| [Fine-tuning](/aigw/integrations/llms/bedrock/fine-tuning) | `bedrock:CreateModelCustomizationJob`, `bedrock:GetModelCustomizationJob`, `bedrock:ListModelCustomizationJobs`, `bedrock:StopModelCustomizationJob`, plus S3 access to the training-data bucket and `iam:PassRole` | +| [Files](/aigw/integrations/llms/bedrock/files) | `s3:PutObject`, `s3:GetObject`, `s3:GetObjectAttributes`, `s3:AbortMultipartUpload`, `s3:ListBucket` on your bucket | + + + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "BedrockInference", + "Effect": "Allow", + "Action": [ + "bedrock:InvokeModel", + "bedrock:InvokeModelWithResponseStream", + "bedrock:CountTokens", + "bedrock:ApplyGuardrail", + "bedrock:Rerank", + "bedrock:ListFoundationModels", + "bedrock:GetFoundationModel", + "bedrock:GetInferenceProfile" + ], + "Resource": "*" + }, + { + "Sid": "BedrockBatchAndFineTuning", + "Effect": "Allow", + "Action": [ + "bedrock:CreateModelInvocationJob", + "bedrock:GetModelInvocationJob", + "bedrock:ListModelInvocationJobs", + "bedrock:StopModelInvocationJob", + "bedrock:CreateModelCustomizationJob", + "bedrock:GetModelCustomizationJob", + "bedrock:ListModelCustomizationJobs", + "bedrock:StopModelCustomizationJob", + "bedrock:TagResource", + "bedrock:UntagResource", + "bedrock:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "S3ForFilesAndJobIO", + "Effect": "Allow", + "Action": [ + "s3:ListBucket", + "s3:PutObject", + "s3:GetObject", + "s3:GetObjectAttributes", + "s3:AbortMultipartUpload" + ], + "Resource": [ + "arn:aws:s3:::", + "arn:aws:s3:::/*" + ] + }, + { + "Sid": "PassBedrockServiceRole", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": "arn:aws:iam:::role/", + "Condition": { + "StringEquals": { + "iam:PassedToService": "bedrock.amazonaws.com" + } + } + } + ] +} +``` + +Replace ``, `` and `` with your own values, and narrow each `Resource` to the model, inference profile and job ARNs you actually use. + + ### Create a new IAM role Choose *AWS account* as the trusted entity type. If you set an external ID be sure to copy it, we will need it later. diff --git a/aigw/product/model-catalog/connect-bedrock-with-amazon-assumed-role.mdx b/aigw/product/model-catalog/connect-bedrock-with-amazon-assumed-role.mdx index 4ebf6e685..9da396f92 100644 --- a/aigw/product/model-catalog/connect-bedrock-with-amazon-assumed-role.mdx +++ b/aigw/product/model-catalog/connect-bedrock-with-amazon-assumed-role.mdx @@ -16,6 +16,8 @@ This role you create will be used by the AI Gateway to execute InvokeModel comma ### Create a permission policy in your AWS account using the following JSON +These actions cover standard inference — chat completions, the Messages API, embeddings, image generation, and token counting: + ```json { "Version": "2012-10-17", @@ -25,7 +27,8 @@ This role you create will be used by the AI Gateway to execute InvokeModel comma "Effect": "Allow", "Action": [ "bedrock:InvokeModel", - "bedrock:InvokeModelWithResponseStream" + "bedrock:InvokeModelWithResponseStream", + "bedrock:CountTokens" ], "Resource": "*" } @@ -33,6 +36,98 @@ This role you create will be used by the AI Gateway to execute InvokeModel comma } ``` + +`bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` also authorize the Converse and ConverseStream APIs, which the AI Gateway uses for most models. + +`bedrock:CountTokens` backs the `/v1/messages/count_tokens` endpoint. Clients such as Claude Code call it on every turn, so omitting it shows up as `is not authorized to perform: bedrock:CountTokens`. It is a recent addition to Bedrock — roles created before it existed need it added explicitly. + + +### Additional permissions by feature + +Add these only if you use the corresponding feature — everything else works with the three actions above. + +| Feature | Additional actions | +| :-- | :-- | +| [Inference profiles](/aigw/integrations/llms/bedrock/aws-bedrock#inference-profiles) (the model ID is an inference profile ARN) | `bedrock:GetInferenceProfile` | +| Model discovery (listing Bedrock models through the AI Gateway) | `bedrock:ListFoundationModels` | +| [Bedrock Guardrails](/aigw/integrations/guardrails/bedrock-guardrails) | `bedrock:ApplyGuardrail` | +| [Reranking](/aigw/integrations/llms/bedrock/rerank) | `bedrock:Rerank` | +| [Batch inference](/aigw/integrations/llms/bedrock/batches) | `bedrock:CreateModelInvocationJob`, `bedrock:GetModelInvocationJob`, `bedrock:ListModelInvocationJobs`, `bedrock:StopModelInvocationJob`, plus S3 access to the input/output bucket and `iam:PassRole` for the Bedrock service role — see the [full batch policy](/aigw/integrations/llms/bedrock/batches#information-about-permissions-and-iam-roles) | +| [Fine-tuning](/aigw/integrations/llms/bedrock/fine-tuning) | `bedrock:CreateModelCustomizationJob`, `bedrock:GetModelCustomizationJob`, `bedrock:ListModelCustomizationJobs`, `bedrock:StopModelCustomizationJob`, plus S3 access to the training-data bucket and `iam:PassRole` | +| [Files](/aigw/integrations/llms/bedrock/files) | `s3:PutObject`, `s3:GetObject`, `s3:GetObjectAttributes`, `s3:AbortMultipartUpload`, `s3:ListBucket` on your bucket | + + + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "BedrockInference", + "Effect": "Allow", + "Action": [ + "bedrock:InvokeModel", + "bedrock:InvokeModelWithResponseStream", + "bedrock:CountTokens", + "bedrock:ApplyGuardrail", + "bedrock:Rerank", + "bedrock:ListFoundationModels", + "bedrock:GetFoundationModel", + "bedrock:GetInferenceProfile" + ], + "Resource": "*" + }, + { + "Sid": "BedrockBatchAndFineTuning", + "Effect": "Allow", + "Action": [ + "bedrock:CreateModelInvocationJob", + "bedrock:GetModelInvocationJob", + "bedrock:ListModelInvocationJobs", + "bedrock:StopModelInvocationJob", + "bedrock:CreateModelCustomizationJob", + "bedrock:GetModelCustomizationJob", + "bedrock:ListModelCustomizationJobs", + "bedrock:StopModelCustomizationJob", + "bedrock:TagResource", + "bedrock:UntagResource", + "bedrock:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "S3ForFilesAndJobIO", + "Effect": "Allow", + "Action": [ + "s3:ListBucket", + "s3:PutObject", + "s3:GetObject", + "s3:GetObjectAttributes", + "s3:AbortMultipartUpload" + ], + "Resource": [ + "arn:aws:s3:::", + "arn:aws:s3:::/*" + ] + }, + { + "Sid": "PassBedrockServiceRole", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": "arn:aws:iam:::role/", + "Condition": { + "StringEquals": { + "iam:PassedToService": "bedrock.amazonaws.com" + } + } + } + ] +} +``` + +Replace ``, `` and `` with your own values, and narrow each `Resource` to the model, inference profile and job ARNs you actually use. + + ### Create a new IAM role Choose *AWS account* as the trusted entity type. If you set an external ID be sure to copy it, we will need it later. diff --git a/aigw/self-hosting/hybrid-deployments/aws/eks.mdx b/aigw/self-hosting/hybrid-deployments/aws/eks.mdx index 645d284cd..7612c25bc 100644 --- a/aigw/self-hosting/hybrid-deployments/aws/eks.mdx +++ b/aigw/self-hosting/hybrid-deployments/aws/eks.mdx @@ -584,7 +584,7 @@ cat >bedrock-access-policy.json <::foundation-model/", "arn:aws:bedrock:::provisioned-model/" diff --git a/api-reference/inference-api/config-object.mdx b/api-reference/inference-api/config-object.mdx index 0b11a27df..21b4a6e89 100644 --- a/api-reference/inference-api/config-object.mdx +++ b/api-reference/inference-api/config-object.mdx @@ -669,6 +669,7 @@ You can find more examples of schemas [below](/api-reference/inference-api/confi | `override_params` | Pass model name and other hyper parameters | object | No | "model", "temperature", "frequency\_penalty", "logit\_bias", "logprobs", "top\_logprobs", "max\_tokens", "n", "presence\_penalty", "response\_format", "seed", "stop", "top_p", etc. | Pass everything that's typically part of the payload | | `default_params` | Inject parameters into the request body only when not already present (respects client-sent values) | object | No | Any payload field | See [Default and drop params](/product/ai-gateway/configs#default-and-drop-params) | | `drop_params` | Remove parameters from the final request body using bracket-notation paths (nested keys, array indices, and `[*]` wildcards) | array of strings | No | Any payload field path | See [Default and drop params](/product/ai-gateway/configs#default-and-drop-params) | +| `auto_caching` | Add Anthropic's top-level `cache_control: {type: "ephemeral"}` to requests that don't already set one | boolean | No | `true`, `false` | Inherited by nested targets. See [Automatic Caching](/integrations/llms/anthropic/prompt-caching#automatic-caching) | ### Strategy Object Details diff --git a/integrations/libraries/claude-code-bedrock.mdx b/integrations/libraries/claude-code-bedrock.mdx index 2393d32d1..87e698434 100644 --- a/integrations/libraries/claude-code-bedrock.mdx +++ b/integrations/libraries/claude-code-bedrock.mdx @@ -27,6 +27,10 @@ Enter your AWS credentials: Create a slug like `bedrock-prod`. + +**IAM permissions:** the credentials (or assumed role) you use here need `bedrock:InvokeModel`, `bedrock:InvokeModelWithResponseStream` **and** `bedrock:CountTokens`. Claude Code calls the count-tokens API on every turn, so a role without it fails with `is not authorized to perform: bedrock:CountTokens`. Add `bedrock:GetInferenceProfile` if your model ID is an inference profile ARN. See [Connect Bedrock with Amazon Assumed Role](/product/model-catalog/connect-bedrock-with-amazon-assumed-role) for the full policy. + + diff --git a/integrations/llms/anthropic/prompt-caching.mdx b/integrations/llms/anthropic/prompt-caching.mdx index e8d043f82..c60f5d9af 100644 --- a/integrations/llms/anthropic/prompt-caching.mdx +++ b/integrations/llms/anthropic/prompt-caching.mdx @@ -204,7 +204,103 @@ Instead of marking individual messages, you can have the gateway turn on Anthrop } ``` -The gateway adds a top-level `cache_control: { "type": "ephemeral" }` to the request. A `cache_control` you set on the request yourself is kept as is. `auto_caching` is inherited by nested targets and can be overridden on a target. It applies to Claude models on Anthropic, Google Vertex AI, Azure AI Foundry, and Claude Platform on AWS for `/chat/completions` and `/messages`, including requests to the Responses API (`/v1/responses`) that are sent to Claude. It also applies to `/messages` on Amazon Bedrock (InvokeModel) and Bedrock Mantle (`anthropic.*` models only). Bedrock Converse is not covered. +The gateway adds a top-level `cache_control: { "type": "ephemeral" }` to the request, and Anthropic places the cache breakpoint on the last cacheable block. A `cache_control` you set on the request yourself is kept as is. `auto_caching` is inherited by nested targets and can be overridden on a target. + +### When to use it + +`auto_caching` is for requests where the **client never sends `cache_control` markers of its own**: + +- **Clients you don't control the request body for** — OpenAI-compatible SDKs, internal applications, and anything that has no way to express Anthropic's cache markers. +- **Codex and other Responses API clients.** Codex calls `/v1/responses`, which the gateway converts internally before sending it to Claude. The Responses API has no `cache_control` field, so the client cannot request caching at all — `auto_caching` injects the marker on that converted request for you. +- **Clients where you also want to control the TTL.** Combine `auto_caching` with `default_params` to set a 1-hour cache without touching the client. See [Controlling the TTL](#controlling-the-ttl). + +If your client already sends its own block-level markers — Claude Code is the common case — read [Clients that send their own cache markers](#clients-that-send-their-own-cache-markers) before enabling this. + +### Supported providers and endpoints + +The endpoint column is the route **your client calls on the gateway**. Requests to `/v1/responses` are converted to chat completions internally before reaching Claude. + +| Provider | `/v1/messages` | `/v1/chat/completions` | `/v1/responses` | +| --- | --- | --- | --- | +| Anthropic | Yes | Yes | Yes | +| Claude Platform on AWS | Yes | Yes | Yes | +| Google Vertex AI | Yes | Yes | Yes | +| Azure AI Foundry | Yes | Yes 1 | Yes 1 | +| Amazon Bedrock | Yes (InvokeModel) 2 | No (Converse) | No | + +1 Anthropic deployments only — the gateway uses Claude's request format when the Foundry deployment URL identifies an Anthropic model. +2 Some older Claude deployments on Bedrock do not support a top-level `cache_control`. The gateway does not check the model before adding it, so the request is sent as is and Bedrock may reject it. If you hit this on an older model, turn `auto_caching` off for that target (and remove any `cache_control` set through `default_params`), and use block-level `cache_control` markers instead. + +Where a cell says No, the request still succeeds — it is simply sent without a top-level `cache_control`. + + +Sending the `x-portkey-beta: use-responses-api-2026-07-30` header with a `/v1/responses` request keeps it on the provider's native Responses API instead of converting it. No `cache_control` is injected on that path for any provider, and providers without a native Responses endpoint return an error. Leave the header off if you want automatic caching. + + + +`auto_caching` is opt-in per config, not an organization-wide default. Before turning it on broadly, consider: + +- **One-shot traffic can cost more.** The cache marker goes after the last block of the request. If your prompt ends in content that is unique to each request, every request pays the 1.25x cache write price and never gets a cache hit. It saves money on multi-turn chats and agent loops. +- **Anthropic can return a 400** if the client already uses all 4 explicit cache breakpoints, or has a `cache_control` on the last block with a different TTL than the automatic one. Requests without cache markers are not affected, so the risk is clients that mix their own markers with `auto_caching`. + + +### Controlling the TTL + +`auto_caching` on its own uses Anthropic's default 5-minute TTL. To cache with a 1-hour TTL, set the top-level `cache_control` through `default_params`: + +```json +{ + "provider": "@anthropic-provider", + "auto_caching": true, + "default_params": { + "cache_control": { "type": "ephemeral", "ttl": "1h" } + } +} +``` + +A 1-hour cache write costs 2x the base input token price, against 1.25x for 5 minutes. Cache reads cost 0.1x at either TTL. + +- Pass `cache_control` as an object. The gateway forwards it unchanged, so the string `"ephemeral"` would reach Anthropic as a plain string. +- Use `default_params` rather than `override_params`. `default_params` only fills in `cache_control` when the request doesn't already have one, while `override_params` always overwrites a `cache_control` the client sent deliberately. See [Default and drop params](/product/ai-gateway/configs#default-and-drop-params). +- `default_params` works on `/v1/messages` for every provider in the table above, including Bedrock Mantle's non-Claude models. On `/v1/chat/completions` and `/v1/responses` it is carried only by the providers marked Yes in that column; elsewhere the value is dropped and the request is sent uncached. + +### Clients that send their own cache markers + +Some clients place `cache_control` markers on individual blocks themselves. **Claude Code** does this on every request — it marks `system` blocks, tool definitions, and `messages` entries, including the `role: "system"` entries it appends mid-conversation. It never sends a top-level `cache_control`, so `auto_caching` will add one *on top of* its markers. + + +Anthropic allows a maximum of **4 cache breakpoints per request**, and the automatic marker consumes one of them. If Claude Code is already using all four, the gateway's injected marker makes five and Anthropic returns a **400**. A marker on the last block with a TTL different from the injected one also returns a 400. + +Claude Code recovers from this by retrying once with the marker moved onto the last conversation message and keeping it there until it exits, so sessions are not broken — but the affected block is billed as uncached input for the rest of the session. + + +You have two options. + +**Option 1 — Keep Claude Code's own caching (recommended).** Leave `auto_caching` off for configs serving Claude Code. Its markers pass through unchanged, and the TTL is configured in the client with the [`promptCacheTtl`](https://code.claude.com/docs/en/settings-reference#promptcachettl) and [`subagentPromptCacheTtl`](https://code.claude.com/docs/en/settings-reference#subagentprompt-cache-ttl) settings, or the `ENABLE_PROMPT_CACHING_1H` environment variable. Claude Code places its breakpoints at stable prefix boundaries, which is usually better tuned than a single automatic breakpoint. + +**Option 2 — Switch to top-level caching controlled by the gateway.** If you want the TTL decided centrally rather than per developer machine, strip the client's block-level markers with `drop_params` so the injected top-level marker is the only one left: + +```json +{ + "provider": "@anthropic-provider", + "auto_caching": true, + "default_params": { + "cache_control": { "type": "ephemeral", "ttl": "1h" } + }, + "drop_params": [ + "system[*].cache_control", + "messages[*].cache_control", + "messages[*].content[*].cache_control", + "tools[*].cache_control" + ] +} +``` + +All four paths are required — omitting `messages[*].cache_control` leaves the marker on mid-conversation `role: "system"` entries. With no block markers left, neither 400 case can occur. Drop `default_params` from the example to use the 5-minute TTL instead. + + +With `drop_params` in place, the client's own TTL settings no longer have any effect, because the TTL lives inside the markers being removed. A developer who sets `ENABLE_PROMPT_CACHING_1H` will still get whatever the config specifies. The gateway becomes the only place the TTL is set, so use separate configs or targets if different teams need different TTLs. + diff --git a/integrations/llms/bedrock/count-tokens.mdx b/integrations/llms/bedrock/count-tokens.mdx index 5d0cbaa2b..a1283090c 100644 --- a/integrations/llms/bedrock/count-tokens.mdx +++ b/integrations/llms/bedrock/count-tokens.mdx @@ -7,6 +7,10 @@ Portkey supports the [AWS Bedrock CountTokens API](https://docs.aws.amazon.com/b The token count returned by this operation matches the token count that would be charged if the same input were sent to the model in an `InvokeModel` or `Converse` request. + +Your Bedrock credentials need the `bedrock:CountTokens` IAM action on the foundation models you call. It is separate from `bedrock:InvokeModel`, so roles created before this action existed will fail with `is not authorized to perform: bedrock:CountTokens`. See [Connect Bedrock with Amazon Assumed Role](/product/model-catalog/connect-bedrock-with-amazon-assumed-role) for the full policy. + + ## Using Count Tokens with Portkey Portkey supports Bedrock's CountTokens endpoint through the Anthropic-compatible format, allowing you to use the same API signature across Bedrock, Vertex AI, and Anthropic providers. diff --git a/product/ai-gateway/configs.mdx b/product/ai-gateway/configs.mdx index c4f6e5a13..6c7946742 100644 --- a/product/ai-gateway/configs.mdx +++ b/product/ai-gateway/configs.mdx @@ -176,6 +176,8 @@ Each config target can shape the request body before it reaches the upstream pro Execution order: `default_params` → `override_params` → `drop_params`. All three fields inherit through nested target configs the same way `override_params` does — values defined on a parent target apply to its children unless overridden. +To set Anthropic's top-level `cache_control` for requests that don't send one, see [Controlling the TTL](/integrations/llms/anthropic/prompt-caching#controlling-the-ttl). For a 5-minute cache, the [`auto_caching`](/integrations/llms/anthropic/prompt-caching#automatic-caching) flag does the same without any params. If your client sends its own block-level cache markers, see [Clients that send their own cache markers](/integrations/llms/anthropic/prompt-caching#clients-that-send-their-own-cache-markers). + ### `drop_params` paths Each entry in `drop_params` is a path into the request body. The gateway deletes the matching property before forwarding the request. Paths that do not exist on the request are silently ignored. diff --git a/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx b/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx index 8b06c9099..6290e9cfa 100644 --- a/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx +++ b/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx @@ -25,6 +25,8 @@ This role you create will be used by Portkey to execute InvokeModel commands on ### Create a permission policy in your AWS account using the following JSON +These actions cover standard inference — chat completions, the Messages API, embeddings, image generation, and token counting: + ```json { "Version": "2012-10-17", @@ -34,7 +36,8 @@ This role you create will be used by Portkey to execute InvokeModel commands on "Effect": "Allow", "Action": [ "bedrock:InvokeModel", - "bedrock:InvokeModelWithResponseStream" + "bedrock:InvokeModelWithResponseStream", + "bedrock:CountTokens" ], "Resource": "*" } @@ -42,10 +45,102 @@ This role you create will be used by Portkey to execute InvokeModel commands on } ``` + +`bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` also authorize the Converse and ConverseStream APIs, which Portkey uses for most models. + +`bedrock:CountTokens` backs the `/v1/messages/count_tokens` endpoint. Clients such as Claude Code call it on every turn, so omitting it shows up as `is not authorized to perform: bedrock:CountTokens`. It is a recent addition to Bedrock — roles created before it existed need it added explicitly. + + +### Additional permissions by feature + +Add these only if you use the corresponding feature — everything else works with the three actions above. + +| Feature | Additional actions | +| :-- | :-- | +| [Inference profiles](/integrations/llms/bedrock/aws-bedrock#inference-profiles) (the model ID is an inference profile ARN) | `bedrock:GetInferenceProfile` | +| Model discovery (listing Bedrock models through Portkey) | `bedrock:ListFoundationModels` | +| [Bedrock Guardrails](/integrations/guardrails/bedrock-guardrails) | `bedrock:ApplyGuardrail` | +| [Reranking](/integrations/llms/bedrock/rerank) | `bedrock:Rerank` | +| [Batch inference](/integrations/llms/bedrock/batches) | `bedrock:CreateModelInvocationJob`, `bedrock:GetModelInvocationJob`, `bedrock:ListModelInvocationJobs`, `bedrock:StopModelInvocationJob`, plus S3 access to the input/output bucket and `iam:PassRole` for the Bedrock service role — see the [full batch policy](/integrations/llms/bedrock/batches#information-about-permissions-and-iam-roles) | +| [Fine-tuning](/integrations/llms/bedrock/fine-tuning) | `bedrock:CreateModelCustomizationJob`, `bedrock:GetModelCustomizationJob`, `bedrock:ListModelCustomizationJobs`, `bedrock:StopModelCustomizationJob`, plus S3 access to the training-data bucket and `iam:PassRole` | +| [Files](/integrations/llms/bedrock/files) | `s3:PutObject`, `s3:GetObject`, `s3:GetObjectAttributes`, `s3:AbortMultipartUpload`, `s3:ListBucket` on your bucket | + + + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "BedrockInference", + "Effect": "Allow", + "Action": [ + "bedrock:InvokeModel", + "bedrock:InvokeModelWithResponseStream", + "bedrock:CountTokens", + "bedrock:ApplyGuardrail", + "bedrock:Rerank", + "bedrock:ListFoundationModels", + "bedrock:GetFoundationModel", + "bedrock:GetInferenceProfile" + ], + "Resource": "*" + }, + { + "Sid": "BedrockBatchAndFineTuning", + "Effect": "Allow", + "Action": [ + "bedrock:CreateModelInvocationJob", + "bedrock:GetModelInvocationJob", + "bedrock:ListModelInvocationJobs", + "bedrock:StopModelInvocationJob", + "bedrock:CreateModelCustomizationJob", + "bedrock:GetModelCustomizationJob", + "bedrock:ListModelCustomizationJobs", + "bedrock:StopModelCustomizationJob", + "bedrock:TagResource", + "bedrock:UntagResource", + "bedrock:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "S3ForFilesAndJobIO", + "Effect": "Allow", + "Action": [ + "s3:ListBucket", + "s3:PutObject", + "s3:GetObject", + "s3:GetObjectAttributes", + "s3:AbortMultipartUpload" + ], + "Resource": [ + "arn:aws:s3:::", + "arn:aws:s3:::/*" + ] + }, + { + "Sid": "PassBedrockServiceRole", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": "arn:aws:iam:::role/", + "Condition": { + "StringEquals": { + "iam:PassedToService": "bedrock.amazonaws.com" + } + } + } + ] +} +``` + +Replace ``, `` and `` with your own values, and narrow each `Resource` to the model, inference profile and job ARNs you actually use. + + ### Create a new IAM role Choose *AWS account* as the trusted entity type. If you set an external ID be sure to copy it, we will need it later. diff --git a/product/enterprise-offering/cloud-marketplace/aws.mdx b/product/enterprise-offering/cloud-marketplace/aws.mdx index e93969437..1ddb965df 100644 --- a/product/enterprise-offering/cloud-marketplace/aws.mdx +++ b/product/enterprise-offering/cloud-marketplace/aws.mdx @@ -263,6 +263,7 @@ Resources: Action: - "bedrock:InvokeModel" - "bedrock:InvokeModelWithResponseStream" + - "bedrock:CountTokens" Resource: "*" - Effect: Allow Action: diff --git a/product/model-catalog/connect-bedrock-with-amazon-assumed-role.mdx b/product/model-catalog/connect-bedrock-with-amazon-assumed-role.mdx index 09e3dba0f..12121951d 100644 --- a/product/model-catalog/connect-bedrock-with-amazon-assumed-role.mdx +++ b/product/model-catalog/connect-bedrock-with-amazon-assumed-role.mdx @@ -23,6 +23,8 @@ This role you create will be used by Porktey to execute InvokeModel commands on ### Create a permission policy in your AWS account using the following JSON +These actions cover standard inference — chat completions, the Messages API, embeddings, image generation, and token counting: + ```json { "Version": "2012-10-17", @@ -32,7 +34,8 @@ This role you create will be used by Porktey to execute InvokeModel commands on "Effect": "Allow", "Action": [ "bedrock:InvokeModel", - "bedrock:InvokeModelWithResponseStream" + "bedrock:InvokeModelWithResponseStream", + "bedrock:CountTokens" ], "Resource": "*" } @@ -40,10 +43,102 @@ This role you create will be used by Porktey to execute InvokeModel commands on } ``` + +`bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` also authorize the Converse and ConverseStream APIs, which Portkey uses for most models. + +`bedrock:CountTokens` backs the `/v1/messages/count_tokens` endpoint. Clients such as Claude Code call it on every turn, so omitting it shows up as `is not authorized to perform: bedrock:CountTokens`. It is a recent addition to Bedrock — roles created before it existed need it added explicitly. + + +### Additional permissions by feature + +Add these only if you use the corresponding feature — everything else works with the three actions above. + +| Feature | Additional actions | +| :-- | :-- | +| [Inference profiles](/integrations/llms/bedrock/aws-bedrock#inference-profiles) (the model ID is an inference profile ARN) | `bedrock:GetInferenceProfile` | +| Model discovery (listing Bedrock models through Portkey) | `bedrock:ListFoundationModels` | +| [Bedrock Guardrails](/integrations/guardrails/bedrock-guardrails) | `bedrock:ApplyGuardrail` | +| [Reranking](/integrations/llms/bedrock/rerank) | `bedrock:Rerank` | +| [Batch inference](/integrations/llms/bedrock/batches) | `bedrock:CreateModelInvocationJob`, `bedrock:GetModelInvocationJob`, `bedrock:ListModelInvocationJobs`, `bedrock:StopModelInvocationJob`, plus S3 access to the input/output bucket and `iam:PassRole` for the Bedrock service role — see the [full batch policy](/integrations/llms/bedrock/batches#information-about-permissions-and-iam-roles) | +| [Fine-tuning](/integrations/llms/bedrock/fine-tuning) | `bedrock:CreateModelCustomizationJob`, `bedrock:GetModelCustomizationJob`, `bedrock:ListModelCustomizationJobs`, `bedrock:StopModelCustomizationJob`, plus S3 access to the training-data bucket and `iam:PassRole` | +| [Files](/integrations/llms/bedrock/files) | `s3:PutObject`, `s3:GetObject`, `s3:GetObjectAttributes`, `s3:AbortMultipartUpload`, `s3:ListBucket` on your bucket | + + + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "BedrockInference", + "Effect": "Allow", + "Action": [ + "bedrock:InvokeModel", + "bedrock:InvokeModelWithResponseStream", + "bedrock:CountTokens", + "bedrock:ApplyGuardrail", + "bedrock:Rerank", + "bedrock:ListFoundationModels", + "bedrock:GetFoundationModel", + "bedrock:GetInferenceProfile" + ], + "Resource": "*" + }, + { + "Sid": "BedrockBatchAndFineTuning", + "Effect": "Allow", + "Action": [ + "bedrock:CreateModelInvocationJob", + "bedrock:GetModelInvocationJob", + "bedrock:ListModelInvocationJobs", + "bedrock:StopModelInvocationJob", + "bedrock:CreateModelCustomizationJob", + "bedrock:GetModelCustomizationJob", + "bedrock:ListModelCustomizationJobs", + "bedrock:StopModelCustomizationJob", + "bedrock:TagResource", + "bedrock:UntagResource", + "bedrock:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "S3ForFilesAndJobIO", + "Effect": "Allow", + "Action": [ + "s3:ListBucket", + "s3:PutObject", + "s3:GetObject", + "s3:GetObjectAttributes", + "s3:AbortMultipartUpload" + ], + "Resource": [ + "arn:aws:s3:::", + "arn:aws:s3:::/*" + ] + }, + { + "Sid": "PassBedrockServiceRole", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": "arn:aws:iam:::role/", + "Condition": { + "StringEquals": { + "iam:PassedToService": "bedrock.amazonaws.com" + } + } + } + ] +} +``` + +Replace ``, `` and `` with your own values, and narrow each `Resource` to the model, inference profile and job ARNs you actually use. + + ### Create a new IAM role Choose *AWS account* as the trusted entity type. If you set an external ID be sure to copy it, we will need it later. diff --git a/self-hosting/hybrid-deployments/aws/eks.mdx b/self-hosting/hybrid-deployments/aws/eks.mdx index d2a0654bf..dd1379efe 100644 --- a/self-hosting/hybrid-deployments/aws/eks.mdx +++ b/self-hosting/hybrid-deployments/aws/eks.mdx @@ -672,7 +672,7 @@ cat >bedrock-access-policy.json <::foundation-model/", "arn:aws:bedrock:::provisioned-model/" diff --git a/self-hosting/hybrid-deployments/aws/marketplace.mdx b/self-hosting/hybrid-deployments/aws/marketplace.mdx index b05f9ed7d..641156bad 100644 --- a/self-hosting/hybrid-deployments/aws/marketplace.mdx +++ b/self-hosting/hybrid-deployments/aws/marketplace.mdx @@ -263,6 +263,7 @@ Resources: Action: - "bedrock:InvokeModel" - "bedrock:InvokeModelWithResponseStream" + - "bedrock:CountTokens" Resource: "*" - Effect: Allow Action: diff --git a/virtual_key_old/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx b/virtual_key_old/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx index 973fc892f..0957d2486 100644 --- a/virtual_key_old/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx +++ b/virtual_key_old/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx @@ -25,6 +25,8 @@ This role you create will be used by Porktey to execute InvokeModel commands on ### Create a permission policy in your AWS account using the following JSON +These actions cover standard inference — chat completions, the Messages API, embeddings, image generation, and token counting: + ```json { "Version": "2012-10-17", @@ -34,7 +36,8 @@ This role you create will be used by Porktey to execute InvokeModel commands on "Effect": "Allow", "Action": [ "bedrock:InvokeModel", - "bedrock:InvokeModelWithResponseStream" + "bedrock:InvokeModelWithResponseStream", + "bedrock:CountTokens" ], "Resource": "*" } @@ -42,10 +45,102 @@ This role you create will be used by Porktey to execute InvokeModel commands on } ``` + +`bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` also authorize the Converse and ConverseStream APIs, which Portkey uses for most models. + +`bedrock:CountTokens` backs the `/v1/messages/count_tokens` endpoint. Clients such as Claude Code call it on every turn, so omitting it shows up as `is not authorized to perform: bedrock:CountTokens`. It is a recent addition to Bedrock — roles created before it existed need it added explicitly. + + +### Additional permissions by feature + +Add these only if you use the corresponding feature — everything else works with the three actions above. + +| Feature | Additional actions | +| :-- | :-- | +| [Inference profiles](/virtual_key_old/integrations/llms/bedrock/aws-bedrock) (the model ID is an inference profile ARN) | `bedrock:GetInferenceProfile` | +| Model discovery (listing Bedrock models through Portkey) | `bedrock:ListFoundationModels` | +| [Bedrock Guardrails](/virtual_key_old/integrations/guardrails/bedrock-guardrails) | `bedrock:ApplyGuardrail` | +| Reranking | `bedrock:Rerank` | +| [Batch inference](/virtual_key_old/integrations/llms/bedrock/batches) | `bedrock:CreateModelInvocationJob`, `bedrock:GetModelInvocationJob`, `bedrock:ListModelInvocationJobs`, `bedrock:StopModelInvocationJob`, plus S3 access to the input/output bucket and `iam:PassRole` for the Bedrock service role | +| [Fine-tuning](/virtual_key_old/integrations/llms/bedrock/fine-tuning) | `bedrock:CreateModelCustomizationJob`, `bedrock:GetModelCustomizationJob`, `bedrock:ListModelCustomizationJobs`, `bedrock:StopModelCustomizationJob`, plus S3 access to the training-data bucket and `iam:PassRole` | +| [Files](/virtual_key_old/integrations/llms/bedrock/files) | `s3:PutObject`, `s3:GetObject`, `s3:GetObjectAttributes`, `s3:AbortMultipartUpload`, `s3:ListBucket` on your bucket | + + + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "BedrockInference", + "Effect": "Allow", + "Action": [ + "bedrock:InvokeModel", + "bedrock:InvokeModelWithResponseStream", + "bedrock:CountTokens", + "bedrock:ApplyGuardrail", + "bedrock:Rerank", + "bedrock:ListFoundationModels", + "bedrock:GetFoundationModel", + "bedrock:GetInferenceProfile" + ], + "Resource": "*" + }, + { + "Sid": "BedrockBatchAndFineTuning", + "Effect": "Allow", + "Action": [ + "bedrock:CreateModelInvocationJob", + "bedrock:GetModelInvocationJob", + "bedrock:ListModelInvocationJobs", + "bedrock:StopModelInvocationJob", + "bedrock:CreateModelCustomizationJob", + "bedrock:GetModelCustomizationJob", + "bedrock:ListModelCustomizationJobs", + "bedrock:StopModelCustomizationJob", + "bedrock:TagResource", + "bedrock:UntagResource", + "bedrock:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "S3ForFilesAndJobIO", + "Effect": "Allow", + "Action": [ + "s3:ListBucket", + "s3:PutObject", + "s3:GetObject", + "s3:GetObjectAttributes", + "s3:AbortMultipartUpload" + ], + "Resource": [ + "arn:aws:s3:::", + "arn:aws:s3:::/*" + ] + }, + { + "Sid": "PassBedrockServiceRole", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": "arn:aws:iam:::role/", + "Condition": { + "StringEquals": { + "iam:PassedToService": "bedrock.amazonaws.com" + } + } + } + ] +} +``` + +Replace ``, `` and `` with your own values, and narrow each `Resource` to the model, inference profile and job ARNs you actually use. + + ### Create a new IAM role Choose *AWS account* as the trusted entity type. If you set an external ID be sure to copy it, we will need it later. diff --git a/virtual_key_old/product/enterprise-offering/cloud-marketplace/aws.mdx b/virtual_key_old/product/enterprise-offering/cloud-marketplace/aws.mdx index 6398b7c8b..aefc9adbe 100644 --- a/virtual_key_old/product/enterprise-offering/cloud-marketplace/aws.mdx +++ b/virtual_key_old/product/enterprise-offering/cloud-marketplace/aws.mdx @@ -263,6 +263,7 @@ Resources: Action: - "bedrock:InvokeModel" - "bedrock:InvokeModelWithResponseStream" + - "bedrock:CountTokens" Resource: "*" - Effect: Allow Action: