From 55eaf5d92e79ecc1db6d607b63402dee62601634 Mon Sep 17 00:00:00 2001 From: Akhil Menon Date: Mon, 2 Nov 2026 16:42:45 +0530 Subject: [PATCH 1/3] Added CB and concurrency related information with examples in legacy and aigw paths --- aigw/product/ai-gateway/circuit-breaker.mdx | 33 +++++++++++++++++++++ product/ai-gateway/circuit-breaker.mdx | 33 +++++++++++++++++++++ 2 files changed, 66 insertions(+) diff --git a/aigw/product/ai-gateway/circuit-breaker.mdx b/aigw/product/ai-gateway/circuit-breaker.mdx index b723695bc..680d6a0da 100644 --- a/aigw/product/ai-gateway/circuit-breaker.mdx +++ b/aigw/product/ai-gateway/circuit-breaker.mdx @@ -5,6 +5,10 @@ description: Automatically stop routing to unhealthy targets until they recover. Circuit breakers prevent cascading failures by temporarily blocking requests to targets that are failing. + +Circuit breakers react to failures only. They do not manage concurrency. See [Circuit Breakers and Concurrency](#circuit-breakers-and-concurrency). + + ## Config Schema | Field | Description | @@ -76,3 +80,32 @@ flowchart TD - Unhealthy targets removed from routing - If all targets are OPEN, circuit breaker is bypassed + +## Circuit Breakers and Concurrency + +Circuit breakers are a **health mechanism, not a concurrency control**. They open only when a target is failing, and they have no concept of how many requests are running at the same time. + +**What circuit breakers do** +- Count failures and successes per strategy path +- Open when `failure_threshold` or `failure_threshold_percentage` is exceeded, counting only responses matching `failure_status_codes` +- Close again after `cooldown_interval` + +**What circuit breakers do not do** +- Limit, queue, or throttle concurrent requests to a target +- Use the number of simultaneous requests as an input to open or close the circuit +- Open because of a burst of parallel requests. Only failures open the circuit. + +### Why concurrency can seem relevant + +The question usually comes up when circuit breakers are combined with [configs](/aigw/product/ai-gateway/configs) and [fallbacks](/aigw/product/ai-gateway/fallbacks). For example, a provider that rejects requests under heavy load may return `429`. If `429` is in `failure_status_codes`, those responses count as failures and can open the circuit. The circuit breaker is reacting to the failures, not to the load that caused them. Any rerouting of traffic that follows comes from your fallback or load balancing strategy. + + +Managing concurrency in a config and fallback setup is outside the scope of circuit breakers. Do not rely on them to cap parallel requests or smooth out traffic spikes. + + +### Managing concurrency and load + +Use the controls built for it: +- [Rate limits](/aigw/product/ai-gateway/virtual-keys/rate-limits) to cap requests or tokens on a provider or integration +- [Load balancing](/aigw/product/ai-gateway/load-balancing) to spread traffic across targets by weight +- [Fallbacks](/aigw/product/ai-gateway/fallbacks) to route to a backup target when a request fails diff --git a/product/ai-gateway/circuit-breaker.mdx b/product/ai-gateway/circuit-breaker.mdx index c24dfd0bc..9c96a9b5e 100644 --- a/product/ai-gateway/circuit-breaker.mdx +++ b/product/ai-gateway/circuit-breaker.mdx @@ -9,6 +9,10 @@ Available on all Portkey [plans](https://portkey.ai/pricing). Circuit breakers prevent cascading failures by temporarily blocking requests to targets that are failing. + +Circuit breakers react to failures only. They do not manage concurrency. See [Circuit Breakers and Concurrency](#circuit-breakers-and-concurrency). + + ## Config Schema | Field | Description | @@ -81,6 +85,35 @@ flowchart TD - Unhealthy targets removed from routing - If all targets are OPEN, circuit breaker is bypassed +## Circuit Breakers and Concurrency + +Circuit breakers are a **health mechanism, not a concurrency control**. They open only when a target is failing, and they have no concept of how many requests are running at the same time. + +**What circuit breakers do** +- Count failures and successes per strategy path +- Open when `failure_threshold` or `failure_threshold_percentage` is exceeded, counting only responses matching `failure_status_codes` +- Close again after `cooldown_interval` + +**What circuit breakers do not do** +- Limit, queue, or throttle concurrent requests to a target +- Use the number of simultaneous requests as an input to open or close the circuit +- Open because of a burst of parallel requests. Only failures open the circuit. + +### Why concurrency can seem relevant + +The question usually comes up when circuit breakers are combined with [configs](/product/ai-gateway/configs) and [fallbacks](/product/ai-gateway/fallbacks). For example, a provider that rejects requests under heavy load may return `429`. If `429` is in `failure_status_codes`, those responses count as failures and can open the circuit. The circuit breaker is reacting to the failures, not to the load that caused them. Any rerouting of traffic that follows comes from your fallback or load balancing strategy. + + +Managing concurrency in a config and fallback setup is outside the scope of circuit breakers. Do not rely on them to cap parallel requests or smooth out traffic spikes. + + +### Managing concurrency and load + +Use the controls built for it: +- [Rate limits](/product/ai-gateway/virtual-keys/rate-limits) to cap requests or tokens on a provider or integration +- [Load balancing](/product/ai-gateway/load-balancing) to spread traffic across targets by weight +- [Fallbacks](/product/ai-gateway/fallbacks) to route to a backup target when a request fails + import PrismaAirsCta from "/snippets/prisma-airs-cta.mdx"; From 4be8558f4564115b7d4775597efe042e39d23c65 Mon Sep 17 00:00:00 2001 From: Akhil Menon Date: Wed, 4 Nov 2026 15:25:17 +0530 Subject: [PATCH 2/3] Addded use case and subtle nudge on concurrecny with CBs --- aigw/product/ai-gateway/circuit-breaker.mdx | 31 ++++----------------- product/ai-gateway/circuit-breaker.mdx | 30 +++----------------- 2 files changed, 9 insertions(+), 52 deletions(-) diff --git a/aigw/product/ai-gateway/circuit-breaker.mdx b/aigw/product/ai-gateway/circuit-breaker.mdx index 680d6a0da..e8bff87e0 100644 --- a/aigw/product/ai-gateway/circuit-breaker.mdx +++ b/aigw/product/ai-gateway/circuit-breaker.mdx @@ -5,10 +5,6 @@ description: Automatically stop routing to unhealthy targets until they recover. Circuit breakers prevent cascading failures by temporarily blocking requests to targets that are failing. - -Circuit breakers react to failures only. They do not manage concurrency. See [Circuit Breakers and Concurrency](#circuit-breakers-and-concurrency). - - ## Config Schema | Field | Description | @@ -81,31 +77,14 @@ flowchart TD - Unhealthy targets removed from routing - If all targets are OPEN, circuit breaker is bypassed -## Circuit Breakers and Concurrency - -Circuit breakers are a **health mechanism, not a concurrency control**. They open only when a target is failing, and they have no concept of how many requests are running at the same time. - -**What circuit breakers do** -- Count failures and successes per strategy path -- Open when `failure_threshold` or `failure_threshold_percentage` is exceeded, counting only responses matching `failure_status_codes` -- Close again after `cooldown_interval` +## Where Circuit Breakers Fit -**What circuit breakers do not do** -- Limit, queue, or throttle concurrent requests to a target -- Use the number of simultaneous requests as an input to open or close the circuit -- Open because of a burst of parallel requests. Only failures open the circuit. +Circuit breakers protect your setup from unhealthy targets. They watch for failures and open when a target stops responding reliably, so the gateway can route around it until it recovers. Since they are driven by failure signals, the number of requests in flight doesn't play a part in when a circuit opens or closes. -### Why concurrency can seem relevant +If your goal is to shape traffic or manage load, for example handling bursts of parallel requests, these work well alongside circuit breakers in a config: -The question usually comes up when circuit breakers are combined with [configs](/aigw/product/ai-gateway/configs) and [fallbacks](/aigw/product/ai-gateway/fallbacks). For example, a provider that rejects requests under heavy load may return `429`. If `429` is in `failure_status_codes`, those responses count as failures and can open the circuit. The circuit breaker is reacting to the failures, not to the load that caused them. Any rerouting of traffic that follows comes from your fallback or load balancing strategy. - - -Managing concurrency in a config and fallback setup is outside the scope of circuit breakers. Do not rely on them to cap parallel requests or smooth out traffic spikes. - - -### Managing concurrency and load - -Use the controls built for it: - [Rate limits](/aigw/product/ai-gateway/virtual-keys/rate-limits) to cap requests or tokens on a provider or integration - [Load balancing](/aigw/product/ai-gateway/load-balancing) to spread traffic across targets by weight - [Fallbacks](/aigw/product/ai-gateway/fallbacks) to route to a backup target when a request fails + +Each handles its own part. For example, if a provider returns `429` under heavy load and `429` is in `failure_status_codes`, the circuit breaker counts those responses as failures and can open the circuit, while your fallback or load balancing strategy decides where traffic goes next. diff --git a/product/ai-gateway/circuit-breaker.mdx b/product/ai-gateway/circuit-breaker.mdx index 9c96a9b5e..62c884826 100644 --- a/product/ai-gateway/circuit-breaker.mdx +++ b/product/ai-gateway/circuit-breaker.mdx @@ -9,10 +9,6 @@ Available on all Portkey [plans](https://portkey.ai/pricing). Circuit breakers prevent cascading failures by temporarily blocking requests to targets that are failing. - -Circuit breakers react to failures only. They do not manage concurrency. See [Circuit Breakers and Concurrency](#circuit-breakers-and-concurrency). - - ## Config Schema | Field | Description | @@ -85,35 +81,17 @@ flowchart TD - Unhealthy targets removed from routing - If all targets are OPEN, circuit breaker is bypassed -## Circuit Breakers and Concurrency - -Circuit breakers are a **health mechanism, not a concurrency control**. They open only when a target is failing, and they have no concept of how many requests are running at the same time. - -**What circuit breakers do** -- Count failures and successes per strategy path -- Open when `failure_threshold` or `failure_threshold_percentage` is exceeded, counting only responses matching `failure_status_codes` -- Close again after `cooldown_interval` - -**What circuit breakers do not do** -- Limit, queue, or throttle concurrent requests to a target -- Use the number of simultaneous requests as an input to open or close the circuit -- Open because of a burst of parallel requests. Only failures open the circuit. +## Where Circuit Breakers Fit -### Why concurrency can seem relevant - -The question usually comes up when circuit breakers are combined with [configs](/product/ai-gateway/configs) and [fallbacks](/product/ai-gateway/fallbacks). For example, a provider that rejects requests under heavy load may return `429`. If `429` is in `failure_status_codes`, those responses count as failures and can open the circuit. The circuit breaker is reacting to the failures, not to the load that caused them. Any rerouting of traffic that follows comes from your fallback or load balancing strategy. - - -Managing concurrency in a config and fallback setup is outside the scope of circuit breakers. Do not rely on them to cap parallel requests or smooth out traffic spikes. - +Circuit breakers protect your setup from unhealthy targets. They watch for failures and open when a target stops responding reliably, so the gateway can route around it until it recovers. Since they are driven by failure signals, the number of requests in flight doesn't play a part in when a circuit opens or closes. -### Managing concurrency and load +If your goal is to shape traffic or manage load, for example handling bursts of parallel requests, these work well alongside circuit breakers in a config: -Use the controls built for it: - [Rate limits](/product/ai-gateway/virtual-keys/rate-limits) to cap requests or tokens on a provider or integration - [Load balancing](/product/ai-gateway/load-balancing) to spread traffic across targets by weight - [Fallbacks](/product/ai-gateway/fallbacks) to route to a backup target when a request fails +Each handles its own part. For example, if a provider returns `429` under heavy load and `429` is in `failure_status_codes`, the circuit breaker counts those responses as failures and can open the circuit, while your fallback or load balancing strategy decides where traffic goes next. import PrismaAirsCta from "/snippets/prisma-airs-cta.mdx"; From e922b419648dad65bf1617f9e7f743c7a4e02ae1 Mon Sep 17 00:00:00 2001 From: Akhil Menon Date: Mon, 9 Nov 2026 15:36:50 +0530 Subject: [PATCH 3/3] Modified the concurrecy wordings to sound less absolute --- aigw/product/ai-gateway/circuit-breaker.mdx | 2 +- product/ai-gateway/circuit-breaker.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/aigw/product/ai-gateway/circuit-breaker.mdx b/aigw/product/ai-gateway/circuit-breaker.mdx index e8bff87e0..380846c9a 100644 --- a/aigw/product/ai-gateway/circuit-breaker.mdx +++ b/aigw/product/ai-gateway/circuit-breaker.mdx @@ -79,7 +79,7 @@ flowchart TD ## Where Circuit Breakers Fit -Circuit breakers protect your setup from unhealthy targets. They watch for failures and open when a target stops responding reliably, so the gateway can route around it until it recovers. Since they are driven by failure signals, the number of requests in flight doesn't play a part in when a circuit opens or closes. +Circuit breakers protect your setup from unhealthy targets. They watch for failures and open when a target stops responding reliably, so the gateway can route around it until it recovers. Circuit breakers are failure-based. They open when failures cross your configured thresholds, and they don't limit how many requests run in parallel. If your goal is to shape traffic or manage load, for example handling bursts of parallel requests, these work well alongside circuit breakers in a config: diff --git a/product/ai-gateway/circuit-breaker.mdx b/product/ai-gateway/circuit-breaker.mdx index 62c884826..9c39ed348 100644 --- a/product/ai-gateway/circuit-breaker.mdx +++ b/product/ai-gateway/circuit-breaker.mdx @@ -83,7 +83,7 @@ flowchart TD ## Where Circuit Breakers Fit -Circuit breakers protect your setup from unhealthy targets. They watch for failures and open when a target stops responding reliably, so the gateway can route around it until it recovers. Since they are driven by failure signals, the number of requests in flight doesn't play a part in when a circuit opens or closes. +Circuit breakers protect your setup from unhealthy targets. They watch for failures and open when a target stops responding reliably, so the gateway can route around it until it recovers. Circuit breakers are failure-based. They open when failures cross your configured thresholds, and they don't limit how many requests run in parallel. If your goal is to shape traffic or manage load, for example handling bursts of parallel requests, these work well alongside circuit breakers in a config: