Concurrency is how many units of work are in flight at once. Designing it well means balancing four forces that pull against each other β security, idempotency, latency and infra capacity β then sizing, testing and hardening the system so a single failure stays isolated. This guide covers all of it, including the new wrinkle: AI agents.
The core tension. More concurrency β more throughput and better hardware use, but also more contention, tail latency, cost, and blast radius when something fails. There's no single "right" number β you design for a target, then find the safe limit empirically.
Framing The four forces
Every concurrency decision trades off these four. Name them explicitly for your system:
π Security & isolation
Concurrent work must not leak across contexts (tenants, users, sessions). Shared mutable state and race conditions cause data bleed and auth bugs.
β»οΈ Idempotency
At scale, retries and duplicates are inevitable. Operations must be safe to run more than once β so a retry never double-charges or double-sends.
β±οΈ Latency
Queueing adds delay. Past a point, more concurrency increases tail latency (p99) even as throughput plateaus.
ποΈ Infra capacity
The scarcest downstream resource β DB connections, memory, an API rate limit β is your real ceiling. Concurrency above it just queues or fails.
Design principle:bound everything (queues, pools, in-flight work), isolate failure (bulkheads), and make work idempotent so you can safely retry. Then tune to the capacity you actually have.
Trade-offs Architecture options
Three common shapes, from simplest/fastest to most isolated/durable. Pick per workload β there's a real performance β security/durability β cost trade-off.
Option A Β· In-process async + limits
An event loop (or small thread pool) handles many requests, capped by a semaphore. Perf β β β Isolation β Cost β β β
Lowest latency & cost, simplest. But weak isolation and durability β a crash loses in-flight work, and one bad request can affect neighbours. Best for stateless, short, low-risk work.
Option B Β· Worker pool + bounded queue
Requests enter a bounded queue; a fixed pool of workers drains it (producer/consumer). Perf β β Isolation β β Cost β β
The queue gives natural backpressure (reject/shed when full); the pool caps concurrency. Good middle ground for most services. Still limited by single-process durability.
Option C Β· Distributed durable queue + isolated workers
A durable queue (SQS/Kafka/β¦) feeds autoscaling, isolated workers (containers or serverless). Perf β Isolation β β β Cost β
Strongest isolation and durability, elastic capacity, survives crashes (redelivery). Costs more latency, money and complexity. Best for long, expensive, multi-tenant work β and for AI agents.
Rule of thumb. Start at the simplest option that meets your isolation and durability needs, not the most scalable one. You can graduate A β B β C as load and risk grow β the concurrency limits and idempotency you build carry over.
Capacity Sizing β how much concurrency?
Start with math, not a guess. Little's Law relates the three quantities:
Little's Law
concurrency (L) = throughput (Ξ») Γ latency (W)
# Example: you want 1,000 req/s and each takes 0.2s of in-flight time
L = 1000 /s Γ 0.2 s = 200 concurrent requests in flight
# So size pools/limits around ~200 β but ONLY if downstream can take it.
Then bound it by reality:
Downstream is the ceiling. Never allow more concurrency than the scarcest dependency (DB connection pool, an LLM API's rate limit) can serve β set your limit at or below it.
I/O-bound thread count β cores Γ (1 + wait_time / compute_time) β lots of waiting means more threads help; CPU-bound work does not.
Diminishing returns. Contention and coordination mean throughput plateaus and can even drop as you add concurrency (the Universal Scalability Law). More is not always more.
Practical recipe: compute a target with Little's Law β cap it at downstream capacity β load-test to find the real knee (next section) β set the operating limit just below it with headroom for spikes.
Verify Testing β identify the limit
You can't reason your way to the exact number β you load-test it. Ramp concurrency and plot throughput, latency percentiles and errors. The curves reveal the knee: where throughput stops rising and latency takes off.
Test
Question it answers
Load / ramp
Where's the knee? What throughput & p99 at each concurrency level?
Stress
What happens past saturation β graceful shedding, or collapse?
Spike
Can it absorb a sudden burst without falling over?
Soak
Does it degrade over hours (leaks, pool exhaustion)?
Chaos
Kill a worker / a dependency β does one failure stay isolated?
Tools:k6, Locust, Gatling, JMeter, vegeta, wrk. Watch internal signals too: queue depth, pool utilization, connection-pool wait time, CPU/memory, GC pauses β these often saturate before the user-facing metrics do.
Harden Resilience β retries for isolated failures
At concurrency, transient failures are constant. The goal: recover from a single, isolated failure without amplifying it into an outage.
Cap in-flight work (bulkhead)
concurrency limit β a semaphore/bulkhead
import asyncio
sem = asyncio.Semaphore(200) # cap in-flight work at the tested limit
async def handle(req):
async with sem: # beyond 200 β callers queue (backpressure)
return await do_work(req)
# Separate semaphores per dependency = "bulkheads": a slow DB can't# starve the threads serving cache hits.
Retry β safely, and only when it helps
exponential backoff + full jitter
import asyncio, random
TRANSIENT = {408, 429, 500, 502, 503, 504} # retry these; NOT 400/401/403/404
async def with_retry(fn, attempts=4, base=0.2, cap=5.0):
for i in range(attempts):
try:
return await fn() # must be idempotent!
except Transient:
if i == attempts - 1: raise
delay = min(cap, base * 2 ** i) * random.random() # full jitter
await asyncio.sleep(delay)
Retries are dangerous at scale. Naive retries cause retry storms that turn a blip into an outage. Always: retry only idempotent ops and transient errors; add jitter; cap attempts and honour a request deadline; and wrap flaky dependencies in a circuit breaker so you stop hammering a service that's already down.
Make retries safe β idempotency keys
idempotency key β dedupe so retries don't double-act
# Client sends a unique Idempotency-Key; server returns the SAME result# for a repeat, so a retried "create payment" never charges twice.
def create_payment(idem_key, body):
if (prev := store.get(idem_key)): # already processed
return prev
result = charge(body)
store.set(idem_key, result, ttl="24h")
return result
Isolation Security under concurrency
Concurrency is a security surface. The failures are subtle because they only show up under load.
Data bleeding between requests (a cached user, a reused buffer) leaks one caller's data to another. Use per-request context, immutability, no globals.
π Race conditions
Two requests checking-then-acting can bypass limits or double-spend. Use atomic ops or optimistic concurrency (version / compare-and-swap).
π Tenant isolation
Per-tenant concurrency caps and scoped credentials stop one tenant starving others (a "noisy neighbour") or reaching their data.
π§ Resource limits = DoS defense
Bounded pools, payload caps and rate limits keep a flood (or an attack) from exhausting memory/connections.
The new wrinkle AI agent concurrency
Agents break the usual assumptions: runs are long (secondsβminutes) and variable, expensive (tokens/cost), stateful (memory), non-deterministic, and gated by LLM API rate limits. And each run holds a rich context that must never bleed into another.
Context isolation is the top concern
Every concurrent agent run needs its own memory, tools, credentials and budget β keyed by session/tenant. Shared context is both a correctness bug (one user's history contaminating another's) and a security breach (cross-tenant data/tool access).
per-run isolated context
async def run_agent(session_id, tenant_id, prompt):
async with tenant_sem[tenant_id]: # per-tenant concurrency cap
ctx = Context(
memory = memory_store.scope(session_id), # isolated per session
tools = tools_for(tenant_id), # scoped, least-privilege
creds = vault.scoped(tenant_id), # tenant-scoped credentials
budget = Budget(max_tokens=100_000, max_usd=2.0),
)
return await agent.run(prompt, ctx) # ctx is the ONLY state; no globals
This is Option C for good reason: durable queues absorb bursts and let long runs survive restarts; isolated workers give context & blast-radius isolation; a gateway enforces per-tenant limits and dedupes triggers (idempotency) so a retried request doesn't spawn two expensive runs.
Agent concern
Solution
Cross-context leakage
Per-session memory, scoped tools & credentials, no shared globals
LLM rate limits / cost blowup
Per-tenant + global concurrency caps; token & spend budgets
Long, variable runs
Async queue + workers (not request threads); durable so they survive restarts
Tool fan-out (sub-agents)
Cap parallel sub-agents; bulkhead tool calls; timeouts
Duplicate triggers
Idempotency key per run so a retry doesn't double-execute
Untrusted tool output
Isolate & validate; don't let it cross into another run's context
Trade-off, agent edition. In-process parallel sub-agents are fastest but share a blast radius and are hard to isolate; a queue-and-workers design costs latency but gives you durability, per-tenant fairness, and true context isolation. For multi-tenant or costly agents, choose isolation. (See loops and agent kits.)