← All tutorials
Concurrency Design

Concurrency design β€” capacity, resilience & agent isolation

Concurrency is how many units of work are in flight at once. Designing it well means balancing four forces that pull against each other β€” security, idempotency, latency and infra capacity β€” then sizing, testing and hardening the system so a single failure stays isolated. This guide covers all of it, including the new wrinkle: AI agents.

The core tension. More concurrency β†’ more throughput and better hardware use, but also more contention, tail latency, cost, and blast radius when something fails. There's no single "right" number β€” you design for a target, then find the safe limit empirically.

Framing  The four forces

Every concurrency decision trades off these four. Name them explicitly for your system:

πŸ”’ Security & isolation

Concurrent work must not leak across contexts (tenants, users, sessions). Shared mutable state and race conditions cause data bleed and auth bugs.

♻️ Idempotency

At scale, retries and duplicates are inevitable. Operations must be safe to run more than once β€” so a retry never double-charges or double-sends.

⏱️ Latency

Queueing adds delay. Past a point, more concurrency increases tail latency (p99) even as throughput plateaus.

πŸ—οΈ Infra capacity

The scarcest downstream resource β€” DB connections, memory, an API rate limit β€” is your real ceiling. Concurrency above it just queues or fails.

Design principle: bound everything (queues, pools, in-flight work), isolate failure (bulkheads), and make work idempotent so you can safely retry. Then tune to the capacity you actually have.

Trade-offs  Architecture options

Three common shapes, from simplest/fastest to most isolated/durable. Pick per workload β€” there's a real performance ↔ security/durability ↔ cost trade-off.

Option A Β· In-process async + limits

An event loop (or small thread pool) handles many requests, capped by a semaphore. Perf β˜…β˜…β˜… Isolation β˜… Cost β˜…β˜…β˜…

Lowest latency & cost, simplest. But weak isolation and durability β€” a crash loses in-flight work, and one bad request can affect neighbours. Best for stateless, short, low-risk work.

Option B Β· Worker pool + bounded queue

Requests enter a bounded queue; a fixed pool of workers drains it (producer/consumer). Perf β˜…β˜… Isolation β˜…β˜… Cost β˜…β˜…

The queue gives natural backpressure (reject/shed when full); the pool caps concurrency. Good middle ground for most services. Still limited by single-process durability.

Option C Β· Distributed durable queue + isolated workers

A durable queue (SQS/Kafka/…) feeds autoscaling, isolated workers (containers or serverless). Perf β˜… Isolation β˜…β˜…β˜… Cost β˜…

Strongest isolation and durability, elastic capacity, survives crashes (redelivery). Costs more latency, money and complexity. Best for long, expensive, multi-tenant work β€” and for AI agents.

Rule of thumb. Start at the simplest option that meets your isolation and durability needs, not the most scalable one. You can graduate A β†’ B β†’ C as load and risk grow β€” the concurrency limits and idempotency you build carry over.

Capacity  Sizing β€” how much concurrency?

Start with math, not a guess. Little's Law relates the three quantities:

Little's Law
concurrency (L)  =  throughput (Ξ»)  Γ—  latency (W)

# Example: you want 1,000 req/s and each takes 0.2s of in-flight time
L = 1000 /s Γ— 0.2 s = 200 concurrent requests in flight

# So size pools/limits around ~200 β€” but ONLY if downstream can take it.

Then bound it by reality:

  • Downstream is the ceiling. Never allow more concurrency than the scarcest dependency (DB connection pool, an LLM API's rate limit) can serve β€” set your limit at or below it.
  • I/O-bound thread count β‰ˆ cores Γ— (1 + wait_time / compute_time) β€” lots of waiting means more threads help; CPU-bound work does not.
  • Diminishing returns. Contention and coordination mean throughput plateaus and can even drop as you add concurrency (the Universal Scalability Law). More is not always more.
Practical recipe: compute a target with Little's Law β†’ cap it at downstream capacity β†’ load-test to find the real knee (next section) β†’ set the operating limit just below it with headroom for spikes.

Verify  Testing β€” identify the limit

You can't reason your way to the exact number β€” you load-test it. Ramp concurrency and plot throughput, latency percentiles and errors. The curves reveal the knee: where throughput stops rising and latency takes off.

Throughput and latency versus concurrency As concurrency rises, throughput increases then plateaus and slightly declines, while latency stays low then rises sharply. The knee β€” where throughput flattens and latency starts climbing β€” marks the safe operating limit. knee β†’ set limit here saturation ⚠️ throughput latency (p99) concurrency β†’
TestQuestion it answers
Load / rampWhere's the knee? What throughput & p99 at each concurrency level?
StressWhat happens past saturation β€” graceful shedding, or collapse?
SpikeCan it absorb a sudden burst without falling over?
SoakDoes it degrade over hours (leaks, pool exhaustion)?
ChaosKill a worker / a dependency β€” does one failure stay isolated?

Tools: k6, Locust, Gatling, JMeter, vegeta, wrk. Watch internal signals too: queue depth, pool utilization, connection-pool wait time, CPU/memory, GC pauses β€” these often saturate before the user-facing metrics do.

Harden  Resilience β€” retries for isolated failures

At concurrency, transient failures are constant. The goal: recover from a single, isolated failure without amplifying it into an outage.

Cap in-flight work (bulkhead)

concurrency limit β€” a semaphore/bulkhead
import asyncio
sem = asyncio.Semaphore(200)          # cap in-flight work at the tested limit

async def handle(req):
    async with sem:                   # beyond 200 β†’ callers queue (backpressure)
        return await do_work(req)
# Separate semaphores per dependency = "bulkheads": a slow DB can't
# starve the threads serving cache hits.

Retry β€” safely, and only when it helps

exponential backoff + full jitter
import asyncio, random
TRANSIENT = {408, 429, 500, 502, 503, 504}   # retry these; NOT 400/401/403/404

async def with_retry(fn, attempts=4, base=0.2, cap=5.0):
    for i in range(attempts):
        try:
            return await fn()                 # must be idempotent!
        except Transient:
            if i == attempts - 1: raise
            delay = min(cap, base * 2 ** i) * random.random()   # full jitter
            await asyncio.sleep(delay)
Retries are dangerous at scale. Naive retries cause retry storms that turn a blip into an outage. Always: retry only idempotent ops and transient errors; add jitter; cap attempts and honour a request deadline; and wrap flaky dependencies in a circuit breaker so you stop hammering a service that's already down.

Make retries safe β€” idempotency keys

idempotency key β€” dedupe so retries don't double-act
# Client sends a unique Idempotency-Key; server returns the SAME result
# for a repeat, so a retried "create payment" never charges twice.
def create_payment(idem_key, body):
    if (prev := store.get(idem_key)):     # already processed
        return prev
    result = charge(body)
    store.set(idem_key, result, ttl="24h")
    return result

Isolation  Security under concurrency

Concurrency is a security surface. The failures are subtle because they only show up under load.

🧩 No shared mutable state

Data bleeding between requests (a cached user, a reused buffer) leaks one caller's data to another. Use per-request context, immutability, no globals.

🏁 Race conditions

Two requests checking-then-acting can bypass limits or double-spend. Use atomic ops or optimistic concurrency (version / compare-and-swap).

🏠 Tenant isolation

Per-tenant concurrency caps and scoped credentials stop one tenant starving others (a "noisy neighbour") or reaching their data.

🚧 Resource limits = DoS defense

Bounded pools, payload caps and rate limits keep a flood (or an attack) from exhausting memory/connections.

The new wrinkle  AI agent concurrency

Agents break the usual assumptions: runs are long (seconds–minutes) and variable, expensive (tokens/cost), stateful (memory), non-deterministic, and gated by LLM API rate limits. And each run holds a rich context that must never bleed into another.

Context isolation is the top concern

Every concurrent agent run needs its own memory, tools, credentials and budget β€” keyed by session/tenant. Shared context is both a correctness bug (one user's history contaminating another's) and a security breach (cross-tenant data/tool access).

per-run isolated context
async def run_agent(session_id, tenant_id, prompt):
    async with tenant_sem[tenant_id]:                 # per-tenant concurrency cap
        ctx = Context(
            memory = memory_store.scope(session_id),   # isolated per session
            tools  = tools_for(tenant_id),             # scoped, least-privilege
            creds  = vault.scoped(tenant_id),          # tenant-scoped credentials
            budget = Budget(max_tokens=100_000, max_usd=2.0),
        )
        return await agent.run(prompt, ctx)            # ctx is the ONLY state; no globals

Reference architecture

client→ gateway: rate-limit · concurrency cap · idempotency→ durable queue→ agent workers (isolated context)→ scoped tools

This is Option C for good reason: durable queues absorb bursts and let long runs survive restarts; isolated workers give context & blast-radius isolation; a gateway enforces per-tenant limits and dedupes triggers (idempotency) so a retried request doesn't spawn two expensive runs.

Agent concernSolution
Cross-context leakagePer-session memory, scoped tools & credentials, no shared globals
LLM rate limits / cost blowupPer-tenant + global concurrency caps; token & spend budgets
Long, variable runsAsync queue + workers (not request threads); durable so they survive restarts
Tool fan-out (sub-agents)Cap parallel sub-agents; bulkhead tool calls; timeouts
Duplicate triggersIdempotency key per run so a retry doesn't double-execute
Untrusted tool outputIsolate & validate; don't let it cross into another run's context
Trade-off, agent edition. In-process parallel sub-agents are fastest but share a blast radius and are hard to isolate; a queue-and-workers design costs latency but gives you durability, per-tenant fairness, and true context isolation. For multi-tenant or costly agents, choose isolation. (See loops and agent kits.)

Problems & solutions β€” a reference

ProblemSymptomSolution
Race conditionCorrupted or double-counted stateAtomic ops, optimistic concurrency (version/CAS), careful locking
Shared mutable stateData bleeds across requestsPer-request context, immutability, no globals
Resource exhaustionOOM, pool starvation, DoSBounded pools & queues, concurrency limits, load shedding
Retry storm / thundering herdA blip cascades into an outageBackoff + jitter, circuit breaker, request coalescing
Duplicate processingDouble charge / double sendIdempotency keys, consumer-side dedup
Tail latency (p99)Slow outliers hurt usersTimeouts, hedged requests, isolate slow deps (bulkhead)
DeadlockRequests hang foreverConsistent lock ordering, lock timeouts, avoid nested locks
Cross-tenant leakageOne tenant affects/sees anotherContext isolation, per-tenant limits & credentials
Cost blowup (agents)Runaway token spendConcurrency + spend caps, budgets, rate limits

Checklist

Design for…Do
CapacityLittle's Law target Β· cap at downstream limit Β· load-test to the knee
LatencyBound queues Β· timeouts everywhere Β· watch p99, not just averages
IdempotencyIdempotency keys Β· dedup Β· design ops to be safely repeatable
Failure isolationBulkheads Β· circuit breakers Β· retry (idempotent + transient + jitter)
SecurityNo shared mutable state Β· atomic ops Β· per-tenant caps & creds
AgentsIsolated per-run context Β· queue + workers Β· token/spend budgets

Knowledge check

Six questions on concurrency design. Pick an answer for each, then check your score.

1. Little's Law says concurrency β‰ˆ …

2. In a load test, the "knee" is where…

3. When is it safe to automatically retry a failed operation?

4. What stops a retried "create payment" from charging twice?

5. The top concurrency concern for AI agents is…

6. Which pattern isolates a slow dependency so it can't starve everything else?

Next steps

☸️ Cloud Native

Scale it out β€” autoscaling workers and orchestration for Option C.

πŸ›‘οΈ Web Β· API Β· AI Security

The security side β€” isolation, rate limits and least privilege.

πŸ”„ Getting started with loops

Running agents β€” where per-run isolation & budgets matter most.

πŸ““ All tutorials

The hub β€” LLMs, agents, APIs, cloud & more.