Backend Engineering — From First Principles

Request Flow End-to-End

Browser → DNS → TCP → TLS → HTTP → LB → server → response. Every hop matters.

The full path

Hops & what each does

The response: structure

HTTP/2 200 OK
content-type: application/json; charset=utf-8
content-length: 87
cache-control: private, max-age=60
x-request-id: 7f3a-1c2b
date: Mon, 11 May 2026 09:14:22 GMT

{"id": 42, "name": "alice", "email": "[email protected]"}

HTTP Protocol

Message structure, headers, methods, CORS, status codes, caching, versions, TLS.

Raw message format

# request
POST /api/users HTTP/1.1
Host: api.example.com
Content-Type: application/json
Authorization: Bearer eyJ...
Content-Length: 41

{"email":"[email protected]","password":"hunter2"}

# response
HTTP/1.1 201 Created
Content-Type: application/json
Location: /api/users/42

{"id":42,"email":"[email protected]"}

Header families

Methods & semantics

CORS — Cross-Origin Resource Sharing

Browser enforces same-origin policy. Server opts other origins in via headers.

Simple request

• Methods GET/HEAD/POST only.
• Only "safelisted" headers (Accept, Content-Language, Content-Type: text/plain | application/x-www-form-urlencoded | multipart/form-data).
• Browser sends directly with Origin: https://app.x.com. Server returns Access-Control-Allow-Origin.

Pre-flight

# browser sends first:
OPTIONS /api/users HTTP/1.1
Origin: https://app.x.com
Access-Control-Request-Method: PUT
Access-Control-Request-Headers: authorization, content-type

# server replies:
HTTP/1.1 204 No Content
Access-Control-Allow-Origin: https://app.x.com
Access-Control-Allow-Methods: GET, POST, PUT, DELETE
Access-Control-Allow-Headers: authorization, content-type
Access-Control-Allow-Credentials: true
Access-Control-Max-Age: 86400

Status codes — ones that matter

Caching: ETag vs max-age

# first response carries ETag
HTTP/1.1 200 OK
ETag: "v1-7f3a"
Cache-Control: max-age=60, must-revalidate

# subsequent request — conditional
GET /api/users/42
If-None-Match: "v1-7f3a"

# server unchanged → no body
HTTP/1.1 304 Not Modified
ETag: "v1-7f3a"

• Strong validators: ETag (byte-exact), Last-Modified (second precision).
• Freshness: max-age=N = fresh for N sec, no server hit.
• private = only end-user cache. public = CDN can cache.
• no-store = never persist (sensitive data). no-cache = revalidate every time.

HTTP versions

Content negotiation & compression

Accept: application/json;q=0.9, application/xml;q=0.5
Accept-Encoding: gzip, br, zstd
Accept-Language: en-US, en;q=0.8

# server picks best match, replies:
Content-Type: application/json
Content-Encoding: br

TLS / HTTPS

• TLS provides confidentiality (encryption), integrity (MAC), authentication (cert chain).
• TLS 1.3 handshake: 1 RTT (vs 2 in 1.2), 0-RTT for session resumption.
• Cert chain: leaf → intermediate → root (trust anchor in OS).
• SNI lets one IP host many TLS sites (sends hostname in ClientHello).
• HSTS header Strict-Transport-Security: max-age=31536000; includeSubDomains; preload forces HTTPS for year.

Routing

URL → handler. Method-aware. Versioned. Grouped. Fast.

Route components

GET /api/v1/users/:userId/posts?status=published&limit=20
     │   │   │     │           │
     │   │   │     │           └─ query params (filters, paging)
     │   │   │     └─ path param (resource id)
     │   │   └─ resource (collection)
     │   └─ version
     └─ namespace

Route types

API versioning strategies

Deprecation pattern

HTTP/1.1 200 OK
Deprecation: true
Sunset: Wed, 01 Jan 2027 00:00:00 GMT
Link: <https://api.x.com/v2/users>; rel="successor-version"
Warning: 299 - "v1 deprecated; migrate to v2 by 2027-01-01"

Route grouping

# pseudo-framework
group("/api/v1", middleware=[logger, requestId]) {
  group("/auth", middleware=[rateLimit(5, "1m")]) {
    POST("/login",   loginHandler)
    POST("/refresh", refreshHandler)
  }
  group("/admin", middleware=[requireAuth, requireRole("admin")]) {
    GET("/users",         listUsers)
    DELETE("/users/:id",  deleteUser)
  }
}

Serialization & Deserialization

Native ↔ wire bytes. Pick format by audience + perf budget.

Text vs binary

JSON deep-dive

{
  "string": "hello",
  "int":    42,
  "float":  3.14,
  "bool":   true,
  "null":   null,
  "array":  [1, 2, 3],
  "nested": { "k": "v" },
  "date":   "2026-05-11T09:14:22Z"   // ISO-8601 with offset
}

Native mapping

Edge cases

• Missing fields — apply defaults; use Optional[T]/pointers to distinguish absent vs null.
• Extra fields — strict mode reject, lenient ignore. Default to reject for inbound user data.
• Numbers — JS number = float64, loses precision past 2^53. Send big ints as strings.
• Dates — ISO-8601 with timezone offset (Z = UTC). Never plain "2026-05-11 09:14:22".
• Floats — money in cents (integer) or decimal-string. Never float for currency.
• Null vs absent — for PATCH, "absent" = leave alone, "null" = clear field.

Schema validation (JSON Schema)

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["email", "age"],
  "additionalProperties": false,
  "properties": {
    "email": { "type": "string", "format": "email", "maxLength": 254 },
    "age":   { "type": "integer", "minimum": 0, "maximum": 150 },
    "tags":  { "type": "array", "items": { "type": "string" }, "maxItems": 20 }
  }
}

Authentication, Authorization, Security

Who you are, what you can do, how attackers try to get past it.

Authentication mechanisms

OAuth 2.0 + PKCE

The authorization code grant with PKCE (Proof Key for Code Exchange, RFC 7636) is the modern default. The client picks a random code_verifier, derives code_challenge = BASE64URL(SHA-256(code_verifier)) with code_challenge_method=S256, and binds the two halves of the flow together.

# 1. client generates a high-entropy secret
code_verifier  = base64url(random 32 bytes)          # 43–128 chars
code_challenge = base64url(sha256(code_verifier))    # S256

# 2. redirect user to /authorize (sends the CHALLENGE only)
GET /authorize?response_type=code
   &client_id=spa
   &redirect_uri=https://app.x.com/cb
   &scope=read:users
   &state=xyz&code_challenge=E9Mq...&code_challenge_method=S256

# 3. user authenticates → redirected back with a one-time code
GET https://app.x.com/cb?code=AUTH_CODE&state=xyz

# 4. client exchanges code at /token (sends the VERIFIER)
POST /token
grant_type=authorization_code&code=AUTH_CODE
   &redirect_uri=https://app.x.com/cb
   &client_id=spa&code_verifier=dBjftJ...
# server recomputes sha256(code_verifier), compares to stored challenge → issues tokens

• Why it works: an attacker who intercepts the authorization code (the redirect leg) still cannot redeem it without the code_verifier, which never left the client. Defeats authorization-code interception on public clients.
• Implicit grant is dead: returning tokens directly in the redirect fragment is removed by the OAuth 2.0 Security BCP and OAuth 2.1 — use code + PKCE instead.
• PKCE for everyone: OAuth 2.1 consolidates the spec and makes PKCE the default even for confidential (server-side) clients, on top of the client secret.

JWT anatomy

header.payload.signature

# header
{ "alg": "RS256", "typ": "JWT", "kid": "key-2026-q2" }

# payload (claims)
{
  "sub": "user_42",
  "iss": "https://auth.x.com",
  "aud": "api.x.com",
  "exp": 1715420000,
  "iat": 1715416400,
  "scope": "read:users write:posts"
}

# signature = sign(base64(header) + "." + base64(payload), private_key)

JWT pitfalls

• Algorithm confusion: validate alg against allow-list. Reject none. Don't trust kid blindly.
• No revocation: tokens valid until exp. Use short TTL (5–15 min) + refresh tokens for logout.
• Stored payload is public: signed, not encrypted. Don't put secrets in claims.
• Clock skew: allow ±60s on exp/nbf.

Password storage

# NEVER: plaintext, MD5, SHA-1, SHA-256 plain
# CORRECT: slow KDF with per-user salt
hash = argon2id(password, salt, m=64MB, t=3, p=1)
# alternatives: bcrypt (cost ≥ 12), scrypt, PBKDF2 (≥ 600k iter)

Authorization models

OWASP-style attacks & defenses

Secure design principles

• Least privilege — give each subject minimum it needs.
• Defense in depth — overlapping layers (WAF, app validation, DB constraints).
• Fail secure — when in doubt, deny. Don't open-default on errors.
• Separation of duties — same human can't approve and execute payouts.
• CSP — Content-Security-Policy: default-src 'self'; script-src 'self' 'nonce-xyz'.
• SameSite cookies — Set-Cookie: session=...; HttpOnly; Secure; SameSite=Lax.

Attack prevention practices

• Audit-log failed logins, privilege escalations, admin actions. Tamper-evident store.
• Generic error messages on auth ("invalid credentials" — don't leak which part wrong).
• Rate limit per-IP + per-account. Exponential backoff. Lock after N failures.
• Constant-time compare for tokens/HMAC (hmac.compare_digest) — avoid timing attacks.

Validation, Transformation, Normalization

Fail fast on bad input. Normalize before processing. Sanitize before storing.

Three validation types

Client vs server

• Client-side validation = UX (instant feedback). Cannot be trusted.
• Server-side validation = security. Always re-validate.
• Fail fast: validate at edge, before middleware does work.

Transform & normalize

email   = email.strip().lower()
phone   = re.sub(r'\D', '', phone)        # digits only
country = country.upper()                  # "us" → "US"
name    = ' '.join(name.split())           # collapse spaces
slug    = slugify(title)                   # "Hello World!" → "hello-world"

Sanitization (escape, don't trust)

# HTML
clean_html = bleach.clean(user_html, tags=['p','b','i'], strip=True)
# SQL — never string-format
cur.execute("SELECT * FROM users WHERE id = %s", (user_id,))
# Shell — avoid; if must, use shlex.quote

Complex rules

• Relationship: password == confirmPassword.
• Conditional: if type == "business", then taxId required.
• Chained: parse → type-check → range-check → cross-field check.

Error aggregation

HTTP/1.1 422 Unprocessable Entity
Content-Type: application/problem+json

{
  "type": "https://x.com/errors/validation",
  "title": "Validation failed",
  "status": 422,
  "errors": [
    { "field": "email", "code": "invalid_format", "message": "not a valid email" },
    { "field": "age",   "code": "out_of_range",   "message": "must be 0–150" }
  ]
}

Middleware

Cross-cutting logic in chain. Order matters more than content.

What middleware does

• Run code before handler (parse, auth, log start).
• Run code after handler (log status, add headers, compress).
• Short-circuit — return early (401, 429, 404) without calling next.

Canonical ordering

Common middlewares

Rate-limiting algorithms

Keep middleware lightweight

• Every middleware runs on every request. 1 ms each × 10 middlewares = 10 ms baseline.
• Heavy work (image processing, external API calls) belongs in handler/job, not middleware.
• Cache decisions (e.g., JWKS public keys) — don't refetch per request.

Request Context

Per-request scratch space that flows with the call — without leaking across requests.

What lives in context

• Metadata: URL, headers, method, remote IP, start time.
• Identity: userId, orgId, scopes after auth middleware.
• Tracing: requestId, traceId, spanId for correlation.
• Cancellation: timeout / abort signal for downstream calls.
• DB conn / transaction: scoped per request so all reads see same snapshot.

Patterns

Request ID propagation

# inbound middleware
const reqId = req.headers['x-request-id'] ?? randomUUID()
res.setHeader('x-request-id', reqId)
ctx.set('requestId', reqId)
logger.child({ reqId })

# outbound HTTP calls
fetch(url, { headers: { 'x-request-id': ctx.get('requestId') }})

Timeouts & cancellation

# Go
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
defer cancel()
row := db.QueryRowContext(ctx, "SELECT ...")    # aborts on timeout

# Node
const ctrl = new AbortController()
setTimeout(() => ctrl.abort(), 2000)
await fetch(url, { signal: ctrl.signal })

MVC, Controllers, REST APIs

Separation of concerns inside request path.

Layered responsibility

CRUD ↔ HTTP mapping

POST   /users           → create
GET    /users           → list (paginated)
GET    /users/:id       → read
PUT    /users/:id       → full replace
PATCH  /users/:id       → partial update
DELETE /users/:id       → remove
POST   /users/:id/reset → action (non-CRUD verb)

Standard list response

{
  "data": [ { "id": 1, "name": "alice" }, ... ],
  "meta": {
    "page":  2,
    "limit": 20,
    "total": 137,
    "hasMore": true
  },
  "links": {
    "self": "/users?page=2&limit=20",
    "next": "/users?page=3&limit=20",
    "prev": "/users?page=1&limit=20"
  }
}

Pagination styles

Search / sort / filter

GET /products?q=phone&category=electronics&minPrice=100&sort=-price,name&page=2

# parsed:
{
  q:          "phone",
  filters:    { category: "electronics", price: { gte: 100 } },
  sort:       [{ field: "price", dir: "desc" }, { field: "name", dir: "asc" }],
  page:       2
}

REST principles

• Resource-oriented — nouns in URLs, verbs in methods.
• Stateless — every request carries enough auth/context.
• Cacheable — GETs use ETags / max-age.
• Uniform interface — consistent shape across resources.
• HATEOAS (optional) — responses embed links to next actions.
• Redact sensitive fields — never serialize password_hash, even hashed.
• OpenAPI spec — define contract first; generate client/server stubs.

Databases

Storage shape, consistency, indexing, query plans, ORMs.

Relational vs non-relational

ACID

• Atomicity — all-or-nothing within txn.
• Consistency — txn moves DB between valid states (constraints hold).
• Isolation — concurrent txns don't see each other mid-flight. Levels: Read Uncommitted → Read Committed → Repeatable Read → Serializable.
• Durability — committed data survives crash (fsync to disk).

Isolation levels vs anomalies

*ANSI permits phantoms at Repeatable Read; Postgres RR is snapshot isolation and happens to block phantoms but still allows write skew.

• MVCC (multi-version concurrency control): each writer creates a new row version, so readers never block writers and writers never block readers — Postgres implements Read Committed and Repeatable Read this way.
• Snapshot isolation: Postgres Repeatable Read = a consistent snapshot taken at txn start. It still permits write skew (two txns each read an overlapping set, then write disjoint rows that jointly violate an invariant).
• SERIALIZABLE in Postgres uses SSI (serializable snapshot isolation): it detects dangerous read/write dependency cycles and aborts one txn — retry on a serialization failure.

CAP theorem

Under network Partition, must choose: Consistency (reject reads) or Availability (serve possibly-stale). Real systems pick on partition — most of the time partitions are rare and you have both.

• CP: HBase, MongoDB (default), etcd, ZooKeeper.
• AP: Cassandra, DynamoDB (tunable), Riak.
• PACELC extension: even without partition (E), trade latency (L) vs consistency (C).

Indexing — rules

• B-tree indexes power range + equality on leading column(s).
• Composite index (a, b, c) serves WHERE a=?, WHERE a=? AND b=?, not WHERE b=? alone.
• Covering index: include all SELECT columns → "index-only scan", no heap fetch.
• Each index costs writes (update on insert/update/delete). Audit unused.
• Hash indexes: equality only. GIN/GIST: full-text, JSONB, arrays. BRIN: huge append-only tables.

Query optimization workflow

EXPLAIN ANALYZE
SELECT u.name, COUNT(o.id)
FROM users u
JOIN orders o ON o.user_id = u.id
WHERE u.country = 'US' AND o.created_at > now() - interval '30 day'
GROUP BY u.name;

# look for:
#   Seq Scan on big table         → missing index
#   high "rows removed by filter" → predicate not pushed to index
#   Sort spilled to disk          → work_mem too low
#   Nested Loop on big rowcounts  → expected Hash/Merge join

Connection pooling

• Opening Postgres conn ≈ 5–50 ms. Pool to reuse.
• Pool size ≈ min(N_cores * 2 + spindles, db_max_connections / instances).
• For serverless / many instances → use PgBouncer in transaction mode.
• Always set acquire timeout + max-lifetime to recycle stale conns.

Constraints & transactions

BEGIN;
-- lock both rows in a fixed order (ascending id) to avoid deadlock:
  SELECT balance FROM accounts WHERE id IN ($1, $2) ORDER BY id FOR UPDATE;
  UPDATE accounts SET balance = balance - 100 WHERE id = $1;
  UPDATE accounts SET balance = balance + 100 WHERE id = $2;
  INSERT INTO transfers (from_id, to_id, amount) VALUES ($1, $2, 100);
COMMIT;

-- table constraints catch invariants:
CHECK (balance >= 0)
UNIQUE (email)
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE

ORMs & migrations

• ORMs (Prisma, SQLAlchemy, GORM, Hibernate) trade SQL control for ergonomics.
• Watch for N+1: users.all() then user.posts in loop → use eager load / JOIN.
• Migrations: forward-only, idempotent, reviewed in code. Tools: flyway, alembic, knex, goose, prisma migrate.
• Online schema changes for big tables: gh-ost, pt-osc, or expand-contract pattern.

Business Logic Layer

Where rules live. Independent of HTTP and DB drivers.

Three-layer architecture

Why split

• Testability — unit-test business rules without spinning up HTTP/DB.
• Reuse — same service from REST, gRPC, CLI, cron.
• Swap adapters — Postgres → DynamoDB; HTTP → message queue. Only boundary changes.

SOLID applied

Error propagation pattern

# BLL throws domain errors
class DomainError(Exception): ...
class NotFound(DomainError): ...
class Forbidden(DomainError): ...
class Conflict(DomainError): ...

# presentation layer maps to HTTP
{
  NotFound:  404,
  Forbidden: 403,
  Conflict:  409,
  Validation: 422,
  DomainError: 500,
}[type(e)]

Caching

Trade staleness for latency. Choose layer, strategy, eviction with intent.

Caching layers

Strategies

Eviction policies

• LRU — discard least-recently used. Good general default.
• LFU — least-frequently used. Better for skewed access.
• TTL — time-based expiry. Combine with LRU.
• FIFO — simple ring; ignores access patterns.
• Manual — explicit cache.delete(key) on write.
• Event-based — pub/sub invalidates across instances.

Invalidation patterns

# by key
cache.delete(f"user:{id}")

# by tag (Redis-stack, Varnish)
cache.delete_by_tag(f"user:{id}")

# fan-out (pub/sub)
pubsub.publish("cache:invalidate", {"keys": [f"user:{id}"]})

# versioned key — never need to delete
cache.set(f"user:{id}:v{version}", data)
# bump version on write → old key TTLs out naturally

Use-case recipes

• Hot read DB joins → store materialized join in Redis hash; TTL 60s.
• API responses → cache JSON by URL + auth-scope; vary on user.
• Session → Redis with TTL = session lifetime; sliding refresh.
• Rate limit counters → Redis INCR with EXPIRE.
• Idempotency keys → Redis 24h to dedupe POST retries.

Queues, Background Jobs, Emails

Don't make user wait. Hand off to workers.

What belongs off request path

• Email / SMS / push notifications.
• Image / video transcoding, thumbnail generation.
• Third-party API calls (especially slow/unreliable ones).
• Heavy DB aggregations, report generation.
• Webhook delivery to customers (with retries).
• Periodic maintenance: backups, cleanups, log rotation.

Architecture

Broker comparison

Job semantics

• At-least-once delivery is realistic default → jobs must be idempotent.
• Idempotency key on producer side: stable hash of payload → dedupe at consumer.
• Retries with exponential backoff + jitter; cap attempts; route exhausted → DLQ.
• Dead Letter Queue: failures land here for human inspection / replay.
• Visibility timeout (SQS) / ack-window: if consumer crashes mid-job, message re-appears.

Transactional outbox

"Update the DB and publish an event atomically" has no safe answer with a naive dual write (write DB, then call the broker): if the commit succeeds but the publish fails — or vice versa — the two stores diverge and there is no rollback across them.

-- write the event in the SAME transaction as the state change
BEGIN;
  INSERT INTO orders (id, status) VALUES ($1, 'paid');
  INSERT INTO outbox (id, topic, payload, published)
    VALUES ($2, 'order.paid', $3, false);
COMMIT;
-- a separate relay (poll the outbox, or Debezium CDC tailing the WAL)
-- reads unpublished rows, publishes to the broker, then marks them done

• The state change and the outbox row commit together, so the event is durable iff the data change is. No dual-write window.
• The relay publishes at-least-once (it may re-publish a row if it crashes before marking it done), so consumers must still be idempotent — this gives effectively-exactly-once processing on top of an at-least-once broker.

Chaining & concurrency

# BullMQ-style flow
const flow = new FlowProducer()
await flow.add({
  name: 'order-complete',
  queueName: 'orders',
  children: [
    { name: 'charge-card',     queueName: 'payments' },
    { name: 'send-receipt',    queueName: 'email'    },
    { name: 'update-warehouse',queueName: 'inventory'},
  ],
})
// parent runs only after all children succeed

Transactional email anatomy

Subject:    Your order #4582 is confirmed
Preheader:  Track shipping below • Need help? Reply to this email.
Body:
  Hi Alice, thanks for your order...
  [ Track shipment ]    ← single CTA
Footer:     Unsubscribe • Address (CAN-SPAM)

• Templating with merge vars ({{firstName}}); HTML + plaintext multipart.
• SPF + DKIM + DMARC on sending domain — else inbox spam.
• Track bounce/complaint webhooks; suppress repeats.

Scheduling

• Cron (system / Kubernetes CronJob) — periodic. Use UTC, deal with DST in app code.
• Delayed jobs — schedule for future timestamp; broker handles dispatch.
• Distributed lock for "run on exactly one instance" — Redis SET NX, etcd lease.

Elasticsearch

Inverted index for full-text + analytics, plus vector / hybrid search for semantic + RAG retrieval at scale.

Internals

• Inverted index: term → list of docs containing it. Built from tokenized + analyzed text.
• Segment: immutable Lucene chunk on disk. Writes create new segments; periodic merge.
• Shard: a Lucene index. ES index is N shards distributed across nodes.
• Replica: shard copy for HA + read scaling.
• Term frequency (TF) + IDF + length norm = BM25 relevance score (default).

Use cases

• Type-ahead / autocomplete (edge n-grams or completion suggester).
• Full-text product / article search with relevance.
• Log analytics — ELK / OpenSearch ingesting JSON logs.
• Fuzzy matching (typos, "did you mean").
• Aggregations: top-N, time-series buckets, percentiles.
• Semantic / RAG retrieval — kNN vector search + hybrid (BM25 + vector) ranking.

Query patterns

POST /products/_search
{
  "query": {
    "bool": {
      "must":   [{ "match": { "title": "wireless headphones" } }],
      "filter": [
        { "term":  { "category": "audio" } },
        { "range": { "price": { "lte": 200 } } }
      ],
      "should": [
        { "match": { "brand": "sony" } }   // boost
      ]
    }
  },
  "aggs": {
    "by_brand": { "terms": { "field": "brand.keyword" } },
    "price_p":  { "percentiles": { "field": "price" } }
  },
  "size": 20,
  "from": 0
}

Field mapping rules

Vector & hybrid search

• Dense vectors: dense_vector field + approximate kNN over an HNSW graph (indexed kNN GA on _search since 8.4). int8_hnsw quantization is the default; BBQ shrinks RAM further for high-dim embeddings.
• Hybrid search: combine lexical BM25 with vector kNN and fuse with Reciprocal Rank Fusion (RRF) — exposed via the retrievers abstraction (RRF + linear/rescore are GA in ES 9.x).
• Sparse semantic: ELSER (learned sparse expansion) via sparse_vector, or the one-line semantic_text field that handles chunking + embedding inference automatically.
• OpenSearch equivalents: knn_vector field, neural search, neural sparse, and hybrid via normalization search pipelines.

POST /products/_search
{
  "retriever": {
    "rrf": {                              // fuse lexical + vector
      "retrievers": [
        { "standard": { "query": { "match": { "title": "wireless headphones" } } } },
        { "knn": { "field": "title_vector", "query_vector": [/* embedding */],
                   "k": 50, "num_candidates": 200 } }
      ]
    }
  },
  "size": 20
}

Tuning

• Define explicit mappings up-front — dynamic mapping creates fields that explode index size.
• Use filter context (bool.filter) for yes/no — skips scoring, cacheable.
• Shard count: hard to change after creation. Aim ~10–50 GB per shard.
• Kibana for ad-hoc exploration; not for prod query path.

Error Handling

Errors are first-class output. Plan them.

Error categories

Strategies

• Fail fast — invalid input rejected at edge; cheaper than half-applied work.
• Fail safe — on unknown error, deny access (auth failures); for non-critical features, degrade.
• Graceful degradation — recs unavailable? Show empty section, not 500.
• Circuit breaker — after N failures to dependency, open circuit for cool-down. Prevents cascading.

Circuit breaker state machine

• Closed: requests pass through; the breaker counts consecutive (or windowed) failures. At the failure threshold it trips to open.
• Open: requests fail fast immediately (no call to the sick dependency) for a cool-down period — this is what stops cascading overload.
• Half-open: after the cool-down, let a limited number of trial requests through. Enough successes → back to closed; any failure → straight back to open.
• Companions: pair with a per-call timeout (so a slow dependency counts as a failure) and a bulkhead (bounded concurrency/thread pool per dependency) to isolate blast radius.

Custom error types

class AppError(Exception):
    code:    str
    status:  int
    message: str
    cause:   Exception | None = None

class NotFound(AppError):
    status = 404
    code   = "NOT_FOUND"

class RateLimited(AppError):
    status = 429
    code   = "RATE_LIMITED"

Global handler

@app.errorhandler(Exception)
def handle(e):
    request_id = g.get("request_id")
    if isinstance(e, AppError):
        log.warn({"code": e.code, "rid": request_id})
        return jsonify(error=e.code, message=e.message), e.status
    log.error({"rid": request_id}, exc_info=True)
    return jsonify(error="INTERNAL", message="something went wrong"), 500

User-facing error response

{
  "error":     "INSUFFICIENT_FUNDS",
  "message":   "Balance $20.00 below $50.00 needed.",
  "requestId": "7f3a-1c2b",
  "docsUrl":   "https://x.com/docs/errors#insufficient_funds"
}

Monitoring & alerting

• Sentry / Bugsnag / Rollbar — exception capture with stack + breadcrumbs.
• ELK / Loki / Datadog — log aggregation, search by request-id.
• PagerDuty / Opsgenie — paging for SLO violations.
• Alert on symptoms (latency, error rate) not causes (CPU). Causes are runbook context.

Config Management

Separate config from code. Environment-aware. Secrets isolated.

Config types

Precedence (12-factor)

defaults (in code)
  ↓ overridden by
config file (config.yaml)
  ↓ overridden by
env vars (DATABASE_URL=...)
  ↓ overridden by
command-line flags (--port 8080)

.env workflow

# .env (NEVER commit)
DATABASE_URL=postgres://app:secret@localhost/app
JWT_SECRET=hunter2

# .env.example (commit this)
DATABASE_URL=
JWT_SECRET=

# loading
load_dotenv()
db_url = os.environ["DATABASE_URL"]    # crash fast on missing
log_level = os.environ.get("LOG_LEVEL", "info")

Feature flags

if flags.enabled("new-checkout", user_id=user.id):
    return new_checkout_flow(order)
return legacy_checkout_flow(order)

# rollout patterns:
#   percentage:    10% of users
#   targeting:     users in cohort "beta"
#   kill switch:   instantly disable broken feature without redeploy

Secret rotation

• Read secret at startup; cache. Reload on SIGHUP or scheduled refresh.
• Support two valid versions during rotation (overlap window).
• Audit-log every secret access.

Logging, Monitoring, Tracing

Three pillars: logs (events), metrics (aggregates), traces (causal chains).

Logs

Levels

Structured logging

# DON'T
log.info(f"user {uid} placed order {oid} for ${amt}")

# DO  — JSON keys are queryable
log.info("order_placed", extra={
  "user_id":   uid,
  "order_id":  oid,
  "amount":    amt,
  "request_id": rid,
  "trace_id":  tid,
})

What NOT to log

• Passwords, tokens, full credit card numbers, PII without need.
• Full request body on auth routes (passwords in body).
• Stack traces to user-facing logs (info leak).

Rotation & retention

• Rotate by size or daily; compress; ship to central store.
• Retention by class: access logs 30d, audit 1–7y depending on compliance.

Metrics

RED / USE / Four Golden Signals

• RED (request-oriented): Rate, Errors, Duration.
• USE (resource-oriented): Utilization, Saturation, Errors.
• Four golden signals: latency, traffic, errors, saturation.

Tracing

trace_id: 4f3a... (one per request, propagated across services)
  ├─ span A: api-gateway      (1.2 ms)
  ├─ span B: auth-service     (3.0 ms)
  └─ span C: user-service     (15.4 ms)
       ├─ span D: postgres    (8.1 ms)
       └─ span E: redis       (0.3 ms)

• Use OpenTelemetry SDK to emit spans; export to Jaeger, Tempo, Honeycomb.
• W3C traceparent header propagates across HTTP / gRPC / queues.
• Sample (head/tail) — 100% tracing too expensive; tail-based samples errors + slow tail.

Graceful Shutdown

Stop without dropping in-flight work.

Signals

Shutdown sequence

1. Mark unhealthy — readiness probe returns 503. LB stops sending new traffic.
2. Drain — keep serving in-flight requests; reject new ones (or 503).
3. Wait grace period — typically 10–30s.
4. Close external resources — DB pool drain, flush log buffers, close file handles, ack pending queue messages.
5. Exit 0.

Pattern (Go)

srv := &http.Server{Addr: ":8080", Handler: mux}
go srv.ListenAndServe()

sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)
<-sigCh

ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()
srv.Shutdown(ctx)        // stop accepting, finish in-flight
db.Close()
log.Sync()
os.Exit(0)

Kubernetes specifics

• K8s sends SIGTERM, waits terminationGracePeriodSeconds (default 30), then SIGKILL.
• Add preStop hook to sleep few seconds — gives LB time to remove pod before SIGTERM.
• Pod removal from Service endpoints is eventually consistent — that sleep helps.

Scaling, Performance, Concurrency

Find bottleneck → fix smallest → measure → repeat.

Find bottleneck

• Measure first. Profile (flame graph) before optimizing.
• Response time breakdown: wait queue + compute + DB + downstream + serialization.
• USE method: utilization, saturation, errors per resource (CPU, RAM, disk, net).
• Top tools: pprof (Go), cProfile/py-spy (Python), Chrome DevTools (Node), async-profiler (Java).

DB optimization

• N+1 — fetching N children individually after listing parents. Fix: eager-load / JOIN / dataloader pattern.
• Indexes for read-heavy paths; benchmark EXPLAIN ANALYZE on real data shapes.
• Batching — replace per-item INSERTs with bulk insert; reduce round-trips.
• Read replicas — fan reads off primary; mind replication lag.
• Sharding / partitioning when single node hits write ceiling.

App-level

• Compress payloads (gzip/br) — usually 5–10× smaller on JSON.
• Close file handles / connections — defer/finally/using/context-manager.
• Avoid loading whole files in memory — stream.
• Cache expensive computations (memoize). Beware staleness.
• Graceful degradation under load — shed non-critical features (recs, analytics).

Concurrency vs parallelism

Scaling axes

• Vertical — bigger box. Cheap until ceiling.
• Horizontal — more boxes behind LB. Requires stateless app.
• Functional — split monolith into services by domain.
• Data — read replicas, sharding, CQRS, event sourcing.

Advanced Integrations

Big files, real-time, push patterns.

Object storage (S3 et al.)

• Direct upload via pre-signed URLs — client uploads to S3 directly. Server never sees bytes.
• Multipart upload for files > 100 MB: split into ≥ 5 MB parts; parallel; resumable.
• Streaming — pipe S3 → response without buffering full file in memory.
• Lifecycle: transition to IA/Glacier; expire old objects automatically.
• Versioning + MFA delete for compliance buckets.

# pre-signed PUT
url = s3.generate_presigned_url(
  "put_object",
  Params={"Bucket": "uploads", "Key": f"users/{uid}/{uuid}.jpg",
          "ContentType": "image/jpeg"},
  ExpiresIn=900,
)
# client then: PUT url -H "Content-Type: image/jpeg" --data-binary @file

Real-time

Pub/Sub architecture

• Producers publish events to topics; subscribers consume independently.
• Decouples services — publisher doesn't know who listens.
• Brokers: Redis Pub/Sub (fire-and-forget), Kafka (replayable), GCP Pub/Sub, NATS.

Webhooks (server-initiated)

Outbound webhook checklist

• HTTPS only. Sign payload (HMAC-SHA256) — receiver verifies.
• Include unique event ID + timestamp (replay protection).
• Retry with exponential backoff (e.g., 1m, 5m, 30m, 6h, 24h); DLQ after.
• Expose dashboard so customers can see deliveries, replay manually.
• Test locally with ngrok / cloudflared tunnels.

# signing
sig = hmac.new(secret, body, sha256).hexdigest()
headers = {
  "X-Webhook-Signature": f"sha256={sig}",
  "X-Webhook-Timestamp": ts,
  "X-Webhook-Id": event_id,
}

# verifying — constant-time compare!
expected = "sha256=" + hmac.new(secret, body, sha256).hexdigest()
hmac.compare_digest(expected, headers["X-Webhook-Signature"])

OpenAPI Standards

Spec-first APIs: describe → generate clients/servers/docs/tests.

Why API-first

• Single source of truth — frontend, backend, partners all consume same spec.
• Parallel work — UI mocks against spec while server is built.
• Auto-generated clients (TS, Python, Java, Go) — no hand-written HTTP.
• Auto-generated server stubs and request validators.
• Diffable in code review — breaking changes are visible.

Evolution

• Swagger 2.0 (2014) → OpenAPI 3.0 (2017) → 3.1 (2021, aligned with JSON Schema 2020-12).
• Tools: Swagger UI, Redoc (docs), Postman/Insomnia (test), oapi-codegen / openapi-typescript / openapi-python-client (codegen).

Document anatomy

openapi: 3.1.0
info:
  title:   Orders API
  version: 1.4.0
servers:
  - url: https://api.x.com/v1

paths:
  /orders/{id}:
    get:
      operationId: getOrder
      parameters:
        - name: id
          in: path
          required: true
          schema: { type: string, format: uuid }
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema: { $ref: '#/components/schemas/Order' }
        '404': { $ref: '#/components/responses/NotFound' }
      security:
        - bearerAuth: []

components:
  schemas:
    Order:
      type: object
      required: [id, status, total]
      properties:
        id:     { type: string, format: uuid }
        status: { type: string, enum: [pending, paid, shipped] }
        total:  { type: number, minimum: 0 }
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      bearerFormat: JWT

Best practices

• Keep spec in repo next to code. Lint with Spectral.
• CI step: regenerate clients on every change; fail PR if drift.
• Use operationId consistently — codegen names methods from it.
• Define error schemas once in components/responses; reference from each endpoint.
• Version via URL segment (/v1); bump on breaking change.

Testing, Code Quality, DevOps

Ship safely. Verify automatically. Operate continuously.

Test types & pyramid

TDD cycle

1. Red — write failing test for desired behavior.
2. Green — minimum code to pass.
3. Refactor — clean up; tests still pass.

CI/CD pipeline shape

Code quality

• Lint — eslint, ruff, golangci-lint, rubocop. Run pre-commit + CI.
• Format — prettier, black, gofmt. Don't argue style; let tools settle it.
• Cyclomatic complexity — keep functions < 10–15; split when over.
• Coverage — useful as floor (e.g., 70%), useless as gospel.
• Mutation testing — flips operators; checks tests actually catch.

12-Factor App

1. Codebase: one app, one repo, many deploys.
2. Dependencies: explicit, isolated (lockfile).
3. Config: in environment, never in code.
4. Backing services: treat as attached resources (DB, queue swap by URL).
5. Build → Release → Run: strict separation.
6. Processes: stateless, share-nothing.
7. Port binding: app exports HTTP itself.
8. Concurrency: scale via process model.
9. Disposability: fast startup, graceful shutdown.
10. Dev/prod parity: same OS, services, data shape.
11. Logs: stream to stdout; let platform aggregate.
12. Admin processes: one-off scripts run in same env.

DevOps stack

Deployment strategies

Horizontal vs vertical scaling

• Vertical — bigger instance. Fast win, hits ceiling, single point of failure.
• Horizontal — more instances + LB. Requires stateless app + shared DB/cache. Near-linear scaling until DB becomes bottleneck.

API Styles

How clients and services talk. Pick the contract style that fits the call pattern.

Every style trades along the same axes: coupling (loose REST vs tight binary stubs), payload shape (resource document vs client-shaped query vs compact binary), streaming need (one-shot vs long-lived push), browser reachability (can a raw fetch hit it, or do you need a proxy), and schema/codegen (hand-written vs generated from an IDL). Map the call pattern to those axes first; the name follows.

The landscape

REST — the default for public surfaces

Resource-oriented: nouns as URLs, HTTP verbs as actions, status codes as outcomes. Depth lives in MVC and REST APIs — here just the contract essentials.

• Idempotency — GET/PUT/DELETE safe to retry; POST is not. Add an Idempotency-Key header to make POST retry-safe.
• Versioning — URL (/v2/users), header (Accept: application/vnd.api+json;version=2), or media type. URL is bluntest but most cache-friendly.
• Caching — wins on GET via ETag/Cache-Control; pairs naturally with Caching and CDNs.

GraphQL — one endpoint, client-shaped queries

Single POST /graphql. The client sends a query naming exactly the fields it wants; the server walks resolvers to fill them. Kills over-fetching (REST returns the whole object) and under-fetching (REST needs N round-trips to stitch related data). Three operation types: query (read), mutation (write), subscription (live push over WS).

# schema (SDL)
type User { id: ID!, name: String!, posts: [Post!]! }
type Post { id: ID!, title: String! }

type Query { user(id: ID!): User }

# client query — ask for exactly these fields
query {
  user(id: "42") {
    name
    posts { title }
  }
}

• N+1 resolver problem — resolving posts per user fires 1 query per user. DataLoader batches the keys collected within a tick into one WHERE id IN (...) and caches per-request.
• HTTP caching is hard — everything is one POST URL, so CDN/ETag caching breaks; you cache inside the server (persisted queries, response cache keyed on the query+vars).
• Cost/depth limiting — a deeply nested query is a DoS vector. Enforce max depth, query-complexity scoring, and persisted-query allowlists.
• Complexity tax — schema, resolvers, dataloaders, and tooling are heavier than a REST handler.

gRPC — binary RPC over HTTP/2

Define the service in a .proto IDL, run protoc to generate typed client + server stubs in any language. Wire format is Protocol Buffers (compact binary), multiplexed over HTTP/2 — so many calls share one connection and the four streaming modes come free.

syntax = "proto3";
service UserService {
  rpc GetUser (UserRequest) returns (User);              // unary
  rpc ListUsers (Query) returns (stream User);           // server-streaming
  rpc Upload (stream Chunk) returns (UploadAck);         // client-streaming
  rpc Chat (stream Msg) returns (stream Msg);            // bidirectional
}
message UserRequest { string id = 1; }
message User { string id = 1; string name = 2; }

• Deadlines/cancellation — every call carries a deadline; the context propagates cancellation across hops so no work is wasted on an abandoned request.
• Status codes — typed enum (OK, NOT_FOUND, DEADLINE_EXCEEDED, UNAVAILABLE, RESOURCE_EXHAUSTED), not HTTP numbers.
• gRPC-Web — browsers cannot speak raw HTTP/2 frames; a proxy (Envoy) translates gRPC-Web ↔ gRPC. That friction is why public APIs stay REST/GraphQL.

Realtime: long-polling vs SSE vs WebSocket

• SSE — one-way, rides plain HTTP, auto-reconnects with replay. Default when only the server pushes (drop one-way WS for this).
• WebSocket — pick when the client also pushes frequently. You own reconnect, heartbeats, and backpressure.
• Long-polling — only as a fallback where streaming is blocked by proxies.

Webhooks — server-to-server event push

Inverted REST: the provider POSTs to a URL you registered when an event fires, instead of you polling. Delivery is at-least-once — providers retry on non-2xx with backoff, so the same event can arrive twice. Verify the signature because the URL is public and anyone could forge a payload.

# HMAC verify — constant-time compare
expected = hmac.new(secret, raw_body, sha256).hexdigest()
if not hmac.compare_digest(expected, header_sig):
    return 401
process_idempotent(event_id)   # dedupe: ignore if event_id already seen

• Verify signatures — HMAC over the raw body with a shared secret proves origin and integrity; without it, the endpoint is spoofable.
• Idempotency on the receiver — store processed event_ids; retries must not double-charge or double-ship.
• Ack fast — return 2xx immediately, enqueue the work to a queue; slow handlers trigger retries and pile up.

SOAP & tRPC — the bookends

• SOAP — XML envelopes described by a WSDL contract + XSD schemas. Heavy, formal, with WS-Security/WS-* extensions. Still alive in banking, telecom, and government integrations; avoid for greenfield.
• tRPC — no IDL, no codegen: the server's TypeScript types flow to the client by inference. End-to-end type safety in a TS monorepo, but TS-only and not a language-agnostic wire contract — it is HTTP+JSON under the hood.

Choosing — decision guide

FastAPI

Python async API framework. Type hints become validation, docs, and serialization for free.

FastAPI sits on two pillars: Starlette (the ASGI toolkit handling routing, requests, middleware, WebSockets) and Pydantic (the validation/serialization engine). You annotate function signatures with type hints; FastAPI reads them at startup to validate inbound data, serialize outbound data, and emit an OpenAPI schema. One source of truth — the signature — drives all three. Async-first, but plays nicely with sync code too.

Minimal app + run

# main.py
from fastapi import FastAPI

app = FastAPI(title="Orders API", version="1.0.0")

@app.get("/health")
async def health() -> dict[str, str]:
    return {"status": "ok"}

# dev:  uvicorn main:app --reload --port 8000
# prod: gunicorn main:app -k uvicorn.workers.UvicornWorker -w 9

Parameters: path, query, body

FastAPI infers a parameter's source from its type and signature position: a name that matches a {placeholder} in the path is a path param; a Pydantic model is the JSON body; everything else with a scalar type is a query param.

from fastapi import FastAPI, Path, Query
from pydantic import BaseModel

class ItemIn(BaseModel):
    name: str
    price: float
    tags: list[str] = []

@app.put("/items/{item_id}")
async def upsert(
    item_id: int = Path(gt=0),                    # path: /items/42
    q: str | None = Query(default=None, max_length=50),  # query: ?q=...
    verbose: bool = False,                        # optional query, defaults False
    body: ItemIn = ...,                           # JSON request body
):
    return {"id": item_id, "q": q, "item": body}

Pydantic v2 models

Models declare the shape of data and the rules it must satisfy. Field carries constraints; custom field_validators handle cross-cutting logic; response_model reshapes and redacts output so internal fields (like password hashes) never leak. See Validation for the deeper Pydantic treatment.

from pydantic import BaseModel, Field, EmailStr, field_validator

class UserIn(BaseModel):
    name: str = Field(min_length=1, max_length=80)
    age: int = Field(gt=0, lt=130)
    email: EmailStr
    password: str = Field(min_length=8)

    @field_validator("name")
    @classmethod
    def no_blank(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("name cannot be whitespace")
        return v.strip()

class UserOut(BaseModel):          # note: no password field
    id: int
    name: str
    email: EmailStr

@app.post("/users", response_model=UserOut, status_code=201)
async def create_user(user: UserIn) -> UserOut:
    row = await db.insert_user(user)   # returns object incl. password hash
    return row                          # serialized through UserOut -> hash dropped

Dependency injection

Depends() lets a route declare what it needs; FastAPI resolves and injects it, caching the result per request. Dependencies can have their own dependencies (sub-deps). A yield-based dep runs setup, hands over the value, then runs teardown after the response — perfect for DB sessions that must always close.

from typing import Annotated
from fastapi import Depends

async def get_db():
    session = SessionLocal()
    try:
        yield session            # handed to the route
    finally:
        await session.close()    # always runs, even on exception

def pagination(skip: int = 0, limit: int = Query(default=20, le=100)):
    return {"skip": skip, "limit": limit}

DB = Annotated[AsyncSession, Depends(get_db)]
Page = Annotated[dict, Depends(pagination)]

@app.get("/items")
async def list_items(db: DB, page: Page):
    return await db.fetch_items(**page)

Async and concurrency

Declare a route async def when you await non-blocking I/O (async DB driver, httpx, async Redis). Declare it plain def when your libraries are synchronous — FastAPI runs plain def routes in a threadpool, so blocking calls don't stall the loop. The cardinal sin: synchronous blocking work inside an async def, which freezes the single event-loop thread for every other request. See Scaling and Concurrency.

Authentication: OAuth2 + JWT

OAuth2PasswordBearer extracts the bearer token from the Authorization header and feeds it to a get_current_user dependency that verifies the JWT and loads the user. Compose scope/role checks as further dependencies. See Auth and Security.

from fastapi import Depends, HTTPException, status
from fastapi.security import OAuth2PasswordBearer
import jwt

oauth2 = OAuth2PasswordBearer(tokenUrl="token")

async def get_current_user(token: Annotated[str, Depends(oauth2)], db: DB):
    cred_exc = HTTPException(status.HTTP_401_UNAUTHORIZED, "Invalid token",
                             headers={"WWW-Authenticate": "Bearer"})
    try:
        payload = jwt.decode(token, SECRET, algorithms=["HS256"])
    except jwt.PyJWTError:
        raise cred_exc
    user = await db.get_user(payload["sub"])
    if user is None:
        raise cred_exc
    return user

def require_role(role: str):
    def checker(user = Depends(get_current_user)):
        if role not in user.roles:
            raise HTTPException(status.HTTP_403_FORBIDDEN, "Forbidden")
        return user
    return checker

@app.get("/admin")
async def admin_panel(user = Depends(require_role("admin"))):
    return {"hello": user.name}

Middleware, CORS, exception handlers

Middleware wraps every request/response; exception handlers map raised errors to clean HTTP responses. Register CORS via CORSMiddleware so browsers permit cross-origin calls. See Middleware and HTTP.

from fastapi import Request
from fastapi.responses import JSONResponse
from fastapi.middleware.cors import CORSMiddleware

app.add_middleware(
    CORSMiddleware,
    allow_origins=["https://app.example.com"],
    allow_methods=["*"], allow_headers=["*"], allow_credentials=True,
)

class OutOfStock(Exception):
    def __init__(self, sku: str): self.sku = sku

@app.exception_handler(OutOfStock)
async def oos_handler(request: Request, exc: OutOfStock):
    return JSONResponse(status_code=409, content={"sku": exc.sku, "error": "out_of_stock"})

Background tasks vs a real queue

BackgroundTasks runs after the response is sent — but in the same process, with no retries or durability. Fine for a welcome email. For heavy, durable, or retryable work, push to a real broker. See Queues.

from fastapi import BackgroundTasks

@app.post("/signup")
async def signup(user: UserIn, tasks: BackgroundTasks):
    saved = await db.create(user)
    tasks.add_task(send_welcome_email, saved.email)  # light, fire-and-forget
    return {"id": saved.id}

Auto docs

• Swagger UI — interactive, "try it out" console at /docs.
• ReDoc — clean reference rendering at /redoc.
• OpenAPI schema — raw JSON at /openapi.json, generated from your models and signatures; feeds client codegen.

Testing

Use TestClient (sync, Starlette/requests-based) for quick tests, or httpx.AsyncClient with ASGITransport to exercise async paths. app.dependency_overrides swaps real deps (DB, auth) for fakes — no monkeypatching internals.

from fastapi.testclient import TestClient
from main import app, get_db

def fake_db():
    yield FakeSession()

app.dependency_overrides[get_db] = fake_db
client = TestClient(app)

def test_health():
    r = client.get("/health")
    assert r.status_code == 200 and r.json() == {"status": "ok"}

def test_create_user():
    r = client.post("/users", json={
        "name": "Ada", "age": 36, "email": "[email protected]", "password": "hunter2!!"})
    assert r.status_code == 201
    assert "password" not in r.json()   # redacted by response_model

Deploy

Production runs multiple worker processes behind a reverse proxy. Either bare uvicorn --workers N, or gunicorn as a process manager using the uvicorn worker class for graceful restarts and supervision. Put nginx in front for TLS termination, static files, and buffering. Use the lifespan context to open/close pools on startup/shutdown.

from contextlib import asynccontextmanager

@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.pool = await create_pool()   # startup
    yield
    await app.state.pool.close()           # shutdown

app = FastAPI(lifespan=lifespan)

# gunicorn main:app -k uvicorn.workers.UvicornWorker -w 9 \
#   --bind 0.0.0.0:8000 --timeout 60

Redis Deep-Dive

Single-threaded in-memory store. Know the data type and the command for each pattern.

Redis runs commands on a single thread via an event loop, so every command is atomic by nature — no locks, no races between commands. Data lives in RAM, hence sub-millisecond reads. Durability is optional (snapshot or append-log). It is more than a cache: it is a data-structure server — strings, hashes, lists, sets, sorted sets, streams — manipulated in place with rich commands. This deepens Caching with concrete mechanics.

Data types — pick the right one

Core command cheat-sheet

# strings + counters
SET user:1 "alice"
GET user:1
SETEX session:1 3600 "tok"      # value + 3600s TTL in one op
TTL session:1                    # seconds left (-1 no TTL, -2 gone)
INCR page:views                 # atomic +1, returns new value

# hash (object)
HSET user:1 name alice age 30
HGETALL user:1

# list as work queue
LPUSH jobs "j1"                 # push left
BRPOP jobs 5                    # block up to 5s, pop right (FIFO)

# set membership
SADD online 42
SISMEMBER online 42            # 1 / 0

# sorted set (leaderboard / time index)
ZADD board 100 alice 250 bob
ZRANGE board 0 -1 WITHSCORES    # asc
ZRANGEBYSCORE board 100 200     # score window

# atomic create-if-absent w/ expiry (locks, idempotency)
SET key val NX EX 30

Cache-aside in Redis

The dominant pattern: app reads cache, on miss loads the DB, writes back with a TTL. See Caching for the general model.

def get_user(uid):
    key = f"user:{uid}"
    cached = r.get(key)
    if cached is not None:
        return None if cached == b"\x00" else json.loads(cached)  # negative cache
    row = db.query(uid)                       # miss -> source of truth
    if row is None:
        r.set(key, b"\x00", ex=30)            # cache the miss, short TTL
        return None
    ttl = 300 + random.randint(0, 60)         # jitter -> avoid synchronized expiry
    r.set(key, json.dumps(row), ex=ttl)
    return row

Rate limiting

Three escalating designs. Fixed-window is cheapest but allows 2x burst at the boundary; sliding-window log is exact but stores every hit; token bucket smooths bursts.

# fixed-window: 100 req / 60s per user (atomic via single thread)
key = f"rl:{uid}:{epoch // 60}"
n = r.incr(key)
if n == 1:
    r.expire(key, 60)          # set TTL only on first hit of window
allowed = n <= 100

# sliding-window log (sketch)
# ZADD rl:{uid} now now ; ZREMRANGEBYSCORE rl:{uid} 0 (now-60000) ; ZCARD rl:{uid}

Distributed lock

Acquire with SET NX PX so only one client wins and the lock auto-expires if the holder dies. Release with a Lua compare-and-delete so you never delete a lock that already expired and was re-acquired by someone else.

# acquire: unique token, 30s safety TTL
SET lock:order:1 <uuid> NX PX 30000

# release: delete ONLY if value is still mine (atomic CAS)
EVAL "if redis.call('get', KEYS[1]) == ARGV[1]
        then return redis.call('del', KEYS[1]) else return 0 end"
     1 lock:order:1 <uuid>

Pub/Sub vs Streams

XADD events * type signup uid 42        # append, * = auto id
XGROUP CREATE events g1 0               # consumer group from start
XREADGROUP GROUP g1 c1 COUNT 10 STREAMS events >   # new msgs for this consumer
XACK events g1 1689-0                   # ack -> remove from pending

Prefer Streams over Pub/Sub when you need durability, replay, or balanced consumers. Reach for Kafka/queues when you need long retention, high partition throughput, or multi-day reprocessing — Redis keeps the log in RAM.

Atomicity and transactions

• MULTI/EXEC — queue commands, run as one isolated batch. No rollback on logic errors; it is isolation, not ACID rollback.
• WATCH — optimistic CAS: watch keys, and EXEC aborts (returns nil) if any changed. Retry loop for check-then-set.
• EVAL (Lua) — true multi-step atomic op; script runs to completion on the single thread. Best for read-modify-write (rate limits, token bucket).

WATCH balance:1
val = GET balance:1
MULTI
SET balance:1 (val - 10)
EXEC            # nil if balance:1 changed since WATCH -> retry

Persistence: RDB vs AOF

Production default: enable both. The hybrid mode writes an RDB preamble inside the AOF (aof-use-rdb-preamble yes) — fast load of the RDB base plus the AOF tail for the last seconds.

Eviction and memory

maxmemory 4gb
maxmemory-policy allkeys-lru     # evict least-recently-used across all keys

• noeviction — reject writes at limit (errors). Safe for a pure datastore, dangerous for a cache.
• allkeys-lru / allkeys-lfu — pure cache; LFU keeps frequently-hot keys better than LRU.
• volatile-ttl / volatile-lru — only evict keys that have a TTL; mixed cache + durable data.
• Always set TTLs on cache entries and watch big keys — a 1M-element list/hash blocks the thread on access and skews eviction.

Scaling

• Replication — async primary→replica; offload reads to replicas (accept slight lag).
• Sentinel — monitors primary, runs automatic failover + promotes a replica; clients ask Sentinel for the current primary. HA without sharding.
• Cluster — 16384 hash slots sharded across primaries; key routed by CRC16(key) % 16384. Multi-key ops must land in one slot — use hash tags {user1}:profile / {user1}:cart to co-locate, else CROSSSLOT error.

Intermediary Systems

The boxes between client and service. Each adds a concern: routing, buffering, resilience.

Intermediaries exist to keep services dumb and focused. Push cross-cutting concerns — TLS, auth, rate limiting, retries, load spreading, async buffering — out of application code and into shared infrastructure that scales independently and absorbs load before it reaches origin. The cost: more hops, more latency, more moving parts to operate. Add each box only when a real concern demands it.

The request chain

Reverse proxy & load balancer

A reverse proxy fronts your services: it terminates TLS, picks a healthy backend, and shields origin topology from clients. Load balancing is the spreading policy on top. L4 routes by IP:port (fast, opaque, protocol-agnostic); L7 parses HTTP so it can route by path/header/cookie, do sticky sessions, and retry idempotent requests.

# nginx: L7 upstream, least-conn, passive health checks (max_fails/fail_timeout)
upstream api {
    least_conn;
    server 10.0.1.10:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
}
server {
    listen 443 ssl;
    ssl_certificate     /etc/ssl/api.crt;   # TLS termination here
    ssl_certificate_key /etc/ssl/api.key;
    location / { proxy_pass http://api; }
}

• Health checks — active (LB probes /healthz) vs passive (eject on observed 5xx/timeouts). Probe a deep endpoint that touches dependencies, not just a static 200.
• TLS termination — decrypt at the edge to offload CPU; re-encrypt inside for zero-trust. See Scaling for horizontal fan-out behind the LB.

API Gateway

A gateway is an L7 proxy that owns the public API contract. A load balancer answers "which instance?"; a gateway answers "should this caller even reach the system, and in what shape?" It centralizes auth, quotas, and routing so every service doesn't reimplement them.

• Single entry point — one TLS surface, one place to revoke a key or shed load.
• Request aggregation — collapse N backend calls into one client response (BFF territory).
• Response transformation — strip internal fields, version translate, GraphQL ↔ REST shim. Pairs with REST APIs and Validation.

BFF — Backend-for-Frontend

One gateway per client type rather than one universal API. A web BFF returns rich, denormalized payloads; a mobile BFF trims fields and pre-aggregates to spare radio and battery. Each frontend team owns its BFF, so shaping changes don't fight over a shared contract.

• Why — a one-size API forces chatty mobile clients into many round trips or bloated responses.
• Cost — duplicated orchestration logic; mitigate with shared client libs, not a shared God-gateway.

Message broker

Brokers decouple producers from consumers in time: the producer fires and forgets, the broker buffers, workers drain at their own pace — absorbing spikes and surviving consumer downtime. Two models dominate. A queue deletes a message once acked (work distribution). A log is an append-only, replayable stream where consumers track their own offset.

• Delivery — at-least-once is the default; consumers must be idempotent. Exactly-once needs transactions + dedup (Kafka EOS) and is costly — usually fake it with idempotency keys.
• DLQ — after N failed redeliveries, park the poison message in a dead-letter queue for inspection instead of blocking the line.
• Backpressure — bounded queues + consumer prefetch limits stop a fast producer from drowning slow workers. More in Queues.

Service mesh

A mesh moves service-to-service concerns out of app code into a sidecar proxy (Envoy) deployed next to every pod. Apps speak plaintext to localhost; the sidecar handles mTLS, retries, timeouts, circuit breaking, and emits uniform metrics/traces. The data plane is the fleet of sidecars carrying traffic; the control plane (Istiod, Linkerd) pushes config and certs to them.

• mTLS everywhere — identity-based encryption between services with zero app changes; the zero-trust foundation under Auth.
• Traffic shifting — weighted routing for canary/blue-green; shift 5% → 100% by config.
• Resilience in the proxy — retries, timeouts, circuit breaking, and outlier ejection are policy, not code. Overlaps with Middleware conceptually but lives in infra.

Resilience at the boundary

• Circuit breaker — after a failure threshold, trip open and fail fast instead of hammering a dead dependency; half-open probes test recovery.
• Bulkhead — isolate resource pools (thread/conn per dependency) so one slow downstream can't exhaust everything.
• Retry + backoff + jitter — exponential delay with randomization to avoid synchronized retry waves.
• Idempotency keys — client-supplied key lets the server dedup safe retries of non-idempotent writes (payments, orders).
• Transactional outbox — write the event to a DB table in the same transaction as the state change; a relay publishes it to the broker — no dual-write inconsistency.

# retry with capped exponential backoff + full jitter
import random, time
def call_with_retry(fn, attempts=5, base=0.1, cap=10):
    for i in range(attempts):
        try:
            return fn()
        except Transient as e:
            if i == attempts - 1:
                raise
            sleep = random.uniform(0, min(cap, base * 2 ** i))  # full jitter
            time.sleep(sleep)

When to add which

Distributed Data & Advanced Concurrency

Sharding, replication topologies, distributed transactions, and the concurrency/observability depth interviewers push on.

Partitioning schemes

Partitioning (sharding) splits one dataset across N nodes so storage and throughput scale horizontally. The whole game is picking a partition key and a mapping function that spreads load evenly, keeps related rows co-located, and survives adding/removing nodes. Get the key wrong and you get hotspots — one shard melts while the rest idle.

• Virtual nodes — give each physical node many ring positions (e.g. 256 vnodes) so load evens out and a departing node's keys spread across all survivors, not one neighbor. See consistent-hash LB and the system-design consistent-hashing module.
• Compound keys — (tenant_id, entity_id) co-locates a tenant's data on one shard (cheap joins) while spreading tenants; but a whale tenant becomes a hotspot. Salt or sub-shard the whale.
• Secondary indexes — local (per-shard) indexes need scatter-gather reads; global (term-partitioned) indexes need a distributed write. Pick per read/write ratio.

Resharding without downtime

The reason hash(key) % N is a trap: bump N from 4 to 5 and roughly (N-1)/N of keys remap — a near-total data shuffle mid-flight. Consistent hashing and directory schemes exist precisely to make rebalancing incremental. Production resharding is a careful choreography, not a flip of a config value.

• Pre-splitting — start with far more logical shards than nodes (e.g. 16384 slots on 8 boxes, Redis Cluster style). "Resharding" then just moves whole slots between nodes — no key-level rehash. Kafka's fixed partition count is the same idea; you can't reduce partitions, so over-provision early.
• Split-brain during move — a key mid-migration must have exactly one owner for writes. Route by an authoritative slot map (gossiped or from the directory), and fence the old owner before cutover.
• Backfill rate — throttle the copy so it doesn't saturate disk/replication and starve live traffic; resumable, chunked, idempotent copies survive restarts.

Replication topologies

Replication keeps copies of each partition on multiple nodes for durability, read scaling, and failover. The topology decides who accepts writes — the single hardest question, because it dictates your conflict and consistency story.

• Sync vs async replication — synchronous = follower must ack before commit (no data loss on leader crash, but leader stalls if a follower lags); asynchronous = commit locally, ship later (fast, but a failover can lose the un-shipped tail). Most systems do semi-sync: one sync follower, rest async.
• Replication lag — async followers trail the leader, so read-your-writes breaks (you POST then GET a stale replica). Fixes: read from leader for a window after a write, or track a version/LSN the replica must have caught up to (monotonic reads).
• Failover hazards — promoting a lagging follower silently drops writes; two nodes both thinking they're leader = split-brain. Need fencing tokens and a consensus-backed leader election (Raft/ZAB), covered in the distributed-systems notebook.

Quorum reads & writes (R+W>N)

Leaderless systems tune consistency per operation with three knobs: N replicas per key, W nodes that must ack a write, R nodes that must respond to a read. When W + R > N the read and write sets are guaranteed to overlap on at least one node — so a read sees the latest acked write. That overlap is the entire mechanism behind "strong-ish" quorum consistency.

N = 3   (replicas per key)

W=2, R=2 → W+R = 4 > 3  ✓ overlap → reads see latest write, tolerate 1 down node
W=3, R=1 → fast reads, write blocks if ANY replica down (no write availability)
W=1, R=1 → W+R = 2 ≤ 3  ✗ no overlap → fast but may read stale (eventual only)

• Read-repair — a quorum read that sees divergent versions writes the freshest back to stale replicas, healing lazily on the read path.
• Hinted handoff — if a target replica is down, a neighbor stores the write with a "hint" and forwards it once the node returns; keeps W satisfied during blips (sloppy quorum).
• Quorum is not linearizable — overlap guarantees you see a recent write, not a total order. Concurrent writes still need version vectors to order/merge; sloppy quorums + hinted handoff can even violate the overlap guarantee.

Distributed transactions: 2PC

When one logical operation must atomically touch multiple databases/services, local ACID isn't enough. Two-phase commit coordinates atomic commit across participants via a coordinator: everyone promises before anyone commits.

• Blocking is the fatal flaw — a participant that voted YES must hold locks until it hears the verdict. If the coordinator crashes after PREPARE but before COMMIT, participants are stuck holding locks indefinitely (in-doubt). 3PC and Paxos-Commit reduce but don't fully remove this.
• Latency & coupling — two round trips plus fsync at every participant; the transaction is only as available as the least available participant (availability multiplies down).
• Where it lives — XA/JTA across two RDBMS, or a single DB with distributed shards. Across microservices owned by different teams, 2PC is usually the wrong tool — use a saga.

Sagas, idempotency & the outbox

A saga replaces one distributed ACID transaction with a sequence of local transactions, each with a compensating action that semantically undoes it (refund, not rollback). You trade atomicity for availability: the system passes through inconsistent intermediate states but converges. Two coordination styles:

# Order saga (orchestration): forward steps + compensations
1 reserve_inventory   ⇄  release_inventory
2 charge_payment      ⇄  refund_payment
3 create_shipment     ⇄  cancel_shipment
# step 3 fails → run comps 2,1 in reverse. Comps MUST be idempotent
# & retryable, since the compensation itself can fail and be retried.

• Idempotency keys — every step is at-least-once, so a retried "charge_payment" must not double-charge. Client sends a unique key; the service stores key → result and replays the stored result on a repeat. This is how you get effectively-once semantics.
• Exactly-once is a lie (mostly) — you cannot guarantee exactly-once delivery over an unreliable network. You get exactly-once effect = at-least-once delivery + idempotent processing (dedup on a message/idempotency ID). Kafka EOS achieves it only inside its own transactional boundary.
• Transactional outbox — the dual-write problem: you can't atomically write the DB and publish to a broker (one may fail). Instead write the event to an outbox table in the same local transaction as the state change; a relay (CDC / polling) reads the table and publishes. DB commit = event guaranteed. See the outbox note in Intermediary Systems.

Concurrency models & backpressure

Two ways a server handles many in-flight requests. Thread-per-request (classic Java/Tomcat, synchronous) gives each request a blocking thread — simple to write, but each thread costs ~1 MB stack and context switches, so ~thousands cap you out and blocking I/O idles CPU. Async / event-loop (Node, Netty, Python asyncio, Go's goroutines as a hybrid) multiplexes thousands of connections on a few threads by never blocking — I/O yields control — winning huge concurrency for I/O-bound work.

• Backpressure — when arrivals outpace service, an unbounded queue just grows until it OOMs and latency explodes (bufferbloat). Bound every queue and propagate "slow down" upstream: reject (429), block the producer, or drop. Reactive Streams and gRPC flow control formalize this.
• Bulkhead — partition resources per dependency (separate thread/connection pools) so one slow downstream drowns only its own pool, not the whole service. Same idea as ship compartments; pairs with the circuit breaker.
• Connection-pool exhaustion — the sneakiest outage. A DB pool of 20; a downstream slows from 10 ms to 500 ms; requests hold connections 50× longer; the pool empties; every new request blocks waiting for a connection and times out — a total outage caused by mere slowness, not an error. Size pools with Little's Law and always set a checkout timeout.

Capacity: Little's Law & load testing

You cannot size pools, threads, or fleets by guessing. Little's Law is the one equation that ties throughput, latency, and concurrency together for any stable system: L = λ × W — average concurrent requests in the system = arrival rate × average time each spends in it. It holds regardless of distribution, so it's your back-of-envelope for every capacity question.

L = λ × W
  λ = arrival rate (req/s),  W = time-in-system (s),  L = concurrent in flight

# How big must the DB connection pool be?
λ = 500 req/s,  W = 20 ms per query = 0.020 s
L = 500 × 0.020 = 10 concurrent  → pool of ~10 (add headroom → 15)

# What if the DB slows to 200 ms?
L = 500 × 0.200 = 100 concurrent  → a pool of 10 now queues 90 → exhaustion
# same request rate, 10× latency ⇒ 10× concurrency demand

• Throughput ceiling — a fixed pool of C connections caps throughput at λ_max = C / W. Beyond it, queueing latency grows without bound. Rearranging Little's Law tells you exactly when you'll fall over.
• Load vs stress vs soak — load = expected traffic (does it meet SLO?); stress = ramp until it breaks (find the knee & failure mode); soak = hold for hours (catch leaks, pool drift, GC creep). Also spike tests for autoscaling reaction.
• Measure the right percentile — averages hide pain; report p95/p99/p999. Watch the knee: latency stays flat then hockey-sticks as you approach saturation — that inflection is your real capacity, not the point where it errors.
• Closed vs open models — closed-loop testers (fixed virtual users) mask overload because they wait for responses; open-model tools (fixed arrival rate — k6, Gatling, wrk2) reveal true behavior under a rate you don't control, avoiding coordinated-omission bias.

Observability: SLOs, error budgets & alerting

Monitoring answers "is it up?"; observability answers "why is it behaving like this?" from the outside, via the three pillars — metrics (cheap aggregates), logs (discrete events), traces (one request across services). The discipline that makes it actionable is defining what "good" means numerically.

• SLI — a measured indicator of health, ideally a ratio of good events / total: request success rate, p99 latency under 300 ms, freshness. Measure it where the user feels it (edge/LB), not deep internals.
• SLO — the target for an SLI over a window: "99.9% of requests succeed over 28 days." Deliberately below 100% — perfection is unaffordable and pointless.
• Error budget — 1 − SLO. 99.9% ⇒ 0.1% ⇒ ~43 min/month of allowed failure. Spend it on shipping features and risk; burn it too fast and you freeze releases to focus on reliability. It turns the dev-vs-ops fight into shared math.
• SLA — the contractual, money-backed promise; always looser than your internal SLO so you have margin before penalties.