Core Tenets

• Embrace risk — 100% reliability is wrong target. Set explicit budget and spend it.
• Service Level Objectives drive engineering priority. Below SLO → freeze features, fix reliability. Above → ship faster.
• Eliminate toil — repetitive manual ops work. Cap toil at <50% per SRE.
• Automate everything — runbooks → code.
• Blameless postmortems — focus on systemic causes, not individual blame.
• Shared ownership — devs run what they build; SREs partner, not absorb.

Error Budget Math

SLO = 99.9% availability over 30 days
Budget = (1 - 0.999) × 30d = 0.001 × 43,200 min = 43.2 min/month

Burn rate = (errors observed) / (budget per same interval)
  Burn rate 10× over 1 hour = consumed 10 hr of budget in 1 hr
  Alert threshold typical: 14.4× over 1 hr = 2% budget burned
  Slow burn: 1× over 6 hr = 5% burned

Definitions

Term	Definition	Example
SLI	Indicator: measured ratio of good events / valid events	p99 latency, success rate, freshness
SLO	Target on SLI over window	p99 < 300 ms over 28 days, 99.9%
SLA	Customer contract; financial penalty if breached	99.9% uptime or 10% credit

Common SLI Types

• Availability: successful_requests / valid_requests.
• Latency: requests_under_threshold / valid_requests. Use p50 / p95 / p99 — never average.
• Quality: graceful degradation rate.
• Freshness: data age < X for batch / streaming pipelines.
• Correctness: consistency check pass rate.
• Throughput: requests/sec served vs offered.

Window Choice

Rolling 28-day or 30-day window standard. Monthly resets create gaming. Multi-window multi-burn-rate alerts: page on fast burn (1 hr × 14.4×), ticket on slow burn (6 hr × 6×).

Worked Scenario: SLOs + Alerting for a Checkout Service

A common open-ended interview prompt is "define SLOs and alerting for service X." Walk it end to end rather than reciting definitions.

1. Pick user-centric SLIs. Checkout = "can a user pay and get an order?" Two SLIs cover it: availability = successful checkouts / valid checkout attempts; latency = checkouts completing under 1 s / valid attempts. Measure at the edge (what the user sees), not an internal hop. Skip vanity SLIs like CPU.
2. Set SLO targets from the journey, not a round number. Checkout is revenue-critical, so 99.95% availability over 28 days; p99 latency < 1 s. Reads/browse can be looser; the strictest target belongs on the money path.
3. Derive the error budget. 99.95% over 28 days = (1 − 0.9995) × 28 × 24 × 60 = 0.0005 × 40,320 min ≈ 20.2 min/month of allowed failed-checkout time. That number, not opinion, gates whether you ship features or freeze.
4. Alert on budget burn, not raw error count. Two-tier multi-burn-rate: page when the 1 h burn rate ≥ 14.4× (would exhaust a 30-day budget in ~2 days; ≈ 2% of budget gone in 1 h) and the short window confirms it; ticket on a slow 6 h burn at ≥ 6× (≈ 5% in 6 h). The fast tier catches outages; the slow tier catches steady erosion without paging on noise.
5. Make alerts actionable. Symptom-based (failed checkouts), every page links a runbook, and the SLA you sell customers (e.g. 99.9%) sits below the internal SLO (99.95%) so the buffer absorbs blips before a contractual breach.

Pipeline Stages

1. Source — webhook on push/PR. Branch protection: required reviews, signed commits, status checks.
2. Build — compile, lint, type-check. Hermetic, reproducible. Cache deps.
3. Test — unit (fast), integration (deps), contract, e2e (slow).
4. Static analysis — SAST, dep scan (Snyk, Dependabot), license check.
5. Package — container image, sign (cosign), SBOM.
6. Deploy staging — smoke tests, integration suite.
7. Deploy prod — progressive rollout.
8. Verify — synthetic checks, error rate, latency.
9. Rollback — automated on SLO breach.

GitHub Actions Pattern

name: ci
on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with: { go-version: '1.22', cache: true }
      - run: go vet ./...
      - run: go test -race -cover ./...

  build:
    needs: test
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v5
        with:
          push: true
          tags: ghcr.io/org/app:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

GitOps

Git as single source of truth for desired infra state. Argo CD / Flux watches repo, reconciles cluster. Benefits: auditable, rollback = revert, no kubectl access for humans in prod.

Patterns

Pattern	How	Risk profile
Recreate	Stop old, start new	Downtime; OK for dev only
Rolling	Replace pods N at a time	Mixed versions during rollout
Blue/Green	Two full envs, swap LB	Instant cutover; 2× capacity cost
Canary	Route 1–5% to new, increase	Limited blast radius; needs metrics
Shadow / mirror	Replay prod traffic to new without serving	Validate without user impact
Feature flag	Deploy dark, toggle per user/cohort	Best for product changes; flag debt risk

Canary Recipe

• Promote on success metric: error rate < baseline + 0.1%, p99 < baseline × 1.1.
• Steps: 1% → 5% → 25% → 50% → 100%, 10 min between.
• Auto-rollback on regression. Don't proceed if statistical signal weak (low traffic).
• Tools: Argo Rollouts, Flagger, LaunchDarkly + ingress weights.

Feature Flag Hygiene

• Every flag has owner + expiry date.
• Long-lived (kill switches, experiments) tagged separately.
• Quarterly cleanup sprint to remove dead flags.
• Avoid nested flag combinations — exponential test matrix.

Resilience Patterns

A new deploy fails safe only if the calling code already tolerates a slow or dead dependency. These five patterns are the standard interview checklist — each answers a different failure mode.

Pattern	Failure it handles	Mechanism
Timeout	A hung dependency blocks the caller forever	Bound every network/IO wait; fail fast so threads aren't pinned
Retry + backoff + jitter	Transient blips (brief 5xx, packet loss)	Retry idempotent calls, but spread attempts to avoid synchronized storms
Circuit breaker	A dependency is down, not just blipping	Stop calling after N failures; fail fast; probe before resuming
Bulkhead	One slow dependency starves all others	Isolate resources (thread pools / connection pools / pods) per dependency
Idempotency key	Retries cause duplicate side effects	Client sends a unique key; server dedups within a window

Retry storms — why jitter, not just backoff

Plain exponential backoff still synchronizes: if 10k clients all fail at the same instant (a dependency hiccup), they all wait the same base×2^n and retry in lockstep, re-hammering the recovering service in waves — a retry storm / thundering herd. Adding randomness (jitter) de-correlates the retries so load spreads smoothly. AWS's recommended "full jitter" picks the sleep uniformly in [0, min(cap, base×2^attempt)]:

base = 100 ms, cap = 20 s   # full jitter: sleep = random(0, min(cap, base*2^attempt))
attempt 0 -> window 100 ms     sleep in [0, 100 ms]
attempt 1 -> window 200 ms     sleep in [0, 200 ms]
attempt 2 -> window 400 ms     sleep in [0, 400 ms]
attempt 7 -> window 12.8 s     sleep in [0, 12.8 s]
attempt 8 -> window 20 s       capped: sleep in [0, 20 s]   # base*2^8 = 25.6 s > cap

Circuit breaker state machine

Closed (normal, requests flow, count failures) → trips to Open after a failure threshold (fail fast, reject immediately, no calls to the dep) → after a cooldown moves to Half-Open (allow a limited number of trial probes) → a probe success returns to Closed; a probe failure returns to Open and restarts the cooldown.

import random, time

# Full-jitter exponential backoff (AWS "Exponential Backoff and Jitter")
def backoff_sleep(attempt, base=0.1, cap=20.0):
    window = min(cap, base * (2 ** attempt))   # cap bounds the growth
    return random.uniform(0, window)           # full jitter spreads retries

class CircuitBreaker:
    def __init__(self, fail_max=5, reset_after=30.0, half_open_max=1):
        self.fail_max = fail_max          # consecutive failures that trip it
        self.reset_after = reset_after    # seconds Open before a trial probe
        self.half_open_max = half_open_max
        self.state = "CLOSED"
        self.failures = 0
        self.opened_at = 0.0
        self.half_open_inflight = 0

    def allow(self, now):
        if self.state == "OPEN":
            if now - self.opened_at >= self.reset_after:
                self.state = "HALF_OPEN"      # cooldown elapsed -> probe
                self.half_open_inflight = 0
            else:
                return False                  # fail fast; don't call a dead dep
        if self.state == "HALF_OPEN":
            if self.half_open_inflight >= self.half_open_max:
                return False                  # only a few probes in flight
            self.half_open_inflight += 1
        return True

    def on_success(self):
        self.failures = 0
        self.state = "CLOSED"                 # probe/normal call OK -> close

    def on_failure(self, now):
        self.failures += 1
        # a failed probe, or too many failures while closed, (re)opens it
        if self.state == "HALF_OPEN" or self.failures >= self.fail_max:
            self.state = "OPEN"
            self.opened_at = now

Pillars

Pillar	Cardinality	Question answered	Tools
Metrics	Low (aggregated)	Is it broken? How much?	Prometheus, Datadog, CloudWatch
Logs	High (per event)	What happened on this request?	Loki, ES, Splunk, CloudWatch Logs
Traces	Per request, span tree	Where is the latency?	OpenTelemetry, Jaeger, Tempo, X-Ray
Profiles	Continuous CPU/heap	Why is this slow / fat?	Pyroscope, Parca, Datadog

RED & USE

• RED for services: Rate, Errors, Duration. Per endpoint.
• USE for resources: Utilization, Saturation, Errors. Per CPU/disk/net/memory.
• Four Golden Signals (Google SRE): latency, traffic, errors, saturation.

Cardinality Trap

Distributed Tracing

• Trace = tree of spans across services. Span = single operation with start, end, tags.
• Context propagation via W3C traceparent header.
• Head-based sampling (decide at ingress) vs tail-based (decide after seeing full trace; better for errors but stateful).
• OpenTelemetry = vendor-neutral SDK + protocol (OTLP).

Logging Discipline

• Structured (JSON / logfmt), not free text.
• Levels: ERROR (page-worthy), WARN (anomaly), INFO (lifecycle), DEBUG (dev).
• Include trace_id, request_id, user_id, tenant_id.
• Sample DEBUG/INFO at high RPS; never sample ERROR.
• Never log secrets, PII unless redacted.

Severity Levels

Sev	Definition	Response
SEV-1	Major outage; revenue / user impact	All-hands, exec page, war room
SEV-2	Significant degradation; partial impact	On-call IC + responders
SEV-3	Minor; no user impact yet	Investigate during business hours
SEV-4	Cosmetic / future risk	Backlog ticket

ICS Roles

• Incident Commander (IC) — coordinates; doesn't fix.
• Operations Lead — drives mitigation.
• Communications Lead — internal + customer comms.
• Scribe — timeline, decisions log.
• Subject-matter experts — pulled in as needed.

Blameless Postmortem

• Timeline (UTC, exact times, who did what).
• Impact: users, requests, $$, duration.
• Root cause + contributing factors (5 Whys).
• What went well + what went poorly.
• Action items: owner + due date. Track to completion.
• Detection delay analysis — why didn't we catch sooner?

On-Call Health

• Page volume target: < 2 per shift. More = alert tuning needed.
• Compensate on-call (time off, stipend).
• Rotate week-on / N-off. Primary + secondary.
• Runbook per alert. Link in alert payload.

Capacity Planning

• Baseline = current peak. Plan = baseline × growth × seasonality × headroom (typically 30–50%).
• Failure scenarios: lose 1 AZ → remaining must absorb traffic. Plan N+1 / N+2.
• Track per resource: CPU, memory, network, disk IOPS, connection count, queue depth.

Load Test Types

Type	Goal
Smoke	System works at all under load
Load	Performance at expected peak
Stress	Find breaking point
Spike	Sudden traffic surge
Soak	Stability over hours/days; memory leaks

Tools: k6, Locust, Vegeta, JMeter, Gatling.

Little's Law

L = λ × W
  L = average concurrent requests in system
  λ = arrival rate (req/s)
  W = average response time (s)

example: 1000 req/s × 200 ms = 200 concurrent. Need ≥200 worker slots
  buffer 30% headroom → 260 workers minimum

Principles

1. Define steady-state (the metric that says "system is fine").
2. Hypothesize steady-state holds in both control + experiment.
3. Inject real-world events (instance kill, network latency, disk full).
4. Try to disprove the hypothesis. Evidence beats opinion.
5. Minimize blast radius. Have abort.

Common Experiments

• Terminate random pod / VM / AZ (Chaos Monkey, AWS FIS, Litmus, Gremlin).
• Inject latency / packet loss between services.
• Drop dependency (database, Redis, downstream service).
• Fill disk, exhaust file descriptors.
• Clock skew, DNS failure.
• GameDay: scheduled, supervised, organization-wide drill.

Supply Chain

• SBOM (software bill of materials) — CycloneDX, SPDX. Generate per build.
• Sign artifacts (cosign, Sigstore). Verify at admission.
• SLSA levels (1–4) describe build integrity. Aim for L3.
• Pin transitive deps. Lockfiles checked in. Renovate / Dependabot.
• Provenance attestations linking artifact → build → source commit.

Versioning

• SemVer (MAJOR.MINOR.PATCH) for libraries.
• CalVer (2026.05.08) or commit-SHA for services.
• Immutable artifacts — never re-tag.
• Promote artifact across envs, don't rebuild.

Syscall & Library Tracing: strace, ltrace, /proc

When a process hangs, spins, or fails silently, tracing tells you exactly which kernel boundary it is stuck at. strace intercepts syscalls (the process↔kernel interface); ltrace intercepts library calls (userspace, e.g. malloc, getenv). The kernel exposes live process state through the /proc pseudo-filesystem — no restart, no debugger needed.

• strace -f -T -tt -p PID — follow forks (-f), print time-in-syscall (-T), wall-clock timestamps (-tt). A process blocked in read() or futex() for seconds is your answer.
• strace -c — summary mode: counts + cumulative time per syscall. If 90% of time is in stat(), you have a filesystem-walk problem, not a CPU problem.
• /proc/PID/status — State (R/S/D/Z), VmRSS (resident memory), voluntary_ctxt_switches vs nonvoluntary (high nonvoluntary = CPU-starved/preempted).
• /proc/PID/fd/ — open file descriptors as symlinks. Counting them (ls | wc -l) diagnoses fd leaks before you hit EMFILE.
• /proc/PID/wchan — the kernel function the task is sleeping in; pairs with State: D (uninterruptible sleep, almost always disk/NFS I/O).

# Why is this process using 100% CPU? Attach and watch.
strace -f -T -tt -p 4821 2>&1 | head -40

# It is spinning on a failing syscall:
# 14:22:01.005 epoll_wait(6, [], 128, 0) = 0 <0.000004>   # returns instantly, 0 events
# 14:22:01.005 epoll_wait(6, [], 128, 0) = 0 <0.000004>   # busy-loop: timeout=0, no backoff

# Count syscalls to find the hot path:
strace -c -f -p 4821   # Ctrl-C after ~10s
# % time   seconds  calls   syscall
#  71.2   0.481000  120250  epoll_wait   <-- busy polling bug

Load Average vs CPU, the OOM Killer & the USE Method

The single most misread metric on Linux is load average. Load is not CPU utilization — it is the number of tasks in the run queue plus tasks in uninterruptible sleep (state D), averaged over 1/5/15 minutes. A load of 8 on an 8-core box can mean "perfectly saturated" or "4 cores idle while 8 tasks wait on a stuck disk."

When memory is exhausted and swap is gone, the kernel invokes the OOM killer. It scores every process by oom_score (roughly proportional to RSS, adjustable via oom_score_adj in -1000..1000) and SIGKILLs the highest. The victim gets no chance to clean up — you find it only in dmesg.

dmesg -T | grep -i "killed process"
# [Wed Jul  2 14:31:22] Out of memory: Killed process 5120 (java)
#   total-vm:8912344kB, anon-rss:7810112kB ... oom_score_adj:0

# Protect a critical daemon (less likely to be killed):
echo -800 > /proc/1234/oom_score_adj

The USE method (Brendan Gregg) is a checklist to avoid tunnel vision: for every resource, check Utilization, Saturation, and Errors. CPU util=busy time; saturation=run-queue length; errors=throttling. Memory util=used; saturation=swap/scan rate; errors=OOM kills. It forces you to check saturation (the queue), which utilization alone hides.

TCP Socket States & Connection Debugging

Half of "the service is slow/unreachable" incidents are visible in the TCP state machine. ss -tan (the modern netstat) shows every socket and its state. Knowing what each state means tells you which side is misbehaving.

• SYN-SENT (client) — sent SYN, no SYN-ACK back. Firewall dropping, or server down. Stuck here = network/reachability.
• SYN-RECV (server) — many of these = SYN flood or backlog overflow; check net.core.somaxconn and the app's listen() backlog.
• ESTABLISHED — connected. Count them against your connection-pool size.
• CLOSE-WAIT — the peer closed, but your app hasn't called close(). Piles of CLOSE-WAIT = an application fd leak, not a network problem.
• TIME-WAIT — the side that closed first waits 2×MSL (~60s) to absorb stray packets. Thousands are normal on a busy client; only a problem if you exhaust the ephemeral port range (~28k ports).

# Distribution of socket states — the fastest triage:
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn
#  1820 ESTAB
#  4300 TIME-WAIT      # closing first a lot; watch ephemeral ports
#   612 CLOSE-WAIT     # <-- app not closing sockets: fd leak

# Who is filling the accept queue? Recv-Q on a LISTEN socket = pending, Send-Q = backlog max
ss -tlnp
# State  Recv-Q Send-Q  Local Address:Port
# LISTEN 129    128     0.0.0.0:8080   <-- Recv-Q > Send-Q: backlog overflowing, drops

Kubernetes Scheduler, Requests & Limits

The scheduler places a Pod by filtering nodes that fit (predicates: does the node have the requested CPU/memory, matching taints/tolerations, affinity?) then scoring the survivors (least-loaded, spread, etc.) and binding to the winner. The number that drives filtering is the Pod's requests — not its limits, and not its actual usage.

• requests — the reservation. Scheduler subtracts it from node allocatable; it is the number that decides where a Pod fits and sets the CPU cgroup weight (shares).
• limits — the ceiling. CPU over-limit is throttled (CFS quota, not killed); memory over-limit is OOMKilled (exit 137). Memory is incompressible, so limit=kill.
• QoS class — Guaranteed (requests==limits for all resources), Burstable (requests<limits), BestEffort (none set). Under node memory pressure the kubelet evicts BestEffort first, then Burstable over-request, and Guaranteed last.

resources:
  requests: { cpu: "250m", memory: "256Mi" }  # scheduled against this
  limits:   { cpu: "1",    memory: "512Mi" }  # throttled/OOMKilled at this

Probes, HPA & VPA

Kubernetes has three probes, and confusing them causes real outages. Each answers a different question, and the kubelet reacts differently to each failure.

The rule: liveness = "restart fixes it," readiness = "wait, don't route here." A liveness probe must depend only on the process itself; if it checks a shared downstream, one downstream hiccup restarts your entire fleet simultaneously.

Autoscaling comes in two axes. The HPA (Horizontal Pod Autoscaler) adds/removes replicas based on a metric; the VPA (Vertical Pod Autoscaler) right-sizes requests/limits of existing Pods.

desiredReplicas = ceil( currentReplicas × currentMetric / targetMetric )
# 4 replicas, avg CPU 90%, target 50%:
# ceil(4 × 90 / 50) = ceil(7.2) = 8 replicas

Services, Ingress & Rollout Strategies

A Pod IP is ephemeral; Services give a stable virtual IP. Understanding the layering — ClusterIP → NodePort → LoadBalancer → Ingress — is a staple whiteboard question.

• ClusterIP — stable in-cluster VIP; kube-proxy programs iptables/IPVS to DNAT it across ready endpoints. Not reachable from outside.
• NodePort — opens the same port (30000–32767) on every node; external LB can target any node.
• LoadBalancer — provisions a cloud L4 LB pointing at the NodePorts. One cloud LB (and bill) per Service.
• Ingress — an L7 router (nginx/Envoy) behind one LB that fans out by host/path to many Services — HTTP routing, TLS termination, path rewrites.

Terraform State, Drift & Vault Dynamic Secrets

Terraform's power and its footguns both come from state — a JSON file mapping your config's resources to real cloud IDs. Terraform diffs desired config against state (not against the cloud) to compute a plan. This is why state must be remote, locked, and never hand-edited.

• Remote backend + locking — S3 + DynamoDB (or TF Cloud). The lock prevents two applys from corrupting state via a race.
• Drift — someone changed a resource in the console; state no longer matches reality. terraform plan (which refreshes) shows it as a diff; terraform apply -refresh-only reconciles state without changing infra.
• Module design — small, single-purpose, versioned (pinned source ref). Inputs via variables, contracts via outputs; never bake environment specifics inside — pass them in. Avoid one god-module.
• import / moved — terraform import adopts existing infra into state; the moved{} block refactors addresses without destroy/recreate.

Static secrets in state or env vars rot and leak. Vault dynamic secrets mint short-lived, per-consumer credentials on demand and revoke them on lease expiry — the credential simply doesn't exist until requested and stops working after its TTL.

# App asks Vault for a fresh DB credential; Vault CREATEs a real DB user with a TTL.
vault read database/creds/readonly
# username    v-token-readonly-x7k2...   (auto-generated)
# password    A1b2-C3d4-...              (never stored anywhere long-term)
# lease_duration  1h                     <-- Vault DROPs this DB user at expiry

# Rotate the root/static credential Vault itself uses — old value is discarded:
vault write -f database/rotate-root/my-postgres

Networking Depth: DNS/LB/CDN/Anycast & Backoff Math

A request's journey: DNS resolves the name (often to an anycast IP — the same address advertised from many locations, so BGP routes you to the nearest PoP), a CDN edge serves cache hits, and a load balancer (L4 by connection, L7 by request) spreads the rest across origins. Each hop needs a timeout and a retry policy, or one slow backend cascades into a full outage.

• Timeout — bound every network wait. No timeout = a hung dependency ties up a worker thread forever → thread-pool exhaustion → the whole service stalls.
• Retry — only retry idempotent, transient failures (timeouts, 503, connection-reset). Retrying a non-idempotent POST can double-charge a customer.
• Exponential backoff — wait grows as base × 2^attempt, so a struggling server gets breathing room instead of a retry stampede.
• Jitter — randomize the delay so thousands of clients that failed at the same instant don't retry in lockstep (the "thundering herd").

import random, time

def backoff(attempt, base=0.1, cap=10.0):
    # "Full jitter": uniform(0, min(cap, base * 2^attempt))
    return random.uniform(0, min(cap, base * (2 ** attempt)))

# attempt 0 -> up to 0.1s   attempt 3 -> up to 0.8s   attempt 6 -> up to 6.4s (capped 10)
for attempt in range(6):
    try:
        return call_dependency(timeout=2.0)   # ALWAYS bound the wait
    except Transient:
        time.sleep(backoff(attempt))          # spread the herd
raise   # give up; let a circuit breaker trip

Connection-pool exhaustion is the failure this all guards against. A pool of, say, 20 connections with a 2s dependency latency serves 20 / 2s = 10 req/s (Little's Law: L = λ × W). If latency spikes to 10s, throughput collapses to 20 / 10 = 2 req/s; new requests block waiting for a free connection, pile up, and time out — an outage caused not by errors but by slowness. Retries without backoff make it worse by consuming pool slots faster.