Tail latency, not the average

If one request fans out to 10 services and each has a 1% chance of being slow, about 10% of requests hit at least one slow call (1 − 0.99¹⁰ ≈ 9.6%). Fan-out turns rare slowness into a common experience, so you design and alert on P99, and you cap fan-out on the critical path.

Availability composes

Dependencies in series multiply: three 99.9% services in a chain give about 99.7%. Redundant copies in parallel give 1 − (1 − a)ⁿ, so two independent 99.9% replicas give 99.9999%. The catch is "independent": shared deployments, shared config and shared regions fail together. The useful moves are taking dependencies off the critical path (async, fallback) and making redundancy truly independent.

Little's Law sizes everything

Requests in flight = arrival rate × time in system. At 10,000 per second and 150 ms, you have 1,500 requests in flight. That number sizes thread pools, database connections and Lambda concurrency. It also explains outages: if a dependency slows to 2 s, in-flight requests jump to 20,000 and you run out of everything at once.

SLIs, SLOs and error budgets

An SLI is what you measure (the share of authorizations answered in under 150 ms). An SLO is the target (99.95% per 30 days). The error budget is what's left: 0.05% of a 30-day month is 21.6 minutes. When the budget is spent, you slow releases and fix reliability. This turns reliability arguments into a shared rule that product and engineering both accept.

Estimates that change decisions

Only estimate what moves a decision: peak writes per second (one primary, or partition?), storage growth (hot store, or archive tier?), in-flight concurrency (pool sizes, quotas). Round hard and say so.

Consistency vs availability, honestly

The CAP theorem only bites during a network partition: then you choose between refusing requests and serving possibly stale data. Most of the time the real trade-off is latency vs consistency (PACELC). In a bank, choose consistency for decisions that move money, and availability plus freshness indicators for views.

Choose the store from access patterns

List the reads and writes first, then pick. Relational (Aurora PostgreSQL) for multi-row transactions, constraints and ad-hoc queries. DynamoDB for known access patterns with predictable latency at any scale, conditional writes and small transactions. OpenSearch for text search. S3 for documents, archives and the data lake. A cache only for data whose staleness you can tolerate.

Replication and replica lag

Leader–follower is the default. With asynchronous followers, a read right after a write can miss it. Fixes: read from the leader after a write (read-your-writes), pass a version or LSN token, or accept staleness and show "as of" times. Multi-leader brings write conflicts; leaderless quorums need R + W > N for overlap.

Partitioning and hot keys

The partition key decides your scaling and your hot spots. Hash keys spread load; range keys support ordered scans. In a bank the natural key is account_id, but corporate accounts with thousands of cards sharing one limit become hot. DynamoDB caps each partition at roughly 1,000 write units and 3,000 read units per second, so a hot account needs a design: split counters, credit tranches across sub-partitions, or a dedicated store.

Isolation anomalies with money

Lost update: two withdrawals read a $100 balance, both subtract $80, both write. Under PostgreSQL's default Read Committed isolation, this read-then-write pattern is exposed. Write skew: two transactions each check a rule across rows ("combined balance stays above zero"), each writes a different row, and the rule breaks. Fixes: atomic conditional updates, SELECT … FOR UPDATE, optimistic version columns, or SERIALIZABLE isolation with retries.

-- Atomic, safe under concurrency: the check and the write are one statement
UPDATE accounts
   SET available = available - :amount, version = version + 1
 WHERE account_id = :id AND available >= :amount;
-- 0 rows updated  →  decline (insufficient funds)

# DynamoDB equivalent
UpdateItem  Key={pk: "ACCT#123"}
  UpdateExpression    "SET available = available - :amt"
  ConditionExpression "available >= :amt"

Consistency from the customer's seat

Linearizable: after the transfer returns, every read sees it. Read-your-writes: you see your change immediately, others may lag. Monotonic reads: you never see time go backwards between refreshes. Eventual: the search index may trail by seconds. Choose per screen, and show freshness where you choose eventual consistency ("Pending", "Updated 10 s ago").

Change data capture feeds everything else

Keep one system of record per entity, and feed read models, search, analytics and caches from its change stream (DynamoDB Streams, or Debezium on PostgreSQL logical replication). This avoids dual writes and gives you replay.

Double-entry ledger

Every movement is a journal entry whose lines balance: total debits equal total credits. Entries are append-only and immutable; you correct mistakes with a new reversing entry, never an update. A balance is the sum of an account's lines. You store a materialized balance for speed, but it's updated in the same transaction as the entry, so the two can't disagree. Internal accounts (fees, suspense, network settlement) make every flow balance.

For deposit accounts the bank owes the customer, so a debit reduces A's balance and a credit increases B's.

Ledger balance vs available balance

Ledger (posted) balance is what has settled. Available balance subtracts holds and pending items. For a card: available credit = limit − posted balance − open authorizations. Decisions use available; statements use posted. Mixing the two is a classic interview mistake.

Idempotency keys

The client sends a unique key with each money-moving request. The server stores the key, a hash of the request and the result, in the same transaction as the effect. A retry with the same key returns the stored result. The same key with a different body is rejected (HTTP 422). A request still in progress gets a 409 or waits on a lock. Keys expire after a window such as 24 hours.

"Exactly once" means at-least-once delivery plus idempotent effects

Networks can't deliver exactly once. Messages will be duplicated, so every consumer either dedupes on a message ID or makes its effect naturally idempotent (an upsert keyed by transaction ID).

Transactional outbox

Writing to the database and then publishing an event is a dual write: a crash in between loses the event. Instead, write the event to an outbox table in the same transaction as the state change, and let a relay (polling or CDC) publish it. You get at-least-once events that never contradict the database.

Sagas instead of distributed transactions

When a flow spans services or external rails, two-phase commit blocks on a coordinator and is impossible across organisations. A saga is a sequence of local transactions with compensations (a reversing entry, a hold release). Orchestrated sagas (Step Functions or a state-machine service) are easier to audit; choreographed ones (services reacting to events) suit simple fan-outs. Use explicit pending states so the customer sees the truth mid-flow.

Unknown outcomes

The hardest case is when you sent money to an external rail and the call timed out. Did it go through? Never assume it failed. Keep the transfer pending, query the rail's status with your reference, retry with the same idempotent reference, and let reconciliation settle it. Compensate only on a definitive failure.

Reconciliation is part of the design

Compare your ledger with external truth (network clearing files, rail statements, bank settlement reports) every day. Mismatches become exceptions with owners and SLAs. Say it in the interview: "The ledger is only right if it reconciles."

Card lifecycle: authorize, clear, settle

Authorization approves the purchase and places a hold; there's no ledger posting yet. Clearing (presentment) arrives later, typically within a few days, and creates the posting, matching and releasing the hold. Settlement moves funds between banks. Holds that never clear expire after a set number of days.

Queue, log or router

Ordering is per key, and that's enough

Logs keep order only within a partition. Choose the key so the things that must stay in order share it: account_id for balance updates. If you think you need global ordering, you almost always need per-entity ordering plus timestamps.

Failures in consumers

Retry transient errors with exponential backoff and jitter. Send messages that keep failing to a dead-letter queue, with an alert and a redrive tool. On an ordered log, one poison message blocks its partition, so park it and move on if the business allows.

Backpressure and load levelling

A queue absorbs spikes so downstream systems see a steady rate. Watch the age of the oldest message; it's a better SLI than queue length. Scale consumers on it, but cap concurrency to protect the database behind them.

Designing events

Events are facts in the past tense (CardAuthorized, TransferSettled). Use an envelope: event ID, type, schema version, time, source, subject ID and correlation ID. Evolve schemas compatibly (add optional fields; never rename or repurpose). Keep PII out: carry tokens and IDs, not names and card numbers. Every topic has one owning team.

{ "id": "evt_01J…", "type": "card.authorization.approved", "version": 3,
  "time": "2026-10-28T14:03:11.204Z", "source": "auth-service",
  "subject": "acct_123", "correlation_id": "nw_8842…",
  "data": { "auth_id": "au_…", "card_token": "tok_…", "amount": "50.00",
            "currency": "CAD", "mcc": "5812" } }

Event sourcing and CQRS, used sparingly

Event sourcing stores the events as the source of truth and derives state. It fits ledgers and audit-heavy domains, but it costs you replay tooling, schema evolution pain and harder queries. CQRS (separate write model and read models) is the lighter, more common win.

Cache patterns

Cache-aside (the app reads the cache, falls back to the database, fills the cache) is the default. Read-through and write-through push that logic into the cache layer. Write-behind acknowledges before the database has the data; avoid it for money. Invalidate with TTLs for simple cases and with change events where staleness matters.

What not to cache in a bank

Never make a money decision from a cache. The authorization check and the transfer's funds check read the system of record. Caching is fine for display ("Available $1,240 · updated 20 s ago"), offer catalogs, product rules and reference data.

Stampedes and hot keys

When a popular key expires, thousands of requests miss at once. Use request coalescing (one loader, the rest wait), jittered TTLs and early refresh. For hot keys, replicate them across cache nodes or add a small in-process cache.

Rate limiting

A token bucket allows bursts up to the bucket size and a sustained rate equal to the refill rate. Limit at the edge (API Gateway usage plans per partner) and inside the service (per customer, per card). Distributed counters live in Redis or DynamoDB. Return 429 with Retry-After. For partners, limits belong in the contract.

Load balancing, pooling and pagination

Layer 7 load balancers route by path and header and do health checks. Pool database connections (RDS Proxy in front of Aurora when Lambda is the client). Paginate with cursors, not offsets, so deep pages stay fast and results don't shift.

Consistent hashing

Map keys and nodes onto a ring so that adding or removing a node only moves about 1/N of the keys. Virtual nodes even out the load. It's how cache clusters and many partitioned stores rebalance without a full reshuffle.

Timeouts, retries, backoff and jitter

Every remote call gets a timeout derived from the caller's own budget. Retry only idempotent operations, with exponential backoff plus random jitter so clients don't retry in lockstep. Retries multiply load at every layer, so retry at one layer and use a retry budget (for example, retries at most 10% of requests).

Circuit breakers, bulkheads and load shedding

A circuit breaker stops calling a failing dependency and fails fast. Bulkheads give each dependency its own pool so one slow call can't consume every thread. Load shedding rejects excess work early (cheaply) instead of accepting it and timing out everything.

Graceful degradation: hard vs soft dependencies

Decide which dependencies are hard (the request fails without them) and turn as many as possible into soft ones. If fraud scoring is slow, fall back to rules with a lower risk appetite. If the notification service is down, authorize anyway and send later. The fallback policy is a business decision; bring the business into it.

Limit the blast radius

Use multiple Availability Zones by default. Cells are independent copies of the stack, each serving a slice of customers, so a bad deploy or poison input hurts one cell. Shuffle sharding gives each customer a random subset of workers, so one noisy customer rarely takes down another. Separate AWS accounts add security and quota isolation.

Multi-region and recovery targets

RTO is how long you may be down; RPO is how much data you may lose. AWS describes four strategies from cheap and slow to expensive and instant (DR whitepaper). Choose per workload, not per company.

*Data corruption still needs point-in-time backups, so true RPO for corruption is never zero.

Active/active without conflicts

Two regions writing the same balance will conflict. The usual pattern is a home region per account: all writes for an account go to its home, replicated asynchronously to the other region. On regional failure you move the home, accepting the replication lag as bounded risk (for example, capping authorization amounts during failover). Fail over with a data-plane mechanism you've practised (Route 53 Application Recovery Controller), not a console click, and design for static stability: the surviving region should already have the capacity it needs.

Most outages start with a change

Deploy with canaries by slice (a percentage of traffic, a BIN range), watch SLOs and business signals, and roll back automatically. Feature flags separate deploy from release. Capital One's Credit Offers API team used canary deployments to release around the clock without disrupting affiliates.

Observability and game days

Track RED metrics (rate, errors, duration) per service, distributed traces with a correlation ID from the edge, structured logs with no PII, and business SLIs. An approval-rate drop is often your first sign of an incident. Practise failure deliberately with chaos experiments and region-evacuation game days.

Threat model the data flow

Draw the flow, mark trust boundaries, then walk STRIDE at each boundary: Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege. Ten minutes of this in an interview reads as senior.

Identity everywhere

Customers: OpenID Connect, MFA, device binding, and step-up authentication for risky actions (adding a payee, changing a phone number). Services: short-lived credentials (IAM roles, mTLS), never shared secrets. Staff: SSO and just-in-time access with approval and logging. Authorization: role- or attribute-based, ideally in a policy engine you can audit.

Least privilege: the 2019 lesson

Capital One's 2019 breach is public and well analysed: a misconfigured web application firewall allowed server-side request forgery, which reached the EC2 instance metadata service and obtained credentials for a role with broad S3 access (KrebsOnSecurity). Architectural lessons: narrowly scoped roles, IMDSv2, egress controls, S3 access through VPC endpoints and bucket policies, and anomaly detection on data access. Capital One also created Cloud Custodian, an open-source policy-as-code tool for cloud guardrails. Mention lessons respectfully and briefly if it's relevant.

Protecting data

TLS in transit. Encryption at rest with KMS. Envelope encryption: a data key encrypts the data and a KMS key encrypts the data key, so keys rotate cheaply and access is auditable. Field-level encryption for the most sensitive fields. Tokenization replaces the card number with a token so most systems never touch real card data, which shrinks PCI DSS scope; encryption alone doesn't (Stripe). Classify data, minimise it, and mask it in logs.

Audit trails

Record who did what, when, from where, and why the system decided. Make records immutable (S3 Object Lock, a separate security account), keep them for the retention period, and make them searchable for investigations.

The Canadian regulatory map

Summary from Torys on B-13 and general knowledge. In an interview, say you'd confirm specifics with the compliance team, and design so residency and retention are configuration, not code.

Building blocks and when not to use them

Serverless-first, with judgment

Capital One has spoken publicly about a serverless-first approach. Serverless wins on spiky, event-driven work and removes server management. Containers win for steady high throughput, long-lived connections (voice streams, WebSockets) and tight latency control. Say it as a rule of thumb, then apply it.

Accounts, networks and guardrails

Use a multi-account landing zone (accounts per workload and environment), service control policies, and central logging and security accounts. Use private subnets and VPC endpoints so traffic to AWS services stays off the internet, with controlled egress. Express guardrails as policy-as-code.

Well-Architected as your review lens

The six pillars (operational excellence, security, reliability, performance efficiency, cost optimization, sustainability) make a good checklist for the final minutes of a design and for ARB reviews.

Cost as a design input

Talk in unit economics: cost per authorization, per statement, per active customer. Watch data transfer (cross-AZ, cross-region, NAT gateways), over-provisioning and logging volume.

Translate your GCP experience

The anatomy of a high-level design

1. Context and goals: the business outcome, with a measure.
2. Scope and non-goals.
3. Requirements: functional, and non-functional with numbers.
4. Context and container diagrams (C4 levels 1–2).
5. Key flows as sequence diagrams.
6. Data: ownership, classification, retention, residency.
7. Interfaces: APIs and events, with contracts and versioning.
8. Security and compliance: threat-model summary and controls.
9. Resilience: SLOs, RTO and RPO, DR strategy, failure modes.
10. Operations: observability, runbooks, on-call ownership.
11. Cost: unit economics and growth.
12. Rollout and migration: transition states, rollback.
13. Risks and open questions, each with an owner.
14. Decisions: links to the ADRs.

Architecture decision records

Michael Nygard's format (2011) is short: Title, Status, Context, Decision, Consequences (arc42 FAQ, Martin Fowler). Most teams add options considered and the criteria used. One decision per record. Records are immutable: you supersede an ADR, you don't edit it.

How review boards work, from both sides

As presenter: in 10 minutes cover the problem, the constraints, two or three options, your decision, the risks and what you need from the board. As reviewer: check the requirements trace to the design, the failure modes and recovery targets, security and data classification, standards and exceptions, cost, and operational ownership. Prefer approve with conditions and time-limited exceptions over rejection. Automate standards as fitness functions and policy-as-code so the board reviews judgment, not checklists.

Capability maps

A business capability is what the business does, independent of how: acquire customers, underwrite, issue cards, authorize, post transactions, bill, collect, service customers, report to regulators. Map systems onto capabilities to see duplication, gaps and where investment should go. Shared capabilities (decisioning, identity, notifications) become platform candidates.

Build or buy

Score options on differentiation (does this make us better than competitors?), time to value, total cost over 3–5 years, integration effort, regulatory and vendor risk (OSFI B-10), and exit cost. Buy the commodity, build the differentiator, and always write down the exit plan.

Migrating without a big bang

The strangler fig pattern routes traffic through a facade and moves capability by capability from legacy to new (Fowler). Supporting techniques: change data capture to keep data in sync, parallel run with automated diffing, dark launches, feature-flag cutover by slice, and a rollback for every step. Think in transition architectures, each a stable state you could stop at.

Communicating at every level

Gregor Hohpe's "architect elevator": the same decision told three ways. To executives, talk risk, cost and time to market. To product, talk capabilities and options. To engineers, talk the design and its trade-offs. Practise telling one decision all three ways.

Real-time scoring on a latency budget

The model call is usually the smallest part. Getting fresh features is the hard part: real-time aggregates (spend in the last hour, new merchant, distance from the last transaction) come from stream processing into an online store, and historical features come from an offline store. Keep training and serving features identical (one feature definition, two materialisations) to avoid training-serving skew. Always have a rules-only fallback.

Fraud is a loop, not a model

Rules and models make the decision; case management handles reviews; confirmed fraud and chargebacks come back weeks later as labels. Thresholds are business decisions trading fraud loss against declined good customers. Run new models in shadow mode first, then champion/challenger.

Model governance

Regulated banks keep a model inventory, validate models independently, document them, monitor drift and performance, and can reproduce any decision (inputs, feature versions, model version, threshold). Your ML platform's data versioning and reproducible experiments are exactly this.

Agent and LLM platforms with guardrails

A gateway in front of all model calls handles authentication, routing, PII redaction, cost and latency budgets, caching and logging. Retrieval respects the user's entitlements. Tools are explicit, versioned APIs with least-privilege scopes; risky tools need confirmation or human approval. Inputs and outputs pass policy checks (prompt injection, data leakage, unsafe advice). Every turn is audited. Evaluation suites gate every model or prompt change, with a clear human handoff path. These are all things you built for Roseline; present them as a platform.

Draw while you talk, and draw less

Use the same layout every time (left to right, numbered steps, keys next to stores). Name boxes by responsibility ("Authorization orchestrator"), not technology, until you've decided the technology. If the interview is virtual, practise in the tool they're likely to use (Excalidraw or a shared whiteboard) until drawing costs you no attention.

Check in every five minutes

Ask a short question that hands the interviewer a choice: "I can go into the ledger model or the failover next. Which is more useful?" You stay in control while showing you're collaborating.

Handle curveballs calmly

When the interviewer changes a requirement ("now it must work across two countries"), restate it, say which part of the design it affects, and adjust only that part. When you don't know something, say what you'd verify and why the design doesn't depend on it.

Recover from mistakes in the open

"I said we'd cache balances; that's wrong for the transfer path, because the decision must read the ledger. Let me fix that." Correcting yourself openly is a strong signal at this level.

Your own system as an HLD

Prepare a 10-minute talk on one of your systems using the HLD outline from Module 9: context, a container diagram, one sequence, three decisions with trade-offs, one failure you lived through, and what you'd change. Rehearse until it's 9 minutes.

1 · Available credit without over-commit

Options: a row lock in Aurora, a DynamoDB conditional write in a transaction, or an in-memory single writer per account. Decision: DynamoDB keyed by account_id, with one TransactWriteItems that conditionally decrements open-to-buy, inserts the hold and writes the decision record. It's correct by construction, takes single-digit milliseconds and scales by account. Accepted cost: no ad-hoc SQL, so analytics reads the change stream. Hot accounts: credit tranches across sub-items (Module 2, drill 3).

2 · The latency budget

No synchronous cross-region calls on this path, and no synchronous calls to anything that isn't needed for the decision.

3 · When fraud scoring is slow or down

Fraud is a soft dependency: a 60 ms timeout and a circuit breaker, then a rules-only fallback with a stricter risk appetite (for example, lower approval ceilings for high-risk merchant categories). The business owns the fallback policy; engineering owns the switch and the alert on fallback rate.

4 · Multi-region

Options: synchronous cross-region writes (correct, but every authorization pays the round trip), or a home region per account with asynchronous replication. Decision: home-region writes, with both regions receiving network traffic and forwarding to the home region. On regional failure the survivor takes over homes and applies a failover limits policy (caps on amounts) to bound the exposure from replication lag. This turns a consistency problem into a bounded, business-approved financial risk, the same idea as network stand-in. Rehearse it with game days.

5 · Idempotency and odd ordering

The network retries with the same network_txn_id, so return the original decision. A reversal can arrive before its authorization; store a reversal marker so the late authorization is cancelled on arrival. Stand-in advices are applied as holds after the fact, and reconciled.

6 · Audit and security

The decision record (inputs, rule-set version, model version, feature snapshot hash, decision) streams to S3 with Object Lock for seven years. The PCI zone is as small as possible: the gateway and the vault. Card verification cryptography runs in HSMs. Services use mTLS and narrowly scoped roles.

1 · Internal transfers need no saga

When both accounts are in the same ledger database, a transfer is a single journal entry in a single transaction. Keeping the ledger in one consistency boundary for as long as scale allows is the simplest correct design. Say this explicitly; it shows you aren't distributing for its own sake.

2 · Concurrency on one account

Use a conditional update on available balance (or SELECT … FOR UPDATE on the balance row), in the same transaction as the entry lines and the idempotency record.

3 · External transfers

An orchestrator (Step Functions Standard, or a state-machine service) holds funds by moving them to an outgoing clearing account, calls the rail with our reference as the idempotency key, and records the outcome. On "unknown" it polls status and waits for reconciliation; it compensates (reverses the entry) only on a definitive failure. The customer sees "Pending" throughout.

4 · History and read models

The outbox feeds a history read model (DynamoDB keyed by account and time) and search. Reads are eventually consistent, with the customer's own new transfer shown immediately from the transfer response (read-your-writes in the client).

5 · Reconciliation and audit

Reconcile the clearing account against the rail's settlement report daily; its balance should return to zero as transfers settle. Every entry carries who initiated it and why; entries are immutable, and corrections are new entries.

6 · When this outgrows one database

Scale vertically and add read replicas first. Next, partition ledger accounts by customer with an internal inter-partition clearing account, so cross-partition transfers become two local entries plus a saga. Mention that purpose-built ledger databases exist, and that you'd evaluate build versus buy.

1 · Keeping the promise

Each pre-approval produces a signed offer token that binds the decision, the policy version and a hash of the inputs, with an expiry. At application, the same engine re-evaluates using the token's policy version. If the applicant's information matches, the outcome matches. The promise's conditions (information unchanged, identity verified) are written into the policy and the customer disclosure.

2 · The bureau as a dependency

A bureau gateway wraps vendor APIs with timeouts, a circuit breaker and contract tests. Options during an outage: tell the customer to try later, or accept the request and notify them by email when it completes. Using a second bureau for resilience means a second policy calibration, so it's a business decision with B-10 third-party-risk implications.

3 · Personal data

Collect the minimum. Record consent with purpose and timestamp. Encrypt bureau responses at field level and keep them only as long as policy allows. Restrict access to the decision service. Analytics uses de-identified data.

4 · Explainable, governed decisions

Store reason codes for every product decision. Policy and model changes go through governance (model risk review and the architecture board) and roll out by version, so any past decision can be reproduced. Experiment freely on presentation; changes to eligibility need approval.

5 · Capability reuse

The same decisioning capability serves the Credit Offers API for affiliates (partner authentication, rate limits, data minimisation in responses). That's a capability-catalog point worth making in an architect interview.