← Back to Portfolio

Engineering Notes

Essays on the backend problems worth getting right: authorization, idempotency, consistency, and the trade-offs that only show up in production.

· 15 min read

How Memory Works in AI Agents: Working, Episodic, Semantic, and Long-Term

The model is the CPU. Memory is the hierarchy you build around it, and the hard part was never storage.

ai-agentsmemoryllmragvector-databasescontext-engineeringsystem-design
Read post
· 15 min read

How Agentic Workflows Are Designed: Plan, Act, Observe, and the Control Loop

The model is one node inside a state machine, and the edges around it are where reliability is won or lost.

agentsllmlanggraphreactcontrol-loopsystem-design
Read post
· 15 min read

AI Guardrails: Input Filtering, Output Validation, and Defending Against Jailbreaks

The model is the untrusted component in your own architecture, and every control that holds is the code that does not run inside it.

ai-guardrailsprompt-injectionllm-securityowaspagentic-aisystem-design
Read post
· 15 min read

REST vs gRPC vs GraphQL: Choosing an API Contract

The question is never which protocol wins. It is which contract belongs at this boundary, because the caller decides.

api-designrestgrpcgraphqlsystem-designdistributed-systems
Read post
· 13 min read

How an API Gateway Works: The Front Door to Your System

One thin entry point implements auth, throttling, TLS, and routing once at the edge so a fleet of services can stay focused on business logic.

api-gatewaymicroservicessystem-designenvoykongawsbff
Read post
· 15 min read

How to Version and Evolve an API Without Breaking Your Clients

The whole problem exists because you cannot redeploy your clients, so every technique here is about making change invisible until they opt in.

api-designversioningrestbackward-compatibilitystripesystem-design
Read post
· 16 min read

Sessions, Tokens, and OAuth2: How Auth Works and Where It Breaks

Every auth design is a bet on one number: how long a stolen credential stays valid after you want it dead.

authoauth2jwtsessionssecuritysystem-design
Read post
· 15 min read

How Autoscaling Works: Reacting to Load Without Falling Over

Autoscaling is a control loop with minutes of dead time, and every hard part follows from that one fact.

autoscalingdistributed-systemskubernetesawsreliabilitysystem-design
Read post
· 13 min read

Backpressure: How Systems Say "Slow Down" Before They Fall Over

When work arrives faster than you can finish it, you get exactly three choices, and picking none of them is how the queue eats your memory.

backpressuredistributed-systemsload-sheddingreliabilitykafkasystem-design
Read post
· 15 min read

Cache Invalidation: The Hardest Easy Problem in Computer Science

A cache is a copy that is allowed to lie, and the entire discipline is bounding how much it lies and for how long.

cachingdistributed-systemscache-invalidationconsistencycdcsystem-design
Read post
· 15 min read

Change Data Capture: Turning Your Database Into an Event Stream

Stop publishing events next to your database writes and start reading the commit decision the database already made.

cdcchange-data-capturekafkapostgresqldebeziumdistributed-systemsevent-drivensystem-design
Read post
· 11 min read

The Stop Condition Is the Feature: Running Claude Code Loops Like You Mean It

A loop that runs is a demo. A loop you can walk away from needs four things the CLI does not give you: a budget, a real stop condition, an independent check, and a way to see what happened.

claude-codeai-agentsautomationdeveloper-toolsllm-ops
Read post
· 15 min read

Row vs Columnar Storage: Why Analytics Databases Are Built Differently

One choice about how a table is laid out on disk decides everything downstream, from compression to CPU execution to which queries are fast.

columnarolapdatabasesparquetclickhousesnowflakesystem-design
Read post
· 15 min read

Consensus for Builders: How Raft Actually Works

Consensus sounds like a theory problem until a stale leader serves a customer the wrong balance, and then it is your problem.

raftconsensusdistributed-systemsetcdreplicationsystem-design
Read post
· 15 min read

Cloud Cost Optimization: A Senior Engineer's Lens on the Bill

Cost is a non-functional requirement you trade against reliability and velocity, not a virtue you chase to zero.

cloudcost-optimizationfinopsawsarchitecturesystem-design
Read post
· 15 min read

CRDTs: How Distributed Systems Agree Without a Leader

They converge without a leader by making merge order stop mattering, and you pay for that in metadata you can never quite throw away.

crdtsdistributed-systemseventual-consistencylocal-firstcollaborationsystem-design
Read post
· 15 min read

Data Modeling: Normalization, Denormalization, and Designing for Access Patterns

The senior skill is not picking a side between normalized and denormalized; it is enumerating the access patterns and choosing per workload.

data-modelingdatabasesnormalizationnosqldynamodbsystem-design
Read post
· 15 min read

How Package Managers Resolve Dependencies: SAT Solving and PubGrub

Picking one version of every package that satisfies every constraint is provably as hard as Boolean satisfiability, and that single fact explains every resolver you have ever cursed at.

package-managersdependency-resolutionsat-solvingpubgrubalgorithmssystem-design
Read post
· 15 min read

Blue-Green, Canary, and Progressive Delivery: Shipping Without Downtime

A deploy strategy is not about avoiding bugs. It is about controlling how many users meet the bug before you can pull it back.

deploymentprogressive-deliverycanaryblue-greenkubernetessresystem-design
Read post
· 15 min read

Design Google Docs: Collaborative Editing Without Losing a Keystroke

Two people typing at the same spot is a distributed systems problem wearing a text cursor, and the answer is never to silently drop one of them.

system-designcrdtsoperational-transformationreal-timecollaborationdistributed-systems
Read post
· 17 min read

Design a Logging and Monitoring System: Centralized Logs at Scale

Logging volume routinely exceeds the production traffic it reports on, and every real design decision falls out of surviving that fact cheaply.

loggingobservabilitysystem-designkafkaclickhousedistributed-systems
Read post
· 15 min read

Design Netflix: Streaming, the CDN, and Open Connect

The hard part of Netflix was never storage or compute, it was moving tens of terabits per second of video to the right eyeballs at the right moment, which is why they built a CDN that lives inside your ISP.

system-designnetflixcdnstreamingvideodistributed-systems
Read post
· 15 min read

Design Uber: Dispatch, Geospatial Indexing, and ETA

Finding the nearest driver is a geometry problem hiding a database problem hiding a streaming problem, and the naive answer is wrong at all three layers.

system-designgeospatialh3uberdistributed-systemsreal-time
Read post
· 15 min read

Design WhatsApp: Real-Time Messaging at Planet Scale

The hard part was never the firehose of messages; it was holding tens of millions of mostly-idle connections cheaply and still guaranteeing an offline recipient eventually gets each one.

system-designdistributed-systemswhatsappwebsocketsmessagingerlang
Read post
· 15 min read

Design YouTube: Upload, Transcode, and Serve at Planet Scale

The write path is an embarrassingly parallel job queue and the read path is a CDN, and the candidates who keep those two halves separate are the ones who pass.

system-designvideotranscodingcdnstreamingdistributed-systems
Read post
· 15 min read

Distributed Locks: Why They Are Harder Than They Look

The timeout that saves you from deadlock is the exact thing that lets two clients hold the lock at once.

distributed-systemsdistributed-locksredisredlockconcurrencysystem-design
Read post
· 15 min read

How Distributed Tracing Works: Spans, Context Propagation, and Sampling

Metrics tell you the p99 got worse and logs scatter the story across twenty services; a trace is the only signal that reassembles one request as a single connected object.

distributed-tracingobservabilityopentelemetrysamplingdistributed-systemssystem-design
Read post
· 16 min read

Distributed Transactions and the Saga Pattern: Trading Isolation for Availability

A saga keeps three letters of ACID and quietly drops the fourth, and the whole pattern is a negotiation over the one it gives up.

sagasdistributed-systemsmicroservicestwo-phase-commitconsistencysystem-design
Read post
· 15 min read

How Embeddings Work: Turning Meaning Into Vectors

An embedding does not store meaning. It stores a geometry of trained similarity, and the difference is the whole engineering problem.

embeddingsvector-searchmachine-learningragsemantic-searchsystem-design
Read post
· 15 min read

How to Build Evaluation Harnesses for LLM and Agent Systems

A system you cannot measure is a system you cannot safely change, and every model swap becomes a blind bet.

evalsllmagentstestingci-cdsystem-design
Read post
· 15 min read

Design a Feature Flag Platform: Decoupling Deploy from Release

Deploy puts dormant code on the server. Release is a runtime data change that turns it on for a cohort, and the gap between them is the whole platform.

feature-flagssystem-designprogressive-deliverydistributed-systemsexperimentationsdk
Read post
· 16 min read

Design a File Storage and Sync System (Dropbox / Google Drive)

A file is not bytes in a database row, it is an ordered list of content hashes, and once you see that the whole design falls into place.

system-designdropboxstoragedistributed-systemsdeduplicationsync
Read post
· 16 min read

The GPU Economics of LLM Inference: Why Serving AI Is So Expensive

You are paying a GPU to drag the model out of memory one token at a time, and the only way to make that cheap is to share each trip across more requests.

llm-inferencegpucost-optimizationsystemsmachine-learningsystem-design
Read post
· 13 min read

HLD vs LLD: What High-Level and Low-Level Design Actually Mean

They are two altitudes of one design, and the thing that separates a staff engineer is the willingness to fly between them on demand.

system-designlow-level-designobject-oriented-designinterviewsarchitecturesoftware-engineering
Read post
· 15 min read

How a CDN Works: Edge Caching, Anycast, and the Last Mile

You cannot out-engineer the speed of light, so the only durable fix is to move the bytes closer.

cdncachinganycastdnsnetworkingsystem-design
Read post
· 15 min read

How a Search Engine Works: The Inverted Index (Lucene and Elasticsearch)

Search looks like a read-time lookup, but every fast query is paid for at write time by an index that did the hard work in advance.

searchinverted-indexluceneelasticsearchbm25system-design
Read post
· 15 min read

Leader Election: How Distributed Systems Pick a Boss and Survive Losing One

Picking a leader is the easy half; the murder is in agreeing the previous one is actually dead.

leader-electiondistributed-systemsconsensusraftzookeepersystem-design
Read post
· 16 min read

How Ranking Works: From BM25 to Learning-to-Rank

Retrieval finds the few hundred candidates worth looking at; ranking decides the exact order, and they are different jobs with different costs.

searchrankinglearning-to-rankbm25information-retrievalmachine-learningsystem-design
Read post
· 15 min read

How to Design a Ledger: Double-Entry Accounting for Engineers

A balance is not a number you store, it is a sum you prove every time someone asks.

ledgerdouble-entryaccountingdatabasesfintechsystem-design
Read post
· 15 min read

How LLM Inference Serving Works: Continuous Batching, KV-Cache, and Paged Attention

Generating a token is mostly waiting on memory, which is the one fact that explains every serving trick worth knowing.

llm-inferencegpuvllmkv-cachepagedattentionsystem-design
Read post
· 19 min read

How LLM Quantization Works: FP16 to INT4, GPTQ, AWQ, and the Accuracy Trade

Naive rounding destroys a large model because of one percent of its numbers, and the whole field is three different ways to protect them.

llmquantizationgptqawqggufinferencemachine-learning
Read post
· 16 min read

Design a Stock Exchange: The Matching Engine, the Order Book, and Determinism

The fastest matching engine in the world runs on one thread, and that is the optimization rather than the compromise.

system-designmatching-enginelow-latencydeterminismevent-sourcingorder-book
Read post
· 15 min read

MCP: How AI Agents Get Hands (the Model Context Protocol)

A tool the model can call is an action it can be tricked into taking, and MCP makes connecting those tools the easy part while trusting them stays hard.

mcpai-agentstool-callingllmsecuritysystem-design
Read post
· 15 min read

How Model Routing and Cascades Work: The Right Query to the Right Model

Paying frontier prices for a FAQ lookup is the default waste, and a good router is the cheapest engineering you will ever ship to fix it.

llmmodel-routingcascadescost-optimizationinferencesystem-design
Read post
· 15 min read

Designing Multi-Agent Systems (and When a Single Agent Is Better)

Most designs that call themselves multi-agent should be one agent with good tools, and the token bill is the reason why.

multi-agentai-agentsllmorchestrationsystem-designagentic
Read post
· 13 min read

Multi-Region and Disaster Recovery: RPO, RTO, and the Cost of Always-On

Active-active is not the summit everyone should climb; for most systems it is a multi-master write problem with a permanent latency tax and a legal minefield.

disaster-recoverymulti-regiondistributed-systemscap-theoremawssystem-design
Read post
· 15 min read

MVCC and Isolation Levels: How a Database Serves Concurrent Transactions

Snapshot isolation feels like serializability right up until two transactions read the same rows, write different ones, and quietly break an invariant nobody guarded.

mvccisolation-levelstransactionspostgresqlmysqldatabasessystem-design
Read post
· 17 min read

Design a Notification System: Push, SMS, and Email at Fan-Out Scale

Every delivery pipeline guarantees at-least-once, and your users feel every duplicate as spam. The whole design is a negotiation between those two facts.

system-designnotificationsdistributed-systemspush-notificationskafkaidempotency
Read post
· 16 min read

Database Sharding: Splitting Data When One Machine Is Not Enough

Sharding does not make your database faster. It trades a capacity ceiling for a permanent tax on every query that forgets the shard key.

shardingpartitioningdistributed-systemsdatabasesscalabilitysystem-design
Read post
· 15 min read

Design a Presence System: Who Is Online at Scale

A senior engineer does not try to make the green dot always correct; they write down exactly how wrong it is allowed to be.

presencedistributed-systemsreal-timerediswebsocketssystem-design
Read post
· 13 min read

Bloom Filters, HyperLogLog, and Count-Min Sketch: Being Wrong on Purpose to Save Memory

The skill is not knowing these structures save memory; it is knowing exactly which lie each one buys you.

data-structuresalgorithmsbloom-filterhyperloglogcount-min-sketchsystem-design
Read post
· 15 min read

How Quantum Computing Differs From the Chip in Your Laptop

It does not try every answer at once, and believing that it does is the fastest way to misjudge what it can actually do.

quantum-computingqubitsalgorithmscomplexity-theorycryptographycomputer-science
Read post
· 16 min read

WebSockets vs SSE vs Long Polling: Choosing a Real-Time Transport

A shallow answer reaches for WebSockets. A senior answer makes you justify why Server-Sent Events were not enough.

websocketsssereal-timesystem-designhttpscalability
Read post
· 16 min read

How Recommendation Systems Work: From Collaborative Filtering to Two-Tower Retrieval

The metric that wins the contest is almost never the product, and the gap between them is the whole education.

recommendation-systemsmachine-learningmatrix-factorizationembeddingsvector-searchsystem-design
Read post
· 15 min read

Retries, Timeouts, and Circuit Breakers: Keeping One Failure From Becoming an Outage

A dependency that fails fast returns your thread. A dependency that hangs holds it, and held threads are how one slow service drains an entire fleet.

resiliencedistributed-systemscircuit-breakerreliabilitysresystem-design
Read post
· 15 min read

Design Search Autocomplete (Typeahead) That Feels Instant

Sub-100ms suggestions are a data-structure problem wearing a latency problem as a disguise.

system-designautocompletetypeaheadtrielatencycaching
Read post
· 15 min read

Secondary Indexes: How Databases Find a Row Without Scanning Every One

An index makes reads fast because it makes writes do more work, and that trade is the whole story.

databasesindexesstorage-enginespostgresqlmysqldynamodbsystem-design
Read post
· 15 min read

SLOs and Error Budgets: Deciding What "Reliable Enough" Means

An error budget turns the endless fight between shipping fast and staying up into a single number both sides spend against.

reliabilitysresloerror-budgetsobservabilitysystem-design
Read post
· 16 min read

R-Trees and Spatial Indexing: How "Find Nearby" Actually Works

A B-tree sorts one column; the plane has no sort order that keeps neighbors close, so proximity needs a structure that reasons about regions.

spatial-indexingr-treesgeohashpostgisdatabasessystem-design
Read post
· 13 min read

How Stream Processing Works: Windows, Watermarks, and Exactly-Once

Every hard part of stream processing is the same problem wearing a different hat: you can never be sure you have seen everything yet.

stream-processingflinkkafkadistributed-systemsevent-timesystem-design
Read post
· 15 min read

Threat Modeling for System Design: The Security Pass a Senior Runs

Most security work happens after the breach. Threat modeling is the half hour at the whiteboard that moves the fight earlier, before any code exists to attack.

securitythreat-modelingstridesystem-designowaspllm-security
Read post
· 15 min read

How Time-Series Databases Work: Why Metrics Need a Different Engine

A metrics workload is append-only at "now" and read in windows, and a purpose-built engine wins by exploiting that shape to do less work and store far fewer bytes.

time-seriesdatabasesobservabilityprometheuscompressionsystem-design
Read post
· 14 min read

How Tokenization Works: Why LLMs See Tokens, Not Words

The tokenizer is a separate frozen artifact bolted to the front of the model, and almost every weird LLM failure traces back to it.

tokenizationllmbpenlpai-engineeringmachine-learning
Read post
· 17 min read

How Tool-Calling Works: Function Schemas, the Model, and the Execution Loop

The model never runs your tool. It emits a JSON object that names one, and everything that matters lives in the loop you wrap around that.

llmtool-callingfunction-callingagentsstructured-outputssystem-design
Read post
· 15 min read

Generating Unique IDs at Scale: UUIDs, Snowflake, and ULID

Picking an ID scheme is a three-way bet between coordination, sortability, and index locality, and no scheme wins all three.

system-designdatabasesdistributed-systemspostgresqluuidscalability
Read post
· 15 min read

CAP, PACELC, and the Consistency Spectrum (Beyond "Pick Two")

A partition forces a choice between consistency and availability, but the rest of the time you are trading latency against consistency on every single operation.

cap-theorempacelcconsistencydistributed-systemsdatabasessystem-design
Read post
· 13 min read

Capacity Estimation on a Napkin (Without Fooling Yourself)

The estimate that cannot change your decision should never be computed. Everything else is theater with extra significant figures.

system-designcapacity-planningestimationdistributed-systemsscalabilityinterviews
Read post
· 15 min read

Consistent Hashing and the Art of Rebalancing

The ring is not the answer. It is the first move in a design space, and knowing what the ring still gets wrong is the whole skill.

consistent-hashingdistributed-systemsshardingload-balancingdynamosystem-design
Read post
· 15 min read

Choosing a Database by Data Structure, Not by Logo

The database name is the last decision you make, and the most reversible one, so stop making it first.

databasessystem-designdata-structureslsm-treeb-treepostgresql
Read post
· 16 min read

Design Instagram: The Feed and the Media Pipeline

The feed never stores a single pixel, and the day you understand why is the day this problem gets tractable.

system-designinstagramfeedblob-storagecdndistributed-systems
Read post
· 15 min read

Design Twitter/X: Timeline Fan-Out and the Hybrid Push/Pull Model

The architecture is a function of the graph, and the celebrity is where every pure strategy goes to die.

system-designdistributed-systemstwitterfan-outcachingscalability
Read post
· 13 min read

Inside a Distributed Cache: Eviction, Sharding, Hot Keys and the Thundering Herd

A cache is a staleness budget you spend on speed, and every interesting failure is the bill arriving at once.

cachingdistributed-systemsredismemcachedconsistent-hashingsystem-design
Read post
· 15 min read

Event-Driven RBAC: Authorization When Every Request Asks Permission

Permissions become a caching problem long before they become a scaling problem. The real skill is knowing which half of that problem is allowed to be slow.

authorizationrbacsystem-designdistributed-systemspostgresql
Read post
· 16 min read

How an LLM Actually Works: Transformer Inference for System Designers

An LLM at inference is a bandwidth-bound streaming loop with a quadratic cost term and a cache that quietly decides your per-token price.

llmtransformersinferencegpusystems-designkv-cache
Read post
· 17 min read

Idempotency and the Exactly-Once Lie

Every system that sells exactly-once is really at-least-once delivery plus a dedup trick at a boundary someone chose carefully.

idempotencyexactly-oncedistributed-systemskafkastream-processingsystem-design
Read post
· 14 min read

Idempotent Webhooks: Making At-Least-Once Delivery Behave Like Exactly-Once

Exactly-once delivery is a myth the network will never grant you. Exactly-once processing is a unique constraint and a little discipline.

webhooksidempotencydistributed-systemsstripepostgresqlsystem-design
Read post
· 15 min read

Log-Based Brokers vs Message Queues: Kafka, SQS and RabbitMQ Under the Hood

The split is not streaming versus messaging. It is whether the broker deletes a message when you acknowledge it, or keeps it and tracks where you are.

kafkasqsrabbitmqdistributed-systemsmessage-queuesystem-design
Read post
· 16 min read

Latency vs Throughput, and the Tyranny of the Tail

The average latency is a number no real request experiences, and at fan-out the tail you ignored becomes the median your users feel.

latencythroughputtail-latencydistributed-systemsperformancesystem-design
Read post
· 16 min read

LSM-Tree vs B-Tree: The Storage Engine Decision That Defines Your Write Path

Every storage engine is a bet on two of three things, and the third one degrades whether you planned for it or not.

databasesstorage-engineslsm-treeb-treerocksdbpostgresqlsystem-design
Read post
· 17 min read

Metrics, Logs and Traces: Designing the Three Pillars Without Going Bankrupt

Each signal bills on a different cost axis, and the senior skill is choosing which one you are buying before observability outgrows the system it watches.

observabilitymetricstracingloggingsloopentelemetrysystem-design
Read post
· 17 min read

The Payment System Nobody Sees Fail: Gateways, Idempotency, Retries and the Ledger

A payment system is judged by what it never does: lose a dollar, double a charge, or trust the gateway over its own books.

paymentsidempotencyledgerdouble-entrydistributed-systemssystem-design
Read post
· 16 min read

Designing a RAG System: From Chunk to Answer, and Every Way It Breaks

Retrieval and generation fail independently, so you have to instrument both or you will blame the model for a retrieval miss.

ragllmretrievalembeddingsvector-searchsystem-design
Read post
· 13 min read

Design a Rate Limiter: Token Bucket vs Sliding Window

The algorithm is the easy half. The interview is won on the race condition between two nodes and the failure mode when Redis is gone.

rate-limitingdistributed-systemsredissystem-designtoken-bucketapi-design
Read post
· 16 min read

Replication Strategies That Survive Failure

Every replication design is a bet about which failure you can tolerate, and the bet only comes due when a node dies.

replicationdistributed-systemsconsistencypostgresqldynamoraftsystem-design
Read post
· 17 min read

The System Design Interview, Reverse-Engineered: A Repeatable Framework

You do not fail system design for not knowing enough. You fail for how you behave under ambiguity.

system-designinterviewsdistributed-systemsstaff-engineerarchitecturecareer
Read post
· 15 min read

Design a URL Shortener (TinyURL / bit.ly): The Read Path Is the Whole Design

The write path can be almost anything; every hard decision in this system lives on the read path, and the candidate who optimizes writes has misread the problem.

system-designurl-shortenercachingdistributed-systemsinterviewsscalability
Read post
· 15 min read

How Vector Databases Work: HNSW, IVF and Product Quantization

Every vector index is the same three-way bet between recall, latency, and memory, and the whole job is knowing which corner to give up.

vector-databasesann-searchhnswragembeddingssystem-design
Read post