step 15 / 32 · 6 min
What breaks first, and in what order
Systems do not fall over all at once. They fail in a fairly predictable order, and knowing it tells you what to fix and when to leave things alone.
The usual order
1. One slow query. Almost always first. A missing index, or a query that was fine at ten thousand rows and is not at ten million. Fixing it costs an afternoon and buys a year.
2. The database runs out of connections. Each application instance holds a pool; add instances and the database hits its limit. Usually solved by pooling properly rather than by a bigger database.
3. Reads outgrow one machine. Add read replicas. Writes still go to one place, reads spread out, and you accept that a replica can be a moment behind. That lag is a real behaviour to design for, not a bug.
4. The same expensive answer, computed repeatedly. Add a cache. Now you own an invalidation problem, which is a genuine cost and worth paying only when the read is both expensive and repeated.
5. Slow work blocking a request. Move it to a queue. The user gets a fast answer, the work happens shortly afterwards, and you now have to handle the job failing after the response was sent.
6. Writes outgrow one machine. Partitioning, and this is the one to delay hardest. It changes every query, complicates transactions, and once done is very difficult to undo.
The point of the order
Each step costs complexity, and complexity is paid by every person who touches the system afterwards. Teams that add a queue and a cache on day one have paid for six months of operational burden to solve a problem they do not have.
What tells you where you are
| Symptom | Step | What it costs you |
|---|---|---|
| one endpoint slow, database busy | fix the query | an afternoon |
| errors acquiring a connection | pool properly | a configuration change |
| reads slow, writes fine | read replicas | replica lag to design for |
| same answer computed repeatedly | cache | invalidation, forever |
| requests waiting on slow work | queue | failure after the response |
| writes slow on one machine | partition | every query, and hard to undo |
Measure rather than guess: slow query logs, pool usage, cache hit rate, queue depth. Guessing which step you are on is how the wrong thing gets built.