Running, waiting, and the cost of swapping

A process is rarely running. On a machine with eight cores and four hundred processes, at most eight are running at any instant and the rest are in one of a few other states.

The states

    new ──> ready <───────┐
             │            │
          scheduled    interrupt or
             ↓         time slice ends
          running ──────┘
             │
        waits for I/O
             ↓
          blocked ──> back to ready when the I/O completes
             │
          terminated

The distinction that matters: ready means it could run and no core is free. Blocked means it could not run even if a core were free, because it is waiting for a disk, a network packet or a lock.

A machine with high load and low CPU usage is full of blocked processes, and adding cores does nothing for it.

The context switch

Changing which process a core runs means saving everything the old one was doing and restoring the new one:

• the program counter and every register
• the stack pointer
• the memory mapping, which means changing the page table base

save registers of A     ~ hundreds of nanoseconds
switch page tables      ~ and this flushes the TLB
restore registers of B

The direct cost is roughly one to ten microseconds. The indirect cost is usually larger and is invisible: process B starts with a cold cache and a flushed TLB, so its first thousand memory accesses are slow. Work that was instant for A has to be fetched again when A returns.

Why this shapes design

The last row is why event driven servers exist. Ten thousand connections handled by eight threads switch rarely; ten thousand threads spend the machine on context switches and never reach the work.

Switching between threads of one process is cheaper, because the memory mapping does not change and the TLB survives. That is one of the main practical arguments for threads over processes.