Everything in Part II so far has happened on one card, and the last meeting broke that assumption: disaggregation needs at least two instances, and once there are two, something has to choose. Load balancing is the oldest solved problem in distributed systems, and almost none of the solutions apply here — because an LLM replica has warm state. A request is not equally cheap everywhere: the replica holding its prefix can serve it in 12 ms of prefill instead of 157, so the router's job is not to equalize queues but to trade queueing against locality, and Lecture 13 will price that trade at 129 ms of tolerable extra queueing for a 4K prefix and 14 ms for a 512-token one. Llumnix then adds a move no web load balancer has: migrate a request that is already running, cache and all, for 1.3 ms over NVLink. By the end you should be able to say why least-connections is wrong here, price a migration against a recompute, write the affinity-versus-balance rule as an inequality, and explain why a sparse model's decode step reads every expert once the batch is larger than about thirty.

Where this sits

The last five meetings built a good single instance. Lecture 10 fixed the allocator and reached B = 164 at the SLO; Sep 28 and Sep 30 made the kernels efficient and left them a fixed cost; Oct 5 re-formed the batch every iteration and recovered 4.05×; Oct 7 stopped prefill and decode from fighting, either by chunking within a step or by separating them onto pools — and that last answer is what makes today unavoidable. A disaggregated deployment has at least two instances by construction, and any real deployment has many.

So the question changes from "how do I run this request well?" to "where should this request run?" That is a placement problem, and it is the first one in Part II whose answer depends on facts about other machines. Today is also the last meeting before the semester turns to the cache itself: Oct 19 shrinks it and Nov 4 reuses it across requests, and both will assume that requests can be steered toward the memory they need. Today is what makes that assumption true.

Instructor notes — Timing plan

75-minute class, student-led. One required paper, and two protected results the instructor should hold: the migration price (§17.4) and the affinity inequality (§17.5). §17.7 is a different topic sharing a word — keep it short or make it reading-only.

Time	Segment	Notes
0–6	The new decision	One card became many. Three things a router trades.
6–20	§17.2 Why this is not web load balancing	Protected. Four differences, each with a number. Kill least-connections explicitly.
20–34	§17.3 Llumnix: migrate a running request	The mechanism, and its dependence on Lecture 10's block table.
34–44	§17.4 The migration price	Protected — derive live. 1.3 / 8.4 / 21.5 ms against 129 ms of recompute.
44–56	§17.5 Affinity against balance	The inequality, then Lecture 13's 129 ms and 14 ms.
56–64	§17.6 Many models on one fleet	AlpaServe's inversion, then S-LoRA's adapter arithmetic.
64–71	§17.7 The other routing: MoE	The (1−k/E)^B result is the whole section. Keep it tight.
71–75	§17.8 Wrap	What a router can observe. Point at Oct 19.

Reading-only, not scheduled: §17.7 beyond the one derivation, and the DynamoLLM/Clipper material.

If running long: cut §17.6 to the S-LoRA arithmetic alone and state §17.7's result without deriving it. Never cut §17.4 or §17.5 — the price of a migration and the affinity rule are what a student should still be able to compute in six months.

Learning objectives

By the end of this class you should be able to:

1. Name three quantities a router trades, and explain why they cannot be reduced to one.
2. Give four reasons least-connections and round-robin are wrong for LLM replicas, each grounded in a number from this course.
3. Describe live migration of a decoding sequence, and state the property of the KV allocator it depends on.
4. Price a migration against killing and recomputing, and compute the break-even bandwidth.
5. Write the cache-affinity-versus-load-balance decision as an inequality, and evaluate it for a given prefix length and queue depth.
6. Explain why splitting a model that already fits can reduce tail latency.
7. Compute the memory cost of serving many LoRA adapters against one base model.
8. Explain why a mixture-of-experts model's decode step reads nearly all expert weights once the batch exceeds a few tens of sequences, and what that means for serving.

17.1 One card became many, and now something must choose

A deployment is a fleet of replicas behind a router. The router sees an arriving request and picks a replica. That is the entire interface, and there are exactly three things it can be trying to do.

Minimize queueing. Send the request where it will start soonest. This is what classical load balancing optimizes, and it is the only one of the three that a stateless system has.

Respect memory. A replica can be compute-idle and memory-full. Lecture 10 §10.1 said admission is a promise about memory that the engine cannot take back; at the fleet level that becomes a placement constraint — a replica with 2 GB of free KV cannot accept a 32K-token prompt no matter how idle its tensor cores are.

Exploit locality. A replica that already holds this request's prefix can skip its prefill. Lecture 13 will make this precise, but the magnitude is already visible: an average agent step's prefill is 157 ms cold and 12.5 ms warm, a 12.6× difference that no amount of queueing cleverness can produce.

Those three conflict, and they conflict in a way that has no scalar objective, because they are denominated in different currencies — milliseconds of delay, bytes of capacity, and milliseconds of avoided work. Most of today is about the third one buying the first one, at an exchange rate you can compute.

Instructor notes

Minutes: 6. Board: Three words in a column — QUEUE / MEMORY / LOCALITY — with "ms", "bytes", "ms of avoided work" beside them. Circle that the units differ. Ask the room: "Which of the three does an HTTP load balancer know about?" Only the first. That is the lecture's premise.

17.2 Why this is not web load balancing

The standard policies — round-robin, least-connections, least-response-time — are correct under assumptions that all fail here. Take them one at a time, with a number each, because the failure modes are specific rather than general.

Requests are long-lived and their duration is unknown at arrival. A web request is milliseconds; a generation request occupies a decode slot for G × TPOT, which at 300 tokens and 22.6 ms is 6.8 seconds, and at Lecture 5's 20-step agent it is far longer. Worse, Lecture 11 §11.7 established that G is unknown when the routing decision is made. So the router is placing a job whose duration it cannot estimate into a replica whose future load it therefore cannot project. Least-response-time policies, which assume the recent past predicts the near future, are estimating from a distribution whose variance is three orders of magnitude wide.

A request owns memory for its whole life, so "load" is two-dimensional. Connection count is a proxy for load only when every connection costs the same. Here a replica running 164 short sequences and one running 20 long ones can have identical connection counts and completely different memory headroom — 164 × 0.328 GB against 20 × 8.6 GB at 32K. Least-connections routes on the wrong dimension, and will happily send a long-prompt request to the replica least able to hold it.

Service time is heterogeneous by three orders of magnitude. A 512-token prefill is 14.2 ms; a 32K-token prefill is 8.4 seconds. Lecture 11 §11.4 showed what that does to a single queue — head-of-line blocking worth 2.6× on mean TTFT — and a router that treats requests as interchangeable reproduces that problem once per replica.

And the decisive one: replicas are not interchangeable, because they have warm state. This is the break with everything classical. A replica's KV cache is a function of what it has recently served, so the cost of a request depends on where you send it. Sending an agent's step 7 to the replica that served steps 1–6 costs 12.5 ms of prefill; sending it anywhere else costs 157 ms. The router is therefore making a caching decision, and a routing policy that ignores it is discarding a 12.6× — which is exactly what Lecture 13 §13.5's self-check 6 describes as a fleet showing 95% hit rate per engine and 30% fleet-wide.

The positive statement is worth making explicitly, because it reframes the rest of the lecture. An LLM router is a cache-placement policy that also has to respect queueing and capacity. That is a genuinely different problem from load balancing, and it is why the required paper is about moving state rather than about choosing queues.

Instructor notes

Minutes: 14. Protected. Board: Four bullets, and beside each the number: 6.8 s, two dimensions, 14.2 ms vs 8.4 s, 12.6×. Ask the room: "Two replicas, same connection count. Which is more loaded?" Unanswerable without memory and context lengths — which is the point. Expect confusion: Students propose "least KV bytes free". Better, still incomplete: it ignores locality, and it will steer requests away from the replica holding their prefix.

17.3 Llumnix: move the request, not just the choice

Every policy so far decides once, at arrival, and lives with it. Llumnix's observation is that the information which makes a placement wrong mostly arrives later: the request turns out to be long, another replica frees up, memory fragments, a node needs draining. So add a mechanism — migrate a running request between replicas — and the placement becomes revisable.

The mechanism has to move a decoding sequence's KV cache without interrupting its generation, and the way it does so is the same trick a live VM migration uses. The sequence keeps decoding on the source replica while its already-written KV blocks are copied to the destination; the blocks produced during the copy are copied in a second, much smaller pass; then execution cuts over. Because generation appends and never rewrites — the prefix property again, and the same reason Lecture 12's chunking was sound — the copy is chasing a target that only grows at one token per step, so it converges after one round.

This is only possible because the cache is paged. Lecture 10 §10.3 made a sequence's KV a list of block references, so migrating it means copying a set of blocks and rebuilding a small table on the destination. Without paging, the cache is a slice of one contiguous per-replica tensor, and there is nothing to copy out — you would have to serialize the sequence's state from the middle of somebody else's array and find contiguous room for it on the far side. The paper is a direct dividend of the allocator, which is the answer to its own reading question.

What the mechanism buys is four things at once, and the paper's real claim is that they are one problem.

Load balance that survives being wrong. A replica that becomes hot can shed running work, not just stop accepting new work. That converts a routing mistake from permanent to temporary.

Defragmentation across replicas. Lecture 10 §10.2 named external fragmentation and paging removed it within a replica. At the fleet level it returns in a new form: 4 GB free on each of eight replicas cannot host a 32K-token request that needs 8.6 GB contiguous in one replica's budget. Migration consolidates, and this is the case that has no other solution.

Priority. A high-priority arrival can displace resident work rather than queue behind it, with the displaced request continuing elsewhere instead of being killed.

Draining. A node can be emptied for maintenance without dropping requests, which is an operational requirement rather than a performance one and is often what actually sells the feature.

Instructor notes

Minutes: 14. Board: Source and destination replicas, a sequence's blocks copying across while a token is appended on the left. Then the four objectives as four words. Ask the room: "Why does the copy converge?" Generation appends one token per step; the copy moves thousands of blocks. Make them see the rate difference. Expect confusion: Students think migration must pause the sequence. It pauses it for the cutover only — the copy overlaps with decoding.

17.4 What a migration costs

Derive it, because the number is small enough to be surprising and it is what licenses the whole mechanism.

Three observations, in order of how often they get missed.

First, migration is cheap in the regime that matters. Over NVLink a 4K GQA-8 sequence moves in 1.3 ms, which is 6% of one decode step — the sequence's user does not perceive it. Even over commodity Ethernet it is 21.5 ms, under one step. A mechanism this cheap can be used liberally, which is why Llumnix can afford to treat migration as a scheduling primitive rather than an emergency measure.

Second, the numbers are identical to Lecture 12 §12.5's and Lecture 10 §10.5's, and that is not a coincidence. Moving a KV cache between instances, shipping it from a prefill pool to a decode pool, and swapping it out to host memory are the same operation over different links, and the same break-even — 16.6 GB/s MHA, 4.2 GB/s GQA-8 — governs all three. This is the fourth appearance of that threshold, and it returns on Nov 4 as fetch-versus-recompute against a remote store and on Nov 23 as keep-versus-swap for a stalled session. It is worth memorizing once.

Third, the cost is linear in context length while the recompute it avoids is quadratic, so migration gets relatively cheaper for long sequences. At 16K GQA-8 the cache is 2.147 GB (5.4 ms over NVLink) while the prefill it replaces is 361.6 TFLOP ≈ 731 ms — a ratio of 135× rather than 99×. Long-context deployments are exactly where migration pays most, which is convenient, because they are also where fragmentation bites hardest.

Instructor notes

Minutes: 10. Protected — derive live. Board: 0.537 GB, then divide by three bandwidths. Then 129 ms beside it. Then the break-even. Ask the room: "Would you rather migrate or restart?" Then: "at what bandwidth does that flip?" Let them compute 4.2 GB/s. Expect confusion: Students think 129 ms is the fair comparison for a decoding request. It is generous to restarting — restarting also loses queue position and re-runs any tool side effects, so the true comparison is worse.

17.5 Cache affinity against load balance

Now the central tension. The replica holding your prefix may be busy. Do you wait for it, or go somewhere cold?

Write it as an inequality. Let q_warm be the queueing delay at the replica holding the prefix, q_cold the delay at the least-loaded replica, and Δ_prefill the prefill you would save by going warm.

The right-hand side is computable from the prefix length, and Lecture 13 §13.6 will do exactly this arithmetic: a 4,096-token prefix is worth up to 129 ms of extra queueing, and a 512-token prefix only 14 ms. That is a factor of nine in tolerable imbalance from one property of the request, which is why a single "affinity weight" tuned once is the wrong shape of solution.

Two structural difficulties follow, and they are what make this a research area rather than a configuration setting.

The router needs information it does not own. Δ_prefill depends on which prefix is resident where, which is a property of eight replicas' caches at this instant. So the router needs a fleet-wide index of resident prefixes, kept current, and consulted per request. That is Preble's hierarchical design — a global scheduler with a prefix map over local per-engine schedulers — and it is why Preble is optional today and required on Nov 4.

Affinity concentrates load by construction. A popular prefix — a shared system prompt, a hot document — attracts every request that shares it, so the replica holding it becomes the hottest in the fleet precisely because it is the most useful. The answer is replication: copy the hot prefix to several replicas and load-balance among them, at a cost of 0.537 GB per 4K GQA-8 copy. That converts a locality problem back into a capacity problem, which is the trade you want, and it is the point where routing and caching stop being separable — the router's decisions determine the cache's contents, and the cache's contents determine the router's decisions.

There is a third difficulty worth naming because it is uncomfortable: a cache hit is observable through TTFT. If two tenants share a system prompt, the second one's faster first token is evidence about the first one's traffic. Lecture 13's discussion seeds return to this; today it is enough to notice that a performance optimization has created a cross-tenant signal, and that the mitigations (partitioning the cache per tenant, or padding TTFT) cost exactly the thing you built it for.

Instructor notes

Minutes: 12. Board: The inequality, then the three-arrival table. Do not give the third row's answer — ask. Ask the room: "What does the router need to know that it currently cannot see?" Which prefixes are resident where. Then name Preble and Nov 4. Expect confusion: "Just always prefer the warm replica." Show the 512-token row: it loses 66 ms.

17.6 Several models on one fleet

Two variants of today's problem arise when the fleet does not serve one model, and both invert an intuition.

AlpaServe: split a model that already fits. Model parallelism is usually a capacity tool — you shard because the weights do not fit on one device (Lecture 2 §2.8's 107.8 GB). AlpaServe's result is that sharding a model which does fit can reduce tail latency, because it changes the queueing structure. With four independent replicas, a burst of four long requests can all land on one replica and queue behind each other; with the model sharded across four devices serving one logical queue, the burst is spread across all four by construction. Parallelism converts variance in placement into throughput, at the price of the collectives Lecture 2 §2.13 counted — 64 small all-reduces per token, landing on decode's critical path. So the trade is tail latency against per-token latency, and which side wins depends on burstiness rather than on size.

S-LoRA: thousands of adapters, one base model. A LoRA adapter replaces a weight update with two low-rank factors, so serving many fine-tunes need not mean many copies of the weights. The arithmetic is worth doing because it explains why this is a serving story at all.

Read both halves of that. Individually an adapter is nearly free, which is why the multi-tenant fine-tune business model exists at all. Collectively a thousand of them is a real memory line item competing with the KV cache — so the serving problem is which adapters to keep resident, with the same admission and eviction structure as every cache in this course, plus a batching problem: a step containing requests for different adapters cannot use one GEMM for the adapter math the way Lecture 11 §11.3's shared weights could. That is what S-LoRA's custom kernels are for, and it is the same ragged-batching shape as attention.

Instructor notes

Minutes: 8. Board: "shard a model that fits?" then AlpaServe's answer in one line. Then 33.6 MB / 0.25% / 1,000 adapters = 33.6 GB. Ask the room: "Adapters are 0.25% each. So what is the problem?" Multiply by a thousand. Expect confusion: Students assume adapter serving is free because the math is small. The math is small and ragged, which is the expensive kind.

17.7 The other routing: tokens to experts

Mostly reading-only; the one derivation below is worth class time.

"Routing" means something else inside the model, and it lands in a serving lecture for a reason worth understanding. In a mixture-of-experts layer, a learned gate sends each token to k of E expert MLPs. GShard and Switch Transformer established the machinery: top-k gating, a capacity factor bounding how many tokens one expert may accept per batch, and the consequence that tokens routed to a full expert are dropped — they skip the layer via the residual connection. Load imbalance across experts is therefore an accuracy problem as well as a performance one, which is unusual and is why the literature spends so much effort on auxiliary balancing losses.

For serving, the important question is what MoE does to the byte count, since Lecture 2 established that decode is bandwidth-bound. The advertised win is FLOPs: with top-2 of 8 experts, a token activates a quarter of the expert parameters. The trap is that bytes are counted per step, not per token, and a step contains many tokens with independent routing.

That result reframes sparse models for serving. MoE buys FLOPs per token, which matters for prefill (compute-bound) and for training. It buys nothing in decode bytes at any batch size a throughput-oriented deployment would run, because the union of activated experts saturates. What it costs is capacity: the full parameter set must be resident or fetchable, so a sparse model with 8× the parameters at 2× the active FLOPs needs 8× the memory. Hence the serving literature on MoE — Fiddler, MoE-Infinity, pre-gated MoE in the readings — is largely about placement and prefetch of expert weights, which makes it a routing problem in today's sense too: the question is which expert weights live on which device, and how a token gets to them.

17.8 What a router can actually observe

Close on the gap between the policies above and what is implementable, because it is where the research is.

The decisions want: each replica's queue depth, its free KV bytes, its resident prefixes, the arriving request's prompt length, and its output length. The first two are cheap — a replica can report them every few hundred milliseconds, and staleness on that timescale is tolerable because the quantities change slowly relative to a routing decision. The third is expensive and is the whole design problem of a fleet-wide prefix index. The fourth is free. The fifth is unavailable (Lecture 11 §11.7), and every policy above therefore has to be robust to not knowing it — which is precisely what migration provides: a mechanism for being wrong cheaply.

That is the honest summary of the lecture. The router cannot know enough to be right, so the architecture that wins is the one that makes mistakes revisable at 1.3 ms rather than the one that tries to make better predictions.

Discussion seeds

1. The policy you would ship. Write a routing rule using only what §17.8 says is cheaply observable, and identify the workload on which it performs worst.
2. Replication as an eviction problem. A hot prefix attracts load until you replicate it, at 0.537 GB per 4K GQA-8 copy. What runtime signal tells you a prefix is hot enough, and what tells you a replica should drop its copy?
3. AlpaServe's inversion, tested. Construct a workload where sharding a model that fits clearly wins on p99, and one where it clearly loses. What single statistic separates them?
4. Migration as a fairness tool. Lecture 11 §11.4 noted that per-request scheduling favours whoever issues more requests, and Nov 18 will make that Autellix's argument. Could migration be used to enforce fairness across programs rather than requests? What would it need to know?
5. The timing channel. A cache hit is visible in TTFT, so a shared prefix leaks information across tenants. Price the mitigations — per-tenant cache partitioning, and TTFT padding — in the units of §17.5's inequality. Is either one acceptable?
6. Sparse models, honestly. §17.7 says MoE's bandwidth advantage vanishes above B ≈ 30. Under what deployment would you still choose a sparse model, and what does your answer say about who MoE is really for?

Key takeaways

Numbers worth memorizing

Quantity	Value	Source
A 300-token request's slot occupancy	300 × 22.6 ms = 6.8 s	§17.2
Service-time spread	14.2 ms (512 tok) to 8.4 s (32K tok) prefill	§17.2
Cold vs warm agent step	157 ms vs 12.5 ms (12.6×)	Lecture 13 §13.5
Migration, 4K GQA-8	1.3 / 8.4 / 21.5 ms (NVLink / PCIe / Ethernet)	§17.4
Kill-and-recompute alternative	129 ms	Lecture 13
Break-even bandwidth	16.6 GB/s MHA, 4.2 GB/s GQA-8	2.147 (0.537) GB ÷ 0.129 s
Migration at 16K GQA-8 vs its recompute	5.4 ms vs 731 ms (135×)	linear vs quadratic
Affinity budget: 4K / 512-token prefix	129 ms / 14 ms of tolerable extra queueing	§17.5
Replication cost	0.537 GB per 4K GQA-8 copy	§17.5
LoRA rank 16, whole model	16.78M params = 33.6 MB = 0.25% of base	§17.6
1,000 adapters	33.6 GB — over half the KV budget	§17.6
P(expert unused), top-2 of 8	(0.75)^B: 0.75 at B = 1, 0.0002 at B = 30	§17.7

Self-check

Exercises

1. A router with only cheap signals. Using only queue depth, free KV bytes, and prompt length — the three things §17.8 calls cheap — write a scoring function for replica selection. Then find the workload on which it does worst, and quantify the loss. Solution sketch: A defensible score is q_i + α · max(0, need − free_i) with a large α making infeasible replicas unattractive, breaking ties toward more free bytes. It behaves correctly on heterogeneous prompt lengths and on memory pressure. It does worst on repeated-prefix traffic, which is exactly agent traffic: with eight replicas and round-robin-ish tie-breaking, an agent's 20 steps scatter, so each step pays 157 ms of prefill instead of 12.5 ms — 20 × 144.5 ms = 2.89 s of avoidable prefill per task, against a total task prefill of 2.64 GPU-s (Lecture 12 §12.6). The policy roughly doubles the agent's prefill bill, and no amount of tuning its two terms recovers it, because the missing quantity is not in the input. This is the concrete argument for the fleet-wide prefix index of Nov 4.
2. When is replication cheaper than chasing? A prefix of S tokens is requested by n concurrent sessions. Compare (a) routing all n to the one warm replica, incurring queueing, against (b) replicating to m replicas at 0.537 GB per 4K copy and balancing. Derive the n at which replication wins for S = 4,096, and state the memory cost. Solution sketch: Under (a) the warm replica serves all n; if each step's prefill saving is 129 ms but the replica's queue grows by roughly the decode work of n sessions, the marginal session's added delay is ≈ n × (its share of a 20.1 ms step). Affinity stops paying when the accumulated queue exceeds 129 ms, i.e. around n ≈ 129/20.1 ≈ 6.4 sessions per replica beyond its balanced share. Under (b), m copies cost m × 0.537 GB = 0.86% of a 62.5 GB budget per copy, and let n sessions spread over m replicas, so the queue term falls by m. So replicate once you expect more than ~6 concurrent sessions on a shared prefix — which for a production system prompt is always. The practical reading: hot prefixes should be replicated to every replica by default, at under 1% of budget each, and affinity routing is for the long tail of session-specific prefixes.
3. Adapter residency as a cache. You serve 500 rank-16 LoRA adapters with a Zipf(1.0) popularity distribution and can hold H of them resident; a miss costs a 33.6 MB load over PCIe at 64 GB/s. Compute the resident memory for H = 50, the hit rate, and the mean added latency per request. Solution sketch: Memory: 50 × 33.6 MB = 1.68 GB, 2.7% of the KV budget — cheap. Zipf(1.0) over 500 items has normalizer H₅₀₀ ≈ 6.79; the top 50 hold H₅₀/H₅₀₀ = 4.50/6.79 = 66.3% of requests. Miss cost = 33.6e6 ÷ 64e9 = 0.525 ms. Mean added latency = 0.337 × 0.525 = 0.18 ms — negligible against a 20.1 ms step. Raising H to 200 costs 6.72 GB (10.8% of budget) for a hit rate of H₂₀₀/H₅₀₀ = 5.88/6.79 = 86.6%, saving only 0.11 ms. The adapter cache is not worth optimizing: the miss is cheap because the object is small, so spend the memory on KV instead. That is the opposite conclusion from every other cache in this course, and the reason is the object size — 33.6 MB against a 2.147 GB KV cache.
4. The batch at which MoE stops helping. For top-k-of-E routing, derive the batch size at which the expected fraction of experts read exceeds 95%, and evaluate for (top-2, E = 8), (top-2, E = 64), and (top-8, E = 256). Solution sketch: Expected fraction read = 1 − (1 − k/E)^B; set ≥ 0.95 → B ≥ ln(0.05) / ln(1 − k/E). (2, 8): ln0.05/ln0.75 = 10.4. (2, 64): ln0.05/ln(0.96875) = 94.4. (8, 256): ln0.05/ln(0.96875) = 94.4 as well, since k/E is identical. So the threshold depends only on the sparsity ratio k/E, not on the absolute counts — a fine-grained MoE with 256 experts and top-8 saturates at the same batch as 64-with-top-2. At Lecture 11's B = 164 all three are saturated, so no production-batch deployment gets a decode bandwidth benefit from sparsity. The only regimes that do are batch-1 latency-critical serving and prefill, which is a precise and somewhat deflating characterization of who MoE serves.
5. Fleet-wide hit rate. Eight replicas, agent traffic where each task issues 20 steps sharing a growing prefix. Compute the fleet-wide prefix-cache hit rate under (a) random routing, (b) perfect session affinity, (c) session affinity with one migration mid-task, and translate each into prefill GPU-seconds per task. Solution sketch: Per-task prefill with no reuse is 2.64 GPU-s (97,000 tokens); with perfect reuse it is 0.21 GPU-s (7,700 distinct). (a) Random: each step lands on the warm replica with probability 1/8, so expected reuse ≈ 12.5% of the recoverable 92%, giving hit rate ≈ 0.115 and prefill ≈ 2.64 − 0.115·2.43 = 2.36 GPU-s. (b) Perfect affinity: hit rate 0.92, 0.21 GPU-s — a 11.2× reduction. (c) One migration mid-task: the prefix moves with the session, so only the step at which the migration happens could miss, and even that need not — hit rate ≈ 0.92 still, ≈0.21 GPU-s, at a cost of one 1.3 ms copy. Session affinity is worth 11×, and migration preserves it while still allowing load balancing — which is the strongest single argument for Llumnix's mechanism, and it is an argument about caching rather than about load.

Reading guide

Required — Llumnix. Read §2–§3 first and, as you go, sort their four motivating problems into "caused by load" and "caused by state" — you should find that all four are the second, which is §17.2's thesis and the reason the paper's answer is a state-moving mechanism. Then §4 on migration: what is copied, when the cutover happens, and why the copy converges (§17.3). Read the scheduling policy for the claim that one mechanism serves load balancing, defragmentation, priority, and draining; the defragmentation case is the one with no alternative solution, so make sure you can state it. Skim the implementation and treat the evaluation's multipliers as measurements on their workloads. Hold this question: migration presupposes a paged KV cache — write down what would have to be copied in a pre-PagedAttention engine, and estimate its cost from Lecture 10 §10.2's contiguous-slot model. That estimate is the paper's real debt to Monday of week five.

Optional — Preble. Required on Nov 4, so read only the workload study now: the measured prefix-sharing fractions are the empirical basis for §17.5's whole argument. Question: their global scheduler keeps a fleet-wide prefix map — what does it cost to keep current, and what happens as it goes stale?

Optional — AlpaServe. Read the argument that model parallelism is a load-balancing tool, and the placement algorithm. Question: §17.6 said the trade is tail latency against per-token latency — find where the paper's model captures the collectives' cost, and decide whether you believe the calibration.

Optional — MuxServe and DynamoLLM. MuxServe multiplexes different models spatially; DynamoLLM re-runs today's decisions with energy as the objective, which connects to Lecture 2 §2.16's joules per token. Question for DynamoLLM: which of §17.1's three currencies does energy behave like?

Optional — Clipper. The pre-LLM ancestor, and the most instructive optional reading today because of what it assumes: stateless replicas, interchangeable models, and latency-bounded adaptive batching. Read §3–§4 and mark each assumption as surviving or broken. Question: Clipper's batching logic is a direct ancestor of Lecture 11's — what did it not need to solve that Orca did?

Optional — GShard and Switch Transformer. Routing in its other sense. Read GShard for the capacity factor and the dropped-token consequence, and Switch for the argument that top-1 suffices. Question: §17.7 showed the decode bandwidth benefit vanishes above B ≈ 30 — do these papers ever claim otherwise, or is the inflated expectation entirely the reader's?

Looking ahead

Monday (KV-cache optimization, Oct 19) is a student-led paper discussion, and it stops routing around the cache and starts attacking it. Today's every number — the 0.537 GB migration, the 129 ms recompute, the 0.86%-of-budget replica copy — is proportional to KV bytes per token, so a lecture that shrinks that quantity moves all of them at once. Mooncake and KIVI are required; the four families (fewer heads, fewer bits, fewer tokens, fewer tokens read) are the taxonomy, and the meeting's real subject is the line between exact and approximate optimizations, which Part II crosses for the first time.

Assignment 4 (serve your own agent) is due Nov 10, 11:59pm, and Assignment 5 (optimize the full stack) goes out that day, due Dec 2. The Oct 28 sharing session is where A4's numbers get presented. If your A4 deployment has more than one replica, today's §17.5 inequality is the most useful thing in these notes for it.

One thing to carry out of the room. Every policy today wanted a number nobody has — the output length, the fleet-wide prefix map, the future load — and the mechanism that won did not try to acquire any of them. It made being wrong cost 1.3 ms. That is a general lesson about systems built on predictions, and it is worth holding when Nov 18 asks whether a language model can do systems work: the question is rarely whether the prediction is good, but whether the architecture can afford for it to be bad.