Monday shrank the cache — fewer heads, fewer bits, fewer tokens — and got 25.6× on bytes per token while leaving the step 59% weights. Today takes the two families that Monday only previewed, and they are the ones that do not shrink anything. Read fewer of the tokens you kept, which changes no term in the cache formula and therefore buys no concurrency at all — only TPOT. And put the bytes somewhere else, which buys capacity elsewhere and makes TPOT worse. Both are worth having and neither is a substitute for Monday, and the reason to spend a meeting on them is that they are where the cache stops being an array and becomes a storage hierarchy — at which point every classical question about tiers, misses, and prefetch becomes live, and the fetch-versus-recompute inequality this course has now derived four times gets its fifth and most general outing. By the end you should be able to say which metric a technique moves before you read its speedup, and price a tier from its bandwidth.

Where this sits

Monday (KV-cache optimization I, Oct 19) set up the taxonomy this meeting completes. Its formula — 2·L·n_kv·d_head·b·S·B — named three quantities a serving system can shrink, and Monday shrank all three: grouped-query attention for 4×, int4 quantization at g = 32 for 3.2×, and eviction at ρ = 0.5 for 2×, composing to 25.6× and taking concurrency at 4K from 29 to 744. It also drew the line this course now lives on either side of: those last two are approximate, so they owe a quality measurement on a frozen set, while everything before Monday was exactly output-preserving.

Today's two families sit outside that formula, which is precisely why they behave differently, and the meeting is the last one before the two that exploit the cache rather than manage it: Nov 4 reuses it across requests, and Nov 9 turns the same quantization machinery on the weights — which after Monday's stack are 59% of a step and therefore the binding term.

Instructor notes — Discussion plan

75-minute class, student-led. Two required papers with a clean division — one is a storage paper, one is a bandwidth paper — and the instructor holds the two arithmetic segments and the closing composition.

Time	Segment	Who	Notes
0–7	Monday's leftovers, and the one diagnostic	instructor	Protected. "How many sequences fit?" is the whole taxonomy in one question.
7–20	§25.2 Capacity against bandwidth	instructor	Protected — derive live. 1.6× vs 2.6×, and the memory-filled invariance.
20–34	Presenter 1: Quest	student	Estimation mechanism. Force: what does it do to concurrency?
34–44	§25.4 InfiniGen and prefetch	room	Where sparse reading meets tiering. Little's law on the prefetch depth.
44–58	Presenter 2: Mooncake	student	Read as storage: tiers, transfer engine, SLO-aware rejection.
58–66	§25.6 The fifth appearance	instructor	The one inequality, five mechanisms. Memorize once.
66–72	§25.7–§25.8 Ring attention, composition	instructor	Sparse reading's headroom after Monday: 2.6× → 1.4×.
72–75	Wrap	instructor	Point at Nov 4.

Questions to force. (1) "Which of today's techniques lets you serve more users?" — neither, and that is the lesson. (2) "Quest's estimate is wrong. What breaks, and is it recoverable?" (3) "Your tiers are HBM, DRAM over PCIe, and remote over Ethernet. Which ones should you ever fetch from?" (4) "After Monday's 25.6×, is today worth doing at all?" Question 4 is the honest one and §25.8 answers it.

Reading-only, not scheduled: §25.7 in full, and the HiSparse/prediction material.

If running long: cut §25.4 to Little's law and drop §25.7 to one sentence. Never cut §25.2 or §25.8 — the capacity/bandwidth distinction is what makes the reading list navigable, and the composition arithmetic is what stops students double-counting Monday's win.

Learning objectives

By the end of this class you should be able to:

1. Decide, from a technique's description alone, whether it improves concurrency, TPOT, or capacity elsewhere — and name the one-question diagnostic that settles it.
2. Compute the TPOT gain from reading a fraction of the cache, and explain why the gain is set by the cache's share of the step rather than by context length.
3. Explain page-level importance estimation and why a wrong estimate here is recoverable where a wrong eviction is not.
4. Size a prefetch pipeline with Little's law, given a tier's latency and a step's duration.
5. Describe a tiered KV store and compute which of its tiers are worth fetching from.
6. State the fetch-versus-recompute inequality and name the five mechanisms in this course that it governs.
7. Explain what ring attention does instead, and which of the other families it makes unnecessary.
8. Compose today's techniques with Monday's and say how much headroom remains.

25.1 What Monday left, and the one question that sorts it

Monday's taxonomy had four attacks and it dispatched three of them. The two remaining families are distinguished from those three by a single structural fact, and it produces the most useful diagnostic in this part of the course.

Ask: how many sequences fit?

If the answer changes, the technique attacked a term in the cache formula and bought capacity — grouped-query attention, quantization, eviction. If the answer is unchanged, it did not, and whatever it bought is something else. Both of today's families leave concurrency exactly as it was.

Reading fewer of the tokens you kept — Quest, InfiniGen, HiSparse — keeps every byte resident and touches a subset per step. Cache size identical, step traffic smaller: it buys TPOT.

Putting the bytes elsewhere — Mooncake, CacheGen — moves them off the GPU into DRAM or across a network. HBM occupancy falls, so the card holds more live sequences, but every access to a displaced block now costs a transfer: it buys capacity elsewhere and makes TPOT worse.

Note the signs, because they are opposite and this is the only place in Part II where two techniques in the same lecture push a metric in opposite directions. That is why "KV-cache optimization" is a bad name for a single thing and why this meeting exists separately from Monday's.

Instructor notes

Minutes: 7. Protected. Instructor holds; no paper yet. Board: The question "how many sequences fit?" then a 2×2: attacks-the-formula vs not, against buys-capacity vs buys-time. Ask the room: "Quest keeps the whole cache. So what does it buy?" Push until someone says TPOT and someone else says "nothing for concurrency". Expect confusion: Both families get filed under "make the cache cheaper". They do not make it cheaper; one reads less of it and one relocates it.

25.2 Capacity and bandwidth are different problems

Derive the bandwidth win properly, because the arithmetic contains a result that is not obvious and that most reporting of this literature gets wrong.

A decode step's traffic is the weights plus every resident sequence's cache. Read only a fraction φ of each cache and the second term scales by φ, while the first is untouched.

The naive reading of those two numbers is "sparse reading is a long-context technique". That is almost right and the precise version is better, because it tells you when to deploy it:

That is worth the time it took, because it converts a vague claim into a deployment test you can apply to your own engine: compute the cache's share of a step. If it is small, no amount of sparsity will help you and Monday's techniques are what you want; if it is large, sparsity is available and its ceiling is 1 + 62.5/13.5 = 5.6× as φ → 0.

Instructor notes

Minutes: 13. Protected — derive live. Board: The step-bytes formula, then the two worked boxes, then the invariance result. Write "76.0 ÷ 29.1" and let the room notice B never appears. Ask the room: "Why is the long-context number bigger?" Then push past "longer context" to "the budget is full". The second answer is the useful one. Expect confusion: Students expect a technique's speedup to be a property of the technique. It is a property of the configuration.

25.3 Quest: estimating which pages matter

The mechanism has to be cheap, because it runs on the critical path of a step that is already bandwidth-bound and has no arithmetic to spare (Lecture 9 §9.8's 1 FLOP/byte). Quest's approach is to summarize each page of the KV cache — a block of contiguous positions, the same blocks Lecture 10 §10.3 introduced — by cheap per-page statistics, typically elementwise minima and maxima of the keys. Given the current query, an upper bound on the attention score achievable anywhere in a page can be computed from those statistics alone, at a cost proportional to the number of pages rather than the number of tokens. Rank pages by that bound, take the top ones, attend over only those.

Two properties make this the right shape of solution.

The estimate is an upper bound, not a guess. A page whose bound is low cannot contain a high-scoring token, so skipping it is defensible rather than hopeful. That is a much stronger position than eviction's heuristics, and it is why the technique degrades gracefully.

The cost is per page, not per token. At a page size of 16 tokens (Lecture 10's default), a 4,096-token sequence has 256 pages, so the estimation touches 256 summaries instead of 4,096 keys — a 16× smaller working set, and the summaries are small enough to stay resident.

And the failure mode is genuinely milder than Monday's, which is the asymmetry the reading question asks about. Eviction is destructive: an evicted token is gone, and if it mattered you get a fluent wrong answer forever. Sparse reading is per step: a page skipped now can be consulted at the next token, so a mistaken estimate costs one step's accuracy rather than the rest of the generation. That argues for being more aggressive with φ than with ρ at equal nominal reduction — a real conclusion that follows from the mechanism rather than from any benchmark, and one worth stating because the two techniques are otherwise easy to confuse.

The honest caveat is the one §25.2 implies: this is still an approximate technique, so Monday's §14.8 standard applies — a frozen task set, weighted toward long-context retrieval, because an estimate error changes which token is attended to.

Instructor notes

Minutes: 14, presenter-led. Board (presenter): A sequence of pages, each with a min/max box; a query arriving; the bound computed per page; the top-k shaded. Force the question: "What does this do to concurrency?" Nothing. Make them say it out loud, then ask why that is not a criticism. Instructor adds: the destructive-vs-per-step asymmetry, and the 256-vs-4,096 working set.

25.4 InfiniGen: where reading less meets storing elsewhere

If a step only needs a quarter of the pages, the other three quarters need not be in HBM at all — they could live in host DRAM and be fetched on demand. That is InfiniGen's observation, and it joins today's two families into one mechanism: use a cheap estimate to decide which pages matter, then prefetch exactly those from the slower tier while the previous layer is still computing.

The design constraint is latency hiding, and Lecture 7's Little's law is the right tool.

That box contains the general lesson: the tier's bandwidth decides whether the prefetch can be exact or must be speculative, and a speculative prefetch that misses costs a stall on the critical path. So the deeper the pipeline, the more the technique starts to resemble §25.3's approximation with a storage penalty attached rather than an accuracy one — which is an unusually clean trade, since a mispredicted prefetch costs time and a mis-estimated skip costs accuracy.

Instructor notes

Minutes: 10, room-led. Board: 0.63 ms per layer, then 0.26 ms (PCIe) and 0.67 ms (Ethernet). Then "1 layer" vs "2 layers, speculative". Ask the room: "What does a missed prefetch cost?" A stall, not an error. Contrast with §25.3.

25.5 Mooncake: the cache becomes a storage system

Mooncake's move is to stop treating the KV cache as a per-GPU array and treat it as a tiered, disaggregated store: HBM, host DRAM, and remote memory across the cluster, with a transfer engine whose job is to move blocks fast enough to be worth the trip. Once you do that, the design questions are the ones every storage system has, and this course has already answered several of them in other costumes.

What is the block? Lecture 10 §10.3's paged block, which is why paging was the enabling change for this too — a contiguous per-sequence tensor cannot be tiered, because there is nothing to move independently.

What is the placement policy? Hot prefixes in HBM, warm in DRAM, cold remote — and "hot" is a prediction, which is why the prediction-based management paper is on the reading list and why Lecture 11 §11.7's asymmetry applies: over-predicting wastes fast memory, under-predicting causes a stall.

What is the eviction policy? Exactly the question Nov 4 will spend a lecture on, with the twist that a prefix cache's objects have ancestry.

And what happens under overload? Mooncake's answer is the one genuinely new policy here: SLO-aware early rejection. When the system cannot meet its latency targets it refuses requests before admitting them, rather than admitting everything and missing every deadline. This is the goodput argument of Lecture 2 §2.16 promoted to an admission rule — and it is the first time in this course that a serving system has been allowed to say no in order to protect the requests it already accepted.

Read the tiering decision as arithmetic rather than architecture, which is §25.6.

Instructor notes

Minutes: 14, presenter-led. Board (presenter): Three tiers with bandwidths, the transfer engine between them, and the block as the unit. Force the question: "Which tier would you fetch a 4K prefix from?" Send them into §25.6. Expect confusion: SLO-aware rejection reads as user-hostile. It is the only policy that keeps a promise to the requests already in flight — Lecture 10 §10.5's "refuse" move, at cluster scale.

25.6 The fifth appearance of one inequality

Every tier raises the same question, and it is the question this course has now asked five times.

The same inequality, in five places, and it is worth listing them once so the pattern is unmistakable: Lecture 10 §10.5 (swap a preempted sequence to host memory or recompute it), Lecture 12 §12.5 (ship the cache from a prefill pool to a decode pool or don't disaggregate), Lecture 17 §17.4 (migrate a running request between replicas or kill it), today (fetch from a tier or recompute), and Nov 4 (fetch a cached prefix from a remote store or recompute it). Five mechanisms, one inequality — and the reason the answer is nearly always "move it" is structural: KV bytes are linear in S while prefill FLOPs are quadratic, so the longer the sequence, the more decisively fetching wins. At 16K GQA-8 the fetch is 2.147 GB against a 731 ms recompute, a break-even of 2.9 GB/s.

CacheGen attacks the numerator instead: compress the cache into a streamable bitstream so a slow tier becomes usable. Note the different error budget — its distortion is paid once per fetch, not once per read, which is a weaker requirement than KIVI's and therefore permits more aggressive compression.

Instructor notes

Minutes: 8. Protected. Board: The inequality, then the four-tier table, then the five mechanisms as five words: swap, disaggregate, migrate, tier, prefix. Ask the room: "Why does fetching win more as sequences get longer?" Linear against quadratic.

25.7 Ring attention: the other answer entirely

Reading-only; not scheduled.

Everything in both meetings assumes the cache must fit, be shrunk, or be moved. Ring attention rejects the premise: distribute the sequence across P devices so each holds S/P positions, and compute attention as a collective in which key/value blocks circulate around a ring while each device accumulates its partial output — using exactly Lecture 9 §9.4's online-softmax rescaling to combine partial results, which is why that mechanism was worth deriving properly.

The consequence is that context length becomes bounded by aggregate memory rather than by one card's, so it scales with the cluster. The cost is a communication pattern on the critical path and a parallelism decision that interacts with everything else — and, for a serving system specifically, the awkward fact that it is a parallelism strategy rather than a cache strategy, so it composes with Monday's techniques and substitutes for tiering rather than the reverse.

It belongs on the reading list mainly as the correct comparison. When a paper claims a cache technique has "solved" long context, the question is whether it beat distributing the sequence — and the honest answer is usually that they address different constraints: ring attention gives you capacity you do not have, and today's techniques give you speed on capacity you do.

25.8 What is left after both meetings

Compose honestly, because Monday's 25.6× and today's 2.6× do not multiply — and understanding why is the last thing this pair of meetings has to teach.

So the two meetings' answer to "what should I do?" is a decision procedure rather than a stack. Compute the cache's share of a step. If it is dominant, both meetings' techniques are available and today's are cheaper to deploy because they need no retraining and no checkpoint change. If it is small — which is what success looks like — the remaining term is the weights, and that is Nov 9.

The other thing that is left is the axis neither meeting touched. Everything in both lectures made a single sequence's cache cheaper to hold, read, or store. Nothing noticed that different requests' caches are often identical: Lecture 5's ledger says 92% of an agent's prompt tokens are re-sent, and identical prefixes have identical caches by a property of causal attention. That is Nov 4, it is orthogonal to everything here, and on the agent ledger it is worth 12.6× on prefill — larger than anything either of these two meetings achieved.

Discussion seeds

1. The diagnostic, applied to your A4. Compute the cache's share of a decode step in your Assignment 4 deployment. Which of the two meetings' techniques is available to you, and which would be a waste of effort?
2. Aggressiveness, justified. §25.3 argues a wrong sparse-read estimate is recoverable where a wrong eviction is not. Turn that into a policy: what φ would you run against what ρ, and what experiment would justify the ratio?
3. The tier you should not have. §25.6 says every tier above ~4 GB/s clears the bar at GQA-8. Name a tier that does not, and say what it is still good for.
4. Rejection as a feature. Mooncake refuses requests to protect its SLO. Write the rule, decide who gets refused, and say what you would tell a user whose request was rejected.
5. Prefetch depth and speculation. §25.4 showed a two-layer prefetch must estimate from an incomplete query. Design the fallback for a mispredicted prefetch, and price it against just reading the whole cache.
6. Does Nov 4 make today unnecessary? Prefix caching is worth 12.6× on the agent ledger. Argue both sides of whether a shop serving agent traffic should invest in today's techniques at all.

Key takeaways

Numbers worth memorizing

Quantity	Value	Source
The diagnostic	"how many sequences fit?" — unchanged ⇒ not a capacity technique	§25.1
Sparse read φ = 0.25, memory full	(13.5+62.5)/(13.5+15.6) = 2.61×, context-independent	§25.2
Its ceiling as φ → 0	1 + 62.5/13.5 = 5.6×	§25.2
Same, at B = 164 with 49 GB unused	1.60× — the configuration, not the technique	§25.2
Quest's working set	256 page summaries vs 4,096 keys at 4K	§25.3
Per-layer time to hide a fetch	20.1 ms ÷ 32 = 0.63 ms	§25.4
Quarter-layer fetch, 16K GQA-8	16.8 MB → 0.26 ms PCIe, 0.67 ms Ethernet	§25.4
Fetch-vs-recompute break-even	16.6 GB/s MHA, 4.2 GB/s GQA-8 (4K)	§25.6
Same at 16,384 tokens	2.9 GB/s — fetching wins more with length	§25.6
Sparse reading after Monday's stack	2.61× → 1.39× at the short-context point	§25.8
The axis neither meeting touched	identical prefixes: 12.6× on the agent ledger	Nov 4

Self-check

Exercises

1. The diagnostic as a formula. Write the sparse-reading gain as a function of the cache's share σ of a step, evaluate at σ = 0.2, 0.5, 0.8, 0.9 for φ = 0.25, and state the σ below which you would not bother. Solution sketch: With cache share σ, gain = 1 ⁄ ((1 − σ) + φσ). At φ = 0.25: σ = 0.2 → 1/(0.8+0.05) = 1.18×; 0.5 → 1/(0.5+0.125) = 1.60×; 0.8 → 1/(0.2+0.2) = 2.50×; 0.9 → 1/(0.1+0.225) = 3.08×. The gain is worth having above roughly σ = 0.5 and is marginal below σ = 0.3, where a 1.2× costs you an approximation and a quality experiment. Note the formula needs only one measurement from your own engine — the cache's share of a step — which makes it the rare piece of this literature you can apply before reading any paper.
2. Price a three-tier store. A 4,096-token GQA-8 prefix, with 60% of accesses hitting HBM, 30% DRAM over PCIe, and 10% remote over 25 GB/s Ethernet. Compute the mean access cost and compare it with recomputing every time, then with keeping everything in HBM. Solution sketch: Fetch costs: HBM 0.537/3350 = 0.16 ms; PCIe 0.537/64 = 8.4 ms; Ethernet 0.537/25 = 21.5 ms. Mean = 0.6(0.16) + 0.3(8.4) + 0.1(21.5) = 0.10 + 2.52 + 2.15 = 4.77 ms, against 129 ms to recompute — a 27× saving. Against all-HBM (0.16 ms) the tiered store is 30× worse per access, which is the honest cost of the capacity it bought. So the tiering decision is not "is fetching cheap?" but "is the capacity worth 4.6 ms of mean added latency?" — and at a 20 ms TPOT budget, 4.6 ms is 23% of it, which is a large bill. The right move is to make the hit rate the design target: raising HBM hits from 60% to 85% takes the mean to 2.03 ms.
3. When prefetch must be speculative. Derive the condition on tier bandwidth W under which a one-layer-ahead prefetch of a fraction φ of a sequence's cache fits inside a layer's compute, for a step of duration T over L layers. Evaluate for T = 20.1 ms, L = 32, GQA-8 at 4,096 and 32,768 tokens, φ = 0.25. Solution sketch: Per-layer budget T/L; bytes per layer φ·KV/L. Condition: φ·KV/(L·W) ≤ T/L → W ≥ φ·KV/T, and the L cancels — a pleasant result: the required bandwidth does not depend on the layer count. At 4,096 tokens (KV = 0.537 GB): W ≥ 0.25(0.537)/0.0201 = 6.7 GB/s — every tier qualifies. At 32,768 tokens (KV = 4.295 GB): W ≥ 0.25(4.295)/0.0201 = 53.4 GB/s — PCIe at 64 GB/s just qualifies, Ethernet at 25 does not. So long context is exactly where exact prefetch stops being possible, which is unfortunate since it is also where the technique is most wanted, and it is the structural reason InfiniGen has to speculate.
4. Does tiering ever beat shrinking? You need to serve 4,096-token contexts and are choosing between (a) Monday's int4 KV at g = 32, and (b) tiering half the cache to DRAM over PCIe. Compare concurrency, TPOT, and the evidence each owes. Solution sketch: GQA-8 baseline: 0.537 GB/seq, B = 116, step at the ceiling = 76.0 GB → 22.7 ms. (a) int4 g=32 (3.2×): 0.168 GB/seq, B = 372, step = 13.5 + 62.5 = 76.0 GB → 22.7 ms — same TPOT, 3.2× the users, and it owes a quality measurement. (b) tier half to DRAM: HBM holds 0.268 GB/seq so B = 233, but half of every access crosses PCIe: the step's HBM traffic is 13.5 + 31.2 = 44.7 GB (13.3 ms) plus 31.2 GB over PCIe at 64 GB/s (487 ms) — catastrophic unless overlapped, and even perfectly overlapped it is bounded by the PCIe term. So (a) wins decisively, and the general rule is that tiering is for capacity you cannot otherwise have (very long contexts, cross-request reuse), not for routine concurrency. Tiering's evidence is a performance measurement; quantization's is a quality one — so they are not even comparable on effort.
5. Compose everything Part II has. Starting from MHA bf16 at 4K (B = 29, step 75.8 GB), apply GQA-8, int4 KV, ρ = 0.5 eviction, φ = 0.25 sparse reading, and 4-bit weights, and report concurrency and TPOT at each step. Then name the binding term at the end. Solution sketch: (0) 512 KiB/token, B = 29, step 75.8 GB → 22.6 ms. (1) GQA-8 → 128 KiB, B = 116, step 76.0 GB → 22.7 ms. (2) int4 g=32 → 40 KiB, B = 372, step 76.0 GB → 22.7 ms. (3) ρ = 0.5 → 20 KiB, B = 744, step 76.0 GB → 22.7 ms. Note TPOT is pinned because each step refills the same 62.5 GB budget — capacity techniques buy users, not speed, once you fill memory. (4) φ = 0.25 sparse read: step = 13.5 + 15.6 = 29.1 GB → 8.69 ms, B still 744. (5) 4-bit weights (3.47 GB): step = 3.47 + 15.6 = 19.1 GB → 5.70 ms. Final: 744 concurrent sequences at 5.70 ms against 29 at 22.6 ms — 25.6× the users at 4.0× the speed. Binding term: the cache read, now 82% of a step, so the next optimization is the one neither meeting touched — not reading it at all, because someone else already computed it. That is Nov 4.

Reading guide

Required — Mooncake. Read it as a storage paper. Architecture figure first, then the KV-cache store: what a block is, which tiers exist, what the transfer engine guarantees. Spend the most time on the policy — placement and eviction across tiers — and on the overload section, where SLO-aware early rejection is a policy this course has not met before and which you should be able to defend to a user. Skim the cluster-operations detail. Hold this question: §25.6 prices every tier against a 129 ms recompute. Walk their hierarchy, decide which tiers clear the 4.2 GB/s bar at GQA-8, and say what the system should do with a tier that does not — the answer is not "delete it".

Required — Quest. Read the estimation mechanism carefully and note that it produces a bound rather than a guess; that distinction is what separates it from Monday's eviction heuristics. Check the cost accounting: the estimate runs on the critical path of a step with no spare arithmetic (Lecture 9 §9.8), so per-page rather than per-token is not an optimization but a requirement. Read the accuracy results with Monday's §14.8 standard in hand — long-context retrieval is the sensitive task. Hold this question, which is the meeting's central one: what does this technique do to the number of users a card can serve, and why is the answer "nothing" not a criticism?

Optional — InfiniGen. Where today's two families meet: estimate, then prefetch from host memory. Read the prefetch pipeline and check it against §25.4's Little's-law budget. Question: Exercise 3 shows exact prefetch becomes impossible exactly at the long contexts the technique targets — how do they handle a miss, and what does it cost?

Optional — Ring attention. The premise-rejecting alternative. Read the ring communication pattern and note that the combination step is Lecture 9 §9.4's online softmax. Question: which of the five families across these two meetings does it make unnecessary, and which does it make more important?

Optional — HiSparse and prediction-based KV-cache management. Sparsity and placement driven by prediction. Read the second for how it behaves when the prediction is wrong — Lecture 11 §11.7's asymmetry says under-prediction stalls and over-prediction wastes, and a paper that does not distinguish them has not finished the design. Question: what would you require of a predictor before letting it decide which tier a block lives in?

Looking ahead

Monday, Oct 26 is a guest speaker, and Wednesday, Oct 28 is the Assignment 3 sharing session — bring the lever you expected to win, the one that did, and the gap between them. Assignment 3 is due Sun Oct 25, 11:59pm; Assignment 4 (serve your own agent) goes out Oct 26, when the final project is also announced, with its proposal due Oct 28.

Then Wednesday, Nov 4 (prefix cache) attacks the bill along the axis these two meetings left completely alone. Everything in both lectures made a single sequence's cache cheaper to hold, to read, or to store. Nov 4 observes that different requests' caches are frequently identical — Lecture 5's ledger puts 92% of an agent's prompt tokens at re-sent — and that identical prefixes have identical caches, exactly, by a property of causal attention. On the agent ledger that is 12.6× on prefill, which is larger than anything either of these meetings achieved, and it requires no approximation at all. Nov 9 (pruning and quantization) then takes the term Exercise 5 leaves binding.

One thing to carry out of the room. Today's two families push TPOT in opposite directions, and no speedup either of them reports is a property of the technique — every one of them is a property of your configuration, recoverable from a single measurement of the cache's share of a step. Take that measurement before you read another paper in this area.