Lecture 18 scheduled a sixteen-way fan-out and got a 3.4× speedup by batching the interior and expediting the sink. It could do that because every map was assumed identical. Today we put the variance back, and with it the barrier: a fan-join does not finish when the average worker finishes, it finishes when the last one does, and that single fact reorganizes every lever in the block. Along the way multi-agent turns out to be simultaneously the worst workload this course has seen — k preambles, k sessions parked at once, k sandboxes — and the best, because the k prompts share almost all of their text by construction. Then the lecture turns the subject on itself and asks the question your Assignment 5 write-up has to answer empirically: is it worth spawning a fleet of agents to optimize a system, and what has to be true for the answer to be yes? The answer is not about the agents. It is about whether you can tell which one won. By the end you should be able to price context isolation against a single long context, convert a fan-out's structural prompt sharing into a prefix-reuse ceiling, compute a fan-join's completion time under a bounded and a heavy-tailed output distribution, pick the mitigation that matches the tail you actually measured, and state the precondition that makes best-of-k optimization worth its tokens.

Where this sits

Lecture 18 declared the DAG on Wednesday, November 18; Lecture 19 handled the stalls inside one session and Lecture 20 placed one session's state, both earlier in today's meeting. All three treated the agent as a single line of execution that occasionally pauses. Today k of them run at once under one parent, which changes three things at the same time: the sessions arrive in a burst rather than a stream, their prompts are near-identical rather than merely similar, and — the part with no precedent anywhere earlier in the course — they must all finish before anything downstream can start. Lecture 22, which shared the Nov 18 meeting with Lecture 18, hands the whole structure to a scheduler that is not told how long any of it will take; read it against today's §21.4, because a barrier is precisely the thing an unstructured scheduler cannot see coming.

Forward, this note is the hinge into the LLMs-for-systems material, which is optional content on this calendar: Lecture 23's subject — "LLM for Systems Research I" — is §21.6's verifier argument turned into the research question.

Instructor notes — Timing plan

75-minute class (Mon/Wed 11:15am–12:30pm, SEC 2.118). Student-led paper discussion: the presenting group walks TokenCake and the instructor holds the framing, the §21.4 derivation, and the §21.6 verifier block. This note shares the Nov 23 meeting with Lectures 19 and 20; the budget below is the standalone-lecture plan and has not been re-cut for the shared slot, so treat the protected rows as the floor and everything else as negotiable.

Time	Segment	Notes
0–5	Framing; A5 checkpoint	One line of admin — A5 due Dec 2, project report Dec 8. Then: "Lecture 18's fan-out had no barrier because every map was identical. Today they are not."
5–13	§21.1 Four shapes	Fast. The table is orientation for the rest.
13–24	§21.2 Context isolation, priced	Derive the 4× live. This is Lecture 4's claim finally getting a number.
24–32	§21.3 The prefix-cache best case	The two r values and their ceilings. Land "worst and best workload simultaneously."
32–50	§21.4 The barrier	Centerpiece — never cut. Both distributions on the board; 1.52× versus 2.72× is the whole point.
50–58	§21.5 Cutting the tail	The expedite-versus-cap split by distribution. Protect the punchline.
58–70	§21.6 Multi-agent for optimization	Protect. The verifier arithmetic — q = 0.7 with 16 agents equals q = 1 with 4 — must reach the board.
70–75	§21.7 in brief + close	The 82% repeat rate, then point at the LLMs-for-systems track (Lecture 23).

Reading-only, not scheduled: §21.8 in full, and the hedging arithmetic in §21.5. The discussion seeds after §21.8 are the student-led half's raw material — assign two of them to the presenting group in advance.

If running long: compress §21.2 to the 4× result without the derivation and drop §21.3's second case. Never cut §21.4 or the verifier block in §21.6 — those are the two things this lecture exists to say, and the second one is the one students will use in December.

Learning objectives

By the end of this class you should be able to:

1. Name the four common multi-agent shapes and say which serving lever each one exercises.
2. Price context isolation against a monolithic context, and explain why the win is a token-ledger effect rather than a parallelism effect.
3. Compute the prompt-repeat rate of a fan-out and convert it to a prefix-reuse ceiling.
4. Compute the expected completion time of a fan-join for bounded and heavy-tailed worker distributions, and say how each scales with k.
5. Choose between expediting, capping, hedging, and partial joins based on which distribution you are in.
6. Decide whether to spawn k optimizer agents, and state the precondition that makes best-of-k worth anything at all.
7. Explain why an optimizer fleet is the friendliest workload in this block on every axis, and why that makes it a misleading benchmark to evaluate a serving system on by itself.
8. Name the three shapes in which multi-agent is the wrong answer, and identify which of the fan-out's advantages each one destroys.

21.1 Four shapes, and which lever each one pulls

"Multi-agent" names a family of dataflow shapes, not one thing. Four cover almost everything in practice, and each stresses a different part of the machinery built in the last three lectures.

Shape	Structure	Dominant serving lever
Orchestrator–worker	One parent fans out k independent children, joins their results	The barrier (§21.4); prefix sharing across children (§21.3)
Pipeline	Fixed stages, each agent's output feeding the next	Lecture 18's chain: dependence edges, expedited decode
Debate or ensemble	k agents answer the same question, a judge selects	Best-of-k and the verifier (§21.6)
Recursive decomposition	Workers spawn workers to unknown depth	Admission and memory: k is not known in advance

Only the first and third are genuinely new today; the pipeline is Lecture 18's chain with a longer prompt, and recursive decomposition is the first shape with an unbounded k, which is a scheduling problem and therefore Lecture 22's (Nov 18).

Note what all four share, and it is the property that makes this lecture necessary: every one of them multiplies the number of concurrent sessions per user task. A single agent task occupied one session in Lecture 19's accounting and parked 2.147 GB while it waited. An eight-way fan-out occupies eight, parks 17.2 GB, and — because the workers were launched together and stall together — parks it in a correlated burst rather than a smooth average. Lecture 19's fleet arithmetic assumed independence between sessions. Multi-agent breaks that assumption, and TokenCake's critical inversion is one consequence: when memory pressure arrives, it arrives for all k children at once.

Instructor notes

Minutes: 8. Keep it moving — the table is orientation, not content. Board: The four shapes as four sketches, then under them one line: "sessions per task × k". Then 1 session → 2.147 GB against 8 sessions → 17.2 GB. Ask the room: "Which of the four does your Assignment 2 agent actually implement?" Almost always orchestrator–worker or nothing at all; make somebody say which and why they chose it. Expect confusion: Students read "multi-agent" as a model-quality technique and wonder why it is on a serving syllabus at all. Answer with the multiplier: whatever it does for quality, it multiplies sessions, preambles, and sandboxes by k, and each of those is a line in the serving budget.

21.2 Context isolation, priced

Lecture 4 §4.4 claimed that the real win of sub-agents is context isolation — "search 40 files, return three lines" — and that parallelism is a side effect. It never priced the claim. Price it now, using Lecture 4 §4.2's own multiplier: a token admitted into a context at step j of an n-step task rides in every subsequent prompt, so its true cost is its length times the number of steps remaining. That multiplier is the same one that turned one uncapped 20,000-token tool output at step 4 of 20 into 320,000 prompt tokens — three times the whole task's baseline.

Three things to read off that block.

The win is not parallelism. Nothing in the arithmetic assumed the eight workers ran simultaneously; run them one after another and the token count is identical. The saving comes entirely from where the tokens live — in a context that is discarded after three steps rather than one that is re-sent sixteen more times.

The cost of isolation is preambles. Eight workers pay eight 2,000-token preambles where the monolithic version paid one. That is 16,000 tokens of pure duplication, and it is the price of the 4× — which is a good deal here and a bad one when the workers are short. The break-even is roughly where k × preamble exceeds the ride-along savings; for one-step workers it usually does.

The parent's context is the scarce thing. Lecture 3 §3.5 measured a mid-session context as 55% tool outputs and 2% user request. Isolation is the mechanism that stops the 55% from growing without bound, and it does so by moving the tool outputs into contexts that die young. That is a memory management strategy, and the fact that it is implemented by prompting rather than by an allocator does not make it less of one.

Instructor notes

Minutes: 11. Board: The two totals, then "×16" circled in the monolithic column. Write "the multiplier is the whole argument." Ask the room: "Would the saving survive if the workers ran sequentially?" Yes, entirely — which is the surprise. Then: "what would kill it?" Short workers, or a parent that needs everything the workers saw.

21.3 The best case for the prefix cache

Multi-agent triples the number of sessions and multiplies the token count. It also produces the highest prompt-repeat rates in this course, and for a structural reason: the k children of one parent are launched by the same code, from the same template, with the same task context. Their prompts do not merely resemble each other — they agree on a prefix, exactly, by construction.

Lecture 18's 1/(1−r) ceiling applies directly.

The gap between 1.17× and 5.05× is entirely a design decision about how much context the parent hands down, and it is a decision usually made for accuracy reasons with no idea that it moves a serving number by 5.05 ÷ 1.17 = 4.3×. That is worth saying to whoever writes your orchestrator — and note which way the incentive points: handing workers more shared context makes almost all of it free, so for once the accuracy-motivated choice and the serving-motivated choice agree.

Two mechanisms exploit it, and they differ in a way worth noticing. RadixAttention (Lecture 13) discovers the shared prefix at runtime by matching tokens — correct, but it must match k times and it may evict between children. ForkKV's copy-on-write approach starts from the structural fact instead: the children are forks of one parent state, so the shared blocks are shared by reference from the moment they are created, and no matching happens at all. That is Lecture 18's guess-versus-fact distinction reappearing at the memory-allocator level, and the fan-out is the case where the fact is most obviously available: the orchestrator knows it is forking.

The pathological version, which TokenCake is about: eight children forked from one parent share blocks until they diverge, and then each one's KV grows independently inside a fixed block pool. If a non-critical child — one whose result the join will barely use — allocates aggressively, it can evict the blocks of a child on the critical path. The scheduler thought it had ordered the work correctly; the allocator undid it. Priority in the scheduler without priority in the allocator is not priority.

Instructor notes

Minutes: 8. Board: The two r values and their ceilings, side by side. Then "the difference is a prompt design decision." Ask the room: "Which of the two does your Assignment 2 agent produce?" Most will not know, which is the point — it is measurable in one line of instrumentation.

21.4 The barrier: why the join waits for the tail

The centerpiece. Lecture 18 batched sixteen maps and computed their decode as though they finished together: 250 tokens each, 7.85 ms per step, 1.96 s for all sixteen. That is right only if every worker generates exactly 250 tokens. Real workers do not, and the join cannot start until the last one lands.

The whole lecture is in the distance between those two words, and it depends entirely on the tail of the worker-length distribution.

Now scale k, because the two distributions diverge in a way that decides how wide a fan-out should be.

k	Uniform: E[max] ÷ mean	Exponential: E[max] ÷ mean
4	1.40×	2.08×
8	1.52×	2.72×
16	1.59×	3.38×
64	1.65×	4.74×
limit	1.67× — saturates	grows as ln k — unbounded

For a bounded distribution the barrier tax saturates. Going from 8 workers to 64 costs 8× the tokens and only 9% more barrier — so widening a fan-out is nearly free in latency and linear in cost, and the decision is purely a token-budget one. For a heavy-tailed distribution it does not saturate, and by k = 64 the join waits nearly five times the mean worker. The tax is a property of the tail, not of k, and the two regimes call for opposite designs.

This also revises Lecture 18's headline honestly. Its sixteen maps each emitted exactly 250 tokens: serialized in the 4.3 ms lane, 2.62 s of prefill plus 16 × 250 × 4.3 ms = 19.8 s; batched at 7.85 ms, 2.62 + 250 × 7.85 ms = 4.58 s — the §18.4 result, 4.3×. Now put variance in while holding the mean at 250 so the comparison stays fair, using the standing bounded distribution shifted to that mean — Uniform(150, 350), same 200-token width — and Exponential(mean 250).

Lecture 18's number was not wrong; it was computed in a world without variance, and it said so.

Instructor notes

Minutes: 18. Never cut. Board: max versus mean, boxed. Then the two E[max] derivations, then the scaling table with the saturating column and the growing column drawn as two curves. Ask the room: Before the table: "Does doubling the fan-out double the barrier?" Most say yes. It adds 7% for bounded and 24% for exponential. The gap between those two answers is the lesson. Ask the room: "Which regime is a code-search sub-agent in?" Heavy-tailed — most files are boring and one has the answer, which is precisely the shape that produces long tails. Expect confusion: Students conflate the barrier tax with poor load balancing. It is not an imbalance in assigned work; it is variance in produced output, which no assignment policy can equalize because the length is decided by the model at generation time.

21.5 Cutting the tail: expedite, cap, hedge, join early

Four mitigations, and the useful result is that which one works depends on which distribution you are in — so the first job is to measure the tail, not to pick a technique.

Expedite the predicted straggler. Lecture 18's payoff 2 said batch the interior and expedite the sink. With a barrier the sink is not the reduce — it is whichever worker finishes last. Give that one worker the 4.3 ms expedited lane and the join is set by the second-longest.

Note the inversion: expediting helps less in the heavy-tailed case, which is the one that needs help most. The reason is that exponential order statistics bunch at the top — the second-longest is nearly as long as the longest — so removing one straggler reveals another. Expediting is a bounded-tail technique.

Cap the worker output. The client-side lever from Lecture 4, and the one with no server-side twin. Capping at 300 tokens truncates the exponential's tail at the source: the join falls from 9.21 s to at most 300 × 22.6 ms = 6.78 s, a 26% cut, and it costs only the information in the truncated tail. For heavy tails, capping beats expediting by 5.8× — it saves 9.21 − 6.78 = 2.43 s where expediting saved 0.42 s — and it is free.

Hedge. Launch a duplicate of a worker that has run long, take whichever finishes. Classic tail mitigation, and it costs a full extra worker's tokens for a partial latency win — worth it when the tail is caused by an unlucky draw and worthless when it is caused by the task genuinely being harder, which is the common case in agent workloads. Prefer capping.

Join early. Proceed once m of k workers have returned. This is the only mitigation that attacks the barrier itself rather than the tail, and it is an application decision, not a serving one: whether 6 of 8 summaries suffice is a question about the task. When it is available it dominates everything else — at m = 6, k = 8, exponential, the join waits for the 6th order statistic, 150 × (H_8 − H_2) = 150 × (2.718 − 1.5) = 182.7 tokens → 4.13 s, better than every other option here. Be honest about what it does not do: 4.13 s is still 22% above the 3.39 s a mean-worker estimate would have promised (4.13 ÷ 3.39 = 1.22), because six of the eight still have to land. A partial join shrinks the barrier; nothing here deletes it.

Instructor notes

Minutes: 8. Board: Four columns — expedite, cap, hedge, join early — and under each the exponential saving: 0.42 s, 2.43 s, "a worker's tokens", 5.08 s. Circle the last two. Ask the room: "Which of these can your serving system do without the application's help?" Only expediting, and it is the weakest one. That asymmetry is the section. Common wrong answer: "Hedge — that is what tail-tolerant systems do." Press on why the tail is long. A slow replica is slow for machine reasons, so its duplicate is likely fast; a slow worker is slow because its subtask is harder, and the duplicate draws the same hard subtask. Hedging transplanted from replicas to agents loses its premise.

21.6 Multi-agent for performance optimization

Now turn the subject on itself. This course asks you to optimize systems — Assignment 5, out since Nov 11 and due Dec 2 — and the obvious modern move is to spawn k agents, let each attempt the optimization, and keep the best. When is that worth the tokens?

Model it. Let each independent attempt produce a speedup S drawn from Uniform(1.0, 2.0) — an attempt is somewhere between useless and twice as fast. Take the best of k.

k	E[best] = 1 + k/(k+1)	Gain from one more agent, 1/((k+1)(k+2))	Surplus per agent, (E[best] − 1) ÷ k
1	1.500×	0.167	0.500
2	1.667×	0.083	0.333
4	1.800×	0.033	0.200
8	1.889×	0.011	0.111
16	1.941×	0.003	0.059

Cost is linear in k; value is concave and saturating. The first agent is 8.5× more cost-effective than the sixteenth, and past k = 8 you are buying thousandths of a multiplier for whole agent runs. That alone is a useful stopping rule — spawn until the marginal expected gain stops paying for the run — but it is not the important part.

The important part is a precondition the arithmetic quietly assumed: taking the max requires knowing which one is the max.

Read that as a statement about your benchmark harness, because that is what it is. If your A5 measurements have enough run-to-run noise that you cannot reliably rank two candidate optimizations, then q is low, and every additional optimizer agent is spending tokens to sample a distribution you cannot read. Variance reduction in the benchmark strictly dominates parallelism in the search — more repetitions, controlled machine state, warm-up excluded, percentiles rather than means. All of which Lecture 5 asked for on other grounds.

This is also why performance optimization is the best possible domain for multi-agent methods. The repeated-sampling literature finds coverage rising steeply with k — many samples contain a correct answer — while the gap to realized accuracy is set by whether you can pick it out. In most domains you cannot: there is no automatic checker for "is this summary good." In systems performance, the verifier is free and exact — you run it and time it. KernelBench is precisely this: generate kernels, compile, check correctness, measure speedup, keep the winner. The reason agents optimize kernels well is not that kernels are easy; it is that kernels are checkable.

Which sets up the boundary condition — the one the LLMs-for-systems track opens with Lecture 23 and closes the note chain on with Lecture 24. A verifier you can game is worse than no verifier: if the benchmark can be satisfied without doing the work — caching the answer, specializing to the input, breaking correctness in a way the test misses — then best-of-k searches for exactly that, because it is searching for whatever scores highest. The larger k is, the harder it searches. Scaling the fleet scales the exploitation of your metric at the same rate it scales the optimization, and only the correctness half of the verifier keeps the two apart.

Instructor notes

Minutes: 12. Protect. Board: The E[max] table, then the q block underneath. Box the two numbers 1.800 and 1.809 and write "4 agents with a good ruler = 16 with a bad one." Ask the room: "How noisy is your A5 benchmark?" Most will not have measured it. That is the assignment feedback, delivered two weeks before the Dec 2 due date, and it is the single most actionable thing in the lecture. Expect confusion: Students hear "the verifier matters" as advice about evaluation methodology. It is a serving argument: q determines whether the tokens you spend on k buy anything, so the verifier is a cost parameter, not a quality parameter.

21.7 Serving an optimizer fleet

If you do spawn k optimizers, what does the serving system see? Unusually good news, on every axis this block has examined.

Prefix sharing is extreme. Eight optimizer agents share the repository context, the profiling output, and the system prompt — say 30,000 tokens — and differ by a 2,000-token instruction. Submitted 256,000 tokens, distinct 46,000, r = 82.0%, ceiling 5.6×.

There is no barrier until the very end. The agents are independent; nothing waits on anything until the final selection. So §21.4's tax does not apply, and workers can be batched at the ambient TPOT without any latency consequence — this is the pure-throughput case Lecture 18's payoff 2 wanted.

Stalls are compile-and-benchmark shaped. Each attempt's tool calls are builds and timed runs: seconds to minutes, well past Lecture 19's 21.3 s keep-versus-discard break-even for the long ones. So optimizer sessions should be swapped out or discarded during benchmarking, and the fleet's KV footprint at any instant is far below k × session size.

Put together: an optimizer fleet is the friendliest agent workload in this course — high sharing, no dependence, long stalls that free memory, and one join at the end. If a serving system cannot do well on this, it will not do well on anything. It is worth using as your first benchmark for exactly that reason — and worth distrusting as your only one, because a system tuned on it has never met a barrier, a dependence chain, or a correlated short stall.

Instructor notes

Minutes: 5, and this is the close. Board: Four ticks — sharing 82%, no barrier, stalls past 21.3 s, one join — then a single cross beside "and therefore it tests nothing hard." Ask the room: "You are benchmarking your Assignment 5 stack. What does an optimizer-fleet trace fail to exercise?" Barriers, dependence, short correlated stalls, admission under burst — each of them a section from Nov 18 or from earlier today, so the answer doubles as the review. Expect confusion: The three bullets sound like three separate pieces of luck. They are one fact seen three ways: the fleet has no dependence edges, and every hard problem in this block came from a dependence edge.

21.8 When multi-agent is the wrong answer

(Reading section — not scheduled in class.)

Three failure modes, each of which converts the fan-out's advantages into its costs.

Workers that must talk to each other. The moment workers exchange intermediate results, the shape stops being a fan-out and becomes a sequence of barriers. Two rounds of exchange among eight workers costs two straggler taxes — 2 × 5.15 s under the bounded distribution, 2 × 9.21 s under the exponential — while still paying eight preambles. You have bought a chain's latency at a fan-out's price. If the workers need each other's outputs, one agent with one context is usually cheaper and always simpler.

Workers whose output the parent must read in full. §21.2's 4× came from returning 200 tokens instead of 10,000. A worker that returns its entire context to the parent has isolated nothing; it has added a preamble and a round trip to a monolithic design. The summarization at the boundary is the mechanism, not an afterthought, and its lossiness is the price of the saving.

Recursive decomposition without a depth bound. Lecture 4 made stopping a safety property for one loop. For a tree it is a capacity property as well: unbounded spawning is a fork bomb with a token budget attached, and the admission controller of §19.8 has no way to reason about a session that may become thirty sessions. Bound the depth and the branching factor in the harness, not in the prompt.

The honest summary is that multi-agent is a context management technique that happens to be parallelizable. Used for what it is good at — keeping large intermediate results out of a long-lived context, and searching a space you can score — it is excellent. Used as a general-purpose way to make things faster, it multiplies every cost in this block by k and adds a barrier.

Discussion seeds

1. Whose priority is it? TokenCake's critical inversion says a schedule is undone by an allocator. Two fixes: teach the scheduler about memory, or teach the allocator about priority. Pick one, name the interface it needs that today's engines do not expose, and say which of Lecture 13's radix-tree rules it breaks — eviction there is LRU on leaves only, and a critical child's blocks may sit below a non-critical child's.
2. Where does the cap belong? §21.5 found capping worker output to be the strongest heavy-tail mitigation and noted it has no server-side twin. Design one. The server can see a worker's token count in flight and can see the barrier in a declared DAG (Lecture 18), so it could stop a straggler itself — under what contract, and what does the application need to have promised for that to be safe rather than a silent truncation of somebody's answer?
3. Fan-out width as a serving parameter. The barrier tax saturates at 1.67× for bounded tails and grows as ln k for heavy ones; cost is linear in k either way. Who chooses k — the application, which knows how much breadth helps accuracy, or the serving system, which knows the measured tail and the current memory pressure? If the answer is "both," write the interface.
4. The verifier as a serving argument. §21.6 claims q is a cost parameter, not a quality parameter. Push on it: is there a workload where a serving system should refuse a best-of-k request because the client has not demonstrated a verifier? What would it ask for, and is that admission control or paternalism?
5. Gaming, at scale. Scaling k scales the search for whatever scores highest, including ways to score highest without doing the work. If you ran a leaderboard's scoring harness against sixteen agent-written submissions per team, what would you add to it — and does your answer generalize to a benchmark you did not write?
6. Isolation without sub-agents. §21.2's 4.0× came from tokens living in a context that dies young, and the derivation never used a second model. Could a single agent get the same win from a context policy alone — a scratchpad the harness discards — and if so, what does a sub-agent still buy that a discard policy does not? Answer in the ledger, not in adjectives.

Key takeaways

Numbers worth memorizing

Quantity	Value	Source
Context isolation, forty-file example	1,280,000 versus 320,800 prompt tokens = 4.0×	§21.2, Lecture 4's ride-along multiplier
Fan-out repeat rate, thin versus thick sharing	14.6% (1.17×) versus 80.2% (5.05×)	§21.3
Barrier tax, k = 8	1.52× bounded, 2.72× exponential	E[max] ÷ mean
Barrier scaling	saturates at 1.67×; grows as ln k unbounded	Order statistics
Expediting one straggler, k = 8	saves 10% bounded, only 4.6% exponential	E[2nd max]
Capping output at 300 tokens, exponential	9.21 s → 6.78 s = 26%	Tail truncation
Partial join, 6 of 8, exponential	4.13 s — best of every mitigation, still 22% over the mean	6th order statistic
Lecture 18's fan-out, re-audited at mean 250	4.3× → 3.75× bounded → 2.14× exponential	§21.4, E[max of 16]
Isolation break-even, forty-file task	pays below ≈13 steps per worker	Exercise 1
Best-of-k speedup, Uniform(1, 2)	1.50 / 1.80 / 1.94× at k = 1 / 4 / 16	1 + k/(k+1)
Verifier equivalence	q = 1 at k = 4 (1.800) ≈ q = 0.7 at k = 16 (1.809)	§21.6
Optimizer fleet repeat rate	82.0%, ceiling 5.6×, no barrier	§21.7

Self-check

Exercises

1. The isolation break-even. Sub-agents cost k preambles and save the ride-along multiplier. For the §21.2 task, find the worker length at which isolation stops paying, holding everything else fixed. Then say what that implies about one-step "tool-wrapper" agents. Solution sketch: Isolation costs k × 2,000 = 16,000 tokens of duplicated preamble and saves 80,000 × 16 − (worker context costs) . Setting the saving to zero with w steps per worker and 10,000 tokens of documents each: worker cost ≈ 8 × w × 12,000, which reaches the monolithic 1,280,000 at w ≈ 13 steps. So isolation pays for anything under about thirteen steps per worker and stops paying beyond it. A one-step wrapper agent is at the opposite extreme — it pays a full preamble to save one ride-along — and is almost always worse than an ordinary tool call.
2. Fit the tail. You instrument your fan-out and find worker output lengths with mean 150 and 95th percentile 520 tokens. (a) Is that closer to Uniform(50, 250) or Exponential(150)? (b) Estimate the barrier tax at k = 8. (c) Which mitigation? Solution sketch: (a) Uniform(50, 250) has a hard maximum of 250, so a 95th percentile of 520 is impossible — it is heavy-tailed. Exponential(150) has a 95th percentile of 150 × ln 20 = 449 tokens, close but still light; the real distribution is heavier than exponential. (b) At least 2.72×, so 9.2 s or worse. (c) Capping, and the cap should be set from the distribution: at 300 tokens you truncate about 13% of workers and cut the join by a quarter. Then ask whether a partial join is acceptable, since it dominates everything.
3. Barrier-aware fan-out sizing. You have a 400,000-token budget for one fan-out step. Workers cost 24,000 prompt tokens each (thick sharing) and produce exponential(150) outputs. (a) How many workers fit before prefix reuse? (b) After, at the §21.3 ceiling? (c) What is the join latency at each, and what do you conclude? Solution sketch: (a) 400,000 ÷ 24,000 = 16 workers. (b) With r = 80.2% the computed cost per worker falls toward the distinct portion, roughly 4,750 tokens amortized, so about 84 workers fit the same compute budget. (c) Join at k = 16 is 150 × 3.38 × 22.6 ms = 11.5 s; at k = 84 it is 150 × H_84 × 22.6 ms ≈ 150 × 5.02 × 22.6 ms = 17.0 s. Prefix reuse bought 5× the workers for a 48% latency increase — a good trade if breadth helps, and the point is that the two effects must be evaluated together, since the cache is what makes the wider fan-out affordable and the tail is what makes it slower.
4. Verifier economics. An optimizer agent run costs $2 in tokens. Your benchmark can be made more reliable by running each candidate m times at $0.10 per run, with q rising as q(m) = 1 − 0.3/m. With a fixed $40 budget, find the split between agents and repetitions that maximizes E[selected] under Uniform(1, 2). Solution sketch: With k agents and m repetitions each, cost = 2k + 0.1km = k(2 + 0.1m) ≤ 40. At m = 1: k = 20, q = 0.7 → 0.7 × 1.952 + 0.3 × 1.5 = 1.816. At m = 4: k = 16.7 → 16, q = 0.925 → 0.925 × 1.941 + 0.075 × 1.5 = 1.908. At m = 10: k = 13, q = 0.97 → 0.97 × 1.929 + 0.03 × 1.5 = 1.916. At m = 20: k = 10, q = 0.985, E[max of 10] = 1.909 → 0.985 × 1.909 + 0.015 × 1.5 = 1.903. The optimum is around m = 10, and note how flat the objective is above m = 4 and how sharply it falls at m = 1 — the first few repetitions are worth far more than the last several agents.
5. Auditing Lecture 18 with variance. Redo Lecture 18's §18.4 fan-out comparison — sixteen maps, 6,000 prompt tokens each, batched decode at 7.85 ms — with exponential(250) output lengths instead of a fixed 250. (a) New batched-map time. (b) New end-to-end and speedup versus the 21.6 s per-request configuration. (c) What survives? Solution sketch: (a) E[max of 16] = 250 × H_16 = 250 × 3.381 = 845 tokens; decode 845 × 7.85 ms = 6.63 s, plus 2.62 s of prefill = 9.25 s for the maps, against 4.58 s in the fixed-length version. (b) Adding the 1.8 s reduce: 11.1 s end-to-end versus 6.4 s, so the speedup over per-request scheduling falls from 3.4× to roughly 1.9× — still a win, and roughly half of the advertised one. (c) The direction survives and the magnitude does not, which is the general lesson of this lecture: any fan-out result computed without a length distribution is an upper bound, and the gap between the bound and reality is entirely the tail.

Reading guide

Required — TokenCake. Read it as a critique of the four notes preceding it — Lectures 18 and 22 from Nov 18, and Lectures 19 and 20 from earlier in this same meeting. Its claim is that agent-aware scheduling is insufficient without agent-aware memory allocation, and the critical-inversion example is the proof: a schedule that correctly identifies the critical agent still loses when a non-critical agent's blocks evict it. Reconstruct that example and check it against §21.3's fork picture — where exactly do the shared blocks stop being shared, and who is charged for the divergence? Then read the spatial scheduler for mechanism. Hold two questions: is this priority inheritance by another name, and would ForkKV's copy-on-write make the problem better or merely later?

Optional — TokenDance. The synchronized-round pattern is §21.4's barrier as a permanent structural feature rather than an occasional cost, and the collective sharing it proposes is §21.3's thick-sharing case taken to its limit. Read for how it handles the redistribution step, which is where the sharing is created.

Optional — ScaleSim. Memory management for large-scale multi-agent simulation, keyed on invocation distance. Read it as a structural estimator for Lecture 20 §20.5's p — how likely is this agent to be called again soon — recovered from the graph rather than from recency. Skim the simulation-specific machinery; the transferable claim is that structure predicts reuse better than history does, which is §21.3's argument about forks. Question: what is the invocation-distance analogue in an orchestrator–worker fan-out, and would it beat a TTL?

Optional — ForkKV. Copy-on-write KV for forked agents — the obvious idea, done properly, and the natural implementation of §21.3. Read the fork and divergence paths closely and compare them against Lecture 13's radix-tree match/split/insert: both end up sharing the same blocks, but one is told and the other has to discover. Question: on the thick-sharing fan-out (r = 80.2%), how much of the 5.05× ceiling does runtime matching actually reach, and where does the difference go — matching cost, or eviction between children?

Optional — large language monkeys. Read the coverage curves and only the coverage curves. The paper's contribution for our purposes is the empirical shape of E[best of k] and the sharp separation between coverage and realized accuracy — which is q, and which is §21.6's whole argument.

Optional — KernelBench. The domain where q is nearly 1. Read the harness design rather than the results: what makes a generated kernel accepted, what makes it timed, and what would a determined optimizer do to score well without being fast? KernelBench is also the central reading of the LLMs-for-systems track (Lecture 23), so this is a first pass rather than the only one — hold on to an answer to that last question, because it is the one Lecture 24 then generalizes to a whole research loop.

Looking ahead

Lecture 23, "LLM for Systems Research I" takes §21.6 as its subject, and on this calendar it is optional content rather than a meeting. SWE-agent and KernelBench are its readings, and the second one is this lecture's q argument with the numbers filled in: the most convincing agent results in the whole literature are in domains where a machine can check the answer, and a kernel that produces the right tensor in less time is the cleanest such domain there is. Part II spent the semester making systems serve models; that track asks what happens when models are pointed back at the systems.

Behind us, one note of the block sits out of chain order. Lecture 22 — Wednesday, November 18 — "Scheduling agentic programs" shared the request-DAG meeting rather than following this one. Everything in the block has assumed the scheduler knows something: the DAG in Lecture 18, the interception type in Lecture 19, the return probability in Lecture 20, the fan-out width and the tail today. Autellix assumes none of it; Teola goes the other way and optimizes the graph inside a single request. Read it against Lecture 18's declared-versus-deduced axis — and against today's §21.4, since a barrier is exactly the structure an unstructured scheduler cannot anticipate.

Deadlines. The final project report is due Dec 8 at 11:59pm and the project is worth 16% of the grade; Assignment 5 is due Dec 2 at 11:59pm, presented the week before in the sharing sessions on Nov 30 and Dec 2. If your project touches prefix reuse, §21.3 is the most useful section here: fan-out traces carry the highest achievable reuse in the whole workload, and a policy that recognizes a fork rather than rediscovering the sharing token by token starts several lengths ahead. For Assignment 5 it is §21.6, and the advice is uncomfortable. You will be tempted to spawn a fleet. Spend the first twenty minutes making your measurement trustworthy instead: a perfect verifier with four attempts beat a 70%-accurate one with sixteen, so a good ruler is worth four times the fleet — and it is the only part of the setup you fully control.

One sentence to carry out of the room. Every parallel speedup in this block was quoted as a mean and paid as a maximum, and every search was quoted as a best-of-k and paid as a best-you-could-identify — so before you widen anything, measure the tail and check the ruler.