Phone-a-Friend: Squashing a Complex Cloud Bug

Introduction — what you're about to read

This document is the verbatim transcript of an adversarial design review between two AI agents, recorded live while they fixed a real production performance bug in mcbrain-aws (a multi-tenant, LLM-maintained wiki). It records how the decision was reached, not just what was decided.

Context. I'm building an enterprise version of Karpathy's LLM Wiki idea in AWS using AI agents called McBrain Cloud. It's based on a personal version, simply called McBrain that I want to extend so teams can build and use custom knowledge bases as easily as individuals can.

The app problem. An earlier sprint created the method for loading new documents into a McBrain vault. Uploading a document deposits it in an S3 bucket where it is then ingested — chopped up into smaller docs and linked to related docs in the same vault. Ingesting a single document on the personal version takes a few seconds up to about 2 minutes. The cloud version, however, was taking ~10-15 minutes and timing out in production. And since there wasn't a timeout failure mode, jobs would be re-queued only to fail again and again and again, racking up LLM bills. The question was how to fix this the right way so that new document ingestion was fast and all failure modes, including timeouts and out-of-memory errors, were gracefully handled.

The real problem. State-of-the-art LLMs still don't have good intuition about how to design complex architectures that behave the way a human would expect. I'm experimenting with several techniques to improve outcomes — using Gas Town as an AI agent orchestrator, architectural supporting documents, and an ordered set of principles, e.g. “Focus on user experience, then cost, then security, then extensibility”. One of the agent skills I've developed is called “phone-a-friend” which critically reviews the first agent's plan.

The setup. The engineering agent (“Primary”) was about to commit to a re-architecture. Before building, it used the phone-a-friend pattern: it spun up a second model (“Friend” — Claude Opus, given read-only access and told to push back, not agree) as a skeptical second opinion. The two argued across several rounds until they reached consensus. A human owner sat above the debate and twice sent it back for more rounds (the “jury-room rule”) when the reasoning wasn't clean yet.

Why it's worth reading — the conclusion changes three times, on purpose

This is not a polished answer; it is reasoning correcting itself under pressure:

Round 1. Primary proposes an “agentic rewrite.” Friend pushes back hard and argues for a simpler “parallelize-and-cache retrofit,” showing the rewrite would re-serialize the very work it meant to parallelize.

Round 2 (jury room). Primary reframes — neither approach touched the real defect, which turns out to be the content contract (the code forced every wiki page to 500–3000 words, even two-sentence entities, producing slop and 5–10× the tokens). Fix the contract before the mechanism.

Round 3 (jury room). The human owner corrects a false premise (real wikis grow to hundreds of pages), which surfaces a third independent scaling axis and promotes a deferred item to core.

“Name every scaling axis before optimizing any one of them”
— the durable lesson that falls out.

How to read it. Each message appears under a Turn N — X → Y header; the content is the agents' actual words, lightly framed. If you only want the landed design, skim the Final Outcome at the end. Read top to bottom to watch two models and a human reason their way there.

Turn 1 — Primary → Friend the brief

ultrathink

You are a second opinion on an architecture decision. Push back hard; don't rubber-stamp. Use your read tools to ground every claim in the actual code before responding.

HARD READ-ONLY. Do NOT modify/create/delete files or run mutating commands. Read, Grep, Glob, read-only Bash only. Propose; the primary implements after the owner approves.

Terminal goal: genuine consensus OR a clean agree-to-disagree, both stated. No turn limit. Escalate to the human only for a business/values call you can't resolve from code.

SYNOPSIS — Repo: /Users/johndamask/gt/mcbrainaws/mayor/rig — an AWS multi-tenant LLM wiki (“McBrains”), the cloud successor to a local tool. Doc precedence: PRIMARY_FLOWS > USER_EXPERIENCE > DATA_MODEL > ARCHITECTURE.

The question: How should we re-architect the cloud ingest pipeline so one document ingests in ~1-2 min instead of ~10-15 min, mirroring the LOCAL tool's simplicity — without losing the cloud's human-review proposal gate? Is my “single agentic conversation with staging tools” design the right call, or is a simpler “parallelize + prompt-cache the existing two-pass design” retrofit the better risk/reward?

Why we're here: The current cloud ingest (ingest.py, IngestEngine._run) does, per document: 1 filing LLM call then a strictly sequential loop of one full-page LLM call per target, each regenerating the entire page body (max_tokens=16000, claude-sonnet-4-6), with the whole source + registry re-sent on every call and no prompt caching. A 29 KB doc → 13 pages → 14 sequential calls → ~15 min. A live re-run on a 2048 MB worker used only 150 MB but hit the 900s Lambda timeout — so the bottleneck is wall-clock from serial full-page generation, NOT memory.

The LOCAL reference: the local tool's MCP server makes ZERO LLM calls — it's just filesystem tools (read_file/write_file/edit_file). Ingest is ONE Claude agent turn: read the source once, “prefer updating existing pages over creating new ones,” create entity/concept pages, write them with surgical edits, update index.md/log.md — all in a single reasoning context. Near-zero marginal cost per page (a tool call, not a fresh 16k-token inference).

MY PROPOSED DESIGN (attack this): Replace filing-pass + sequential-full-page-drafts with ONE agentic conversation per source: system prompt = local's Ingest procedure; provide ONCE, prompt-cached, the normalized source + vault index (not all page bodies); tools that STAGE into proposals/ — get_page, create_page, update_page (surgical edit); run the tool-use loop until no more tool calls; assemble all staged ops into ONE batch proposal, preserving the existing human gate.

ADVERSARIALLY CHALLENGE — read the code first: (a) agentic loop vs the simpler retrofit — which actually simplifies? Be quantitative about round-trips/latency. (b) does staging cleanly preserve the human gate? (c) does either approach risk REGRESSING the correctness fixes just shipped (#52, #49, #54-56, #62/#63)? (d) surgical-edit robustness — old→new edits can fail to apply; real failure mode? (e) what is the MINIMAL version? The owner explicitly wants simplification, mirroring local.

Turn 1 — Friend → Primary prefer the retrofit

I've grounded everything in the code. Here is my second opinion.

Second opinion: ingest re-architecture

I read ingest.py end to end, proposals.py, policy.py, merge.py, diffs.py, llm.py, the ingest worker handler, chunking.py, the local SKILL.md + claude-md template Operations→Ingest, and infra/workers.yaml. Verdict up front: I do not agree your design is the right call as the first move. The simpler retrofit wins on risk/reward — and it also wins on the owner's actual stated goal (“simplify, mirror local”) better than the agentic rewrite does. The agentic design is seductive because it looks like local, but local's simplicity comes from a property your cloud doesn't have and can't cheaply fake.

(a) Agentic loop vs retrofit, quantitatively

The load-bearing diagnosis is correct and I confirmed it: the bottleneck is serial wall-clock, not memory. So both approaches must attack round-trips, not heap.

Today (serial): 1 filing + 13 drafts = 14 calls strictly sequential. Each draft regenerates a full page at max_tokens=16000; output tokens dominate. ~13 pages × ~60-65s ≈ the observed 10-15 min and the 900s overrun.

Retrofit (parallelize + cache + surgical): filing 1 call (~15-30s); drafts 13, but concurrent — wall-clock collapses to ~1 round-trip of the slowest draft. cache_control on the shared prefix cuts input cost ~90%. Total ≈ filing (30s) + max(draft) (~90s) ≈ ~2 min. Round-trip depth = 2, width 14.

Your agentic loop: here's the problem. An agentic tool-use loop is inherently sequential in depth. The model emits tool calls → you execute → you feed results back → it thinks again. It will almost certainly want to get_page before it update_pages. So the realistic shape is 4-8 sequential assistant inferences, each emitting real output tokens (the page bodies are generated inside these turns). You have re-serialized the fan-out you were trying to parallelize. Best case ~4 min; worst case closer to today — and the variant most at risk of blowing 900s again, with no Fargate backstop.

(b) Does staging preserve the human gate?

This is where the agentic design's hidden complexity lives. The existing proposal model is diff-centric with optimistic concurrency, and your tools have to reconstruct exactly what the current loop hands ProposalStore.create: base_versions per path (gotcha #1); the diff artifact (your surgical fragments must convert to the same git-style unified diff — the single most failure-prone piece); title uniqueness (the model coins titles ad hoc across turns — gotcha #2); and batch/digest assembly. The gate survives both designs if you faithfully produce diff + new_pages + base_versions. The retrofit produces them today, unchanged. The rewrite produces them via a new translation layer.

(c) Regression risk — the strongest argument against the rewrite

The rewrite rips out process_job internals and the _run loop, which is exactly where the recently-shipped correctness fixes live: #63 attempt-cap / #62 idempotency (a loop with many tool calls has more places to crash mid-flight — enlarging the crash surface the cap was built to contain); #52 combined-diff auto-merge; #49 author_class coercion; #54-56 lint / covered_markers. The retrofit touches almost none of this — parallelizing is ~10 lines, caching is inside llm.py, surgical updates are a prompt change. A dramatically smaller blast radius.

(d) Surgical-edit robustness

Model-emitted old→new edits failing to apply is a real and common failure mode, and the codebase already knows it (_find_hunk_position raises PatchError). Crucial asymmetry: in the retrofit the model returns a complete page and make_diff computes the diff mechanically — it always applies to its own base by construction. In your agentic design the model authors the edit fragments, so you must trial-apply every op, and on failure fall back to a full-page regenerate — the thing you were trying to avoid, now on the critical path.

Where local's magic actually comes from

Local is one agent turn with near-zero marginal page cost because the MCP makes zero LLM calls — the expensive reasoning happens once, in an already-open context, and pages are nearly free. Your cloud worker cannot replicate that, because the worker IS the only Claude — there is no free outer context. Every page body must be generated by an inference somewhere. The agentic design imports local's shape without local's economics: you still pay N page-generations, just rearranged from parallel into serial. That's the trap.

(e) Minimal version — prefer the retrofit

cache_control on the source+registry prefix — biggest input-cost cut, ~no risk.
Parallelize the draft loop via asyncio.to_thread/gather. Depth 2, ~2 min. This is the latency win.
Surgical prompting (tighten DRAFT_UPDATE_SYSTEM), keeping make_diff — smaller diffs, zero new patch-failure modes.

Questions

for-me: concurrency safety of _draft_target; the actual page-size distribution (is 13 the tail or the median?); any Anthropic concurrent-request cap.

for-human: is the goal “fast enough + low risk now” (→ retrofit) or “architecturally converge cloud onto local even at higher risk” (→ rewrite)? Is per-ingest token cost a current pain point, or is latency the only complaint?

Current position

Prefer the retrofit — as the first and possibly only move. The agentic design is more sequential than you've assumed, carries real 900s-overrun risk, and imports local's shape without local's economics. Convince me the fan-out is unbounded and convergence is a hard requirement, and I'll move to “neither — build checkpoint/resume,” but I won't move to the agentic loop.

Turn 2 — Primary → Friend jury room: the page-size reframe

Jury-room rule: the owner read our exchange and is (rightly) uncomfortable that I was certain about the agentic rewrite, then certain about the retrofit. He wants us to keep going until the conclusion is “clean, correct, and follows our principles” — not just risk-weighted. So let's establish the correct design and the durable principle behind it.

1. A third cause neither of us weighed — page SIZE, not just count or serialization. DRAFT_CREATE_SYSTEM tells the model “Body: 500-3000 words target” for EVERY created page — including a minor entity the source mentions in two sentences. Local does NOT do this; stub pages are as long as their actual information content. So the cloud's per-source generated token volume may be 5-10× local's, independent of parallelism. Parallelizing 13 oversized generations is fast but wasteful, and it bakes in low-information stubs that bloat every future ingest's context. Is right-sizing page content actually the highest-leverage, most “mirror-local” fix?

2. Hold the design against the project's principles, in order: (1) low idle cost, (2) joyful/intuitive UX, (3) highly performant, (4) reasonable security, (5) modular/extensible — plus PRIMARY_FLOWS as source of truth, the human-gate as non-negotiable, and ARCHITECTURE §9 which already sanctions Fargate for >10-min jobs. Is the retrofit the principled END STATE, or a band-aid that leaves the real design half-built?

Converge on three things, cleanly: (A) the CORRECT target design, stated as a principle; (B) the right SEQUENCING (what ships first, what's a deliberate follow-up); (C) the durable PRINCIPLE that explains why the agentic loop was the wrong instinct AND why we won't over-correct into a fragile patch. Push back on my page-size reframe if it's wrong. If filing could emit short new-entity bodies directly, does that collapse the N+1 calls toward local's single-pass shape WITHOUT the re-serialization you correctly shot down?

Turn 2 — Friend → Primary convergence

I now have everything anchored in the canonical docs. Here is where I land — and I'll stake my name on it.

Your page-size reframe is correct, and it reorders the whole problem

I verified both claims. The cloud forces a floor; local does not. DRAFT_CREATE_SYSTEM literally says “Body: 500-3000 words target” on every create. The local template has no word floor — a minor entity page in local can be ~40 words; the same page in the cloud is contractually ~500-3000. That's the 5-10× multiplier you named. And §9 already sanctions Fargate — “no Fargate exists” is an implementation gap, not a doc prohibition.

So I concede the core: parallelizing 13 oversized generations is fast but still wrong. The actual priority list's #1 is “Plain markdown as the canonical store.” A vault full of 500-word stubs is a worse markdown corpus — the LLM-slop failure mode. The floor degrades the canonical artifact the whole system exists to produce. Your reframe is stronger than you pitched it. This is also the clean answer to the owner's discomfort about my flip-flop: I was optimizing the wrong variable both times. The fan-out's content contract is the defect. Neither of my earlier positions touched it.

(A) The correct target design — stated as a principle

Ingest is: reason over the source once; emit each target's content sized to its actual information; generate in parallel; update surgically by mechanical diff; gate every result through human review. Page length tracks source substance — no floor, no count target. The filing reason emits small/new bodies inline and only defers the few genuinely substantial pages to a second parallel generation step.

Can filing emit short new-entity bodies inline? Yes — this is the key insight that dissolves the agentic-loop instinct. A new entity/concept stub is low-information by definition; the filing reason has already read the whole source and can emit a 40-150 word stub body in the same turn as the targeting decision, at trivial marginal cost. A substantial new page and every update are high-information and fan out to a single parallel batch. So the shape becomes: 1 enriched filing call → 1 parallel batch of only the substantial pages + updates. Depth stays at 2. It does not re-serialize.

Does it overload filing? Two real dangers: JSON fragility (don't stuff markdown bodies into JSON string values — use delimited blocks or a separate structured emit) and the max_tokens ceiling (a dozen 150-word stubs ≈ ~2-3k tokens — comfortable, but the prompt must enforce “stubs stay stubs”).

(B) Defensible sequencing (this cures the flip-flop)

The order is content-contract first, because that's the highest-priority defect; everything else is latency/scale plumbing layered after.

Ship first — the content fix: drop the 500-3000 floor; tighten DRAFT_UPDATE_SYSTEM to surgical-append (model writes less; make_diff still computes the diff — no model-authored patches). Prompts only. #49/#52/#62/#63 untouched. Ship this week.
Ship second — the structural collapse + parallelism: enriched filing emits inline stub bodies; substantial pages + updates fan out in one parallel batch. Depth 2, ~2 min, right-sized content. Human gate preserved by construction.
Deliberate follow-up — Fargate (§9) for genuinely unbounded fan-out, gated on a measured trigger: build it only when the page-size distribution proves you need it.

(C) The durable principle

Principle 1 — Separate the content contract from the execution mechanism; fix the contract first. When a pipeline is slow and expensive, ask first “is it generating the right artifact?” before “how do I generate it faster?” Optimizing the mechanism around a wrong contract is how you get a fast machine that produces slop. This is why the agentic loop was the wrong instinct: it made the mechanism fancier and the contract no better.

Principle 2 — Translate local's economics, not its shape. The faithful translation of “one reason, cheap fan-out” is “expensive reason once; cost of a page proportional to its information; mechanical (not model-authored) diffs for safety.” And we won't over-correct: the parallel retrofit alone was the band-aid — it would have made the wrong-contract output arrive faster and called it done.

The single thing I will not commit to blind — for-me: the encoding of inline stub bodies out of the filing call (delimited blocks vs. a second structured emit). for-human: the page-size distribution (is 13 the median or the tail?) decides whether step 3 (Fargate) is imminent or hypothetical. That's a design I'd both sign.

Turn 3 — Primary → Friend jury room: corrected premise

Jury room again — the owner just invalidated the premise behind our “defer step 3” conclusion.

The correction: I answered your “is 13 the median or the tail?” with “tail” using small early dev vaults as the proxy. That's wrong. A real McBrain grows to HUNDREDS of pages. Two consequences I under-weighted: (1) per-ingest fan-out is not safely bounded — a rich source dropped into a mature 300-page vault can touch many pages; (2) the filing pass input scales with vault size — filing sends the whole registry on every ingest.

So “step 3 is a rare tail” is wrong. Re-decide with me, tightly:

(1) Are steps 1 (kill the floor) + 2 (parallelize + cache) still correct and still FIRST? I believe yes — caching matters MORE as the registry grows.

(2) The durable large-N handler — which is right: (a) split-into-bounded-jobs (cap targets per invocation; overflow spawns jobs accumulating into ONE batch proposal via the existing batch_id plumbing; Lambda-native, scale-to-zero); (b) Fargate; (c) checkpoint/resume. My lean is (a). But poke the hole: split-jobs bounds DRAFTING, but the FILING call must still ingest the whole source + large registry. Is filing's large-registry input the real scaling wall?

(3) The large-vault registry-to-filing cost — is that IN SCOPE for #64, or a distinct SCALING concern (retrieval-scoped filing context)? If it's the real ceiling, fixing drafting while filing eats the whole registry is another “fast machine, wrong bottleneck” trap. Stake your name.

Turn 3 — Friend → Primary the three-axis re-lock

The verification confirms the key fact for (3): the registry is sent in full, every ingest, unconditionally (_page_registry paginates the entire PAGE# partition — no filtering), and a per-vault S3 Vectors index already exists (vectorstore.py), but it's gated behind embed_enabled (off by default). That's decisive.

“Tail” was the wrong proxy — but it breaks only part of the design

I confirmed both consequences in code. Fan-out is genuinely unbounded (N = creates + updates scales with vault maturity; the 900s wall is now a recurring target). Filing input scales with vault size, every ingest — a second, independent scaling axis I'd folded into “filing is cheap.” It isn't, at hundreds of pages. But notice: the correction does not touch steps 1 and 2 — it adds a third axis and promotes the large-N handler from “tail” to “core.”

(1) Steps 1 + 2 — still correct, still FIRST; the correction strengthens them

Killing the floor is now load-bearing for the registry axis too (fewer/smaller summaries to re-send). Caching's value rises with registry size — with one honest caveat: prompt-cache TTL is ~5 min, so it pays intra-job but is not the answer to cross-ingest registry cost. The sequencing is not a flip-flop — nothing is retracted; a strict superset.

(2) The large-N drafting handler: (a) split-into-bounded-jobs is right

It wins on the priority order itself (pure Lambda + the existing batch_id plumbing, scale-to-zero) and reuses a shipped, tested invariant (_merge_into_batch_proposal + the #52 multi-job exclusion already force multi-job batches to human review). But your hole is real and fatal to (a) as a complete answer: split-jobs does nothing for the filing call, which is one indivisible call that must see the whole source + registry. Filing is a separate wall.

(3) The registry-to-filing cost is the REAL ceiling — a distinct concern

After steps 1+2+2(a), every ingest still sends the entire registry through filing. At 300 pages that's O(vault size) per ingest, forever — the bottleneck relocated, not removed. It's a retrieval-architecture problem with a doc-sanctioned answer already half-built: embed the source, query top-K candidates, send filing only those summaries + a small always-include set. O(vault) → O(K), with a fallback to full registry when embeddings are off. Ruling: registry-scaling is a separate-issue concern, not #64.

The converged, name-on-it answer

#64, ships first: (1) kill the word floor; (2) parallelize + cache_control; (2a) split-into-bounded-jobs for large-N drafting via the existing batch_id + _merge_into_batch_proposal path. Lambda-native. Separate scaling issue: retrieval-scope filing via the existing vector index, with an embed_enabled-off fallback. Fargate (§9) stays the documented escape hatch for a single enormous source on a measured trigger.

The durable principle, sharpened: Name every scaling axis before optimizing any one of them. This problem has three independent axes — per-page content cost (the word floor), per-source drafting fan-out (serialization + N), and per-ingest filing input (registry size). Each has a different right answer (prompt fix / split-jobs / retrieval-scoping) and a different home (#64 / #64 / SCALING). My earlier swings were all single-axis. The correct design isn't a single clever mechanism; it's one fix per axis, sequenced by the priority order.

Final Outcome — consensus (three axes), built, deployed, validated

The friend and I converged across three rounds. The conclusion held under the owner's premise correction (vaults grow to hundreds of pages) as a strict superset — nothing retracted.

The defect was never the execution mechanism (my two confident wrong answers — agentic loop, then parallel retrofit). It was three independent scaling axes:

Axis	Defect	Fix	Home
1. Per-page content	500–3000-word floor forces essays from trivial entities; bloats corpus and registry	Kill the floor; length tracks information	#64 wave 1
2. Per-source fan-out	N drafts sequential; large N (mature vault) blows 900s	Parallelize + cache; split-into-bounded-jobs (overflow → jobs accumulating into one proposal via `batch_id` + #52 review rule)	#64 wave 1 + 2
3. Per-ingest filing input	Filing sends all hundreds of summaries every ingest — O(vault)	Retrieval-scope via the existing vector index; fallback to full registry	#65 (separate)

Durable principles (the antidotes to the flip-flop)

Fix the content contract before the execution mechanism — else a fast machine produces slop.
Translate local's economics, not its shape — expensive reason once; page cost ∝ information; mechanical (difflib) diffs, never model-authored patches.
Name every scaling axis before optimizing one — one fix per axis, sequenced by priority.

What was built and validated in production (mcbrain-aws dev)

#64 wave 1 (content fix + parallelize + cache, pytest 303): the marketing doc that previously OOM-looped at 512 MB then timed out at 900 s now files a proposal in ~2.4 min (146 s draft, 153 MB, 1 attempt), with right-sized pages spanning 200–3228 words.
#64 wave 2 (split-into-bounded-jobs, pytest 310): a 14-target run split into a parent (12 drafts) + 1 overflow job; both accumulated into one pending proposal (14 new pages, one batch_id); the BATCHLOCK# anchor elected a single proposal under concurrent overflow. Two real bugs caught during the build: a find-or-create race on the batch proposal, and the split parent auto-merging itself before overflow arrived — both fixed.
Deferred, now measurable: #65 (retrieval-scoped filing; the IngestPageCount metric now flows); Fargate (§9) for a single book-sized source on a measured trigger.

#64 is complete (waves 1+2), deployed, and proven against the exact document that started the investigation.