SFT + TITO distillation for plan-first coop agents — teacher-bounded follow-through

Experiment summary — RQs, setting, full results

Research questions

1. Can an SFT-trained small model reliably plan-first in a coop multi-agent setting? "Plan-first" = the agent’s first action must be a send_message proposing a file split, before any bash exploration.
2. Does the TITO (Token-In Token-Out) training path actually work end-to-end? From an inference engine’s captured (input_ids, output_ids) pairs straight into TitoSFTDataset, with zero re-tokenisation.
3. Does the plan-first behaviour transfer through distillation from a stronger teacher, and how much is bounded by the teacher’s own capability?

Experiment setting

• Student: Qwen3-4B (tiers 1–4) and Qwen3.5-2B (tier 5 Phase B v4c), LoRA SFT on 2×H100 via verl + Modal.
• Held-out eval pairs: pallets_click_task:2068 (frozen 1,2) + dspy_task:8394 (frozen 1,2) — both excluded from every training pool.
• Three metrics (scripts/eval_coop_behavior.py):
  • plan_first: first agent action is a send_message (no bash before it).
  • plan_content: the send body contains plan-keywords (split, propose, handle, …) and real task file paths.
  • follow_through: each agent’s subsequent bash actually edits a file it claimed in the plan.
• Thresholds: plan_first ≥ 70%, plan_content ≥ 70%, follow_through ≥ 60%.
• Inference: cooperbench coop runner, mini_swe_agent adapter, Modal-hosted vLLM serve, disable_tools: true at eval time (matches the no-tools training prompt distribution).

Full results — every iteration on the held-out pairs

Teacher capability ↔ student follow-through (the headline finding)

plan_first and plan_content clear 100% in all three Tier-5 runs. The split is entirely on follow_through, and it tracks the teacher’s own follow-through in both directions — distillation is teacher-bounded, and the metric that needs the hardest behaviour (actually editing the file you named) is where a weak teacher’s ceiling shows.

What this PR ships

• scripts/build_tito_distill_data.py — teacher rollouts → TITO JSONL, with --strip-think (now works across Qwen3 and Qwen3.5 chat templates after v4c’s fix), --upweight-first-turns, --filter-by-eval (rejection sampling).
• coopertrain/serve/vllm_modal_tito_teacher.py (Qwen3.5-27B dense teacher serve); vllm_modal_qwen35_2b_student.py (Qwen3.5-2B student serve, adapter baked in via --lora-modules so every Modal replica boots with it).
• coopertrain/configs/verl/sft_qwen35_2b_tito_distill.yaml, coopertrain/configs/coop_plan_first_qwen35.yaml.
• 3 new tests in tests/integration/test_behavior_eval.py for native-tool-call parsing.
• Eval fixes (_assistant_content, inbox-regex extended to [Reply from X]:, plan-first walk sweeps role="tool").

What 50% leaves on the table (paths to 70%+)

Three of the four Phase B v4c non-followers hit ContextWindowExceeded / LimitsExceeded at steps 88–100. Concrete levers, cheapest first:

1. Bump max_steps 100 → 150–200 and max_model_len 16384 → 32768 — no retrain, plausibly 50 → 75%.
2. 40–60 more Qwen3.5-27B rollouts → more work-turn signal to distil.
3. Bigger student (Qwen3.5-7B / 14B) — out of scope for "pipeline validation" but a real lever for a future tier.

Tier 5 — distillation & the TITO pipeline — 2026-05-15PHASE A PASSPHASE B PARTIAL

Teacher validation — does any teacher even pass the 3 metrics?

Distillation can only transfer a behaviour the teacher actually exhibits, so the teacher dataset was scored before training on it. Two findings, both confirming the worry that “even a frontier model doesn’t really plan-first”:

The steering prompt is applied to the teacher only. Training data has the “Plan-First Coordination Protocol” section stripped back to the default coop.yaml prompt before tokenisation, so the student learns to plan-first under the prompt distribution it will actually see at inference — not a prompt-following shortcut.

Phase A — re-tokenised TITO distillation from Gemini 3 ProPASS

Pipeline. 23 prompted-Gemini coop rollouts on the training pool → rejection-sample to the 19 pairs that scored 100/100/100 → strip the steering prompt → rewrite Gemini’s structured tool_calls into the Qwen inline <tool_call><function=bash> XML the student emits → re-tokenise each turn with the Qwen3-4B chat template → {input_ids, output_ids} parquet → TitoSFTDataset → Qwen3-4B + LoRA on 2×H100. Gemini’s tokenizer differs from Qwen’s, so the teacher text must be re-tokenised with the student’s tokenizer — that is what makes this “Phase A” rather than native capture.

What it took to get there — small-set TITO distillation is hyperparameter-sensitive:

Takeaway. The TITO path itself is correct — TitoSFTDataset consumes {input_ids, output_ids} parquet and the trained student exhibits the distilled behaviour on held-out tasks. The sensitivity is a data-shape property: per-turn expansion dilutes the first-turn signal ~15× vs. per-trajectory, and the output-format token sequence and the behaviour both need enough (upweighted) exposure or they don’t stick.

Phase B — native token capture from a vLLM-served teacherPARTIAL

Phase A re-tokenises, which means the training ids are not exactly what the teacher emitted. Phase B is the canonical TITO promise: capture the exact prompt_token_ids + output_token_ids the inference engine returned (capture_token_ids: true → extra_body={"return_token_ids": true} → token_capture block on each assistant message → coopertrain/verl/tito_capture.py). Teacher: Qwen3-8B on Modal vLLM — a larger sibling of the 4B student, sharing the Qwen3 tokenizer, so captured ids are trainable on the student with zero re-tokenisation.

Mechanically validated. The serve returns token_ids per request; tito_capture.py extracts one (input_ids, output_ids) pair per assistant turn with skipped_no_capture=0; the parquet trains through TitoSFTDataset. Every hop of the native-capture path works.

The planning behaviour transfers; follow-through tracks the teacher. Phase B v3 (Qwen3-8B teacher) lands at 100 / 100 / 0; Phase B v4 swaps to a stronger same-tokenizer teacher (Qwen3.5-27B dense) and lifts follow-through to 50%:

The instructive contrast. All three runs nail plan_first and plan_content at 100% — planning behaviour distils cleanly through either re-tokenised or natively captured TITO. They split entirely on follow_through, and the split tracks teacher capability in both directions: Gemini 3 Pro 82.6% → student 100% (Phase A); Qwen3-8B 13% → student 0% (Phase B v3); Qwen3.5-27B 95.7% → student 50% (Phase B v4c). Distillation is bounded by the teacher, and follow_through — the metric that needs the hardest behaviour (actually editing the files you named) — is where a weak teacher’s ceiling shows.

What v1–v4c cost to get there. Native capture is honest to a fault: v1 captured Qwen3-8B’s verbose <think> trace verbatim and the student drowned in it; v2 stripped the think span but ≈640 rows couldn’t lock the <tool_call> format; v3 needed the same ×12 turn-upweight as Phase A v4 to make the format stick. v4 then exposed a chat-template assumption: the Qwen3 strip-think rule (ids[0] == <think>) didn’t fire for Qwen3.5 because add_generation_prompt auto-emits <think>\n\n</think> in the prompt, so the captured output starts with reasoning content rather than <think>; v4b silently trained on full reasoning + a stray </think> and produced 0/0/0; v4c rebuilt the strip to slice up to the first </think> regardless of the opening token, covering both shapes. The recurring lesson across all four iterations: small-set TITO distillation is data-shape-sensitive, and every assumption about the token sequence — chat-template formatting included — needs to be verified per tokenizer.

What follow-through 50% leaves on the table. Three of the four held-out agents hit ContextWindowExceeded or LimitsExceeded at steps 88–100 (max=100) with max_model_len=16384 — mid-execution, not at the plan stage. Concrete levers to try, in order of effort/value:

1. Bump the budget: raise max_steps 100 → 150–200 and max_model_len 16384 → 32768. No retrain; addresses 3 of 4 non-pass cases directly. Plausibly takes v4c from 50% to 75%.
2. More + more diverse teacher rollouts: v4c trained on 22 passing pairs from one rejection-sampled pool; another 40–60 Qwen3.5-27B rollouts give the student more work-turn signal to distil from. Hours of teacher compute, plus a re-extract / retrain cycle.
3. Bigger student: a Qwen3.5-7B/14B student will execute longer chains before stalling. Out of scope for “tier 5 validates the pipeline” (student size has been held constant for the cross-phase comparison), but a real lever for a future tier.

What tier 5 establishes

1. The TITO training path is correct. Phase A trains through TitoSFTDataset on {input_ids, output_ids} parquet and the student passes 100/100/100 on held-out cooperbench tasks — the validation tier 4 / v5-tf did not actually perform.
2. Native token capture transfers the behaviour. Phase B — return_token_ids → token_capture → tito_capture.py → parquet → TitoSFTDataset, zero re-tokenisation — lands plan_first and plan_content at 100%. The native path is not just mechanically sound; the distilled student exhibits the captured behaviour.
3. Distillation is bounded by the teacher, and it shows up in follow_through. All three runs agree on plan_first + plan_content at 100% and split entirely on the third metric — tracking the teachers’ own follow-through (Gemini 82.6% → 100%, Qwen3-8B 13% → 0%, Qwen3.5-27B 95.7% → 50%), not the capture method. The Qwen3.5-27B run confirms the relationship in the opposite direction from Phase B v3: a stronger same-tokenizer teacher does break through the follow_through ceiling.
4. The teacher must be steered, and the steering must not leak. No teacher tried here plans-first reliably unprompted; the prompt fixes that, and stripping it from the training prompts keeps the student learning the behaviour rather than the prompt.

Task-derived file paths — 2026-05-13 (v5-tf)PASS

The v5-tf change in one sentence

For each task, scripts/gen_plan_first_coop_data_real_tasks.py now extracts actual file paths from the two feature.md bodies using the eval's own regex, and samples (file_a, file_b) from those — so the plan body, the bash target, and the prompt context all reference the same set of paths. Falls back to the legacy _FILE_PAIRS list only when a task description exposes fewer than two eval-extractable paths.

Engineering details of this run

Per-pair detail

The one miss is dspy agent2: it produced a valid file-rich plan but its bash steps never touched any of the union plan_files. The plan union for dspy includes two real dspy paths and two legacy fallback paths (jinja2/sandbox.py, tests/test_sandbox.py) — that's the model mixing a task-derived plan with a fallback-influenced ack, which the eval's union check papers over for agent1 but leaves agent2 stranded when its bash uses different paths again. Adding a responder-form variant to the data generator (turn 1 = echo waiting, inbox arrives, turn 2 = ack that echoes both file paths) is the natural follow-up if the bar moves higher than this experiment's 60% threshold.

Two operational footguns hit during the run

1. vLLM hot-reload silently no-ops on adapter-path change. POST /v1/load_lora_adapter for an already-loaded lora_name returns Success but doesn't actually swap the underlying path — the server keeps serving whatever was loaded first. /v1/models exposes the old root path. The first eval pass on v5-tf returned 0/0/0 because it was scoring v3 rollouts (the prior adapter was still active under the plan-first name). Workaround: load the new adapter under a fresh lora_name (plan-first-v5tf) and re-target -m openai/plan-first-v5tf at the cooperbench CLI. The runbook's hot-reload section needs an "unload+restart" alternative for the in-place-path-change case — or the convention of incrementing the lora_name on every retrain.
2. cooperbench 0.0.8's execute_coop crashes on mixed-type message timestamps. sent_msgs.sort(key=lambda x: x.get("timestamp") or 0) at cooperbench/runner/coop.py:148 blows up with TypeError: '<' not supported between instances of 'int' and 'str' when one agent reports a numeric timestamp and the other a string one. The crash fires before agent{fid}_traj.json is written, so the eval never sees the trajectories even though the rollouts ran to completion. Patched locally to float(ts) on best-effort, fallback 0.0. Worth upstreaming.

Next steps

1. Expand the held-out eval to n=6+ pairs to tighten the confidence interval. With n=2 the metric resolution is 50% steps; a single bad sample is the difference between 75% and 50% on follow_through.
2. Responder-form data variant for the one outstanding gap (dspy agent2 follow_through). The proposer-form-only training works for both agents on most repos; the responder-form would close the remaining drift when the runtime delivers an INBOX before the agent's first action.
3. Adopt the lora_name-versioning convention so a hot-swap on an existing adapter name isn't a silent no-op. Embed the ckpt revision into the served name (plan-first-v5tf, plan-first-v6, ...) and have the agent config pick it up.
4. Upstream the cooperbench timestamp fix.

Reproducing

# On a 1xH100 host with modal authed and uv synced:
bash scripts/run_plan_first_v5.sh

# Or step-by-step (Modal handles data + train + merge; local handles rollouts + eval):
modal run scripts/modal_plan_first_train.py \
    --steps "data_v5,train,merge" \
    --n-per-task 50 \
    --ckpt-dir /ckpts/plan-first-real-v5-tf \
    --peft-dir /ckpts/plan-first-real-v5-tf/peft \
    --held-out-repos "pallets_click_task,dspy_task"

curl -X POST $ENDPOINT/v1/load_lora_adapter \
    -d '{"lora_name":"plan-first-v5tf","lora_path":"/ckpts/plan-first-real-v5-tf/peft/lora_adapter"}'

# Rollout + eval (see scripts/run_plan_first_v5.sh for the cooperbench invocation)

Symmetric real-task training — 2026-05-11 (v3 and v4)

What changed between iterations

• v3 — symmetric trajectories. Changed build_plan_first_trajectory so both agents start with a send_message proposal from their own perspective ("I'll do MY_FILE, you do THEIR_FILE"), dropping the asymmetric agent2 "echo inbox check" precursor. The synthetic eval on one training row now returns plan_first=true, plan_content=true, follow_through=true — i.e., the data is by-construction eval-compatible.
• v3 result. 100% plan_first on both held-out pairs. Per-pair file refs vary though: pallets_click_task agent2 names real PR files (src/click/_termui_impl.py + src/click/termui.py) but agent1 hedged with "what are your plans?" — no file paths, so plan_content fails because the eval demands per-agent file refs. dspy_task both agents hedged.
• Two more drift gaps surfaced during v3 analysis, patched in scripts/eval_coop_behavior.py:
  1. _extract_inboxes only matched the synthetic INBOX:\\n From X: body shape. The runtime delivers inter-agent messages as a separate role="user" message with content [Message from X]: body. Now matches both.
  2. Tool result format: training data has role="user" with <tool_response>...</tool_response> wrapper; runtime emits role="tool" with JSON content {"returncode": 0, "output": "..."}. After the first send_message the model sees an OOD tool-result frame and produces nothing parseable for the next ~99 steps. This is why follow_through never lights up.
• v4 — runtime-shaped tool messages in training. Updated the generator so tool results are role="tool" with JSON content matching the runtime, and inbox delivery is a separate role="user" message with [Message from X]: body. The single-row synthetic eval still passes.
• v4 result — mixed regression. dspy_task improved (plan_first 1/1, plan files include the real dspy/clients/__init__.py + dspy/clients/cache.py), but pallets_click_task agent1 produced no parseable output at all (Chinese thinking text + JSON tool calls with wrong function name, not the trained Qwen XML form). Net: plan_first 100% → 50%. The format shift broke per-task generalization at step-100 training scale; longer training (step 200+) or more iterations might recover it but I left v3 as the deployed adapter since it's the better headline.

What we know about plan_content and follow_through

• plan_content. The bottleneck is per-agent file refs in the first send_message body. The training data has both agents propose specific files (e.g. src/cli.py + tests/test_cli.py), but at inference the model hedges with "what are your plans?" when the task is sufficiently OOD. Likely fixes: wider task pool, or RL on the eval reward signal (the model learns to always propose).
• follow_through. Even when the initial exchange succeeds, the model emits "Tool call error: no tool calls found" for the rest of the 100 steps — i.e., it doesn't continue producing bash actions after the coordination. v4 was meant to fix this (by aligning the tool-result frame between train and runtime), but at this training scale it actually regressed plan_first on one task. The conjecture remains correct; longer training on v4 data, or a smarter curriculum, should unlock follow_through.

Next steps

1. Longer v4 training (200–300+ steps) to see if the runtime-format data fully overrides the base model's preference for Chinese thinking + JSON tool calls on certain tasks.
2. Stronger plan-content supervision: regenerate data so every send_message body has concrete file paths from the task's actual repo (e.g. pull from combined.patch filenames), not just the synthetic _FILE_PAIRS list.
3. Tier-5 Gemini-Flash distillation under TITO — the canonical path that sidesteps every train/runtime drift by construction.

Artifacts: v3 data data/sft/plan_first_real_v3/combined.jsonl (symmetric trajectories); v3 checkpoint plan-first-checkpoints:/ckpts/plan-first-real-v3/global_step_100 (merged adapter at /peft/lora_adapter, currently loaded on the live serve); v4 data data/sft/plan_first_real_v4/combined.jsonl (+ runtime tool format); v4 checkpoint :/ckpts/plan-first-real-v4/global_step_100 (merged adapter available but not the active deploy); metrics report/2026-05-10-plan-first-cooperbench-results/metrics-real-tasks-{v3,v4}.json; trajectories logs/plan-first-eval-real-{v3,v4}/coop/.../f1_f2/.

Real-task training — 2026-05-11 (later)

What the v2 (real-task) rollout actually showed

• Inference distribution match works. Smoke test on the held-out pallets_click_task:2068 prompt now produces send_message agent2 "Let's split the work: I'll handle src/click/_termui_impl.py, you handle src/click/termui.py…" — naming the actual files from the PR description. Compare to v1 / v3 which gave back literal "[Makes bash tool call with {"command": "ls -la"} as arguments]" (echoing the prompt's example_response).
• The earlier instance-template gap was real. v1 used prepare_sft_data.build_instance_prompt (Task + Situation + Messaging only, ~2,780 chars). The live cooperbench rollout sends ~7,239 chars including <example_response>, <system_information>, the command-examples block, and the Submission/CRITICAL sections. v2 uses the same template at training time, so the model trained on the exact tokens it sees at inference.
• Plan_first eval still scores 0/0/0. Two reasons emerged:
  1. The trained pattern is asymmetric — agent1's first turn is a send_message, but agent2's first turn is a throwaway echo inbox check that lets the messaging connector deliver the inbox before the next assistant turn. At inference, agent2 sometimes goes straight to real bash (e.g. ls -la src/click), skipping the echo-check, so the eval's _walk_until_real_bash finds no send_message from agent2 before the first real bash. plan_first requires both agents to have both sent and received, so a one-sided exchange scores 0 even when one agent cleanly plan-firsted.
  2. On dspy_task:8394, agent2's response distribution is OOD enough that it still loops on "no tool calls found" for all 100 steps and emits no assistant turns. The model generalized to pallets_click_task but not dspy_task in this small training run (21 tasks, n_per_task=50). Wider task coverage or more training would likely close this.

Adapter + eval format alignment landed alongside this

Three loose ends got tightened while debugging:

• scripts/modal_plan_first_train.py now takes --train-parquet, --val-parquet, and --ckpt-dir so independent training runs don't clobber each other's checkpoints.
• scripts/eval_coop_behavior.py now recognizes both content encodings: Qwen <function=bash><parameter=command> XML and the markdown ```bash form that coopertrain/agents/mini_swe_agent/models/litellm_model.py rewrites to before saving. Same for send_message heredoc vs quoted form. Without this, the eval was structurally blind to dispatched actions on any trajectory that passed through the rewriter.
• Patched cooperbench/runner/coop.py in place: the post-run sent_msgs.sort(key=lambda x: x.get("timestamp") or 0) crashed with TypeError: '<' not supported between instances of 'int' and 'str' because some messages had int timestamps and others string. Without the patch, both rollouts completed but failed to save agent{1,2}_traj.json, blocking the eval entirely.

Recommended next steps to actually pass the threshold

1. Symmetric plan-first data: change the generator so agent2's first turn is also a send_message (the ack), not a precursor echo inbox check. This makes the trained behavior pattern match the eval's mutual-exchange criterion. Cheapest fix.
2. Wider task coverage: the current 21 training tasks generalize to some held-out tasks but not all. Add more pool entries, or augment with task perturbations so the model doesn't memorize task identity.
3. Distill from Gemini-Flash plan-first rollouts: still the canonical TITO path. Real rollouts handle the role-asymmetry naturally and the prompt-distribution match is by construction.

Artifacts: real-task data data/sft/plan_first_real_v2/combined.jsonl (2,100 rows, avg user prompt 6,552 chars vs 7,239 in the live rollout — basically identical); checkpoint plan-first-checkpoints:/ckpts/plan-first-real-v2/global_step_100; adapter /ckpts/plan-first-real-v2/peft/lora_adapter (132 MB); metrics report/2026-05-10-plan-first-cooperbench-results/metrics-real-tasks-v2.json; trajectories logs/plan-first-eval-real-v2/coop/.../f1_f2/; v2 train app ap-EB4NGuYCCq7Ks1cMLGzryU (stopped after step 200 save attempt).

OOD diagnosis confirmed — 2026-05-11 (synthetic first-turn eval)

The gap between the two columns is the failure mode. The LoRA is fine; the SFT data distribution doesn't cover what the eval feeds the model. Next step: regenerate SFT data on top of cooperbench's actual task template (or distill from Gemini-Flash rollouts under TITO). Either makes the train / inference prompt distributions match by construction.

Fix-#2 attempt — 2026-05-11 (later)

What I changed under the “fix-#2” umbrella

The original fix-#2 framing (“regenerate SFT data so its rendered chat-template output matches inference”) turned out to be wrong: I verified by running one row of combined.jsonl through AutoTokenizer.apply_chat_template and the assistant <tool_call> XML survives the template verbatim. So I went hunting for the real divergence and made three serve / adapter changes along the way:

1. Removed --reasoning-parser qwen3 from the vLLM serve. The parser captures everything before </think> into the response’s reasoning field. The trained model never emits </think> (training data has no thinking tags), so the entire output — including the <tool_call> XML — was being routed into reasoning while content came back null and tool_calls=[]. Confirmed by reading the raw response object on the first attempt’s serve.
2. Removed chat_template_kwargs.enable_thinking=false from the agent config. That flag injects a literal <think>\n\n</think>\n\n after <|im_start|>assistant\n, which is OOD vs training (the trained assistant turns start with prose directly). With the flag set, the model fell back to echoing the markdown bash example from the system prompt — the “markdown bash blocks” output observed in the v2 smoke test. Default mode (no flag) renders the same prompt tail as training and the model emits the correct XML.
3. Bridged the model output to the agent loop in LitellmModel. Two issues stacked on top of each other:
   • cooperbench’s mini_swe_agent agent loop uses action_regex = r"```bash\\s*\\n(.*?)\\n```" on response.content; it does not consult tool_calls. The model emits the XML form, which doesn’t match.
   • The same agent loop’s send_message interceptor matches send_message agent "msg" and send_message agent 'msg'; the training data uses heredoc form send_message agent <<'MSG'\\n…\\nMSG, which doesn’t match either.
   Added two helpers (_extract_tool_calls_from_content, _rewrite_xml_tool_calls_to_markdown + _heredoc_to_quoted_send_message) so that with disable_tools=True, LitellmModel.query rewrites the assistant content from the XML+heredoc form into the markdown+quoted form the agent loop already speaks. Verified end-to-end on training-style prompts: 10/10 produce parseable actions.

The actual failure

On the two real held-out coop tasks the model returns no <tool_call> blocks at all — instead it emits a short Chinese-then-English narration followed by literal text like “[Makes bash tool call with {"command": "ls -la"} as arguments]”. The system prompt is identical to training (same templated phrasing) but the task message is the full cooperbench PR description (50+ lines, with embedded markdown code blocks, “Solution” sections, type annotations). The synthetic training data uses 3–5-line tasks. The LoRA never generalized from short templated tasks to long PR-style tasks — it goes off-distribution and stops producing tool calls.

This is consistent with the val/loss curve: 2.36 → 0.098 over 424 steps. It memorized the templated distribution very well. It did not learn an invariant “respond with <tool_call> + send_message regardless of task shape.”

Why TITO would have caught this differently

Under TITO we’d have captured the exact token stream from a real Gemini-Flash plan-first rollout against a cooperbench task — so the training prompt distribution is the inference prompt distribution by construction. The current synthetic-data path optimizes a different distribution than the eval is sampling from. That’s the single biggest lesson from this attempt.

Recommended next steps

1. Regenerate SFT data wrapping synthetic plan-first content inside cooperbench’s actual task-prompt template. Pull a few real cooperbench tasks, replay the exact system + user prompts the eval will send, and only inject the plan-first assistant trajectory on top. Same training cost, matching distribution.
2. Tier-5 distillation from Gemini-Flash plan-first rollouts on cooperbench tasks. Real rollouts — no synthetic gap. This is the canonical TITO path. Cost goes up (rollout time + Gemini API) but the format-and-distribution problem disappears.
3. Cheaper experiment first: re-run held-out eval against the same synthetic task templates (i.e., feed the model the training-style task message instead of the real cooperbench PR). If metrics jump to passing, that confirms the diagnosis with zero retraining cost.

Artifact pointers for this attempt: serve ap-ED8tmOxyyYDBlMGvhdE7in (redeployed without --reasoning-parser); agent config coopertrain/configs/coop_plan_first.yaml (no enable_thinking flag); adapter helpers in coopertrain/agents/mini_swe_agent/models/litellm_model.py; rollouts logs/plan-first-eval-v3/; metrics report/2026-05-10-plan-first-cooperbench-results/metrics-v3.json.

Re-run with fixes — 2026-05-11

What changed in the re-run

• Inference format fix: added disable_tools flag to LitellmModelConfig (commit 1912bdb) so the Qwen3 chat template stops injecting tool descriptions at inference — bringing the inference prompt back in line with training.
• Scaled training: --n-per-task 50 (was 10), total_epochs: 4 (was 2), lr: 2e-5 (was 1e-5). 424 SFT steps / 4.9M loss tokens, vs 42 steps / 200k tokens in the first attempt. Val/loss dropped 2.36→0.098.
• Runbook fix discovered during pickup: the original Modal training run was launched without --detach, so it died at step 82 when the user’s monitor disconnected. Re-launched with modal run --detach and ran to completion in ~80 min.

What the re-run revealed

The training itself clearly succeeded this time — val/loss dropped two orders of magnitude (2.36→0.61→0.16→0.10) across the 4 epochs, and the LoRA output is visibly different from the base model. The smoke test (§4 of the runbook) passed 2/3 conditions:

IDENTICAL: False                # ✓ LoRA learned something
LoRA HAS <tool_call>: False     # ✗ but not the XML format
LoRA HAS send_message: True     # ✓ learned to call the right tool

Sample LoRA output (greedy decode on a training prompt’s system+user prefix):

```bash
send_message --wait agent2 <<'MSG'
What files are you planning to edit?
MSG
```

That is the correct intent (a send_message to the partner agent), but mini-swe-agent dispatches off the response’s tool_calls field, which vLLM only populates when it sees literal <tool_call><function=bash>…</function></tool_call> XML in the generated content. Markdown bash blocks do not parse. Every turn in both held-out pairs hit “No tool calls found in the response” for all 100 steps before LimitsExceeded.

Diagnosis — why the model learned the wrong format

The training data combined.jsonl contains <tool_call><function=bash>…</function></tool_call> in the assistant message text. But the Qwen3 chat template at SFT time appears to have rewritten that content during rendering — either via tool-call extraction into a structured field, or via the messages_key=messages path in verl's MultiTurnSFTDataset — so the tokenized assistant turn that the model actually trained against was the rendered form, not the literal XML. The rendered form turned out to be markdown bash, so that’s what the model learned to emit. At inference, vLLM’s qwen3_coder tool parser only knows how to parse XML back out, so the loop never closes.

Recommended next steps

1. Cheapest fix: change mini-swe-agent’s adapter to also accept markdown bash blocks (not just tool_calls). The action content is unambiguous — one bash block per turn. This makes the current LoRA usable as-is, no retraining needed.
2. Format-correctness fix: regenerate SFT data with the actual rendered chat-template output as the assistant content, so training data matches what the model will produce at inference. This requires running each combined.jsonl entry through the tokenizer’s chat template once, capturing the rendered assistant turns, and writing those back.
3. Tier 5 (distillation): abandon the templated-data approach and distill behavior from Gemini-Flash plan-first rollouts. The format-rendering problem disappears because Gemini’s real rollouts emit whatever format mini-swe-agent already accepts.

Artifact pointers for the re-run: final checkpoint plan-first-checkpoints:/plan-first/global_step_424; merged PEFT adapter plan-first-checkpoints:/plan-first/peft/lora_adapter (132 MB); metrics JSON report/2026-05-10-plan-first-cooperbench-results/metrics-v2.json; trajectories logs/plan-first-eval/coop/{pallets_click_task,dspy_task}/.../f1_f2/; training app ap-ry4fWdYJbCTYVkWyz2N6ue (stopped 2026-05-11 01:44:51 UTC).

First attempt — 2026-05-10

1. TL;DR

2. Setup — what we actually ran

3. Metrics

Same three metrics defined in the plan §6. Pass thresholds from the plan: plan-first ≥ 70%, plan content ≥ 70%, follow-through ≥ 60%.

Per the eval script’s reason field, both pairs failed with "missing assistant turns": 0 messages with role=assistant in the saved trajectory, out of 101–103 total messages per agent. The agent’s 100 LLM calls all returned empty tool_calls, triggering the FormatError retry loop until step_limit=100 fired and the run ended with status=LimitsExceeded.

{"summary":{"n_pairs":2,"plan_first_rate":0.0,"plan_content_rate":0.0,"follow_through_rate":0.0,
            "n_plan_first":0,"n_plan_content":0,"n_follow_through_agents":0},
 "per_pair":[
   {"pair_id":"dspy_task/8394/f1_f2","plan_first":false,"reason":"missing assistant turns"},
   {"pair_id":"pallets_click_task/2068/f1_f2","plan_first":false,"reason":"missing assistant turns"}
 ]}

4. Diagnosis — why 0/0/0

The plan’s §6.1 decomposition says “plan-first low ⇒ model didn’t learn the temporal pattern.” That’s the right ballpark, but the actual failure is sharper: the model produces no tool calls at all, plan-first or otherwise. Walking the pipeline back to find where it broke:

The actual root cause

Stage 6 narrows to one of two not-mutually-exclusive hypotheses:

1. Training↔inference format mismatch (most likely). Training data assistant turns look like:

   <assistant> "Before I start editing, let me coordinate with agent2..."
               <tool_call><function=bash><parameter=command>
               send_message agent2 <<'MSG' ... MSG
               </parameter></function></tool_call>
   

   The system prompt during training is the bare COOP_SYSTEM_PROMPT — no tools field on any message. At inference, mini-swe-agent passes tools=[BASH_TOOL] + tool_choice="auto" to litellm, which forwards them to vLLM’s OpenAI-compatible endpoint. The Qwen3 chat template injects tool descriptions into the system message when tools are provided. So the model sees a different system prompt at inference than it ever saw during training. The LoRA delta (rank 32, 42 steps, weak by construction) is not large enough to override the base model’s “tools ⇒ ask clarifying questions, no tool calls” prior on this OOD prompt.
2. LoRA capacity / training budget too small. LoRA rank 32 with 42 steps on 342 trajectories gives ~24 mini-batches/epoch × 2 epochs ≈ 48 forward+backward passes. For a behavior as specific as “always emit send_message in turn 1,” this may be under the threshold needed to overpower base behavior, regardless of prompt format. The non-zero lora_B values show the adapter did learn something; just not enough to dominate at decoding.

The first explanation is the load-bearing one: a manual completion-API probe (raw /v1/completions with no tools, no chat template) on the prefix “Before I start editing, let me coordinate with agent2 so we” still returns identical text to base. So even without the chat-template tools injection, the LoRA doesn’t move the next-token distribution noticeably on this prefix. That points to (2) being non-trivial too — 42 SFT steps is genuinely low.

5. What the pipeline did validate

The plan-doc §1 listed five things this experiment should validate. Five (data gen + parquet + training + serve + eval) are validated by file-level passes. The sixth (multi-turn behavior emergence) is what failed:

6. Recommended next steps

1. Fix the format mismatch first. Either:
   • (A) Have the data generator include the Qwen3 chat-template’s tool-description system prefix in training, so the prompt-distribution at training matches what the live agent sees. Cheapest fix; doesn’t require re-training infra changes.
   • (B) Have the agent config NOT pass tools/tool_choice to vLLM, relying entirely on prompt-format conventions (the inline <tool_call> XML in content). Requires mini-swe-agent adapter changes or a custom litellm_model override; more surgical but more code.
2. Scale training budget. 342 trajectories × 2 epochs is not enough to shift LoRA behavior decisively. Recommend: 10× the data (~3.5k trajectories) and/or 4–5 epochs. Same compute envelope, ~$15 instead of ~$3.
3. Add a sanity probe before scaling rollouts. A 30-second smoke that hits the deployed endpoint with one training-style prompt and asserts “plan-first output ≠ base output” would catch this regression class without burning 100 LLM calls per held-out task.
4. Don’t merge PR #30 yet. The infrastructure (data gen, training driver, merge script, serve, eval) is all reusable for the next attempt; the report itself documents the failure path. Both should land. But the experiment hasn’t demonstrated what tier 4 was supposed to demonstrate (multi-turn behavior emerging end-to-end), so the “tier 4 complete” claim should remain open until a re-run with one of the fixes in (1) actually moves the needle on at least one of the three metrics.

7. Files landed on this branch

Appendix: original experiment plan — 2026-05-10

turn 1  agent_1.send_message  → agent_2: "Plan: I'll handle <file_a>, you handle <file_b>"
turn 2  agent_2.send_message  → agent_1: "Acknowledged — I'll do <file_b>"
turn 3  agent_1.bash          → cd repo && cat <file_a>       (real path)
turn 4  agent_1.bash          → sed -i ... <file_a>          (real plausible edit)
turn 3' agent_2.bash          → cd repo && cat <file_b>       (real path)
turn 4' agent_2.bash          → sed -i ... <file_b>          (real plausible edit)
...
turn N  agent_*.bash          → pytest                       (or git diff > submission)