10 KiB
End-to-End Audit: Delegation Router + Research Versioning — Gap Analysis
Date: 2026-06-24 Auditor: Pi Scope: Both design systems delivered this session
Audit Method: Simulated End-to-End Trace
I traced 5 user flows across both systems, examining every step from prompt submission to output persistence. Verified against the actual files on disk (not just the design docs).
Findings
Finding 1 (HIGH) — UserPromptSubmit hook output injection is unverified
System: Delegation Router — Claude-side trigger
Flow: Task-shaped prompt → dispatch-trigger.sh → [dispatch-check] marker
Gap: The dispatch-trigger.sh script prints its output to stdout, expecting it to be injected into Claude's conversation context. But the actual behavior of Claude Code's UserPromptSubmit hook is not documented in this setup. If the hook's stdout goes to a log, stderr, or is silently discarded, the [dispatch-check] marker never reaches the model and the trigger never fires.
Why it matters: This is the entire trigger mechanism. If the hook doesn't inject stdout into context, the system has no trigger at all.
Fix needed before building: Claude must verify how UserPromptSubmit hooks work in Claude Code — specifically whether command stdout is injected into the conversation. Test with a simple echo hook before building the real one.
Finding 2 (HIGH) — dispatch-trigger.sh $1 assumption unverified
System: Delegation Router — Claude-side trigger
Flow: UserPromptSubmit hook → bash script → $1
Gap: The script assumes the prompt is passed as the first command argument (PROMPT="$1"). If Claude Code'sUserPromptSubmit hook passes the prompt via environment variable, stdin, or not at all, $1 is empty and the script always exits without firing.
Why it matters: Same as Finding 1 — the trigger silently never works.
Fix: Verify hook argument API as part of Finding 1.
Finding 3 (MEDIUM) — Pi extension hooks tool_call, not tool_result (auto-stage never fires)
System: Research Versioning — Layer 3 (Pi extension)
Flow: Pi writes research file → extension auto-stages
Gap: The research-versioning.ts extension hooks tool_call, which fires before the tool executes. The file doesn't exist yet, so git add fails silently (caught by try/catch). The auto-stage never actually works for Pi.
Why it matters: Layer 3 is completely non-functional for Pi mid-session writes. Only the Stop hook (Layer 1) catches Pi's research artifacts, and only at session end.
Fix: Change from tool_call to tool_result event, which fires after the file has been written.
Finding 4 (MEDIUM) — Dispatch thresholds need merge, not append (conflicting guidance)
System: Delegation Router — dispatch skill edits
Flow: Model reads dispatch skill → two trigger mechanisms
Gap: The dispatch skill currently has an Auto-invoke section that says "Auto-invoke (even without being typed) when..." — which relies on the model's own judgment. The redesign replaces this with "When you see the [dispatch-check] marker in context..." — which relies on an external hook. The design says "append Thresholds section" but the old and new auto-invoke rules coexist and can conflict:
- Old rule: model decides when to auto-invoke (broad, implicit)
- New rule: hook decides when to flag, model only acts on explicit marker (narrow, explicit)
If both sections survive, the model gets mixed signals. The old section must be fully replaced, not just supplemented.
Why it matters: Model confusion = inconsistent delegation behavior.
Fix: When Claude edits dispatch SKILL.md, the old Auto-invoke bullet must be completely deleted and replaced by the new Auto-invoke (dispatch check fired by UserPromptSubmit hook) section. Do NOT leave both.
Finding 5 (LOW) — signal/docs/ is not covered by research auto-versioning
System: Research Versioning — all three layers
Flow: Pi writes design doc to signal/docs/ → auto-versioning
Gap: The research-versioning extension watches research/, docs/compliance/, and content/ — but NOT signal/docs/. Design documents written by Pi or Claude to signal/docs/ are not auto-staged by Layer 3, not auto-committed by Layer 1 (Stop hook), and only covered by the manual discipline of Layer 2.
The design-review-persist skill tells Pi to manually commit — but Claude has no equivalent prompt to do the same.
Why it matters: signal/docs/ is the canonical home for design documents. If neither agent auto-versions it, design docs drift.
Fix: Add signal/docs/ to Layer 1 stop hook directory scope and Layer 3 extension watched paths. The exclusion patterns already prevent raw data and large files from being committed.
Finding 6 (LOW) — Mid-session research commits stay local until next push cycle
System: Research Versioning — Layer 1 (Stop hook)
Flow: Agent commits mid-session → Stop hook sees clean tree → exits without pushing
Gap: When an agent follows the research-versioning discipline and commits mid-session, the working tree is clean. The Stop hook (research-autopush.sh) exits immediately because git diff --quiet shows no changes. Those commits are never pushed until the NEXT session triggers a research change that creates a new commit-and-push cycle.
Why it matters: If the machine crashes or the session is interrupted between sessions, mid-session commits exist only in the local repo and are lost.
Fix (optional): The Stop hook could run git push even when it doesn't commit. But this adds latency to every session close. Risk is low — acceptable for now.
Finding 7 (LOW) — Chief-of-staff sequential dependency handling is untested
System: Delegation Router — dispatch thresholds
Flow: Multi-domain sequential task → chief-of-staft orchestration
Gap: The thresholds route "sequential handoffs between specialists" to chief-of-staff. But the actual capability of the skyremote chief-of-staff-kit plugin to handle dependencies (specialist B depends on specialist A's output) is unknown. The plugin may only support parallel dispatch, not sequential workflows.
Why it matters: If chief-of-staff can't sequence, multi-domain sequential tasks break.
Fix: Discovery task for Claude: test chief-of-staff with a sequential brief. If it handles ordering, document it. If not, add a note that sequential multi-domain tasks need manual chaining.
Finding 8 (LOW) — Pi model behavior with injected messages is untested
System: Delegation Router — Pi-side trigger
Flow: Pi extension injects [dispatch-check-pi] → model may or may not respond
Gap: The dispatch-trigger.ts extension injects a customType message with display: true. Whether the model (deepseek-v4-flash) treats this as an actionable instruction or ignores it as session metadata is unknown without testing.
Why it matters: The Pi-side trigger may silently have zero effect.
Fix: Testing step: submit a task-shaped prompt to Pi and verify the model responds to the dispatch-check. If ignored, switch from message injection to systemPrompt modification.
Finding 9 (LOW/KNOWN) — Skills directory is not git-versioned
System: Both — shared infrastructure
Flow: Any skill file in ~/.claude/skills/
Gap: All deployed skills (dispatch edits, delegation-discipline, design-review-persist, research-versioning) live in ~/.claude/skills/ which is NOT under version control. A machine failure loses all skill work.
Status: TRACKED. This is phase 2 "skills-as-git-repo" from the workflow-consolidation plan.
Summary
| # | Severity | Finding | System | Fix needed |
|---|---|---|---|---|
| 1 | HIGH | UserPromptSubmit hook output injection unverified | Delegation Router (Claude) | Verify hook behavior before building |
| 2 | HIGH | dispatch-trigger.sh $1 assumption unverified | Delegation Router (Claude) | Verify hook argument API before building |
| 3 | MEDIUM | Pi extension hooks tool_call not tool_result | Research Versioning (Pi) | Change event to tool_result |
| 4 | MEDIUM | Dispatch thresholds need merge, not append | Delegation Router (dispatch) | Delete old Auto-invoke section, don't append |
| 5 | LOW | signal/docs/ not covered by research auto-versioning | Research Versioning (all layers) | Add docs/ to watched directories |
| 6 | LOW | Mid-session commits stay local until next push | Research Versioning (Stop hook) | Optional: push even on clean tree |
| 7 | LOW | Chief-of-staff sequential handling untested | Delegation Router (thresholds) | Discovery test by Claude |
| 8 | LOW | Pi model behavior with injected messages untested | Delegation Router (Pi trigger) | Testing step |
| 9 | LOW/KNOWN | Skills not git-versioned | Both | Phase 2: skills-as-git-repo |
Recommended Next Steps
Critical (blockers to resolve before building):
- Claude: Verify UserPromptSubmit hook behavior — does stdout inject into context? Is
$1the prompt? - Pi: Fix research-versioning extension from
tool_calltotool_result
Design refinements (fix in design docs before handoff):
3. Clarify dispatch threshold merge instructions (replace old section, not append)
4. Add signal/docs/ to research versioning scope (all 3 layers)
Discovery tasks for Claude: 5. Test chief-of-staff sequential dependency handling 6. Document the hook argument API after verification
Files Affected by Fixes
| Finding | File to Change | Change Type |
|---|---|---|
| 3 | ~/.pi/agent/extensions/research-versioning.ts |
Edit: tool_call → tool_result |
| 4 | signal/docs/delegation-router-design-2026-06-24.md |
Edit: clarify merge vs append |
| 5 | signal/docs/research-versioning-design-2026-06-24.md |
Edit: add docs/ to scope |
| 5 | ~/.pi/agent/extensions/research-versioning.ts |
Edit: add docs/ to RESEARCH_PATTERNS |
| 5 | signal/docs/research-versioning-design-2026-06-24.md |
Edit: add docs/ to Stop hook |
| 1,2 | signal/docs/delegation-router-design-2026-06-24.md |
Add: verification step before build |