170 lines
10 KiB
Markdown
170 lines
10 KiB
Markdown
# End-to-End Audit: Delegation Router + Research Versioning — Gap Analysis
|
|
|
|
**Date:** 2026-06-24
|
|
**Auditor:** Pi
|
|
**Scope:** Both design systems delivered this session
|
|
|
|
---
|
|
|
|
## Audit Method: Simulated End-to-End Trace
|
|
|
|
I traced 5 user flows across both systems, examining every step from prompt submission to output persistence. Verified against the actual files on disk (not just the design docs).
|
|
|
|
---
|
|
|
|
## Findings
|
|
|
|
### Finding 1 (HIGH) — UserPromptSubmit hook output injection is unverified
|
|
|
|
**System:** Delegation Router — Claude-side trigger
|
|
**Flow:** Task-shaped prompt → dispatch-trigger.sh → [dispatch-check] marker
|
|
**Gap:** The dispatch-trigger.sh script prints its output to stdout, expecting it to be injected into Claude's conversation context. But the actual behavior of Claude Code's `UserPromptSubmit` hook is **not documented in this setup**. If the hook's stdout goes to a log, stderr, or is silently discarded, the `[dispatch-check]` marker never reaches the model and the trigger never fires.
|
|
|
|
**Why it matters:** This is the entire trigger mechanism. If the hook doesn't inject stdout into context, the system has no trigger at all.
|
|
|
|
**Fix needed before building:** Claude must verify how `UserPromptSubmit` hooks work in Claude Code — specifically whether command stdout is injected into the conversation. Test with a simple echo hook before building the real one.
|
|
|
|
---
|
|
|
|
### Finding 2 (HIGH) — dispatch-trigger.sh $1 assumption unverified
|
|
|
|
**System:** Delegation Router — Claude-side trigger
|
|
**Flow:** UserPromptSubmit hook → bash script → $1
|
|
**Gap:** The script assumes the prompt is passed as the first command argument (`PROMPT="$1"`). If Claude Code's`UserPromptSubmit` hook passes the prompt via environment variable, stdin, or not at all, `$1` is empty and the script always exits without firing.
|
|
|
|
**Why it matters:** Same as Finding 1 — the trigger silently never works.
|
|
|
|
**Fix:** Verify hook argument API as part of Finding 1.
|
|
|
|
---
|
|
|
|
### Finding 3 (MEDIUM) — Pi extension hooks tool_call, not tool_result (auto-stage never fires)
|
|
|
|
**System:** Research Versioning — Layer 3 (Pi extension)
|
|
**Flow:** Pi writes research file → extension auto-stages
|
|
**Gap:** The `research-versioning.ts` extension hooks `tool_call`, which fires **before** the tool executes. The file doesn't exist yet, so `git add` fails silently (caught by try/catch). The auto-stage never actually works for Pi.
|
|
|
|
**Why it matters:** Layer 3 is completely non-functional for Pi mid-session writes. Only the Stop hook (Layer 1) catches Pi's research artifacts, and only at session end.
|
|
|
|
**Fix:** Change from `tool_call` to `tool_result` event, which fires after the file has been written.
|
|
|
|
---
|
|
|
|
### Finding 4 (MEDIUM) — Dispatch thresholds need merge, not append (conflicting guidance)
|
|
|
|
**System:** Delegation Router — dispatch skill edits
|
|
**Flow:** Model reads dispatch skill → two trigger mechanisms
|
|
**Gap:** The dispatch skill currently has an `Auto-invoke` section that says "Auto-invoke (even without being typed) when..." — which relies on the model's own judgment. The redesign replaces this with "When you see the `[dispatch-check]` marker in context..." — which relies on an external hook. The design says "append Thresholds section" but the old and new auto-invoke rules coexist and can conflict:
|
|
|
|
- Old rule: model decides when to auto-invoke (broad, implicit)
|
|
- New rule: hook decides when to flag, model only acts on explicit marker (narrow, explicit)
|
|
|
|
If both sections survive, the model gets mixed signals. The old section must be fully replaced, not just supplemented.
|
|
|
|
**Why it matters:** Model confusion = inconsistent delegation behavior.
|
|
|
|
**Fix:** When Claude edits dispatch SKILL.md, the old `Auto-invoke` bullet must be completely deleted and replaced by the new `Auto-invoke (dispatch check fired by UserPromptSubmit hook)` section. Do NOT leave both.
|
|
|
|
---
|
|
|
|
### Finding 5 (LOW) — signal/docs/ is not covered by research auto-versioning
|
|
|
|
**System:** Research Versioning — all three layers
|
|
**Flow:** Pi writes design doc to signal/docs/ → auto-versioning
|
|
**Gap:** The research-versioning extension watches `research/`, `docs/compliance/`, and `content/` — but NOT `signal/docs/`. Design documents written by Pi or Claude to `signal/docs/` are not auto-staged by Layer 3, not auto-committed by Layer 1 (Stop hook), and only covered by the manual discipline of Layer 2.
|
|
|
|
The `design-review-persist` skill tells Pi to manually commit — but Claude has no equivalent prompt to do the same.
|
|
|
|
**Why it matters:** signal/docs/ is the canonical home for design documents. If neither agent auto-versions it, design docs drift.
|
|
|
|
**Fix:** Add `signal/docs/` to Layer 1 stop hook directory scope and Layer 3 extension watched paths. The exclusion patterns already prevent raw data and large files from being committed.
|
|
|
|
---
|
|
|
|
### Finding 6 (LOW) — Mid-session research commits stay local until next push cycle
|
|
|
|
**System:** Research Versioning — Layer 1 (Stop hook)
|
|
**Flow:** Agent commits mid-session → Stop hook sees clean tree → exits without pushing
|
|
**Gap:** When an agent follows the research-versioning discipline and commits mid-session, the working tree is clean. The Stop hook (research-autopush.sh) exits immediately because `git diff --quiet` shows no changes. Those commits are never pushed until the NEXT session triggers a research change that creates a new commit-and-push cycle.
|
|
|
|
**Why it matters:** If the machine crashes or the session is interrupted between sessions, mid-session commits exist only in the local repo and are lost.
|
|
|
|
**Fix (optional):** The Stop hook could run `git push` even when it doesn't commit. But this adds latency to every session close. Risk is low — acceptable for now.
|
|
|
|
---
|
|
|
|
### Finding 7 (LOW) — Chief-of-staff sequential dependency handling is untested
|
|
|
|
**System:** Delegation Router — dispatch thresholds
|
|
**Flow:** Multi-domain sequential task → chief-of-staft orchestration
|
|
**Gap:** The thresholds route "sequential handoffs between specialists" to chief-of-staff. But the actual capability of the skyremote `chief-of-staff-kit` plugin to handle dependencies (specialist B depends on specialist A's output) is unknown. The plugin may only support parallel dispatch, not sequential workflows.
|
|
|
|
**Why it matters:** If chief-of-staff can't sequence, multi-domain sequential tasks break.
|
|
|
|
**Fix:** Discovery task for Claude: test chief-of-staff with a sequential brief. If it handles ordering, document it. If not, add a note that sequential multi-domain tasks need manual chaining.
|
|
|
|
---
|
|
|
|
### Finding 8 (LOW) — Pi model behavior with injected messages is untested
|
|
|
|
**System:** Delegation Router — Pi-side trigger
|
|
**Flow:** Pi extension injects [dispatch-check-pi] → model may or may not respond
|
|
**Gap:** The dispatch-trigger.ts extension injects a customType message with `display: true`. Whether the model (deepseek-v4-flash) treats this as an actionable instruction or ignores it as session metadata is unknown without testing.
|
|
|
|
**Why it matters:** The Pi-side trigger may silently have zero effect.
|
|
|
|
**Fix:** Testing step: submit a task-shaped prompt to Pi and verify the model responds to the dispatch-check. If ignored, switch from `message` injection to `systemPrompt` modification.
|
|
|
|
---
|
|
|
|
### Finding 9 (LOW/KNOWN) — Skills directory is not git-versioned
|
|
|
|
**System:** Both — shared infrastructure
|
|
**Flow:** Any skill file in ~/.claude/skills/
|
|
**Gap:** All deployed skills (dispatch edits, delegation-discipline, design-review-persist, research-versioning) live in `~/.claude/skills/` which is NOT under version control. A machine failure loses all skill work.
|
|
|
|
**Status:** TRACKED. This is phase 2 "skills-as-git-repo" from the workflow-consolidation plan.
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
| # | Severity | Finding | System | Fix needed |
|
|
|---|----------|---------|--------|------------|
|
|
| 1 | **HIGH** | UserPromptSubmit hook output injection unverified | Delegation Router (Claude) | Verify hook behavior before building |
|
|
| 2 | **HIGH** | dispatch-trigger.sh $1 assumption unverified | Delegation Router (Claude) | Verify hook argument API before building |
|
|
| 3 | **MEDIUM** | Pi extension hooks tool_call not tool_result | Research Versioning (Pi) | Change event to `tool_result` |
|
|
| 4 | **MEDIUM** | Dispatch thresholds need merge, not append | Delegation Router (dispatch) | Delete old Auto-invoke section, don't append |
|
|
| 5 | **LOW** | signal/docs/ not covered by research auto-versioning | Research Versioning (all layers) | Add docs/ to watched directories |
|
|
| 6 | **LOW** | Mid-session commits stay local until next push | Research Versioning (Stop hook) | Optional: push even on clean tree |
|
|
| 7 | **LOW** | Chief-of-staff sequential handling untested | Delegation Router (thresholds) | Discovery test by Claude |
|
|
| 8 | **LOW** | Pi model behavior with injected messages untested | Delegation Router (Pi trigger) | Testing step |
|
|
| 9 | LOW/KNOWN | Skills not git-versioned | Both | Phase 2: skills-as-git-repo |
|
|
|
|
## Recommended Next Steps
|
|
|
|
**Critical (blockers to resolve before building):**
|
|
|
|
1. Claude: Verify UserPromptSubmit hook behavior — does stdout inject into context? Is `$1` the prompt?
|
|
2. Pi: Fix research-versioning extension from `tool_call` to `tool_result`
|
|
|
|
**Design refinements (fix in design docs before handoff):**
|
|
3. Clarify dispatch threshold merge instructions (replace old section, not append)
|
|
4. Add `signal/docs/` to research versioning scope (all 3 layers)
|
|
|
|
**Discovery tasks for Claude:**
|
|
5. Test chief-of-staff sequential dependency handling
|
|
6. Document the hook argument API after verification
|
|
|
|
---
|
|
|
|
## Files Affected by Fixes
|
|
|
|
| Finding | File to Change | Change Type |
|
|
|---------|---------------|-------------|
|
|
| 3 | `~/.pi/agent/extensions/research-versioning.ts` | Edit: tool_call → tool_result |
|
|
| 4 | `signal/docs/delegation-router-design-2026-06-24.md` | Edit: clarify merge vs append |
|
|
| 5 | `signal/docs/research-versioning-design-2026-06-24.md` | Edit: add docs/ to scope |
|
|
| 5 | `~/.pi/agent/extensions/research-versioning.ts` | Edit: add docs/ to RESEARCH_PATTERNS |
|
|
| 5 | `signal/docs/research-versioning-design-2026-06-24.md` | Edit: add docs/ to Stop hook |
|
|
| 1,2 | `signal/docs/delegation-router-design-2026-06-24.md` | Add: verification step before build
|