Signal/docs/pilot-e2e-validation-2026-06-21.md

Signal Pilot — End-to-End Validation

Date: 2026-06-21 Author: Claude Code (live validation + two parallel audit lanes) Purpose: An honest, evidence-backed answer to "how ready is the pilot, end to end" — not a checklist re-read. This document is the input for Pi to design a remediation plan; Claude Code then builds it. Method: Live exercise of the deployed stack (Vercel front-end, Railway API, Supabase, Clerk) + a backend-correctness audit + a compliance/production audit. Every claim below is backed by a live result or a file:line.

---

1. Verdict in one paragraph

The Signal engine is genuinely good: the deployed API scores 5,000 patients with zero skips, returns sensible worklist reasons and actions, keeps patient data to patient_id only, hashes identifiers, and the Confirm-Visit and Export paths work against the live database. But the pilot is not operable today and not yet safe for a real supplier's real data, for three independent reasons:

1. The live app is down. https://signal-ui-xi.vercel.app renders a blank screen for everyone because a half-finished Clerk production migration points the front-end at a custom domain that doesn't resolve.
2. The core gap logic has three correctness blockers that only surface on messy real-world data (Medicare Advantage misclassification, duplicate-patient collisions, and staff overrides that don't actually clear the gap). On clean demo data they're invisible; on a real Brightree export they produce visibly wrong answers.
3. The compliance and trust layer has open items, including a published privacy page that claims a BAA that does not exist, an audit log that is never actually written, and unexecuted LOI/NDA.

"100% pilot-ready (13/13)" is true only for the product build checklist, and even there one item is overstated. It is not true for live operability or compliance. Bottom line: with synthetic/de-identified data and the fixes below, a controlled Gaboro pilot is reachable. Real patient data is gated to the separate Production Readiness Whitepaper.

---

2. What was tested live (evidence)

Test	Result
Backend /health	200 {"status":"ok"}
Backend /health/db (Supabase)	200 {"status":"ok","org_count":1} — DB connected
Upload without auth	401 Authentication required — auth genuinely enforced in prod ✓
Upload sample-green.csv (authorized)	200 — 1 record, RESUPPLY_READY → "Clear to Ship", correct
Upload gaboro-stress-test.csv (authorized)	200 — 5,000 patients, 0 skipped, full flag distribution (994 lapsed, 1,161 resupply-ready, 1,994 prescriber-action, etc.)
PHI in API response	None — patient_id only, no name/DOB/SSN ✓
/api/export	200, text/csv, correct filename header, clean header + data rows ✓
/api/confirm-visit	200 — GAB-000001 moved RENEWAL_CRITICAL/"Escalate" (est.) → ACTIVE/"On Track" (confidence: confirmed), persisted to Supabase ✓
Front-end https://signal-ui-xi.vercel.app	200 HTTP but blank render, 8 console errors (see §3, P0-1)
/privacy page	200 (published, but see §5 for a false BAA claim)

Side effect to clean up: the uploads + confirm-visit wrote synthetic rows (PT-GREEN-1, GAB-*) into the pilot Supabase. This ties to open item C-7 (deletion procedure never tested).

---

3. P0 — Pilot cannot proceed until these are fixed

P0-1. The live app is a blank screen (broken Clerk production migration)

• Symptom: https://signal-ui-xi.vercel.app renders nothing. Console: 8× Failed to load Clerk JS … ERR_CONNECTION_CLOSED from https://clerk.sttilsolutions.com/npm/@clerk/clerk-js@6/....
• Root cause (verified by DNS): clerk.sttilsolutions.com is CNAME'd to Vercel (signal.sttilsolutions.com → cname.vercel-dns.com → 76.76.21.142), not to Clerk's frontend-api.clerk.services. The deployed front-end was rebuilt with a production Clerk key (pk_live_, custom domain clerk.sttilsolutions.com), but that domain never got the correct Clerk DNS. accounts.sttilsolutions.com is correctly pointed at Clerk — so exactly one of the required records is wrong, and it's the one that loads the auth library.
• Second, independent defect: the back-end still validates tokens against the development Clerk instance (CLERK_JWKS_URL = https://eternal-goblin-1.clerk.accounts.dev/...). Even with front-end DNS fixed, production Clerk tokens would fail backend validation. The two halves are on different Clerk instances.
• Fix paths (decision for Kisa, Clerk work owned by Pi):
  • A — Restore the working demo fast (recommended for an imminent Gaboro demo): repoint the deployed front-end to the dev Clerk key (pk_test_, eternal-goblin-1), matching the backend, and redeploy. Live demo works again in minutes. Production Clerk is then completed deliberately later.
  • B — Finish the production migration properly: fix clerk.sttilsolutions.com CNAME → frontend-api.clerk.services, add the remaining Clerk DNS records, switch the backend CLERK_JWKS_URL to the production instance, redeploy both. This is the Production Whitepaper path; do it as a project, not under call pressure.
• Acceptance: the public URL renders the sign-in screen, a test user logs in, and an authenticated upload round-trips through the UI.

P0-2. Medicare Advantage plans misclassify and drop the PA requirement (VERIFIED LIVE)

• Evidence (run live against coverage_calculator._normalize_payer):
  • WellCare Medicare → medicare → doc-state pa = "Not Required" ❌ (MA requires PA)
  • Aetna Medicare → medicare → pa = "Not Required" ❌
  • Humana Gold Plus HMO, UnitedHealthcare Dual Complete, Anthem MediBlue, Cigna HealthSpring, Kaiser Senior Advantage → commercial → PA required but pecos = "N/A" (right answer, wrong reason; MA needs PECOS)
• File: python-backend/core/coverage_calculator.py:144-148 — only the literal phrase "medicare advantage" is detected; real MA plan brand names are not.
• Why it's a blocker: Signal's entire job is to flag the documentation gap that causes a denial. For an MA patient mislabeled as Medicare FFS, Signal says "no PA needed" and would show Clear to Ship on a claim that will be denied. This is the product failing at its core promise.
• Acceptance: a CSV of common MA brand names classifies as medicare_advantage (PA required, PECOS required); when ambiguous, default to PA-required and surface "verify payer type" rather than silently clearing it.

P0-3. Duplicate patient_id collides — wrong order's docs attach to a row

• File: python-backend/api/main.py:429-442 — record_lookup = {r.patient_id: r for r in records} keeps only the last order per patient.
• Why it's a blocker: A real Brightree export routinely has multiple orders per patient (sensor + transmitter, refills). Every row for that patient then shows the SWO/PA/visit/device/order-number of a different order. In a demo this is a visibly wrong document status next to the wrong order number.
• Acceptance: key the pairing on the source order (e.g. index or (patient_id, shipment_date, device, order_number)), verified against a multi-order CSV.

P0-4. Staff doc-status override updates the badge but not the gap or the next step

• Files: python-backend/api/main.py:307-313, 337; core/doc_state_machine.py.
• Behavior: after a staff member marks SWO "On File," the badge turns green, but cascade and recommended_next_step_code are still computed from the un-overridden state, so Signal still says "obtain SWO from prescriber" and the export carries the contradiction.
• Why it's a blocker: the override workflow is a named pilot feature ("Signal's core persistent value"). It currently contradicts itself the moment a user does what Signal asked.
• Acceptance: overrides feed the recompute (cascade + next-step), not just the label; the status vocab (requested/on_file vs on file/pending/expired) is reconciled.

---

4. P1 — Should fix before a real supplier, not demo-blocking

#	Finding	File	Fix / acceptance
P1-1	Mapping override dropdown is a no-op (checklist counts it PASS)	signal-ui/src/CSVImport.jsx:117-123	Wire the override to actually remap, or correct the checklist claim
P1-2	Visit window inconsistent: 180 (rules) vs 183 (confirm-visit)	payer_rules.json, coverage_calculator.py:309 (hardcoded 180), main.py:584-596 (183)	One canonical window (183 = "6 months"), read from config everywhere; remove hardcoded 180
P1-3	Ambiguous date parsing defaults US %m/%d/%Y silently	api/normalizer.py:154-166	Detect day>12 to lock format per batch; warn when ambiguous
P1-4	No upload size cap; whole file read + decoded twice (OOM risk)	api/main.py:400-404, persistence.py:104	Reject > N MB (413) + row cap
P1-5	NO_RECENT_SHIPMENT priority (5) sorts above clean ACTIVE (0)	coverage_calculator.py:301	Route through _compute_priority or set below ACTIVE
P1-6	patient_id aliases include bare id/patient (could mis-map a name column to the crosswalk key)	api/normalizer.py:20-26	Drop overly generic aliases or flag low-confidence patient_id mapping
P1-7	Root vercel.json serves a static demo (signal-ui/demo/), not the live React app	vercel.json	Reconcile: the live deploy uses a different config; root file is stale/misleading and should match what actually ships

---

5. Compliance & trust — open before any supplier uploads

Genuinely solid (verified): PHI allowlist at the normalizer (name/DOB/SSN never mapped or stored), patient_id SHA-256 hashed before storage and logging, visit-date priority chain correct, backend auth enforced, empty/malformed CSV handled safely.

Must close (verified gaps):

• C-1. Privacy page claims a BAA that does not exist. privacy-policy.md:88 / Privacy.jsx assert an executed BAA with cloud providers. No BAA exists. This is a material misrepresentation a supplier (or regulator) could rely on. Fix the copy now. Also reconcile retention (Privacy.jsx says 6 years, privacy-policy.md says 7).
• C-2. The audit log is never written to the database. audit_logger.log_event() only persists when a db_conn is passed, and no caller in main.py ever passes one. Entries go to stdout only. A "6-year audit retention sink" cannot be claimed yet. Live calls also hardcode "demo_user"/"0.0.0.0".
• C-3. WORM is not effective. The RLS migration exists (supabase/migrations/20260607000001_audit_log_worm.sql) but the app connects with the Supabase service role, which bypasses RLS; and the table receives no rows (C-2). Docs also disagree on the table name (audit_log vs audit_events).
• C-4. Latent master-key leak path. signal-ui/src/lib/api.js:8 reads VITE_SIGNAL_API_KEY and sends it as X-API-Key, which the backend treats as a full-access service account bypassing Clerk. It's empty today (safe), but if anyone ever sets it in Vercel, the master key ships in public JS. Delete this code path.
• C-5. MFA not verified on any admin console (Supabase, Railway, Vercel, Clerk, GitHub/Forgejo). ~30 min, no engineering. Required even at zero-PHI — these consoles hold the keys.
• C-6. PHI-at-normalizer is implicit, not enforced. PHI columns are dropped (good) but an upload containing name/dob/ssn succeeds silently rather than being refused. Downgrade checklist PASS → PARTIAL; add an explicit reject-on-PHI guard.
• C-7. Deletion procedure never tested (policy written, never exercised) — and synthetic rows from this validation are now in the DB to delete.
• C-8. Tabletop incident exercise not run; production access scope not documented.
• C-9. LOI + NDA not executed with Gaboro, and FDA CDS exemption memo not written/attorney-reviewed.

---

6. Readiness by audience

• Internal demo on clean data: works the moment P0-1 is resolved (revert to dev Clerk). The engine is demo-quality.
• Controlled pilot, Gaboro, synthetic/de-identified data: reachable after P0-1..P0-4 + C-1, C-4, C-5, C-6, C-7, C-9, and confirming the Railway env vars are set. No BAA/host change required at zero-PHI.
• Real patient data: not on this stack. Gated entirely to the Production Readiness Whitepaper (BAA, host, Clerk prod, FIDO2 MFA, effective WORM, attorney review).

---

7. Remediation backlog (priority order — for Pi to design, Claude Code to build)

1. Restore the live app (P0-1, decision A or B from Kisa).
2. Fix MA payer classification (P0-2) — highest product-credibility risk.
3. Fix duplicate-patient record pairing (P0-3).
4. Make doc-status overrides recompute the gap (P0-4).
5. Correct the privacy page (C-1) and delete the VITE_SIGNAL_API_KEY path (C-4) — fast, high-trust-impact.
6. Wire the audit log to actually write + real user/IP (C-2), then effective WORM (C-3).
7. Confirm MFA on all consoles (C-5), test deletion (C-7), run tabletop (C-8), document access scope (C-8).
8. P1 correctness cleanups (override dropdown, 180/183, date parsing, size cap, priority, aliases, vercel.json).
9. Paperwork: execute LOI + NDA, write FDA CDS memo (C-9) — attorney-gated, parallel track.

---

Cross-reference: an internal quality review at docs/reviews/2026-06-11-quality-review.md independently flagged several of these (audit-log wiring, demo_user placeholders, override dropdown no-op, VITE_SIGNAL_API_KEY, implicit PHI guard). This validation confirms them live and adds the P0-1 live-outage and P0-2 MA-classification findings.