Signal/docs/pilot-e2e-validation-2026-06-21.md

# Signal Pilot — End-to-End Validation

**Date:** 2026-06-21
**Author:** Claude Code (live validation + two parallel audit lanes)
**Purpose:** An honest, evidence-backed answer to "how ready is the pilot, end to end" — not a checklist re-read. This document is the input for Pi to design a remediation plan; Claude Code then builds it.
**Method:** Live exercise of the deployed stack (Vercel front-end, Railway API, Supabase, Clerk) + a backend-correctness audit + a compliance/production audit. Every claim below is backed by a live result or a `file:line`.

---

## 1. Verdict in one paragraph

The Signal **engine** is genuinely good: the deployed API scores 5,000 patients with zero skips, returns sensible worklist reasons and actions, keeps patient data to `patient_id` only, hashes identifiers, and the Confirm-Visit and Export paths work against the live database. But the pilot is **not operable today and not yet safe for a real supplier's real data**, for three independent reasons:

1. **The live app is down.** `https://signal-ui-xi.vercel.app` renders a blank screen for everyone because a half-finished Clerk production migration points the front-end at a custom domain that doesn't resolve.
2. **The core gap logic has three correctness blockers** that only surface on messy real-world data (Medicare Advantage misclassification, duplicate-patient collisions, and staff overrides that don't actually clear the gap). On clean demo data they're invisible; on a real Brightree export they produce visibly wrong answers.
3. **The compliance and trust layer has open items**, including a published privacy page that claims a BAA that does not exist, an audit log that is never actually written, and unexecuted LOI/NDA.

"100% pilot-ready (13/13)" is true only for the **product build checklist**, and even there one item is overstated. It is not true for live operability or compliance. **Bottom line: with synthetic/de-identified data and the fixes below, a controlled Gaboro pilot is reachable. Real patient data is gated to the separate Production Readiness Whitepaper.**

---

## 2. What was tested live (evidence)

| Test | Result |
|---|---|
| Backend `/health` | `200 {"status":"ok"}` |
| Backend `/health/db` (Supabase) | `200 {"status":"ok","org_count":1}` — DB connected |
| Upload **without** auth | `401 Authentication required` — auth genuinely enforced in prod ✓ |
| Upload `sample-green.csv` (authorized) | `200` — 1 record, `RESUPPLY_READY → "Clear to Ship"`, correct |
| Upload `gaboro-stress-test.csv` (authorized) | `200` — **5,000 patients, 0 skipped**, full flag distribution (994 lapsed, 1,161 resupply-ready, 1,994 prescriber-action, etc.) |
| PHI in API response | **None** — `patient_id` only, no name/DOB/SSN ✓ |
| `/api/export` | `200`, `text/csv`, correct filename header, clean header + data rows ✓ |
| `/api/confirm-visit` | `200` — `GAB-000001` moved `RENEWAL_CRITICAL/"Escalate"` (est.) → `ACTIVE/"On Track"` (confidence: **confirmed**), persisted to Supabase ✓ |
| Front-end `https://signal-ui-xi.vercel.app` | `200` HTTP but **blank render, 8 console errors** (see §3, P0-1) |
| `/privacy` page | `200` (published, but see §5 for a false BAA claim) |

> Side effect to clean up: the uploads + confirm-visit wrote synthetic rows (`PT-GREEN-1`, `GAB-*`) into the pilot Supabase. This ties to open item C-7 (deletion procedure never tested).

---

## 3. P0 — Pilot cannot proceed until these are fixed

### P0-1. The live app is a blank screen (broken Clerk production migration)
- **Symptom:** `https://signal-ui-xi.vercel.app` renders nothing. Console: 8× `Failed to load Clerk JS … ERR_CONNECTION_CLOSED` from `https://clerk.sttilsolutions.com/npm/@clerk/clerk-js@6/...`.
- **Root cause (verified by DNS):** `clerk.sttilsolutions.com` is CNAME'd to **Vercel** (`signal.sttilsolutions.com → cname.vercel-dns.com → 76.76.21.142`), not to Clerk's `frontend-api.clerk.services`. The deployed front-end was rebuilt with a **production** Clerk key (`pk_live_`, custom domain `clerk.sttilsolutions.com`), but that domain never got the correct Clerk DNS. `accounts.sttilsolutions.com` **is** correctly pointed at Clerk — so exactly one of the required records is wrong, and it's the one that loads the auth library.
- **Second, independent defect:** the **back-end** still validates tokens against the **development** Clerk instance (`CLERK_JWKS_URL = https://eternal-goblin-1.clerk.accounts.dev/...`). Even with front-end DNS fixed, production Clerk tokens would fail backend validation. The two halves are on different Clerk instances.
- **Fix paths (decision for Kisa, Clerk work owned by Pi):**
  - **A — Restore the working demo fast (recommended for an imminent Gaboro demo):** repoint the deployed front-end to the **dev** Clerk key (`pk_test_`, `eternal-goblin-1`), matching the backend, and redeploy. Live demo works again in minutes. Production Clerk is then completed deliberately later.
  - **B — Finish the production migration properly:** fix `clerk.sttilsolutions.com` CNAME → `frontend-api.clerk.services`, add the remaining Clerk DNS records, switch the **backend** `CLERK_JWKS_URL` to the production instance, redeploy both. This is the Production Whitepaper path; do it as a project, not under call pressure.
- **Acceptance:** the public URL renders the sign-in screen, a test user logs in, and an authenticated upload round-trips through the UI.

### P0-2. Medicare Advantage plans misclassify and drop the PA requirement (VERIFIED LIVE)
- **Evidence (run live against `coverage_calculator._normalize_payer`):**
  - `WellCare Medicare` → `medicare` → doc-state `pa = "Not Required"` ❌ (MA requires PA)
  - `Aetna Medicare` → `medicare` → `pa = "Not Required"` ❌
  - `Humana Gold Plus HMO`, `UnitedHealthcare Dual Complete`, `Anthem MediBlue`, `Cigna HealthSpring`, `Kaiser Senior Advantage` → `commercial` → PA required but `pecos = "N/A"` (right answer, wrong reason; MA needs PECOS)
- **File:** `python-backend/core/coverage_calculator.py:144-148` — only the literal phrase "medicare advantage" is detected; real MA plan brand names are not.
- **Why it's a blocker:** Signal's entire job is to flag the documentation gap that causes a denial. For an MA patient mislabeled as Medicare FFS, Signal says "no PA needed" and would show **Clear to Ship on a claim that will be denied**. This is the product failing at its core promise.
- **Acceptance:** a CSV of common MA brand names classifies as `medicare_advantage` (PA required, PECOS required); when ambiguous, default to PA-required and surface "verify payer type" rather than silently clearing it.

### P0-3. Duplicate patient_id collides — wrong order's docs attach to a row
- **File:** `python-backend/api/main.py:429-442` — `record_lookup = {r.patient_id: r for r in records}` keeps only the **last** order per patient.
- **Why it's a blocker:** A real Brightree export routinely has multiple orders per patient (sensor + transmitter, refills). Every row for that patient then shows the SWO/PA/visit/device/order-number of a **different** order. In a demo this is a visibly wrong document status next to the wrong order number.
- **Acceptance:** key the pairing on the source order (e.g. index or `(patient_id, shipment_date, device, order_number)`), verified against a multi-order CSV.

### P0-4. Staff doc-status override updates the badge but not the gap or the next step
- **Files:** `python-backend/api/main.py:307-313, 337`; `core/doc_state_machine.py`.
- **Behavior:** after a staff member marks SWO "On File," the badge turns green, but `cascade` and `recommended_next_step_code` are still computed from the un-overridden state, so Signal still says "obtain SWO from prescriber" and the export carries the contradiction.
- **Why it's a blocker:** the override workflow is a named pilot feature ("Signal's core persistent value"). It currently contradicts itself the moment a user does what Signal asked.
- **Acceptance:** overrides feed the recompute (cascade + next-step), not just the label; the status vocab (`requested`/`on_file` vs `on file`/`pending`/`expired`) is reconciled.

---

## 4. P1 — Should fix before a real supplier, not demo-blocking

| # | Finding | File | Fix / acceptance |
|---|---|---|---|
| P1-1 | Mapping **override dropdown is a no-op** (checklist counts it PASS) | `signal-ui/src/CSVImport.jsx:117-123` | Wire the override to actually remap, or correct the checklist claim |
| P1-2 | Visit window inconsistent: 180 (rules) vs 183 (confirm-visit) | `payer_rules.json`, `coverage_calculator.py:309` (hardcoded 180), `main.py:584-596` (183) | One canonical window (183 = "6 months"), read from config everywhere; remove hardcoded 180 |
| P1-3 | Ambiguous date parsing defaults US `%m/%d/%Y` silently | `api/normalizer.py:154-166` | Detect day>12 to lock format per batch; warn when ambiguous |
| P1-4 | No upload size cap; whole file read + decoded twice (OOM risk) | `api/main.py:400-404`, `persistence.py:104` | Reject > N MB (413) + row cap |
| P1-5 | `NO_RECENT_SHIPMENT` priority (5) sorts above clean `ACTIVE` (0) | `coverage_calculator.py:301` | Route through `_compute_priority` or set below ACTIVE |
| P1-6 | `patient_id` aliases include bare `id`/`patient` (could mis-map a name column to the crosswalk key) | `api/normalizer.py:20-26` | Drop overly generic aliases or flag low-confidence patient_id mapping |
| P1-7 | Root `vercel.json` serves a **static demo** (`signal-ui/demo/`), not the live React app | `vercel.json` | Reconcile: the live deploy uses a different config; root file is stale/misleading and should match what actually ships |

---

## 5. Compliance & trust — open before any supplier uploads

**Genuinely solid (verified):** PHI allowlist at the normalizer (name/DOB/SSN never mapped or stored), `patient_id` SHA-256 hashed before storage and logging, visit-date priority chain correct, backend auth enforced, empty/malformed CSV handled safely.

**Must close (verified gaps):**
- **C-1. Privacy page claims a BAA that does not exist.** `privacy-policy.md:88` / `Privacy.jsx` assert an executed BAA with cloud providers. No BAA exists. This is a material misrepresentation a supplier (or regulator) could rely on. **Fix the copy now.** Also reconcile retention (Privacy.jsx says 6 years, privacy-policy.md says 7).
- **C-2. The audit log is never written to the database.** `audit_logger.log_event()` only persists when a `db_conn` is passed, and no caller in `main.py` ever passes one. Entries go to stdout only. A "6-year audit retention sink" cannot be claimed yet. Live calls also hardcode `"demo_user"`/`"0.0.0.0"`.
- **C-3. WORM is not effective.** The RLS migration exists (`supabase/migrations/20260607000001_audit_log_worm.sql`) but the app connects with the Supabase **service role**, which bypasses RLS; and the table receives no rows (C-2). Docs also disagree on the table name (`audit_log` vs `audit_events`).
- **C-4. Latent master-key leak path.** `signal-ui/src/lib/api.js:8` reads `VITE_SIGNAL_API_KEY` and sends it as `X-API-Key`, which the backend treats as a **full-access service account** bypassing Clerk. It's empty today (safe), but if anyone ever sets it in Vercel, the master key ships in public JS. **Delete this code path.**
- **C-5. MFA not verified** on any admin console (Supabase, Railway, Vercel, Clerk, GitHub/Forgejo). ~30 min, no engineering. Required even at zero-PHI — these consoles hold the keys.
- **C-6. PHI-at-normalizer is implicit, not enforced.** PHI columns are dropped (good) but an upload containing `name`/`dob`/`ssn` succeeds silently rather than being refused. Downgrade checklist PASS → PARTIAL; add an explicit reject-on-PHI guard.
- **C-7. Deletion procedure never tested** (policy written, never exercised) — and synthetic rows from this validation are now in the DB to delete.
- **C-8. Tabletop incident exercise** not run; **production access scope** not documented.
- **C-9. LOI + NDA not executed** with Gaboro, and **FDA CDS exemption memo** not written/attorney-reviewed.

---

## 6. Readiness by audience

- **Internal demo on clean data:** works the moment P0-1 is resolved (revert to dev Clerk). The engine is demo-quality.
- **Controlled pilot, Gaboro, synthetic/de-identified data:** reachable after P0-1..P0-4 + C-1, C-4, C-5, C-6, C-7, C-9, and confirming the Railway env vars are set. No BAA/host change required at zero-PHI.
- **Real patient data:** **not on this stack.** Gated entirely to the Production Readiness Whitepaper (BAA, host, Clerk prod, FIDO2 MFA, effective WORM, attorney review).

---

## 7. Remediation backlog (priority order — for Pi to design, Claude Code to build)

1. **Restore the live app** (P0-1, decision A or B from Kisa).
2. **Fix MA payer classification** (P0-2) — highest product-credibility risk.
3. **Fix duplicate-patient record pairing** (P0-3).
4. **Make doc-status overrides recompute the gap** (P0-4).
5. **Correct the privacy page** (C-1) and **delete the `VITE_SIGNAL_API_KEY` path** (C-4) — fast, high-trust-impact.
6. **Wire the audit log to actually write + real user/IP** (C-2), then **effective WORM** (C-3).
7. **Confirm MFA on all consoles** (C-5), **test deletion** (C-7), **run tabletop** (C-8), **document access scope** (C-8).
8. **P1 correctness cleanups** (override dropdown, 180/183, date parsing, size cap, priority, aliases, vercel.json).
9. **Paperwork:** execute LOI + NDA, write FDA CDS memo (C-9) — attorney-gated, parallel track.

---

*Cross-reference: an internal quality review at `docs/reviews/2026-06-11-quality-review.md` independently flagged several of these (audit-log wiring, demo_user placeholders, override dropdown no-op, VITE_SIGNAL_API_KEY, implicit PHI guard). This validation confirms them live and adds the P0-1 live-outage and P0-2 MA-classification findings.*