design+state: dedup decision locked (patient_id+device_type+DOS). Readiness model UNBLOCKED [Pi]

context/current-state.md

@@ -2,12 +2,13 @@ v:1
 
 # current-state
 
-ACTIVE: Pi designed the modular Convex-Signal spike from the brief (docs/convex-signal-spike-design-2026-06-24.md). Design covers module map with swappable seams, data-access interface (Repository portability hedge), Convex schema matching the readiness model, auth seam (Convex default, Clerk swappable), phased 40h spike plan (Phase 1 proves core readiness loop). Claude builds Phase 1 once repo exists.
+ACTIVE: Kisa locked the dedup key to (patient_id, device_type, date_of_service) — scales to all DMEPOS categories, order_number is display-only. Pi designed the dedup logic (tiebreakers, quantity merge guard, CoverageLine assignment, 6 test cases) at docs/dedup-design-2026-06-24.md. This UNBLOCKS the readiness-model build.
 
-NEXT (in order): (1) Kisa: dedup decision (same patient + device + DOS under two order numbers -> one worklist line or two?) -> unblocks Claude's readiness-model build. (2) Kisa: send the Convex BAA email + /pilot-prep labABLE Fri 2026-06-26 1:30 PM. (3) Claude: build the readiness model once dedup is decided from its brief (rename coverage->readiness, rules-to-config, patient/line/shipment grouping+verdict, device-keyed override, require-plan-type, frontend nesting). (4) Claude: create signal-convex repo (GitHub via `gh`, Forgejo needs Kisa's token) + build Phase 1 of the Convex-Signal spike from Pi's design (docs/convex-signal-spike-design-2026-06-24.md). (5) Claude: research-versioning Layer-1 + add signal/docs/ to its scope (Audit Finding 5 — the reason design docs need manual commits).
+NEXT (in order): (1) Claude: build the readiness model — dedup decision is LOCKED (docs/dedup-design-2026-06-24.md). Build sequence: rename coverage->readiness -> extract rules to addressable config (IDs) -> patient/line/shipment grouping + required-vs-input verdict -> device-keyed override + recompute + rollup -> require-plan-type + client value-mapping + remove guesser/default -> frontend nesting + "plan type needed" state. (2) Kisa: send the Convex BAA email + /pilot-prep labABLE Fri 2026-06-26 1:30 PM. (3) Claude: create signal-convex repo (GitHub via `gh`, Forgejo needs Kisa's token) + build Phase 1 of the Convex-Signal spike from Pi's design (docs/convex-signal-spike-design-2026-06-24.md). (4) Claude: research-versioning Layer-1 + add signal/docs/ to its scope (Audit Finding 5).
 
 ## Decisions (last 8)
 
+- 2026-06-24 [Pi/Kisa]: Dedup decision LOCKED — key = (patient_id, device_type, date_of_service). Order_number is display-only, never in dedup key. Scales to all DMEPOS categories. Dedup design (tiebreakers, quantity merge guard, test cases) at docs/dedup-design-2026-06-24.md. Readiness-model build is UNBLOCKED.
 - 2026-06-24 [Pi]: Convex-Signal spike full design persisted to docs/convex-signal-spike-design-2026-06-24.md. 12 sections covering module map with 6 swappable seams (auth, data, readiness, storage, automations, notifications), Repository interface for backend portability, Convex schema (8 tables matching the readiness model), auth seam decision (Convex first, Clerk swappable), phased 40h/5-day spike plan (Phase 1 proves core readiness loop on Convex). Claude builds Phase 1 from this.
 - 2026-06-24 [CC]: SHIPPED the Claude-side delegation trigger — dispatch-trigger.sh reads stdin JSON (fixed Pi's $1 bug), settings.json UserPromptSubmit hook wired, dispatch SKILL.md thresholds added, Finding-4 clarifying line so old+new dispatch rules don't conflict. Verified live. Audit Findings 1/2/4 (this build) closed; 3/8 Pi-side, 5/6/7/9 research-versioning/phase-2 and non-blocking.
 - 2026-06-24 [CC]: NocoDB leads diagnosed — Signal-base data source bdmhi02nks1pg6g is corrupted, creates PK-less tables (old leads + a fresh test table both no-PK; content base healthy). SQLite can't add a PK to a populated table (Route 1 failed). Capture still works (n8n webhook signal-cgm-lead inserts ok); 2 test rows un-deletable. Folded into the platform decision.
@@ -22,7 +23,7 @@ NEXT (in order): (1) Kisa: dedup decision (same patient + device + DOS under two
 
 - CONVEX-SIGNAL SPIKE (2026-06-24): brief ready (docs/convex-signal-instance-brief-2026-06-24.md); separate repo signal-convex; Pi designs next. Deps: Kisa's Convex BAA email; repo creation post-approval. Goal: prove Convex on the real product, low-risk parallel, before any platform commitment.
 - DATA-PLATFORM DECISION (2026-06-24): NocoDB leads broke (no-PK, corrupted source). Options: repaired-NocoDB / Supabase Postgres / Neon / AWS (already set up) / Convex. Leads = non-PHI. Informed by the Convex spike. Interim: capture works; 2 test rows un-deletable.
-- READINESS MODEL BUILD: BLOCKED on Kisa's dedup decision. Design ready: docs/readiness-model-brief-2026-06-23.md.
+- READINESS MODEL BUILD: UNBLOCKED. Dedup decision locked (docs/dedup-design-2026-06-24.md). Ready for Claude to build.
 - DELEGATION / RESEARCH-VERSIONING CLEANUP: trigger SHIPPED; remaining = research-versioning Layer-1 + signal/docs scope (Finding 5), low-priority Findings 6/7/9, Pi-side 3/8. Ledger: docs/ship-status-ledger.md.
 - LabABLE Meeting Fri 2026-06-26 1:30 PM — /pilot-prep.
 

docs/dedup-design-2026-06-24.md

@@ -0,0 +1,270 @@
+# Dedup Design — Patient + Device + DOS Grouping
+
+**Date:** 2026-06-24
+**Design by:** Pi (decision locked by Kisa)
+**Builds by:** Claude (Phase: patient/line/shipment grouping + required-vs-input verdict)
+**Source:** docs/readiness-model-brief-2026-06-23.md, Kisa dedup decision 2026-06-24
+
+---
+
+## 1. The Locked Dedup Key
+
+```
+(patient_id, device_type, date_of_service)
+```
+
+Three fields, no more. Order_number is displayed as an attribute, never used in dedup logic.
+
+Scales to all DMEPOS categories (CGM, CPAP, wheelchair, oxygen, wig, bed, ostomy). The rationale is documented in the audit at `docs/review-audit-delegation-research-versioning-2026-06-24.md` and the Kisa decision in `context/current-state.md`.
+
+---
+
+## 2. Where Dedup Fits in the Pipeline
+
+The current pipeline grades ONE record at a time (the known bug). The fixed pipeline:
+
+```
+CSV rows
+  │
+  ├── Step 1: Normalize (map field names, parse dates, validate required fields)
+  │
+  ├── Step 2: DEDUP (this design)
+  │     Group rows by (patient_id, device_type, DOS)
+  │     → MergedShipment (one per group)
+  │
+  ├── Step 3: Assign to CoverageLines
+  │     Each MergedShipment → coverageLine (patient + device + plan)
+  │     Latest DOS per line → isCurrent = true
+  │
+  ├── Step 4: Grade current line
+  │     gradeLine() → verdict per CoverageLine
+  │
+  └── Step 5: Roll up
+        rollupPatient() → patient-level status
+```
+
+---
+
+## 3. MergedShipment Shape (After Dedup)
+
+```python
+@dataclass
+class MergedShipment:
+    """One deduped shipment after grouping CSV rows by (patient_id, device_type, DOS)."""
+    patient_id: str
+    device_type: str
+    shipment_date: date
+    total_quantity: int                     # summed from all rows in the group
+    components: list[str]                   # combined set, deduped
+    order_numbers: list[str]                # collected, displayed only
+    hcpcs_codes: list[str]                  # collected, displayed only
+    payer: str                              # taken from the LATEST row in the group
+    # CSV doc fields — merged from all rows, latest non-null wins
+    csv_visit_date: date | None
+    csv_swo_status: str | None
+    csv_pecos_verified: str | None
+    csv_pa_status: str | None
+    csv_diagnosis_on_file: str | None
+    csv_transfer_from: str | None
+```
+
+---
+
+## 4. Tiebreaker Rules (Within a Dedup Group)
+
+When multiple CSV rows have the same `(patient_id, device_type, DOS)`:
+
+| Field | Rule | Why |
+|-------|------|-----|
+| `quantity` | **Sum** across rows | Each row represents components; total is the sum of all components delivered |
+| `components` | **Combine into set**, dedup duplicates | A row with [sensor] + a row with [sensor, transmitter] → [sensor, transmitter] |
+| `order_numbers` | **Collect into list**, dedup duplicates | Displayed as "ORD-100, ORD-101" — collapsed to "ORD-100, ORD-101 (2 orders)" if same number |
+| `hcpcs_codes` | **Collect into list**, dedup duplicates | Displayed as "A4230, A4231" |
+| `payer` | **Latest non-null wins** | Last row processed in the group. Rows are processed in ingestion order. |
+| `csv_*` doc fields | **Latest non-null wins** | If one row has csv_swo_status="On File" and another has null, the non-null wins. If both have different values, latest wins. |
+| `shipment_date` | **Must be identical** after normalization to date | If two rows have different dates after normalization, they belong to different groups. If same calendar date but different timestamps, normalize to date first. |
+
+### Quantity edge cases
+
+| Scenario | Input | Result | Rationale |
+|----------|-------|--------|-----------|
+| Normal split | Row A qty=3, Row B qty=1 | total_qty = 4 | Sensor + transmitter = 4 components |
+| Duplicate | Row A qty=3, Row B qty=3 (same components) | total_qty = 6 | ⚠️ This IS the risk of summing: a data-entry duplicate doubles the quantity. Mitigation: flag rows where components are identical across rows in the same group. If all fields except order_number match exactly, it is a likely duplicate → take max, not sum. |
+| Zero row | Row A qty=3, Row B qty=0 | total_qty = 3 | Zero-quantity rows are informational (e.g., a cancelation) and do not add to quantity |
+| Wildly different | Row A qty=3, Row B qty=100 | total_qty = 103 | This is almost certainly an error. Flag for review. For the pilot, take max (100) and log a warning. |
+
+### Recommended default for quantity ambiguity
+
+**Sum, but guard against identical-component duplicates.** The rule:
+
+```python
+def merge_quantities(rows_in_group: list[CSVRow]) -> int:
+    """Sum quantities within a dedup group.
+    If two rows have identical components, take max instead of sum
+    (likely a data-entry duplicate, not a real split)."""
+    component_sets = [set(row.components.split(",")) for row in rows_in_group]
+    unique_component_configs = set(frozenset(cs) for cs in component_sets)
+
+    if len(unique_component_configs) < len(rows_in_group):
+        # At least one component config appears more than once
+        # Take the maximum quantity, not the sum (defense against dupes)
+        return max(row.quantity for row in rows_in_group)
+
+    return sum(row.quantity for row in rows_in_group)
+```
+
+---
+
+## 5. How This Feeds CoverageLines
+
+Each `MergedShipment` is assigned to a `CoverageLine` keyed by `(patient_id, device_type, plan_type_id)`.
+
+```python
+def assign_to_coverage_lines(merged_shipments: list[MergedShipment]) -> dict[CoverageLineKey, CoverageLine]:
+    """Group shipments into CoverageLines by (patient_id, device_type, plan_type)."""
+    lines: dict[CoverageLineKey, list[MergedShipment]] = {}
+
+    for shipment in merged_shipments:
+        # Capline: plan_type comes from client's CSV mapping, not from guessing
+        key = CoverageLineKey(
+            patient_id=shipment.patient_id,
+            device_type=shipment.device_type,
+            plan_type_id=shipment.payer,  # The client-mapped plan type, not the raw payer name
+        )
+        lines.setdefault(key, []).append(shipment)
+
+    result = {}
+    for key, shipments in lines.items():
+        # Sort by DOS, latest = current
+        shipments.sort(key=lambda s: s.shipment_date, reverse=True)
+        current = shipments[0]
+        gradeable = is_cgm(key.device_type)  # CGM only for Phase 1; expand later
+
+        result[key] = CoverageLine(
+            patient_id=key.patient_id,
+            device_type=key.device_type,
+            plan_type=key.plan_type_id,
+            gradeable=gradeable,
+            current_shipment=current,
+            prior_shipments=shipments[1:],
+            line_status=None,   # computed by gradeLine()
+            timing_flag=None,   # computed by gradeLine()
+        )
+
+    return result
+```
+
+---
+
+## 6. Frontend Display After Dedup
+
+Before dedup (current behavior — the bug):
+
+```
+Patient  | Device   | DOS       | Order  | Status
+ABC-123  | Dex G7   | 2026-06-01| ORD-100| Resupply Ready
+ABC-123  | Dex G7   | 2026-06-01| ORD-101| Resupply Ready
+```
+
+After dedup:
+
+```
+Patient  | Device   | DOS       | Orders             | Qty | Components          | Status
+ABC-123  | Dex G7   | 2026-06-01| ORD-100, ORD-101   | 4   | sensor, transmitter  | Resupply Ready
+```
+
+**Order numbers column:** Shows a comma-separated list. If >3 order numbers, show "ORD-100, ORD-101 +2 more" with expand/collapse.
+
+**Components column:** Shows the combined set. If only one component, show "sensor". If multiple, show "sensor, transmitter".
+
+**Quantity column:** Shows the sum. Hover shows detail: "3 sensors + 1 transmitter" is derived from components + quantity (not explicitly stored, but inferred from the component splits).
+
+---
+
+## 7. Edge Cases Log
+
+| Edge Case | Handled? | How |
+|-----------|----------|-----|
+| Only one row for the key | ✅ | No merging needed; MergedShipment = the single row |
+| Missing quantity | ✅ | Treat quantity as 1 (a row without a qty is still a shipment) |
+| Missing components | ✅ | Default to the device_type's default component from payer_rules.json |
+| Missing order_number | ✅ | Order_numbers list is empty; display shows "—" |
+| No order_numbers at all in CSV | ✅ | Entire column hidden |
+| DOS different due to timezone | ✅ | Normalize to YYYY-MM-DD UTC before comparison |
+| Three+ orders for same patient+device+DOS | ✅ | Sum quantities, combine all order numbers |
+| Same patient, same DOS, different device | ✅ | Different device = different CoverageLine; no dedup conflict |
+| Same patient, same device, different DOS | ✅ | Different DOS = different MergedShipment; no dedup conflict |
+
+---
+
+## 8. Implementation Notes for Claude
+
+### Where the change lives
+
+| Layer | File | Change |
+|-------|------|--------|
+| Dedup step | NEW: `python-backend/core/dedup.py` | Group rows → MergedShipment |
+| Pipeline orchestration | `python-backend/core/coverage_calculator.py` | Call dedup BEFORE gradeLine, pass MergedShipments instead of raw rows |
+| Data model | `python-backend/core/coverage_calculator.py` | Add MergedShipment dataclass, add CoverageLine dataclass |
+| Normalizer | `python-backend/api/normalizer.py` | No change — still normalizes individual rows; dedup happens after |
+| Frontend | `signal-ui/src/components/WorklistTable.jsx` | Display order_numbers as list, components as set, quantity as sum |
+
+### Test scenarios to cover
+
+```python
+def test_dedup_component_split():
+    """Two rows, same patient+device+DOS, different components → one MergedShipment."""
+    rows = [
+        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
+        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=1, component="transmitter"),
+    ]
+    result = dedup(rows)
+    assert len(result) == 1
+    assert result[0].total_quantity == 4
+    assert set(result[0].components) == {"sensor", "transmitter"}
+    assert len(result[0].order_numbers) == 2  # Two different orders
+
+def test_dedup_identical_duplicate():
+    """Two rows, identical data → max quantity, not sum (duplicate guard)."""
+    rows = [
+        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
+        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
+    ]
+    result = dedup(rows)
+    assert len(result) == 1
+    assert result[0].total_quantity == 3  # max, not 6
+
+def test_dedup_different_dos():
+    """Same patient+device, different DOS → two separate shipments."""
+    rows = [
+        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
+        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-07-01", qty=3, component="sensor"),
+    ]
+    result = dedup(rows)
+    assert len(result) == 2
+
+def test_dedup_different_device():
+    """Same patient+DOS, different device → two separate shipments."""
+    rows = [
+        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
+        CSVRow(patient_id="P1", device="omnipod_5", dos="2026-06-01", qty=10, component="pod"),
+    ]
+    result = dedup(rows)
+    assert len(result) == 2
+
+def test_dedup_no_quantity():
+    """Missing quantity → default to 1."""
+    rows = [
+        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=None, component="sensor"),
+    ]
+    result = dedup(rows)
+    assert result[0].total_quantity == 1
+
+def test_dedup_missing_order_number():
+    """No order_number in CSV → order_numbers list is empty, no crash."""
+    rows = [
+        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor", order_number=None),
+    ]
+    result = dedup(rows)
+    assert result[0].order_numbers == []
+```

docs/ship-status-ledger.md

@@ -18,7 +18,7 @@ Canonical, git-tracked record of **what shipped, what didn't, and the reason or
 | 6 | Convex-Signal spike **design** | Pi | **SHIPPED** | docs/convex-signal-spike-design-2026-06-24.md. 40h/3-phase plan; Module map + Repository interface + Convex schema + Auth seam (Convex default, Clerk swappable) + Phase 1 (core readiness loop) = what Claude builds first. | Claude builds Phase 1 from it |
 | 7 | `signal-convex` repo creation | Claude (GitHub) + Kisa/token (Forgejo) | **PENDING** | Dependency: Pi design + Kisa approval (create once, correctly). Claude can do GitHub via `gh`; Forgejo has no API token here. | after design approved |
 | 8 | Convex BAA email (plan/cost/signing) | Kisa | **IN PROGRESS** | Kisa handling | Kisa sends |
-| 9 | Readiness-model build (rename coverage→readiness, rules-to-config, grouping+verdict, device override, require-plan-type, frontend nesting) | Claude | **BLOCKED** | Dependency: Kisa's dedup decision (same patient + device + DOS under two order numbers → one worklist line or two?). Design ready: `docs/readiness-model-brief-2026-06-23.md` | Kisa decides → Claude builds |
+| 9 | Readiness-model build (rename coverage→readiness, rules-to-config, grouping+verdict, device override, require-plan-type, frontend nesting) | Claude | **UNBLOCKED** | Dedup decision locked: (patient_id, device_type, DOS). Order_number is display-only. Scales to all DMEPOS categories. Design: `docs/dedup-design-2026-06-24.md`. | Claude builds from dedup design + readiness model brief |
 | 10 | NocoDB leads / CRM cleanup | Claude + Kisa | **BLOCKED** | Reason: leads table has no primary key (Signal-base data source `bdmhi02nks1pg6g` is corrupted — a fresh table there also came up PK-less; content base is healthy). SQLite can't add a PK to a populated table (Route 1 failed). Interim: capture still works (inserts via n8n `signal-cgm-lead`); the 2 test rows can't be deleted. | resolved by the data-platform decision (item 11) |
 | 11 | Data-platform decision (leads + future PHI backend) | Kisa (with Claude/Pi analysis) | **OPEN** | Options analyzed: repaired-NocoDB / Supabase Postgres / Neon / AWS (already set up, BAA+RDS+S3 parked) / Convex ($25/mo incl. BAA). Leads = non-PHI (stay outside PHI boundary). Decision informed by the Convex spike. | run spike → decide |
 | 12 | Research-versioning Layer-1 (`research-autopush.sh` + Stop-hook entry) | Claude | **NOT STARTED** | Dependency: Pi design exists (`docs/research-versioning-design-2026-06-24.md`). Lower priority than product. | next infra slot |