diff --git a/context/current-state.md b/context/current-state.md index 84f8411..09c9306 100644 --- a/context/current-state.md +++ b/context/current-state.md @@ -2,12 +2,13 @@ v:1 # current-state -ACTIVE: Pi designed the modular Convex-Signal spike from the brief (docs/convex-signal-spike-design-2026-06-24.md). Design covers module map with swappable seams, data-access interface (Repository portability hedge), Convex schema matching the readiness model, auth seam (Convex default, Clerk swappable), phased 40h spike plan (Phase 1 proves core readiness loop). Claude builds Phase 1 once repo exists. +ACTIVE: Kisa locked the dedup key to (patient_id, device_type, date_of_service) — scales to all DMEPOS categories, order_number is display-only. Pi designed the dedup logic (tiebreakers, quantity merge guard, CoverageLine assignment, 6 test cases) at docs/dedup-design-2026-06-24.md. This UNBLOCKS the readiness-model build. -NEXT (in order): (1) Kisa: dedup decision (same patient + device + DOS under two order numbers -> one worklist line or two?) -> unblocks Claude's readiness-model build. (2) Kisa: send the Convex BAA email + /pilot-prep labABLE Fri 2026-06-26 1:30 PM. (3) Claude: build the readiness model once dedup is decided from its brief (rename coverage->readiness, rules-to-config, patient/line/shipment grouping+verdict, device-keyed override, require-plan-type, frontend nesting). (4) Claude: create signal-convex repo (GitHub via `gh`, Forgejo needs Kisa's token) + build Phase 1 of the Convex-Signal spike from Pi's design (docs/convex-signal-spike-design-2026-06-24.md). (5) Claude: research-versioning Layer-1 + add signal/docs/ to its scope (Audit Finding 5 — the reason design docs need manual commits). +NEXT (in order): (1) Claude: build the readiness model — dedup decision is LOCKED (docs/dedup-design-2026-06-24.md). Build sequence: rename coverage->readiness -> extract rules to addressable config (IDs) -> patient/line/shipment grouping + required-vs-input verdict -> device-keyed override + recompute + rollup -> require-plan-type + client value-mapping + remove guesser/default -> frontend nesting + "plan type needed" state. (2) Kisa: send the Convex BAA email + /pilot-prep labABLE Fri 2026-06-26 1:30 PM. (3) Claude: create signal-convex repo (GitHub via `gh`, Forgejo needs Kisa's token) + build Phase 1 of the Convex-Signal spike from Pi's design (docs/convex-signal-spike-design-2026-06-24.md). (4) Claude: research-versioning Layer-1 + add signal/docs/ to its scope (Audit Finding 5). ## Decisions (last 8) +- 2026-06-24 [Pi/Kisa]: Dedup decision LOCKED — key = (patient_id, device_type, date_of_service). Order_number is display-only, never in dedup key. Scales to all DMEPOS categories. Dedup design (tiebreakers, quantity merge guard, test cases) at docs/dedup-design-2026-06-24.md. Readiness-model build is UNBLOCKED. - 2026-06-24 [Pi]: Convex-Signal spike full design persisted to docs/convex-signal-spike-design-2026-06-24.md. 12 sections covering module map with 6 swappable seams (auth, data, readiness, storage, automations, notifications), Repository interface for backend portability, Convex schema (8 tables matching the readiness model), auth seam decision (Convex first, Clerk swappable), phased 40h/5-day spike plan (Phase 1 proves core readiness loop on Convex). Claude builds Phase 1 from this. - 2026-06-24 [CC]: SHIPPED the Claude-side delegation trigger — dispatch-trigger.sh reads stdin JSON (fixed Pi's $1 bug), settings.json UserPromptSubmit hook wired, dispatch SKILL.md thresholds added, Finding-4 clarifying line so old+new dispatch rules don't conflict. Verified live. Audit Findings 1/2/4 (this build) closed; 3/8 Pi-side, 5/6/7/9 research-versioning/phase-2 and non-blocking. - 2026-06-24 [CC]: NocoDB leads diagnosed — Signal-base data source bdmhi02nks1pg6g is corrupted, creates PK-less tables (old leads + a fresh test table both no-PK; content base healthy). SQLite can't add a PK to a populated table (Route 1 failed). Capture still works (n8n webhook signal-cgm-lead inserts ok); 2 test rows un-deletable. Folded into the platform decision. @@ -22,7 +23,7 @@ NEXT (in order): (1) Kisa: dedup decision (same patient + device + DOS under two - CONVEX-SIGNAL SPIKE (2026-06-24): brief ready (docs/convex-signal-instance-brief-2026-06-24.md); separate repo signal-convex; Pi designs next. Deps: Kisa's Convex BAA email; repo creation post-approval. Goal: prove Convex on the real product, low-risk parallel, before any platform commitment. - DATA-PLATFORM DECISION (2026-06-24): NocoDB leads broke (no-PK, corrupted source). Options: repaired-NocoDB / Supabase Postgres / Neon / AWS (already set up) / Convex. Leads = non-PHI. Informed by the Convex spike. Interim: capture works; 2 test rows un-deletable. -- READINESS MODEL BUILD: BLOCKED on Kisa's dedup decision. Design ready: docs/readiness-model-brief-2026-06-23.md. +- READINESS MODEL BUILD: UNBLOCKED. Dedup decision locked (docs/dedup-design-2026-06-24.md). Ready for Claude to build. - DELEGATION / RESEARCH-VERSIONING CLEANUP: trigger SHIPPED; remaining = research-versioning Layer-1 + signal/docs scope (Finding 5), low-priority Findings 6/7/9, Pi-side 3/8. Ledger: docs/ship-status-ledger.md. - LabABLE Meeting Fri 2026-06-26 1:30 PM — /pilot-prep. diff --git a/docs/dedup-design-2026-06-24.md b/docs/dedup-design-2026-06-24.md new file mode 100644 index 0000000..55e1b44 --- /dev/null +++ b/docs/dedup-design-2026-06-24.md @@ -0,0 +1,270 @@ +# Dedup Design — Patient + Device + DOS Grouping + +**Date:** 2026-06-24 +**Design by:** Pi (decision locked by Kisa) +**Builds by:** Claude (Phase: patient/line/shipment grouping + required-vs-input verdict) +**Source:** docs/readiness-model-brief-2026-06-23.md, Kisa dedup decision 2026-06-24 + +--- + +## 1. The Locked Dedup Key + +``` +(patient_id, device_type, date_of_service) +``` + +Three fields, no more. Order_number is displayed as an attribute, never used in dedup logic. + +Scales to all DMEPOS categories (CGM, CPAP, wheelchair, oxygen, wig, bed, ostomy). The rationale is documented in the audit at `docs/review-audit-delegation-research-versioning-2026-06-24.md` and the Kisa decision in `context/current-state.md`. + +--- + +## 2. Where Dedup Fits in the Pipeline + +The current pipeline grades ONE record at a time (the known bug). The fixed pipeline: + +``` +CSV rows + │ + ├── Step 1: Normalize (map field names, parse dates, validate required fields) + │ + ├── Step 2: DEDUP (this design) + │ Group rows by (patient_id, device_type, DOS) + │ → MergedShipment (one per group) + │ + ├── Step 3: Assign to CoverageLines + │ Each MergedShipment → coverageLine (patient + device + plan) + │ Latest DOS per line → isCurrent = true + │ + ├── Step 4: Grade current line + │ gradeLine() → verdict per CoverageLine + │ + └── Step 5: Roll up + rollupPatient() → patient-level status +``` + +--- + +## 3. MergedShipment Shape (After Dedup) + +```python +@dataclass +class MergedShipment: + """One deduped shipment after grouping CSV rows by (patient_id, device_type, DOS).""" + patient_id: str + device_type: str + shipment_date: date + total_quantity: int # summed from all rows in the group + components: list[str] # combined set, deduped + order_numbers: list[str] # collected, displayed only + hcpcs_codes: list[str] # collected, displayed only + payer: str # taken from the LATEST row in the group + # CSV doc fields — merged from all rows, latest non-null wins + csv_visit_date: date | None + csv_swo_status: str | None + csv_pecos_verified: str | None + csv_pa_status: str | None + csv_diagnosis_on_file: str | None + csv_transfer_from: str | None +``` + +--- + +## 4. Tiebreaker Rules (Within a Dedup Group) + +When multiple CSV rows have the same `(patient_id, device_type, DOS)`: + +| Field | Rule | Why | +|-------|------|-----| +| `quantity` | **Sum** across rows | Each row represents components; total is the sum of all components delivered | +| `components` | **Combine into set**, dedup duplicates | A row with [sensor] + a row with [sensor, transmitter] → [sensor, transmitter] | +| `order_numbers` | **Collect into list**, dedup duplicates | Displayed as "ORD-100, ORD-101" — collapsed to "ORD-100, ORD-101 (2 orders)" if same number | +| `hcpcs_codes` | **Collect into list**, dedup duplicates | Displayed as "A4230, A4231" | +| `payer` | **Latest non-null wins** | Last row processed in the group. Rows are processed in ingestion order. | +| `csv_*` doc fields | **Latest non-null wins** | If one row has csv_swo_status="On File" and another has null, the non-null wins. If both have different values, latest wins. | +| `shipment_date` | **Must be identical** after normalization to date | If two rows have different dates after normalization, they belong to different groups. If same calendar date but different timestamps, normalize to date first. | + +### Quantity edge cases + +| Scenario | Input | Result | Rationale | +|----------|-------|--------|-----------| +| Normal split | Row A qty=3, Row B qty=1 | total_qty = 4 | Sensor + transmitter = 4 components | +| Duplicate | Row A qty=3, Row B qty=3 (same components) | total_qty = 6 | ⚠️ This IS the risk of summing: a data-entry duplicate doubles the quantity. Mitigation: flag rows where components are identical across rows in the same group. If all fields except order_number match exactly, it is a likely duplicate → take max, not sum. | +| Zero row | Row A qty=3, Row B qty=0 | total_qty = 3 | Zero-quantity rows are informational (e.g., a cancelation) and do not add to quantity | +| Wildly different | Row A qty=3, Row B qty=100 | total_qty = 103 | This is almost certainly an error. Flag for review. For the pilot, take max (100) and log a warning. | + +### Recommended default for quantity ambiguity + +**Sum, but guard against identical-component duplicates.** The rule: + +```python +def merge_quantities(rows_in_group: list[CSVRow]) -> int: + """Sum quantities within a dedup group. + If two rows have identical components, take max instead of sum + (likely a data-entry duplicate, not a real split).""" + component_sets = [set(row.components.split(",")) for row in rows_in_group] + unique_component_configs = set(frozenset(cs) for cs in component_sets) + + if len(unique_component_configs) < len(rows_in_group): + # At least one component config appears more than once + # Take the maximum quantity, not the sum (defense against dupes) + return max(row.quantity for row in rows_in_group) + + return sum(row.quantity for row in rows_in_group) +``` + +--- + +## 5. How This Feeds CoverageLines + +Each `MergedShipment` is assigned to a `CoverageLine` keyed by `(patient_id, device_type, plan_type_id)`. + +```python +def assign_to_coverage_lines(merged_shipments: list[MergedShipment]) -> dict[CoverageLineKey, CoverageLine]: + """Group shipments into CoverageLines by (patient_id, device_type, plan_type).""" + lines: dict[CoverageLineKey, list[MergedShipment]] = {} + + for shipment in merged_shipments: + # Capline: plan_type comes from client's CSV mapping, not from guessing + key = CoverageLineKey( + patient_id=shipment.patient_id, + device_type=shipment.device_type, + plan_type_id=shipment.payer, # The client-mapped plan type, not the raw payer name + ) + lines.setdefault(key, []).append(shipment) + + result = {} + for key, shipments in lines.items(): + # Sort by DOS, latest = current + shipments.sort(key=lambda s: s.shipment_date, reverse=True) + current = shipments[0] + gradeable = is_cgm(key.device_type) # CGM only for Phase 1; expand later + + result[key] = CoverageLine( + patient_id=key.patient_id, + device_type=key.device_type, + plan_type=key.plan_type_id, + gradeable=gradeable, + current_shipment=current, + prior_shipments=shipments[1:], + line_status=None, # computed by gradeLine() + timing_flag=None, # computed by gradeLine() + ) + + return result +``` + +--- + +## 6. Frontend Display After Dedup + +Before dedup (current behavior — the bug): + +``` +Patient | Device | DOS | Order | Status +ABC-123 | Dex G7 | 2026-06-01| ORD-100| Resupply Ready +ABC-123 | Dex G7 | 2026-06-01| ORD-101| Resupply Ready +``` + +After dedup: + +``` +Patient | Device | DOS | Orders | Qty | Components | Status +ABC-123 | Dex G7 | 2026-06-01| ORD-100, ORD-101 | 4 | sensor, transmitter | Resupply Ready +``` + +**Order numbers column:** Shows a comma-separated list. If >3 order numbers, show "ORD-100, ORD-101 +2 more" with expand/collapse. + +**Components column:** Shows the combined set. If only one component, show "sensor". If multiple, show "sensor, transmitter". + +**Quantity column:** Shows the sum. Hover shows detail: "3 sensors + 1 transmitter" is derived from components + quantity (not explicitly stored, but inferred from the component splits). + +--- + +## 7. Edge Cases Log + +| Edge Case | Handled? | How | +|-----------|----------|-----| +| Only one row for the key | ✅ | No merging needed; MergedShipment = the single row | +| Missing quantity | ✅ | Treat quantity as 1 (a row without a qty is still a shipment) | +| Missing components | ✅ | Default to the device_type's default component from payer_rules.json | +| Missing order_number | ✅ | Order_numbers list is empty; display shows "—" | +| No order_numbers at all in CSV | ✅ | Entire column hidden | +| DOS different due to timezone | ✅ | Normalize to YYYY-MM-DD UTC before comparison | +| Three+ orders for same patient+device+DOS | ✅ | Sum quantities, combine all order numbers | +| Same patient, same DOS, different device | ✅ | Different device = different CoverageLine; no dedup conflict | +| Same patient, same device, different DOS | ✅ | Different DOS = different MergedShipment; no dedup conflict | + +--- + +## 8. Implementation Notes for Claude + +### Where the change lives + +| Layer | File | Change | +|-------|------|--------| +| Dedup step | NEW: `python-backend/core/dedup.py` | Group rows → MergedShipment | +| Pipeline orchestration | `python-backend/core/coverage_calculator.py` | Call dedup BEFORE gradeLine, pass MergedShipments instead of raw rows | +| Data model | `python-backend/core/coverage_calculator.py` | Add MergedShipment dataclass, add CoverageLine dataclass | +| Normalizer | `python-backend/api/normalizer.py` | No change — still normalizes individual rows; dedup happens after | +| Frontend | `signal-ui/src/components/WorklistTable.jsx` | Display order_numbers as list, components as set, quantity as sum | + +### Test scenarios to cover + +```python +def test_dedup_component_split(): + """Two rows, same patient+device+DOS, different components → one MergedShipment.""" + rows = [ + CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"), + CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=1, component="transmitter"), + ] + result = dedup(rows) + assert len(result) == 1 + assert result[0].total_quantity == 4 + assert set(result[0].components) == {"sensor", "transmitter"} + assert len(result[0].order_numbers) == 2 # Two different orders + +def test_dedup_identical_duplicate(): + """Two rows, identical data → max quantity, not sum (duplicate guard).""" + rows = [ + CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"), + CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"), + ] + result = dedup(rows) + assert len(result) == 1 + assert result[0].total_quantity == 3 # max, not 6 + +def test_dedup_different_dos(): + """Same patient+device, different DOS → two separate shipments.""" + rows = [ + CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"), + CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-07-01", qty=3, component="sensor"), + ] + result = dedup(rows) + assert len(result) == 2 + +def test_dedup_different_device(): + """Same patient+DOS, different device → two separate shipments.""" + rows = [ + CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"), + CSVRow(patient_id="P1", device="omnipod_5", dos="2026-06-01", qty=10, component="pod"), + ] + result = dedup(rows) + assert len(result) == 2 + +def test_dedup_no_quantity(): + """Missing quantity → default to 1.""" + rows = [ + CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=None, component="sensor"), + ] + result = dedup(rows) + assert result[0].total_quantity == 1 + +def test_dedup_missing_order_number(): + """No order_number in CSV → order_numbers list is empty, no crash.""" + rows = [ + CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor", order_number=None), + ] + result = dedup(rows) + assert result[0].order_numbers == [] +``` diff --git a/docs/ship-status-ledger.md b/docs/ship-status-ledger.md index 5e55ded..957c101 100644 --- a/docs/ship-status-ledger.md +++ b/docs/ship-status-ledger.md @@ -18,7 +18,7 @@ Canonical, git-tracked record of **what shipped, what didn't, and the reason or | 6 | Convex-Signal spike **design** | Pi | **SHIPPED** | docs/convex-signal-spike-design-2026-06-24.md. 40h/3-phase plan; Module map + Repository interface + Convex schema + Auth seam (Convex default, Clerk swappable) + Phase 1 (core readiness loop) = what Claude builds first. | Claude builds Phase 1 from it | | 7 | `signal-convex` repo creation | Claude (GitHub) + Kisa/token (Forgejo) | **PENDING** | Dependency: Pi design + Kisa approval (create once, correctly). Claude can do GitHub via `gh`; Forgejo has no API token here. | after design approved | | 8 | Convex BAA email (plan/cost/signing) | Kisa | **IN PROGRESS** | Kisa handling | Kisa sends | -| 9 | Readiness-model build (rename coverage→readiness, rules-to-config, grouping+verdict, device override, require-plan-type, frontend nesting) | Claude | **BLOCKED** | Dependency: Kisa's dedup decision (same patient + device + DOS under two order numbers → one worklist line or two?). Design ready: `docs/readiness-model-brief-2026-06-23.md` | Kisa decides → Claude builds | +| 9 | Readiness-model build (rename coverage→readiness, rules-to-config, grouping+verdict, device override, require-plan-type, frontend nesting) | Claude | **UNBLOCKED** | Dedup decision locked: (patient_id, device_type, DOS). Order_number is display-only. Scales to all DMEPOS categories. Design: `docs/dedup-design-2026-06-24.md`. | Claude builds from dedup design + readiness model brief | | 10 | NocoDB leads / CRM cleanup | Claude + Kisa | **BLOCKED** | Reason: leads table has no primary key (Signal-base data source `bdmhi02nks1pg6g` is corrupted — a fresh table there also came up PK-less; content base is healthy). SQLite can't add a PK to a populated table (Route 1 failed). Interim: capture still works (inserts via n8n `signal-cgm-lead`); the 2 test rows can't be deleted. | resolved by the data-platform decision (item 11) | | 11 | Data-platform decision (leads + future PHI backend) | Kisa (with Claude/Pi analysis) | **OPEN** | Options analyzed: repaired-NocoDB / Supabase Postgres / Neon / AWS (already set up, BAA+RDS+S3 parked) / Convex ($25/mo incl. BAA). Leads = non-PHI (stay outside PHI boundary). Decision informed by the Convex spike. | run spike → decide | | 12 | Research-versioning Layer-1 (`research-autopush.sh` + Stop-hook entry) | Claude | **NOT STARTED** | Dependency: Pi design exists (`docs/research-versioning-design-2026-06-24.md`). Lower priority than product. | next infra slot |