design+state: dedup decision locked (patient_id+device_type+DOS). Readiness model UNBLOCKED [Pi]
This commit is contained in:
Kisa
parent
04d204e965
commit
0e5c2f13b7
3 changed files with 275 additions and 4 deletions
|
|
@ -2,12 +2,13 @@ v:1
|
|||
|
||||
# current-state
|
||||
|
||||
ACTIVE: Pi designed the modular Convex-Signal spike from the brief (docs/convex-signal-spike-design-2026-06-24.md). Design covers module map with swappable seams, data-access interface (Repository portability hedge), Convex schema matching the readiness model, auth seam (Convex default, Clerk swappable), phased 40h spike plan (Phase 1 proves core readiness loop). Claude builds Phase 1 once repo exists.
|
||||
ACTIVE: Kisa locked the dedup key to (patient_id, device_type, date_of_service) — scales to all DMEPOS categories, order_number is display-only. Pi designed the dedup logic (tiebreakers, quantity merge guard, CoverageLine assignment, 6 test cases) at docs/dedup-design-2026-06-24.md. This UNBLOCKS the readiness-model build.
|
||||
|
||||
NEXT (in order): (1) Kisa: dedup decision (same patient + device + DOS under two order numbers -> one worklist line or two?) -> unblocks Claude's readiness-model build. (2) Kisa: send the Convex BAA email + /pilot-prep labABLE Fri 2026-06-26 1:30 PM. (3) Claude: build the readiness model once dedup is decided from its brief (rename coverage->readiness, rules-to-config, patient/line/shipment grouping+verdict, device-keyed override, require-plan-type, frontend nesting). (4) Claude: create signal-convex repo (GitHub via `gh`, Forgejo needs Kisa's token) + build Phase 1 of the Convex-Signal spike from Pi's design (docs/convex-signal-spike-design-2026-06-24.md). (5) Claude: research-versioning Layer-1 + add signal/docs/ to its scope (Audit Finding 5 — the reason design docs need manual commits).
|
||||
NEXT (in order): (1) Claude: build the readiness model — dedup decision is LOCKED (docs/dedup-design-2026-06-24.md). Build sequence: rename coverage->readiness -> extract rules to addressable config (IDs) -> patient/line/shipment grouping + required-vs-input verdict -> device-keyed override + recompute + rollup -> require-plan-type + client value-mapping + remove guesser/default -> frontend nesting + "plan type needed" state. (2) Kisa: send the Convex BAA email + /pilot-prep labABLE Fri 2026-06-26 1:30 PM. (3) Claude: create signal-convex repo (GitHub via `gh`, Forgejo needs Kisa's token) + build Phase 1 of the Convex-Signal spike from Pi's design (docs/convex-signal-spike-design-2026-06-24.md). (4) Claude: research-versioning Layer-1 + add signal/docs/ to its scope (Audit Finding 5).
|
||||
|
||||
## Decisions (last 8)
|
||||
|
||||
- 2026-06-24 [Pi/Kisa]: Dedup decision LOCKED — key = (patient_id, device_type, date_of_service). Order_number is display-only, never in dedup key. Scales to all DMEPOS categories. Dedup design (tiebreakers, quantity merge guard, test cases) at docs/dedup-design-2026-06-24.md. Readiness-model build is UNBLOCKED.
|
||||
- 2026-06-24 [Pi]: Convex-Signal spike full design persisted to docs/convex-signal-spike-design-2026-06-24.md. 12 sections covering module map with 6 swappable seams (auth, data, readiness, storage, automations, notifications), Repository interface for backend portability, Convex schema (8 tables matching the readiness model), auth seam decision (Convex first, Clerk swappable), phased 40h/5-day spike plan (Phase 1 proves core readiness loop on Convex). Claude builds Phase 1 from this.
|
||||
- 2026-06-24 [CC]: SHIPPED the Claude-side delegation trigger — dispatch-trigger.sh reads stdin JSON (fixed Pi's $1 bug), settings.json UserPromptSubmit hook wired, dispatch SKILL.md thresholds added, Finding-4 clarifying line so old+new dispatch rules don't conflict. Verified live. Audit Findings 1/2/4 (this build) closed; 3/8 Pi-side, 5/6/7/9 research-versioning/phase-2 and non-blocking.
|
||||
- 2026-06-24 [CC]: NocoDB leads diagnosed — Signal-base data source bdmhi02nks1pg6g is corrupted, creates PK-less tables (old leads + a fresh test table both no-PK; content base healthy). SQLite can't add a PK to a populated table (Route 1 failed). Capture still works (n8n webhook signal-cgm-lead inserts ok); 2 test rows un-deletable. Folded into the platform decision.
|
||||
|
|
@ -22,7 +23,7 @@ NEXT (in order): (1) Kisa: dedup decision (same patient + device + DOS under two
|
|||
|
||||
- CONVEX-SIGNAL SPIKE (2026-06-24): brief ready (docs/convex-signal-instance-brief-2026-06-24.md); separate repo signal-convex; Pi designs next. Deps: Kisa's Convex BAA email; repo creation post-approval. Goal: prove Convex on the real product, low-risk parallel, before any platform commitment.
|
||||
- DATA-PLATFORM DECISION (2026-06-24): NocoDB leads broke (no-PK, corrupted source). Options: repaired-NocoDB / Supabase Postgres / Neon / AWS (already set up) / Convex. Leads = non-PHI. Informed by the Convex spike. Interim: capture works; 2 test rows un-deletable.
|
||||
- READINESS MODEL BUILD: BLOCKED on Kisa's dedup decision. Design ready: docs/readiness-model-brief-2026-06-23.md.
|
||||
- READINESS MODEL BUILD: UNBLOCKED. Dedup decision locked (docs/dedup-design-2026-06-24.md). Ready for Claude to build.
|
||||
- DELEGATION / RESEARCH-VERSIONING CLEANUP: trigger SHIPPED; remaining = research-versioning Layer-1 + signal/docs scope (Finding 5), low-priority Findings 6/7/9, Pi-side 3/8. Ledger: docs/ship-status-ledger.md.
|
||||
- LabABLE Meeting Fri 2026-06-26 1:30 PM — /pilot-prep.
|
||||
|
||||
|
|
|
|||
270
docs/dedup-design-2026-06-24.md
Normal file
270
docs/dedup-design-2026-06-24.md
Normal file
|
|
@ -0,0 +1,270 @@
|
|||
# Dedup Design — Patient + Device + DOS Grouping
|
||||
|
||||
**Date:** 2026-06-24
|
||||
**Design by:** Pi (decision locked by Kisa)
|
||||
**Builds by:** Claude (Phase: patient/line/shipment grouping + required-vs-input verdict)
|
||||
**Source:** docs/readiness-model-brief-2026-06-23.md, Kisa dedup decision 2026-06-24
|
||||
|
||||
---
|
||||
|
||||
## 1. The Locked Dedup Key
|
||||
|
||||
```
|
||||
(patient_id, device_type, date_of_service)
|
||||
```
|
||||
|
||||
Three fields, no more. Order_number is displayed as an attribute, never used in dedup logic.
|
||||
|
||||
Scales to all DMEPOS categories (CGM, CPAP, wheelchair, oxygen, wig, bed, ostomy). The rationale is documented in the audit at `docs/review-audit-delegation-research-versioning-2026-06-24.md` and the Kisa decision in `context/current-state.md`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Where Dedup Fits in the Pipeline
|
||||
|
||||
The current pipeline grades ONE record at a time (the known bug). The fixed pipeline:
|
||||
|
||||
```
|
||||
CSV rows
|
||||
│
|
||||
├── Step 1: Normalize (map field names, parse dates, validate required fields)
|
||||
│
|
||||
├── Step 2: DEDUP (this design)
|
||||
│ Group rows by (patient_id, device_type, DOS)
|
||||
│ → MergedShipment (one per group)
|
||||
│
|
||||
├── Step 3: Assign to CoverageLines
|
||||
│ Each MergedShipment → coverageLine (patient + device + plan)
|
||||
│ Latest DOS per line → isCurrent = true
|
||||
│
|
||||
├── Step 4: Grade current line
|
||||
│ gradeLine() → verdict per CoverageLine
|
||||
│
|
||||
└── Step 5: Roll up
|
||||
rollupPatient() → patient-level status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. MergedShipment Shape (After Dedup)
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class MergedShipment:
|
||||
"""One deduped shipment after grouping CSV rows by (patient_id, device_type, DOS)."""
|
||||
patient_id: str
|
||||
device_type: str
|
||||
shipment_date: date
|
||||
total_quantity: int # summed from all rows in the group
|
||||
components: list[str] # combined set, deduped
|
||||
order_numbers: list[str] # collected, displayed only
|
||||
hcpcs_codes: list[str] # collected, displayed only
|
||||
payer: str # taken from the LATEST row in the group
|
||||
# CSV doc fields — merged from all rows, latest non-null wins
|
||||
csv_visit_date: date | None
|
||||
csv_swo_status: str | None
|
||||
csv_pecos_verified: str | None
|
||||
csv_pa_status: str | None
|
||||
csv_diagnosis_on_file: str | None
|
||||
csv_transfer_from: str | None
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Tiebreaker Rules (Within a Dedup Group)
|
||||
|
||||
When multiple CSV rows have the same `(patient_id, device_type, DOS)`:
|
||||
|
||||
| Field | Rule | Why |
|
||||
|-------|------|-----|
|
||||
| `quantity` | **Sum** across rows | Each row represents components; total is the sum of all components delivered |
|
||||
| `components` | **Combine into set**, dedup duplicates | A row with [sensor] + a row with [sensor, transmitter] → [sensor, transmitter] |
|
||||
| `order_numbers` | **Collect into list**, dedup duplicates | Displayed as "ORD-100, ORD-101" — collapsed to "ORD-100, ORD-101 (2 orders)" if same number |
|
||||
| `hcpcs_codes` | **Collect into list**, dedup duplicates | Displayed as "A4230, A4231" |
|
||||
| `payer` | **Latest non-null wins** | Last row processed in the group. Rows are processed in ingestion order. |
|
||||
| `csv_*` doc fields | **Latest non-null wins** | If one row has csv_swo_status="On File" and another has null, the non-null wins. If both have different values, latest wins. |
|
||||
| `shipment_date` | **Must be identical** after normalization to date | If two rows have different dates after normalization, they belong to different groups. If same calendar date but different timestamps, normalize to date first. |
|
||||
|
||||
### Quantity edge cases
|
||||
|
||||
| Scenario | Input | Result | Rationale |
|
||||
|----------|-------|--------|-----------|
|
||||
| Normal split | Row A qty=3, Row B qty=1 | total_qty = 4 | Sensor + transmitter = 4 components |
|
||||
| Duplicate | Row A qty=3, Row B qty=3 (same components) | total_qty = 6 | ⚠️ This IS the risk of summing: a data-entry duplicate doubles the quantity. Mitigation: flag rows where components are identical across rows in the same group. If all fields except order_number match exactly, it is a likely duplicate → take max, not sum. |
|
||||
| Zero row | Row A qty=3, Row B qty=0 | total_qty = 3 | Zero-quantity rows are informational (e.g., a cancelation) and do not add to quantity |
|
||||
| Wildly different | Row A qty=3, Row B qty=100 | total_qty = 103 | This is almost certainly an error. Flag for review. For the pilot, take max (100) and log a warning. |
|
||||
|
||||
### Recommended default for quantity ambiguity
|
||||
|
||||
**Sum, but guard against identical-component duplicates.** The rule:
|
||||
|
||||
```python
|
||||
def merge_quantities(rows_in_group: list[CSVRow]) -> int:
|
||||
"""Sum quantities within a dedup group.
|
||||
If two rows have identical components, take max instead of sum
|
||||
(likely a data-entry duplicate, not a real split)."""
|
||||
component_sets = [set(row.components.split(",")) for row in rows_in_group]
|
||||
unique_component_configs = set(frozenset(cs) for cs in component_sets)
|
||||
|
||||
if len(unique_component_configs) < len(rows_in_group):
|
||||
# At least one component config appears more than once
|
||||
# Take the maximum quantity, not the sum (defense against dupes)
|
||||
return max(row.quantity for row in rows_in_group)
|
||||
|
||||
return sum(row.quantity for row in rows_in_group)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. How This Feeds CoverageLines
|
||||
|
||||
Each `MergedShipment` is assigned to a `CoverageLine` keyed by `(patient_id, device_type, plan_type_id)`.
|
||||
|
||||
```python
|
||||
def assign_to_coverage_lines(merged_shipments: list[MergedShipment]) -> dict[CoverageLineKey, CoverageLine]:
|
||||
"""Group shipments into CoverageLines by (patient_id, device_type, plan_type)."""
|
||||
lines: dict[CoverageLineKey, list[MergedShipment]] = {}
|
||||
|
||||
for shipment in merged_shipments:
|
||||
# Capline: plan_type comes from client's CSV mapping, not from guessing
|
||||
key = CoverageLineKey(
|
||||
patient_id=shipment.patient_id,
|
||||
device_type=shipment.device_type,
|
||||
plan_type_id=shipment.payer, # The client-mapped plan type, not the raw payer name
|
||||
)
|
||||
lines.setdefault(key, []).append(shipment)
|
||||
|
||||
result = {}
|
||||
for key, shipments in lines.items():
|
||||
# Sort by DOS, latest = current
|
||||
shipments.sort(key=lambda s: s.shipment_date, reverse=True)
|
||||
current = shipments[0]
|
||||
gradeable = is_cgm(key.device_type) # CGM only for Phase 1; expand later
|
||||
|
||||
result[key] = CoverageLine(
|
||||
patient_id=key.patient_id,
|
||||
device_type=key.device_type,
|
||||
plan_type=key.plan_type_id,
|
||||
gradeable=gradeable,
|
||||
current_shipment=current,
|
||||
prior_shipments=shipments[1:],
|
||||
line_status=None, # computed by gradeLine()
|
||||
timing_flag=None, # computed by gradeLine()
|
||||
)
|
||||
|
||||
return result
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Frontend Display After Dedup
|
||||
|
||||
Before dedup (current behavior — the bug):
|
||||
|
||||
```
|
||||
Patient | Device | DOS | Order | Status
|
||||
ABC-123 | Dex G7 | 2026-06-01| ORD-100| Resupply Ready
|
||||
ABC-123 | Dex G7 | 2026-06-01| ORD-101| Resupply Ready
|
||||
```
|
||||
|
||||
After dedup:
|
||||
|
||||
```
|
||||
Patient | Device | DOS | Orders | Qty | Components | Status
|
||||
ABC-123 | Dex G7 | 2026-06-01| ORD-100, ORD-101 | 4 | sensor, transmitter | Resupply Ready
|
||||
```
|
||||
|
||||
**Order numbers column:** Shows a comma-separated list. If >3 order numbers, show "ORD-100, ORD-101 +2 more" with expand/collapse.
|
||||
|
||||
**Components column:** Shows the combined set. If only one component, show "sensor". If multiple, show "sensor, transmitter".
|
||||
|
||||
**Quantity column:** Shows the sum. Hover shows detail: "3 sensors + 1 transmitter" is derived from components + quantity (not explicitly stored, but inferred from the component splits).
|
||||
|
||||
---
|
||||
|
||||
## 7. Edge Cases Log
|
||||
|
||||
| Edge Case | Handled? | How |
|
||||
|-----------|----------|-----|
|
||||
| Only one row for the key | ✅ | No merging needed; MergedShipment = the single row |
|
||||
| Missing quantity | ✅ | Treat quantity as 1 (a row without a qty is still a shipment) |
|
||||
| Missing components | ✅ | Default to the device_type's default component from payer_rules.json |
|
||||
| Missing order_number | ✅ | Order_numbers list is empty; display shows "—" |
|
||||
| No order_numbers at all in CSV | ✅ | Entire column hidden |
|
||||
| DOS different due to timezone | ✅ | Normalize to YYYY-MM-DD UTC before comparison |
|
||||
| Three+ orders for same patient+device+DOS | ✅ | Sum quantities, combine all order numbers |
|
||||
| Same patient, same DOS, different device | ✅ | Different device = different CoverageLine; no dedup conflict |
|
||||
| Same patient, same device, different DOS | ✅ | Different DOS = different MergedShipment; no dedup conflict |
|
||||
|
||||
---
|
||||
|
||||
## 8. Implementation Notes for Claude
|
||||
|
||||
### Where the change lives
|
||||
|
||||
| Layer | File | Change |
|
||||
|-------|------|--------|
|
||||
| Dedup step | NEW: `python-backend/core/dedup.py` | Group rows → MergedShipment |
|
||||
| Pipeline orchestration | `python-backend/core/coverage_calculator.py` | Call dedup BEFORE gradeLine, pass MergedShipments instead of raw rows |
|
||||
| Data model | `python-backend/core/coverage_calculator.py` | Add MergedShipment dataclass, add CoverageLine dataclass |
|
||||
| Normalizer | `python-backend/api/normalizer.py` | No change — still normalizes individual rows; dedup happens after |
|
||||
| Frontend | `signal-ui/src/components/WorklistTable.jsx` | Display order_numbers as list, components as set, quantity as sum |
|
||||
|
||||
### Test scenarios to cover
|
||||
|
||||
```python
|
||||
def test_dedup_component_split():
|
||||
"""Two rows, same patient+device+DOS, different components → one MergedShipment."""
|
||||
rows = [
|
||||
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
|
||||
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=1, component="transmitter"),
|
||||
]
|
||||
result = dedup(rows)
|
||||
assert len(result) == 1
|
||||
assert result[0].total_quantity == 4
|
||||
assert set(result[0].components) == {"sensor", "transmitter"}
|
||||
assert len(result[0].order_numbers) == 2 # Two different orders
|
||||
|
||||
def test_dedup_identical_duplicate():
|
||||
"""Two rows, identical data → max quantity, not sum (duplicate guard)."""
|
||||
rows = [
|
||||
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
|
||||
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
|
||||
]
|
||||
result = dedup(rows)
|
||||
assert len(result) == 1
|
||||
assert result[0].total_quantity == 3 # max, not 6
|
||||
|
||||
def test_dedup_different_dos():
|
||||
"""Same patient+device, different DOS → two separate shipments."""
|
||||
rows = [
|
||||
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
|
||||
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-07-01", qty=3, component="sensor"),
|
||||
]
|
||||
result = dedup(rows)
|
||||
assert len(result) == 2
|
||||
|
||||
def test_dedup_different_device():
|
||||
"""Same patient+DOS, different device → two separate shipments."""
|
||||
rows = [
|
||||
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
|
||||
CSVRow(patient_id="P1", device="omnipod_5", dos="2026-06-01", qty=10, component="pod"),
|
||||
]
|
||||
result = dedup(rows)
|
||||
assert len(result) == 2
|
||||
|
||||
def test_dedup_no_quantity():
|
||||
"""Missing quantity → default to 1."""
|
||||
rows = [
|
||||
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=None, component="sensor"),
|
||||
]
|
||||
result = dedup(rows)
|
||||
assert result[0].total_quantity == 1
|
||||
|
||||
def test_dedup_missing_order_number():
|
||||
"""No order_number in CSV → order_numbers list is empty, no crash."""
|
||||
rows = [
|
||||
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor", order_number=None),
|
||||
]
|
||||
result = dedup(rows)
|
||||
assert result[0].order_numbers == []
|
||||
```
|
||||
|
|
@ -18,7 +18,7 @@ Canonical, git-tracked record of **what shipped, what didn't, and the reason or
|
|||
| 6 | Convex-Signal spike **design** | Pi | **SHIPPED** | docs/convex-signal-spike-design-2026-06-24.md. 40h/3-phase plan; Module map + Repository interface + Convex schema + Auth seam (Convex default, Clerk swappable) + Phase 1 (core readiness loop) = what Claude builds first. | Claude builds Phase 1 from it |
|
||||
| 7 | `signal-convex` repo creation | Claude (GitHub) + Kisa/token (Forgejo) | **PENDING** | Dependency: Pi design + Kisa approval (create once, correctly). Claude can do GitHub via `gh`; Forgejo has no API token here. | after design approved |
|
||||
| 8 | Convex BAA email (plan/cost/signing) | Kisa | **IN PROGRESS** | Kisa handling | Kisa sends |
|
||||
| 9 | Readiness-model build (rename coverage→readiness, rules-to-config, grouping+verdict, device override, require-plan-type, frontend nesting) | Claude | **BLOCKED** | Dependency: Kisa's dedup decision (same patient + device + DOS under two order numbers → one worklist line or two?). Design ready: `docs/readiness-model-brief-2026-06-23.md` | Kisa decides → Claude builds |
|
||||
| 9 | Readiness-model build (rename coverage→readiness, rules-to-config, grouping+verdict, device override, require-plan-type, frontend nesting) | Claude | **UNBLOCKED** | Dedup decision locked: (patient_id, device_type, DOS). Order_number is display-only. Scales to all DMEPOS categories. Design: `docs/dedup-design-2026-06-24.md`. | Claude builds from dedup design + readiness model brief |
|
||||
| 10 | NocoDB leads / CRM cleanup | Claude + Kisa | **BLOCKED** | Reason: leads table has no primary key (Signal-base data source `bdmhi02nks1pg6g` is corrupted — a fresh table there also came up PK-less; content base is healthy). SQLite can't add a PK to a populated table (Route 1 failed). Interim: capture still works (inserts via n8n `signal-cgm-lead`); the 2 test rows can't be deleted. | resolved by the data-platform decision (item 11) |
|
||||
| 11 | Data-platform decision (leads + future PHI backend) | Kisa (with Claude/Pi analysis) | **OPEN** | Options analyzed: repaired-NocoDB / Supabase Postgres / Neon / AWS (already set up, BAA+RDS+S3 parked) / Convex ($25/mo incl. BAA). Leads = non-PHI (stay outside PHI boundary). Decision informed by the Convex spike. | run spike → decide |
|
||||
| 12 | Research-versioning Layer-1 (`research-autopush.sh` + Stop-hook entry) | Claude | **NOT STARTED** | Dependency: Pi design exists (`docs/research-versioning-design-2026-06-24.md`). Lower priority than product. | next infra slot |
|
||||
|
|
|
|||
Loading…
Reference in a new issue