270 lines
11 KiB
Markdown
270 lines
11 KiB
Markdown
# Dedup Design — Patient + Device + DOS Grouping
|
|
|
|
**Date:** 2026-06-24
|
|
**Design by:** Pi (decision locked by Kisa)
|
|
**Builds by:** Claude (Phase: patient/line/shipment grouping + required-vs-input verdict)
|
|
**Source:** docs/readiness-model-brief-2026-06-23.md, Kisa dedup decision 2026-06-24
|
|
|
|
---
|
|
|
|
## 1. The Locked Dedup Key
|
|
|
|
```
|
|
(patient_id, device_type, date_of_service)
|
|
```
|
|
|
|
Three fields, no more. Order_number is displayed as an attribute, never used in dedup logic.
|
|
|
|
Scales to all DMEPOS categories (CGM, CPAP, wheelchair, oxygen, wig, bed, ostomy). The rationale is documented in the audit at `docs/review-audit-delegation-research-versioning-2026-06-24.md` and the Kisa decision in `context/current-state.md`.
|
|
|
|
---
|
|
|
|
## 2. Where Dedup Fits in the Pipeline
|
|
|
|
The current pipeline grades ONE record at a time (the known bug). The fixed pipeline:
|
|
|
|
```
|
|
CSV rows
|
|
│
|
|
├── Step 1: Normalize (map field names, parse dates, validate required fields)
|
|
│
|
|
├── Step 2: DEDUP (this design)
|
|
│ Group rows by (patient_id, device_type, DOS)
|
|
│ → MergedShipment (one per group)
|
|
│
|
|
├── Step 3: Assign to CoverageLines
|
|
│ Each MergedShipment → coverageLine (patient + device + plan)
|
|
│ Latest DOS per line → isCurrent = true
|
|
│
|
|
├── Step 4: Grade current line
|
|
│ gradeLine() → verdict per CoverageLine
|
|
│
|
|
└── Step 5: Roll up
|
|
rollupPatient() → patient-level status
|
|
```
|
|
|
|
---
|
|
|
|
## 3. MergedShipment Shape (After Dedup)
|
|
|
|
```python
|
|
@dataclass
|
|
class MergedShipment:
|
|
"""One deduped shipment after grouping CSV rows by (patient_id, device_type, DOS)."""
|
|
patient_id: str
|
|
device_type: str
|
|
shipment_date: date
|
|
total_quantity: int # summed from all rows in the group
|
|
components: list[str] # combined set, deduped
|
|
order_numbers: list[str] # collected, displayed only
|
|
hcpcs_codes: list[str] # collected, displayed only
|
|
payer: str # taken from the LATEST row in the group
|
|
# CSV doc fields — merged from all rows, latest non-null wins
|
|
csv_visit_date: date | None
|
|
csv_swo_status: str | None
|
|
csv_pecos_verified: str | None
|
|
csv_pa_status: str | None
|
|
csv_diagnosis_on_file: str | None
|
|
csv_transfer_from: str | None
|
|
```
|
|
|
|
---
|
|
|
|
## 4. Tiebreaker Rules (Within a Dedup Group)
|
|
|
|
When multiple CSV rows have the same `(patient_id, device_type, DOS)`:
|
|
|
|
| Field | Rule | Why |
|
|
|-------|------|-----|
|
|
| `quantity` | **Sum** across rows | Each row represents components; total is the sum of all components delivered |
|
|
| `components` | **Combine into set**, dedup duplicates | A row with [sensor] + a row with [sensor, transmitter] → [sensor, transmitter] |
|
|
| `order_numbers` | **Collect into list**, dedup duplicates | Displayed as "ORD-100, ORD-101" — collapsed to "ORD-100, ORD-101 (2 orders)" if same number |
|
|
| `hcpcs_codes` | **Collect into list**, dedup duplicates | Displayed as "A4230, A4231" |
|
|
| `payer` | **Latest non-null wins** | Last row processed in the group. Rows are processed in ingestion order. |
|
|
| `csv_*` doc fields | **Latest non-null wins** | If one row has csv_swo_status="On File" and another has null, the non-null wins. If both have different values, latest wins. |
|
|
| `shipment_date` | **Must be identical** after normalization to date | If two rows have different dates after normalization, they belong to different groups. If same calendar date but different timestamps, normalize to date first. |
|
|
|
|
### Quantity edge cases
|
|
|
|
| Scenario | Input | Result | Rationale |
|
|
|----------|-------|--------|-----------|
|
|
| Normal split | Row A qty=3, Row B qty=1 | total_qty = 4 | Sensor + transmitter = 4 components |
|
|
| Duplicate | Row A qty=3, Row B qty=3 (same components) | total_qty = 6 | ⚠️ This IS the risk of summing: a data-entry duplicate doubles the quantity. Mitigation: flag rows where components are identical across rows in the same group. If all fields except order_number match exactly, it is a likely duplicate → take max, not sum. |
|
|
| Zero row | Row A qty=3, Row B qty=0 | total_qty = 3 | Zero-quantity rows are informational (e.g., a cancelation) and do not add to quantity |
|
|
| Wildly different | Row A qty=3, Row B qty=100 | total_qty = 103 | This is almost certainly an error. Flag for review. For the pilot, take max (100) and log a warning. |
|
|
|
|
### Recommended default for quantity ambiguity
|
|
|
|
**Sum, but guard against identical-component duplicates.** The rule:
|
|
|
|
```python
|
|
def merge_quantities(rows_in_group: list[CSVRow]) -> int:
|
|
"""Sum quantities within a dedup group.
|
|
If two rows have identical components, take max instead of sum
|
|
(likely a data-entry duplicate, not a real split)."""
|
|
component_sets = [set(row.components.split(",")) for row in rows_in_group]
|
|
unique_component_configs = set(frozenset(cs) for cs in component_sets)
|
|
|
|
if len(unique_component_configs) < len(rows_in_group):
|
|
# At least one component config appears more than once
|
|
# Take the maximum quantity, not the sum (defense against dupes)
|
|
return max(row.quantity for row in rows_in_group)
|
|
|
|
return sum(row.quantity for row in rows_in_group)
|
|
```
|
|
|
|
---
|
|
|
|
## 5. How This Feeds CoverageLines
|
|
|
|
Each `MergedShipment` is assigned to a `CoverageLine` keyed by `(patient_id, device_type, plan_type_id)`.
|
|
|
|
```python
|
|
def assign_to_coverage_lines(merged_shipments: list[MergedShipment]) -> dict[CoverageLineKey, CoverageLine]:
|
|
"""Group shipments into CoverageLines by (patient_id, device_type, plan_type)."""
|
|
lines: dict[CoverageLineKey, list[MergedShipment]] = {}
|
|
|
|
for shipment in merged_shipments:
|
|
# Capline: plan_type comes from client's CSV mapping, not from guessing
|
|
key = CoverageLineKey(
|
|
patient_id=shipment.patient_id,
|
|
device_type=shipment.device_type,
|
|
plan_type_id=shipment.payer, # The client-mapped plan type, not the raw payer name
|
|
)
|
|
lines.setdefault(key, []).append(shipment)
|
|
|
|
result = {}
|
|
for key, shipments in lines.items():
|
|
# Sort by DOS, latest = current
|
|
shipments.sort(key=lambda s: s.shipment_date, reverse=True)
|
|
current = shipments[0]
|
|
gradeable = is_cgm(key.device_type) # CGM only for Phase 1; expand later
|
|
|
|
result[key] = CoverageLine(
|
|
patient_id=key.patient_id,
|
|
device_type=key.device_type,
|
|
plan_type=key.plan_type_id,
|
|
gradeable=gradeable,
|
|
current_shipment=current,
|
|
prior_shipments=shipments[1:],
|
|
line_status=None, # computed by gradeLine()
|
|
timing_flag=None, # computed by gradeLine()
|
|
)
|
|
|
|
return result
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Frontend Display After Dedup
|
|
|
|
Before dedup (current behavior — the bug):
|
|
|
|
```
|
|
Patient | Device | DOS | Order | Status
|
|
ABC-123 | Dex G7 | 2026-06-01| ORD-100| Resupply Ready
|
|
ABC-123 | Dex G7 | 2026-06-01| ORD-101| Resupply Ready
|
|
```
|
|
|
|
After dedup:
|
|
|
|
```
|
|
Patient | Device | DOS | Orders | Qty | Components | Status
|
|
ABC-123 | Dex G7 | 2026-06-01| ORD-100, ORD-101 | 4 | sensor, transmitter | Resupply Ready
|
|
```
|
|
|
|
**Order numbers column:** Shows a comma-separated list. If >3 order numbers, show "ORD-100, ORD-101 +2 more" with expand/collapse.
|
|
|
|
**Components column:** Shows the combined set. If only one component, show "sensor". If multiple, show "sensor, transmitter".
|
|
|
|
**Quantity column:** Shows the sum. Hover shows detail: "3 sensors + 1 transmitter" is derived from components + quantity (not explicitly stored, but inferred from the component splits).
|
|
|
|
---
|
|
|
|
## 7. Edge Cases Log
|
|
|
|
| Edge Case | Handled? | How |
|
|
|-----------|----------|-----|
|
|
| Only one row for the key | ✅ | No merging needed; MergedShipment = the single row |
|
|
| Missing quantity | ✅ | Treat quantity as 1 (a row without a qty is still a shipment) |
|
|
| Missing components | ✅ | Default to the device_type's default component from payer_rules.json |
|
|
| Missing order_number | ✅ | Order_numbers list is empty; display shows "—" |
|
|
| No order_numbers at all in CSV | ✅ | Entire column hidden |
|
|
| DOS different due to timezone | ✅ | Normalize to YYYY-MM-DD UTC before comparison |
|
|
| Three+ orders for same patient+device+DOS | ✅ | Sum quantities, combine all order numbers |
|
|
| Same patient, same DOS, different device | ✅ | Different device = different CoverageLine; no dedup conflict |
|
|
| Same patient, same device, different DOS | ✅ | Different DOS = different MergedShipment; no dedup conflict |
|
|
|
|
---
|
|
|
|
## 8. Implementation Notes for Claude
|
|
|
|
### Where the change lives
|
|
|
|
| Layer | File | Change |
|
|
|-------|------|--------|
|
|
| Dedup step | NEW: `python-backend/core/dedup.py` | Group rows → MergedShipment |
|
|
| Pipeline orchestration | `python-backend/core/coverage_calculator.py` | Call dedup BEFORE gradeLine, pass MergedShipments instead of raw rows |
|
|
| Data model | `python-backend/core/coverage_calculator.py` | Add MergedShipment dataclass, add CoverageLine dataclass |
|
|
| Normalizer | `python-backend/api/normalizer.py` | No change — still normalizes individual rows; dedup happens after |
|
|
| Frontend | `signal-ui/src/components/WorklistTable.jsx` | Display order_numbers as list, components as set, quantity as sum |
|
|
|
|
### Test scenarios to cover
|
|
|
|
```python
|
|
def test_dedup_component_split():
|
|
"""Two rows, same patient+device+DOS, different components → one MergedShipment."""
|
|
rows = [
|
|
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
|
|
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=1, component="transmitter"),
|
|
]
|
|
result = dedup(rows)
|
|
assert len(result) == 1
|
|
assert result[0].total_quantity == 4
|
|
assert set(result[0].components) == {"sensor", "transmitter"}
|
|
assert len(result[0].order_numbers) == 2 # Two different orders
|
|
|
|
def test_dedup_identical_duplicate():
|
|
"""Two rows, identical data → max quantity, not sum (duplicate guard)."""
|
|
rows = [
|
|
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
|
|
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
|
|
]
|
|
result = dedup(rows)
|
|
assert len(result) == 1
|
|
assert result[0].total_quantity == 3 # max, not 6
|
|
|
|
def test_dedup_different_dos():
|
|
"""Same patient+device, different DOS → two separate shipments."""
|
|
rows = [
|
|
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
|
|
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-07-01", qty=3, component="sensor"),
|
|
]
|
|
result = dedup(rows)
|
|
assert len(result) == 2
|
|
|
|
def test_dedup_different_device():
|
|
"""Same patient+DOS, different device → two separate shipments."""
|
|
rows = [
|
|
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
|
|
CSVRow(patient_id="P1", device="omnipod_5", dos="2026-06-01", qty=10, component="pod"),
|
|
]
|
|
result = dedup(rows)
|
|
assert len(result) == 2
|
|
|
|
def test_dedup_no_quantity():
|
|
"""Missing quantity → default to 1."""
|
|
rows = [
|
|
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=None, component="sensor"),
|
|
]
|
|
result = dedup(rows)
|
|
assert result[0].total_quantity == 1
|
|
|
|
def test_dedup_missing_order_number():
|
|
"""No order_number in CSV → order_numbers list is empty, no crash."""
|
|
rows = [
|
|
CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor", order_number=None),
|
|
]
|
|
result = dedup(rows)
|
|
assert result[0].order_numbers == []
|
|
```
|