Signal/docs/dedup-design-2026-06-24.md

Dedup Design — Patient + Device + DOS Grouping

Date: 2026-06-24 Design by: Pi (decision locked by Kisa) Builds by: Claude (Phase: patient/line/shipment grouping + required-vs-input verdict) Source: docs/readiness-model-brief-2026-06-23.md, Kisa dedup decision 2026-06-24 Verification fixes (Claude, 2026-06-24): (1) added a separate client-mapped plan_type field, distinct from display payer, so CoverageLine keys on the mapped plan type and never the payer name (locked 2026-06-23); (2) fixed merge_quantities to sum across distinct component configs instead of taking a blanket max.

---

1. The Locked Dedup Key

(patient_id, device_type, date_of_service)

Three fields, no more. Order_number is displayed as an attribute, never used in dedup logic.

Scales to all DMEPOS categories (CGM, CPAP, wheelchair, oxygen, wig, bed, ostomy). The rationale is documented in the audit at docs/review-audit-delegation-research-versioning-2026-06-24.md and the Kisa decision in context/current-state.md.

---

2. Where Dedup Fits in the Pipeline

The current pipeline grades ONE record at a time (the known bug). The fixed pipeline:

CSV rows
  │
  ├── Step 1: Normalize (map field names, parse dates, validate required fields)
  │
  ├── Step 2: DEDUP (this design)
  │     Group rows by (patient_id, device_type, DOS)
  │     → MergedShipment (one per group)
  │
  ├── Step 3: Assign to CoverageLines
  │     Each MergedShipment → coverageLine (patient + device + plan)
  │     Latest DOS per line → isCurrent = true
  │
  ├── Step 4: Grade current line
  │     gradeLine() → verdict per CoverageLine
  │
  └── Step 5: Roll up
        rollupPatient() → patient-level status

---

3. MergedShipment Shape (After Dedup)

@dataclass
class MergedShipment:
    """One deduped shipment after grouping CSV rows by (patient_id, device_type, DOS)."""
    patient_id: str
    device_type: str
    shipment_date: date
    total_quantity: int                     # summed from all rows in the group
    components: list[str]                   # combined set, deduped
    order_numbers: list[str]                # collected, displayed only
    hcpcs_codes: list[str]                  # collected, displayed only
    payer: str                              # raw payer NAME from the LATEST row — DISPLAY ONLY
    plan_type: str                          # REQUIRED client-mapped plan type (medicare|medicare_advantage|medicaid|commercial|unknown); NEVER derived from payer name; used for CoverageLine keying + grading
    # CSV doc fields — merged from all rows, latest non-null wins
    csv_visit_date: date | None
    csv_swo_status: str | None
    csv_pecos_verified: str | None
    csv_pa_status: str | None
    csv_diagnosis_on_file: str | None
    csv_transfer_from: str | None

---

4. Tiebreaker Rules (Within a Dedup Group)

When multiple CSV rows have the same (patient_id, device_type, DOS):

Field	Rule	Why
quantity	Sum across rows	Each row represents components; total is the sum of all components delivered
components	Combine into set, dedup duplicates	A row with [sensor] + a row with [sensor, transmitter] → [sensor, transmitter]
order_numbers	Collect into list, dedup duplicates	Displayed as "ORD-100, ORD-101" — collapsed to "ORD-100, ORD-101 (2 orders)" if same number
hcpcs_codes	Collect into list, dedup duplicates	Displayed as "A4230, A4231"
payer	Latest non-null wins	Last row processed in the group. Rows are processed in ingestion order.
csv_* doc fields	Latest non-null wins	If one row has csv_swo_status="On File" and another has null, the non-null wins. If both have different values, latest wins.
shipment_date	Must be identical after normalization to date	If two rows have different dates after normalization, they belong to different groups. If same calendar date but different timestamps, normalize to date first.

Quantity edge cases

Scenario	Input	Result	Rationale
Normal split	Row A qty=3, Row B qty=1	total_qty = 4	Sensor + transmitter = 4 components
Duplicate	Row A qty=3, Row B qty=3 (same components)	total_qty = 6	⚠️ This IS the risk of summing: a data-entry duplicate doubles the quantity. Mitigation: flag rows where components are identical across rows in the same group. If all fields except order_number match exactly, it is a likely duplicate → take max, not sum.
Zero row	Row A qty=3, Row B qty=0	total_qty = 3	Zero-quantity rows are informational (e.g., a cancelation) and do not add to quantity
Wildly different	Row A qty=3, Row B qty=100 (same component)	total_qty = 100	Almost certainly an error. Same component config, so take max (100, not sum); flag for review and log a warning.

Recommended default for quantity ambiguity

Sum, but guard against identical-component duplicates. The rule:

def merge_quantities(rows_in_group: list[CSVRow]) -> int:
    """Sum quantities across DISTINCT component configs within a dedup group,
    while collapsing exact-duplicate rows.

    Group rows by their component set. Within one config, identical rows are
    likely data-entry duplicates, so take the MAX for that config (not the sum).
    Then SUM across distinct configs (a real component split, e.g. sensor + transmitter).

    Fixes the mixed case: two duplicate [sensor] rows + one [transmitter] row
    -> max(sensor)=3 + max(transmitter)=1 = 4, not a blanket global max of 3.
    """
    from collections import defaultdict

    by_config: dict[frozenset, list[int]] = defaultdict(list)
    for row in rows_in_group:
        config = frozenset(row.components.split(","))
        by_config[config].append(row.quantity)

    # Per distinct config: max (defend against duplicate rows). Across configs: sum.
    return sum(max(qtys) for qtys in by_config.values())

---

5. How This Feeds CoverageLines

Each MergedShipment is assigned to a CoverageLine keyed by (patient_id, device_type, plan_type_id).

def assign_to_coverage_lines(merged_shipments: list[MergedShipment]) -> dict[CoverageLineKey, CoverageLine]:
    """Group shipments into CoverageLines by (patient_id, device_type, plan_type)."""
    lines: dict[CoverageLineKey, list[MergedShipment]] = {}

    for shipment in merged_shipments:
        # plan_type is the client-mapped REQUIRED field, never the payer name (locked 2026-06-23)
        key = CoverageLineKey(
            patient_id=shipment.patient_id,
            device_type=shipment.device_type,
            plan_type_id=shipment.plan_type,  # mapped plan type; payer name is display-only
        )
        lines.setdefault(key, []).append(shipment)

    result = {}
    for key, shipments in lines.items():
        # Sort by DOS, latest = current
        shipments.sort(key=lambda s: s.shipment_date, reverse=True)
        current = shipments[0]
        gradeable = is_cgm(key.device_type)  # CGM only for Phase 1; expand later

        result[key] = CoverageLine(
            patient_id=key.patient_id,
            device_type=key.device_type,
            plan_type=key.plan_type_id,
            gradeable=gradeable,
            current_shipment=current,
            prior_shipments=shipments[1:],
            line_status=None,   # computed by gradeLine()
            timing_flag=None,   # computed by gradeLine()
        )

    return result

---

6. Frontend Display After Dedup

Before dedup (current behavior — the bug):

Patient  | Device   | DOS       | Order  | Status
ABC-123  | Dex G7   | 2026-06-01| ORD-100| Resupply Ready
ABC-123  | Dex G7   | 2026-06-01| ORD-101| Resupply Ready

After dedup:

Patient  | Device   | DOS       | Orders             | Qty | Components          | Status
ABC-123  | Dex G7   | 2026-06-01| ORD-100, ORD-101   | 4   | sensor, transmitter  | Resupply Ready

Order numbers column: Shows a comma-separated list. If >3 order numbers, show "ORD-100, ORD-101 +2 more" with expand/collapse.

Components column: Shows the combined set. If only one component, show "sensor". If multiple, show "sensor, transmitter".

Quantity column: Shows the sum. Hover shows detail: "3 sensors + 1 transmitter" is derived from components + quantity (not explicitly stored, but inferred from the component splits).

---

7. Edge Cases Log

Edge Case	Handled?	How
Only one row for the key	✅	No merging needed; MergedShipment = the single row
Missing quantity	✅	Treat quantity as 1 (a row without a qty is still a shipment)
Missing components	✅	Default to the device_type's default component from payer_rules.json
Missing order_number	✅	Order_numbers list is empty; display shows "—"
No order_numbers at all in CSV	✅	Entire column hidden
DOS different due to timezone	✅	Normalize to YYYY-MM-DD UTC before comparison
Three+ orders for same patient+device+DOS	✅	Sum quantities, combine all order numbers
Same patient, same DOS, different device	✅	Different device = different CoverageLine; no dedup conflict
Same patient, same device, different DOS	✅	Different DOS = different MergedShipment; no dedup conflict

---

8. Implementation Notes for Claude

Where the change lives

Layer	File	Change
Dedup step	NEW: python-backend/core/dedup.py	Group rows → MergedShipment
Pipeline orchestration	python-backend/core/coverage_calculator.py	Call dedup BEFORE gradeLine, pass MergedShipments instead of raw rows
Data model	python-backend/core/coverage_calculator.py	Add MergedShipment dataclass, add CoverageLine dataclass
Normalizer	python-backend/api/normalizer.py	No change — still normalizes individual rows; dedup happens after
Frontend	signal-ui/src/components/WorklistTable.jsx	Display order_numbers as list, components as set, quantity as sum

Test scenarios to cover

def test_dedup_component_split():
    """Two rows, same patient+device+DOS, different components → one MergedShipment."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=1, component="transmitter"),
    ]
    result = dedup(rows)
    assert len(result) == 1
    assert result[0].total_quantity == 4
    assert set(result[0].components) == {"sensor", "transmitter"}
    assert len(result[0].order_numbers) == 2  # Two different orders

def test_dedup_identical_duplicate():
    """Two rows, identical data → max quantity, not sum (duplicate guard)."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
    ]
    result = dedup(rows)
    assert len(result) == 1
    assert result[0].total_quantity == 3  # max, not 6

def test_dedup_different_dos():
    """Same patient+device, different DOS → two separate shipments."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-07-01", qty=3, component="sensor"),
    ]
    result = dedup(rows)
    assert len(result) == 2

def test_dedup_different_device():
    """Same patient+DOS, different device → two separate shipments."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
        CSVRow(patient_id="P1", device="omnipod_5", dos="2026-06-01", qty=10, component="pod"),
    ]
    result = dedup(rows)
    assert len(result) == 2

def test_dedup_no_quantity():
    """Missing quantity → default to 1."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=None, component="sensor"),
    ]
    result = dedup(rows)
    assert result[0].total_quantity == 1

def test_dedup_missing_order_number():
    """No order_number in CSV → order_numbers list is empty, no crash."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor", order_number=None),
    ]
    result = dedup(rows)
    assert result[0].order_numbers == []