# Dedup Design — Patient + Device + DOS Grouping

**Date:** 2026-06-24
**Design by:** Pi (decision locked by Kisa)
**Builds by:** Claude (Phase: patient/line/shipment grouping + required-vs-input verdict)
**Source:** docs/readiness-model-brief-2026-06-23.md, Kisa dedup decision 2026-06-24
**Verification fixes (Claude, 2026-06-24):** (1) added a separate client-mapped `plan_type` field, distinct from display `payer`, so CoverageLine keys on the mapped plan type and never the payer name (locked 2026-06-23); (2) fixed `merge_quantities` to sum across distinct component configs instead of taking a blanket max.

---

## 1. The Locked Dedup Key

```
(patient_id, device_type, date_of_service)
```

Three fields, no more. Order_number is displayed as an attribute, never used in dedup logic.

Scales to all DMEPOS categories (CGM, CPAP, wheelchair, oxygen, wig, bed, ostomy). The rationale is documented in the audit at `docs/review-audit-delegation-research-versioning-2026-06-24.md` and the Kisa decision in `context/current-state.md`.

---

## 2. Where Dedup Fits in the Pipeline

The current pipeline grades ONE record at a time (the known bug). The fixed pipeline:

```
CSV rows
  │
  ├── Step 1: Normalize (map field names, parse dates, validate required fields)
  │
  ├── Step 2: DEDUP (this design)
  │     Group rows by (patient_id, device_type, DOS)
  │     → MergedShipment (one per group)
  │
  ├── Step 3: Assign to CoverageLines
  │     Each MergedShipment → coverageLine (patient + device + plan)
  │     Latest DOS per line → isCurrent = true
  │
  ├── Step 4: Grade current line
  │     gradeLine() → verdict per CoverageLine
  │
  └── Step 5: Roll up
        rollupPatient() → patient-level status
```

---

## 3. MergedShipment Shape (After Dedup)

```python
@dataclass
class MergedShipment:
    """One deduped shipment after grouping CSV rows by (patient_id, device_type, DOS)."""
    patient_id: str
    device_type: str
    shipment_date: date
    total_quantity: int                     # summed from all rows in the group
    components: list[str]                   # combined set, deduped
    order_numbers: list[str]                # collected, displayed only
    hcpcs_codes: list[str]                  # collected, displayed only
    payer: str                              # raw payer NAME from the LATEST row — DISPLAY ONLY
    plan_type: str                          # REQUIRED client-mapped plan type (medicare|medicare_advantage|medicaid|commercial|unknown); NEVER derived from payer name; used for CoverageLine keying + grading
    # CSV doc fields — merged from all rows, latest non-null wins
    csv_visit_date: date | None
    csv_swo_status: str | None
    csv_pecos_verified: str | None
    csv_pa_status: str | None
    csv_diagnosis_on_file: str | None
    csv_transfer_from: str | None
```

---

## 4. Tiebreaker Rules (Within a Dedup Group)

When multiple CSV rows have the same `(patient_id, device_type, DOS)`:

| Field | Rule | Why |
|-------|------|-----|
| `quantity` | **Sum** across rows | Each row represents components; total is the sum of all components delivered |
| `components` | **Combine into set**, dedup duplicates | A row with [sensor] + a row with [sensor, transmitter] → [sensor, transmitter] |
| `order_numbers` | **Collect into list**, dedup duplicates | Displayed as "ORD-100, ORD-101" — collapsed to "ORD-100, ORD-101 (2 orders)" if same number |
| `hcpcs_codes` | **Collect into list**, dedup duplicates | Displayed as "A4230, A4231" |
| `payer` | **Latest non-null wins** | Last row processed in the group. Rows are processed in ingestion order. |
| `csv_*` doc fields | **Latest non-null wins** | If one row has csv_swo_status="On File" and another has null, the non-null wins. If both have different values, latest wins. |
| `shipment_date` | **Must be identical** after normalization to date | If two rows have different dates after normalization, they belong to different groups. If same calendar date but different timestamps, normalize to date first. |

### Quantity edge cases

| Scenario | Input | Result | Rationale |
|----------|-------|--------|-----------|
| Normal split | Row A qty=3, Row B qty=1 | total_qty = 4 | Sensor + transmitter = 4 components |
| Duplicate | Row A qty=3, Row B qty=3 (same components) | total_qty = 6 | ⚠️ This IS the risk of summing: a data-entry duplicate doubles the quantity. Mitigation: flag rows where components are identical across rows in the same group. If all fields except order_number match exactly, it is a likely duplicate → take max, not sum. |
| Zero row | Row A qty=3, Row B qty=0 | total_qty = 3 | Zero-quantity rows are informational (e.g., a cancelation) and do not add to quantity |
| Wildly different | Row A qty=3, Row B qty=100 (same component) | total_qty = 100 | Almost certainly an error. Same component config, so take max (100, not sum); flag for review and log a warning. |

### Recommended default for quantity ambiguity

**Sum, but guard against identical-component duplicates.** The rule:

```python
def merge_quantities(rows_in_group: list[CSVRow]) -> int:
    """Sum quantities across DISTINCT component configs within a dedup group,
    while collapsing exact-duplicate rows.

    Group rows by their component set. Within one config, identical rows are
    likely data-entry duplicates, so take the MAX for that config (not the sum).
    Then SUM across distinct configs (a real component split, e.g. sensor + transmitter).

    Fixes the mixed case: two duplicate [sensor] rows + one [transmitter] row
    -> max(sensor)=3 + max(transmitter)=1 = 4, not a blanket global max of 3.
    """
    from collections import defaultdict

    by_config: dict[frozenset, list[int]] = defaultdict(list)
    for row in rows_in_group:
        config = frozenset(row.components.split(","))
        by_config[config].append(row.quantity)

    # Per distinct config: max (defend against duplicate rows). Across configs: sum.
    return sum(max(qtys) for qtys in by_config.values())
```

---

## 5. How This Feeds CoverageLines

Each `MergedShipment` is assigned to a `CoverageLine` keyed by `(patient_id, device_type, plan_type_id)`.

```python
def assign_to_coverage_lines(merged_shipments: list[MergedShipment]) -> dict[CoverageLineKey, CoverageLine]:
    """Group shipments into CoverageLines by (patient_id, device_type, plan_type)."""
    lines: dict[CoverageLineKey, list[MergedShipment]] = {}

    for shipment in merged_shipments:
        # plan_type is the client-mapped REQUIRED field, never the payer name (locked 2026-06-23)
        key = CoverageLineKey(
            patient_id=shipment.patient_id,
            device_type=shipment.device_type,
            plan_type_id=shipment.plan_type,  # mapped plan type; payer name is display-only
        )
        lines.setdefault(key, []).append(shipment)

    result = {}
    for key, shipments in lines.items():
        # Sort by DOS, latest = current
        shipments.sort(key=lambda s: s.shipment_date, reverse=True)
        current = shipments[0]
        gradeable = is_cgm(key.device_type)  # CGM only for Phase 1; expand later

        result[key] = CoverageLine(
            patient_id=key.patient_id,
            device_type=key.device_type,
            plan_type=key.plan_type_id,
            gradeable=gradeable,
            current_shipment=current,
            prior_shipments=shipments[1:],
            line_status=None,   # computed by gradeLine()
            timing_flag=None,   # computed by gradeLine()
        )

    return result
```

---

## 6. Frontend Display After Dedup

Before dedup (current behavior — the bug):

```
Patient  | Device   | DOS       | Order  | Status
ABC-123  | Dex G7   | 2026-06-01| ORD-100| Resupply Ready
ABC-123  | Dex G7   | 2026-06-01| ORD-101| Resupply Ready
```

After dedup:

```
Patient  | Device   | DOS       | Orders             | Qty | Components          | Status
ABC-123  | Dex G7   | 2026-06-01| ORD-100, ORD-101   | 4   | sensor, transmitter  | Resupply Ready
```

**Order numbers column:** Shows a comma-separated list. If >3 order numbers, show "ORD-100, ORD-101 +2 more" with expand/collapse.

**Components column:** Shows the combined set. If only one component, show "sensor". If multiple, show "sensor, transmitter".

**Quantity column:** Shows the sum. Hover shows detail: "3 sensors + 1 transmitter" is derived from components + quantity (not explicitly stored, but inferred from the component splits).

---

## 7. Edge Cases Log

| Edge Case | Handled? | How |
|-----------|----------|-----|
| Only one row for the key | ✅ | No merging needed; MergedShipment = the single row |
| Missing quantity | ✅ | Treat quantity as 1 (a row without a qty is still a shipment) |
| Missing components | ✅ | Default to the device_type's default component from payer_rules.json |
| Missing order_number | ✅ | Order_numbers list is empty; display shows "—" |
| No order_numbers at all in CSV | ✅ | Entire column hidden |
| DOS different due to timezone | ✅ | Normalize to YYYY-MM-DD UTC before comparison |
| Three+ orders for same patient+device+DOS | ✅ | Sum quantities, combine all order numbers |
| Same patient, same DOS, different device | ✅ | Different device = different CoverageLine; no dedup conflict |
| Same patient, same device, different DOS | ✅ | Different DOS = different MergedShipment; no dedup conflict |

---

## 8. Implementation Notes for Claude

### Where the change lives

| Layer | File | Change |
|-------|------|--------|
| Dedup step | NEW: `python-backend/core/dedup.py` | Group rows → MergedShipment |
| Pipeline orchestration | `python-backend/core/coverage_calculator.py` | Call dedup BEFORE gradeLine, pass MergedShipments instead of raw rows |
| Data model | `python-backend/core/coverage_calculator.py` | Add MergedShipment dataclass, add CoverageLine dataclass |
| Normalizer | `python-backend/api/normalizer.py` | No change — still normalizes individual rows; dedup happens after |
| Frontend | `signal-ui/src/components/WorklistTable.jsx` | Display order_numbers as list, components as set, quantity as sum |

### Test scenarios to cover

```python
def test_dedup_component_split():
    """Two rows, same patient+device+DOS, different components → one MergedShipment."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=1, component="transmitter"),
    ]
    result = dedup(rows)
    assert len(result) == 1
    assert result[0].total_quantity == 4
    assert set(result[0].components) == {"sensor", "transmitter"}
    assert len(result[0].order_numbers) == 2  # Two different orders

def test_dedup_identical_duplicate():
    """Two rows, identical data → max quantity, not sum (duplicate guard)."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
    ]
    result = dedup(rows)
    assert len(result) == 1
    assert result[0].total_quantity == 3  # max, not 6

def test_dedup_different_dos():
    """Same patient+device, different DOS → two separate shipments."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-07-01", qty=3, component="sensor"),
    ]
    result = dedup(rows)
    assert len(result) == 2

def test_dedup_different_device():
    """Same patient+DOS, different device → two separate shipments."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
        CSVRow(patient_id="P1", device="omnipod_5", dos="2026-06-01", qty=10, component="pod"),
    ]
    result = dedup(rows)
    assert len(result) == 2

def test_dedup_no_quantity():
    """Missing quantity → default to 1."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=None, component="sensor"),
    ]
    result = dedup(rows)
    assert result[0].total_quantity == 1

def test_dedup_missing_order_number():
    """No order_number in CSV → order_numbers list is empty, no crash."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor", order_number=None),
    ]
    result = dedup(rows)
    assert result[0].order_numbers == []
```
