# Dedup Design — Patient + Device + DOS Grouping

**Date:** 2026-06-24
**Design by:** Pi (decision locked by Kisa)
**Builds by:** Claude (Phase: patient/line/shipment grouping + required-vs-input verdict)
**Source:** docs/readiness-model-brief-2026-06-23.md, Kisa dedup decision 2026-06-24

---

## 1. The Locked Dedup Key

```
(patient_id, device_type, date_of_service)
```

Three fields, no more. Order_number is displayed as an attribute, never used in dedup logic.

Scales to all DMEPOS categories (CGM, CPAP, wheelchair, oxygen, wig, bed, ostomy). The rationale is documented in the audit at `docs/review-audit-delegation-research-versioning-2026-06-24.md` and the Kisa decision in `context/current-state.md`.

---

## 2. Where Dedup Fits in the Pipeline

The current pipeline grades ONE record at a time (the known bug). The fixed pipeline:

```
CSV rows
  │
  ├── Step 1: Normalize (map field names, parse dates, validate required fields)
  │
  ├── Step 2: DEDUP (this design)
  │     Group rows by (patient_id, device_type, DOS)
  │     → MergedShipment (one per group)
  │
  ├── Step 3: Assign to CoverageLines
  │     Each MergedShipment → coverageLine (patient + device + plan)
  │     Latest DOS per line → isCurrent = true
  │
  ├── Step 4: Grade current line
  │     gradeLine() → verdict per CoverageLine
  │
  └── Step 5: Roll up
        rollupPatient() → patient-level status
```

---

## 3. MergedShipment Shape (After Dedup)

```python
@dataclass
class MergedShipment:
    """One deduped shipment after grouping CSV rows by (patient_id, device_type, DOS)."""
    patient_id: str
    device_type: str
    shipment_date: date
    total_quantity: int                     # summed from all rows in the group
    components: list[str]                   # combined set, deduped
    order_numbers: list[str]                # collected, displayed only
    hcpcs_codes: list[str]                  # collected, displayed only
    payer: str                              # taken from the LATEST row in the group
    # CSV doc fields — merged from all rows, latest non-null wins
    csv_visit_date: date | None
    csv_swo_status: str | None
    csv_pecos_verified: str | None
    csv_pa_status: str | None
    csv_diagnosis_on_file: str | None
    csv_transfer_from: str | None
```

---

## 4. Tiebreaker Rules (Within a Dedup Group)

When multiple CSV rows have the same `(patient_id, device_type, DOS)`:

| Field | Rule | Why |
|-------|------|-----|
| `quantity` | **Sum** across rows | Each row represents components; total is the sum of all components delivered |
| `components` | **Combine into set**, dedup duplicates | A row with [sensor] + a row with [sensor, transmitter] → [sensor, transmitter] |
| `order_numbers` | **Collect into list**, dedup duplicates | Displayed as "ORD-100, ORD-101" — collapsed to "ORD-100, ORD-101 (2 orders)" if same number |
| `hcpcs_codes` | **Collect into list**, dedup duplicates | Displayed as "A4230, A4231" |
| `payer` | **Latest non-null wins** | Last row processed in the group. Rows are processed in ingestion order. |
| `csv_*` doc fields | **Latest non-null wins** | If one row has csv_swo_status="On File" and another has null, the non-null wins. If both have different values, latest wins. |
| `shipment_date` | **Must be identical** after normalization to date | If two rows have different dates after normalization, they belong to different groups. If same calendar date but different timestamps, normalize to date first. |

### Quantity edge cases

| Scenario | Input | Result | Rationale |
|----------|-------|--------|-----------|
| Normal split | Row A qty=3, Row B qty=1 | total_qty = 4 | Sensor + transmitter = 4 components |
| Duplicate | Row A qty=3, Row B qty=3 (same components) | total_qty = 6 | ⚠️ This IS the risk of summing: a data-entry duplicate doubles the quantity. Mitigation: flag rows where components are identical across rows in the same group. If all fields except order_number match exactly, it is a likely duplicate → take max, not sum. |
| Zero row | Row A qty=3, Row B qty=0 | total_qty = 3 | Zero-quantity rows are informational (e.g., a cancelation) and do not add to quantity |
| Wildly different | Row A qty=3, Row B qty=100 | total_qty = 103 | This is almost certainly an error. Flag for review. For the pilot, take max (100) and log a warning. |

### Recommended default for quantity ambiguity

**Sum, but guard against identical-component duplicates.** The rule:

```python
def merge_quantities(rows_in_group: list[CSVRow]) -> int:
    """Sum quantities within a dedup group.
    If two rows have identical components, take max instead of sum
    (likely a data-entry duplicate, not a real split)."""
    component_sets = [set(row.components.split(",")) for row in rows_in_group]
    unique_component_configs = set(frozenset(cs) for cs in component_sets)

    if len(unique_component_configs) < len(rows_in_group):
        # At least one component config appears more than once
        # Take the maximum quantity, not the sum (defense against dupes)
        return max(row.quantity for row in rows_in_group)

    return sum(row.quantity for row in rows_in_group)
```

---

## 5. How This Feeds CoverageLines

Each `MergedShipment` is assigned to a `CoverageLine` keyed by `(patient_id, device_type, plan_type_id)`.

```python
def assign_to_coverage_lines(merged_shipments: list[MergedShipment]) -> dict[CoverageLineKey, CoverageLine]:
    """Group shipments into CoverageLines by (patient_id, device_type, plan_type)."""
    lines: dict[CoverageLineKey, list[MergedShipment]] = {}

    for shipment in merged_shipments:
        # Capline: plan_type comes from client's CSV mapping, not from guessing
        key = CoverageLineKey(
            patient_id=shipment.patient_id,
            device_type=shipment.device_type,
            plan_type_id=shipment.payer,  # The client-mapped plan type, not the raw payer name
        )
        lines.setdefault(key, []).append(shipment)

    result = {}
    for key, shipments in lines.items():
        # Sort by DOS, latest = current
        shipments.sort(key=lambda s: s.shipment_date, reverse=True)
        current = shipments[0]
        gradeable = is_cgm(key.device_type)  # CGM only for Phase 1; expand later

        result[key] = CoverageLine(
            patient_id=key.patient_id,
            device_type=key.device_type,
            plan_type=key.plan_type_id,
            gradeable=gradeable,
            current_shipment=current,
            prior_shipments=shipments[1:],
            line_status=None,   # computed by gradeLine()
            timing_flag=None,   # computed by gradeLine()
        )

    return result
```

---

## 6. Frontend Display After Dedup

Before dedup (current behavior — the bug):

```
Patient  | Device   | DOS       | Order  | Status
ABC-123  | Dex G7   | 2026-06-01| ORD-100| Resupply Ready
ABC-123  | Dex G7   | 2026-06-01| ORD-101| Resupply Ready
```

After dedup:

```
Patient  | Device   | DOS       | Orders             | Qty | Components          | Status
ABC-123  | Dex G7   | 2026-06-01| ORD-100, ORD-101   | 4   | sensor, transmitter  | Resupply Ready
```

**Order numbers column:** Shows a comma-separated list. If >3 order numbers, show "ORD-100, ORD-101 +2 more" with expand/collapse.

**Components column:** Shows the combined set. If only one component, show "sensor". If multiple, show "sensor, transmitter".

**Quantity column:** Shows the sum. Hover shows detail: "3 sensors + 1 transmitter" is derived from components + quantity (not explicitly stored, but inferred from the component splits).

---

## 7. Edge Cases Log

| Edge Case | Handled? | How |
|-----------|----------|-----|
| Only one row for the key | ✅ | No merging needed; MergedShipment = the single row |
| Missing quantity | ✅ | Treat quantity as 1 (a row without a qty is still a shipment) |
| Missing components | ✅ | Default to the device_type's default component from payer_rules.json |
| Missing order_number | ✅ | Order_numbers list is empty; display shows "—" |
| No order_numbers at all in CSV | ✅ | Entire column hidden |
| DOS different due to timezone | ✅ | Normalize to YYYY-MM-DD UTC before comparison |
| Three+ orders for same patient+device+DOS | ✅ | Sum quantities, combine all order numbers |
| Same patient, same DOS, different device | ✅ | Different device = different CoverageLine; no dedup conflict |
| Same patient, same device, different DOS | ✅ | Different DOS = different MergedShipment; no dedup conflict |

---

## 8. Implementation Notes for Claude

### Where the change lives

| Layer | File | Change |
|-------|------|--------|
| Dedup step | NEW: `python-backend/core/dedup.py` | Group rows → MergedShipment |
| Pipeline orchestration | `python-backend/core/coverage_calculator.py` | Call dedup BEFORE gradeLine, pass MergedShipments instead of raw rows |
| Data model | `python-backend/core/coverage_calculator.py` | Add MergedShipment dataclass, add CoverageLine dataclass |
| Normalizer | `python-backend/api/normalizer.py` | No change — still normalizes individual rows; dedup happens after |
| Frontend | `signal-ui/src/components/WorklistTable.jsx` | Display order_numbers as list, components as set, quantity as sum |

### Test scenarios to cover

```python
def test_dedup_component_split():
    """Two rows, same patient+device+DOS, different components → one MergedShipment."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=1, component="transmitter"),
    ]
    result = dedup(rows)
    assert len(result) == 1
    assert result[0].total_quantity == 4
    assert set(result[0].components) == {"sensor", "transmitter"}
    assert len(result[0].order_numbers) == 2  # Two different orders

def test_dedup_identical_duplicate():
    """Two rows, identical data → max quantity, not sum (duplicate guard)."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
    ]
    result = dedup(rows)
    assert len(result) == 1
    assert result[0].total_quantity == 3  # max, not 6

def test_dedup_different_dos():
    """Same patient+device, different DOS → two separate shipments."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-07-01", qty=3, component="sensor"),
    ]
    result = dedup(rows)
    assert len(result) == 2

def test_dedup_different_device():
    """Same patient+DOS, different device → two separate shipments."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor"),
        CSVRow(patient_id="P1", device="omnipod_5", dos="2026-06-01", qty=10, component="pod"),
    ]
    result = dedup(rows)
    assert len(result) == 2

def test_dedup_no_quantity():
    """Missing quantity → default to 1."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=None, component="sensor"),
    ]
    result = dedup(rows)
    assert result[0].total_quantity == 1

def test_dedup_missing_order_number():
    """No order_number in CSV → order_numbers list is empty, no crash."""
    rows = [
        CSVRow(patient_id="P1", device="dexcom_g7", dos="2026-06-01", qty=3, component="sensor", order_number=None),
    ]
    result = dedup(rows)
    assert result[0].order_numbers == []
```