Data Quality Review · Mindex Code Challenge

Run summary

Every figure below is produced by a single reproducible pipeline run anchored on as-of date 2026-06-02 — never on wall-clock time, so the numbers are the same on every execution.

Defect classes detected

17 / 17

Detected count equals the seeded expectation for every class.

Raw rows ingested

553

stores 16 · products 32 · transactions 505

Rows loaded to warehouse

553 removed: exact duplicates dropped, integrity failures and non-events quarantined. Nothing deleted silently.

Discount preserved (TX-03)

—

revenue_reconciliation metric absent from the bundle.

Defect coverage

Detected count versus the count seeded by scripts/seed_data.py, per defect class. A red cell means the pipeline's own assertion failed — this strip is the reason the pipeline exits non-zero on a miss.

critical4high6medium6low1

Critical findings

The four defects where the obvious handling is the wrong handling. Each row states the decision taken and links to the code that takes it.

Code	Finding	Rows	Decision
ST-02	Near-duplicate primary key with conflicting attributes	1	Apply an explicit, order-independent survivorship rule instead of keep='first': for each store_id, prefer the record with the fewest nulls; break ties on the earliest opened_date; break remaining ties on the lexicographically first store_name. Here that elects 'Downtown Rochester'. The losing variant and the reason it lost are both written to the audit ledger.
PR-02	Undocumented price change masquerading as a duplicate	1	Treat it as a slowly-changing attribute, not a duplicate. Both versions are quarantined to output/quarantine/products__PR-02.csv with the delta stated. For dim_product, which must have one row per product_id, the later-appearing (higher) price is elected as the current list price on the explicit assumption that the appended record is the newer extract; the assumption is written into the audit note so it can be contradicted.
TX-01	Three date formats in one column	20	Parse per format with explicit format strings and zero reliance on inference. All 20 non-ISO rows are recovered; the parser asserts that no row is left unparsed.
TX-03	Silent discount: reported total does not equal quantity x unit price	20	PRESERVE the reported total_amount verbatim -- it is authoritative for revenue. Add extended_amount = quantity * unit_price as the list value, expose discount_amount = extended_amount - total_amount, and set has_discount = True. Nothing is overwritten and no row is removed.

Row reconciliation

Raw in, clean out, per dataset. Every row of the difference is accounted for by a defect decision on the Defect Explorer — no row disappears without a ledger entry.

Dataset	Raw rows	Delta
stores	16	-16
products	32	-32
transactions	505	-505

Headline counters

Coverage strip

Critical findings

Row reconciliation