Loaded and validated 39.8 million real mortgage records in PostgreSQL, then measured and fixed the slowest queries, up to 177x faster.
The question. Could the rigor I use on regulated healthcare data at my day job hold up on a different regulated domain, at real production scale, if I built the whole pipeline myself? I picked CFPB’s public HMDA mortgage-lending data, three years of it, 2022 through 2024, because it’s large enough to force real engineering decisions and messy enough that a clean answer isn’t free.
What I built. A normalized PostgreSQL schema loaded with 39,793,397 real mortgage applications through an idempotent Python pipeline that counts and logs every cleaning decision instead of silently dropping rows. HMDA’s data turned out to have several undocumented sentinel values (literal "NA" and "Exempt" strings, a numeric 1111 code) that the load script had to detect and handle explicitly. A six-check data-quality suite runs after every load. Twelve analytical SQL queries, each opening with the business question and a real recorded finding, cover window functions, multi-step CTEs, every join type, and GROUPING SETS. Then I measured the four heaviest queries with EXPLAIN (ANALYZE, BUFFERS), applied a materialized view, composite indexes, and year-range partitioning, and re-measured. One query dropped from 5,659ms to 32ms, a real 177.58x, not an estimate.
What I learned. The biggest risks weren’t in the SQL. They were in the assumptions between steps. A missing index turned a routine reload into a multi-hour operation. A statistics-stale query looked like a failed optimization until VACUUM ANALYZE fixed it. A materialized view built one step too early silently detached from the table it was supposed to summarize. None of that shows up until you actually run the thing end to end and check the numbers against reality, which is the same discipline that matters most in production data engineering, on any dataset.