The pre-flight checklist
Seven rules I run before trusting any dataset. Every one was learned from a real catch in the nine studies on this site, and none of those errors announced itself: nothing crashed, no warning appeared. The numbers simply came out wrong, and confidently so.
Read the questionnaire, never just the variable name
In the rent and health study, the variable I first picked as "housing hardship" turned out, on a careful read of the actual survey questions, to measure general money trouble. I switched to the housing-specific measure before running anything; the study's core claim would have been mislabeled otherwise.
The habit: open the codebook and the questionnaire before the data.
Strip and normalize strings before matching anything
Toronto's planning tracker writes some status labels with an invisible trailing space. My exact-match filter silently missed 3,799 records and flipped a key odds ratio upside down (the full story, with the before-and-after chart). One line of cleaning code fixed it.
The habit: trim, case-fold, and de-punctuate keys as a reflex, never as a repair.
Count your join failures out loud
Joining building audits to neighbourhood incomes, 239 buildings dropped away because two city datasets punctuate the same neighbourhood names differently (building standards study). A silent drop of 7% of the sample would have survived to publication if the join had not been made to report its own losses.
The habit: assert the expected row count after every merge; investigate every miss.
Ask whether the measurement itself changed mid-series
RentSafeTO rebuilt its scoring scheme in 2023. Scores from the two eras look alike, sit in similar columns, and average fifteen points apart for reasons that have nothing to do with buildings (scores over time). Every trend claim on that page stays inside one era.
The habit: before charting any series, ask what changed about the ruler.
Distrust dates that arrive in crowds
308 planning applications share a single "submission" date in May 2020: an administrative batch, wearing the costume of an intake surge (planning queue study). Medians shrug it off; the affected percentile figures are flagged on the chart rather than passed along as fact.
The habit: histogram every date column before believing any of them.
Learn what the source hides on purpose, and what it cannot know yet
Federal asylum counts are rounded to fives with small cells suppressed (shelter study); opioid deaths for recent quarters revise upward as coroners' investigations close (opioid study). Both facts sit in the source documentation, and both change what the newest numbers are allowed to mean.
The habit: read the privacy rules and the reporting-lag notes as part of the data.
Check hardest the results that flatter your expectations
The whitespace bug in rule 02 was caught by a smell test, never by the code: an intermediate figure came out at 98%, which was too tidy to be true. The same instinct killed a straight-line model in the shelter study when the fitted line visibly disagreed with the data it claimed to describe.
The habit: the more a result confirms what I expected, the more checking it earns.
Why keep mistakes on display? Because every analyst makes them, and the ones who say otherwise are the ones whose errors ship. What separates careful work is the machinery for catching them: asserts, smell tests, documentation reading, and a written record. Each study's page tells its own catch in full.