How I try to keep myself from over-interpreting a result.
This is the short reference for the research-design choices I use across the site. I keep it separate so every project does not need to explain the same statistical ideas from scratch.
I separate discovery from validation
If I found a pattern while exploring a dataset, I treat it as exploratory. A stronger test freezes the relevant specification and checks it on data that did not help create the candidate.
I keep failed results visible
Nulls, broken designs, and failed holdouts stay in the record. I do not want the site to become a collection of only the experiments that happened to work.
I report effect shrinkage
If a relationship moves from about −0.188 in discovery to −0.061 in validation, the smaller number is part of the result, not an inconvenient detail.
I state important limitations
Survivorship, incomplete historical membership, measurement constraints, dependence, and non-causal designs are disclosed when they change how a result should be read.
A rank-based correlation. Negative ρ means higher values of one variable tend to line up with lower values of the other. It does not mean a 1% change in X causes a fixed percentage change in Y.
Data that did not help discover or tune the candidate. A holdout makes it harder for me to keep adjusting a specification until one sample looks good.
A test evaluated only on observations that occur after the rule is frozen. This is stronger than a cross-sectional holdout drawn from the same historical period.
Using information that would not have been known at the historical decision point. This can create a very convincing backtest that could never have existed in real time.
A resampling procedure that keeps related observations together, for example by stock, quarter, or sector. I use it when treating every row as independent would understate dependence.
Studying only entities that survive into a later universe can make history look cleaner than it was. The equity-weakness project uses current Nifty 500 membership and explicitly documents that limitation.