A method that looks brilliant in a single backtest is, more often than not, exploiting something that will not survive contact with the future: a leaked feature, a lucky split, or an evaluation rule that quietly bent toward the result. We have learned to treat an unusually good first result as a reason for more scrutiny, not less.
Where it is scientifically appropriate for the method being tested, our checks can include an intentionally-leaky control run: if a deliberately broken, leakage-permitting version of an experiment performs suspiciously close to the clean version, that is a signal something in the clean pipeline may not be as clean as assumed. This is a targeted diagnostic, not a step we run in every research cycle.
We also require chronological evaluation throughout — training only ever sees the past relative to what it is being tested against — and we freeze evaluation rules before a result is seen, so the bar cannot move to fit whatever the data happened to produce.
Finally, a method that survives all of that still has to hold up under robustness testing: shifts in level and scale, added noise, missing data, and being asked to generalize beyond the exact conditions it was built on. Most methods that pass the first checks lose some of their shine here — and that is the point.