PATTERN 001 / MODEL VALIDATION
Build the Skeptic
Into the Model.
Every predictive model should arrive with a safe, symmetric counterargument—an inverse control that asks what would happen if we took the opposite side using exactly the same information.
When a predictive model recommends an action, the most useful question is not always, “How can we make it more confident?” Sometimes the better question is:
What would happen if we took the opposite side using exactly the same information?
That question gives us a reusable engineering pattern. An inverse control is not a second production strategy. It is a paper-only counterfactual that receives the same decision-time snapshot, reverses the base probability, and evaluates the opposite action at a price that was actually available.
WHY IT EXISTS
A model can look right for the wrong reasons.
The underlying events may have been unusually easy. Market prices may already contain most of the useful information. A forecast can be directionally correct but worthless after fees, spread, and slippage. Several model rows may even represent the same event.
None of those outcomes proves future profitability. Each one teaches us more than an isolated win rate.
THE IMPLEMENTATION
Symmetry matters more than code volume.
def evaluate_inverse(snapshot, evaluate_base):
base = evaluate_base(snapshot)
if snapshot.get("opposite_ask") is None:
return {"entry": False,
"reason": "missing executable quote"}
probability = 1.0 - base["probability"]
price = snapshot["opposite_ask"]
edge = probability - price - fees(price) - slippage(snapshot)
return {
"paper_only": True,
"probability": probability,
"expected_value": edge,
"entry": edge >= snapshot.get("minimum_edge", 0.04),
}The inverse cannot receive a later forecast, a cleaner quote, or knowledge of the outcome. It must remain inside the original information boundary.
SAFE LIFECYCLE
Generate the skeptic. Do not fund it automatically.
- 01CREATE TOGETHERBuild the base evaluator and offline inverse control from one interface.
- 02VALIDATE SYMMETRICALLYUse identical chronological snapshots, costs, and missing-data rules.
- 03OBSERVE SAFELYRun bankroll-free backtests or isolated forward paper controls.
- 04COMPARE HONESTLYMeasure calibration, drawdown, independent events, and cost-adjusted return.
- 05REVIEW SEPARATELYRequire explicit approval before scheduling even a paper lane.
MEASUREMENT
Win rate is not enough.
Backtests, holdouts, forward paper fills, exchange quotes, and actual fills must remain separate evidence classes. Small samples should be labeled insufficient rather than promoted into a story.
THE WEATHER PROJECT LESSON
From one-off experiment to reusable infrastructure.
We initially ran a dedicated inverse lane for one weather model. It helped expose a counting problem: an internal control appeared in operational totals even though users saw a ten-model product.
The improvement was architectural, not predictive. We archived the dedicated lane and moved its logic into the standard model scaffold. Every new model can now challenge itself without silently consuming capital or dashboard space.
BEYOND MARKETS
The pattern travels.
The inverse is not always literally 1 − probability. Multiclass and continuous systems need a domain-specific counterfactual. The durable idea is to define the strongest meaningful alternative, evaluate it with the same information, and keep it observational until evidence supports a larger role.