Developer documentation API v1

Forecast and strategy evaluation

Measure predictive skill, economic performance and operational reliability separately.

Chronological evaluation windows

Training, validation and held-out test windows are separated in time. A label whose outcome crosses the cutoff is removed from training. — Illustrative split. Purging concerns overlapping label horizons; a temporal gap is not a substitute for checking which information each observation actually contains. View full size ↗

Proper scoring and point error

MAE evaluates numerical point predictions. Brier scores binary probabilities p against outcomes o∈{0,1}; lower is better. Pinball loss evaluates a τ-quantile q with 0<τ<1. Scores compare matched targets, cases and horizons against a prespecified baseline. Results include sample counts and missing-outcome rates.

\begin{aligned}\operatorname{MAE}&=\frac1N\sum_i|y_i-\widehat y_i|\\ \operatorname{Brier}&=\frac1N\sum_i(p_i-o_i)^2\\ \operatorname{Pinball}_{\tau}(y,q)&=(\tau-\mathbf1[y<q])(y-q)\end{aligned}

Chronological validation and leakage

Chronological evaluation uses rolling or expanding training windows followed by validation and an untouched out-of-time segment. Transformations, imputation and calibration are fitted within their training partitions. Training labels whose outcome intervals overlap a held-out interval are purged; an embargo excludes adjacent observations where the split structure would otherwise admit leakage.

Universe membership, delistings, revisions and publication delays belong to the evaluation snapshot. Duplicated stories and related observations are assigned to the same partition group, preventing near-identical evidence from appearing on both sides of a split. Historical eligibility does not remove knowledge embedded in a pretrained language model, so prospective evaluation commits predictions before their outcomes and reduces the opportunity for retrospective contamination.

Calibration and uncertainty

Calibration compares predicted probabilities with realized frequencies, while discrimination measures ordering between outcomes. Interval evaluation records both coverage and width. An extremely wide interval can cover most observations without carrying useful information.

Overlapping horizons induce dependence between forecast errors. Uncertainty estimates preserve this structure through a declared temporal resampling scheme. Evaluation history retains attempted configurations, selection criteria and missing outcomes so selecting the best result from many trials is visible rather than treated as an untouched test.

Reliability diagram and sampling uncertainty

Five synthetic probability bins of 50 independent binary events each, with observed frequencies 14, 34, 48, 64 and 82 percent and 95 percent Wilson intervals. The diagonal represents perfect calibration. — Synthetic teaching fixture, not Seer performance. Bin means are 0.1, 0.3, 0.5, 0.7 and 0.9; positive counts are 7, 17, 24, 32 and 41. Wilson intervals assume independent Bernoulli observations. Financial evaluation with dependent or overlapping outcomes uses the declared temporal resampling method. Calibration alone does not establish trading value. View full size ↗

From forecast to net decision value

q is currency notional, s∈{−1,+1} is exposure direction and R_h is simple return for the actual traded instrument. Short exposure requires a supported instrument and its financing model. Do not substitute expected log return for expected simple return. Costs already included in executable quotes must not be subtracted twice. This formulation supports a predictive trade; a risk-reducing rebalance can have a different objective.

\operatorname{EV}(q,h)=q\,\mathbb E[sR_h]-C_{\mathrm{entry}}-\mathbb E[C_{\mathrm{exit}}]-C_{\mathrm{hold}}-C_{\mathrm{operational}}

Evaluation modes

Mode	Required record	Principal limit
Historical simulation	Point-in-time inputs, frozen rules and a fill/cost model.	Historical selection and incomplete data can bias results.
Prospective shadow	Predictions persisted before their outcomes.	No simulated or actual capital path by itself.
Paper trading	Simulated orders, fills, fees, positions and cash ledger.	Assumed liquidity and fills may differ from live execution.
Live evaluation	Actual orders, fills, fees, funding and reconciled positions.	Results remain specific to period, capacity and market conditions.

Evaluation manifest

json

{
  "evaluation_id": "eval_example_1",
  "mode": "historical_simulation",
  "strategy_revision": "strategy-example-r3",
  "forecast_revision": "aura-example-r1",
  "data_snapshot": "snap_example_1",
  "baseline": "no_change",
  "cost_model_revision": "cost-example-r2",
  "split_policy": "chronological_purged",
  "status": "incomplete",
  "limitations": ["Historical source vintages unavailable for one input"]
}

Strategy and execution metrics

Strategy records combine net return, drawdown, turnover, exposure, liquidity assumptions and cost sensitivity. No-trade decisions, rejected orders, partial fills and failed workflows remain part of the evaluation population.

Execution metrics compare filled quantities with the decision benchmark using side-adjusted prices and explicit fees. Unfilled quantities are reported separately. Paper histories retain their fill assumptions and simulated portfolio state; live histories retain observed venue fills and reconciled account movements.

Coverage and baseline-relative skill

Coverage is the fraction of outcomes inside their forecast intervals. Skill uses a lower-is-better loss on the same eligible cases; it is undefined when baseline loss is zero. Positive skill means improvement over that baseline for this sample and metric, not guaranteed trading profitability.

\widehat C=\frac1N\sum_{i=1}^{N}\mathbf1[l_i\le y_i\le u_i],\qquad \operatorname{Skill}=1-\frac{\overline L_{\mathrm{model}}}{\overline L_{\mathrm{baseline}}}

Continuous ranked probability score

CRPS evaluates an entire predictive distribution against one realized value and has the same units as the target. It rewards informative distributions while penalizing misplaced probability; lower values are better. The distribution, not only a few quantiles, is needed to evaluate the integral.

\operatorname{CRPS}(F,y)=\int_{-\infty}^{\infty}\left(F(z)-\mathbf1[y\le z]\right)^2\,dz

Interpreting a small evaluation sample

If 90 of 100 outcomes fall inside nominal 90% intervals, empirical coverage is 90%. That single number does not establish calibration: the intervals may be unnecessarily wide, coverage may differ by regime, and overlapping horizons may reduce the effective independent sample size.

A baseline loss of 0.020 and model loss of 0.016 produce skill 0.20 on that loss scale. The result describes a 20% relative loss reduction on matched observations. It is neither a 20% expected return nor evidence that fees, latency and sizing have been accounted for.

Mathematical and systems references

Gneiting and Raftery: proper scoring rulesDistributional forecast evaluation and the continuous ranked probability score.