Developer documentation API v1

Reproducible strategy research

Freeze the experiment, account for selection and compare net outcomes.

The experiment is the unit of comparison

research.evaluate receives an immutable strategy revision, evidence snapshot, decision schedule, initial simulated capital, baseline, split policy and execution-model revision. The strategy revision pins prompts, model versions, features, parameters and sizing rules. The evaluation records every attempted variant, including failed runs and discarded thresholds. A final selected result remains linked to that experiment family.

Evaluate a frozen strategy

POST /v1/tool-invocations. The calculation produces a job and tool-result artifact. Reference IDs are selected from the workspace’s immutable research registry.

json

{
  "agent_revision_id": "agr_example_3",
  "tool": "research.evaluate",
  "input": {
    "strategy_revision": "strategy-example-r3",
    "evidence_snapshot_id": "snap_example_1",
    "mode": "historical_simulation",
    "start_at": "2026-01-01T00:00:00Z",
    "end_at": "2026-10-01T00:00:00Z",
    "baseline_revision": "baseline-no-change-r1",
    "execution_model_revision": "execution-example-r2",
    "split_policy_revision": "walk-forward-example-r1",
    "initial_cash": {
      "asset": "USD",
      "atoms": "1000000",
      "scale": 2
    },
    "experiment_family_id": "experiment_example_1"
  }
}

Evaluation contract

Component	Required behavior
Universe	Time-valid membership, delistings and availability; no current-survivor substitution.
Labels and partitions	Decision/label intervals, chronological windows, purging and the declared embargo.
Transformations	Fit normalization, selection and calibration inside training partitions.
Execution	Decision time precedes order eligibility; explicit delay, spread, fees, depth, partial fills and expiry.
Accounting	Self-financing cash/position ledger; corporate actions and financing where applicable.
Comparison	Matched periods, assets, budgets and cost assumptions; preserve missing/no-trade cases.
Artifacts	Manifest, predictions, orders, fills, NAV, metrics, exclusions and trial history.

Historical LLM knowledge is a separate leakage channel

Restricting retrieved documents to a cutoff does not remove later facts in model parameters. Historical LLM-assisted runs therefore retain model release/training-window information when known, anonymization controls and an explicit contamination assessment. A missing training boundary remains unknown. Prospective shadow runs commit outputs before outcomes and keep their history separate from retrospective simulations.

Selection-adjusted evidence

Deflated Sharpe diagnostics account for a selected result’s track-record length, non-normal returns and the effective number of trials. The record retains raw trial count, dependence adjustment, estimator revision and assumptions. If these inputs are unavailable, the diagnostic is unavailable, not a default confidence score.

Probability-of-backtest-overfitting analysis compares selection and evaluation rankings across a declared partition scheme. It diagnoses instability of selection; it does not replace chronological deployment tests or correct an invalid dataset. Repeated inspection of a held-out segment makes it part of the selection process.

Net performance and research cost

V is marked account value after trading and financing costs; F is an external flow applied at the end of the interval. With intraperiod flows, use subperiod chain-linking rather than this simplified formula. Research costs are reported separately unless actually charged to the investment account, preventing double subtraction. Cost per completed decision includes failed and abstaining research attempts in its cost numerator.

R_{\mathrm{net},t}=\frac{V_t-F_t}{V_{t-1}}-1,\qquad C_{\mathrm{decision}}=C_{\mathrm{data}}+C_{\mathrm{inference}}+C_{\mathrm{compute}}

Ablations, sensitivity and capacity

Compare a price-only baseline with additions such as company events, social features and execution context. Remove one feature family at a time under the same decision budget and partitions. Sensitivity runs perturb execution delay, cost, signal thresholds and capital size. Capacity reports include participation, unfilled orders and the capital scale at which constraints bind.

Report matched-sample forecast loss, rank IC, net return, drawdown, expected shortfall, turnover and fees with temporal block uncertainty intervals. A higher Sharpe ratio on a short or selectively chosen window does not establish a durable strategy.

Reading an ablation benchmark

Synthetic baseline loss of 0.020 compared with company-event loss 0.017, social-feature loss 0.019 and full-feature loss 0.016. Relative skills are 0, 15, 5 and 20 percent. — Illustrative comparison on a common loss scale, not a measured Seer benchmark. Each single-family addition is relative to the price-only baseline; the full set includes both. Point estimates show how to read the report, not statistical significance. A real comparison retains matched observations, trial history, cost assumptions and uncertainty. Forecast-loss improvement is not investment return. View full size ↗

Research references

Bailey and López de Prado: Deflated Sharpe RatioSelection bias, track-record length and non-normal returns.Bailey et al.: Probability of Backtest OverfittingSelection instability under repeated strategy searches.Glasserman and Lin: LLM look-ahead biasHistorical model knowledge and company-identity effects.Citadel: inside quantitative researchHypothesis formation, live out-of-sample evaluation and market frictions.

Retrieve the research record

The completed tool result includes evaluation_id, a frozen manifest, metrics and record_refs for predictions, orders, fills, NAV, trials and exclusions. Resolve each reference with GET /v1/research-records/{record_id}. Each paginated collection supplies its row_schema and schema_revision so a client can validate its rows. Currency amounts use exact strings; ratios declare their units. No raw source text is redistributed beyond its entitlement.

Experiment family identity

An experiment family is a registry object created in the workspace Research registry before the first trial. Discover it with GET /v1/research-registry?kind=experiment_family. Its immutable ID groups the thesis and selection protocol; individual trials reference that ID even when their strategy revisions differ. Changing a family label does not reset the recorded trial history or turn an already-inspected sample into a new holdout.