Developer documentation API v1

Durable jobs and event delivery

Explicit state machines, duplicate-tolerant consumers and recoverable progress.

Durable publication without a distributed commit

A local transaction commits an aggregate change and outbox event. The publisher can redeliver; a consumer inbox deduplicates before applying its own transition. — A crash after publish but before acknowledgment causes redelivery. The consumer’s event identity prevents the same local effect from being applied twice. View full size ↗

Job union

typescript

type Job =
  | { job_id: string; state: "accepted" | "running" | "waiting" }
  | { job_id: string; state: "completed"; result_ref: string }
  | { job_id: string; state: "failed"; error_code: string }
  | { job_id: string; state: "cancelled" };

type DomainEvent = {
  event_id: string;
  schema_version: string;
  aggregate_id: string;
  aggregate_revision: number;
  occurred_at: string;
  type: string;
  data_ref: string;
};

Admission and checkpoints

Admission validates identity, inputs, entitlements and resource limits before persisting a stable job identifier and its configuration revision. The acknowledgment refers to that durable record. Checkpoints associate completed stages with result artifacts, allowing replacement workers to resume without reconstructing state from conversation text.

Scheduled triggers carry a temporal window and an overlap policy: skip, coalesce or enqueue distinct instances. Their stable trigger identity prevents duplicate logical work. Scheduling cadence and evidence freshness are independent constraints.

Transactional state-change publication

The transactional outbox records an event in the same local transaction as its aggregate state change. An asynchronous publisher delivers it; a consumer inbox deduplicates it before applying the receiving transition. External effects retain their own operation identity and reconciliation state.

text

begin local transaction
  require owned aggregate at expected revision
  apply permitted state transition
  append event with unique event_id
  append pending publication record
commit

publisher delivers pending event
consumer atomically records (consumer_id, event_id) with its update
duplicate delivery returns the previously recorded outcome

Delivery and ordering

At-least-once delivery permits duplicates. Per-aggregate revisions help detect gaps and stale events, but do not imply a total order across accounts. A consumer receiving an unexpected revision can retrieve the authoritative aggregate instead of applying events blindly.

A database commit and event publication are not one remote atomic operation. The outbox closes the local commit/publication gap; it does not produce end-to-end exactly-once execution at a venue.

Worker recovery and cancellation

Worker leases and fencing tokens identify the current owner of a job. After ownership changes, writes carrying a stale token are rejected. Recovery starts from the latest committed checkpoint with the original configuration and input revisions.

Cancellation is cooperative and may race with completion. A request to cancel is distinct from a terminal cancelled state. If a job has already initiated a financial action, that action continues to retain its execution and settlement history independently of the job’s cancellation.

Operational evidence

Job telemetry records queue age, time in state, retries, publication delay and classified failures using correlated identifiers. Financial history remains in the authoritative domain records rather than diagnostic logs.

Completion means that the job reached its defined terminal computation state. Coverage, execution and settlement each have their own state, so a completed research job cannot be mistaken for a filled order or settled payment.

Fenced worker mutation

f is the ownership fencing token and r the aggregate revision. A replacement worker advances ownership; a delayed mutation from the old worker then fails even if its process is still running. The predicate applies at the authoritative write boundary. This is logical eligibility for a write, not a guarantee that storage succeeds.

\operatorname{writeEligible}(c)\iff f_c=f_{\mathrm{current}}\land r_c=r_{\mathrm{current}}\land\operatorname{valid}(c)

Safety, liveness and delivery multiplicity

Safety invariants describe events that cannot occur, such as applying the same journal charge twice. Liveness describes eventual progress under stated conditions, such as a healthy dependency and an active worker eventually completing eligible work. A persistent outage can prevent progress without justifying a violation of the safety invariant.

At-least-once delivery permits repeated messages. Exactly-once local application is obtained through a stable event identity and an atomic consumer transition; it is not an assertion of exactly-once network delivery or external venue execution. Aggregate revisions establish local order, while causal references relate records across aggregates.

Mathematical and systems references

Lamport: time, clocks and event orderingCausal partial order and the distinction between logical order and physical time.

Read the overview