The Background Execution Reconciler
The reconciler is a background service running inside the API process that periodically inspects running executions and reconciles platform state with the workflow engine's actual state. It's the platform's answer to an unavoidable reality: callbacks can be missed, engines can crash, and a run can otherwise be left stranded as running forever. It's also the clearest expression of the platform's most important operating rule.
The Truthfulness Invariant
The reconciler exists to uphold one invariant: the platform must never mark an execution terminal while the engine may still be running. A platform-terminal state over a still-running engine is a critical lie; a stale running state after the engine exited is wrong but operationally safer. Every decision it makes is biased toward that safer error.
What it reconciles, and how it decides
On each tick it scans running executions and classifies each by three questions — does it have an external_execution_id, has it breached a limit, and has the engine finished? — then acts deterministically:
| External ID? | Limit breached? | Engine finished? | Action |
|---|---|---|---|
| yes | no | true | terminate locally (failed_transient), no remote stop |
| yes | yes | false | terminate with remote stop |
| yes | yes | true | terminate locally (failed_transient) |
| no | any | unknown | leave running; log for operator intervention |
Two rules keep it honest. The engine-finished condition is stoppedAt != null AND status is terminal — both, because n8n's Wait nodes set stoppedAt while merely paused, and a naive check would kill waiting runs. And it only ever produces failed_transient, never completed: engine status = success does not authorise the platform to claim completion, because completion needs the result payload that only /complete delivers. Without an external ID it can neither verify nor stop the engine, so it leaves the run alone and flags it for a human.
Single-runner by design
The reconciler must not double-process when several API instances share a database, so each tick acquires a PostgreSQL advisory lock; if the lock is busy, the tick is skipped. The loop is strictly sequential — tick, sleep, tick — with no overlap, and it releases the lock in a finally (destroying the client if unlock fails, which ends the session and frees the lock anyway). Normal endpoints are never blocked by this lock. It's always enabled, runs on a configurable interval (default 120s), and emits telemetry under a reconciler domain — the modest, robust design you'd want from a component whose whole job is to be the thing that's still correct when everything else has already gone wrong.