Failure handling

Section 18 of 22

Common recovery states

Every executable job needs a deadline, retry allowance, spending limit, and recovery owner. Its permitted outcomes include successful completion, correction, cancellation, and explicit inability to proceed. The parent flow records which outcome occurred.

Use the following shared states without losing the original step:

WAITING_FOR_INPUT: a named contributor or authority must supply something specific before a deadline.
RETRYING: the failure is considered temporary and the next attempt remains within its budget.
RECOVERY_REQUIRED: automatic retries have failed or authority must change. Record the permitted recovery actor and action.
RECONCILING: an external action may have succeeded. Establish its actual outcome before attempting another effect.
PAUSED_FOR_SAFETY: an identified condition prevents the affected operation from continuing.
CLOSED_UNRESOLVED: this attempt cannot safely complete under its rules. Preserve the evidence, reason, and separately recorded obligations.

These states make stalled work visible and actionable. They do not turn timeout into permission or turn a failed payment attempt into cancellation of a debt.

Failure requirements

Failure and affected steps	Automatic response	Authorized recovery or intervention	Deadline outcome
Gas wallet lacks ETH. U08, U14, settlement, and recovery transactions.	Check gas before dispatch. Request a bounded top-up from an already-authorized operating allowance. Queue the original action once.	Treasury or operations authority supplies funds if the allowance is empty. A gas funder can pay transaction costs without receiving release authority.	Mark the action unfunded if no source exists. Keep unpaid entitlements due. Do not promote on the strength of an unsubmitted transaction.
TEE server is unavailable. Runner, council, or signer jobs.	Detect loss of service, end the worker's assignment when permitted, and assign an authorized replacement.	Recover or replace the key through the previously approved attestation and role policy. Revoke obsolete access when required.	Close the candidate attempt or keep the affected service paused if no authorized replacement exists.
Inference endpoint fails. U02, U05, U10, challenges, and E05.	Retry within count, time, and cost limits. Preserve completed valid runs and mark missing runs incomplete.	Use only a substitute allowed by the applicable policy. A change to the verifier's own selected model creates a changed candidate requiring evaluation.	Unfinished required work prevents acceptance. Never infer a pass from an empty response or expired request.
Required signers coordinate a refusal. U14, governance changes, and discretionary payments.	Publish missing participation and invoke any preauthorized replacement or recovery route. The ordinary executor cannot manufacture the missing signatures.	A separate recovery authority replaces or restores capability under its existing rules. It cannot depend on consent from the same blocking quorum.	Without such a route, the action remains unauthorized. Close the attempt where possible and declare the affected capability unavailable.
A key is lost or compromised. Submission, execution, governance, or payout.	Disable compromised authority where the incident rules permit. Stop using an uncertain recipient key for disputed payments.	Use authenticated role replacement or participant recovery. Update permissions before resuming queued actions.	Preserve pending work and entitlements. A delay alone is insufficient evidence to grant a replacement key.
A transaction's outcome is unknown. U14 and all transfers.	Enter `RECONCILING`. Read the expected state and transaction identity before resubmitting.	Operations resolves inconsistent provider observations under the confirmation policy.	Keep the result unconfirmed. Retrying without reconciliation risks duplicate effects.
Chain access or required confirmations are unavailable. U08, U14, issuance, and payments.	Retry permitted providers and retain pending jobs. Stop transitions that require fresh confirmed state.	Restore access. Follow an explicit product policy for any already-issued evidence that cannot be checked.	Do not label an unknown validity check valid. Defer the action or close the attempt without granting new authority.
Git push, build, or merge fails. U04 through U07.	Retry transport or build work where the same inputs remain valid. Record tool failures separately from failed behavior tests.	Contributor or maintainer resolves conflicts. A changed source tree returns to evaluation.	Cancel an unfinished assignment under its terms after the deadline. Accepted earlier milestones remain recorded.
Two executors race or events are delivered twice. Any execution.	Use the action identifier and current state to claim one execution. Reject stale assignments and repeated effects.	Reconcile any external outcome already produced. Restore the correct permission record.	One recorded outcome per authorized effect. Never allocate a second reward because the same event was delivered twice.
Test endpoint or essential evidence disappears during the window. U11 and C01 through C04.	Stop counting unavailable access time and attempt restoration. A member-window expiry cannot clear this condition.	Restore equivalent access to the same target and disclose the interruption.	Extend for missing qualifying access or restart when equivalence cannot be established. Close the candidate if access cannot be restored.
A council abstains, disagrees, or misses its deadline. Conditional judgments and challenges.	Apply its recorded agreement rule and bounded reassignment allowance. An abstention is not an acceptance.	The designated replacement or appeal authority handles the specific unresolved question.	Record unresolved and close the affected attempt. Existing production continues only if independently permitted.
Candidate or policy becomes stale. U07 through U15.	Detect changes to the base, target, permissions, or applicable rule. Suspend obsolete eligibility.	Re-evaluate the changed dependency or obtain an explicit transition rule through governance.	Supersede or close the stale attempt. Do not reuse signatures for a different candidate.
An actual test failure is found. U06, U10, U11, or E05.	Record the failure and prevent the affected transition.	Implementer corrects the code or corpus authority resolves a mistaken expectation under the proper rules.	Reject or supersede the candidate if no accepted correction arrives. Infrastructure retry limits must not be used to rerun away real failures.
Deployment only partly succeeds. U15 or E07.	Compare contract authorization with actual routing and endpoint state. Disable the affected route where necessary.	Complete activation or restore a still-authorized fallback through the proper permissions.	Declare continued degraded or paused service. Retain the actual state of every completed action.
Token conversion cannot obtain the required payout within its limit. F6 and F7.	Stop the conversion and retry only under the agreed pricing and cost limits.	Use a funded fallback permitted by the quote or obtain the recipient's agreement to amended settlement.	Entitlement remains due or disputed under its terms. An execution limit cannot erase it.
A resource budget is exhausted. Testing, attack bots, hosting, or appeals.	Stop discretionary work and new spending commitments. Keep required unfinished evaluation incomplete.	Authorized budget authority adds funds or reduces future scope through a valid rule change.	Close the attempt or maintain containment. Preserve a separately authorized allowance for essential incident and recovery work.

The failure table applies before, during, and after an action. Before dispatch, check authority and funds. During execution, track the assignment and deadline. After interruption, reconcile the actual effect before choosing a retry or reversal. A successful off-chain job cannot compensate for a failed on-chain authorization, and the reverse can also leave a partial deployment.

Availability and deliberate obstruction

Removing routine human signatures reduces opportunities for people to stall individual releases. It does not eliminate control concentrated in hosting, key recovery, governance, or upgrade authority.

To protect routine operation, preauthorize bounded routine execution, keep replaceable workers, and provide recovery through an authority that does not require the failed actors' cooperation. Recovery permissions are consequential powers. They need explicit limits and notice because an actor who can replace signers may also take control through that replacement.

There is a hard limit to the promise of continuous progress. If every authority needed to act refuses, and no independent recovery authority already exists, software cannot both respect the authorization rule and execute anyway. A contract does not acquire permission from elapsed time unless that permission was built into its rules beforehand.

The achievable requirement is therefore that a failed attempt reaches a visible resolution and that common failures have a preauthorized recovery route. It is not a guarantee that every proposed update will eventually reach production. Sometimes the correct resolution is rejection, cancellation, or a continued pause. The DAO should remain able to receive reports and pursue recovery without pretending the blocked capability is healthy.