Unexpected Interruptions and Auto-Resume

"What should I do if the run is not completed once?" is a question that must be answered in production-level workflow. Braidrun provides two solutions for handling platform interruptions:

• Default: safe break — Mark the execution as INTERRUPTED and wait until you see it clearly before manually "retrying" from a certain step in the middle.
• Optional: Automatically Continue Running — When the deployment enables resilience, the workflow does not exit, and the recovery point meets the idempotent requirements, the platform can continue from the step boundary.

Default Behavior: Safe Break

When a platform-side interrupt occurs, the platform:

1. Mark the execution status as INTERRUPTED;
2. Keep the results, artifacts, and logs of every completed step;
3. You can see the INTERRUPTED status in the execution history and details pages;
4. After confirming safety, re-execute from the specified step, and the platform will reuse the previously retained step results.

This default is available to all workflows and requires no configuration. Suitable for:

• Workflow contains external side effects (sending emails/deducting money/modifying database)
• When production is not yet sure "whether it is safe to restart and continue running"

Optional: Automatically Continue Running

If the workflow is a pure data pipeline (read-only API + pure calculation + report generation), you can use automatic continuation. Resume occurs at step boundaries: the interrupted step is re-executed from the beginning and does not continue from the middle of the code or model call.

Three gates for automatic continuation

Automatic continuation after an unplanned crash or restart requires both of the following:

1. Deployment level: Administrator enables automatic recovery; default deployment can turn off this capability.
2. workflow level: AutoResumeOnRestart is not set to false, and ABORT is not selected. It is recommended to explicitly write out the policy in the production workflow to avoid relying on default values.
3. step level: When using RESUME_FROM_LAST_INCOMPLETE, steps that may be replayed must be marked with idempotent: true.

Complete Example

name: daily-asa-digest
version: "1.0.0"

recovery:
  autoResumeOnRestart: true
  policy: RESUME_FROM_LAST_INCOMPLETE
  maxAutoResumeAttempts: 3

workflow:
  - step: fetch_asa_data
    sub_workflow:
      name: dingyue-module-asa-fetch
    idempotent: true           # 只读拉取，可重跑

  - step: build_excel
    sub_workflow:
      name: dingyue-module-excel-report
    idempotent: true           # 生成文件到产物目录，可重跑
    depends_on: [fetch_asa_data]

  - step: notify_team
    sub_workflow:
      name: dingyue-module-slack-deliver
    depends_on: [build_excel]
    # 不写 idempotent —— 默认 false，发消息不可重跑

Three Recovery Policies

idempotent How To Mark

The gold standard for determining whether a step is idempotent:

"If the same input is run twice, will there be any side effects?"

If you can (send external messages, charge money, write orders), don't add idempotent.

Several Situations That Will Not Occur During Automatic Continuation

Even if you configure recovery, the following executions will still not continue running automatically:

• The trigger source is a Webhook - the Webhook payload may be at-most-once, and the platform does not know whether the upstream will retry.
• The trigger source is the Scheduler - the scheduler will re-fire itself in the next round of cron.
• Currently paused at manual_approval—crash recovery never bypasses approval automatically.
• The workflow definition file was modified during the outage - the platform detected a hash mismatch and is unsure if the new version is compatible.

Manual "Retry from a Step" (Available at Any Time)

For final records that support re-execution, the execution details page provides an entrance to re-execute from the specified step:

1. Open an INTERRUPTED / FAILED execution;
2. Click "Retry from this step" next to the failed/interrupted step;
3. The result of the previously COMPLETED step will be reused; this step starts to run again.

Even if the step isn't idempotent, you can manually retry it - because that's when you as a human consciously decide it's "worth rerunning".

Continue Running Limit

Prevent the system from restarting the same execution in an "infinite loop": each execution will automatically continue running up to 3 times. If it still fails, stop and wait for you to deal with it. You can override (1-20) in recovery.maxAutoResumeAttempts.

Audit

Audit events will be recorded every time automatic continuation is performed:

• execution.auto_resume — Successfully continued running
• execution.auto_resume.skipped — According to the rules, you should not continue running, with reason

Admin can filter and view the audit logs by these two event_types.

Practical Suggestions

• Pure data pipeline (ASA daily, RSS summary): full recovery + idempotent, rest assured that it will continue running automatically.
• If there are external side effects (deductions, sending messages, placing orders): do not open auto resume. Interrupting human decision-making.
• Add manual_approval before the key side effect steps - approval itself is a "natural blocking point" for discontinuous running.

Next

• Execution Monitoring and Logging — How to view execution in INTERRUPTED state
• Best Practices — Comprehensive suggestions for idempotent annotation/splitting/modularization