It is 7:04 a.m. on a Monday and the payments channel is on fire. Card authorizations are failing across the board. The on-call engineer traces it back within twenty minutes: the quarterly key-rotation script ran over the weekend. It generated a new signing key, wrote it to the vault, and then crashed on a network timeout before it pushed the new key reference to three downstream services. Worse, the old key had already been marked for expiry earlier in the same script. The vault is correct. The apps are not. Nobody got an alert; the only log lived on a build agent recycled Sunday night.
The fix takes ten minutes. Two months later, the auditor asks for evidence that every production key was rotated on schedule last year, with timestamps. The team has a cron expression, a Git history, and a lot of hope.
If you run IT or DevOps automation, some version of this story is probably sitting in your scheduler right now.
Why operational scripts break the way they do
Most operational automation starts as a reasonable script written to solve one problem fast. Cloud cost attribution, secret rotation, user provisioning, pulling files off a legacy SFTP server, checking that last night’s backups actually restore. Each one is a few hundred lines of Bash, PowerShell, or Python, triggered by cron, a serverless function on a timer, or a CI pipeline someone repurposed as a scheduler.
But these tasks are multi-step processes touching several systems, and the script treats them as one long transaction with no memory.
The root causes tend to repeat:
- No state between steps or runs. If the process dies after step three of six, the next run has no idea what already happened. It either starts over (and repeats side effects) or skips work it assumes is done.
- Partial failure is invisible. The script exits non-zero, but which systems were changed? The vault was updated, two of five apps were updated, the old key was not revoked. That state lives only in the head of whoever reads the logs.
- Retries are either missing or dangerous. A blanket retry loop around the whole script can generate a second key, provision a user twice, or upload the same file to the ERP again.
- Logic is scattered. Schedule, credentials, retry behavior, and alerting all live in different places.
- There is no record of what ran. Logs rotate, build agents get recycled, and “did this succeed last quarter?” becomes an archaeology project.
Why the usual workarounds fall short
Teams usually respond by hardening the script: more try/catch blocks, a status file on disk, a retry helper, a chat alert on failure. Each helps a little; none solves the core problem. A status file on local disk disappears with the VM. A homegrown checkpoint table works until two runs overlap. Chat alerts tell you something broke, not what state the world is in. Retry helpers wrapped around individual API calls do not know how to undo the steps that already succeeded.
Moving the script into a serverless function or CI pipeline gives you nicer logs and a managed trigger, but a rerun still starts from the top with no memory of what already ran.
What you actually need is not a better script. It is a different execution model.
The approach: treat automation as durable workflows
A durable workflow is code that describes a multi-step process, executed by an engine that records every step’s result as it goes. If the process crashes, the engine replays the recorded history and resumes from the last completed step instead of starting over. The idea is vendor-neutral.
Applied to operational automation, it looks like this:
- Each step is an activity. “Generate key,” “write to vault,” “update app config,” “verify health,” “revoke old key.” Each activity does one piece of I/O and gets its own retry policy, with backoff and a cap on attempts.
- The orchestrator holds the logic. The sequence, branching, and fan-out live in one deterministic function. It does no I/O itself; it only decides what to call next.
- Compensation handles partial failure. If updating the apps fails after the vault was changed, the workflow runs explicit undo steps: restore the previous config, disable the new key, alert a human. This is the saga pattern, and it turns “half-done” into a defined state.
- Durable timers replace sleeps and cron gaps. A workflow can wait 24 hours for propagation or 90 days until the next rotation without holding a process open. The timer survives restarts.
- Human approval becomes a step. Risky actions, such as revoking a key used by a customer-facing system, can pause for an external approval event with a timeout and escalation path.
- History is your evidence. Every run has an ID, inputs, step results, and timestamps. When the auditor asks about key rotation, you export workflow history instead of reconstructing it.
Implementing it with Dapr Workflow
Dapr, the CNCF graduated Distributed Application Runtime, includes a workflow engine with SDKs for .NET, Java, Python, Go, and JavaScript. Workflow history is event-sourced into a pluggable state store, and after a crash the engine replays that history so completed activities are not executed again.
Here is that weekend key rotation as a Python workflow. Apps are updated in parallel, verified, and only then is the old key revoked. If anything after the vault write fails, the workflow rolls the apps back and disables the new key.
import dapr.ext.workflow as wf
from datetime import timedelta
wfr = wf.WorkflowRuntime()
retry = wf.RetryPolicy(first_retry_interval=timedelta(seconds=2), max_number_of_attempts=5,
backoff_coefficient=2, max_retry_interval=timedelta(minutes=1))
@wfr.workflow(name=”rotate_signing_key”)
def rotate_signing_key(ctx: wf.DaprWorkflowContext, data: dict):
new_key = yield ctx.call_activity(generate_key, input=data, retry_policy=retry)
yield ctx.call_activity(store_in_vault, input=new_key, retry_policy=retry)
try:
yield wf.when_all([ctx.call_activity(update_app_config, input={“app”: app, “key_id”: new_key[“id”]},
retry_policy=retry) for app in data[“apps”]])
yield ctx.call_activity(verify_apps, input=data, retry_policy=retry)
except Exception:
yield wf.when_all([ctx.call_activity(restore_app_config, input={“app”: app, “key_id”: data[“old_key_id”]},
retry_policy=retry) for app in data[“apps”]])
yield ctx.call_activity(disable_key, input=new_key, retry_policy=retry)
raise
yield ctx.call_activity(revoke_key, input={“key_id”: data[“old_key_id”]}, retry_policy=retry)
@wfr.activity(name=”update_app_config”)
def update_app_config(ctx: wf.WorkflowActivityContext, data: dict): …
Notice that the old key is revoked last, only after verification, which removes the exact failure from the opening scene. Each app update retries independently, so one slow service does not force the others to redo work. And if the worker process dies halfway through the fan-out, the engine picks up where it left off on restart rather than generating a second key.
The same shape covers other jobs. Cost attribution becomes a scheduled workflow that fans out across subscriptions, tags resources, and writes a report. A legacy file integration becomes a pipeline of “fetch,” “validate,” “transform,” and “load” activities, where a failed load does not cause a duplicate fetch. For example, an IT services and DevOps team used Dapr Workflow for Azure cost attribution, security key rotation and legacy file integration with fault-tolerant pipelines.
You can run Dapr yourself on Kubernetes, VMs, or self-hosted machines, or use a managed environment. Azure Container Apps, for instance, offers Dapr as a built-in integration you enable per app. If you would rather not operate the runtime and its state store yourself, a managed durable workflow platform built on Dapr is another route. Whatever you choose, run your own failure tests before trusting it with production keys.
Where to start: a practical roadmap
Do not migrate everything at once. Pick the script that hurts most and prove the model there.
- List your automation inventory. Every cron entry, scheduled function, and pipeline-as-scheduler. Note what each touches.
- Rank by blast radius. Secret rotation and anything touching payments or identity usually wins.
- Break the chosen script into steps. Write each side effect down as a named activity. If a step does two things, split it.
- Define the undo for each step. For every activity that changes state, decide what compensation looks like. Some steps cannot be undone; those need to run last or behind an approval.
- Make activities idempotent. Retries will happen. Use stable IDs so a repeated “create user” or “upload file” is safe.
- Set retry policies per activity. Fast backoff for flaky APIs, fewer attempts for expensive calls, no automatic retry for actions that need a human.
- Add approvals for risky steps. Revoking production credentials or deleting data can wait for an external event with a timeout.
- Wire up observability. Dapr emits Prometheus metrics and OpenTelemetry traces. Alert on failed and stuck workflows, not just on process exit codes.
- Retire the old script fully. Remove the cron entry. Two schedulers running the same job is worse than one.
Before you call a workflow production-ready, check it against this list:
- ☐ Every step that changes an external system is a separate activity
- ☐ Each activity has an explicit retry policy
- ☐ Compensation exists for every reversible step
- ☐ Irreversible steps run last or behind approval
- ☐ Killing the worker mid-run and restarting it does not duplicate side effects
- ☐ Workflow history is retained long enough for your audit window
- ☐ Failed and long-running instances trigger alerts
Conclusion
Fragile automation usually fails not because its authors were careless, but because multi-step processes were squeezed into an execution model built for one-shot commands. Durable workflows give those processes memory, retries with intent, a defined answer to “what happens if it breaks halfway,” and a history you can hand to an auditor.
Start with the script that woke someone up most recently. Rewrite it as a workflow, break it on purpose in a test environment, and watch it recover. Open-source Dapr is enough to get there, and providers such as Diagrid offer managed options if you want to compare. Either way, the next Monday morning should be a quiet one.
