Heartbeats, Not Hope: Scheduling Autonomous Work That Actually Runs
Heartbeats, Not Hope: Scheduling Autonomous Work That Actually Runs
Every agent platform has the same embarrassing secret in its first year: the "scheduled" jobs that nobody checked. A competitor monitor that silently stopped fetching six weeks ago. A nightly triage that has been failing since a token expired. The schedule existed; the work didn't happen. Cron is a promise, not a delivery mechanism.
Autonomous operations need something stronger than cron. They need **heartbeats**: scheduled checks that run, report structured results, and — critically — *raise the alarm about their own absence*.
The anatomy of a heartbeat
A heartbeat is not a job; it's a contract with four clauses:
1. **A playbook** — the explicit list of checks to execute ("uptime of all tenant domains," "CI status across repos," "queue depth under threshold").
2. **A structured report** — each check returns pass/fail plus a detail line. Not a log blob; data. `{"name": "uptime:feeltrack", "passed": false, "detail": "HTTP 502, 4.1s"}`.
3. **A roll-up status** — `ok`, `degraded`, or `critical`, with rules defined ahead of time so the agent doesn't get to grade its own homework leniently.
4. **An escalation path** — degraded opens a task; critical opens a task *and* pages the human. The report is the evidence; the task is the work item.
The difference from a cron job is clause 2 and 3: the output is designed to be *acted on by software*, not skimmed by a tired human.
The reflexive problem: who watches the watcher?
Here's the failure mode that bites everyone: the heartbeat scheduler itself wedges. A deploy hangs the worker, a Redis connection leaks, and now zero heartbeats run — and because the system is quiet, nobody notices. The monitoring died of the same thing it was monitoring.
The fix is a **self-health check that lives outside the thing it checks**. Ours is embarrassingly simple: aggregate heartbeat runs over the last hour. If `runs == 0`, the verdict is `failed` — a wedged scheduler is indistinguishable from a healthy one only if you forget to count. Time-since-last-run is a metric; alert on its silence, not just on loud failures.
Any autonomous system that can't detect its own stillness isn't autonomous — it's abandoned.
Designing playbooks that earn their keep
A few rules from running these in production across a multi-tenant fleet:
- **Checks must be cheap.** A heartbeat that takes ten minutes to run won't run hourly, and hourly is the minimum cadence that catches real drift. Shell out, probe, exit.
- **Checks must be specific.** "System healthy?" is unanswerable. "All 12 tenant domains returned 200 in under 3s" is a fact.
- **Failures must become work, not vibes.** A failed check that doesn't create a task with the evidence attached is a failure that will happen again next hour. The heartbeat's job is to convert anomalies into tracked work items with owners.
- **Playbooks rot.** Every quarter, delete the checks that have never failed and add checks for the incident you just had. Your heartbeat coverage should be a scar map of your actual outage history.
Beyond health: heartbeats as the agent's alarm clock
Once the reporting contract exists, heartbeats stop being only defensive. The same mechanism drives proactive work: a weekly heartbeat that checks competitor watch pages for changed content hashes, a daily one that sweeps stale branches, a Monday one that drafts the content batch for the week. The schedule fires, the playbook runs, structured results land, exceptions become tasks.
This is the bridge from "agent that responds" to "agent that tends": a fleet of small, honest, self-reporting loops, each one too simple to hide a failure.
The honest summary
Cron tells you something was *supposed* to run. A heartbeat tells you what ran, what it found, and what's being done about it — and it notices when it itself stops beating. In supervised autonomy, that's the difference between scheduled work and wishful thinking.
Hope is not an operations strategy. Heartbeats are.
---
*Part of the FlukeBase supervised-autonomy series. Next: cross-project orchestration — one operator, ten products.*