When “Completed”
Stops Meaning Done
AEGIS queues coding work for autonomous agents and records a status for every task. I stopped trusting that status field and measured all 909 tasks straight from the production database. The pipeline had worked. Then it quietly stopped — and its most recent success turned out to be a false completion.
Is the factory actually delivering?
AEGIS turns GitHub issues and its own findings into queued tasks, runs them through autonomous coding agents, and opens pull requests for review. Every task carries a status: pending, completed, failed, or cancelled. It is easy to read that column as a scoreboard.
A status is a claim the harness makes about itself. I wanted a measurement, so I skipped the dashboard and queried the production task table directly:
- Separated canaries — synthetic health-check tasks — from everything else
- Counted tasks that actually started, not tasks that were queued
- Counted completions that produced a pull request a human could review
- Grouped everything by month and by executor
It worked — then it stopped.
| Month | Started | Reported complete | With a PR | Cancelled |
|---|---|---|---|---|
| Mar 2026 | 424 | 326 | 132 | 85 |
| Apr 2026 | 131 | 89 | 17 | 70 |
| May 2026 | 5 | 1 | 0 | 24 |
| Jun 2026 | 13 | 6 | 1 | 38 |
| Jul 2026 | 2 | 0 | 0 | 29 |
| Aug 2026 | 2 | 0 | 0 | 24 |
| Sep 2026 | 0 | 0 | 0 | 5 |
In March and April the headless Claude Code executor started 555 tasks. 415 reported complete, and 149 produced a pull request. By May, starts had fallen to five. No non-canary task has completed since July 1.
Nothing alerted, because the queue never looked empty. Tasks kept arriving — proposed, prioritized, waiting. Queue depth looked like activity. Only the started column showed that nothing was moving.
Admission, not execution, was the biggest sink.
259 cancelled tasks never started
Of 275 cancellations, 259 died before execution. In the most recent 100 tasks, 43 of the 50 cancellations were proposals waiting on an approval that never came. The approval gate removed more work than the agents ever failed.
Lesson: an admission gate needs its own throughput metric48 tasks routed to a free executor, none started
From July through September, 48 tasks were assigned to a zero-cost Workers AI executor. Not one started. Routing work to the cheapest executor only saves money if that executor runs.
Lesson: confirm an executor drains its queue before routing work to itThe productive executor went quiet
The headless Claude Code executor behind March's 424 starts has not started a task since June, even though its queue kept receiving work.
Lesson: alert on starts per executor, not total queue sizeThe replacement executor passed 11 of 40 canaries.
I rebuilt execution around a cloud sandbox executor. Before trusting it with real work, I ran canary tasks: small, well-specified test additions with a known correct outcome.
The canary passed 11 times in 40 runs. Across the 31 failures in the most recent 100 tasks, the causes were tool loops that exhausted their iteration budget (10), sandbox and transport errors (8), environment bootstrap failures (8), planning failures (2), and one each of git clone, git push, and no-change failures.
One fix did hold in production: each publish branch is now named from a hash of its staged diff, so an identical re-run attaches to the existing pull request instead of opening a duplicate.
A green test run is not a delivered change.
The prompt
Add two new tests covering how rejected git pushes are classified.
The agent
Made one edit: an import statement.
The summary
Reported that both tests had been added.
The harness
Ran the focused test file. It passed with 15 tests — unchanged — and the task was marked completed.
The check answered the wrong question. A passing test file proves the edit broke nothing. It does not prove the requested change exists. The model's summary and the diff disagreed, and the harness believed the summary.
The fix is not a stronger prompt. It is a checker that compares the result against the task's stated acceptance criteria — here, that the test count rose by two. I logged it as a named gap in the system's governance register instead of patching around it.
It also changes how to read the history above. Every earlier completion used the same signal, so 422 is an upper bound on delivered work. The 150 completions that produced a pull request are the ones a human could actually review.
What I measure now
A harness reporting its own success is testimony. Query the underlying records before trusting the dashboard.
A full queue looks like progress. Tasks started per executor per week is the number that moves when a pipeline stalls.
Approval gates remove work silently. Track how long proposals wait and how many die unstarted.
Compare the diff against acceptance criteria. Green checks on touched files prove safety, not delivery.
Canaries that open real pull requests turn monitoring into review debt. Debugging this executor left seven open PRs to triage.
How the numbers were measured
Source: the production cc_tasks table, queried on September 13, 2026. All 909 tasks created
from March 8 to September 12, 2026 are included. Months are grouped by task creation date.
“Canary” means a task titled as a canary: a synthetic health check. “Non-canary” still includes a few smoke tests, which is another reason completion counts are an upper bound on delivered work.
More Case Studies
See the agent system this pipeline belongs to, or a migration where measurement pointed the other way.