From 18 LLM Calls
to Zero
My project scaffolding pipeline chained 18 to 24 LLM calls to turn a description into a specification. When I measured what downstream consumers actually used, they kept the structured facts and threw away the prose. So I replaced inference with structure.
Good output, bad economics
The original pipeline generated six specification modes in sequence: product requirements, UX patterns, risk analysis, architecture decisions, test plans, and sprint tasks. Each mode took two to four inference calls.
- About 7 minutes end to end
- 18 to 24 LLM calls per scaffold
- Roughly 84,000 characters of prose per run
- No reproducibility — different output every run
- About a 5% failure rate — one failed call broke the whole run
The quality was good. But every call was also a point of failure, a latency penalty, and a cost that grew with usage.
Consumers kept the facts and discarded the prose
I measured what downstream consumers actually used. They used component names, priority rankings, framework choices, test frameworks, CI stages, task estimates, and threat categories — structured key-value pairs. They ignored about 90% of the prose.
The pipeline spent 18 calls generating text that consumers immediately decomposed back into key-value pairs. So I inverted it: structured facts became the primary output, and prose became optional.
A symbolic engine with no inference in the loop
Evaluate
The description is evaluated against curated domain knowledge encoded as typed data, not a model's training data.
Generate
The engine produces 400 to 600+ structured facts across six dimensions.
Materialize
Facts render into nine deployable project files, governance files first.
Publish
The files land in a GitHub repository as a single atomic commit.
Deploy
npm install, then wrangler deploy — a live Cloudflare Worker in about two minutes.
An optional single inference call adds natural-language polish for anyone who wants prose. Most API consumers skip it.
Measured side by side
| Metric | LLM pipeline | Symbolic engine |
|---|---|---|
| Latency | ~7 minutes | 323 ms |
| LLM calls | 18–24 | 0 |
| Structured facts | 52, extracted from prose | 400–600+, native |
| Composite quality | 0.54 | 0.77 |
| Failure rate | ~5% | <0.1% |
| Reproducibility | None | Seed + SHA-256 receipt |
| Cost per scaffold | $0.01–0.02 | $0.00 |
| Time to deployed Worker | ~15 minutes | ~2 minutes |
Quality was scored on 3,000 decisions per engine — 15 scenarios × 50 seeds — across six weighted metrics: diversity, coherence, acceptance, degeneracy, latency, and constraint satisfaction.
One caveat: the composite quality score is a metric I defined for this comparison, not an external benchmark. The latency, call-count, cost, and reproducibility results do not depend on it.
Governance before code
Each scaffold produces nine files: three .ai/ governance files holding architectural constraints,
requirements, security policies, and the backlog; package.json, tsconfig.json, and
wrangler.toml with bindings inferred from the description; an entry point; test stubs; and a README.
Governance files are generated first, deliberately. The first thing a developer sees is the rules of the system, not an empty source folder.
What carried over
When a consumer needs structured data, generating prose and parsing it back is an anti-pattern. Start with the structure.
The goal is not identical output. It is variation you can replay from a seed and verify by receipt hash.
Curated, typed domain knowledge produced better Cloudflare Workers architecture decisions than a general model's training data.
Putting constraint files ahead of source files shifts the first question from “what should I build?” to “what are the rules?”
One optional polish call serves the users who want prose. Everyone else gets faster, cheaper, verifiable output.
More Case Studies
See the agent system this engine runs inside, or how I measure whether autonomous work actually gets done.