A prototype agent can carry a demo and still fall over in production. The useful work was refactoring it from a loosely-prompted loop into an orchestrated pipeline — planner, retrieval, tool use, verifier, response — with scoped context per step and a RAGAS-based eval suite running in CI as a release gate.
The agent was burning tokens on repeated context, weak routing, and tool calls that should have been resolved earlier in the flow.
The team needed a way to catch hallucination, retrieval drift, and tool-call regressions before any of it went user-facing.
Reliability had to improve without locking the system down so hard it lost the flexibility that made an agent worth building in the first place.
Orchestration core
We split the workflow into planner, retrieval, tool-use, verifier, and response steps, then put strict limits on what context each step was allowed to see.
Evaluation harness
A RAGAS-based test suite scored retrieval relevance, factual grounding, answer quality, and regression behavior against curated task sets we could trust.
Release gates
Eval failures, tool-call anomalies, and citation gaps were treated as release blockers — not as observations someone might inspect later, after it had already shipped.
- Instrumented traces first so the team could actually see token waste, bad branches, and repeated context.
- Refactored the orchestration layer around explicit state, tool boundaries, and verifier passes.
- Added eval suites to the build pipeline so reliability checks ran before any production change went out.
Token usage dropped about thirty percent from context control and better routing alone.
The team got a regression harness for hallucination, retrieval quality, and tool-call behavior.
The agent moved out of demo logic and into something traceable, testable, and noticeably cheaper to run.
Autonomy without evals is theater. The actual production work is orchestration, context control, release gates, and making failures visible early enough that someone can still fix them.
