Nobody can prove it still works
The system runs, the logs look fine, and no one can say whether quality today matches quality at launch. Without an evaluation suite that question has no answer.
03 · Managed AI Support
The part that starts after launch
AI systems degrade in ways ordinary software does not. Providers change models, prompts drift, data shifts, edge cases accumulate. This is the discipline that keeps a deployed system right.
We can operate a system we did not build, but only after a review that establishes what it does, how it fails, and what correct looks like. Where no evaluation set exists, building one is the first engagement rather than an optional extra.
Deployed, traced, evaluated, and round again
The loop, not a claim about results. Most releases pass. The one that does not is the reason the loop exists.
01What it is
What this is, concretely
A deployed AI system is not finished. The same input can return a different answer next quarter because a provider retired a model, adjusted a default, or shipped a version that reasons differently. Meanwhile your data moves, your users find inputs nobody designed for, and the failure that matters is the quiet one: output that is still fluent and now wrong.
This is operations work, not a helpdesk. It means owning the evaluation suite, watching the numbers that move before complaints do, testing provider changes before they reach production, cutting cost where it is being wasted, and responding when something breaks, including the decision to roll back to a model everyone had stopped thinking about.
02The readout
Six numbers, watched properly.
Uptime tells you the service responded. Only these tell you it responded correctly, which is the question that matters here.
- Task success rate
- Against the evaluation suite, per release
- Escalation rate
- Moves before users start complaining
- Tool error rate
- Integration failures, kept separate from model failures
- Latency, p95
- Per task type, not averaged into meaninglessness
- Cost per task
- Attributed per feature so it can be cut on purpose
- Provider deprecations
- Tracked with a migration date ahead of the cutoff
03Where it fits
What goes wrong after launch.
The provider changed something
A model is deprecated, a default shifts, pricing moves. Each of these lands on your production system on their schedule, not yours.
Cost grew without a decision
Usage patterns change, retries stack up, context grows. Spend rises steadily and no line item explains it.
The team that built it moved on
A change is needed and nobody left understands the prompts, the evaluation set, or the failure history.
04Examples
Systems of this shape
Build patterns and capabilities, not client projects.
- Evaluation suite and regression gate
- A versioned set of real cases with expected behaviour, run on every prompt, model or configuration change. Nothing reaches production without passing.
- Production monitoring
- Success and escalation rates, latency, spend per task, tool errors and refusals, tracked over time with alerts on the trends that precede a bad week.
- Provider and model change management
- Deprecations tracked, candidates tested against your evaluation set in staging, and migrations run with a measured before and after.
- Incident response and rollback
- A defined path when output goes wrong: contain, roll back to a known-good configuration, diagnose from the run record, fix, add the case to the suite.
- Continuous improvement
- Escalations and corrections reviewed to find the recurring failures, then the tools, prompts or data that cause them get fixed and proved against the suite.
05Approach
How the engagement actually runs.
- 01
Evaluations are the contract
Before anything is operated, correct is established as a runnable suite. Everything after that is measured against it.
- 02
Watch the leading indicators
Escalation, refusal and retry rates move before users complain. Those are the numbers on the dashboard, not just uptime.
- 03
Change control for prompts and models
Prompts, tool definitions and model versions are versioned artefacts with a review step. A prompt edit is a deployment and is treated like one.
- 04
Cost is an engineering metric
Spend is attributed per feature and per task so it can be reduced deliberately: smaller models where they suffice, caching, shorter context.
06In production
What you keep
We can operate a system we did not build, but only after a review that establishes what it does, how it fails, and what correct looks like. Where no evaluation set exists, building one is the first engagement rather than an optional extra.
A known-good configuration always exists
Every model, prompt and tool version that passed evaluation is retained and can be restored.
Response times agreed in advance
Severity levels and response windows are set at the start of an engagement and written down.
Reporting is honest
Regular reporting covers what degraded as well as what improved, with the numbers behind both.
You keep the keys
Provider accounts, repositories and infrastructure stay yours. We operate inside your environment, and you can end the engagement without losing the system.
07Where and next
Where we do this work
Operating engagements run mostly with organisations in the UAE and Saudi Arabia, and remotely for a small number of clients in the United States and Europe. Time zone overlap is agreed at the start, because the response windows depend on it.
Start a projectThe other two lines
- AI Agent DevelopmentGive AI the tools, permissions and context to finish real work inside your systems.
- Custom AI ApplicationsProduction software where AI is a working part of the product, not a chat box bolted on the side.
- All three, comparedWhat each line is, when it fits, and what it needs from you.
Something is already live and nobody can prove it still works?
That is a normal place to start. The first engagement establishes what correct looks like as a runnable evaluation suite. After that, every change has an answer instead of an opinion.