03 · Managed AI Support
AI systems degrade in ways ordinary software does not. Providers change models, prompts drift, data shifts, edge cases accumulate. This is the discipline that keeps a deployed system right.
We can operate a system we did not build, but only after a review that establishes what it does, how it fails, and what correct looks like. Where no evaluation set exists, building one is the first engagement rather than an optional extra.
Monitored, evaluated, caught, restored, and round again
Illustrative incident · system status
Restored
01 · Live
Running in production
02 · Monitor
Every run traced
03 · Evaluate
Scored against expected answers
04 · Review
A person looks at what changed
A model update, a data shift, a new edge case.
05 · Correct
Fix it, or roll back
06 · Restored
Known-good version live
The loop, not a claim about results. Most releases pass. The one that does not is the reason the loop exists.
After launch
Every change has an owner, a record and a way back.
Someone stays answerable for the system for as long as it runs, not only until it ships.
A deployed AI system is not finished. The same input can return a different answer next quarter because a provider retired a model, adjusted a default, or shipped a version that reasons differently. Meanwhile your data moves, your users find inputs nobody designed for, and the failure that matters is the quiet one: output that is still fluent and now wrong.
This is operations work, not a helpdesk. It means owning the evaluation suite, watching the numbers that move before complaints do, testing provider changes before they reach production, cutting cost where it is being wasted, and responding when something breaks, including the decision to roll back to a model everyone had stopped thinking about.
Uptime tells you the service responded. These tell you it responded correctly.
A regression rarely announces itself. It arrives looking like a slightly worse week.
Before a change
Every prompt, model or configuration change is run against the versioned evaluation set first. A change that drops a score does not reach production on the grounds that it looked fine in a demonstration.
After it ships
Escalation, refusal and retry rates move before anybody files a complaint. Those are the numbers on the dashboard, not uptime on its own.
When something slips
When something does slip past, the input that exposed it is added to the suite. That particular failure cannot come back quietly, which is the only durable form of a fix.
Always
Every model, prompt and tool version that passed evaluation is retained. Rolling back is a decision someone takes calmly, not a salvage operation at the end of a bad week.
AI operations is not a helpdesk with a different name. It is knowing, with evidence, that a system which worked in March still works in November.
The system runs, the logs look fine, and no one can say whether quality today matches quality at launch. Without an evaluation suite that question has no answer.
A model is deprecated, a default shifts, pricing moves. Each of these lands on your production system on their schedule, not yours.
Usage patterns change, retries stack up, context grows. Spend rises steadily and no line item explains it.
A change is needed and nobody left understands the prompts, the evaluation set, or the failure history.
Build patterns and capabilities, not client projects.
Pattern 01
A versioned set of real cases with expected behaviour, run on every prompt, model or configuration change. Nothing reaches production without passing.
Pattern 02
Success and escalation rates, latency, spend per task, tool errors and refusals, tracked over time with alerts on the trends that precede a bad week.
Pattern 03
Deprecations tracked, candidates tested against your evaluation set in staging, and migrations run with a measured before and after.
Pattern 04
A defined path when output goes wrong: contain, roll back to a known-good configuration, diagnose from the run record, fix, add the case to the suite.
Pattern 05
Escalations and corrections reviewed to find the recurring failures, then the tools, prompts or data that cause them get fixed and proved against the suite.
Before anything is operated, correct is established as a runnable suite. Everything after that is measured against it.
Escalation, refusal and retry rates move before users complain. Those are the numbers on the dashboard, not just uptime.
Prompts, tool definitions and model versions are versioned artefacts with a review step. A prompt edit is a deployment and is treated like one.
Spend is attributed per feature and per task so it can be reduced deliberately: smaller models where they suffice, caching, shorter context.
We can operate a system we did not build, but only after a review that establishes what it does, how it fails, and what correct looks like. Where no evaluation set exists, building one is the first engagement rather than an optional extra.
Every model, prompt and tool version that passed evaluation is retained and can be restored.
Severity levels and response windows are set at the start of an engagement and written down.
Regular reporting covers what degraded as well as what improved, with the numbers behind both.
Provider accounts, repositories and infrastructure stay yours. We operate inside your environment, and you can end the engagement without losing the system.
Operating engagements run mostly with organisations in the UAE and Saudi Arabia, and remotely for a small number of clients in the United States and Europe. Time zone overlap is agreed at the start, because the response windows depend on it.
Start a projectThe other two lines
That is a normal place to start. The first engagement establishes what correct looks like as a runnable evaluation suite. After that, every change has an answer instead of an opinion.
