Why your AI agent demo isn’t a product, and how to fix it

The demo is not the hard part

Anyone can build an agent demo in an afternoon now: pick a model, write a prompt, wire a tool, show it on the examples you chose. It works. Then a client asks for the same quality on their content, every day, inside the tools their team already uses, at a price they can put in a budget, and the demo has nothing to say.

I have been on both sides of that gap. At App2Check we turned FAQs into a working chatbot, EVA, and had to make it something a bank or a carmaker could run. At DATAmundi I led the research team behind AIDA Agents and its implementation: a multi-agent translation platform that beat every system on 10 of 11 language pairs of a public benchmark and now runs in production. In both cases the model was the easy part. The distance between the demo and the product was four specific gaps: orchestration, quality control, evaluation and integration. Here is what each one looks like, and how we closed it.

Gap 1: one prompt is not an orchestration

A demo asks one model to do everything in one call. A product gives the work to specialized agents with defined roles, and decides what passes between them.

AIDA Agents runs up to three translator agents in parallel to produce candidates, an arbiter that rates them, a post-editor that improves the best one, and a second rater that checks the result. The roles matter less than the hand-offs: each agent receives the previous agent’s output plus the same client context, retrieved for every segment from the translation memory, the glossary of approved and forbidden terms, and the style guide. That is how the platform adapts to a new client or domain with no fine-tuning at all, which is also why it can be deployed in days rather than after a training cycle.

Gap 2: a demo trusts the first answer

The single most valuable component in a production agent system is the one that says no. In AIDA Agents that is the rater.

The rater scores every candidate on criteria borrowed from the MQM quality framework used by professional linguists: accuracy, fluency, terminology and style, with weights the client can change. It produces an explainable quality report, not just a score. The post-editor reads that report, all the candidates and the guidelines, and fixes what the report flagged. A second rater then re-evaluates. The loop is what turns “the model usually gets it right” into “here is why this segment is publication-ready, and here is the one that is not.” The report is also what the client’s reviewers read, so the human work moves from redoing everything to checking exceptions.

Gap 3: a screenshot is not an evaluation

Buyers ask three questions in a fixed order: how does it compare with others, how does it do on our content, and what changes in our workflow. Each needs its own evidence.

For the first we used WMT24++, a public benchmark across eleven languages, with no translation memory or terminology available to any system: AIDA Agents was best on ten of the eleven. For the second we built an industrial benchmark from a real project, ninety-six segments per language across six languages, with the client’s translation memory, terminology and style guides, rated blind by three linguists per language on a five-point scale, against a commercial NMT engine and a competitor’s system that included human post-editing. Between 70% and 98% of our segments were publication-ready with no human intervention; for German that beat the human-post-edited competitor. For the third we measured what a production system changes: on the WMT25 terminology task the terminology-constrained version reached 99.4% terminology accuracy, and the EAMT paper estimates that the automated workflow could cut post-editing effort by 50–70%.

Gap 4: it has to live where the work happens, at a price

Translators work in CAT tools and exchange XLIFF files. A product that asks them to paste text into a chat window is a demo with extra steps.

AIDA Agents connects to the CAT tool natively and returns the translation with the XLIFF structure preserved, so a linguist sees it in the same place and the same format as before. It also comes in three tiers, because cost is a product decision: a balanced tier with three model calls per segment, a maximum-quality tier with up to six, and a lowest-cost tier that uses a neural machine translation engine with a post-editor and a rater on top. Professional human post-editing plugs into all three. A client chooses per job, and the price is a number of calls per segment, which finance can forecast.

How to fix yours

Most agent demos can become products. The order below is the one that worked for us.

1. Name the roles.Split the single prompt into agents with one job each, and write down what passes between them. If you cannot say what the hand-off contains, you do not have an orchestration yet.
2. Add the agent that says no.A rater with explicit criteria and an explainable report, and a loop that acts on it. Without it every error reaches the customer.
3. Evaluate on three levels.A public benchmark for comparability, a benchmark built from the client’s real content for credibility, and a production signal you can measure every week.
4. Inject the client’s context instead of fine-tuning.Translation memories, glossaries, style guides, documents: retrieved per request, they make one system work for many clients and keep deployment to days.
5. Live inside the existing tool.Preserve the file formats and the workflow the users already have. Integration is most of the engineering, and all of the adoption.
6. Price by unit, and keep a human path.Tiers expressed in calls per unit make the cost forecastable; a professional review step makes the product acceptable where quality is contractual.

The receipts

The system, the papers and the recognition, all public.