Piloting an AI project: what to prove in a month before deciding to go on

Short answer: a pilot of an AI project runs three to six weeks, and its output is a decision, go on or stop, taken on numbers. Three things have to be in place before it starts. One task in one department. A set of fifty to a hundred real questions with the answers an experienced employee would give. And an agreement on what share of correct answers counts as success. Without them the pilot ends in a demo everybody liked, and nobody can say what to do next.

We quote in roubles, so the figures below are given as proportions and timelines rather than converted amounts — convert at the rate of the day if you need an absolute number, or ask us for a quote.

One task, one department

A pilot across “all the company’s documents” proves nothing: the questions are too different, and an average score over them says nothing at all. Pick a task where a mistake is visible and the cost of the mistake is understood. Answers to sales staff about product specifications and availability, say, or finding a clause in standard contracts for the legal team.

A good pilot task comes up often, takes noticeable time from people, and has a checkable answer. If the answer to a question depends on judgement, there will be nothing to score the assistant against.

Benchmark questions

This is the client’s main piece of work in a pilot. The head of the department collects the questions people actually ask: out of chats, email, requests to senior colleagues. Each one gets the correct answer written next to it, and the document that answer lives in.

The set has to exist before the vendor starts configuring anything. Then the result is measured on questions nobody tuned the system for. Half the set is worth keeping back and not showing the vendor until the final check.

What to measure

Share of correct answers. Each answer is compared with the reference. Count correct, partly correct and wrong separately.

Share of honest refusals. When the answer is not in the documents, the assistant should say it does not know and pass the question to a human. A confident invented answer is worse than a refusal, and the two have to be counted apart.

The source link. The user has to see which document the answer came from and be able to check it inside a minute.

Response time and cost per query. In a pilot these look like details, and they decide what the full department-wide setup will cost to run.

What the pilot budget covers

Connecting the data sources, parsing the documents, configuring retrieval and the model, a simple interface for the department, and a run against the benchmark set. The server for the model is usually rented from a Russian cloud for the duration; buying hardware at this stage is early.

The most underestimated part is parsing the documents. Scans, tables inside PDFs and superseded versions of internal rules produce more errors than the choice of model does. There is a detailed account of that in RAG on real corporate documents.

When to stop

A pilot with a poor result is still useful when the reason is clear. There are three usual reasons.

The documents holding the answers do not exist: the knowledge lives in people’s heads. Writing the knowledge base comes first then, and that is months of work.

The answers exist but contradict each other: three versions of the price list, two procedures for the same operation. Then the sources need order, and an assistant will not supply it.

The task turned out to be rare: five questions a week, and the saving will not cover the running cost. Then the project closes, and that is a legitimate outcome.

After a successful pilot

Moving to a working system takes two to four months. Access rights come in, the remaining sources, scheduled index updates, a question log and quality monitoring. The pilot question set becomes the regression set: every update is checked against it. Price guidance for the full project is in what RAG costs.

Common questions

What share of correct answers is good? It depends on the cost of a mistake. For prompts to a sales manager with a source link attached, starting around eighty per cent correct with honest refusals on the rest is reasonable. For legal answers the bar is higher, and a person checks every answer.

Can the pilot run on a cloud model and move to a local one later? Yes, if the pilot documents contain no personal data and no secrets. The local model’s quality then has to be measured again on the same question set.

Who is needed on the client side? The head of the department, who collects the questions and accepts the result, and two or three people who use the assistant in their work.

The rule

Collect the benchmark questions before the pilot starts and agree the success threshold in advance. Then the decision at the end of the pilot is made on numbers.