In a chatbot call for tenders, vendors' resolution rates are only comparable if you impose the definition and check how each vendor measures it. One vendor claims 95%, another 80%, a third 55%, and the selection panel scores the first one highest without knowing how each of them counted. Here are twelve questions to put to vendors, a grid to score their answers and a clause ready to paste into your specifications.

Most specifications describe in great detail what the chatbot must do, and almost never how you will know whether it does it. Our guide to writing an AI chatbot RFP sets out the six building blocks of good technical specifications (the CCTP, in French public procurement). This article adds the seventh: quality measurement.

Key takeaways

• A resolution rate without a definition cannot be compared: impose your own in the technical specifications and make every bidder answer with it.

• Twelve questions across four themes check each vendor's method. Scored out of 24, they carry 15% of the overall mark.

• At the presentation, ask to see the unresolved conversations, and have the vendor justify a "resolved" conversation picked at random.

• A contractually guaranteed rate pushes the vendor towards the most generous definition: require the measurement method instead.

• With a strict definition, ViaSay customers in production achieve resolution rates between 50 and 85%.

The same chatbot can show 95%, 80% or 55% depending on the definition used

Three definitions of resolution rate circulate under the same name: deflection, containment and verified resolution. Applied to the same conversations from the same chatbot, they produce three different figures. Two bids can therefore only be compared on the same definition.

• Deflection counts as resolved every conversation that did not reach a human, abandoned ones included. Typical range: 80 to 95%.

• Containment counts every conversation that stayed within the bot, with no transfer. Typical range: 80 to 95%.

• Verified resolution only counts conversations where the request was actually dealt with, as confirmed by a review. Typical range: 40 to 85%.

The denominator, meaning the conversations included in the calculation, also moves the figure.

The market is shifting towards the third definition. Intercom changed its calculation in July 2026: in its official example, the same bot goes from 33% to 50% once conversations where it had no chance to answer are removed from the denominator. Since May 2026, Zendesk only counts "Verified Resolutions" confirmed by a language model.

A vendor who claims 95% without stating the definition is, in almost every case, giving you a containment rate. The figure is not a lie, but it measures something other than resolution.

The three definitions are explained in detail in our article on measuring an AI chatbot's resolution rate. In your tender, impose one definition and make every vendor answer with it.

Which resolution rate questions should you ask in a chatbot call for tenders?

Twelve questions let you check how a vendor measures the resolution rate it claims. They cover four themes: definition, coverage, auditability (the ability to verify every figure) and improvement. For each question, here is the answer to expect and the red flag to watch for.

Definition (questions 1 to 4)

1. How do you define a "resolved" conversation?

• Expected answer: a written definition, with criteria: a direct answer, a valid redirection, or an escalation correctly handed over to an agent.

• Red flag: "A conversation with no transfer to an agent."

2. Does an escalation to an agent count as a failure?

• Expected answer: no, if it is correct: the bot recognised its limit and handed over cleanly. Yes, if it refused without offering an alternative, transferred a question it knew how to handle, or merely said that no agent was available.

• Red flag: every escalation counts as a failure, which pushes the bot to hold on to the customer, or every escalation counts as a success, which inflates the figure.

3. What is excluded from the denominator?

• Expected answer: conversations with no stated request, or those where the bot had no chance to answer. The containment rate is still calculated on the total, as a cross-check.

• Red flag: nothing is excluded, so the figure also measures how much time users have to spare; or everything is excluded, with no counterweight.

4. Does the customer's silence after the last answer count as a resolution?

• Expected answer: never by default. The assessment looks at what the customer was given before leaving, and a departure during a pending step (a question, an identifier, a choice) is set aside.

• Red flag: "No further question, so it's resolved."

Coverage (questions 5 to 7)

5. What share of conversations is assessed?

• Expected answer: 100% of completed conversations.

• Red flag: a sample, or only the conversations where the customer left a rating.

6. Are scripted journeys, written in advance, assessed in the same way as AI-generated answers?

• Expected answer: yes, against the same criteria, with a comparison of the two.

• Red flag: only generative AI answers are measured.

7. Who delivers the verdict?

• Expected answer: an AI evaluator separate from the bot, using a fixed set of criteria, complemented by ratings from real customers.

• Red flag: the bot assesses itself, or a human reviews 2% of conversations.

Auditability (questions 8 to 10)

8. Can I open the conversations behind every figure?

• Expected answer: yes, from any metric, with the conversations already filtered.

• Red flag: the rate is just a number in a monthly report.

9. Is every verdict justified?

• Expected answer: yes. The evaluator explains in one sentence why it judged the conversation resolved, unresolved or lacking information.

• Red flag: a percentage, with no explanation for each conversation.

10. Does the rate tell you why a conversation failed?

• Expected answer: yes, distinguishing missing information in the knowledge base (a knowledge gap), misunderstanding and dead-end journeys.

• Red flag: a single overall figure, with no cause.

Improvement (questions 11 and 12)

11. How do you get from the figure to a fix?

• Expected answer: topics ranked by number of lost conversations (volume × unresolved), not by percentage. Each topic links directly to the content or journey to fix.

• Red flag: "Your CSM will send you a report", the CSM being the vendor's customer success manager.

12. What rate do your customers in production achieve with this definition?

• Expected answer: an honest range, measured with the definition from question 1.

• Red flag: a single, round figure above 90%.

At ViaSay, we answer question 12 with a range: between 50 and 85% depending on the customer. This rate is measured conversation by conversation by an AI evaluator, correct escalations included.

The bottom of the range corresponds to chatbots we have only just started measuring. The top corresponds to those where one topic has been fixed every month for a year.

For questions 8 and 9, "auditable" means that every conversation carries its verdict, and hovering over it shows the reason.

A 24-point grid lets you score vendors' answers

Each question is scored from 0 to 2 points, giving a total out of 24 that fits the usual weighted selection grids. We recommend giving this "quality measurement" block 15% of the overall mark, as much as integrations. It is the block that will tell you, six months after go-live, whether the project is keeping its promises.

• 0 points: no answer, or a red flag.

• 1 point: a partial answer, or a promise without a demonstration.

• 2 points: a complete answer, demonstrated at the presentation on real conversations.

At the presentation (the bidders' oral pitch), two tests are particularly revealing.

1. Ask to see the resolution rate measured on the demo, not just the demo. A vendor who genuinely measures will show you the unresolved conversations screen without hesitation; the others will show you a bot that answers well.

2. Have them open a "resolved" conversation picked at random, then ask why it counts as resolved. The answer, or the lack of one, will tell you as much as the twelve questions.

A clause in the specifications sets the definition and the measurement method

The clause below writes into the technical specifications the definition of resolution, coverage of 100% of conversations and access to every verdict. Paste it into the "Functional requirements" or "Management and reporting" block: every bidder will then answer with the same definition.

Measuring conversation quality. The contractor provides a resolution rate calculated on 100% of completed conversations, including both scripted journeys and generated answers.

A conversation counts as resolved when the request expressed by the user has been dealt with: a direct answer, a redirection to a valid resource, or an escalation correctly handed over to an agent. The user's silence does not count as resolution.

Each verdict can be viewed individually, with its justification, from the dashboard. The contractor distinguishes conversations left unresolved because of missing information in the knowledge base from those left unresolved for another reason. The containment rate is provided separately and calculated on the total number of conversations.

Also add this requirement to the list of documents to be submitted:

"The bidder states the resolution rate achieved by three customers in production, using the definition above, and specifies the evaluation method."

The Excel selection grid template that comes with our GDPR and AI chatbot RFP guide can take this block in an additional tab.

A contractually guaranteed resolution rate almost always backfires

Some specifications require a contractual resolution rate, for example "80% minimum, with penalties below that". The intention is good, but the effect is almost always the opposite: the vendor picks the definition that gives it that figure, and the bot learns to avoid escalations.

You end up with a guaranteed 90% containment rate and no way of knowing how many customers left without an answer. And a bot pushed to "hit the number" avoids handing over to an agent, which is exactly what your customers hate.

Require the measurement method rather than the result: a strict definition, 100% coverage, auditable verdicts and a quarterly review of the rate on that basis. When the figure can be read conversation by conversation, it improves on its own.

See what "measured" means on a real instance

A 30-minute demo of AI Quality Monitoring, with the unresolved conversations screen and the justification for every verdict.

👉 Book a demo

Frequently asked questions

What resolution rate should you ask for in an AI chatbot call for tenders? It is better not to impose any figure. Ask for a strict definition (request dealt with, correct escalations included, silence not counted), coverage of 100% of conversations and auditable verdicts. With this definition, a range of 50 to 85% is credible; a figure above 90% almost always measures something else.

What is the difference between containment rate and resolution rate in a set of specifications? The containment rate counts conversations that stayed within the bot with no transfer; the resolution rate counts those where the request was dealt with. Ask for both, with the first calculated on the total number of conversations.

Should you require a contractual resolution rate with penalties? No. The vendor would pick the most generous definition and tune the bot to avoid escalations. Require the measurement method and a quarterly review of the rate instead.

How can you check at the presentation that a vendor genuinely measures resolution? Ask for the unresolved conversations screen from the demo, then have them open a conversation counted as resolved and ask why it is.

Should scripted journeys be assessed in the same way as generative AI answers? Yes, against the same criteria. It is the only way to compare the two on your own topics, and to decide where to roll out generative AI next.