Measuring resolution rate and CSAT with AI agents
If you cannot measure it, you cannot trust it. These are the few numbers that tell you whether your automation is actually helping customers.
Deflection is a vanity metric
Deflection counts conversations that never reached your team. It says nothing about whether the customer got help. A confusing bot that makes people give up scores a great deflection rate while quietly damaging your brand.
Resolution asks the harder, more useful question: was the problem actually solved? That is the number automation should be judged on, because it is the number customers experience.
Define resolution strictly
A conversation counts as resolved when the customer got a correct, complete answer and did not come back about the same issue. In practice: no reopen and no repeat contact on the same topic within a set window, for example seven days.
Be strict about the definition, because a loose one flatters the numbers and hides the gaps. A customer who asks in chat, gets a half answer, and then emails the next day is not a resolution. It is two conversations and one frustrated person.
Measure CSAT honestly
Ask right after the conversation ends, keep it to one question with an optional comment, and use the same scale everywhere so numbers stay comparable.
Then read the comments, especially on automated conversations. A single sentence like the bot kept repeating itself tells you more than a decimal shift in the average. Compare automated and human CSAT side by side: the gap between them tells you honestly how far your automation has come.
Watch the handoff line
The handoff rate, the share of conversations escalated to a person, is your most direct signal of where automation reaches its limits. The absolute number matters less than the trend and the reasons.
Rising handoffs on one topic usually mean a missing or outdated knowledge entry, and that is good news: it is a fixable gap with a name, not a vague quality problem.
A dashboard that fits on a sticky note
You do not need forty charts. Five numbers, reviewed weekly, will tell you almost everything:
- Resolution rate for automated conversations.
- CSAT, split by automated and human.
- Handoff rate, with the top three handoff reasons.
- First response time across channels.
- Coverage: the share of incoming questions your knowledge base can answer.