Inconvenient truths about how we measure Customer Service, and how AI will save us

Customer Service leaders care most about one thing: the quality of the customer experience they deliver.

That’s wonderful. That’s the good news.

The bad news is that most don’t measure it properly, or well, or sometimes at all. If that’s you and before you rage close this, think about it, we both know it’s true. We have some inconvenient truths to face up to.

The better news is that AI is going to save us, I’ll get to that at the end.

Inconvenient truth 1: Everyone uses different metrics

If this was an excellent disciplined area, we would all agree on the most important metric, the second most important, and so on. But we don’t. We all care about completely different metrics, and we bend them to suit ourselves.

Some obsess about First Response Time. Some don’t even look at it.

Some obsess about Average Handling Time. Some don’t even look at it.

What about First Contact Resolution? Same story.

And don’t forget CSAT, we use it, but it seems we all agree in private that it is a terrible metric.

Inconvenient truth 2: We don’t measure the actual experience of our customers

What’s a good customer experience? A satisfied customer. So let’s talk about CSAT.

CSAT has always been a metric people love to hate. It is a crude measure that captures no nuance, nor whether the customer query was actually resolved. Customers famously only answer 1 or 5 on the scale. 1 being extremely dissatisfied, and the experience was terrible. 5 being satisfied. OK, but sometimes that satisfaction isn’t real, it is a customer not wanting to get someone in trouble, hitting the 5 despite being entirely dissatisfied.

The worst thing about CSAT though is that we’re only getting answers from the 10% of customers who bother to fill in the survey. So for 90% of customers, the vast majority of them, we’re completely blind as to whether we delivered a great experience.

Inconvenient truth 3: Everyone adds on a lot of intuition and vibes

Once we analyse our numbers, FRT, AHT, or whatever you care about, we then add a new lens: subjectivity and vibes. We use our gut, our instincts. Makes sense: everything quantitative we use is a proxy, so we need to layer on how we feel things are going. 

The numbers say x, but it doesn’t feel like that. So we report on how we feel, as much as on what we see. We bring some art into the science. But measurement should be science.

OK, we are where we are, here is the good news.

The good great news is that all these metrics were designed for the world of 100% human support, and that world doesn’t exist anymore. We are already deep into a world of AI and Humans delivering customer service together.

So we need, and are getting, new metrics. Not only that, we’re getting some superpowers to go with them.

Resolution Rate

A hard metric that measures whether the customer query was resolved. Customers are explicitly asked if their query was resolved. And we can use AI to double check.

AI CSAT

Now this metric is like magic. It solves both of the problems with CSAT (low coverage and unreliable) and is really a ‘Customer Experience Score’.

AI can analyse every single customer conversation (100% coverage!), and can reliably and accurately do two things:

• It can determine the customer sentiment. Was the customer happy? Sad? Did they start with one emotion and end up with another?
• It can determine whether the query was actually resolved. Because AI knows from conversation history what an excellent, accurate resolution looks like for any given query, we can have AI check if the query was properly resolved. No fake dissatisfaction here.

Also, people don’t care if they get the AI in trouble, so they are honest in their appraisal. 

‘AI CSAT’ though? We might need a new name.