The AI agent buyer’s guide: How to make the right choice for your support needs

Working with AI is no longer an option for customer service teams – it’s a necessity. With an influx of new entrants promising powerful AI agents, choosing the right solution has become increasingly complex.

Making the right choice is a challenge for today’s support leaders for a number of reasons. Solutions range from basic chatbots to much more sophisticated AI agents, but elaborate marketing claims make it hard to tell them apart without hands-on testing.

“To make an empowered decision, support leaders need a systematic way to cut through the noise”

Integration requirements vary widely, cost structures are complex, and it’s difficult to predict which solutions will remain viable long-term. Faced with a market that’s moving incredibly fast, you need to trust not only that your vendor can keep pace with what’s current, but also that their solution truly aligns with your unique business needs, company values, and customer expectations.

To make an empowered decision, support leaders need a systematic way to cut through the noise. In this comprehensive guide, we detail exactly how to evaluate AI agents based on what matters most. To distinguish between truly powerful performers and basic tools wrapped in shiny (thin) wrappers, we’ll walk you through how to:

• Audit your support profile to help you understand exactly what you need from an AI agent.
• Identify your evaluation criteria to ensure the options you’re considering tick all the boxes.
• Test different AI agent options by applying a rigorous framework that’ll reveal how they really stand up.

Choosing the right AI agent to keep ahead of your competitors and deliver excellent customer service is crucial. Let’s explore everything you need to know to make the best decision.

Understanding AI agents

Automation in customer service has evolved dramatically with the advent of generative AI and large language models (LLMs). While the conversational interface looks the same, AI agents fundamentally differ from the chatbots that came before them, and understanding how they operate is important as you try to assess the options in market.

Key differences between chatbots and AI agents

 

• Traditional chatbots rely on rigid, rules-based systems, using decision trees and pre-scripted responses to simulate conversations. They require extensive manual configuration to detect keywords and deliver relevant manually curated responses.
• Modern AI agents are powered by LLMs and can understand natural language, interpret and remember context, and generate human-like responses. They can process and synthesize vast amounts of information from sources such as knowledge bases, enabling them to automatically understand and resolve many informational customer queries in natural conversations.

This technological leap has brought with it massive efficiency gains. For instance, Intercom’s Fin AI Agent resolves an average of 51% of customer queries out of the box with 99.9% accuracy.

As AI agents become more sophisticated, they are expanding beyond conversations and will be able to take actions and handle even more complex tasks, further transforming customer service.

Audit your support profile

Before you dive into exploring options from different vendors, take some time to understand what your “support profile” looks like. Performing an analysis of the queries your team encounters every day will help you identify exactly what your needs are and where an AI agent can help.

Consider the following four factors:

Support query volume

The number of queries you receive will be a major factor in your choice of AI agent. You’ll need to compare the AI agent’s potential resolution rates, pricing models, and their ability to work with your human support team.

Support channels

Where do your customer queries come from? If you’re working across more than one channel, as most support teams are, you’ll need to know the AI agent’s capabilities for handling your queries and ensuring a consistent customer experience.

Query complexity

Informational queries are at the heart of customer service – for most businesses, the majority of tickets involve customers seeking answers to questions. That’s where AI agents already excel.

Take a close look at how your team currently handles queries about things like product features, pricing, policies, and technical specifications. Do seemingly simple questions often require more nuanced responses? Are your team members having to combine information from multiple sources to piece together answers?

“As AI agents evolve, they will become more and more capable of handling personalized and action-based queries”

Use these insights as you start evaluating different AI agents. You’ll discover when you reach the testing phase that different options vary significantly in their ability to resolve informational queries – there can be significant variation in accuracy and resolution rates.

As AI agents evolve, they will become more and more capable of handling personalized and action-based queries, so you should also consider whether the vendors are discussing features such as this in their product roadmaps.

Knowledge content health

The current state of your support content will impact how quickly and effectively you can deploy an AI agent. Consider these questions when assessing your knowledge base:

• Is the content up-to-date?
• What major gaps need to be filled?
• Is everything centralized in one location, or is it scattered across different platforms?
• Do you follow a clear process for maintaining your content and keeping the information current?
• How easy is it to find the information you’re looking for?
• Are there valuable knowledge sources outside your knowledge base that could enhance your AI agent’s capabilities, such as website content, internal process documents, or PDFs that explain product features?

Many teams find that preparing to implement an AI agent provides them with a great opportunity to audit and optimize their knowledge content. A consolidated, well-maintained knowledge base not only makes it easier to train your AI agent, but also enables it to provide the most accurate, consistent responses to customers as soon as it’s up and running.

It’s also worth noting that modern AI agents’ ability to draw from multiple sources, like pricing pages, technical documentation, and product guides, enables them to construct richer, more comprehensive answers than traditional support tools. This empowers support teams to leverage content from across the entire organization, expanding their influence and ability to deliver expert assistance on any topic the company has documented.

Identify your AI agent evaluation criteria

Once you’ve understood what your team’s unique needs are, you can start assessing your options. This evaluation process involves two distinct parts:

1. Your internal adoption requirements.
2. The customer-facing capabilities you need.

Internal adoption considerations will help you determine how smoothly the AI agent can be implemented and maintained by your team, while customer-facing features will ensure the solution can deliver the best level of service for your customers.

1: Internal adoption requirements

There are four foundational areas to consider, namely:

• How long the AI agent will take to set up.
• How easily it can integrate with your existing support platform.
• How much it costs.
• How it handles data privacy and security.

Setup time: Choose an option that’s quick to get started with

Working in customer service is busy (to put it lightly), and most teams don’t have a spare minute, let alone hours, to dedicate to a complex AI agent setup process. Setup times can range from a few minutes (in the case of Fin AI Agent) to days, or even weeks – so be sure you know what you’re letting yourself in for.

“The sooner your AI agent can assimilate with your team’s processes, the sooner you can start to see real results and return on your investment”

Consider the resources you have on hand to get your AI agent up and running: which team members can you spare, and for how long? How much time can you give to team training? If the answer is “not much”, it’s worth prioritizing an AI agent that can hit the ground running.

The sooner your AI agent can assimilate with your team’s processes, the sooner you can start to see real results and return on your investment.

Find out:

• How easy is it to get started with this AI agent?
• Will this AI agent require training before it can begin to bring real value to your team?
• What kind of resolution rate can you expect to see right out of the box – and how will it improve over time?
• Will the AI agent require ongoing technical configuration to operate effectively? If so, does your team have the necessary resources?

Ease of integration: Get up and running without having to overhaul your current support platform

Every support team has carefully planned processes and workflows that are optimized for efficiency and an excellent customer experience. It’s natural to fear that bringing in a new AI agent could disrupt the way the team operates.

An AI agent should work with – not parallel to – your current setup. Some AI agents operate as add-ons to whichever platform you’re using, while others are native to their platforms.

Find out:

• How will this AI agent work with your existing tech stack?
• Do you need to purchase other tools alongside the AI agent to maximize effectiveness?
• What kinds of integration capabilities will you need to get the most from your AI agent?

Cost: Pick a pricing model that works best for you

When it comes to AI agents, it’s incredibly important to understand the kind of pricing model that will provide the best ROI for your team.

AI agents are being priced in many different ways, but these models essentially boil down to two pricing philosophies:

• Pricing against outcomes: This approach is based on measurable results that impact your team performance. An example would be pricing against resolutions, which is what we’ve chosen to do at Intercom. This means you only pay when your customer has received a satisfactory answer to their question without having to be passed to your support team. This approach is value-based, measurable, and makes it easy to tell the kind of ROI your AI agent is providing to your team.
• Pricing against usage: This approach involves pricing based on API requests or messages. It can often be difficult to ascertain ROI when your AI agent is priced against these metrics because they don’t account for actual resolutions. It’s hard to be sure whether your customer got the answer they needed, or just left the conversation frustrated.

“Our most successful Fin customers think about the return on investment through two lenses: increased bandwidth and cost efficiency”

Consider what will ensure the highest ROI for your team by comparing prices against the cost of adding headcount, and against the savings you’ll gain from improving team efficiency and dedicating more time to driving customer value and satisfaction.

At Intercom, our most successful Fin customers think about the return on investment through two lenses: increased bandwidth and cost efficiency. Their goal is to run an efficient organization where every dollar is put to good use. Here are two important things to consider when quantifying the ROI of AI first customer service:

1. Purchase price vs long-term value: When you’re assessing AI agents, it can be tempting to look solely at the price per resolution. But what moves the bottom line the most – and delivers the greatest ROI – is actually the tool with the best resolution rate and performance rate.
2. Total cost of ownership: To get the full picture, you also need to think about the total cost of ownership, which includes all of the “hidden” costs that come with adopting any new tool.

Download a copy of our new guide, The New Economics of Customer Service, for more insight into how AI has shaken up the linear growth model of customer service and revolutionized how teams think about the ROI of AI.

Find out:

• What metric are you using to determine how much your team will pay for the AI agent?
• How are you defining this metric?
• What is your cost to serve? How does it compare to the price per unit, i.e. per resolution, per message, or per deflection, depending on your chosen AI agent?

Protecting your customers: Be upfront about your data privacy and security requirements

Data privacy and security is a priority for every company, but depending on the nature of your business, it may be your number one concern.

AI agents use LLMs to respond to the questions they receive, meaning the information your customers offer will be processed through any of the increasing number of LLMs available, whether it’s Claude 3.5, GPT-4o, Gemini, or any other.

In order to assure your customers that their data will be safe, it’s important to understand the LLM your chosen AI agent works with, and how it handles your data. Below are some questions you can ask to dive a little deeper into the suitability of an AI agent for your business.

Find out:

• How will your customer data be used? Will it be shared with, or stored by, the LLM provider?
• Is your customer data being used to build AI models by your current provider? Are you comfortable with this?
• Will customer messages be encrypted?
• Does the AI agent vendor have a partnership with an LLM provider?
• Where is your data hosted, and will the location affect your ability to use the AI agent?
• What legal agreements, including NDAs, are required before testing the AI agent with real customer data?

Pro tip: Start discussions about NDAs and data protection agreements early in your evaluation process. Many companies find that getting these agreements in place takes longer than expected, which can delay testing and implementation. Having these conversations upfront will help with a smooth transition from evaluation to deployment once you’ve decided which AI agent you want to go with.

2: Customer-facing capabilities

After ruling out any AI agents that don’t suit your internal requirements, you can move onto evaluating the quality of support the AI agents can actually deliver. There are a million ways vendors are marketing these features, but ultimately, characteristics of exceptional frontline support fall into these categories:

• Knowledge
• Behavior
• Actions
• Insights

Consider what each AI agent on your list offers within these categories and how you can leverage this to provide the best experience for your customers.

Knowledge: Intuitively deliver the most accurate, comprehensive answers

All AI agents rely on quality knowledge content to function. But how they use this content to deliver results can differ drastically.

Strong performers operate like your most experienced team member, seamlessly drawing from your entire knowledge ecosystem to craft complete, accurate answers. They can instantly learn from all available sources, including help centers, internal documents, PDFs, and URLs and process this information at lightning speed. Most importantly, they’re able to come up with responses to even the most complex questions by combining relevant information from multiple sources, ensuring customers receive comprehensive answers that solve their problems completely the first time.

Fin has the ability to combine knowledge from multiple content sources to create tailored answers for your customers.

Not all AI agents can handle this level of complexity. These options tend to offer single-source responses rather than true information synthesis. They typically require lengthy training periods before delivering real value, need constant manual updates to stay current, and frequently miss crucial context by failing to connect related information across different sources. This limitation will lead to incomplete answers, frustrated customers, and increased workload for human agents who need to fill in the gaps.

Find out:

• What types of knowledge sources can the AI agent integrate with (help centers, internal documents, PDFs, URLs)?
• How quickly can the AI agent process and begin utilizing new information?
• Does the AI create answers by combining information from multiple sources, or is it limited to single-source responses?
• How easy is it to update or modify the AI’s knowledge base as your product evolves and processes scale?
• What mechanisms are in place to ensure the AI provides accurate, up-to-date information?

Behavior: Mirror your human team to ensure a seamless interaction experience for customers

The spectrum of AI behavior control ranges from basic tone-of-voice settings to comprehensive workflow management. Some AI agents operate with fixed responses and limited flexibility, while others can be trained like human team members – learning your policies, adapting their communication style, and making nuanced decisions about when to handle issues versus when to escalate them. The key difference lies in how naturally these behaviors can be implemented and modified.

“When an AI agent can be trained like a human team member, it doesn’t just handle inquiries; it becomes an integral part of your customer experience strategy”

Consider your current support team guidelines: What policies must be strictly followed? Which situations require human intervention? How do you handle multilingual support? Your AI agent’s ability to understand context, follow workflows, and seamlessly integrate with your existing processes will determine its effectiveness as a true front-line support solution.

When an AI agent can be trained like a human team member – understanding policies, detecting customer sentiment, and adapting its behavior accordingly – it doesn’t just handle inquiries; it becomes an integral part of your customer experience strategy. The right combination of guardrails and flexibility means you can trust your AI agent to represent your brand while maintaining compliance and consistency.

Find out:

• How can you control the AI agent’s tone and communication style to match your brand voice?
• What mechanisms exist for creating and modifying behavioral workflows and rules?
• How reliable and robust are the mechanisms to ensure workflow and policy adherence and prevent unauthorized or mistaken actions?
• How does the AI agent handle multilingual support and cultural nuances?
• Can the AI agent detect and appropriately respond to customer sentiment?
• How does the AI agent determine when to resolve issues versus when to escalate to human support?
• Can you establish different behavioral rules based on customer segments or channels?

Fin supports multilingual interactions, automatically translating your content in real-time to match the customer’s language.

Actions: Take meaningful steps independently to resolve customer issues

AI agents are developing fast, and the next phase will see them increasingly able to take independent actions in response to customer queries.

This capability is still emerging, but forward-thinking support teams are already planning for a future where AI agents will progress from information retrieval to executing complex, multi-step processes.

When considering AI solutions today, it’s crucial to understand vendors’ plans for developing these capabilities. You want to be sure their roadmaps are aligned with your future needs so that the AI agent you choose now continues to serve you in the long-term.

Find out:

• What is the vendor’s roadmap for developing action-taking capabilities?
• Which third-party integrations are they prioritizing?
• How are they approaching security and authorization for automated actions?
• What is their timeline for implementing these features?
• How do they plan to give customers control over AI permissions?
• What safeguards will they put in place for automated actions?

Fin will be able to retrieve customer data and provide answers specific to each customer, like checking recent orders.

Insights: Get a deeper understanding of performance metrics to drive better results

The ability to measure and improve your AI agent’s performance isn’t just about tracking metrics – it’s about understanding the true impact on your customer experience and business outcomes. For support teams, comprehensive insights mean the difference between flying blind and having a clear view of service quality across every single customer interaction.

When choosing your AI agent, consider what kind of reporting you’ll need for the following areas:

• Determining the ROI of your AI agent: You’ll need a reporting system that offers robust insights into how the AI agent is affecting your most important metrics.
• Areas for improvement: Some AI agents have the capacity to improve as they ingest more of your support material, and as your team strengthens your support center and and optimizes workflows to boost its performance. A solid reporting system will indicate the areas where the AI agent could be offering more value, allowing you to maximize its strengths.
• Customer satisfaction (CSAT) measurement: Look for systems that are moving towards providing comprehensive CSAT measurements across both human and AI interactions. Soon, the most advanced solutions will be able to offer AI-generated CSAT scoring that analyzes 100% of conversations, giving you way more insight than you currently get from a few filled-out customer surveys.
• Unified performance view: Since humans and AI will work together to handle support, you’ll need holistic reporting that shows how both are performing. A unified view will help you understand the overall health of your customer service operation and how different parts of your support system complement each other.
• Sharing results with the wider company: As every customer service manager knows, pulling these insights is only half the work – the rest is sharing the reports with management and the wider company so everyone can get behind the support team’s efforts and appreciate the value AI is bringing to the customer experience.

The analytics capabilities of AI agents can vary – from basic resolution rate tracking to sophisticated AI-powered analysis of conversations. While some solutions offer only standard metrics, advanced solutions are progressing to a point where they’ll soon be able to evaluate customer sentiment, measure true resolution rates, and provide detailed quality assessments across both AI and human interactions.

Find out:

• What kind of reports are available?
• What metrics are available for comparing AI and human agent performance?
• How will the reporting system work with your current reports?
• How easily can reports be shared with stakeholders across the organization?
• How granular are the quality analysis capabilities?
• Can the system identify specific areas needing content or workflow improvements?

Test different AI agent options

It’s relatively easy to make an AI agent look impressive in a controlled demo, which we call the “AI demo problem.” Similar to self-driving cars that blew people away with their performance on closed circuits, many AI agents can look deceptively fast and impressive in a demo environment answering basic questions.

“Before you make the final call on which AI agent is the best fit, make sure you’re confident about how it actually performs”

To truly test its capabilities, you need to see how it handles real-world challenges and push it with complex or ambiguous questions – and know what to look for in its responses. A good AI agent will respond accurately or admit it doesn’t know. A poor AI agent will confidently provide misinformation.

Before you make the final call on which AI agent is the best fit for you, make sure you’re confident about how it actually performs. You want to be sure that the tool you’re bringing on board will truly be a great addition in reality and not just “passable.”

Here are some steps to thoroughly assess how an AI agent will perform in practice.

Pro tip: Remember that an AI agent’s performance is only as good as the content that it’s trained on. Before running any tests, review the help articles, documentation, and data sources your AI agent will be drawing from. Are they up-to-date? Do they cover all the necessary information? Are there any gaps in your content that need to be filled? Having comprehensive, well-organized content is essential for accurate responses, and will impact the results you get.

Step 1: Prepare your test cases

To truly test how an AI agent will perform in your environment, you need to get specific.

To ensure you’re covering both common and current scenarios, gather 10-20 of your most frequently asked questions and a selection of your most recent queries. This combination should hopefully give you a range of questions that vary from basic to complex, and focus on different areas of your product or service offering.

To go more in depth, you might also want to prepare:

• Complex queries that typically require multiple touchpoints from different team members.
• Phatic or vague queries that don’t contain any “real” information and require further clarification from the customer to resolve.
• Queries spread across multiple turns.
• Edge cases that have been difficult for your human team to resolve.
• A few sensitive scenarios, such as billing disputes and cases where customers have become angry.
• Examples of queries in different languages, if you provide multilingual support.

Step 2: Test variations of the same question

Once you’ve gathered your list of questions, prepare a few variations. Real customers rarely ask questions in the exact same way, so you need to ensure your AI agent can handle different types of communication.

For each test case you prepared in the first step, try:

• Difficult questions that require information from multiple sources to answer.
• Different phrasings of the same question.
• Incomplete or fragmented queries.
• Questions with typos or grammatical errors.
• Various levels of formality.

Step 3: Simulate real conversation flows

Customer conversations rarely follow a straight line from question to resolution. Put your AI agent through realistic scenarios by:

• Starting with vague queries that require clarification.
• Testing follow-up questions to see if context is maintained.
• Introducing new information mid-conversation.
• Changing the topic or circling back to previous points.
• Expressing frustration or confusion during the interaction.

Step 4: Challenge the guardrails

Understanding your AI agent’s limitations is just as important as knowing its capabilities. You want to be sure the AI agent won’t answer something it shouldn’t. Test its boundaries by:

• Requesting information it shouldn’t have access to.
• Asking about competitors or sensitive topics.
• Trying to bypass security measures.
• Using slang, technical jargon, or industry-specific terms.
• Asking it to perform an action it hasn’t been configured for.

Step 5: Document and analyze

For each test scenario, make note of:

• The exact query or action attempted.
• The AI agent’s response or action taken.
• Response time and number of steps required.
• Any unexpected behavior or errors.
• Areas where human intervention was needed.
• Opportunities for improvement.

As you’re working through the testing process, remember that the goal isn’t to find an AI agent that passes every test perfectly, but rather to understand exactly how it will perform in real-world situations with your customers. Pay special attention to how it handles uncertainty, maintains conversation context, and knows when to escalate to your human team.

Speaking of your human team, while it’ll require some planning and dedicated time, it’s a good idea to get them involved in this testing process. If possible, have multiple team members run through these tests independently. Different people will interact with the AI agent in different ways, which will help you uncover potential issues from various perspectives.

Getting your human agents involved is also a great way to bring them along with you on the journey and get their input on which AI agent is the best fit. As the experts who are on frontlines, they’ll be the ones working most closely with it every day, so it’s critical to secure their buy-in.

Welcome aboard, AI agent

Hopefully, this guide has set you up with everything you need to make a confident decision about which AI agent is best for you.

Taking the time to thoroughly evaluate your options will help you lay the foundation for a transformative shift in how you serve your customers. When implemented thoughtfully, an AI agent can revolutionize your customer experience by providing instant, accurate resolutions, while simultaneously empowering your human team to focus on complex, high-impact work.

The result is a win-win-win: happier customers who get faster resolutions, more engaged support teams doing meaningful work, and a business that can scale efficiently while providing world-class service that leaves competitors in the dust.