Inside the Black Box: Analyzing your LLM Bot

From Traditional Bots to LLMs

Transitioning from intent-based bots to Large Language Models (LLMs) is a shift from constrained, knowable models with defined outputs to open-ended systems which reference knowledge bases to answer user queries flexibly. New tools are needed to curate these knowledge bases to ensure LLMs are leveraging high quality data - not noise. Similarly, new tools are needed to store, transform, and analyze LLM output; Dashbot’s tools are uniquely positioned to demonstrate what your users say, how well your bots respond, and many ways of analyzing bot performance and automation opportunities. Before understanding the most effective ways to deploy and optimize a LLM bot, the traditional approach needs to be summarized for context and comparison.

Traditional, intent-based chatbots feature a straightforward structure which simplifies the process of model performance analysis. Each intent is explicitly designed and composed of training data either written by humans or synthetically generated. This explicit definition of intents is what determines the scope of the bot. Before releasing an intent model to the public, stress testing can be conducted in various ways, including via inbuilt NLP provider metrics or testing the appropriateness of your training data with NLP tools.

Once your model is live, real interactions with human users provide the highest quality test data - this allows for constant updating of model scope and intent health. For example, new intents may be designed to match unforeseen user input, or overlapping training data across different intents can be disentangled.

Need for LLM Bot Analysis (Including CustomGPTs)

The explicitly designed scope and structure of an intent model is in principle absent in LLM bots. Even in the scenario where users create LLM bots with the functionality of custom GPTs - complete with established guardrails, relevant knowledge bases, and predefined actions and functions; you still have no idea how your LLM performs in the real world or how well it sticks to the rules and data you supposedly constrained it with - hence the importance of analytics.

Dashbot’s Solutions

Dashbot presents a number of critical analyses for LLM-based chatbots:

Data Slicer: User Input & Satisfaction Analysis

While LLMs have no intents to analyze, Dashbot can break down any text data (both what your users say and what your bot says) into topics and ‘reasons’ (reasons represent what the user is seeking to achieve) - these are roughly the analogues to intents in more natural human conversation. The Data Slicer provides as general or as fine grained an analysis as desired and can match topics/reasons with many other dimensions of analysis, such as: user frustration score, customer effort score, AI-predicted CSAT, sentiment, resolution rate, and many more. Dashbot also provides custom metrics and dimensions of analysis to suit specific needs.

Sample Data Slicer page:

Knowledge Base Analysis: What are the biases?

Explore the reasons, topics, and semantic relationships within your knowledge base. As you build out the articles that your LLM pulls from, there may be hidden biases or areas of improvement that aren't obvious without deep analysis. This feature gives you a scope breakdown of your LLM’s knowledge. Gain insights into topic distribution and identify areas requiring attention.

LLM Output Analysis & Conversation Flows

Assess the relevance and accuracy of LLM responses. Our advanced tools allow us to determine how well your LLM bot is answering user questions based on automatically detected issue resolution rate and other metrics.

We are currently investigating a tool to determine whether the LLM is answering based on your knowledge base or hallucinating.

Our Flows tool visualizes user journeys based on how users progress through your traditional bot’s intents. We are in the process of making Flows LLM-friendly by visualizing user journey through topics/reasons extracted from your user conversations with LLM bots. Gain insights into user engagement patterns and optimize bot flows accordingly.

Sample Flow:

Conclusion

By leveraging Dashbot's suite of analytics solutions you can understand your LLM bot traffic to ensure you meet user expectations while driving meaningful outcomes.

Ready to unlock the full potential of your LLM-based chatbot? Sign up for a demo with Dashbot today and take your conversational AI to the next level.