Why Thematic is superior to OpenAI for analyzing feedback
With easy access to OpenAI's ChatGPT product, organizations are exploring its abilities. Can they use it as a tool to quickly analyze data sets, such as survey responses and customer feedback?
Yes, OpenAI is a powerful tool. It is useful to analyze smaller datasets and to get a quick overview. However, it is not capable of the accurate and detailed analysis that can be achieved with specialized feedback analysis tools.
We had a chat with Thematic CTO Nathan Holmberg and Rob Dumbleton, Head of Data Science & Research to understand this more. We talked about how to use ChatGPT to analyze a dataset of feedback comments, how it can help and what to watch out for. Nathan and Rob also shared how to overcome the limitations of OpenAI in analyzing feedback with Thematic.
Let's delve into what Nathan and Rob discussed:
First, what are we talking about when we say ‘OpenAI’?
OpenAI has become a catch-all term for new generative AI models and approaches.
OpenAI is the parent company, while their products bundle their generative AI into an interface where you engage with the large language model (LLM); ChatGPT is the model that most people are now familiar with. ChatGPT3.5 is the free, most commonly used chat interface, while ChatGPT4 is a more advanced version with a larger (and more expensive) underlying model.
Anyone who has experimented with ChatGPT, particularly 3.5, will know that it still comes with the potential for frequent inaccuracy and hallucination. While ChatGPT4 offers improved accuracy, the downside of this larger model is slower processing - around twice the time it would take on the smaller model.
Understanding OpenAI’s strengths and weaknesses
ChatGPT has its place for doing simple analysis of a limited data set. However Nathan and Rob shared that it does come with some clear limitations.
- ChatGPT is clever but lacks the insight you need
ChatGPT excels at generating human-like text based on the input it receives. It can understand context, generate creative content, and even engage in meaningful conversations. However, when it comes to feedback analysis, OpenAI's capabilities are broad and general, lacking the specific focus required for in-depth insights into user sentiment and preferences.
- Get a quick analysis (but don’t expect it to be reliable)
ChatGPT’s prerogative is to give you a satisfying answer, but with feedback analysis, what we’re really looking for is an accurate one; we need a true picture of customer needs.
ChatGPT becomes problematic when it draws on its training data, rather than solely extracting from the data you’ve provided it. This means responses can be questionable in their accuracy and their consistency.
Good, clear prompts help you to correctly filter the data and rely on it for relevancy, but finding the right prompt for every question is an inefficient way to manage the analysis process. Particularly when you have a large dataset and many questions to answer.
- Helpful for small, simple data samples
It’s super simple to take comments from a survey or reviews, input it into ChatGPT and ask it to provide you with general sentiment based on those responses. For example, you might ask, "What’s the biggest concern for our clients?”. This works well for initial exploratory research if you’re working with 100 or fewer responses, where you can also take the time to cross-check validity yourself. It will give you a good feel for overall sentiment, but not an overly accurate report.
- A longer way round to the answer you need
ChatGPT’s reliability drops as your dataset grows, and it becomes unrealistic to have to read and validate its analysis for accuracy yourself. You can also prompt ChatGPT to provide you with evidence. For example, if you are extracting themes from responses, you can ask Chat GPT to show the relevant data it used to reach its conclusions. However, again, it’s not the most efficient way to cross-check accuracy.
- Good for one-time inputs
ChatGPT has also proven ineffective when it comes to tracking responses over time, comparing time periods or drilling down into quantifiable details, for example asking it to analyze and compare data across different regions. ChatGPT will struggle if you have to input data across multiple prompts or need to add new data sets, and can’t segment by specific criteria or responses.
What can Thematic do that ChatGPT can’t?
As insights professionals, your role is to accurately represent and report on the voice of your customer. If analysis isn’t accurate, and insights can’t be trusted, your customer's voice will essentially be ignored - the very opposite of what you’re trying to achieve. That’s why insights reporting should be precise, reliable and consistent in the type of data they draw on.
If stakeholders detect inaccuracy or inconsistency in the analysis, they will ignore the insights outright. So AI insights must build confidence and trust, and be able to be traced back to the original interaction."
Robert Dumbleton, Head of Data Science and Research, Thematic
Insight analysis about consistency, validation and evidence. Big, strategic decisions need to be based on quantifiable facts, not generalizations. At Thematic, we consider ChatGPT as a tool, but not a solution that gives you the level of accuracy and insight you need.
- Accurate and fast insights with LLMs and our own algorithms combined
To deliver specific and accurate answers, we use large language models (LLMs) such as GPT4 along with our own algorithms. This makes it easier and faster to get granular and reliable answers. That’s because our proprietary algorithms do the work to ensure that generative AI only analyzes useful and clean data.
Part of this process - a really important part - is that a human validates Thematic's AI's analysis. With our built-in tools to guide the AI, a human can check the predictions, and refine themes to fit their company's decision needs. This makes it easier to get specific insights for decision-making and capture the full voice of the customer. It dramatically reduces the time it takes to produce reports relating to a specific inquiry.
- Analyze large volumes of feedback
A significant accuracy advantage comes from Thematic’s capacity to process larger amounts of feedback, without needing to do batch analysis. Thematic can generate all the relevant themes (up to 2000, if you wish) while ChatGPTs has a very limited scope.
If GPT did analyze all the unstructured feedback data at once, it would get all the noise and irrelevant comments, resulting in irrelevant themes. If you ask GPT to rerun the analysis, results will be inconsistent, because it will draw on a different batch of feedback.
- Reliable data security & compliance
Importantly, you know that your data is secure. Thematic uses OpenAI through Azure, meaning it is GDPR and SOC2 Type II compliant. With GPT, there are risks around the security of your data and compliance with EU regulations and data is potentially used for training purposes. If your organization cares about privacy and protection of data (and it should), OpenAI offers you no guarantee about where your data is consequently stored or used.
- Visualization for insights & decision making
Thematic’s user interface is built specifically to help decision makers get all the layers of data insights they need to deliver to their stakeholders, by identifying and visualizing the drivers of satisfaction or score changes. Emerging trends in customer feedback and their impact on the business can easily be gleaned through reporting. We've identified what insights leaders need to deliver to their stakeholders and ensured that they get all that in one platform.
Dive deeper into Thematic with LLMs
Thematic CEO and Co-founder, Alyona, recently shared her experience experimenting with ChatGPT to analyze feedback - read about her experience, how it worked and where it fell short.
If you’d like to explore the insights you can get with Thematic with generative AI, book some time with our helpful team to get a free demo on your data.
