How Thematic finds insights in large datasets: Analyzing 130k bank app reviews

Feedback Analysis Thematic

Extracting key insights from large datasets sounds daunting but it doesn’t have to be. With access to the right feedback analysis software, you’re hours away from having valuable and actionable insights to power your CX interventions.

The ticket to the game is finding a partner who can funnel large volumes of feedback from diverse sources. Once your data is in one place, the real job is to analyze it effectively.

Sounds like a no brainer but it was only during Alyona's (Thematic CEO & co-founder) PhD research in natural language processing (NLP) that she got it: The thing that makes the biggest difference to extracting key insights and executing on CX strategy, is the way the qualitative data is sorted, interpreted and shared. That was the Eureka! moment that led to the creation of Thematic and the design of our transparent approach to delivering meaningful insights.

To help find clear focus, and deliver better customer experiences, we take all the raw, unstructured feedback - from reviews, survey responses, chats, complaints and support tickets etc. Then, our AI text analytics solution reveals all the themes they contain, plus the sentiment attached to them, so you can see what matters to your customers and what keeps them coming back.

The review, the themes and the tagged sentiment within Thematic

By combining the artificial intelligence of NLP and the human intelligence of our insights experts, Thematic is able to go beyond providing businesses with nuggets of relevant CX info. We provide valuable actionable insights that make it easier to zone in on the business issues and determine those you want to prioritise.

How does finding insights in large datasets work in practice?

We analyzed the public reviews of 35 mobile banking apps to see what matters most to their users:

To define the scope of the dataset, we limited our search to Bank and Credit Union apps with more than 100 current consumer reviews (from June 2020 to May 2021) on Google Play and iOS store sites. That yielded 130k reviews and out of the gate we were able to see: the bank; the store/branch; the country; the review date (including a star rating) plus the App version used.

Some of the banking app brands that Thematic analyzed the reviews of

Then, Thematic started structuring and filtering the information: first by identifying the themes and sub-themes contained in the reviews and second by attributing a positive or negative sentiment to each of them.

Rather than simply relying on keywords to provide direction, Thematic automatically highlights key themes as they come up. The difference being that a theme can be expressed in more than one way, which increases the relevant feedback we are able to capture. And color-coding the sentiment (red for negative, green for positive) makes it easy to see what’s working and what’s not.

Here, our starting point was to identify what made customers happy or unhappy and their 1 - 3 star (Poor) or 4 - 5 star (Good) ratings gave us the “why?” behind the reviews of their banking app experience.

Unsurprisingly, the happiest are those who have a “positive user experience” and the driver (or sub-theme) that came up most was the “ability to deposit checks”. Congratulations to JP Morgan who outperformed all the other banks and got the highest number of positive mentions for this.

The unhappiest are those who experience problems using their App. The data shows the lowest scores come from “issues” around it not working properly. When we dig into the sub-themes, we can see these are mostly associated with “authentication” glitches that occur when Apps are “updated” and users have to “reinstall” it before they can “log in” to their accounts.

“Account information” is another hot button theme and, from the sub-themes that come up, we can see that some of the greatest causes of unhappiness occur when “balance” or “transaction” information is inaccurate.

Getting granular

Using Thematic it was effortless to extract super-granular data providing answers to some of the compelling questions bank app product managers are asking. Like, what makes for a better mobile bank outcome? Which banks are delivering on key features? Where are the biggest opportunities and issues?

Had we been working with an internal dataset, we’d have been able to pipe metadata into the platform and give clients the ability to filter and analyze feedback on variables such as customers’ transaction volume, value or age. Because the purpose of this article is to demonstrate how key insights are extracted from large datasets, we’ve opted for less granularity. But simply knowing whether a customer uses an iPhone or an Android device for mobile banking turns out to be full of interesting and valuable insights.

For starters, there’s a significant difference in their ratings! iPhone users are less happy across all mobile banks (averaging 3.6 stars) while Android users give 4 stars.

When we look at why, the data circles back to the “deposit check” feature, revealing that Apple users are three times more likely to mention it and, when they do, it’s most often in a negative light.

Apple users mention 'deposit checks' 4% more than Google Play users, and 73.5% of the time it's not positive

Why? From the comments expressed by this iOS reviewer, perhaps it’s that iPhone users have higher expectations or are harder to please. The negative sentiment attached to the “features” theme and “deposit checks” sub-theme would suggest so.

Discovering unknown unknowns

One of the delights of a large dataset is the untold treasure that lies within! Benchmarking can be a powerful way to reveal unknown unknowns and get answers to business-altering questions such as: Do bigger banks deliver a better mobile app experience? How could a Credit Union use a mobile app as a competitive advantage? How can an app’s strengths and weaknesses be validated?

Benchmark Example 1 – Top players vs the rest

To assess whether size matters when it comes to the mobile experience a bank can deliver, we looked at the apps of the five biggest banks (US Bank, Citi, Wells Fargo, Bank of America and JP Morgan). Their ratings are, indeed, higher at 4.4 stars versus an average of 4 for the others. And a glance at the base theme “ease of use” tells us this is because they generally perform the way they are supposed to and generate fewer complaints about core issues, login difficulties or broken updates.

Benchmark Example 2 – Category leaders: big Bank vs small Credit Union

But does that mean banks need a big app budget to deliver a great user experience and get amazing reviews? To explore that, we used the data to compare the best rated Credit Union and Bank apps and what an illuminating benchmark that turned out to be!

Though ESL and Citi both scored the best ratings in their category (4.7 for ESL, 4.6 for Citi) they are vastly different in size, resources and approach. So it was fascinating to see that the much smaller Credit Union app earned twice as many positive reviews for “reliability”, putting it on top for user experience.

Benchmark Example 3 – One bank vs the rest

Another revealing yardstick in a large dataset like this can be to understand what drives the performance of one bank against the rest. At random, we looked at TD Bank to explore the bank’s unique strengths and weaknesses. It is well rated for being “faster” than other apps, which gives it a clear competitive advantage.

However, a look at the negatives shows that the sub-theme with the most mentions was “does not work”, an indication that it’s less reliable than other apps. From the timeline we can see they got most of their 1 to 3 star ratings for “does not work” in the last 30 days of the period under review.

14.8% of 'issues' for TD Bank mention the sub-theme 'does not work'

Digging deeper into the themes we can see that, once again, it’s “mobile check deposits” that is the issue and a sizeable 7% of comments mention it.

What’s interesting here is the direct correlation between the number of times this core feature is given a negative mention and the impact it has on the bank’s rating. The timeline data shows the problem was at its worst in October and, now that TD Bank is fixing its issues, the good news is that its ratings are climbing.

Conclusion

This has been a quick look at how Thematic extracts key takeaways from large volumes of data using a combination of artificial and human intelligence. And, ironic as it may sound, the most profound insight is … get the basics right!

By identifying key themes and experiences across 35 banks, we’ve uncovered the most important messages mobile bank customers want their banks to understand:

More than anything, they want a “reliable”, “easy to use” “check deposit” feature
“Authentication”, “reliability” and “speed” can all be competitive advantages
“Updates” need to be tightly tracked so issues can be identified and fixed before they have a negative impact on user experience.

It took just three hours to set up the dataset, refine themes and distill these critical insights from the feedback. And closer analysis would reveal far more. So just imagine what we can do with your customer feedback. If you’d like to take a deep dive into any aspect of your business, you can receive a private walkthrough of your data by booking a demo of Thematic.