skip to Main Content

KODEE XPLAN CHATBOT

The first week, as told by data

November 13, 2017 |  matthew-townsend

Looking through the data lens

Kodee – the experimental xmerge chatbot – was recently introduced to the greater xplan community and made available to all, 24/7.

The initial response and usage of Kodee has been positive and already yielded the first insights into the how, what, where’s and when of conversational xplan and xmerge engagement.  The below is a sample of some broad data points:

ACTIVE SESSIONS
887
27 Oct – 3 Nov

XMERGE SEARCHES
1,623
27 Oct – 3 Nov

Some positive numbers to kick off Kodees first week.

Over the course of the week the average searches per user increased; likely as people moved from quickly checking out Kodee to using it / testing it more thoroughly.  I would expect a decline in users over the coming weeks but with a steady level out.

Xmerge Searches

insights into what and how users search for xmerge.

One of the exciting things about this experiment is the potential of being able to analyse and visualise data around what xmerge users are searching for, looking for help on or just curious about.

The word cloud to the right shows the aggregated areas of xplan from all the xmerge searches over the 1st week.  It’s no surprise that ‘client details‘ would be the most frequently searched category – this includes things like client age, date of birth, preferred name etc.

We can always expect some level of client details in the search results, although overtime I would anticipate that will reduce as users start to search for xmerge syntax that isn’t as accessible or as documented.

XPLAN XMERGE Word Cloud

Click the image to view in full-size

Breakdown of how users search xmerge:

Specific search patterns 94%
Phrase (NLP) search patterns 6%

Types of user searching patterns

The earliest versions of Kodee were designed heavily around NLP and machine learning, so users could ask Kodee complex questions (phrases) like “what’s the xmerge for today date plus six months“.

Early user testing revealed an unexpected twist: in the vast majority of cases, users would search with specific search patterns such as: “clients date of birth” or “family tree syntax“.  Even when the testing was biased towards phrase based searches (to test the NLP) they still mostly ended up in specific patterns.

The data from the 1st week strongly supports the decision to re-design around specific search types first and look to build up to the advanced patterns over time.

Most searched query:

‘First Name’

Would imagine most of this is users just throwing Kodee some easy hits to test it out initially.

Surprise popular search:

‘IPS’

Modelling tool  syntax was not prioritised for Kodee’s initial launch and those looking for IPS could see that from the limited results returned.

Most Random query:

‘Xports’

Not sure what xmerge people feel should exist for this area, but great question for an xplan bot 😉

Search Stats

Lots of successful search results returned but a higher than expected failure…set back rate.

Whilst the overall failed response rates are higher than I would have liked, it wasn’t unexpected.  An early design decision in the way users need to initiate Kodee’s ‘xmerge search’ has contributed to the majority of the failure rate.  This did come down over the week, users who spent more time on Kodee worked out the back to base search function.  With UX and response improvements to Kodee, I believe we can get this to drop right down in the coming weeks.

The other large contributor to users not getting a successful result is around content that simply doesn’t exist yet. As Kodee is an ongoing experiment, so too are the further xmerge and content additions to the xmerge database – each week more are added to close the gaps, based on the ranking from what users searched for.

Response rate to xmerge searches:

  • Successful Response Rate (%)
  • No content available (%)
  • Failed response rate (%)

USER POLITENESS RATING:
100%
27 Oct – 3 Nov

Thank you from Kodee for being such polite users! No users swore or used profanity in chatting with Kodee, even though its a common exception handler and an area where you can have some fun with your bot…

CONVERSATIONAL USAGE:
8%
27 Oct – 3 Nov

It can be common for users to ask chatbots a variety of non specific, conversational queries such as: greetings, how are you’s, right through to tell me a joke. As you can see, Kodees users are all down to business with very few veering from the xmerge search to ask Kodee how he’s going.

EXISTENTIAL QUERIES:
13
27 Oct – 3 Nov

Some users sought Kodees sage advice on the more philosophical or existential questions such as: “Why am I here?”, “Can you solve world hunger?” and “just the meaning of life please Kodee?”

Locations

The majority of users were Australian based but Kodee managed to pique interest across most of the locales that XPLAN operates in.  Special thanks to those on social who shared, liked and commented on Kodee.

  • Australia
  • United Kingdon
  • New Zealand
  • Africa
  • Europe
  • Asia

About the author

Matthew Townsend on EmailMatthew Townsend on Linkedin
Matthew Townsend
Advice Technology Consultant | Digital Advice at Create Something
Matthew is an experienced and innovative Xplan consultant and developer, having worked on and developed some of the largest advice projects in the industry. Passionate about building great experiences in xplan that enable businesses and clients to get the most out of this powerful software.
Back To Top