The DataHour Synopsis: Build Your First Chatbot: Open Source Tool

Overview

Analytics Vidhya has long been at the forefront of imparting data science knowledge to its community. With the intent to make learning data science more engaging to the community, we began with our new initiative “DataHour.”

DataHour is a series of webinars by top industry experts where they teach and democratize data science knowledge. On 11 May 2022, we were joined by Dr. Rachael Tatman from the US  for a DataHour session on “Building your first chatbot: The Rasa (Open Source tool) approach”.

Rachael has PhD in linguistics from the University of Washington. She began texting voice assistants and speech recognition systems like Amazon and Alexa from a research perspective in 2016. Currently, she is a staff developer advocate at Rasa, where she helps professionals use open source systems to build great AI conversational projects. Previously, she was a developer advocate for Kaggle, a part of Goggle. She worked to grow the Kaggle platform and help the Kagglers develop professional data science stuff. 

Are you excited to dive deeper into the world of Chatbot? We got you covered. Let’s get started with the major highlights of this session: Building your first chatbot: The Rasa (Open Source tool) approach.

Introduction

We’ll learn about chatbots, why chatbots are, and how to build them using open source through this session. Here Rasa approach is an open-source tool that we’ll use in building the chatbots. This is the new approach to dealing with the earlier manual assistance. The ML algo’s have flooded the markets with far better and improved approaches.

This session focuses on:

• Concepts of Rasa and how it works?
• How can chatbots work?
• Live coding for a chatbot

What do you Understand about a Chatbot?

A chatbot or chatterbot is a software application used to conduct an online chat conversation via text or text-to-speech in place of providing direct contact with a live human agent. A chatbot is software that can help customers by automating conversations and interacting with them through messaging platforms.  Designed to simulate how a human behaves as a conversational partner convincingly, chatbot systems typically require continuous tuning and testing, and many in production remain unable to converse adequately. At the same time, none of them can pass the standard Turing test.

Why Chatbots?

Chatbots are used in dialogue systems, including customer service, request routing, or information gathering. While some chatbot applications use extensive word-classification processes, natural-language processors, and sophisticated AI, others scan for general keywords and generate responses using common phrases obtained from an associated library or database.

Most chatbots are accessed online via website popups or through virtual assistants. They can be classified into usage categories: commerce (e-commerce via chat), education, entertainment, finance, health, news, and productivity.

How to Build Chatbots using RASA(Open Source Tool)?

Earlier Approach:

To build chatbots, you must have catalogues or listed things you want your system to do. Earlier, humans looked at data and wrote some rules for that. For example, ELIZA- is an early chatbot system. This system would have taken the user’s things to the chatbot. Then it would purse it using a purse string. Then reform that statement into question and answer. And then, the same process repeats till the results are met. This approach was old enough and was manual.

Today’s approach:

Now machine learning has changed the dynamics of chatbots and improvised them into more advanced and faster resolution systems. Here we’ll take ML algorithms and train them on language or conversational data and move that menu step for more flexibility.

Rasa-Open Source Tool

Rasa is an open-source machine learning framework for automated text and voice-based conversations. Understand messages, hold conversations, and connect to messaging channels and APIs. You build your first chatbot using the Open Source tool for the example mentioned below.

For example:

Source: Rachael’s Presentation

There are two problems statement in the example mentioned above. We are here to build a chatbot that will serve both purposes. So, the question here is whether it is possible to build a system like this or not?

Well, the answer is Yes. But do we’ll be having only two problem statements always, particularly not, so for this we need to do:

• Build a state machine that maps out all conversational flows your assistant can handle.
• It should be slow and costly to update.

Source: Rachael’s presentation

Rasa’s approach gives more flexibility and a different approach to chatbots. The system built on this approach has the capability/benefit of tracking the user on the right question if the user had diverted from its main query.

The general recommendations to start with are:

• Undoubtedly, first, we’ll start with a few example conversations & finally move on to their learnings. And then, we’ll put the findings in the system accordingly.
• Then we’ll put our assistant in front of test users as soon as possible. So that they can begin interacting with it and then have a look at what conversations they have.
• We’ll keep using CI/CD to constantly retrain and improve our assistants. Besides, build a chatbot that is up-to-date with the required updations.
• More focus should be on the dialogue box, furthermore, how to label entities, etc.

Conversation-Driven Development

Conversation-Driven Development (CDD) is listening to your users and using those insights to improve your AI assistant. It is the overarching best practice approach for chatbot development. The principle behind CDD is that users are telling you—in their own words—exactly what they want in every conversation. By practising CDD at every stage of bot development, you orient your assistant towards actual user language and behaviour.

It includes these basics steps:

● Build a working prototype
● Share with testers
● Review conversations
● Annotate real data
● Test with end-to-end conversations
● Track desired outcomes
● Fix any problems that arise

Now we’ll look into:

What’s in a Conversational AI?

Note: Difference between chatbot and conversational AI. While the chatbot is primarily suitable for text-based conversations, Conversational AI can be operated via text and speech.

Conversational AI combines natural language processing (NLP) with traditional software like chatbots, voice assistants, or an interactive voice recognition system to help customers through a spoken or typed interface. For example, Amazon Alexa, Apple’s Siri, etc.

Steps Involved in Making a Chatbot Conversational AI

Natural language understanding: Firstly, take text input and convert it to a computer-readable format. It’s more like taking a text input and vectorising it somehow. We don’t need to vectorization it in English only, and we can do this in any language. You can use pre-trained embeddings to do so or can train your own on your training data.

Dialogue policy: How to decide what the chatbots will have to say next

• Memorization policy: If we’ve seen this sequence of turns before, repeat it exactly.
• TED policy: A flexible transformer-based policy guesses the best response based on the previous conversation.
• Fallback policy: What the assistant should do when unsure about the best response.

Assistant responses: It deals with- what the assistant says. You can produce it using Natural Language Generation (NLG) but more commonly by providing verbatim.

Time for Real-time Coding

Whatever real-time coding you need to perform, all that is already there in the video. You can go through the same and perform what-so-ever operations you need to perform to build your chatbot.

Note: Try the code on your own because mugging up the data doesn’t make sense. To get clarity on the approach, try it and build a model of your own.

Conclusion

I hope you have understood the concept “Build your first chatbot using the Rasa (Open Source tool) approach” and will apply the same in your chatbots. Moreover, a few links would help you further in the process.