Building a Serverless Chatbot w/ AWS, Zappa, Telegram, and api.ai

If you’ve ever had to set up and maintain a web server before, you know the hassle of keeping it up-to-date, installing security patches, renewing SSL certificates, dealing with downtime, rebooting when things go wrong, rotating logs and all of the other ‘ops’ that come along with managing your own infrastructure. Even if you haven’t had to manage a web server before, you probably want to avoid all of these things.

For those who want to focus on building and running code, serverless computing provides fully-managed infrastructure that takes care of all of the nitty-gritty operations automatically.

In this tutorial, we’ll show you how to build a chatbot which performs currency conversions. We’ll make the chatbot available to the world via AWS Lambda, meaning you can write the code, hit deploy, and never worry about maintenance again. Our bot’s brain will be powered by api.ai, a natural language understanding platform owned by Google.

Overview

In this post we’ll walk you through building a Telegram Bot. We’ll write the bot in Python, wrap it with Flask and use Zappa to host it on AWS Lambda. We’ll add works-out-the-box AI to our bot by using api.ai.

By the end of this post, you’ll have a fully-functioning Chatbot that will respond to Natural Language queries. You’ll be able to invite anyone in the world to chat with your bot and easily edit your bot’s “brain” to suit your needs.

Before We Begin

To follow along with this tutorial, you’ll have to have a valid phone number and credit card (we’ll be staying within the free usage limits of all services we use, so you won’t be charged). Specifically, you’ll need:

• …to sign up with Amazon Web Services. The signup process can be a bit long, and requires a valid credit card. AWS offers a million free Lambda requests per month, and our usage will stay within this free limit.
• …to sign up with api.ai. Another lengthy sign-up process, as it requires integration with the Google Cloud Platform. You’ll be guided through this process when you sign up with api.ai. Usage is currently free.
• …to sign up with Telegram, a chat platform similar to the more popular WhatsApp. You’ll need to download one of their apps (for Android, iPhone, Windows Phone, Windows, MacOS, or Linux) in order to register, but once you have an account you can also use it from web.telegram.org. You’ll also need a valid phone number. Telegram is completely free.
• …basic knowledge of Python and a working Python environment (that is, you should be able to run Python code and install new Python packages). Preferably, you should have used Python virtual environments before, but you should be able to keep up even if you haven’t. All our code examples use Python 3, but most things should be Python 2 compatible.

If you’re aiming to learn how to use the various services covered in this tutorial, we suggest you follow along step by step, creating each component as it’s needed. If you’re impatient and want to get a functioning chatbot set up as fast as possible, you can clone the GitHub repository with all the code presented here and use that as a starting point.

Building an Echo Bot

When learning a new programming language, the first program you write is one which outputs the string “Hello, World!” When learning to build chatbots, the first bot you build is one that repeats everything you say to it.

Achieving this proves that your bot is able to accept and respond to user input. After that, it’s simple enough to add the logic to make your bot do something more interesting.

Getting a Token for Our New Bot

The first thing you need is a bot token from Telegram. You can get this by talking to the @BotFather bot through the Telegram platform.

In your Telegram app, open a chat with the official @BotFather Chatbot, and send the command /newbot. Answer the questions about what you’ll use for your new bot’s name and username, and you’ll be given a unique token similar to 14438024:AAGI6Kh8ew4wUf9-vbqtb3S4sIM7nDlcXj3. We’ll use this token to prove ownership of our new bot, which allows us to send and receive messages through the Bot.

We can now control our new bot via Telegram’s HTTP API. We’ll be using Python to make calls to this API.

Writing the First Code for Our New Bot

Create a new directory called currencybot to house the code we need for our bot’s logic, and create three Python files in this directory named config.py, currencybot.py, and bot_server.py The structure of your project should be as follows:

currency/
bot_server.py
config.py
currencybot.py

in config.py we need a single line of code defining the bot token, as follows (substitute with the token you received from BotFather).

bot_token = "14438024:AAGI6Kh8ew4wUf9-vbqtb3S4sIM7nDlcXj3"

In currencybot.py we need to put the logic for our bot, which revolves around receiving a message, handling the message, and sending a message. That is, our bot receives a message from some user, works out how to respond to this message, and then sends the response. For now, because we are building an echo bot, the handling logic will simply return any input passed to it back again.

Add the following code to currencybot.py:

import requests
import config

# The main URL for the Telegram API with our bot's token
BASE_URL = "https://api.telegram.org/bot{}".format(config.bot_token)

def receive_message(message):
    """Receive a raw message from Telegram"""
    try:
        message = str(msg["message"]["text"])
        chat_id = msg["message"]["chat"]["id"]
        return message, chat_id
    except Exception as e:
        print(e)
        return (None, None)
 
def handle_message(message):
    """Calculate a response to the message"""
    return message
 
def send_message(message, chat_id):
    """Send a message to the Telegram chat defined by chat_id"""
    data = {"text": message.encode("utf8"), "chat_id": chat_id}
    url = BASE_URL + "/sendMessage"
    try:
        response = requests.post(url, data).content
    except Exception as e:
        print(e)
        
def run(message):
    """Receive a message, handle it, and send a response"""
    try:
        message, chat_id = receive_message(message)
        response = handle_message(message)
        send_message(response, chat_id)
    except Exception as e:
        print(e)

Finally, bot_server.py is a thin wrapper for our bot that will allow it to receive messages via HTTP. Here we’ll run a basic Flask application. When our bot receives new messages, Telegram will send these via HTTP to our Flask app, which will pass them on to the code we wrote above. In bot_server.py, add the following code:

from flask import Flask
from flask import request
from currencybot import run

app = Flask(__name__)

@app.route("/", methods=["GET", "POST"])
def receive():
    try:
        run(request.json)
        return ""
    except Exception as e:
        print(e)
        return ""

This is a minimal Flask app that imports the main run() function from our currencybot script. It uses Flask’s request module (distinct from the requests library we used earlier, though the names are similar enough to be confusing) to grab the POST data from an HTTP request and convert this to JSON. We pass the JSON along to our bot, which can extract the text of the message and respond to it.

Deploying Our Echo Bot

We’re now ready to deploy our bot onto AWS Lambda so that it can receive messages from the outside world.

We’ll be using the Python library Zappa to deploy our bot, and Zappa will interact directly with our Amazon Web Services account. In order to do this, you’ll need to set up command line access for your AWS account as described here: https://aws.amazon.com/blogs/security/a-new-and-standardized-way-to-manage-credentials-in-the-aws-sdks/.

To use Zappa, it needs to be installed inside a Python virtual environment. Depending on your operating system and Python environment, there are different ways of creating and activating a virtual environment. You can read more about how to set one up here. If you’re using MacOS or Linux and have used Python before, you should be able to create one by running the following command.

virtualenv ~/currencybotenv

You should see output similar to the following:

~/git/currencybot g$ virtualenv ~/currencybotenv

Using base prefix '/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6'

New python executable in /Users/g/currencybotenv/bin/python3.6

Also creating executable in /Users/g/currencybotenv/bin/python

Installing setuptools, pip, wheel...done.

The result is that a clean Python environment has been created, which is important so Zappa will know exactly what dependencies to install on AWS Lambda. We’ll install the few dependencies we need for our bot (including Zappa) inside this environment.

Activate the environment by running:

source ~/currencybotenv/bin/activate

You should see your Terminal’s prompt change to indicate that you’re now working inside that environment. Mine looks like this:

(currencybotenv) ~/git/currencybot g$

Now we need to install the dependencies for our bot using pip. Run:

pip install zappa requests flask

At this point, we need to initialize our Zappa project. We can do this by running:

zappa init

This will begin an interactive process of setting up options with Zappa. You can accept all of the defaults by pressing Enter at each prompt. Zappa should figure out that your Flask application is inside bot_server.py and prompt to use bot_server.app as your app’s function.

You’ve now initialized the project and Zappa has created a zappa_settings.json file in your project directory. Next, deploy your bot to Lambda by running the following command (assuming you kept the default environment name of ‘dev’):

zappa deploy dev

This will package up your bot and all of its dependencies, and put them in an AWS S3 bucket, from which it can be run via AWS Lambda. If everything went well, Zappa will print out the URL where your bot is hosted. It should look something like https://l19rl52bvj.execute-api.eu-west-1.amazonaws.com/dev. Copy this URL because you’ll need to instruct Telegram to post any messages sent to our bot to this endpoint.

In your web browser, change the setting of your Telegram bot by using the Telegram API and your bot’s token. To set the URL to which Telegram should send messages to your bot, build a URL that looks like the following, but with your bot’s token and your AWS Lambda URL instead of the placeholders.

https://api.telegram.org/bot<your-bot-token>/setWebhook?url=<your-zappa-url>

For example, your URL should look something like this:

http://api.telegram.org/bot14438024:AAGI6Kh8ew4wUf9-vbqtb3S4sIM7nDlcXj3/setWebhook?url=https://l19rl52bvj.execute-api.eu-west-1.amazonaws.com/dev

Note that the string bot must appear directly before the token.

Testing Our Echo Bot

Visit your bot in the Telegram client by navigating to t.me/<your-bot’s-username>. You can find a link to your bot in the last message sent by BotFather when you created the bot. Open up a Chat with your bot in the Telegram client and press the /start button.

Now you can send your bot messages and you should receive the same message as a reply.

If you don’t, it’s likely that there’s a bug in your code. You can run zappa tail dev in your Terminal to view the output of your bot’s code, including any error messages.

Teaching Our Bot About Currencies

You’ll probably get bored of chatting to your echo bot pretty quickly. To make it more useful, we’ll teach it how to send us currency conversions.

Add the following two functions to the currencybot.py file. These functions allow us to use the Fixer API to get today’s exchange rates and do some basic calculations.

def get_rate(frm, to):
    """Get the raw conversion rate between two currencies"""
    url = "http://api.fixer.io/latest?base={}&symbols={}".format(frm, to)
    try:
        response = requests.get(url)
        js = response.json()
        rates = js['rates']
        return rates.popitem()[1]
    except Exception as e:
        print(e)
        return 0

def get_conversion(quantity=1, frm="USD", to="GBP"):
    rate = get_rate(frm.upper(), to.upper())
    to_amount = quantity * rate
    return "{} {} = {} {}".format(quantity, frm, to_amount, to)

We’ll now expect the user to send currency conversion queries for our bot to compute. For example, if a user sends “5 USD GBP” we should respond with a calculation of how many British Pounds are equivalent to 5 US Dollars. We need to change our handle_message() function to split the message into appropriate parts and pass them to our get_conversion() function. Update handle_message() in currencybot.py to look like this:

def handle_message(message):
    """Calculate a response to a message"""
    try:
        qty, frm, to = message.split(" ")[:3]
        qty = int(qty)
        response = get_conversion(qty, frm, to)
    except Exception as e:
        print(e)
        response = "I couldn't parse that"
    return response

This function now parses messages that match the required format into the three parts. If the message doesn’t match what we were expecting, we inform the user that we couldn’t deal with their input.

Save the code and update the bot by running the following command (make sure you are still within your Python virtual environment, in your project directory).

zappa update dev

Testing Our Currency Converter Bot

After the update has completed, you’ll be able to chat with your bot and get currency conversions. You can see an example of the bot converting US Dollars to South African Rands and US Dollars to British Pounds below:

Adding AI to Our Bot

Our bot is more useful now, but it’s not exactly smart. Users have to remember the correct input format and any slight deviations will result in the “I couldn’t parse that” error. We want our bot to be able to respond to natural language queries, such as “How much is 5 dollars in pounds?” or “Convert 3 USD to pounds”. There are an infinite number of ways that users might ask these questions, and extracting the three pieces of information (the quantity, from-currency, and to-currency) is a non-trivial task.

This is where Artificial Intelligence and Machine Learning can help us out. Instead of writing rules to account for each variation of the same question, machine learning lets us learn patterns from existing examples. Using machine learning, we can teach a program to extract the pieces of information that we want by ‘teaching’ it with a number of existing examples. Luckily, someone else has already done this for us, so we don’t need to start from scratch.

Create an account with api.ai, and go through their setup process. Once you get to the main screen, select the “Prebuilt Agents” tab, as shown below

Select the “Currency Converter” agent from the list of options, and choose a Google Cloud Project (or create a new one) to host this agent. Now you can test your agent by typing in a query in the top right-hand corner of the page, as indicated below:

Hit the “Copy Curl” link, which will copy a URL with the parameters you need to programmatically make the same request you just made manually through the web page. It should have copied a string that looks similar to the following into your clipboard.

curl 'https://api.api.ai/api/query?v=20150910&query=convert%201%20usd%20to%20zar&lang=en&sessionId=fed2f39e-6c38-4d42-aa97-0a2076de5c6b&timezone=2017-07-15T18:12:03+0200' -H 'Authorization:Bearer a5f2cc620de338048334f68aaa1219ff'

The important part is the Authorization argument, which we’ll need to make the same request from our Python code. Copy the whole token, including Bearer into your config.py file, which should now look similar to the following:

bot_token = "14438024:AAGI6Kh8ew4wUf9-vbqtb3S4sIM7nDlcXj3"

apiai_bearer = "Bearer a5f2cc620de338048334f68aaa1219ff"

Add the following line to the top of your currencybot.py file:

from datetime import datetime

And add a parse_conversion_query() function below in the same file, as follows:

def parse_conversion_query(query):
    url_template = "https://api.api.ai/api/query?v=20150910&query={}&lang=en&sessionId={}"
    url = url_template.format(query, datetime.now())
    headers = {"Authorization":  config.apiai_bearer}
    response = requests.get(url, headers=headers)
    js = response.json()
    currency_to = js['result']['parameters']['currency-to']
    currency_from = js['result']['parameters']['currency-from']
    amount = js['result']['parameters']['amount']
    return amount, currency_from, currency_to

This reconstructs the cURL command that we copied from the api.ai site for Python. Note that the v=20150910 in the url_template is fixed and should not be updated for the current date. This selects the current version of the api.ai API. We omit the optional timezone argument but use datetime.now() as a unique sessionId.

Now we can pass a natural language query to the api.ai API (if you think that’s difficult to say, just look at the url_template which contains api.api.ai/api/!) It will work out what the user wants in terms of quantity, from-currency and to-currency, and return structured JSON for our bot to parse. Remember that api.ai doesn’t do the actual conversion–its only role is to extract the components we need from a natural language query, so we’ll pass these pieces to the fixer.io API as before. Update the handle_message() function to use our new NLU parser. It should look as follows:

def handle_message(message):
    """Calculate a response to a message"""
    try:
        qty, frm, to = parse_conversion_query(message)
        qty = int(qty)
        response = get_conversion(qty, frm, to)
    except Exception as e:
        print(e)
        response = "I couldn't parse that"
    return response

Make sure you’ve saved all your files, and update your deployment again with:

zappa update dev

Testing Our Bot’s AI

Now our bot should be able to convert between currencies based on Natural Language queries such as “How much is 3 usd in Indian Rupees”.

If this doesn’t work, run zappa tail dev again to look at the error log and figure out what went wrong.

Our bot is by no means perfect, and you should easily be able to find queries that break it and cause unexpected responses, but it can handle a lot more than the strict input format we started with! If you want to teach it to handle queries in specific formats, you can use the api.ai web page to improve your bot’s understanding and pattern recognition.

Conclusion

Serverless computing and Chatbots are both growing in popularity, and in this tutorial you learned how to use both of them.

We showed you how to set up a Telegram Chatbot, make it accessible to the world, and plug in a prebuilt brain.

You can now easily do the same using the other pre-built agents offered by api.ai, or start building your own. You can also look at the other Bot APIs offered by Facebook Messenger, Skype, and many similar platforms to make your Bots accessible to a wider audience.