From NLP To Chatbots

That’s quite common today, when calling the bank or other companies, to hear a robot on the other end of the line, greeting you as follows: “Hello, I am your digital assistant. Please ask your question.” Yeah, robots can now not only speak human language, but also interact with users in human language. This is due to Natural language processing (NLP) — the technology that lies at the heart of any digital assistant, allowing it to understand and generate natural language programmatically.

The article covers an example of how you might extract meaning from user input, using spaCy, the leading open source Python library for NLP.

How to Extract Meaning from User Input

Extracting meaning from user input programmatically can be quite challenging, but not impossible though. It’s fairly obvious that you cannot rely on the meaning of individual words in a sentence — the same word may express different meanings, depending on its syntactic function in a particular sentence. This can be best understood by example. Look at the following two utterances:

I’d like to order a cake.
I want to cancel my order.

In both utterances, you can see word “order”. In each case, however, it has a different syntactic function and carries a different meaning. In the first case, “order” is an action (transitive) verb that acts upon noun “cake” — the direct object of the sentence. In contrast, “order” in the second utterance is a noun that receives the action of the sentence — that is, it acts as the direct object of the sentence, where “cancel” is the transitive verb.

Linguistic characteristics of the words in a sentence — like transitive verb or direct object in the previous example — are also known as linguistic features. spaCy automatically assigns linguistic features to each token in a sentence to which the spaCy’s text-processing pipeline is being applied. Then, analyzing linguistic features can help in recognizing the meaning of words in this particular sentence. We’ll discuss how to use linguistic features for the meaning extraction task in the Using Linguistic Features in NLP section later in this article.

Preparing Your Working Environment

To follow along with the code provided in this article, you’ll need to install the following software components on your machine:

Python 2.7+∕3.4+

spaCy v2.0+

A pre-trained English model for spaCy

You’ll find installation instructions on the respective sites. The quickest way to make sure your environment is ready, you can enter the following code lines into a Python session:

import spacy
nlp = spacy.load('en')

If everything works fine, you should have no error messages.

Using Linguistic Features in NLP

Linguistic features, such as part-of-speech tags and syntactic dependency labels are specifically designed to enable development of applications capable of processing raw text intelligently. The following script illustrates how you can use spaCy to extract linguistic features for each word in a sentence:

import spacy
nlp = spacy.load('en')
doc = nlp(u'I have to send them a notification.')
for token in doc:
  print(token.text, token.pos_, token.tag_, token.dep_)

In the above script, you extract and then output the coarse-grained part-of-speech tags (pos_), fine-grained part-of-speech tags (tag_), and syntactic dependency labels (dep_) for each token in the submitted sentence. So, the script should give you the following output (tabulated for readability):

I            PRON   PRP  nsubj
have         VERB   VBP  ROOT
to           PART   TO   aux
send         VERB   VB   xcomp
them         PRON   PRP  dative
a            DET    DT   det
notification NOUN   NN   dobj
.            PUNCT  .    Punct

If you’re new to spaCy, the fine-grained part-of-speech tags and syntactic dependency labels outputted above in the third and fourth columns respectively, may look a bit confusing. To learn what the values in these columns mean, you can check with the spaCy’s documentation at https://spacy.io/api/annotation/ or use the spacy.explain() function, which returns a description for a given linguistic feature. In the following loop, you output the description of the fine-grained part-of-speech tag for each token in our sample sentence:

for token in doc:
  print(token.text, spacy.explain(token.tag_))

This should give you the following output:

I            pronoun, personal
have         verb, non-3rd person singular present
to           infinitival to
send         verb, base form
them         pronoun, personal
a            determiner
notification noun, singular or mass
.            punctuation mark, sentence closer

Similarly, you can use the spacy.explain() function to obtain descriptions for coarse-grained part-of-speech tags and syntactic dependency labels.

Extracting Intent from an Utterance

Let’s now look at an example of how you can take advantage of linguistic features to extract meaning from user input. Suppose you need to extract the intent from a submitted utterance. For example, a user of a food-ordering chatbot submits the following utterance:

I want to order a photo cake.

Obviously, the words “order” and “cake” best describe the intent expressed in this utterance. These words represent the transitive verb and the direct object respectively, in this particular case. Actually, the transitive verb/direct object pair in most cases is the most descriptive when it comes to determining the intent expressed in a request utterance. Diagrammatically, this might look as follows:

In many request utterances, the transitive verb and its direct object best describe the intent of a phrase.

The operation depicted in the above figure can be easily performed in a Python script that employs spaCy, as follows:

import spacy
nlp = spacy.load('en')
doc = nlp(u'I want to order a photo cake.')
for token in doc:
  if token.dep_ == 'dobj':
    print(token.head.text + token.text.capitalize())

In this script, you apply the text-processing pipeline to the sample sentence and then iterate over the tokens, looking for the one whose dependency label is dobj. When it’s found, you determine the corresponding transitive verb by obtaining the direct object’s syntactic head. Finally, you concatenate the transitive verb and its direct object to express the intent in the form of a single word (this is often a requirement of a processing script).

As a result, the script should generate:

orderCake

In a real application, your users might use a wide set of phrases for each intent. This means that a real application must recognize synonymous phrases in user input. For these details, you might check out my new book Natural Language Processing Using Python that includes a lot of examples on using spaCy for different NLP tasks.

Also, a real example of where the technique of intent extraction might be used in practice can be found in the Generating Intents and Entities for an Oracle Digital Assistant Skill article I recently wrote for Oracle Magazine.