Distributions plays a very crucial role in data science and analytics. Statistical distributions shows us the underlying relationship between the data points. The main purpose of checking distributions is to get an idea on how the data points of the target are spread across, so that appropriate model could be used to fit. For non data-science folks, by target we are referring the variable upon which we are interested to do some prediction or estimation.

The examples below will justify the need for distribution:

The majority of popular statistical models including the most common kinds, say linear regression make an assumption of normally distributed error terms. In another example of insurance industry, if we want to see the distribution of number of claims that come for each policy then ideally it will look like Poisson distribution, as we are concerned about the count of claim, which is discrete in nature.

Blog-post Content

Although, the list of distributions are huge, but that scope of this blog-post is to cover the ones which we use more commonly in the industry. The list shortens to 5 major distributions and one bonus distribution which is very common to insurance industry. They are:

1. Normal Distribution
2. Binomial Distribution
3. Uniform Continuous Distribution (Rectangular Distribution)
4. Poisson Distribution
5. Gamma Distribution
6. Tweedie Distribution

Normal, rectangular and gamma are used for continuous variables, poisson and binomial are for discrete variables and tweedie distribution is one such distribution which models both poisson and gamma together.

For each of the distribution, we will be looking at:

1. Brief explanation with mathematical description
2. Generating each distribution in Python.
3. Plotting with Seaborn library

1. Normal Distribution

Normal distribution is one of the most common distribution and is used for continuous variables. It is bell shaped curve which is neither too skewed not too flat and has parameters:  (mean) and  (Standard Deviation)

The probability density function for normal distribution is :

Properties of Normal Distrbution

1. Symmetric around the mean i.e. all the basic summary stats: mean, median and mode lies at the center and are equal to each other
2. Standard deviation represents how closely packed or loosely floating are the data points to or away from the mean
3. Since the distribution is symmetric, so area under the curve is equal to 1
4. When  and  then the distribution is called standard normal distrbution

We will be using np.random.normal() from numpy package to generate random sample of normally distributed variates. The np.random.normal() has three parameters : loc (mean), scale (SD) and size (no.of obs to be generated).

The below code will show how to generate random sample for normal distribution and the shape of the distribution.

 #Import Libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib.ticker import FuncFormatter
import seaborn as sns
import warnings
warnings.filterwarnings("ignore")

# Normal Distribution
#Intializing the parameters: mean and SD
mu1,sigma1 = 2,2
np.random.seed(seed=32)
s_normal= np.random.normal(mu1, sigma1, 200000)
plt.subplots(figsize=(10,5))
plot=sns.distplot(pd.DataFrame(s_normal)).set_title('Normal Distribution with $\mu$=%.1f,$\sigma$=%.1f'%(mu1,sigma1))
fig=plot.get_figure()

We know that . In order to find out the mean and variance in python we will use the command

print(np.mean(s_normal), np.var(s_normal))

we are getting mean as 2.007 and variance as 4.005. The values obtained here are matching with the mathematical formula given above.

# Standard Normal Distribution
#Initializing the parameters:mean and SD
mu,sigma=0,1
np.random.seed(seed=32)
s_stdnorm= np.random.normal(mu, sigma, 1000)
plt.subplots(figsize=(10,5))
plot=sns.distplot(pd.DataFrame(s_stdnorm)).set_title('Standard Normal Distribution with $\mu$=%.1f and $\sigma^2$=%.1f' %(mu,sigma))
fig=plot.get_figure()

2. Uniform Continuous Distribution

In case of uniform continuous distribution, which is also know as rectangular distribution, the probability associated with each data points within a fixed interval is equal.

The probability density function for uniform distribution is:

For example: In winter, a person can gain weight post Christmas anywhere between 1lb to 50lb. Each of the numbers between 1 to 50 are equally probable. So in this example,  and . If we are interested to find out the probabilty of gaining weight anywhere between 25lb to 35lb then the above formula will turn to:

; where  and  represents 25 and 35 respectively.

Properties of Uniform Distribution

1. Mean and median of the distribution is given by 
2. Since the datapoints within the interval [a,b] are equally likely, so there is no skewness present in the distribution
3. Variance of the distribution is given by 

We will use the command np.random.uniform() to generate a random sample which follows rectangular distribution.

#Here parameters a and b has been assigned the values 1 and 6
s_uniform=np.random.uniform(1, 6, 500000)
np.random.seed(seed=32)
plt.subplots(figsize=(10,5))
plot=sns.distplot(pd.DataFrame(s_uniform)).set_title('Uniform Distribution with $a$=%.1f and $b$=%.1f'%(1,6))
fig=plot.get_figure()

The mean and variance of uniform distribution is calculated as

print(np.mean(s_uniform),np.var(s_uniform))

and the values we obtain are 3.49 (mean) and 2.08(variance). The values obtained here are matching with the mathematical formula given above.

3. Binomial Distribution

Binomial Distribution is a discrete probability distribution, which gives the sum of the outcomes obtained from n Bernoulli trials. For example, if a drug’s effect of curing a cancer is being tested on a patient, the result might be a success or a failure. The same experiment when conducted on 1000 cancer patients where n (trials) is large and we are interested to see exactly 5 successful cases, then the distribution becomes a binomial distribution. Here p measures drug’s success of curing cancer.

The probability density function of binomial distribution is:

The above equation is reduced to bernoulli distribution when n=k=1 and the equation becomes:

Properties of Binomial Distribution

1. The probability of success is independent in each trial which implies the samples drawn randomly have been done with replacement
2. Mean() of the distribution is given by 
3. Variance() of the distribution is given by . The lower the variance of the distribution, the highly skewed will be the distribution. Skewness is given by 

Relation between mean and variance:

1. When p=0 then =0 and =0 which implies  = 
2. When p=1 then =n and =0 which implies  > 
3. When 0 < 

Since the third condition can be re-modeled as =, so whatever be the value of ,  is  times  and will always be less than .

We will generate random sample which are binomially distributed using the command np.random.binomial().

#Initializing the parameters 'number of trials' and 'probaility of success'
n,p = 10,0.5
np.random.seed(seed=32)
s_binomial=np.random.binomial(n,p,1000)
plt.subplots(figsize=(10,5))
plot=sns.distplot(pd.DataFrame(s_binomial)).set_title('Binomial Distribution with $n$=%.1f,$p$=%.1f' % (n,p))
fig=plot.get_figure()

To get the mean and variance of a distribution, we will use the below code:

print(np.mean(s_binomial), np.var(s_binomial))

We find that mean is 5.036 and variance is 2.468 which proves that mean>variance in binomial. The values obtained here are matching with the mathematical formula given above.

4. Poisson Distribution

Poisson distribution is a discrete probability distribution which models number of times an event occurs in fixed time interval. It is a binomial approximated distribution which occurs when number of trials () becomes sufficiently large and probability of success () successively becomes small such that  becomes  (constant), the distribution becomes poisson.

The probability density function of poisson distribution is:

As an example: the number of file server virus infection at a data center during a 24-hour period is  poisson distributed.

Properties of Poisson Distribution

1. Probability of occurrence of one event is independent of the probability of occurrence of the other
2. Mean () and variance () both are equal to 
3. Skewness is represented by . The higher the constant term(), higher would be the skewness

The command used to generate poisson distribution in python was np.random.poisson()

#Intializing the parameters for Poisson distribution
mean,k=2,100000
np.random.seed(seed=32)
s_poisson = np.random.poisson(mean,k)
plt.subplots(figsize=(10,5))
plot=sns.distplot(pd.DataFrame(s_poisson)).set_title('Poisson Distribution with $\lambda$=%.1f and $k$=%.1f'%(mean,k))
fig=plot.get_figure()

We can check the mean and variance of poisson distribution:

print(np.mean(s_poisson),np.var(s_poisson))

We will see that values 2.00 (mean) and 2.00 (variance) are  very close value of  which proves the proposition that mean and variance are equal in poisson distribution. The values obtained here are matching with the mathematical formula given above.

5. Gamma Distribution

Gamma distribution is a right skewed distribution used for continuous variables. This is due to its flexibility in the choice of the shape and scale parameters. The scale parameter determines where the bulk of the observations lies and the shape parameter determines how the distribution will look.

The probability density function of Gamma distribution:
Here  is the shape parameter,  is the scale parameter and  is the gamma function.

For example, to estimate rainfall for a location at select time interval, based on past data of the same, gamma distribution is a popular choice because of the flexibility in parameters: shape and scale.

Properties of Gamma Distribution

1. Mean of the distribution is given by 
2. Variance of the distribution is given by 
3. Skewness is given by . The lesser the shape parameter, the more skewed the distribution will be

We will use the command np.random.gamma() to generate random samples which follows gamma distribution.

#Initializing the parameters for Gamma distribution
shape, scale = 1, 2.
plt.subplots(figsize=(10,5))
np.random.seed(seed=32)
s_gamma = np.random.gamma(shape, scale, 50000)
plot=sns.distplot(pd.DataFrame(s_gamma)).set_title('Gamma Distribution with $shape$=%.1f and $scale$=%.1f'%(shape,scale))
fig=plot.get_figure()

We will compute the mean and variance by :

print(np.mean(s_gamma),np.var(s_gamma))

and the values we obtain are 1.994(mean) and 3.959(variance). The values obtained here are matching with the mathematical formula given above.

Below we will show how the change in each of the parameters  and , we can change shape of the distribution.

Case1: Change in shape parameter with no change in scale parameter

s_gamma1=np.random.gamma(5,scale,5000)
np.random.seed(seed=32)
fig,(ax1,ax2)= plt.subplots(1,2, figsize=(15, 4))

plot=sns.distplot(pd.DataFrame(s_gamma),ax=ax1).set_title('Gamma Distribution with shape parameter =1')
fig=plot.get_figure()

plot1=sns.distplot(pd.DataFrame(s_gamma1),ax=ax2).set_title('Gamma Distribution with shape parameter =5')
fig1=plot1.get_figure()

It is clear from the above graph that with change in shape parameter, skewness of the distribution reduces.

Case2: Change in scale parameter with no change in shape parameter

s_gamma2=np.random.gamma(shape,5,5000)
np.random.seed(seed=32)
fig, (ax1,ax2) =plt.subplots(1,2,figsize=(15,4))

plot=sns.distplot(pd.DataFrame(s_gamma),ax=ax1).set_title('Gamma Distribution with scale parameter=2')
fig=plot.get_figure()

plot1=sns.distplot(pd.DataFrame(s_gamma2),ax=ax2).set_title('Gamma Distribution with scale parameter=5')
fig1=plot1.get_figure()

From the above figure, we can infer that with change in scale parameter, the tails of the distribution became more elongated with no  change in skewness.

6. Tweedie Distribution

Tweedie distributions belongs to the family of linear exponential distributions with a dispersion parameter .

A variable Y is said to follow tweedie distribution if

Here  refers to the mean-variance relationship of the distribution and  is a constant, also called as variance power. The above-mentioned distribution which belongs to the tweedie family can be achieved with a change in the parameter of the constant . The tweedie distribution can be achieved when the value of p belongs to the interval (1,2).

In simple terms, the tweedie distribution can be explained as a sum of N independent gamma random variates where N follows a poisson distribution and N and gamma random variate are independent. In other words, for each poisson random variate, we are trying to find a corresponding gamma variate. For example, in case of modelling insurance claims, the number of incumbent claims can be modeled independently using poisson distribution as its a count(discrete) distribution and the severity(claim cost) associated with the claims can be modeled independently using gamma distribution as claim cost is continuous parameter, but when we look at them together then tweedie distribution views the situation as : “for a claim, it tries to find the product of poisson number of claims and gamma sized claim amount”. So if 5 claims have come, the tweedie will view these 5 claims as 5 claims with their associated claim cost.

However, the below code is focused on Poisson-Gamma distribution.

def tweedie(n,p,mu,phi):
    np.random.seed(seed=32)
    #checking the value of variance power between 1-2
    if(p=2):
        print('p must be between (1,2)')
        pass
    else:
        rt=np.full(n,np.nan)
    # calculating mean of poisson distribution
        lambdaa=mu**(2-p)/(phi*(2-p))
    # shape parameter of gamma distribution
        alpha=(2-p)/(1-p)
    # scale parameter of gamma distribution
        gam= phi*(p-1)*(mu**(p-1))
    # Generating Poisson random sample
        N=np.random.poisson(lambdaa,n)
        for i in range(n):
        # Generate single data point of gamma distribution using poisson random variable
            rt[i]=np.random.gamma(N[i]*np.abs(alpha),gam,1)
        return(rt)

The above function has 4 parameters: n, p, mu, phi. Here  is the number of points to be generated,  refers to the constant parameter.In the above function, we can’t assign values 1 or 2 to the value p.It has to be between 1 and 2, else it will throw a warning (mentioned in the code).  refers to the mean and  refers to the dispersion parameter.

Below we will see that how varying the constant parameter p between 1 and 2, the distribution moves from extreme poisson to extreme gamma distribution.

vals=[1.01,1.2,1.5,1.9]
fig, axes =plt.subplots(2,2, figsize=(15,6))
axes=axes.flatten()
for ax,i in zip(axes,vals):
    s_tweedie=tweedie(50000,i,3,2)
    sns.distplot(pd.DataFrame(s_tweedie),ax=ax).set_title('Tweedie Distribution with $p$=%.2f,$\mu$=%.1f and $\phi$=%.1f'%(i,3,2))
plt.tight_layout()
plt.show()

The values of  and  can be changed as per required.

 We show here the variance of tweedie with p=1.5, which is obtained using the command

print(np.var(s_tweedie))

The variance is 10.40 and the value obtained here is matching with the mathematical formula given above.

Sign off

There are various other distributions which can be looked for (listed here). I hope the above explanations helps the readers to have a good understanding about each of these distributions. This blog-post will help beginners in their data science journey as they will be able to identify, relate and differentiate among these distributions and apply these effectively in modeling data science problems.

If you liked the post, follow this blog to get updates about upcoming articles. Also, share it so that it can reach out to the readers who can actually gain from this. Please feel free to discuss anything regarding the post. I would love to hear feedback from you.

Happy learning distributions 

This blog-post demonstrate the finbert-embedding pypi package which extracts token and sentence level embedding from FinBERT model (BERT language model fine-tuned on financial news articles). The finbert model was trained and open sourced by Dogu Tan Araci (University of Amsterdam).

BERT, published by Google, is conceptually simple and empirically powerful as it obtained state-of-the-art results on eleven natural language processing tasks.

The objective of this project is to obtain the word or sentence embeddings from FinBERT, pre-trained model by Dogu Tan Araci. FinBERT, which is a BERT language model further trained on financial news articles for adapting financial domain. It achieved the state-of-the-art on FiQA sentiment scoring and Financial PhraseBank dataset. Paper here.

Instead of building and perform fine-tuning for an end-to-end NLP model, You can directly utilize word embeddings from Financial BERT to build NLP models for various downstream tasks eg. Text clustering or Word clustering or Extractive summarization etc.

Features

• Creates an abstraction to remove dealing with inferencing the pre-trained FinBERT model.
• Require only two lines of code to get sentence or token-level encoding for a text sentence.
• The package takes care of OOVs (out of vocabulary) inherently.
• Downloads and installs FinBERT pre-trained model (during first initialization, usage in next section).

Install

(Recommended to create a conda env to have isolation and avoid dependency clashes)

pip install finbert-embedding==0.1.4

Note: If you get error in installing this package (common error with Tensorflow):

Installing collected packages: wrapt, tensorflow
Found existing installation: wrapt 1.10.11
ERROR: Cannot uninstall ‘wrapt’. It is a distutils installed project……

then, just do this:

pip install wrapt --upgrade --ignore-installed
pip install finbert-embedding==0.1.4

Usage 1

Token/Sentence Embedding Extraction

Word embeddings generated are list of 768 dimensional embeddings for each word.
Sentence embedding generated is 768 dimensional embedding which is average of each token.

One can also set model_path parameter in FinbertEmbedding() class to extract token/sentence embeddings from any other fine-tuned or even original BERT model. FinBERT model automatically gets downloaded from the dropbox as the class initialization is done i.e. finbert = FinbertEmbedding()

from finbert_embedding.embedding import FinbertEmbedding

text = "Another PSU bank, Punjab National Bank which also reported numbers managed to see a slight improvement in asset quality."

# Class Initialization (You can set default 'model_path=None' as your finetuned BERT model path while Initialization)
finbert = FinbertEmbedding()

word_embeddings = finbert.word_vector(text)
sentence_embedding = finbert.sentence_vector(text)

print("Text Tokens: ", finbert.tokens)
# Text Tokens:  ['another', 'psu', 'bank', ',', 'punjab', 'national', 'bank', 'which', 'also', 'reported', 'numbers', 'managed', 'to', 'see', 'a', 'slight', 'improvement', 'in', 'asset', 'quality', '.']

print ('Shape of Word Embeddings: %d x %d' % (len(word_embeddings), len(word_embeddings[0])))
# Shape of Word Embeddings: 21 x 768

print("Shape of Sentence Embedding = ",len(sentence_embedding))
# Shape of Sentence Embedding =  768

Usage 2

Similarity or Text Comparison

A decent representation for a downstream task doesn’t mean that it will be meaningful in terms of cosine distance. Since cosine distance is a linear space where all dimensions are weighted equally. if you want to use cosine distance anyway, then please focus on the rank not the absolute value.

Namely, do not use:
if cosine(A, B) > 0.9, then A and B are similar

Please consider the following instead:
if cosine(A, B) > cosine(A, C), then A is more similar to B than C.

from finbert_embedding.embedding import FinbertEmbedding

text = "After stealing money from the bank vault, the bank robber was seen fishing on the Mississippi river bank."
finbert = FinbertEmbedding()
word_embeddings = finbert.word_vector(text)

from scipy.spatial.distance import cosine
diff_bank = 1 - cosine(word_embeddings[9], word_embeddings[18])
same_bank = 1 - cosine(word_embeddings[9], word_embeddings[5])

print('Vector similarity for similar bank meanings (bank vault & bank robber):  %.2f' % same_bank)
print('Vector similarity for different bank meanings (bank robber & river bank):  %.2f' % diff_bank)

# Vector similarity for similar bank meanings (bank vault & bank robber):  0.92
# Vector similarity for different bank meanings (bank robber & river bank):  0.64

Warning

According to BERT author Jacob Devlin: I'm not sure what these vectors are, since BERT does not generate meaningful sentence vectors. It seems that this is doing average pooling over the word tokens to get a sentence vector, but we never suggested that this will generate meaningful sentence representations. And even if they are decent representations when fed into a DNN trained for a downstream task, it doesn't mean that they will be meaningful in terms of cosine distance. (Since cosine distance is a linear space where all dimensions are weighted equally).

However, with the [CLS] token, it does become meaningful if the model has been fine-tuned, where the last hidden layer of this token is used as the “sentence vector” for downstream sequence classification task. This package encode sentence by average pooling over all word tokens. It would be definitely good to experiment with sentence embeddings for similarity purposes.

To Do (Next Version)

• Extend it to give word embeddings for a paragraph/Document (Currently, it takes one sentence as input). Chunkize your paragraph or text document into sentences using Spacy or NLTK before using finbert_embedding.
• Adding batch processing feature.
• More ways of handing OOVs (Currently, uses average of all tokens of a OOV word).

Future Goal

• Create generic downstream framework using FinBERT language model for any financial labelled text classification task like sentiment classification, Financial news classification, Financial Document classification etc.

One can check the package here at PyPi and codes for token/sentence embedding extraction at GitHub here.

Happy Deep Learning

Text summarization is one of famous NLP application which had been researched a lot and still at its nascent stage compared to manual summarization. In simple terms, the objective is to condense unstructured text of an article into a summary automatically. There are two types of summarization techniques.

1. Extractive Summarization
2. Abstractive Summarization

Extractive Summarization

Extractive summarization is extracting the most important sentences from whole document which would coherently represent the document. The number of sentences picked may depend on the compression ratio of summary. There could be multiple approaches depending on the summarization application. For example,

• For summarizing the reviews, we may want to pick highly positive and negative sentences as summary.
• Another approach could be picking only the sentences which contains some object/entity (Company name, person, dates, location etc).
• Also, if we want to summarize a report or an article, we may want to pick the important or say main sentences from the article. Here, the problem of summarization converges to the problem “How do we know the main/important sentences (Text Rank algorithm)”. This problem is very similar to “How does google searches for important web pages from so many web pages having similar content” (Page Rank algorithm). It is an unsupervised approach.

Abstractive Summarization

This is a hard problem. We want to produce a summary which corresponds to how a professional writer would summarize an article. Abstractive summarizers are generative models as it may come up with a summary containing new words (as opposed to picking existing sentences). Also, an abstractive summarization requires supervised learning procedure to learn how a professionally supervised summary looks like given long articles. There are lot of deep learning models which had been researched for this purpose. Interested readers can check the links here, here and here. Still, it is an active area of research.

This blog post demonstrate extractive summarization using TextRank algorithm (similar to pagerank) in an unsupervised manner. The underlying metric for calculation of ranking is similarity matrix which is similarity of each sentence with all the other sentence in article. We use glove vectors to calculating similarity scores among sentences.

1. Imports

I downloaded the pretrained glove vectors (1.9M vocab words, 300 dim) from here.

import os
import re
import string
import numpy as np
import networkx as nx
import matplotlib.pyplot as plt
from nltk.corpus import stopwords
from nltk.tokenize import sent_tokenize
from nltk.stem.wordnet import WordNetLemmatizer
from sklearn.metrics.pairwise import cosine_similarity

gloveFile = "../glove.42B.300d.txt"
dim=300

2. Custom Functions: Preprocessing

Below are the 3 python functions for:

• removing non-ascii characters,
• preprocessing and cleaning stop words, punctuation marks and digits
• creating and loading word to glove embedding dictionary.

stop = set(stopwords.words('english'))
exclude = set(string.punctuation)
lemma = WordNetLemmatizer()

def rem_ascii(s):
    return "".join([c for c in s if ord(c) < 128 ])

# Cleaning the text sentences so that punctuation marks, stop words and digits are removed.
def clean(doc):
    stop_free = " ".join([i for i in doc.lower().split() if i not in stop])
    punc_free = ''.join(ch for ch in stop_free if ch not in exclude)
    processed = re.sub(r"\d+","",punc_free)
    return processed

def loadGloveModel(gloveFile):
    word_embeddings = {}
    f = open(gloveFile, encoding='utf-8')
    for line in f:
        values = line.split()
        word = values[0]
        coefs = np.asarray(values[1:], dtype='float32')
        word_embeddings[word] = coefs
    f.close()
    return word_embeddings

word_embeddings = loadGloveModel(gloveFile)

The above steps are important as we want to only choose important words of sentence for similarity scoring. Readers can check the vocabulary size of this glove model below (1.9M).

print("Vocab Size = ",len(word_embeddings))

Vocab Size =  1917494

Next, we chose a news article which talks about blocking of customer debit cards of certain banks from year 2020 to meet international payment standards. We used NLTK sentence tokenizer to tokenize the article paragraph into sentences.

text = """SBI, PNB, HDFC, ICICI,  Debit Card Customers: If you are using a bank debit card which doesn't have EMV (Europay, Mastercard and Visa), then you may have to face a problem during money withdrawal from the ATM after 31st December 2019 as your debit card may be blocked from 1st january 2020. According to the Reserve Bank of India (RBI) guidelines, all Indian banks need to replace the magnetic debit card of their customers with a new EMV card. This debit card replacement is mandatory as it is aimed at meeting the international payment standards. Hence, SBI, PNB, HDFC Bank, ICICI Bank or any other bank customers who are using a magnetic debit card are advised to replace their debit card otherwise they will have to face difficulty in money withdrawal from the ATM. The RBI guidelines say all Indian banks will have to replace all magnetic chip-based debit cards with EMV and PIN-based cards by 31st December 2019. Keeping in view of the continuing online frauds on magnetic stripe cards, the RBI has proposed to deactivate them by 31st December 2019. So, all magnetic chip-based debit cards will be deactivated from 1st January 2020 (irrespective of the validity of the existing magnetic SBI debit cards). All banks are sending messages to their customers via various means asking them to replace their magnetic chip-based debit card by a new EMV debit card. The SBI warned its debit cardholders through a tweet citing, "Apply now to change your Magnetic Stripe Debit Cards to the more secure EMV Chip and PIN-based SBI Debit card at your home branch by 31st December 2019. Safeguard yourself with guaranteed authenticity, greater security for online payments and added security against fraud". So, the SBI has made it clear that SBI debit card will be blocked if it is a magnetic card. In fact, the SBI has already started deactivating the SBI cards of those SBI accounts in which PAN or Form 60 is not updated."""
print(text)
sentences = sent_tokenize(text)
cleaned_texts = [rem_ascii(clean(sentence)) for sentence in sentences]

3. Sentence Embeddings from Glove

Idea is very simple. We simply calculate the average of embeddings of all the words present the sentence. We have already removed stop words, digits etc which do not contribute to similarity scoring.

sentence_vectors = []
for i in cleaned_texts:
    if len(i) != 0:
        v = sum([word_embeddings.get(w, np.zeros((dim,))) for w in i.split()])/(len(i.split())+0.001)
    else:
        v = np.zeros((dim,))
    sentence_vectors.append(v)

4. Similarity Matrix

We have 300 dimensional vector for each sentence of article now. We create a similarity matrix which keeps cosine distance of each sentences to every other sentence. It is obvious that the matrix is symmetric in nature. Below codes produces matrix and graph to display how a similarity matrix would look like.

sim_mat = np.zeros([len(cleaned_texts), len(cleaned_texts)])
for i in range(len(sentences)):
    for j in range(len(sentences)):
        if i != j:
            sim_mat[i][j] = cosine_similarity(sentence_vectors[i].reshape(1,dim),sentence_vectors[j].reshape(1,dim))[0,0]
sim_mat = np.round(sim_mat,3)
print(sim_mat)

# Creating the network graph
nx_graph = nx.from_numpy_array(sim_mat)
plt.figure(figsize=(10, 10))
pos = nx.spring_layout(nx_graph)
nx.draw(nx_graph, with_labels=True, font_weight='bold')
nx.draw_networkx_edge_labels(nx_graph,pos,font_color='red')
plt.show()

5. Text Rank Algorithm

Text rank algorithm is inspired from the famous Google’s PageRank algorithm. The underlying assumption of PageRank algorithm is that more important websites are likely to receive more links from other websites. It counts the number and quality of links to a page to determine a rough estimate of how important the website is. An illustrative image (taken from wikipedia) showing the importance of webpage can be seen below.

Similar to PageRank, The underlying assumption of text rank algorithm is that the summary sentences are similar (or linked) to most of the other sentences. Essentially, it runs PageRank on similarity graph designed in previous section. It builds a graph using some set of text sentences as vertices. Edges are based on measure of semantic similarity (glove embedding based) between the text sentence vertices.

scores = nx.pagerank(nx_graph)
print(scores)

6. Result

As we got the ranked score of all the 12 sentences in the article. There are simply two steps which had been implemented next.

1. Sort the sentences in decreasing order of their score. (Also, preserve the index of sentences).
2. Choose half of the sentences from top (Compression ratio 0.5) and sort it again based on their increasing indices. This will ensure that summary is coherent with original article (picked sentences are in order).

ranked_sentences = sorted(((scores[i],i) for i,s in enumerate(sentences)), reverse=True)
arranged_sentences = sorted(ranked_sentences[0:int(len(sentences)*0.5)], key=lambda x:x[1])
print("\n".join([sentences[x[1]] for x in arranged_sentences]))

At the End

The application of text summarization could be limitless. The blog-post demonstrates one technique to do so. Also, this is not enough as it only picks the main (kind of) sentences. Abstractive summarization is a way to go but it’s at early stage and an activate area of research. As far as sentence based extractive summarization is concerned, The similarity measure among sentences could be one of the various metrics available. For example,

• tf-idf score
• word2vec based similarity
• doc2vec (sentence to vec)
• word embeddings from deep learning models like ELMO or BERT based.

If you liked the post, follow this blog to get updates about upcoming articles. Also, share it so that it can reach out to the readers who can actually gain from this. Please feel free to discuss anything regarding the post. I would love to hear feedback from you.

Happy machine learning

I will be covering this task in couple of blog-posts. The purpose of the blog-post series is to build the question answering toy application understanding each step and provide an insight of implementation of the same to our readers.

The Task

In 2015, Facebook came up with a bAbI data-set and 20 toy tasks for testing text understanding and reasoning in the bAbI project. The tasks are described in detail in the paper here and GitHub link here. The aim is that each task tests a unique aspect of text and reasoning, and hence test different capabilities of learning models. In this blog-post, we will demonstrate one of the toy task named : “Basic factoid QA with single supporting fact“. The task is very simple illustrated below.

Sandra travelled to the kitchen. 
Sandra travelled to the hallway. 
Mary went to the bathroom. 
Sandra moved to the garden. 

Where is Sandra ?
Ground Truth: Garden (based on single supporting fact 4)

Looking at this reasoning task per se, is not generic in nature and answer is only based on single supporting fact from the story. The learning model should be able to learn the sequence of events and then answer the question “Where is Actor ?“.  Factoid QA means the answer would be single word.

bAbI Dataset

For the aforementioned task, We have 10,000 training examples and 1000 testing examples.

The file format for the task is as follows:

ID text
ID text
ID text
ID question[tab]answer[tab]supporting_fact ID.
...

Each sentence is provided with an ID. The IDs for a given “story” start at 1 and increase. When the IDs in a file reset back to 1 you can consider the following sentences as a new “story”. Supporting fact ID refer to the sentences within a “story”.

1 Mary moved to the bathroom.
2 John went to the hallway.
3 Where is Mary?        bathroom        1
4 Daniel went back to the hallway.
5 Sandra moved to the garden.
6 Where is Daniel?      hallway         4
7 John moved to the office.
8 Sandra journeyed to the bathroom.
9 Where is Daniel?      hallway         4
10 Mary moved to the hallway.
11 Daniel travelled to the office.
12 Where is Daniel?     office          11
13 John went back to the garden.
14 John moved to the bedroom.
15 Where is Sandra?     bathroom        8
1 Sandra travelled to the office.
2 Sandra went to the bathroom.
3 Where is Sandra?      bathroom        2

Further in the following sections, we will go step by step from

• Getting data
• Getting User Stories : Pre-Processing
• Feature Extraction
• Building Memory Network model
• Train the Model
• Tests & Results

1. Getting Data

Importing Libraries

import re
import tarfile
import numpy as np
from functools import reduce
from keras.utils.data_utils import get_file
from keras.preprocessing.sequence import pad_sequences

Downloading Data

try:
    #tar.gz data-set get saved on "~/.keras/datasets/" path
    path = get_file('babi-tasks-v1-2.tar.gz', origin='https://s3.amazonaws.com/text-datasets/babi_tasks_1-20_v1-2.tar.gz')
except:
    print('Error downloading dataset, please download it manually:\n'
          '$ wget http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz\n'
          '$ mv tasks_1-20_v1-2.tar.gz ~/.keras/datasets/babi-tasks-v1-2.tar.gz')
    raise

#reading a tar.gz file
tar = tarfile.open(path)

If you are not able to download, The data-set can be downloaded from here manually.

2. Getting User Stories : Pre-processing

This is a crucial step where we need to ingest the train and test file and extract stories out of it. The raw format in which text stories, questions and answers are kept is aforementioned. Initially, we will write some helper functions.

def tokenize(sent):
    '''
    argument: a sentence string
    returns a list of tokens(words)
    '''
    return [ x.strip() for x in re.split('(\W+)?', sent) if x.strip()]

def parse_stories(lines):
    '''
    - Parse stories provided in the bAbI tasks format
    - A story starts from line 1 to line 15. Every 3rd line,
      there is a question & answer.
    - Function extracts sub-stories within a story and
      creates tuples
    '''
    data = []
    story = []
    for line in lines:
        line = line.decode('utf-8').strip()
        nid, line = line.split(' ', 1)
        nid = int(nid)
        if nid == 1:
            # reset story when line ID=1 (start of new story)
            story = []
        if '\t' in line:
            # this line is tab separated Q, A & support fact ID
            q, a, supporting = line.split('\t')
            # tokenize the words of question
            q = tokenize(q)
            # Provide all the sub-stories till this question
            substory = [x for x in story if x]
            # A story ends and is appended to global story data-set
            data.append((substory, q, a))
            story.append('')
        else:
            # this line is a sentence of story
            sent = tokenize(line)
            story.append(sent)
    return data

def get_stories(f):
    '''
    argument: filename
    returns list of all stories in the argument data-set file
    '''
    # read the data file and parse 10k stories
    data = parse_stories(f.readlines())
    # lambda func to flatten the list of sentences into one list
    flatten = lambda data: reduce(lambda x, y: x + y, data)
    # creating list of tuples for each story
    data = [(flatten(story), q, answer) for story, q, answer in data]
    return data

Further, we use the above helper functions to get the train and test stories.

challenge = 'tasks_1-20_v1-2/en-10k/qa1_single-supporting-fact_{}.txt'
print('Extracting stories for the challenge: single_supporting_fact_10k')
# Extracting train stories
train_stories = get_stories(tar.extractfile(challenge.format('train')))
# Extracting test stories
test_stories = get_stories(tar.extractfile(challenge.format('test')))

To validate that the stories are extracted properly, we can simply check the number of stories in train and test stories variables. Also, let us see how a story looks like as of now.

print('Number of training stories:', len(train_stories))
print('Number of test stories:', len(test_stories))
train_stories[0]

3. Feature Extraction

Let us first write a helper function to vectorize each stories in order to fetch it to memory network model which we will be creating later.

def vectorize_stories(data, word_idx, story_maxlen, query_maxlen):
    # story vector initialization
    X = []
    # query vector initialization
    Xq = []
    # answer vector intialization
    Y = []
    for story, query, answer in data:
        # creating list of story word indices
        x = [word_idx[w] for w in story]
        # creating list of query word indices
        xq = [word_idx[w] for w in query]
        # let's not forget that index 0 is reserved
        y = np.zeros(len(word_idx) + 1)
        # creating label 1 for the answer word index
        y[word_idx] = 1
        X.append(x)
        Xq.append(xq)
        Y.append(y)
    return (pad_sequences(X, maxlen=story_maxlen),
            pad_sequences(Xq, maxlen=query_maxlen), np.array(Y))

The following snippet creates a vocabulary dictionary and extract word indices vectors as input features.

# creating vocabulary of words in train and test set
vocab = set()
for story, q, answer in train_stories + test_stories:
    vocab |= set(story + q + [answer])

# sorting the vocabulary
vocab = sorted(vocab)

# Reserve 0 for masking via pad_sequences
vocab_size = len(vocab) + 1

# calculate maximum length of story
story_maxlen = max(map(len, (x for x, _, _ in train_stories + test_stories)))

# calculate maximum length of question/query
query_maxlen = max(map(len, (x for _, x, _ in train_stories + test_stories)))

# creating word to index dictionary
word_idx = dict((c, i + 1) for i, c in enumerate(vocab))

# creating index to word dictionary
idx_word = dict((i+1, c) for i,c in enumerate(vocab))

# vectorize train story, query and answer sentences/word using vocab
inputs_train, queries_train, answers_train = vectorize_stories(train_stories,
                                                               word_idx,
                                                               story_maxlen,
                                                               query_maxlen)
# vectorize test story, query and answer sentences/word using vocab
inputs_test, queries_test, answers_test = vectorize_stories(test_stories,
                                                            word_idx,
                                                            story_maxlen,
                                                            query_maxlen)

We have prepared the feature vectors for our toy question and answering system. Let us display some analysis on the extracted vectors. The below python code snippets are used to explore and see how these vectors and vocabulary looks like.

print('-------------------------')
print('Vocabulary:\n',vocab,"\n")
print('Vocab size:', vocab_size, 'unique words')
print('Story max length:', story_maxlen, 'words')
print('Query max length:', query_maxlen, 'words')
print('Number of training stories:', len(train_stories))
print('Number of test stories:', len(test_stories))
print('-------------------------')

print('-------------------------')
print('inputs: integer tensor of shape (samples, max_length)')
print('inputs_train shape:', inputs_train.shape)
print('inputs_test shape:', inputs_test.shape)
print('input train sample', inputs_train[0,:])
print('-------------------------')

print('-------------------------')
print('queries: integer tensor of shape (samples, max_length)')
print('queries_train shape:', queries_train.shape)
print('queries_test shape:', queries_test.shape)
print('query train sample', queries_train[0,:])
print('-------------------------')

print('-------------------------')
print('answers: binary (1 or 0) tensor o<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>f shape (samples, vocab_size)')
print('answers_train shape:', answers_train.shape)
print('answers_test shape:', answers_test.shape)
print('answer train sample', answers_train[0,:])
print('-------------------------')

What’s Next ?

Hope it was easy to go through tutorial as I have tried to keep each step precisely and explainable in order to understand and reproduce (till feature extraction). Most of the codes are commented to make the logic clear. In the subsequent blog-post (PART 2), we will walk through the following steps using Keras to train a neural network model for question answering system.

1. Understanding & Building Memory Network Model
2. Training the Model
3. Visualization
4. Tests Results & Demo

If you liked the post, follow this blog to get updates about upcoming articles. Also, share it so that it can reach out to the readers who can actually gain from this. Please feel free to discuss anything regarding the post. I would love to hear feedback from you.

Happy deep learning

Motivation: The idea came from my previous post “Performing OCR by running parallel instances of Tesseract 4.0 : Python“. Saying that following could be some use cases where you may have to build such search engine on top of other applications. e.g.

• You have built an OCR app and converted millions of images into text files. You may want to build a search engine over converted text files to search contents of images.
• You have built a speech to text system where you are converting thousands of recorded audios into text data. You may like to search contents of audio in real time.

Here is a video demonstration of an desktop app developed in QT. It is a whoosh python implementation working in back end.

Introduction: Whoosh

Some of you might have heard about a popular java based library “Lucene” which is a search engine library written entirely in Java. You may find a python wrapper for Lucene. If you are looking for similar pythonic library, “Whoosh” is the one. Whoosh is a fast, featureful full-text indexing and searching library implemented in pure Python. Programmers can use it to easily add search functionality to their applications and websites.

Whoosh pypi package can simply be installed with pip:
pip install Whoosh

For the example demonstrated in this blog-post, You can download  a data-set of 70,000 text files which were taken from simple wiki articles from .

1. Creating Indexed Data: Whoosh

It is easy to index all your text files with Whoosh. Initially, the schema of the index has to be defined. Schema defines list of fields to be indexed or stored for each text file. It’s similar to how we define it for database. A field is a piece of information for each document in the index, such as its title or text content. Indexing of a field means it can be searched and it is also returned with results if defined as argument (stored=True) in schema. You only need to create the schema once while creating the index.

Finally, all the text documents are added to index writer in loop. Documents are indexed as per schema and has to be added as per schema design. Below is the python implementation for indexing  all the text documents of a directory.

import os
from whoosh.index import create_in
from whoosh.fields import Schema, TEXT, ID
import sys

def createSearchableData(root):   

    '''
    Schema definition: title(name of file), path(as ID), content(indexed
    but not stored),textdata (stored text content)
    '''
    schema = Schema(title=TEXT(stored=True),path=ID(stored=True),\
              content=TEXT,textdata=TEXT(stored=True))
    if not os.path.exists("indexdir"):
        os.mkdir("indexdir")

    # Creating a index writer to add document as per schema
    ix = create_in("indexdir",schema)
    writer = ix.writer()

    filepaths = [os.path.join(root,i) for i in os.listdir(root)]
    for path in filepaths:
        fp = open(path,'r')
        print(path)
        text = fp.read()
        writer.add_document(title=path.split("\\")[1], path=path,\
          content=text,textdata=text)
        fp.close()
    writer.commit()

root = "corpus"
createSearchableData(root)

2. Querying Indexed Data : Whoosh

Querying a indexed data has two important parts which you may like to look upon.

Query String : It is passed while searching the indexed data. Query string can be a single word, a single sentence to be matched exactly, multiple words with ‘AND’, multiple words with ‘OR’ etc. For examples –

Query : politics (returns if the word occurs)Query : sports OR games OR play (returns if any one of the strings occur)
Query : alpha beta gamma (return if a document contains all strings)
Query : “alpha beta gamma“ (returns if all strings occur together in a document).

Scoring : Each document is ranked according to a scoring function. There are quite a few types of scoring function supported by whoosh.

1. Frequency : It simply returns the count of the terms occurred in the document. It does not perform any normalization or weighting.
2. Tf-Idf scores : It returns tf * idf scores of each document. To know more read here or wiki page here.
3. BM25F scoring : It is the by default ranking function used by whoosh. BM stands for best matching. It is based on tf-idf along with bunch of factors like length of document in words, average length of documents in the collection. It also has free parameters k = 1.2 and b = 0.75. To read more check here.
4. Cosine scoring : It is useful for finding document similar to your search query.

There are few more scoring algorithms which has been implemented. Check here to know more.

Below is the python implementation for searching a query in the indexed database.

from whoosh.qparser import QueryParser
from whoosh import scoring
from whoosh.index import open_dir

ix = open_dir("indexdir")

# query_str is query string
query_str = sys.argv[1]
# Top 'n' documents as result
topN = int(sys.argv[2])

with ix.searcher(weighting=scoring.Frequency) as searcher:
    query = QueryParser("content", ix.schema).parse(query_str)
results = searcher.search(query,limit=topN)
for i in range(topN):
    print(results[i]['title'], str(results[i].score), results[i]['textdata'])

QueryParser class of whoosh implements query language very similar to java Lucene’s.

3. Glossary

Below are the basic terminologies you will always come across in discussions involving searching and indexing documents (taken from whoosh docs).

Corpus
The set of documents you are indexing.

Documents
The individual pieces of content you want to make searchable. The word “documents” might imply files, but the data source could really be anything – articles in a content management system, blog posts in a blogging system, chunks of a very large file, rows returned from an SQL query, individual email messages from a mailbox file, or whatever. When you get search results from Whoosh, the results are a list of documents, whatever “documents” means in your search engine.

Fields
Each document contains a set of fields. Typical fields might be “title”, “content”, “url”, “keywords”, “status”, “date”, etc. Fields can be indexed (so they’re searchable) and/or stored with the document. Storing the field makes it available in search results. For example, you typically want to store the “title” field so your search results can display it.

Forward index
A table listing every document and the words that appear in the document. Whoosh lets you store term vectors that are a kind of forward index.

Indexing
The process of examining documents in the corpus and adding them to the reverse index.

Postings
The reverse index lists every word in the corpus, and for each word, a list of documents in which that word appears, along with some optional information (such as the number of times the word appears in that document). These items in the list, containing a document number and any extra information, are called postings. In Whoosh the information stored in postings is customizable for each field.

Reverse Index
Basically a table listing every word in the corpus, and for each word, the list of documents in which it appears. It can be more complicated (the index can also list how many times the word appears in each document, the positions at which it appears, etc.) but that’s how it basically works.

Schema
Whoosh requires that you specify the fields of the index before you begin indexing. The Schema associates field names with metadata about the field, such as the format of the postings and whether the contents of the field are stored in the index.

Term vector
A forward index for a certain field in a certain document. You can specify in the Schema that a given field should store term vectors.

At the End

Hope it was an easy read and good heads-up to start with. So, what can we do more from here:

1. Searching for alike documents instead of exact term searches only.
2. Exploring hierarchical search in file system.
3. Correcting errors in the queries. Did you mean…?
4. Search N-grams for getting fast, “search as you type” functionality

Update: Readers can download the backup of QT application built from below link. It was a old work and don’t have a working app now. You may have to figure out codes in order to reproduce it. Good luck with that.

If you liked the post, follow this blog to get updates about upcoming articles. Also, share it so that it can reach out to the readers who can actually gain from this. Please feel free to discuss anything regarding the post. I would love to hear feedback from you.

Happy machine learning 

Introduction

An face emotion recognition system comprises of two step process i.e. face detection (bounded face) in image followed by emotion detection on the detected bounded face. The following two techniques are used for respective mentioned tasks in face recognition system.

1. Haar feature-based cascade classifiers : It detects frontal face in an image well. It is real time and faster in comparison to other face detector. This blog-post uses an implementation from Open-CV.
2. Xception CNN Model (Mini_Xception, 2017) : We will train a classification CNN model architecture which takes bounded face (48*48 pixels) as input and predicts probabilities of 7 emotions in the output layer.

Data-set

One can download the facial expression recognition (FER) data-set from Kaggle challenge here. The data consists of 48×48 pixel gray scale images of faces. The faces have been automatically registered so that the face is more or less centered and occupies about the same amount of space in each image. The task is to categorize each face based on the emotion shown in the facial expression in to one of seven categories (0=Angry, 1=Disgust, 2=Fear, 3=Happy, 4=Sad, 5=Surprise, 6=Neutral).

The training set consists of 35,888 examples. train.csv contains two columns, “emotion” and “pixels”. The “emotion” column contains a numeric code ranging from 0 to 6, inclusive, for the emotion that is present in the image. The “pixels” column contains a string surrounded in quotes for each image. The contents of this string a space-separated pixel values in row major order

Loading FER Data-set

The below code loads the data-set and pre-process the images for feeding it to CNN model. There are two definitions in the code snippet here:

1. def load_fer2013 : It reads the csv file and convert pixel sequence of each row in image of dimension 48*48. It returns faces and emotion labels.

2. def preprocess_input: It is a standard way to pre-process images by scaling them between -1 to 1. Images is scaled to [0,1] by dividing it by 255. Further, subtraction by 0.5 and multiplication by 2 changes the range to [-1,1]. [-1,1] has been found a better range for neural network models in computer vision problems.

import pandas as pd
import cv2
import numpy as np

dataset_path = 'fer2013/fer2013.csv'
image_size=(48,48)

def load_fer2013():
        data = pd.read_csv(dataset_path)
        pixels = data['pixels'].tolist()
        width, height = 48, 48
        faces = []
        for pixel_sequence in pixels:
                face = [int(pixel) for pixel in pixel_sequence.split(' ')]
                face = np.asarray(face).reshape(width, height)
                face = cv2.resize(face.astype('uint8'),image_size)
                faces.append(face.astype('float32'))
        faces = np.asarray(faces)
        faces = np.expand_dims(faces, -1)
        emotions = pd.get_dummies(data['emotion']).as_matrix()
        return faces, emotions

def preprocess_input(x, v2=True):
    x = x.astype('float32')
    x = x / 255.0
    if v2:
        x = x - 0.5
        x = x * 2.0
    return x

faces, emotions = load_fer2013()
faces = preprocess_input(faces)
xtrain, xtest,ytrain,ytest = train_test_split(faces, emotions,test_size=0.2,shuffle=True)

5 expression samples of each of the 7 emotions in the data-set can be seen below.

Originally in the dataset provided in kaggle link, each image is given as string which is a row 1×2304 which is 48×48 image stored as row vector. The strings in the .csv files can be converted into images using the code in github link here.

Training CNN model : Mini Xception

Here comes the exciting architecture which is comparatively small and achieves almost state-of-art performance of classifying emotion on this data-set. The below architecture was proposed by Octavio Arragia et al. in this paper.

One can notice that the center block is repeated 4 times in the design. This architecture is different from the most common CNN architecture like one used in the blog-post here. Common architectures uses fully connected layers at the end where most of parameters resides. Also, they use standard convolutions. Modern CNN architectures such as Xception leverage from the combination of two of the most successful experimental assumptions in CNNs: the use of residual modules and depth-wise separable convolutions.

There are various techniques that can be kept in mind while building a deep neural network and is applicable in most of the computer vision problems. Below are few of those techniques which are used while training the CNN model below.

1. Data Augmentation : More data is generated using the training set by applying transformations. It is required if the training set is not sufficient enough to learn representation. The image data is generated by transforming the actual training images by rotation, crop, shifts, shear, zoom, flip, reflection, normalization etc.
2. Kernel_regularizer  : It allows to apply penalties on layer parameters during optimization. These penalties are incorporated in the loss function that the network optimizes. Argument in convolution layer  is nothing but L2 regularisation of the weights. This penalizes peaky weights and makes sure that all the inputs are considered.
3. BatchNormalization : It normalizes the activation of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1. It addresses the problem of internal covariate shift. It also acts as a regularizer, in some cases eliminating the need for Dropout. It helps in speeding up the training process.
4. Global Average Pooling : It reduces each feature map into a scalar value by taking the average over all elements in the feature map. The average operation forces the network to extract global features from the input image.
5. Depthwise Separable Convolution : These convolutions are composed of two different layers: depth-wise convolutions and point-wise convolutions. Depth-wise separable convolutions reduces the computation with respect to the standard convolutions by reducing the number of parameters. A very nice and visual explanation of the difference between standard and depth-wise separable convolution is given in the paper.

Below python codes implements the above architecture in Keras.

from keras.callbacks import CSVLogger, ModelCheckpoint, EarlyStopping
from keras.callbacks import ReduceLROnPlateau
from keras.preprocessing.image import ImageDataGenerator
from sklearn.model_selection import train_test_split
from keras.layers import Activation, Convolution2D, Dropout, Conv2D
from keras.layers import AveragePooling2D, BatchNormalization
from keras.layers import GlobalAveragePooling2D
from keras.models import Sequential
from keras.layers import Flatten
from keras.models import Model
from keras.layers import Input
from keras.layers import MaxPooling2D
from keras.layers import SeparableConv2D
from keras import layers
from keras.regularizers import l2
import pandas as pd
import cv2
import numpy as np

# parameters
batch_size = 32
num_epochs = 110
input_shape = (48, 48, 1)
verbose = 1
num_classes = 7
patience = 50
base_path = 'models/'
l2_regularization=0.01

# data generator
data_generator = ImageDataGenerator(
                        featurewise_center=False,
                        featurewise_std_normalization=False,
                        rotation_range=10,
                        width_shift_range=0.1,
                        height_shift_range=0.1,
                        zoom_range=.1,
                        horizontal_flip=True)

# model parameters
regularization = l2(l2_regularization)

# base
img_input = Input(input_shape)
x = Conv2D(8, (3, 3), strides=(1, 1), kernel_regularizer=regularization, use_bias=False)(img_input)
x = BatchNormalization()(x)
x = Activation('relu')(x)
x = Conv2D(8, (3, 3), strides=(1, 1), kernel_regularizer=regularization, use_bias=False)(x)
x = BatchNormalization()(x)
x = Activation('relu')(x)

# module 1
residual = Conv2D(16, (1, 1), strides=(2, 2), padding='same', use_bias=False)(x)
residual = BatchNormalization()(residual)
x = SeparableConv2D(16, (3, 3), padding='same', kernel_regularizer=regularization, use_bias=False)(x)
x = BatchNormalization()(x)
x = Activation('relu')(x)
x = SeparableConv2D(16, (3, 3), padding='same', kernel_regularizer=regularization, use_bias=False)(x)
x = BatchNormalization()(x)
x = MaxPooling2D((3, 3), strides=(2, 2), padding='same')(x)
x = layers.add([x, residual])

# module 2
residual = Conv2D(32, (1, 1), strides=(2, 2), padding='same', use_bias=False)(x)
residual = BatchNormalization()(residual)
x = SeparableConv2D(32, (3, 3), padding='same', kernel_regularizer=regularization, use_bias=False)(x)
x = BatchNormalization()(x)
x = Activation('relu')(x)
x = SeparableConv2D(32, (3, 3), padding='same', kernel_regularizer=regularization, use_bias=False)(x)
x = BatchNormalization()(x)
x = MaxPooling2D((3, 3), strides=(2, 2), padding='same')(x)
x = layers.add([x, residual])

# module 3
residual = Conv2D(64, (1, 1), strides=(2, 2),padding='same', use_bias=False)(x)
residual = BatchNormalization()(residual)
x = SeparableConv2D(64, (3, 3), padding='same',kernel_regularizer=regularization,use_bias=False)(x)
x = BatchNormalization()(x)
x = Activation('relu')(x)
x = SeparableConv2D(64, (3, 3), padding='same',kernel_regularizer=regularization,use_bias=False)(x)
x = BatchNormalization()(x)
x = MaxPooling2D((3, 3), strides=(2, 2), padding='same')(x)
x = layers.add([x, residual])

# module 4
residual = Conv2D(128, (1, 1), strides=(2, 2),padding='same', use_bias=False)(x)
residual = BatchNormalization()(residual)
x = SeparableConv2D(128, (3, 3), padding='same',kernel_regularizer=regularization,use_bias=False)(x)
x = BatchNormalization()(x)
x = Activation('relu')(x)
x = SeparableConv2D(128, (3, 3), padding='same',kernel_regularizer=regularization,use_bias=False)(x)
x = BatchNormalization()(x)
x = MaxPooling2D((3, 3), strides=(2, 2), padding='same')(x)
x = layers.add([x, residual])
x = Conv2D(num_classes, (3, 3), padding='same')(x)
x = GlobalAveragePooling2D()(x)
output = Activation('softmax',name='predictions')(x)

model = Model(img_input, output)
model.compile(optimizer='adam', loss='categorical_crossentropy',metrics=['accuracy'])
model.summary()

# callbacks
log_file_path = base_path + '_emotion_training.log'
csv_logger = CSVLogger(log_file_path, append=False)
early_stop = EarlyStopping('val_loss', patience=patience)
reduce_lr = ReduceLROnPlateau('val_loss', factor=0.1, patience=int(patience/4), verbose=1)
trained_models_path = base_path + '_mini_XCEPTION'
model_names = trained_models_path + '.{epoch:02d}-{val_acc:.2f}.hdf5'
model_checkpoint = ModelCheckpoint(model_names, 'val_loss', verbose=1,save_best_only=True)
callbacks = [model_checkpoint, csv_logger, early_stop, reduce_lr]

model.fit_generator(data_generator.flow(xtrain, ytrain,batch_size),
                        steps_per_epoch=len(xtrain) / batch_size,
                        epochs=num_epochs, verbose=1, callbacks=callbacks,
                        validation_data=(xtest,ytest))

The model gives 65-66% accuracy on validation set while training the model. The CNN model learns the representation features of emotions from the training images. Below are few epochs of training process with batch size of 64.

Testing the Model

On Images

While performing tests on the trained model, I felt that model detects the emotion of faces as neutral if the expressions are not made distinguishable enough. The model gives probabilities of each emotion class in the output layer of trained mini_xception CNN model. Below are the 18 facial expressions taken from google images to validate the trained model.

In order to detect emotion in a single image, one can execute the python code below.

from keras.preprocessing.image import img_to_array
from keras.models import load_model
import imutils
import cv2
import numpy as np
import sys

# parameters for loading data and images
detection_model_path = 'haarcascade_files/haarcascade_frontalface_default.xml'
emotion_model_path = 'models/_mini_XCEPTION.106-0.65.hdf5'
img_path = sys.argv[1]

# hyper-parameters for bounding boxes shape
# loading models
face_detection = cv2.CascadeClassifier(detection_model_path)
emotion_classifier = load_model(emotion_model_path, compile=False)
EMOTIONS = ["angry","disgust","scared", "happy", "sad", "surprised","neutral"]

#reading the frame
orig_frame = cv2.imread(img_path)
frame = cv2.imread(img_path,0)
faces = face_detection.detectMultiScale(frame,scaleFactor=1.1,minNeighbors=5,minSize=(30,30),flags=cv2.CASCADE_SCALE_IMAGE)

if len(faces) > 0:
    faces = sorted(faces, reverse=True,key=lambda x: (x[2] - x[0]) * (x[3] - x[1]))[0]
    (fX, fY, fW, fH) = faces
    roi = frame[fY:fY + fH, fX:fX + fW]
    roi = cv2.resize(roi, (48, 48))
    roi = roi.astype("float") / 255.0
    roi = img_to_array(roi)
    roi = np.expand_dims(roi, axis=0)
    preds = emotion_classifier.predict(roi)[0]
    emotion_probability = np.max(preds)
    label = EMOTIONS[preds.argmax()]
    cv2.putText(orig_frame, label, (fX, fY - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 0, 255), 2)
    cv2.rectangle(orig_frame, (fX, fY), (fX + fW, fY + fH),(0, 0, 255), 2)

cv2.imshow('test_face', orig_frame)
cv2.imwrite('test_output/'+img_path.split('/')[-1],orig_frame)
if (cv2.waitKey(2000) & 0xFF == ord('q')):
    sys.exit("Thanks")
cv2.destroyAllWindows()

On Video

In order to detect emotion in webcam, one can execute the python code here.

References

The demonstration codes has been ingested from following sources.

[1] https://github.com/oarriaga/face_classification
[2] https://github.com/omar178/Emotion-recognition
[3] Real-time Convolutional Neural Networks for Emotion and Gender Classification

If you liked the post, follow this blog to get updates about upcoming articles. Also, share it so that it can reach out to the readers who can actually gain from this. Please feel free to discuss anything regarding the post. I would love to hear feedback from you.

Happy deep learning

• 1  Importing Libraries
• 2  User Defined Functions
• 3  Reading Data
  • 3.1  Checking the event rate
• 4  Displaying the attributes
• 5  Checking Data Quality
• 6  Missing Value Treatment
• 7  Looking at attributes (EDA)
• 8  Preparing Data for Modeling
• 9  Model 1 – XGB Classifier

HR Analytics : Hackathon Challenge

I participated in WNS Analytics Wizard hackathon, “To predict whether an employee will be promoted or not” and hence I am coming up with this blog-post of the solution submitted which ranked me 138 (Top 11%) in the challenge. The leader board ranking was decided on the F1-score which is harmonic mean of precision and recall.

About Data
The data-set consists of 54808 rows where each row had 14 attributes including target variable (i.e “is_promoted”). There are 4668 cases where employees have been promoted (8.5%). The data-set is provided in GitHub link here.

Let’s get started in building the data analytics pipeline end to end.

Importing Libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from IPython.display import display

from sklearn.metrics import confusion_matrix, f1_score, precision_recall_curve
from sklearn.model_selection import GridSearchCV, train_test_split,cross_val_score

import xgboost as xgb
import lightgbm as lgb

import warnings
warnings.filterwarnings("ignore")

# Set all options
%matplotlib inline
plt.style.use('seaborn-notebook')
plt.rcParams["figure.figsize"] = (20, 3)
pd.options.display.float_format = '{:20,.4f}'.format
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
sns.set(context="paper", font="monospace")

User Defined Functions

def convert_categorical_to_dummies(d_convert):

    """
    Author: Abhijeet Kumar
    Description: returns Dataframe with all categorical variables converted into dummies
    Arguments: Dataframe (having categorical variables)
    """

    df = d_convert.copy()
    list_to_drop = []
    for col in df.columns:
        if df[col].dtype == 'object':
            list_to_drop.append(col)
            df = pd.concat([df,pd.get_dummies(df[col],prefix=col,prefix_sep='_', drop_first=False)], axis=1)
            df = df.drop(list_to_drop,axis=1)
    return df

def quality_report(df):

    """
    Author: Abhijeet Kumar
    Description: Displays quality of data in terms of missing values, unique numbers, datatypes etc.
    Arguments: Dataframe
    """
    dtypes = df.dtypes
    nuniq = df.T.apply(lambda x: x.nunique(), axis=1)
    total = df.isnull().sum().sort_values(ascending = False)
    percent = (df.isnull().sum()/df.isnull().count()*100).sort_values(ascending = False)
    quality_df = pd.concat([total, percent, nuniq, dtypes], axis=1, keys=['Total', 'Percent','Nunique', 'Dtype'])
    display(quality_df)

def score_on_test_set(model, file_name, out_name):

    """
    Author: Abhijeet Kumar
    Description : It runs same steps of preprocessing as in training, scores
    on the test data provided in hackathon and generates the submission file.
    Argument : model, test data file, submission file
    """

    test_data = pd.read_csv(file_name)

    # Treating the missing values of education as a separate category
    test_data['education'] = test_data['education'].replace(np.NaN, 'NA')

    # Treating the missing values of education as a separate category
    test_data['previous_year_rating'] = test_data['previous_year_rating'].fillna(0)

    # Creating dummy variables for all the categorical columns, droping that column
    master_test_data = convert_categorical_to_dummies(test_data)

    # Removing the id attributes
    df_test_data = master_test_data.drop(['employee_id'],axis=1)
    if out_name == "submission_lightgbm.csv":
        y_pred = model.predict_proba(df_test_data.values, num_iteration=model.best_iteration_)
    else:
        y_pred = model.predict_proba(df_test_data.values)
    submission_df = pd.DataFrame({'employee_id':master_test_data['employee_id'],'is_promoted':y_pred[:,1]})
    submission_df.to_csv(out_name, index=False)

    score = model.predict_proba(df_test_data.values)
    return test_data,score

Reading Data

data = pd.read_csv("train.csv")
print("Shape of Data = ",data.shape)
data.sample(5)

Shape of Data =  (54808, 14)

Checking the event rate

plt.figure(figsize=(6,3))
sns.countplot(x='is_promoted',data=data)
plt.show()

# Checking the event rate : event is when claim is made
data['is_promoted'].value_counts()

0    50140
1     4668
Name: is_promoted, dtype: int64

Displaying the attributes

# Checking the attribute names
pd.DataFrame(data.columns)

Checking Data Quality

# checking missing data
quality_report(data)

Missing Value Treatment

# Treating the missing values of education as a separate category
data['education'] = data['education'].replace(np.NaN, 'NA')

# Treating the missing values of previous year rating as 0
data['previous_year_rating'] = data['previous_year_rating'].fillna(0)

Looking at attributes (EDA)

Can we make some inferences from EDA ?

• Promotions are worst in Legal department (5.1%). Best promotions are in technology department (10.7%).
• Region 9 is worst (1.9%) and region 4 is best (14.4%) in terms of promotions.
• Although Master’s & above has greater promotion percentage but difference is not much.
• Employees having previous years rating greater than 5 will have better chances of promotion than others.
• Employess having KPI greater than 80% has good chances of promotions (16%)
• Employees winning awards are promoted more (44%).

for col in data.drop('is_promoted',axis=1).columns:
    if data[col].dtype == 'object' or data[col].nunique()
        xx = data.groupby(col)['is_promoted'].value_counts().unstack(1)
        per_not_promoted = xx.iloc[:, 0] *100/xx.apply(lambda x: x.sum(), axis=1)
        per_promoted = xx.iloc[:, 1]*100/xx.apply(lambda x: x.sum(), axis=1)
        xx['%_0'] = per_not_promoted
        xx['%_1'] = per_promoted
        display(xx)

Preparing Data for Modeling

# Creating dummy variables for all the categorical columns, droping that column
master_data = convert_categorical_to_dummies(data)
print("Total shape of Data :",master_data.shape)

# dropping the target from dataset
labels = np.array(master_data['is_promoted'].tolist())

# Removing the id attributes
df_data = master_data.drop(['is_promoted','employee_id'],axis=1)
print("Shape of Data:",df_data.shape)
df = df_data.values

Total shape of Data : (54808, 61)
Shape of Data: (54808, 59)

Model 1 – XGB Classifier

xgb_model = xgb.XGBClassifier()
print(xgb_model)

# Cross validation scores
f1_scores = cross_val_score(xgb_model, df, labels, cv=5, scoring='f1')
print("F1-score = ",f1_scores," Mean F1 score = ",np.mean(f1_scores))

# Training the models
xgb_model.fit(df,labels)

# Scoring on test set
test_data,score_xgb = score_on_test_set(xgb_model,"test.csv","submission_xgb.csv")

XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
       colsample_bytree=1, gamma=0, learning_rate=0.1, max_delta_step=0,
       max_depth=3, min_child_weight=1, missing=None, n_estimators=100,
       n_jobs=1, nthread=None, objective='binary:logistic', random_state=0,
       reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,
       silent=True, subsample=1)
F1-score = [ 0.4526749   0.41547519  0.43579122  0.43012552  0.43427621]  
Mean F1 score =  0.433668606717

XGB Classifier : Parameter Tuning

Our goal is usually to set the model parameters to optimal values that enable a model to complete learning task in the best way possible. Thus, tuning XGboost classifier can optimize the parameters that impact the model in order to enable the algorithm to perform the best.
I performed lot of iterations patiently which led to fine tuning of parameters: n_estimators, max_depth and L1 regularization. A norm is to take baby steps to learn (small learning rate) and tune the parameters. Here, I found that with large number of trees (n_estimators), the F1-scores were improving.

# Create parameters to search
params = {
     'learning_rate': [0.01],
     'n_estimators': [900,1000,1100],
     'max_depth':[7,8,9],
     'reg_alpha':[0.3,0.4,0.5]
    }

# Initializing the XGBoost Regressor
xgb_model = xgb.XGBClassifier()

# Gridsearch initializaation
gsearch = GridSearchCV(xgb_model, params,
                    verbose=True,
                    cv=5,
                    n_jobs=2)

gsearch.fit(df, labels)

#Printing the best chosen params
print("Best Parameters :",gsearch.best_params_)

params = {'objective':'binary:logistic', 'booster':'gbtree'}

# Updating the parameter as per grid search
params.update(gsearch.best_params_)

# Initializing the XGBoost Regressor
xgb_model = xgb.XGBClassifier(**params)
print(xgb_model)

# Cross validation scores
f1_scores = cross_val_score(xgb_model, df, labels, cv=5, scoring='f1',n_jobs=2)
print("F1_scores per fold : ",f1_scores," \nMean F1_score= ",np.mean(f1_scores))

# Fitting model on tuned parameters
xgb_model.fit(df, labels)

# Scoring on test set
test_data,score_xgb_tuned = score_on_test_set(xgb_model,"test.csv","submission_xgb_tuned.csv")

Fitting 5 folds for each of 1 candidates, totalling 5 fits

[Parallel(n_jobs=2)]: Done   5 out of   5 | elapsed: 13.0min finished

Best Parameters :{'learning_rate': 0.01, 'max_depth': 8, 'n_estimators': 1000, 'reg_alpha': 0.4}

XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
       colsample_bytree=1, gamma=0, learning_rate=0.01, max_delta_step=0,
       max_depth=8, min_child_weight=1, missing=None, n_estimators=1000,
       n_jobs=1, nthread=None, objective='binary:logistic', random_state=0,
       reg_alpha=0.4, reg_lambda=1, scale_pos_weight=1, seed=None,
       silent=True, subsample=1)
F1_scores per fold : [ 0.51014041  0.48657188  0.49528302  0.53054911  0.51130164]  
Mean F1_score= 0.506769210361

XGB Classifier : Setting threshold

How does XGBoost classifier predicts the class (‘promoted’ or ‘not promoted’) ? It predicts a probability between 0 and 1 for the unseen cases. Further, it predicts 0 and 1 by putting a threshold at 0.5 by default (1 if probability > 0.5). In unbalance data-set as in here, it may be a biased setting as it would be difficult to capture rare event with 0.5 threshold.

• We can change the by default threshold of 0.5 by finding the optimal threshold to increase F1-score.
• We need to find the threshold where f1-score is highest.
• I tried submissions on few optimal cut-offs to get maximum possible improved F1-score.

The following python code splits the data in 90:10 and trains XGBoost classifier with tuned parameters. It calculates precision and recall at different thresholds and plots the precision recall curve. Further, we calculate F1-score for the same using precision and recall values.

# Splitting the dataset in order to use early stopping round
X_train, X_test, y_train, y_test = train_test_split(df, labels, test_size=0.10, stratify=labels)

xgb_model = xgb.XGBClassifier(**params)

# Training the models
xgb_model.fit(X_train, y_train)
y_pred = xgb_model.predict_proba(X_test)
precision, recall, thresholds = precision_recall_curve(y_test, y_pred[:,1])

thresholds = np.append(thresholds, 1)
f1_scores = 2*(precision*recall)/(precision+recall)
plt.step(recall, precision, color='b', alpha=0.4, where='post')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.ylim([0.0, 1.05])
plt.xlim([0.0, 1.0])
plt.title('2-class Precision-Recall curve')
plt.show()

Getting optimal threshold

We plot F1-scores with respect to threshold in x-axis to check the F1-score peak. The below python codes gets the threshold value where the F1-score was highest.

scrs = pd.DataFrame({'precision' : precision, 'recal' : recall, 'thresholds' : thresholds, 'f1_score':f1_scores})
print("Threshold cutoff: ",scrs.loc[scrs['f1_score'] == scrs.f1_score.max(),'thresholds'].iloc[0])
print("Max F1-score at cut-off : ",scrs.f1_score.max())
scrs.plot(x='thresholds', y='f1_score')

Threshold cutoff:  0.340377241373
Max F1-score at cut-off :  0.53791130186

Once you get the optimal threshold, use it for test set probability predictions as a cutoff to predict class labels 0 and 1 for the final submission.

What did not work ?

I tried the following other techniques which did not work and hence my final submissions were based on single model “XGBoost classifier” as described in this post.

• I tried logistic regression and SVM, f1 score was low (less than 0.4).
• I tried Random Forest. F1-score was comparatively low.
• I tried LightGBM model. In default setting, It gave 0.50 f1-score but somehow it was not improving with parameter tuning. Little improvement was there when early_stopping_rounds was used. We consider best iteration for predictions on test set.
• I created some interaction variables like if previous_year_rating == 5 and KPI > 80 == 1 then 1 else 0, if awards_won? == 1 and KPI > 80 == 1 then 1 else 0. It did not help.
• Finally, I took the best tuned params of all three (RF, XGboost and LightGBM) and stacked them with ‘Logistics Regression’ as classifier. It did not gave better f1 than individual XGB Classifier model.

At the End

Readers are also encouraged to download the data-set and check if they can reproduce the results. Also, I would love to check in comments if you can surpass the F1-score achieved here in the blog-post. There are following other things which one can try.

• Generally, Stacking improves scores when there are lot of models. One can train say 100s of models of XGBoost and LightGBM (with different close by parameters) and then apply logistic regression on top of that (I tried with only 3 models, failed).
• Also, one can try an interaction variable by calculating total score achieved in training (Number of training * Avg. training score).
• One can try setting “early_stopping_rounds”  in XGBoost classifier training which I did not tried. It prevents over-fitting and can improve results.

The full implementation of the followed approach along with LightGBM model example (jupyter notebook) can be downloaded from GitHub link here.

If you liked the post, follow this blog to get updates about upcoming articles. Also, share it so that it can reach out to the readers who can actually gain from this. Please feel free to discuss anything regarding the post. I would love to hear feedback from you.

Happy data analytics 

This blog-post implements the above two tasks using well-known machine algorithms: K-NN and K-Means respectively. We will walk through the following steps:

1. A simple text cleaning definition.
2. Feature Extraction.
3. Training the models.
4. Testing the models.

For training K-NN and K-Means models, the following 30 sentences were collected from 3 categories, namely Cricket, Artificial Intelligence and Chemistry.

1. Cricket is a bat and ball game played between two teams of eleven players each on a cricket field.
2. Each phase of play is called an innings during which one team bats, attempting to score as many runs as possible.
3. The teams have one or two innings apiece and, when the first innings ends, the teams swap roles for the next innings
4. Before a match begins, the two team captains meet on the pitch for the toss of a coin to determine which team will bat first.
5. Two batsmen and eleven fielders then enter the field and play begins when a member of the fielding team, known as the bowler, delivers the ball.
6. The most common dismissal in cricket match are bowled, when the bowler hits the stumps directly with the ball and dislodges the bails. Batsman gets out.
7. Runs are scored by two main methods: either by hitting the ball hard enough for it to cross the boundary, or by the two batsmen swapping ends.
8. The main objective of each team is to score more runs than their opponents.
9. If the team batting last is all out having scored fewer runs than their opponents, they are said to have "lost by n runs".
10. The role of striker batsman is to prevent the ball from hitting the stumps by using his bat and, simultaneously, to strike it well enough to score runs
11. Artificial intelligence is intelligence exhibited by machines, rather than humans or other animals. 
12. the field of AI research defines itself as the study of "intelligent agents": any device that perceives its environment and takes actions that maximize its chance of success at some goal
13. The overall research goal of artificial intelligence is to create technology that allows computers and machines to function in an intelligent manner.
14. Natural language processing[77] gives machines the ability to read and understand human language and extract intelligence from it.
15. AI researchers developed sophisticated mathematical tools to solve specific subproblems. These tools are truly scientific, in the sense that their results are both measurable and verifiable.
16. An intelligent agent is a system that perceives its environment and takes actions which maximize its chances of success.
17. AI techniques have become an essential part of the technology industry, helping to solve many challenging problems in computer science.
18. Recent advancements in AI, and specifically in machine learning, have contributed to the growth of Autonomous Things such as drones and self-driving cars.
19. AI research was revived by the commercial success of expert systems,[28] a form of AI program that simulated the knowledge and analytical skills of human experts.
20. Advanced statistical techniques (loosely known as deep learning), access to large amounts of data and faster computers enabled advances in machine learning and perception.
21. A compound is a pure chemical substance composed of more than one element and the properties of a compound bear little similarity to those of its elements.
22. Since the properties of an element are mostly determined by its electron configuration, the properties of the elements likewise show recurring patterns or periodic behaviour.
23. The property of inertness of noble gases makes them very suitable in chemicals where reactions are not wanted.
24. The atom is also the smallest entity that can be envisaged to retain the chemical properties of the element, such as electronegativity, ionization potential and preferred oxidation state.
25. The nucleus is made up of positively charged protons and uncharged neutrons (together called nucleons), while the electron cloud consists of negatively charged electrons which orbit the nucleus
26. The atom is the basic unit of chemistry. It consists of a dense core called the atomic nucleus surrounded by a space called the electron cloud.
27. A chemical reaction is a transformation of some substances into one or more different substances.
28. Chemistry is sometimes called the central science because it bridges other natural sciences, including physics, geology and biology.
29. Chemistry includes topics such as the properties of individual atoms and how atoms form chemical bonds to create chemical compounds.
30. Chemistry is a branch of physical science that studies the composition, structure of atoms, properties and change of matter.

A very simple step by step procedure to build the pipeline for training a machine learning model for text analytics applications can be followed like this:

Texts  ==>  Stop words removal ==> Punctuation free ==> Word Lemmatization ==> Digit removal ==> Feature Extraction (Tf-Idf) ==> Model training

Lets start building the pipeline in Python.

1. A Text cleaning definition.

Data pre-processing (a.k.a data cleaning) is one of the most significant step in text analytics. The purpose is to remove any unwanted words or characters which are written for human readability, but won’t contribute to the classification or clustering task in any way.
In general, there are mainly 4 cleaning steps that need to be done on text sentences:

a) Removal of stop words – Stop words like “and”, “if”, “the”, etc are very common in all English sentences and are not very meaningful in deciding the theme of the article, so these words can be removed from the articles.

b) Removal of Punctuation Characters – Exclude all punctuation marks from the set([‘!’, ‘#’, ‘”‘, ‘%’, ‘$’, “‘”, ‘&’, ‘)’, ‘(‘, ‘+’, ‘*’, ‘-‘, ‘,’, ‘/’, ‘.’, ‘;’, ‘:’, ‘=’, ‘<‘, ‘?’, ‘>’, ‘@’, ‘[‘, ‘]’, ‘\\’, ‘_’, ‘^’, ‘`’, ‘{‘, ‘}’, ‘|’, ‘~’]).

c) Lemmatization – It is the process of grouping together the different inflected forms of a word so they can be analyzed as a single item. For example, “include”, “includes,” and “included” would all be represented as “include”. The context of the sentence is also preserved in lemmatization as opposed to stemming (another buzz word in text mining which does not consider the meaning of the sentence).

d) Removal of digits from the text sentence.

The following Python code defines a function clean() for cleaning the text article passed as an argument to it:

# Importing libraries
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.neighbors import KNeighborsClassifier
from sklearn.cluster import KMeans
from nltk.corpus import stopwords
from nltk.stem.wordnet import WordNetLemmatizer
import string
import re
import numpy as np
from collections import Counter

stop = set(stopwords.words('english'))
exclude = set(string.punctuation)
lemma = WordNetLemmatizer()

# Cleaning the text sentences so that punctuation marks, stop words & digits are removed
def clean(doc):
    stop_free = " ".join([i for i in doc.lower().split() if i not in stop])
    punc_free = ''.join(ch for ch in stop_free if ch not in exclude)
    normalized = " ".join(lemma.lemmatize(word) for word in punc_free.split())
    processed = re.sub(r"\d+","",normalized)
    y = processed.split()
    return y

2. Tf-Idf Feature Extraction.

The most popular and widely used word weighing scheme in text mining problems, term frequency and inverse document frequency (tf-idf), is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. The tf-idf value increases proportionally to the number of times a word appears in the document (tf), but is often offset by the frequency of the word in the whole corpus (idf), which helps to adjust for the fact that some words appear more frequently in general. I have explained tf-idf weighing scheme in detail in this blog-post of sentiment analysis application. The following Python code cleanses the text sentences using the definition provided in Section 1. Further, it extracts tf-idf features using scikit-learn library.

print "There are 10 sentences of following three classes on which K-NN classification and K-means clustering"\
         " is performed : \n1. Cricket \n2. Artificial Intelligence \n3. Chemistry"
path = "Sentences.txt"

train_clean_sentences = []
fp = open(path,'r')
for line in fp:
    line = line.strip()
    cleaned = clean(line)
    cleaned = ' '.join(cleaned)
    train_clean_sentences.append(cleaned)

vectorizer = TfidfVectorizer(stop_words='english')
X = vectorizer.fit_transform(train_clean_sentences)

# Creating true labels for 30 training sentences
y_train = np.zeros(30)
y_train[10:20] = 1
y_train[20:30] = 2

3. Training the Classification (K-NN) & Clustering (K-Means) models.

As we have discussed earlier also, Text classification is a supervised learning task, whereas text clustering is an unsupervised task. We are investigating two machine learning algorithms here: K-NN classifier and K-Means clustering.
In k-NN classification, the output is a category membership. A text is classified by a majority vote of its neighbors, with the text being assigned to the class most common among its k nearest neighbors.
The goal of clustering is to determine the intrinsic grouping in a set of unlabeled data (feature vectors). In K-Means clustering, ‘K’ cluster centers are discovered which is centroid of data points belonging to that cluster. A test data (feature-vector) is assigned to that cluster whose centroid is at minimum Euclidean distance from it.

The following Python code snippet trains both the models using scikit-learn library from the tf-idf features extracted in Section 2.

# Clustering the document with KNN classifier
modelknn = KNeighborsClassifier(n_neighbors=5)
modelknn.fit(X,y_train)

# Clustering the training 30 sentences with K-means technique
modelkmeans = KMeans(n_clusters=3, init='k-means++', max_iter=200, n_init=100)
modelkmeans.fit(X)

4. Testing on Unseen Texts.

Once the model has been trained,  we demonstrate the concept of classification and clustering with above conventional methods. We tested it on the following few unseen text sentences.

Chemical compounds are used for preparing bombs based on some reactions.
Cricket is a boring game where the batsman only enjoys the game.
Machine learning is a area of Artificial intelligence.

# Predicting it on test data : Testing Phase
test_sentences = ["Chemical compunds are used for preparing bombs based on some reactions",\
"Cricket is a boring game where the batsman only enjoys the game",\
"Machine learning is a area of Artificial intelligence"]

test_clean_sentence = []
for test in test_sentences:
cleaned_test = clean(test)
cleaned = ' '.join(cleaned_test)
cleaned = re.sub(r"\d+","",cleaned)
test_clean_sentence.append(cleaned)

Test = vectorizer.transform(test_clean_sentence)

true_test_labels = ['Cricket','AI','Chemistry']
predicted_labels_knn = modelknn.predict(Test)
predicted_labels_kmeans = modelkmeans.predict(Test)

print "\nBelow 3 sentences will be predicted against the learned nieghbourhood and learned clusters:\n1. ",\
test_sentences[0],"\n2. ",test_sentences[1],"\n3. ",test_sentences[2]
print "\n-------------------------------PREDICTIONS BY KNN------------------------------------------"
print "\n",test_sentences[0],":",true_test_labels[np.int(predicted_labels_knn[0])],\
"\n",test_sentences[1],":",true_test_labels[np.int(predicted_labels_knn[1])],\
"\n",test_sentences[2],":",true_test_labels[np.int(predicted_labels_knn[2])]

print "\n-------------------------------PREDICTIONS BY K-Means--------------------------------------"
print "\nIndex of Cricket cluster : ",Counter(modelkmeans.labels_[0:10]).most_common(1)[0][0]
print "Index of Artificial Intelligence cluster : ",Counter(modelkmeans.labels_[10:20]).most_common(1)[0][0]
print "Index of Chemistry cluster : ",Counter(modelkmeans.labels_[20:30]).most_common(1)[0][0]

print "\n",test_sentences[0],":",predicted_labels_kmeans[0],\
"\n",test_sentences[1],":",predicted_labels_kmeans[1],\
"\n",test_sentences[2],":",predicted_labels_kmeans[2]

While testing, we simply follow the same steps as those in training. Once the sentences have been cleaned, the tf-idf weighted features are extracted from text sentences using the pre-trained dictionary. The feature vectors are then assigned (predicted) to a category (in case of classification) or to a group (in case of  clustering).

The output snippet of the sequentially combined code from section 1, 2, 3 and 4 is given below:

One can easily observe that all the three sentences are classified as well as clustered in correct categories.

Summary

Hope it was very easy for readers to follow the implementation of text classification and clustering task. If you have gone through it well, you might have understood by now the basic difference between  supervised (K-NN) and unsupervised (K-Means) learning. In supervised learning, the category of the input text can be identified whereas in unsupervised model, only the similar group can be found.

For text classification or clustering tasks, these above posted methods are conventional. Much recently, a lot of methods and text representations using deep learning have been proposed and had provided state-of-the-art results for the same task. Few of the approaches that one can explore after having a basic understanding of this blog-post are:

1. Word2Vec text representation
2. Glove vectors text representation
3. FastText n-gram representation
4. Deep learning techniques for classification (Fully Connected, 1-D CNN, LSTM etc.)

You can get the full python implementation of this blog-post from GitHub link here.

If you liked the post, follow this blog to get updates about upcoming articles. Also, share it so that it can reach out to the readers who can actually gain from this. Please feel free to discuss anything regarding the post. I would love to hear feedback from you.

Happy machine learning

1. If you want to develop large scale AI applications then the hardware configuration would be high-end server based with multi GPU system (mostly for commercial applications).
2. If you want to learn, develop prototypes and experiment with open source data-sets a lot (mostly for personal usage) then a single GPU based system would suffice.
3. For a novice, who has started to get into deep learning, I would suggest to get started with free resources like Google Colab or Kaggle Kernels.

I am not a big fan of cloud based DL as it is expensive if you are using it for personal/research purposes. At times, we need to run lot of experiments in order to conclude or come up with an accurate model. Going for cloud based GPUs may incur huge cost. A comparison between AWS cloud and on-premise DL system has been penned down in last section of this blog-post.

Through this concise blog-post, I will share my personal experience of building a deep learning based system which may help people who want to purchase the same for personal usage. This hardware set up is intended for learning, prototyping and research purposes.

1. Configuration

We will see each component individually which are required for building a deep learning desktop.

Processor:

I purchased Intel Core i7 8700 processor. This processor is 12 hyper-threaded with 6 physical cores. With base frequency of 3.20 GHz and maximum turbo frequency of 4.60 GHz, this one would be a good choice in Intel family. Also, for a multi-GPU based system, one should consider processor like Intel Xeon Processor E5-2630 v4 which is 20 hyper-threaded with 10 physical cores.

It is important to note that Intel Core i7 8700 has 16 PCI Express lanes which would be sufficient for single GPU. Processor and graphic card (GPU) communicate or push data over 1X16 or 2X8 lanes. For dual-GPU system, It is better to have Intel Xeon E5 V4 processor as it has 40 PCI lanes and it can accommodate parallel communications with GPUs. 

For my usage, I went with Intel core I7 8700 processor as I built a single GPU system and had budget constraints. Also, there are cheaper multi-core AMD processors which one can consider but there are lots of packages, libraries or software which are Intel compatible by default (not AMD compatible by default).

GPU:

Nvidia GTX-1080 Ti. Depending on your budget, one can purchase GPU. I would suggest to get at least GTX 1080 (Video RAM 8GB) in order to set up deep learning experiments. I got ZOTAX MINI GTX 1080TI (Video RAM 11 GB).

Motherboard:

I bought “MSI Z 370 PC PRO” which supports Intel 8th gen processors along with 2 Graphics card slots.  The board comes with two PCIe x16 slots, of which one comes with 16 lanes connected to the chipset (processor), while the other uses four lanes. So, it offers the option of installing multiple graphics cards. The extra PCI express lanes accommodates communication with other peripherals like SSD cards.

Cabinet:

Bought “Cooler Master 590 III Black Window” computer case. Pre-installed with front blue LED 120mm x 2 Fan Support. This would be good enough to accommodate for single GPU based system.

SMPS:

SMPS (switched-mode power supply) is an electronic power supply that incorporates a switching regulator to convert electrical power efficiently. 500W SMPS would be sufficient enough to regulate power supply for RAM (10W), GPU(250W), I7 processor (70W), SSD, HDD and other circuits in Motherboard. I purchased “Cooler Master GOLD 750W” SMPS.

RAM:

I bought 32 GB RAM consisting two Corsair RAM 16GB DDR4 3000MHz.

SSD:

One of the important recommendations for deep learning system is solid state drive because it fastens read write operations from disk. One may have to work on experiments where you need to access a million audios/images from disk. Moreover, It is recommended to install operating system on SSD. Also, as obvious, all the working data-sets can be kept in SSD. One can keep the other archived data-sets in HDD.

The two types of SSD are SATA based SSD and PCI-E based SSD NVME. NVMe drives are much faster than SATA SSD because it uses the PCI-E bus which has greater speed. I bought Western Digital 240GB SATA SSD (did not know about SATA vs NVME difference at that time). One can purchase 500GB NVME SSD if budget allows.

Hard-Disk:

Purchased Western Digital 1TB HDD. Again, if budget allows, go for 2 TB. It’s worth it as there are large data-sets out there which occupies HDD quickly.

Monitor:

As per choice you can take one. I bought “L.G 22 INCH MONITOR”. Look out for monitors which can be connected to the system with HDMI cable. You may not want a monitor with only VGA port which has to be connected with VGA-to-HDMI converter in order to connect to the system cabinet.

UPS:

Got “APC UPS 1100VA”. For GPU based system, It is important to take at least 1100 VA which can give backup to 600W devices for 5 minutes. You will have enough time to safely abort the process and shutdown the system.

Finally, do not forget to include sound and Wi-Fi cards in your system.

2. Installations

Once all the installations are done, you may want to install linux operating system (I prefer ubuntu),  nvidia drivers, CUDA, CuDNN and finally the neural network frameworks like tensorflow or pytorch. Additionally, readers can follow these links to set up the deep learning systems from scratch.

1. Ubuntu 18.04 Deep Learning Environment Setup
   2. Up and Running with Ubuntu, Nvidia, Cuda, CuDNN, TensorFlow, and PyTorch
2. Github link

3. Cloud GPU Vs On-Premise GPU Comparison

Here is a comparison of running same task of training a “Deepspeech” BiLSTM model for automatic speech recognition on AWS cloud as well as my personal deep learning system. The data-set used was 260 hours of telephonic conversations and its transcripts from switchboard data-set. For bench-marking purpose, we kept all the parameters of the model same. For example, number of layers, number of hidden nodes in each layer and train, development, test batch size etc.

Performance

The task took approximately 20 minutes on GTX-1080Ti (above specified DL system in blog-post). Below screenshot shows the elapsed time.

The task took approximately 10 minutes on AWS Cloud P2.8xlarge instance with 8 K-80 GPU cards in parallel and 32 vCPUs system. Below screenshot shows the same.

We can conclude that AWS cloud P2.8Xlarge instance with 8 parallel K-80 cards are 2X faster than the on-premise single GPU GTX 1080Ti system. Note that we get only twice the performance even after using 8 GPU cards on AWS cloud.

Cost

The cost of running above AWS instance for 65 hours (up till now) is approximately 500 US dollars (It will be 1000 US dollars if using instance from ‘Asia Mumbai’ region).
The cost of building a DL system specified in this blog-post is 2500 US dollars only. It is including all the components mentioned.

In a months time, the cost incurred on AWS cloud will surpass the whole purchasing cost of DL system. In return, we are only getting twice the performance with AWS cloud instance. Note that life of on-premise system will be a resource for few years. I do concur that cloud would always be a faster option as far as the concern is deployment and setting up the machine.

If you liked the post, follow this blog to get updates about upcoming articles. Also, share it so that it can reach out to the readers who can actually gain from this. Please feel free to discuss anything regarding the post. I would love to hear feedback from you.

Happy deep learning

So, let’s start with understanding and building the model proposed by Facebook AI research in 2015 in this paper “End to End Memory Networks“.

1. Understanding Memory Network Model

To understand the end to end memory network model, let us define few memory representation vectors and its computation first.

Input Memory Representation

The story vectors derived from vocabulary (previous article) are converted into memory vectors of dimension ‘d’ (here, assume embedding_size ‘d’=50) computed by embedding layer. An embedding layer takes story vector (max length 68) as input and generates a fixed ‘d’ dimensional vector for each input story vector. Let us call this embedding ‘m’. Dimension of ‘m’ would be (max_story_length, embedding_size) i.e. (68,50).

Question Memory Representation

The query vector is also embedded (again, via another embedding layer in keras with the same dimensions as ‘d’ (here d=50) to obtain an embedding ‘q’. The embedding layer takes query vector (max length 4) as input and generates an embedding ‘q’ of ‘d’ dimension. Dimension of ‘q’ would be (query_length, embedding_size) i.e. (4,50).

Further, In the embedding space, we compute the probability vector ‘p’ i.e. match between ‘q’ and each memory ‘m’ by taking the inner product followed by a softmax function:

p = Softmax(dot(m,q))

Dimension of ‘p’ would be (max_story_length, query_length) i.e. (68,4)

Output Memory Representation

The story vector are also converted into memory vector of dimension query max length i.e. 4 computed by a separate embedding layer. This embedding layer takes story vector (max length 68) vector as input and generates an embedding of ‘max length=4’ dimension. Let us call this embedding ‘c’. Dimension of ‘c’ would be (max_story_length, query_length) i.e. (68,4).

The output vector or response vector  ‘o’ is then calculated by sum over the transformed embedding ‘c’ , weighted by the probability vector ‘p’ calculated earlier:

o = add(c,p)

Dimension of ‘o’ would be (max_story_length, query_length) i.e. (68,4)

Final Prediction

The output or response vector ‘o’ and the input vector ‘m’ are concatenated to form a result vector ‘r’ which is further passed to LSTM layer and finally to output Dense layer of vocabulary size which gives softmax probability of each word in the vocab. We consider the word predicted with greater accuracy as answer word. Dimension after concatenation would be (query_length,  embedding_size + max_story_length) i.e. (4, 118).

answer = Softmax(Dense(LSTM(concat(o,q))))

or we can write it as

answer = Softmax(W(concat(o,q)))

A below pictorial diagram showcases the above procedure of building the model.

2. Building the Model

Hope the above section gave pretty clear illustration on prediction of answer word through the memory network model. Let us quickly build an end to end single layer memory network model.

Importing Libraries

import IPython
import matplotlib.pyplot as plt
import pandas as pd

import keras
from keras.models import Sequential, Model
from keras.layers.embeddings import Embedding
from keras.layers import Permute, dot, add, concatenate
from keras.layers import LSTM, Dense, Dropout, Input, Activation

Config file (Parameters)

# number of epochs to run
train_epochs = 100
# Training batch size
batch_size = 32
# Hidden embedding size
embed_size = 50
# number of nodes in LSTM layer
lstm_size = 64
# dropout rate
dropout_rate = 0.30

Defining the Model

# placeholders
input_sequence = Input((story_maxlen,))
question = Input((query_maxlen,))

print('Input sequence:', input_sequence)
print('Question:', question)

# encoders
# embed the input sequence into a sequence of vectors
input_encoder_m = Sequential()
input_encoder_m.add(Embedding(input_dim=vocab_size,
                              output_dim=embed_size))
input_encoder_m.add(Dropout(dropout_rate))
# output: (samples, story_maxlen, embedding_dim)

# embed the input into a sequence of vectors of size query_maxlen
input_encoder_c = Sequential()
input_encoder_c.add(Embedding(input_dim=vocab_size,
                              output_dim=query_maxlen))
input_encoder_c.add(Dropout(dropout_rate))
# output: (samples, story_maxlen, query_maxlen)

# embed the question into a sequence of vectors
question_encoder = Sequential()
question_encoder.add(Embedding(input_dim=vocab_size,
                               output_dim=embed_size,
                               input_length=query_maxlen))
question_encoder.add(Dropout(dropout_rate))
# output: (samples, query_maxlen, embedding_dim)

# encode input sequence and questions (which are indices)
# to sequences of dense vectors
input_encoded_m = input_encoder_m(input_sequence)
print('Input encoded m', input_encoded_m)
input_encoded_c = input_encoder_c(input_sequence)
print('Input encoded c', input_encoded_c)
question_encoded = question_encoder(question)
print('Question encoded', question_encoded)

# compute a 'match' between the first input vector sequence
# and the question vector sequence
# shape: `(samples, story_maxlen, query_maxlen)
match = dot([input_encoded_m, question_encoded], axes=-1, normalize=False)
print(match.shape)
match = Activation('softmax')(match)
print('Match shape', match)

# add the match matrix with the second input vector sequence
response = add([match, input_encoded_c])  # (samples, story_maxlen, query_maxlen)
response = Permute((2, 1))(response)  # (samples, query_maxlen, story_maxlen)
print('Response shape', response)

# concatenate the response vector with the question vector sequence
answer = concatenate([response, question_encoded])
print('Answer shape', answer)

answer = LSTM(lstm_size)(answer)  # Generate tensors of shape 32
answer = Dropout(dropout_rate)(answer)
answer = Dense(vocab_size)(answer)  # (samples, vocab_size)
# we output a probability distribution over the vocabulary
answer = Activation('softmax')(answer)

In the above code snippet, you can see we have two inputs : input_sequence (story) and question (query). Also, we have ‘answer’ as a final model which combines response model and encoded query as input and predicts the answer word.

3. Training the Model

The below codes complies the model and train it using input features vectors as well as labels extracted in the first part of this blog-post series.

# build the final model
model = Model([input_sequence, question], answer)
model.compile(optimizer='rmsprop', loss='categorical_crossentropy',
              metrics=['accuracy'])

print(model.summary())

# start training the model
model.fit([inputs_train, queries_train],
         answers_train, batch_size, train_epochs,
         callbacks=[TrainingVisualizer()],
         validation_data=([inputs_test, queries_test], answers_test))

# save model
model.save('model.h5')

4. Visualization

The below python function shows visualization of training and validation categorical cross entropy loss as well as  train and validation accuracy over iterations. It is a callback function while model training.

class TrainingVisualizer(keras.callbacks.History):
    def on_epoch_end(self, epoch, logs={}):
        super().on_epoch_end(epoch, logs)
        IPython.display.clear_output(wait=True)
        pd.DataFrame({key: value for key, value in self.history.items() if key.endswith('loss')}).plot()
        axes = pd.DataFrame({key: value for key, value in self.history.items() if key.endswith('acc')}).plot()
        axes.set_ylim([0, 1])
        plt.show()

5. Tests Results

I hope you have enjoyed the model building process till now. Let us check few of the test stories and see how the model performs in predicting the right answer to the query.

for i in range(0,10):
    current_inp = test_stories[i]
    current_story, current_query, current_answer = vectorize_stories([current_inp], word_idx, story_maxlen, query_maxlen)
    current_prediction = model.predict([current_story, current_query])
    current_prediction = idx_word[np.argmax(current_prediction)]
    print(' '.join(current_inp[0]), ' '.join(current_inp[1]), '| Prediction:', current_prediction, '| Ground Truth:', current_inp[2])
    print("--------------------------------------------------------------")

John travelled to the hallway . Mary journeyed to the bathroom . Where is John ? | Prediction: hallway | Ground Truth: hallway
---------------------------------------------------------------------
John travelled to the hallway . Mary journeyed to the bathroom . Daniel went back to the bathroom . John moved to the bedroom . Where is Mary ? | Prediction: bathroom | Ground Truth: bathroom
---------------------------------------------------------------------
John travelled to the hallway . Mary journeyed to the bathroom . Daniel went back to the bathroom . John moved to the bedroom . John went to the hallway . Sandra journeyed to the kitchen . Where is Sandra ? | Prediction: kitchen | Ground Truth: kitchen
---------------------------------------------------------------------
John travelled to the hallway . Mary journeyed to the bathroom . Daniel went back to the bathroom . John moved to the bedroom . John went to the hallway . Sandra journeyed to the kitchen . Sandra travelled to the hallway . John went to the garden . Where is Sandra ? | Prediction: hallway | Ground Truth: hallway
---------------------------------------------------------------------
John travelled to the hallway . Mary journeyed to the bathroom . Daniel went back to the bathroom . John moved to the bedroom . John went to the hallway . Sandra journeyed to the kitchen . Sandra travelled to the hallway . John went to the garden . Sandra went back to the bathroom . Sandra moved to the kitchen . Where is Sandra ? | Prediction: kitchen | Ground Truth: kitchen
---------------------------------------------------------------------
Sandra travelled to the kitchen . Sandra travelled to the hallway . Where is Sandra ? | Prediction: hallway | Ground Truth: hallway
---------------------------------------------------------------------
Sandra travelled to the kitchen . Sandra travelled to the hallway . Mary went to the bathroom . Sandra moved to the garden . Where is Sandra ? | Prediction: garden | Ground Truth: garden
---------------------------------------------------------------------
Sandra travelled to the kitchen . Sandra travelled to the hallway . Mary went to the bathroom . Sandra moved to the garden . Sandra travelled to the office . Daniel journeyed to the hallway . Where is Daniel ? | Prediction: hallway | Ground Truth: hallway
---------------------------------------------------------------------
Sandra travelled to the kitchen . Sandra travelled to the hallway . Mary went to the bathroom . Sandra moved to the garden . Sandra travelled to the office . Daniel journeyed to the hallway . Daniel journeyed to the office . John moved to the hallway . Where is Sandra ? | Prediction: office | Ground Truth: office
---------------------------------------------------------------------
Sandra travelled to the kitchen . Sandra travelled to the hallway . Mary went to the bathroom . Sandra moved to the garden . Sandra travelled to the office . Daniel journeyed to the hallway . Daniel journeyed to the office . John moved to the hallway . John travelled to the bathroom . John journeyed to the office . Where is Daniel ? | Prediction: office | Ground Truth: office
---------------------------------------------------------------------

Reference

[1] End to End Memory Networks
[2] Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
[3] Example code : bAbI_rnn.py from keras-team/keras GitHub repository

Final Thoughts

Hope it was easy to follow this blog-post series on developing question and answering system for a toy task on bAbI data-set. If you are totally unaware of neural networks or Q&A systems then I would suggest to first read more on question answering system from the abundant material available online. Before signing off,  I would like to put few more thoughts on this post and for future:

• The data-set is a synthetic data-set with limited data (10k stories only) and has a very limited vocabulary of 22 unique words.
• It would be great to check the performance on reasoning tasks where prediction is based on multiple facts.
• A more generic Q&A system should allow machine reading comprehension (MRC) abilities where system should be able to predict span of words (sentence) from reading a paragraph (based on multiple supporting facts).
• Readers who are encouraged to explore Q&A systems more can check an open source SQUAD2.0 data-set which is much more rigorous task and would require further research.

You can get the full python implementation (jupyter notebook) from GitHub link here.

Intent of this blog-post series was to perform experiments and demonstrate a fair idea of question answering system. Personally, I am more interested in MRC application where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable. Hopefully, I will be writing solution to MRC task on Squad data-set in near future. Stay tuned !!.

If you liked the post, follow this blog to get updates about upcoming articles. Also, share it so that it can reach out to the readers who can actually gain from this. Please feel free to discuss anything regarding the post. I would love to hear feedback from you.

Happy deep learning