Machine Learning with Python: Meeting TF-IDF for Text Mining

3Hi all,

This month I was studying about information retrieval and text mining, specially how to convert the textual representation of information into a Vector Space Model (VSM).  The VSM is an algebraic model representing the importance of a term (tf-idf) or even the absence or presence (Bag of Words) of it in a document. I'd like to mention the excellent post from the researcher Christian Perone at his blog Pyevolve about Machine learning and Text Mining with TF-IDF, a great post to read.

I decided in this post to be shorter and give some examples using Python . I expect at the end of this post you feel confortamble to use tf-idf at your tasks handling with text mining.

By the way, I extremely recommend you to check the scikit.learn machine learning toolkit. There is a whole package to work with text classification, including TF-IDF with Python!

What is TF-IDF ?

Term Frequency - Inverse Document Frequency is a weighting scheme that is commonly used in information retrieval tasks. The goal is to model each document into a vector space, ignoring the exact ordering of the words in the document while retaining information about the occurrences of each word.

It is composed by two terms: one first computes the normalized Term Frequency, which is the number of times a word appears in a documnet, divided by the total number of words in that document. Then, the second term is the Inverse Document Frequency, which is computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the term ti appears. Or, in symbols:

and 

The TF-IDF gives how important is a word to a document in a collection, since it takes in consideration not only the isolated term but also the term within the document collection. The intuition is that a term that occurs frequently in many documents is not a good discriminator ( why emphasize a term which is almost present in the entire corpus of your documents ?)  So it will scale down the frequent terms while scaling up the rare terms; for instance, a term that occurs 10 times more than another isn't 10 times more important thant it.

For computing the TF-IDF weights for each document in the corpus, it is required in the corpus a series of steps:  1) Tokenize the corpus  2)  Model the Vector  Space  and 3) Compute the TF-IDF weight for each document in the corpus.

Let's going through each step:


Tokenization

First we need to tokenize the text. To do this, we can use the NLTK library which is a collection of natural language processing algorithms written in Python. The process of tokenizing the documents in the corpous is a two steps:  First the text is splint into sentences, and then the sentences are split into the individual words. It is important to notice that there are several words that are not relevant, that is, terms like "the, is, at, on", etc...  aren't going to help us, so in the information extraction, we ignore them. Those words are commonly called stop words and they are present in almost all documents, so it is not relevant for us. In portuguese we also have those stop words such as (a, os , as , os, um , umas, que, etc.)

So considering our example below:

We will tokenize this collection of documents and represent them as vectors (rows) of a matrix with |D| x F shape, where |D|  is the cardinality of the document space, or how many documents we have and the F is the number of features, in our example it is represented by the vocabulary size.

So the matrix representation of our vectors above is:

As you have noticed, these matrices representing the term frequencies (tf) tend to be very sparse (lots of  zero-elements),  so you will usually see the representation of these matrices as sparse matrices. The code shown below will tokenize each document in the corpus and compute the term frequencies.

	auth = OAuthHandler(CLIENT_ID, CLIENT_SECRET, CALLBACK)
	auth.set_access_token(ACCESS_TOKEN)
	api = API(auth)
	venue = api.venues(id='4bd47eeb5631c9b69672a230')
	stopwords = nltk.corpus.stopwords.words('portuguese')
	tokenizer = RegexpTokenizer("[\w’]+", flags=re.UNICODE)
	def freq(word, tokens):
	return tokens.count(word)
	#Compute the frequency for each term.
	vocabulary = []
	docs = {}
	all_tips = []
	for tip in (venue.tips()):
	tokens = tokenizer.tokenize(tip.text)
	bi_tokens = bigrams(tokens)
	tri_tokens = trigrams(tokens)
	tokens = [token.lower() for token in tokens if len(token) > 2]
	tokens = [token for token in tokens if token not in stopwords]
	bi_tokens = [' '.join(token).lower() for token in bi_tokens]
	bi_tokens = [token for token in bi_tokens if token not in stopwords]
	tri_tokens = [' '.join(token).lower() for token in tri_tokens]
	tri_tokens = [token for token in tri_tokens if token not in stopwords]
	final_tokens = []
	final_tokens.extend(tokens)
	final_tokens.extend(bi_tokens)
	final_tokens.extend(tri_tokens)
	docs[tip.text] = {'freq': {}}
	for token in final_tokens:
	docs[tip.text]['freq'][token] = freq(token, final_tokens)
	print docs

view raw term_frequency.py hosted with ❤ by GitHub

Model the Vector Space

Now that each of the documents in the corpus has been tokenized, the next step is to compute the document frequency quantity, that is, for each term, how many documents that term appears in. Before going to IDF, it is important to normalize the term-frequencies. Why ?  Imagine that we have a repeated term in document with porpuse of improving its ranking on an Information Retrieval System or even create a bias torwards long documents, making them look more important than they are just because of the high frequency of the term in the document. By normalizing the TF vector we can overcome this problem.
The code.

	auth = OAuthHandler(CLIENT_ID, CLIENT_SECRET, CALLBACK)
	auth.set_access_token(ACCESS_TOKEN)
	api = API(auth)
	venue = api.venues(id='4bd47eeb5631c9b69672a230')
	stopwords = nltk.corpus.stopwords.words('portuguese')
	tokenizer = RegexpTokenizer("[\w’]+", flags=re.UNICODE)
	def freq(word, tokens):
	return tokens.count(word)
	def word_count(tokens):
	return len(tokens)
	def tf(word, tokens):
	return (freq(word, tokens) / float(word_count(tokens)))
	#Compute the frequency for each term.
	vocabulary = []
	docs = {}
	all_tips = []
	for tip in (venue.tips()):
	tokens = tokenizer.tokenize(tip.text)
	bi_tokens = bigrams(tokens)
	tri_tokens = trigrams(tokens)
	tokens = [token.lower() for token in tokens if len(token) > 2]
	tokens = [token for token in tokens if token not in stopwords]
	bi_tokens = [' '.join(token).lower() for token in bi_tokens]
	bi_tokens = [token for token in bi_tokens if token not in stopwords]
	tri_tokens = [' '.join(token).lower() for token in tri_tokens]
	tri_tokens = [token for token in tri_tokens if token not in stopwords]
	final_tokens = []
	final_tokens.extend(tokens)
	final_tokens.extend(bi_tokens)
	final_tokens.extend(tri_tokens)
	docs[tip.text] = {'freq': {}, 'tf': {}}
	for token in final_tokens:
	#The Frequency computed for each tip
	docs[tip.text]['freq'][token] = freq(token, final_tokens)
	#The True-Frequency (Normalized Frequency)
	docs[tip.text]['tf'][token] = tf(token, final_tokens)
	print docs

view raw term_frequency_normalization.py hosted with ❤ by GitHub

Compute the TF-IDF

Now that you saw how the vector normalization was applied, we will now have to compute the second term of tf-idf: the inverse document frequency. The code is provided below:

	auth = OAuthHandler(CLIENT_ID, CLIENT_SECRET, CALLBACK)
	auth.set_access_token(ACCESS_TOKEN)
	api = API(auth)
	venue = api.venues(id='4bd47eeb5631c9b69672a230')
	stopwords = nltk.corpus.stopwords.words('portuguese')
	tokenizer = RegexpTokenizer("[\w’]+", flags=re.UNICODE)
	def freq(word, doc):
	return doc.count(word)
	def word_count(doc):
	return len(doc)
	def tf(word, doc):
	return (freq(word, doc) / float(word_count(doc)))
	def num_docs_containing(word, list_of_docs):
	count = 0
	for document in list_of_docs:
	if freq(word, document) > 0:
	count += 1
	return 1 + count
	def idf(word, list_of_docs):
	return math.log(len(list_of_docs) /
	float(num_docs_containing(word, list_of_docs)))
	#Compute the frequency for each term.
	vocabulary = []
	docs = {}
	all_tips = []
	for tip in (venue.tips()):
	tokens = tokenizer.tokenize(tip.text)
	bi_tokens = bigrams(tokens)
	tri_tokens = trigrams(tokens)
	tokens = [token.lower() for token in tokens if len(token) > 2]
	tokens = [token for token in tokens if token not in stopwords]
	bi_tokens = [' '.join(token).lower() for token in bi_tokens]
	bi_tokens = [token for token in bi_tokens if token not in stopwords]
	tri_tokens = [' '.join(token).lower() for token in tri_tokens]
	tri_tokens = [token for token in tri_tokens if token not in stopwords]
	final_tokens = []
	final_tokens.extend(tokens)
	final_tokens.extend(bi_tokens)
	final_tokens.extend(tri_tokens)
	docs[tip.text] = {'freq': {}, 'tf': {}, 'idf': {}}
	for token in final_tokens:
	#The frequency computed for each tip
	docs[tip.text]['freq'][token] = freq(token, final_tokens)
	#The term-frequency (Normalized Frequency)
	docs[tip.text]['tf'][token] = tf(token, final_tokens)
	vocabulary.append(final_tokens)
	for doc in docs:
	for token in docs[doc]['tf']:
	#The Inverse-Document-Frequency
	docs[doc]['idf'][token] = idf(token, vocabulary)
	print docs

view raw idf.py hosted with ❤ by GitHub

The TF-IDF is the product between the TF and IDF.  So a high weight of the tf-idf  is reached when you have a high term frequency (tf) in the given document and low document frequency of the term in the whole collection. Now let's see the tf-idf computed for each term present in the vector space.

The code.

	auth = OAuthHandler(CLIENT_ID, CLIENT_SECRET, CALLBACK)
	auth.set_access_token(ACCESS_TOKEN)
	api = API(auth)
	venue = api.venues(id='4bd47eeb5631c9b69672a230')
	stopwords = nltk.corpus.stopwords.words('portuguese')
	tokenizer = RegexpTokenizer("[\w’]+", flags=re.UNICODE)
	def freq(word, doc):
	return doc.count(word)
	def word_count(doc):
	return len(doc)
	def tf(word, doc):
	return (freq(word, doc) / float(word_count(doc)))
	def num_docs_containing(word, list_of_docs):
	count = 0
	for document in list_of_docs:
	if freq(word, document) > 0:
	count += 1
	return 1 + count
	def idf(word, list_of_docs):
	return math.log(len(list_of_docs) /
	float(num_docs_containing(word, list_of_docs)))
	def tf_idf(word, doc, list_of_docs):
	return (tf(word, doc) * idf(word, list_of_docs))
	#Compute the frequency for each term.
	vocabulary = []
	docs = {}
	all_tips = []
	for tip in (venue.tips()):
	tokens = tokenizer.tokenize(tip.text)
	bi_tokens = bigrams(tokens)
	tri_tokens = trigrams(tokens)
	tokens = [token.lower() for token in tokens if len(token) > 2]
	tokens = [token for token in tokens if token not in stopwords]
	bi_tokens = [' '.join(token).lower() for token in bi_tokens]
	bi_tokens = [token for token in bi_tokens if token not in stopwords]
	tri_tokens = [' '.join(token).lower() for token in tri_tokens]
	tri_tokens = [token for token in tri_tokens if token not in stopwords]
	final_tokens = []
	final_tokens.extend(tokens)
	final_tokens.extend(bi_tokens)
	final_tokens.extend(tri_tokens)
	docs[tip.text] = {'freq': {}, 'tf': {}, 'idf': {},
	'tf-idf': {}, 'tokens': []}
	for token in final_tokens:
	#The frequency computed for each tip
	docs[tip.text]['freq'][token] = freq(token, final_tokens)
	#The term-frequency (Normalized Frequency)
	docs[tip.text]['tf'][token] = tf(token, final_tokens)
	docs[tip.text]['tokens'] = final_tokens
	vocabulary.append(final_tokens)
	for doc in docs:
	for token in docs[doc]['tf']:
	#The Inverse-Document-Frequency
	docs[doc]['idf'][token] = idf(token, vocabulary)
	#The tf-idf
	docs[doc]['tf-idf'][token] = tf_idf(token, docs[doc]['tokens'], vocabulary)
	#Now let's find out the most relevant words by tf-idf.
	words = {}
	for doc in docs:
	for token in docs[doc]['tf-idf']:
	if token not in words:
	words[token] = docs[doc]['tf-idf'][token]
	else:
	if docs[doc]['tf-idf'][token] > words[token]:
	words[token] = docs[doc]['tf-idf'][token]
	for item in sorted(words.items(), key=lambda x: x[1], reverse=True):
	print "%f <= %s" % (item[1], item[0])

view raw tf_idf.py hosted with ❤ by GitHub

Putting everything together, the following code will compute the TF-IDF weights for each document. And the result matrix it will be:

A row of this matrix would be:

I ommited the zero-values elements of the row.

If we would decide to check the most relevant words for this place, by using the tf-idf I could see that the place has a nice hot chocolate drink (0.420955 <= chocolate quente ótimo), the soft drink nega maluca is also delicious (0.315716 - nega maluca uma delicia),  its Cheese bun is also quite good (0.252573 - pao de queijo muito bom).

And that is how we comput  our  matrix.  You can take a look at this link and this one to know how to use it with GenSim and Scikit.Learn respectively.

That's all,  I hope that  you enjoyed this article and help more people to know how to implement the tf-idf weight to mine your collection of texts.  Feel free to comment and make suggestions.

The source code of this example is also available.

Regards,

Marcel Caraciolo