Scipy Conference 2011 and my participation!

Hi all,

Last week I was at the Scipy 2011 Conference at Austin, Tx. My first international conference as also my first lecture international! The Scipy Conference is an annual meeting for scientific computing developers and researchers that use python scientific packages in their research or work.  It was a great opportunity for meeting new python developers, know more about what's happening in scientific python nowadays and to learn about Scipy, Numpy and Matplotlib, considered the standard libraries for developers who wants start to develop in the scientific world.

At the first day of the conference, I had the opportunity to learn more about Numpy, a widely used library for numerical computations in Python as also learn more about the Scikit-learn framework, a great open-source toolkit for machine learning developers written in Python, Numpy and Scipy.  

You can access both tutorials available here at the Scipy Conference Tutorials WebPage.  Numpy is an amazing library, and what I learned I started already applied at the library I am currently working on called Crab for building recommender systems.   The Scikit-learn is also an interesting framework written in Scipy, Numpy and Matplotlib with several machine learning techniques and has as one main features the easy-to-use interface with lots of examples and tutorials for starters and beginners in machine learning.  It works so smoothly that I decided to use it as dependency of the Crab framework.

The second day started with more advanced tutorials, specially on Global Arrays with Numpy for  High performance computation. A quite powerful effort in this feature and I believe that soon will be added to the Numpy core. 

The another tutorial was about an introduction to Traits, Matplotlib and Chaco - great tools for creating nice user interfaces and plotting charts. One of the best parts of this tutorial was easily to create nice interfaces and animated plottings with a few lines of code.  Take a look of what you can do here or even see a real-time animated plotting with Matplotlib.

Traits and Chaco are part of the EPD package developed by the company Enthought, whose one of the co-founders is one of the main developers and founders of Numpy! Yeah :D Those frameworks allow easily create nice interfaces only using models concepts. If you want to learn more, please check out the tutorials as the official website about how to download, install and use it.

Another keynote interesting was about the Ipython, the incremented shell for scientific Python developers. What amazed me was when he showed the matplotlib embedded at the shell instead of opening a new window! The work around the Ipython has been fantastic, with several features for python developers! I extremely recommend!

The rest of the conference was dedicated to keynotes and talks about currently works on data science, core technologies and data mining with Python, Scipy , Numpy and related libraries.  I had the opportunity of giving the lecture - Crab - A Python Framework for Building Recommender Systems written by me, Bruno Melo and Ricardo Caspirro, actually the main contributors for this work.  The idea is to provide for python developers a recommender toolkit so they can easily create, test and deploy recommender engines with simple interfaces written with the scientific python packages such as Numpy, Scipy and Matplotlib.

Introduction to Crab - Python Framework for Building Recommender Systems

View more presentations from Marcel Caraciolo

You can check out my slides at the Scipy Conference here.

The project is currently being developed by the non-profitable organization called Muriçoca, that we decided  to create to manage and develop the Crab Framework. 

One of the best keynotes was the presentation of Hilary Manson, the Data Scientist at bit.ly.  She gave a funny lecture about her work and the current challenges with handling with large data sets and lots of URL-shortening happening at the backend of Bit.ly. It is quite amazing the amount of data and what you can do and extract useful information from all this data.

At least, I decided also to give a lighting talk about Mining the Scipy Lectures. A simple lecture to show what you can do with the data from the Scipy Conference Schedule and play with it. I used some NLP techniques and clustered based on the most frequent topics to check how was distributed the lectures at Scipy based on the keywords from their titles.  To visualize I used the Graph Visualization tool Ubigraph to show in 3D the clusters generated (by the way I used the K-means algorithm to cluster). 

Mining Scipy Lectures

View more presentations from Marcel Caraciolo

The slides are also available here and the source code here.

3D Lectures Clusters

Soon I will release the PDF with the article submitted as also the video with both keynotes that I presented.  It was an amazing conference at Austin, making new friends and lots of new partners! :D I expect to be there next year, absolutely!  One of my goals this year also is to prepare a scientific computing course using Python, wait for more information soon here at the blog (it will include matplotlib, scipy and numpy)!

Cheers,

Marcel Caraciolo

Recommender Engines and Data mining with Python at PythonBrasil Conference

Hi all,

Last week I was at an important event in Curitiba called Python Brasil. It is a annual event where it joins several brazilian developers to discuss about technology and of course about the programming language Python.

I also had the opportunity  to lecture three presentations about several topics of my interest.   The official presentation was about recommendation engines with Python.  This work shows how developers could use python in developing recommendation engines with several examples and explaining the main concepts behind this subject.  The best part was the demos where I used real data from the web such as Twitter, to suggest users that are similar to me among the PythonBrasil followers.  The other example is based on collective buying, which i crawled some popular brazilian web sites and gathered real offers at Curitiba. The main idea is to recommend new offers according to my interests and what people similar to me also liked. It is the classical example of the collaborative filtering, commonly used at several e-commerces today including Amazon.

The presentation was great with lots of feedback.  If you want to take a look at the slides (it is on portuguese) please take a look here:

Python em Sistemas de Recomendação: A Cobra é Inteligente!

View more presentations from Marcel Caraciolo.

The main contribution of this work is a new library for building recommendation engines in Python language called Crab.  I've decided with some colleagues to develop this library in order to be a powerful tool for developers to use python as the main language to build and use classical recommender algorithms in their applications. Besides it is extensible so developers can add easily new algorithms to the engine. There is also a easy API so users can plug with their web apps running in different platforms such as Django, AppEngine, Web2Py, etc. 

If you want to collaborate or interest about this subject please feel free to join us at this work. The project is hosted at GitHub with the link:

http://github.com/marcelcaraciolo/crab

My second presentation was a lighting talk. What's that ?  There is a extra category of presentations in PythonBrasil where you have 5 minutes to speak about any topic you want.  The one rule is 5 minutes, no more!   I was challenged so I one day before the presentation to develop a web crawler in order to scrap all the lectures submitted and approved at PythonBrasil conference.  With all this data in my hands I've decided to make an analysis to answer three questions in my mind:

a) Which are the main and  frequent topics showed at the keynotes at Python Brazil ?

b) Based on this information, how we could organize the speakers based on those topics ? That is group speakers with similar topics. A classical problem of clustering.

c) What information we could also extract such as level of expertise of the lectures, total time spent in the lectures, etc.

For all those questions I was seeking to answer I decided to use Python, Matplotlib and Ubigraph (A 3D Visualization tool for graphs).   It was really interesting because I really could find some groups based on similar interests.  The main subjects was Entrepreneurship, Hardware, Web, Design Patterns, Data Mining, Django and Artificial Intelligence.

With those subjects I could now group the speakers using a simple clustering algorithm such as K-means and organize them based what their topics were. I've recorded a simple video to present the result using the tool Ubigraph. Take a look:

The presentation in portuguese  you can see here:

Análise das Palestras do PythonBrasil [6]

View more presentations from Marcel Caraciolo.

In the end I think the event was awesome some great keynotes and of course lots of new contacts at my network.  I have to say it is a great opportunity to meet great people and share ideas and technology!

Next year it will be in São Paulo, Brazil. I expect to be there !

Best regards,

Marcel Caraciolo

Playing with the Twitter Data [Final]: Clustering/Grouping my friends with Python!

Hi all,

One of the problems I'm now facing with Twitter is how organize my friends (the users that i follow) in a way that I could easily read their posts based on similar interests or keywords. 

The idea is identify common groups of users and look at a segmentation based on user biography keywords. That biography information would allow us to segment Twitter users in groups of similar interests, professions and qualities.

I could do this manually checking each user and their last statuses and organize them in lists in accordance to their history data on Twitter. But for me, that I have more than 100 friends, it might be painful to check and place it one by one. 

So far we have seen various examples of using analytics to gain insights from Twitter. Using cluster analysis is one of my favorites.  By doing cluster analysis, it will be possible to reveal similar Twitter users and as result the possibility of creating Twitter Lists based on the groups generated by those clustering algorithms.  

Using text mining and some machine learning techniques, I'll present a simple tool to get all your Twitter Friends and organize them in lists segmented by the similar behavior. 

Step 1: Getting the Twitter User Data

In this part, i'll show the code written in Python for getting the Twitter User data. I used the python-twitter library with some custom modifications, allowing to download the Twitter user data and his last statuses/posts. In order to get my friends statuses, I've developed some functions and custom threads, accelerating the process of downloading the user profiles and dumping them into data files.

Take a look into the code provided in TwitterCollector.py. Due to rate limits (150 requests per hour) from Twitter, depending on how many friends you have, it will be not possible to fetch all of them. So for this article, i'll be using a subset of 100 friends randomly picked from my Twitter user profile ('marcelcaraciolo').  

The number of statuses selected will be up to the 200 last statuses from each user. I've chosen this number, because i thought it was a good number to summarize the user behavior on Twitter. But you can change this number if you desire, just be aware that the script may fail during the process because of the request limits for Twitter Web Service.

The Twitter Collector is really simple and effective. There is a pool of threads  (TwitterCollector) where each one grabs the statuses of the user, and then places it into a queue. This queue is repository for holding the statuses that will be parsed. The parsing involves the extraction of the words for what the user has written at his posts and remove the stop words (using some natural language techniques). Another pool of threads (TwitterMiner) joins on this queue, and is responsible for do the parsing work on the statuses. 

After the TwitterMiner finishes his job, it places it into another queue, where it will be dumped into a data file, in this case handled by the module 'pickle'. The module pickle is very efficient on storing and retrieving objects (serialization) into data files.

Step 2: Building the user profiles


Now let's start building the user profiles. In this step we'll build the user profile for each user fetched in the previous step. The user profile is summarized by the statuses collected, where for each user there's a table of word frequencies. This table will allow to check if there are groups of users that frequently write about similar subjects or write in similar styles (e.g. english or portuguese languages).

For instance, consider three users: me ('marcelcaraciolo') that writes more about python and data mining,  'symbian' writes more about  mobile and symbian stuff and the user 'parties' that write about beer and parties around the world. It will happen that after a clustering analysis it may be possible that me and symbian will be in the same group where 'parties' will be placed at another group due to the dissimilarity in behavior between us on Twitter.

The code provided on TwitterOrganizer.py will be responsible to open the data file with the statuses collected in the first step and build the user profiles by creating a table for each user with the word frequencies.

Step 3: Clustering Analysis

Now it's time to discover the groups of users that exhibit the same behavior. Until now i've presented some techniques for clustering analysis like Hierarchical clustering and MultiDimensional Scaling. But both of them have a couple of disadvantages specially on visualization and computational processing.  An alternative method of clustering is K-means clustering. The main difference from Hierarchical Clustering is the number of distinct clusters which is told in advance.  

Let's talk a little bit about  How does K-means work? 

The idea of K-means clustering is to determine the size of the clusters based on the structure of the data. It begins with k  randomly placed centroids (points in space that represent the center of the cluster), and assigns every item to the nearest one. After the assignment, the centroids are moved to the average location of all the nodes assigned to them, and the assignments are redone. Those steps repeats until the assignments stop changing.  The result is a set of clusters within the ranges of each of the variables.  The number of iterations it takes to produce the final result is quite small compared to hierarchical clustering.  It's important to notice that since the k-means uses random centroids to start with, the order of the results returned will be different most of the times. This happens because of the initial locations of the centroids. The same happens with the number of clusters (k) that may affect the results.

You can see more about k-means process here at this link.

I've run a couple of times the algorithm, and i selected one of the partitions generated by the k-means algorithm with the Twitter user profiles generated at the step 02. The k selected on this example is 15. For presenting the results, i've decided to make the things more interesting. I've used the ubigraph framework, a python library for displaying graphs and networks at 3D.  As you can see at the video below, the clusters are generated during the k-means process. Each node represents a Twitter user, and each cluster is differentiated from each other by the color. 

Grouping Twitter Users with Machine Learning Algorithms from marcel Caraciolo on Vimeo.

After running the graph, i've decided to play more with the Twitter data. In the end of the video playback, you can see that I'm clicking on some cluster centroids (represented by the non-labeled cubes). You can see that it shows the most frequently words that appears for each group. It's notable that they're are joined exactly because of the frequency of those words at their user profiles. It will be useful for the last step of this article: Creating the Twitter Lists.

Step 4: Creating the Twitter Lists

Now that i have my friends grouped by similar subjects posted on Twitter, it's time to make this data clustering useful. I've decided to create Twitter Lists, which is a new feature launched from Twitter recently. The idea is creating group of users whose the owner can follow and so other users if they are interested on the topics discussed on what the members of the list posted. It's a handful tool for organizing your friends in order to read their statuses based on what they post. For instance, my friends that talk about python i could group them altogether in a list with the name 'PythonUsers'.

Based on the clustering algorithm, i'll create the lists with members shown at each cluster. Since, the python-twitter library doesn't have at its last release support with Lists API. I've decided to create one on my own. The TwitterListAPI is a simple python wrapper for the Twitter API handling some operations with lists like: creating, updating and removing lists and adding/removing users from the list. You can check the implementation in the module twitterList.py.

You can run it after running the clustering algorithm. The methods readFile() and readFiles() are responsible for reading the files generated at the result of the last step. They contain the name of the new list that you want create, as also the description of the list, which is optional and at least the ids of the user that you would like to add to the list. If you don't have the userID, don't worry you can use the method GetUserID() to get this information for each username. You may also want edit those files at your preference in order to remove or add users, edit the list name or description.

I've run with my twitter friends, and you can see at the figure below one of the lists created after the clustering algorithm.

Conclusion

So, this is my initial attempt in order to manage my Twitter Friends and also playing with Twitter Data by clustering the users based on their statuses. It uses a initial study of text mining and a machine learning technique called K-means, clustering algorithm, for group the user profiles, in this case summarized by user statuses and word frequencies. I believe there's more work to do, specially about running other clustering algorithms and distance metrics to compare the results. I think that the K (number of the clusters) should be found automatically in a way to balance the trade-off: The number of lists x the number of friends. A lot of friends with many non-related topics in a same group could not be  useful as lists with a few members it's not necessary. The best value for 'k' is a subject for future studies. Other good result of this work is the use of 3D techniques to show the clustering process and also the new API for handling with Twitter Lists.  At least, the pre-processing of the statuses will be expanded in order to improve the identification of the topics and reduce the number of topics - variables (dimensionality), a issue for machine learning techniques due to heavy processing and time costing.

You can download all the code used at this article here. 

That's all,

Special thanks for Luciana Nunes and Ricardo Caspirro for the support during this work!

Marcel Pinheiro Caraciolo