Google Group Members Evolution Visualization using Gource: Our Python Local User Group Analyzed!

Hi all,

Yesterday I lectured a keynote introducing Python Platform and 8 reasons of why I should know about it at our local group meeting of Python user at Pernambuco, Brazil.  It was a great lecture and I believe that attracted more and more people to learn Python platform. However, while I was preparing my presentation I decided also to present some statistics about our local group (PUG-PE).  It is a local community of python developers that was created at July's 2007 and since its beginning it started to grow up quickly by many events, specifically live events, lectures and dojos organized by the group.

In my last lectures I always presented a simple line chart showing the evolution at the number of new members joining the group through the time, such as like this one below:

PUG-PE Discussion List Members Growing since 2007

But sometime ago I came into a great visualization tool called Gource, a project for visualizing the control's version history of softwares (using GIT, SVN, etc.).  It is a beautiful visualization, specifically because it shows interactively all the history of commits of your project.  Then, I realized that I could adapt the tool to process a custom log of all the history of the local group and generate a visualization of the simulation in the evolution of number of new members  since the foundation of group until now.  The data was extracted for our discussion list, that it is hosted at Google Groups.  Writing some lines of code in Python I created a simple custom log as input for the Gource tool.  You can find the script here. 

The source data was fetched by exporting the list all members at the section 'Management Tasks' of the Google Group. 

Each node represents a new user that joined the group and the unique users represents the events that triggered the new participations in the group such as local courses, local meetings, etc.  For generating the video you can easily use the command presented here at this blog ( I recorded the video using the ffmpeg tool at Ubuntu 10.04).  You may also easily adapt the script above for your personal use to check out the evolution of your discussion list at Google Groups.  

The result of this experiment is shown here at this video:

Don't forget when you run the command to put the flag -i  with a big value in seconds (e.g. -i 600 ) for maintaining the number of nodes presented during all the simulation (If you suppress this flag the nodes will disappear after a certain time).

I expect you enjoy,

Marcel Caraciolo

An Custom Python SMS/E-mail Alert Manager using the Google Calendar API's

Hi all,

This week I was working on a recommendation engine project (By the way it is the activity that I am focused on for about two months) and I faced a simple problem during the deploy. Since I was the administrator responsible for the execution of the scripts scheduled for processing the recommendations  it was important to me to be alerted whether the execution of these scripts was successful or a failure.  I haven't searched deeply more for a solution in the web so I decided to develop a simple script in python that would solve my problem in a fast way.

Here in the company we have a custom e-mail responsible for receiving and sending any kind of e-mails related to problems or reports about the operation of the website.  Since the e-mail was an account registered at Google Webmail, I remembered that I could use the Google Calendar Reminders/Events features for this task. 

How did I manage that ? 

Simple, first I looked for the GoogleCalendar APIs and found a API written in Python that could help me. The API is gdata and can be downloaded at this link. After I wrote this simple script generic enough for any kind of events that could send me an e-mail or  a text message to my cellphone anytime that the server scripts faced some problems during their execution.

Using the gdata API, the idea is to set an event to start after 1 hour and 15 minutes after the problem occurred (so if the problem occurred now, I would put probably in the except clause to set this event to start 1 hour and 15 min later).  The script also set a reminder that it is defined by the number of minutes before the event start that you want to be alerted of. So by the default I put 60 minutes (1 hour) before the event (but you can change if the problem occurred is a critical issue).  If you pay attention, you will receive the message, in this case, 15 minutes after the problem happened at the server.  In other words, it is the difference between the Start Time and the Reminder Time  (75 - 60 = 15 min).  

So after 15 minutes, you receive an alert (SMS message at your cellphone registered of course at the Google Calendar and an email if you set up with the message that you define).

I know it isn't a sophisticated solution and maybe there are other solutions out there better than this one. But it solved my problem with a few of lines of code and uses the Google Infrastructure to handle with the SMS/E-mail manager with no costs. It is a simple solution special for system analysts that want to monitor the statuses of their databases or servers.    

I've put the code in Github and is totally open-source. I expect it may help anyone facing this problem and wants a simple solution for an e-mail/text message report manager!

Link for Github Project.

That's all! Any suggestions or improvements feel free to comment or ask!

Marcel Caraciolo 

Free Online Book for Introducing Data Mining!

For anyone who wants to start on the data mining topics, I extremely recommend this  book a available online for free.  I found it last week and one of the interesting features of this book that caught my attention was its structure and visualization.  They provided a table of content which is actually a Data Mining Map.

The book is practical and more suited for people who wants to be introduced to Data Mining in a easy way. There are a lot of pictures, tables , definitions and exercises.   This project was created by the Department of Chemical Engineering and Applied Chemistry at the University of Toronto.

Great work and finding!

Marcel Caraciolo

Developing a Computer-Assisted Twitter User . Meet @caocurseiro !

Hi all,

It has been a while since my last post but the reason is that I've been working in some new stuff related to social recommendation and at my master thesis. But this post is to talk about a recent project that I developed for a brazilian social network called AtePassar.  AtePassar is a brazilian social network for people who wants apply for positions at brazilian civil (government) services.  One of the great features of this social network is because people can share their interests about studies and meet people all around Brazil with same interests or someone that will apply for the same exam as him.  Can you imagine the possibilities ?

AtePassar  Main Page

It is a social network for students into a virtual space where there are several relations of friendship, studies and even exam partners. All the network services are free for the users and there is also a e-commerce store where the users can buy resources for helping them in their studies like video lectures, papers, books, etc. My current job there is to apply intelligence in the social network, specifically collective intelligence. Right now I am working in a big project for a social recommender at AtePassar (it will be soon posted here explaining how I managed to build it) and small projects for helping the social network to fetch more new users.   

One of them that I'd like to comment is the new Twitter Bot that I developed that tries to simulate a human, specifically controlling all the actions of a twitter account.  Let me explain more about it.  The goal is not to develop a SPAM bot or a bot that only talk about specified terms.  I imagined it as a bot that could post new updates about open public positions at Brazil, self-helping quotes for stimulating people and simple mentions related to the AtePassar. Also it would retweet tweets from another users that speak about open public exams in Brazil, send mentions to people that are interest about those subjects presenting them the AtePassar social network and even following those users.

Written in Python and hosted at Google AppEngine I created the bot @caocurseiro, who is the mascot of the social network. What does it do ?

• Post 1 to n updates in a random time set by a interval defined by the administrator.
• Send 1 to n mentions to new users  (@newuser) in a random time set by a interval defined by the administrator
• Follow 1 to n new users in a random time set by a interval defined by the administrator
• Retweet 1 to n tweets in a random time set by a interval defined by the administrator.

Of course, since the Twitter politics is against at all kinds of SPAM, I developed this bot to obey the rules and the rare limits established by them.  Other feature is that  he even sleeps during the night (our Brazilian local time)  for a random period.  I am still thinking how to improve it by answering mentions when is directed to him and improve the quality of the posts that sometimes come repeated.

It has been online for about two days our mascot and he's already following 42 users and it's followed by 25 users.  As soon as I put a new retweets rules which it would retweet posts from another users about  subjects related to open public exams and related stuff, I think it will improve much more the quality of the bot.

Unfortunately the code is not yet open-source, but I am working on it to provide it as open-source.  I think the biggest contribution is how to develop a computer driven Twitter User to be similar as human user as possible. Although he still uses static rules, maybe I could put some intelligence to him by answering real questions about any open public exams at Brazil ( future ideas).

Caocurseiro Twitter Computer's Driven

That's all,

Marcel Caraciolo

My lecture about Recommender Systems at IX Pernambuco Python User Group Meeting and my contributions.

Hi all,

It has been while since my last post, but my master thesis is taking a lot of time available. Soon I will come back with posts and content related to what I am working now.

But the main reason of this post is to publish my lectures that were recorded at the IX Pernambuco Python User Group Meeting (PUG-PE) last month in November.  I had the opportunity to talk more about what I am studying, which is related to the topics of recommender systems and a lighting talk about lighting talks!

But since this blog concerns about artificial intelligence,  I will focus on the recommender systems. In this lecture I've introduced the main concepts behind recommender systems, how it works, the advantages and drawbacks of each classing filtering algorithm.  Both examples presented were used in my lecture at PythonBrasil (a main meeting that joins all Python Users of Brazil).  The result of this project will be explained in the future in two posts. But let me explain my main contributions in this field.

One is my work currently at the startup Orygens, where I am developing a novel recommender system applied on social networks. The idea is to recommend users and content to the users of a brazilian social network called AtePassar.

Main Profile of AtePassar

The other contribution is the development of a framework written purely in Python called Crab.  It was originally idealized by me to be a simple easy-to-use recommendation engine framework in order to  be applied on any domains. Besides it will be used to test new approaches, since it will be easy to plug-in new recommender algorithms and test them with the evaluation tools available.  This project is open-source is completely available at my account on GitHub.com.

The main page of the Crab Project

Today we have four collaborators and we are planning to keep going forward with some demos and a distributed computing framework totally integrated with Crab.  More information I will provide soon here at my blog with some demonstrations.

My video about Recommender Systems.  The video is in portuguese.

Wait for news!  Please any further information, contact me!

Marcel Caraciolo