Google Group Members Evolution Visualization using Gource: Our Python Local User Group Analyzed!

Hi all,

Yesterday I lectured a keynote introducing Python Platform and 8 reasons of why I should know about it at our local group meeting of Python user at Pernambuco, Brazil.  It was a great lecture and I believe that attracted more and more people to learn Python platform. However, while I was preparing my presentation I decided also to present some statistics about our local group (PUG-PE).  It is a local community of python developers that was created at July's 2007 and since its beginning it started to grow up quickly by many events, specifically live events, lectures and dojos organized by the group.

In my last lectures I always presented a simple line chart showing the evolution at the number of new members joining the group through the time, such as like this one below:

PUG-PE Discussion List Members Growing since 2007

But sometime ago I came into a great visualization tool called Gource, a project for visualizing the control's version history of softwares (using GIT, SVN, etc.).  It is a beautiful visualization, specifically because it shows interactively all the history of commits of your project.  Then, I realized that I could adapt the tool to process a custom log of all the history of the local group and generate a visualization of the simulation in the evolution of number of new members  since the foundation of group until now.  The data was extracted for our discussion list, that it is hosted at Google Groups.  Writing some lines of code in Python I created a simple custom log as input for the Gource tool.  You can find the script here. 

The source data was fetched by exporting the list all members at the section 'Management Tasks' of the Google Group. 

Each node represents a new user that joined the group and the unique users represents the events that triggered the new participations in the group such as local courses, local meetings, etc.  For generating the video you can easily use the command presented here at this blog ( I recorded the video using the ffmpeg tool at Ubuntu 10.04).  You may also easily adapt the script above for your personal use to check out the evolution of your discussion list at Google Groups.  

The result of this experiment is shown here at this video:

Don't forget when you run the command to put the flag -i  with a big value in seconds (e.g. -i 600 ) for maintaining the number of nodes presented during all the simulation (If you suppress this flag the nodes will disappear after a certain time).

I expect you enjoy,

Marcel Caraciolo

My new experiment: TweetTalk : A Twitter Post Chatter ;D Update tweets from your Google Talk Account!

Hi all,

During my recent studies about Chatter Bots, I've inspired myself to build a new one now integrating Twitter and Google Talk IM Service.  His name is  TweetTalk.  What's the catch ?  

TweetTalk is a Jabber IM Bot for anyone who wants to quickly update a post on your Twitter Account. So instead of going to a separated client, directly from anywhere you can access the Google Talk Engine (Web mail or Desktop Client) you can just write your tweet and the bot will responsible of sending the post to the Twitter.

Let's see it in action:

The tweet status on Twitter:

As you can see, it is really fast now for me to send a tweet from my webmail gmail. It is a simple experiment of how those type of bots can improve your life and work daily.

If you wanna try it, please add to your contacts:   tweettalk@bot.im  

By the way I am using Python for developing the main logic of the Bot.  For web communication I used Django + Google AppEngine + Twitter API.  And as the bot infra-structure the Imified API.

I am writing some new articles about performance evaluation, recommendation engines, REST APIs and SVM with Keyword/Term Extraction. 

Stay tuned !

Marcel Caraciolo

Google launches their new Google Prediction API: the machine learning as cloud services!

Hi all,


I'd like to share some news that I saw yesterday about the launch of the new Google Prediction API. During the Google I/O anual event that have started yesterday, Google has released new web services including this new API.

So, What is the Google Prediction API ? The Prediction API enables access to Google's machine learning algorithms to analyze your historic data and predict likely future outcomes.  It makes possible for developers and researchers to upload their data to Google Storage for Developers (another service launched during the event), and with the Prediction API , it helps them to make real-time decisions such as recommending products, evaluating user sentiment from blogs or even tweets, routing messages or assessing suspicious activities. 

The Prediction API implements supervised learning algorithms as a RESTful web service to let you leverage patterns in your data, providing more relevant information to your users. Run your predictions on Google's infrastructure and scale effortlessly as your data grows in size and complexity.

A simple screenshot (extracted from the Google Prediction HomePage) shows the idea of the service. In this example, it assess the language of the text passed as parameter.

Google Prediction API Workflow

According to the official home page of the API,  it only implements supervised learning algorithms (no unsupervised like clustering algorithms) as a RESTful web service so you can run your predictions on Google's infrastructure and scale effortlessly as your data grows in size and complexity. 

They don't say about the specific algorithms they are using or how they select the one from several available machine learning techniques (I am very curious about it).  It supports almost the most used types of inputs: numeric or data or unstructured text. Their outputs can be hundreds of discrete categories (doesn't  work with continuous output). And the best it is accessible from many platforms like Google App Engine, web , desktop apps ( mobile apps are included?) and command line.

At least, Google introduced another tool for analyzing your data: BigQuery.  This API enables fast, interactive analysis over huge datasets (Imagine trillions of records). Using SQL-like commands via a RESTful API,  you can quickly explore and understand your massive data. It can help you, for example, analyze your network logs, identify seasonal sales trends, etc.

My opinion about this ? Google made a huge step forward to help the current applications in order to use their historical data for improving the usability, decisions and make money, of course! A new generation of applications using those techniques will appear in the next few years, using Natural Language Processing and Machine Learning for improving their services.  A lot of data is available for users and Google is helping them to analyze this data in order to quickly make decisions. With this  RESTful interface, even a young boy with some lines of code could develop a simple application to predict the weather in its city or a twitter-spam filter. Imagine the possibilities!  Now, you don't need to be under a lot of machine learning and statistics books in order to give intelligence to your application or analysis at your data.

It's the intelligence now injected in black boxes for anyone with basic knowledge of programming. Let's see what happens with this step. Anyway,  Google has made a step forward to the Cloud Data Analysis Computing (CDAC) ( I invented this name).

What do you think about it ? Let's wait for the next chapters!

Regards,

Marcel Caraciolo