Atepassar Social Network Friendship Connections Visualizations using GeoLocalization!

Hi all


I've been looking after some visualization tools for social networks in order to present a visual representation of the the AtePassar social network, helping me to see how the users are connected and  visualize the friendships between them.  However, sometime ago I found this post about a new visualization created by the Facebook team which has explored new types of visualization. They plotted a new visualization that showed how geography and political borders affected where people lived relative to their friends.  This visualization focused on which cities all around the world had a lot of friendships between them.

The result is shown here:

Facebook Friendship Visualization

If you want to know more about the how they managed to create this map, you can check the Facebook's blog.  Inspired by this work I decided to create one by my own analyzing the AtePassar network. AtePassar is a famous brazilian social network where I work for as a data mining analyst creating and bringing collective intelligence to improve the features of the website.

AtePassar Social Network

So I have created a Python script which exports the data from the AtePassar Profile users and then convert it to a structured file with information of each user's current city and summed the number of friends between each pair of cities. Then, I merged the data with the longitude and latitude of each city. The coordinates of the brazilian cities were obtained at the Datasus website, a Brazilian data repository    

for the government with statistics and data files about Brazil's  population, health, geography, etc.  You can download the database with the information of the cities here.  To open and read it you can use a third-party library called dbfpy, which handles with .dbf data files. The script is available for download at my personal repository at Github. You can use and modify it for your needs. 

The result of all the experiment is shown in the figure below.  There are some interesting insights about it:

Atepassar SocialNetwork until Feb 2011 - Friendship Visualization

• There are several black areas in the map. Since Brazil is a huge country and there are several places, specially in the North region where we have the Amazon Forest, the demography there is quite low, so we don't have many users around there.  Also, in the North region is the region with the lowest number of users at AtePassar. We see only in the capitals the presence of users, so we believe the access to internet is still a problem around that region or maybe our network is not yet released there.
• We have a great number of users in Recife (PE), São Paulo (SP), Rio de Janeiro (RJ) and Brasília (DF).  As you may see the white shinning lines that interconnect those states in contrast to another cities states in the map. We believe Recife is important specially because the team working behind AtePassar is from Recife, PE, so the marketing around network there is more present than other cities. Another reason is because of the videos available at Atepassar, which the provider (the course and teachers staff) is also quite famous around Recife, PE.  São Paulo, Rio de Janeiro and Brasília are considered currently the cities that have the greatest number of students registering for public exams according to a research made by a popular news site  CorreioWeb, specialized in news about public exams.

After seeing the Perone's post at his blog using the visualization tool Gource to create a new visualization for the Google Analytics,  I realized that project could help me to tell the history of AtePassar Social Network. After writing some python code,  I decided to represent the users by using the states of the users and I also changed the default user icon from Gource to brazilian state flags (You can download them here). 

The social network started at 2009 and launched for public in middle of 2010, where today the network have more than 30 thousand users registered.  We modified the Gource in a way that it could represent the history users registering of all social network by showing the users and his hometowns. Unfortunately, Gource does not work with more than ~= 15.000 nodes, so I decided to show only a period of the social network since its launch until April 2010.  

I've also tried the visualization tool 3D Ubigraph, however since there were thousands of nodes, it didn't work for long periods. This time I've tried to present the network in a different aspect by checking the friendship between the users. It is clear in the video below that the network centers around between two users, by the way, the founders of the social network rjcf and marcoscampello. Another aspect to see is that there are many users but with low degree of friendship. This happens because the timeline of the socialnetwork is the same for all users. Different from Twitter, the user in AtePassar can see what everyone posts in the timeline. We believed that in the beginning of the social network in order to estimulate the interaction between users, we decided to show the posts of all users at Atepassar. But the team is looking carefully if the timeline stream becomes overloaded. The video is presented below.

I was so excited with the results that I decided to use the Gource tool for presenting the history of all users that joined our local community of Python Technology here at Pernambuco-Brazil to present in a lecture of one of our meetings. You can read more about it in this post.


I'd like to mention Andreas Kaltenbrunner for supporting me in this work, giving me some insights on how drawing the brazilian map using coordinates.  He did a similar work on a spanish social network called Tuenti. You can see his post about it here.

I hope you like it,

Cheers, 

Marcel Caraciolo

Google Group Members Evolution Visualization using Gource: Our Python Local User Group Analyzed!

Hi all,

Yesterday I lectured a keynote introducing Python Platform and 8 reasons of why I should know about it at our local group meeting of Python user at Pernambuco, Brazil.  It was a great lecture and I believe that attracted more and more people to learn Python platform. However, while I was preparing my presentation I decided also to present some statistics about our local group (PUG-PE).  It is a local community of python developers that was created at July's 2007 and since its beginning it started to grow up quickly by many events, specifically live events, lectures and dojos organized by the group.

In my last lectures I always presented a simple line chart showing the evolution at the number of new members joining the group through the time, such as like this one below:

PUG-PE Discussion List Members Growing since 2007

But sometime ago I came into a great visualization tool called Gource, a project for visualizing the control's version history of softwares (using GIT, SVN, etc.).  It is a beautiful visualization, specifically because it shows interactively all the history of commits of your project.  Then, I realized that I could adapt the tool to process a custom log of all the history of the local group and generate a visualization of the simulation in the evolution of number of new members  since the foundation of group until now.  The data was extracted for our discussion list, that it is hosted at Google Groups.  Writing some lines of code in Python I created a simple custom log as input for the Gource tool.  You can find the script here. 

The source data was fetched by exporting the list all members at the section 'Management Tasks' of the Google Group. 

Each node represents a new user that joined the group and the unique users represents the events that triggered the new participations in the group such as local courses, local meetings, etc.  For generating the video you can easily use the command presented here at this blog ( I recorded the video using the ffmpeg tool at Ubuntu 10.04).  You may also easily adapt the script above for your personal use to check out the evolution of your discussion list at Google Groups.  

The result of this experiment is shown here at this video:

Don't forget when you run the command to put the flag -i  with a big value in seconds (e.g. -i 600 ) for maintaining the number of nodes presented during all the simulation (If you suppress this flag the nodes will disappear after a certain time).

I expect you enjoy,

Marcel Caraciolo

An Custom Python SMS/E-mail Alert Manager using the Google Calendar API's

Hi all,

This week I was working on a recommendation engine project (By the way it is the activity that I am focused on for about two months) and I faced a simple problem during the deploy. Since I was the administrator responsible for the execution of the scripts scheduled for processing the recommendations  it was important to me to be alerted whether the execution of these scripts was successful or a failure.  I haven't searched deeply more for a solution in the web so I decided to develop a simple script in python that would solve my problem in a fast way.

Here in the company we have a custom e-mail responsible for receiving and sending any kind of e-mails related to problems or reports about the operation of the website.  Since the e-mail was an account registered at Google Webmail, I remembered that I could use the Google Calendar Reminders/Events features for this task. 

How did I manage that ? 

Simple, first I looked for the GoogleCalendar APIs and found a API written in Python that could help me. The API is gdata and can be downloaded at this link. After I wrote this simple script generic enough for any kind of events that could send me an e-mail or  a text message to my cellphone anytime that the server scripts faced some problems during their execution.

Using the gdata API, the idea is to set an event to start after 1 hour and 15 minutes after the problem occurred (so if the problem occurred now, I would put probably in the except clause to set this event to start 1 hour and 15 min later).  The script also set a reminder that it is defined by the number of minutes before the event start that you want to be alerted of. So by the default I put 60 minutes (1 hour) before the event (but you can change if the problem occurred is a critical issue).  If you pay attention, you will receive the message, in this case, 15 minutes after the problem happened at the server.  In other words, it is the difference between the Start Time and the Reminder Time  (75 - 60 = 15 min).  

So after 15 minutes, you receive an alert (SMS message at your cellphone registered of course at the Google Calendar and an email if you set up with the message that you define).

I know it isn't a sophisticated solution and maybe there are other solutions out there better than this one. But it solved my problem with a few of lines of code and uses the Google Infrastructure to handle with the SMS/E-mail manager with no costs. It is a simple solution special for system analysts that want to monitor the statuses of their databases or servers.    

I've put the code in Github and is totally open-source. I expect it may help anyone facing this problem and wants a simple solution for an e-mail/text message report manager!

Link for Github Project.

That's all! Any suggestions or improvements feel free to comment or ask!

Marcel Caraciolo 

Free Online Book for Introducing Data Mining!

For anyone who wants to start on the data mining topics, I extremely recommend this  book a available online for free.  I found it last week and one of the interesting features of this book that caught my attention was its structure and visualization.  They provided a table of content which is actually a Data Mining Map.

The book is practical and more suited for people who wants to be introduced to Data Mining in a easy way. There are a lot of pictures, tables , definitions and exercises.   This project was created by the Department of Chemical Engineering and Applied Chemistry at the University of Toronto.

Great work and finding!

Marcel Caraciolo

Developing a Computer-Assisted Twitter User . Meet @caocurseiro !

Hi all,

It has been a while since my last post but the reason is that I've been working in some new stuff related to social recommendation and at my master thesis. But this post is to talk about a recent project that I developed for a brazilian social network called AtePassar.  AtePassar is a brazilian social network for people who wants apply for positions at brazilian civil (government) services.  One of the great features of this social network is because people can share their interests about studies and meet people all around Brazil with same interests or someone that will apply for the same exam as him.  Can you imagine the possibilities ?

AtePassar  Main Page

It is a social network for students into a virtual space where there are several relations of friendship, studies and even exam partners. All the network services are free for the users and there is also a e-commerce store where the users can buy resources for helping them in their studies like video lectures, papers, books, etc. My current job there is to apply intelligence in the social network, specifically collective intelligence. Right now I am working in a big project for a social recommender at AtePassar (it will be soon posted here explaining how I managed to build it) and small projects for helping the social network to fetch more new users.   

One of them that I'd like to comment is the new Twitter Bot that I developed that tries to simulate a human, specifically controlling all the actions of a twitter account.  Let me explain more about it.  The goal is not to develop a SPAM bot or a bot that only talk about specified terms.  I imagined it as a bot that could post new updates about open public positions at Brazil, self-helping quotes for stimulating people and simple mentions related to the AtePassar. Also it would retweet tweets from another users that speak about open public exams in Brazil, send mentions to people that are interest about those subjects presenting them the AtePassar social network and even following those users.

Written in Python and hosted at Google AppEngine I created the bot @caocurseiro, who is the mascot of the social network. What does it do ?

• Post 1 to n updates in a random time set by a interval defined by the administrator.
• Send 1 to n mentions to new users  (@newuser) in a random time set by a interval defined by the administrator
• Follow 1 to n new users in a random time set by a interval defined by the administrator
• Retweet 1 to n tweets in a random time set by a interval defined by the administrator.

Of course, since the Twitter politics is against at all kinds of SPAM, I developed this bot to obey the rules and the rare limits established by them.  Other feature is that  he even sleeps during the night (our Brazilian local time)  for a random period.  I am still thinking how to improve it by answering mentions when is directed to him and improve the quality of the posts that sometimes come repeated.

It has been online for about two days our mascot and he's already following 42 users and it's followed by 25 users.  As soon as I put a new retweets rules which it would retweet posts from another users about  subjects related to open public exams and related stuff, I think it will improve much more the quality of the bot.

Unfortunately the code is not yet open-source, but I am working on it to provide it as open-source.  I think the biggest contribution is how to develop a computer driven Twitter User to be similar as human user as possible. Although he still uses static rules, maybe I could put some intelligence to him by answering real questions about any open public exams at Brazil ( future ideas).

Caocurseiro Twitter Computer's Driven

That's all,

Marcel Caraciolo