Mobile Recommenders and current challenges

Hi all,

It has been a while that I've been studying about recommendation engines and how can they be applied on mobile apps.   More and more data is exchanged between those platforms. Great examples of mobile apps that are using recommender engines are Google Hotpot  and Foursquare. They not only connect people to each other, but help users discover places around them.

Bizzy: A mobile recommender for Places

However, they only scratch the surface in this field, which is considered a novel research area in recommender systems. There are  several topics to be explored such as:

•  The location-based recommenders suggest items based on how far away we are from them (sometimes this can be manually changed). This could be a problem, if you consider the distance as the main factor for your recommender. Let me explain with an example. Imagine that you receive music concerts recommendations from your app around your in radius of 2km. When you're looking for live music, there's a band playing 1 km which will be recommended, but your favorite band, which is playing 2.1 km aways, will be out of the final list. And worst, if my  favorite band will play tomorrow and I am at home looking for recommendations, it won't be suggested in this case because of the distance.  It is necessary those systems to consider our habits in order to provide recommendations based on the most checked-in places that I visit (one possibility).

• The mobile recommenders must consider the context where the users are inserted. However the recommenders currently must receive what people are looking for, before receiving the suggestions. Wouldn't be interesting to consider the time events or even the historical habits ? Are those factors enough ?

•  The information about the location and place (content) is also important in the recommendation computation. Imagine you exploring places around you at Foursquare and there are trending places around you (lot's of people there). This recommendation will be received considering only distance ? It is necessary to consider the temporal information associated to the place.

• Just because I've been many times at the Nipon's Sushi , it doesn't mean that I don't want to receive the recommendation again. The process of discovery and re-discovery is important either. The current systems, in general, don't consider the user's familiarity with the locations the user frequent.

• Venues are venues. Events are the "main" item interested by the user, not the venue. Ok, I like receiving a recommendation of a place, but sometimes I'd like to know what's happening there. Furthermore, the current check-ins and rating systems reflect what's happening now, but not what I am planning in the future. The discovery process and prediction decision are main important issues when you're designing a mobile recommender, specially dealing with temporal short-life like events.

• Noisy data. Recommendations of "my apartment", "my mother house". Recommendations can suffer with those types of places.

•  How do we collect the data ? It will be by check-ins, 5 *scale ? Or passive using GPS ? This brings issues about the recommendation interface and how the data will be influenced by the social-signalling noise.

• Some systems use the user's search history to recommend places. This can be noisy, considering that sometimes that what the user searches online often do not match what the user needs when he wants to go out. Just because I looked for information about the Recife airport, it does mean that I'd like to receive airport recommendations.  The point is: the relevance of places that you search for online doesn't match places that you would like to discover in the real world.

Those are some of several challenges faced by mobile recommenders. Mobile recommender systems are still growing and there's lots of research around it.  But one important observation to make is that the best techniques for recommenders in the web sometimes are not suited for mobile recommenders. It's required that the recommendation designer be able to balance between the distance and the preferences from the user and the related items, understand the context where the user is inserted and help him to discover and re-find places and events hidden in his city!

There will be a workshop during the ACM Recommender Systems 2011 about mobile recommenders. Unfortunately this year I won't be attending , but it is on my plans! By the way, you still can submit your paper until July's 25th!

PS: I've found this great blog about mobile recommendations and mobile data mining : Urban Mining. I recommend!

PS2: Recommended reading about why people check-in. An interesting research about why people are interested in checking-in mobile applications.

I hope you enjoyed this article,

Marcel Caraciolo

Atepassar Social Network Friendship Connections Visualizations using GeoLocalization!

Hi all


I've been looking after some visualization tools for social networks in order to present a visual representation of the the AtePassar social network, helping me to see how the users are connected and  visualize the friendships between them.  However, sometime ago I found this post about a new visualization created by the Facebook team which has explored new types of visualization. They plotted a new visualization that showed how geography and political borders affected where people lived relative to their friends.  This visualization focused on which cities all around the world had a lot of friendships between them.

The result is shown here:

Facebook Friendship Visualization

If you want to know more about the how they managed to create this map, you can check the Facebook's blog.  Inspired by this work I decided to create one by my own analyzing the AtePassar network. AtePassar is a famous brazilian social network where I work for as a data mining analyst creating and bringing collective intelligence to improve the features of the website.

AtePassar Social Network

So I have created a Python script which exports the data from the AtePassar Profile users and then convert it to a structured file with information of each user's current city and summed the number of friends between each pair of cities. Then, I merged the data with the longitude and latitude of each city. The coordinates of the brazilian cities were obtained at the Datasus website, a Brazilian data repository    

for the government with statistics and data files about Brazil's  population, health, geography, etc.  You can download the database with the information of the cities here.  To open and read it you can use a third-party library called dbfpy, which handles with .dbf data files. The script is available for download at my personal repository at Github. You can use and modify it for your needs. 

The result of all the experiment is shown in the figure below.  There are some interesting insights about it:

Atepassar SocialNetwork until Feb 2011 - Friendship Visualization

• There are several black areas in the map. Since Brazil is a huge country and there are several places, specially in the North region where we have the Amazon Forest, the demography there is quite low, so we don't have many users around there.  Also, in the North region is the region with the lowest number of users at AtePassar. We see only in the capitals the presence of users, so we believe the access to internet is still a problem around that region or maybe our network is not yet released there.
• We have a great number of users in Recife (PE), São Paulo (SP), Rio de Janeiro (RJ) and Brasília (DF).  As you may see the white shinning lines that interconnect those states in contrast to another cities states in the map. We believe Recife is important specially because the team working behind AtePassar is from Recife, PE, so the marketing around network there is more present than other cities. Another reason is because of the videos available at Atepassar, which the provider (the course and teachers staff) is also quite famous around Recife, PE.  São Paulo, Rio de Janeiro and Brasília are considered currently the cities that have the greatest number of students registering for public exams according to a research made by a popular news site  CorreioWeb, specialized in news about public exams.

After seeing the Perone's post at his blog using the visualization tool Gource to create a new visualization for the Google Analytics,  I realized that project could help me to tell the history of AtePassar Social Network. After writing some python code,  I decided to represent the users by using the states of the users and I also changed the default user icon from Gource to brazilian state flags (You can download them here). 

The social network started at 2009 and launched for public in middle of 2010, where today the network have more than 30 thousand users registered.  We modified the Gource in a way that it could represent the history users registering of all social network by showing the users and his hometowns. Unfortunately, Gource does not work with more than ~= 15.000 nodes, so I decided to show only a period of the social network since its launch until April 2010.  

I've also tried the visualization tool 3D Ubigraph, however since there were thousands of nodes, it didn't work for long periods. This time I've tried to present the network in a different aspect by checking the friendship between the users. It is clear in the video below that the network centers around between two users, by the way, the founders of the social network rjcf and marcoscampello. Another aspect to see is that there are many users but with low degree of friendship. This happens because the timeline of the socialnetwork is the same for all users. Different from Twitter, the user in AtePassar can see what everyone posts in the timeline. We believed that in the beginning of the social network in order to estimulate the interaction between users, we decided to show the posts of all users at Atepassar. But the team is looking carefully if the timeline stream becomes overloaded. The video is presented below.

I was so excited with the results that I decided to use the Gource tool for presenting the history of all users that joined our local community of Python Technology here at Pernambuco-Brazil to present in a lecture of one of our meetings. You can read more about it in this post.


I'd like to mention Andreas Kaltenbrunner for supporting me in this work, giving me some insights on how drawing the brazilian map using coordinates.  He did a similar work on a spanish social network called Tuenti. You can see his post about it here.

I hope you like it,

Cheers, 

Marcel Caraciolo

Mining data from Web 2.0 and Location Web Services for Services Recommendation and Products Offer via Mobile media

Hi all,

It has been a while since my last post, but I've returned.  During this period, I was working on master thesis project plan (and finally decided what I will research and work on) as also lecturing a Python training course for a company here at Recife - Brazil. In this post, I will talk more about what I'm planning to do at my master thesis and present some concepts related to Mobile Marketing, Web, Services, Social Media and Recommendation.

Web  2.0  and Location Web Services [ Photo from blog Arrobazona]

Here, I present a resume of my master degree plan.

With the advent of the latest Web 2.0 technologies [1] and social activities ocurring all over the world, more and more people are sharing information and building relationships. They're taking a important role in part of our lives as helping to answer critical questions such as 'what' , 'how', 'where', 'where', 'why' and 'who'.  However, regardless of these questions, one critical issue is how to give all those answers (information) effectively and recommend in a way that may interest people.

One of the possible targets for these activities are the mobile phones. They are a perfect recipient for fetching a variety of data from mobile information like location and ubiquitous content like small text messages, photos, etc. The new generation of multimedia mobile phone, like Iphone, has begun to integrate online web services and location data acquired from location providers such as  Global Positioning System (GPS) and mobile networks.  These new services formed a known and independent research area name as Location Based Services (LBS)[2] [3].  A perfect example of LBS is the Google Maps [4], which aims to help mobile users access to their destinations with real-time traffic information and road conditions. 

Futhermore, the  GPS software vendors, mobile operators and content providers have also gradually to try for the mobile terminal application development. With content created by combining GPS location-based services and latest Web 2.0 technologies (blogs, tagging, comments, social networks, etc.)  it would be possible to provide timely and personalized information and sharing services based on the user's location information. Or even more, use the content provided of the mobile user, to inform the vicinity of restaurants, entertainment and shopping information, etc.

If we look at the existing location-based services,  such as Foursquare [6], Yelp [7] , Gowalla [8] and others, its information is derived from a single content providers (such as map makers or service providers) so there are some relevant limitations [5].  Based on the traditional information retrieving, the location-based-services and companies are giving more emphasis on the dynamics of information and diversity more than the real-time and targeted content services.  Although, the  users want to be able to obtain contextual and identifying content, not just the indexed information based simply on a static database.  

Recently, those LBS services are looking to how to improve their systems by using some game components and   foucusing on the user experience and engagement with augmented-reality functionalities [9]. However, the rise of a large number of Web 2.0 applications (blogs, microblogs, Taggins, forums, Web albums, etc.) indicates that the users have the urgent requirements of direct, fast, useful and personalized information recommendation and sharing services.

So there is a big question here: How to efficiently combine new Web 2.0 applications (Twitter, Facebook, etc.) with location based services and apply to mobile phone ?  Since there are heterogeneous data and services in various formats and different application platforms, how to integrate all this data that can be used as platform-transparency specially for the user? And how to display all this information in a limited display screen of mobile devices, without prejudicing the usability and  the associated costs for the traffic data. Finally,  how to deploy a mobile discovery content  provider by identifying the user preferences and his location in a intelligent way ?

Those questions are doubtless part of a important research topic, and will have a very wide market prospect. Creating mobile advertisements to target a specific audience and a group of users is also one of the challenges in this area and in the Mobile Marketing research field.

Considering the previous statements, my proposal is to study the use of data mining techniques and recommendation engines in order to develop a  recommender system  integrated with Web technologies and location web services in the mobile enviroment. To solve that I will apply a variety of data analysis tools, algorithms to discover valid, novel, potentially useful and understandable patterns and relationships in data.  Design and implement a collaborative recommender algorithm that can analyze the user value-added data obtained from many Web 2.0 applications. Finally,  prototype a location-based data and service middleware based on web services protocols (SOA) to group all this heterogeneous data and services and publish them as one transparent-platform web service. Atacking those fields, I believe at the end of this project, to develop a real case demo and present a complete tool set for mobile data analysis.

That's all, There are many important topics to research and a lot of work to do. My aim is to build a recommender system for events/places/users using data from Twitter/Foursquare and Yelp and other possibility for recommend/offer products in ubiquitous enviroments with prices, items and shopping advertisements. I believe that there's a incredible and promising to research, specially with the  rise of new mobile social web services.

Best regards,

Marcel Caraciolo

References

[1] Tim O'Reilly (2005-09-30). "What Is Web 2.0". O'Reilly Network. 

 http://www.oreillynet.com/pub/a/oreilly/tim/news/2005/09/30/what-is-web-20.html.

[2]  Shiode, N., Li, C., Batty, M., Longley, P., & Maguire, D. The impact and penetration of  location-based services. In H. A. Karimi & A.  Hammad (Eds.), Telegeoinformatics:  location-based computing and services, 2004,  pp. 349–366, CRC Press.

[3] Jiang, B., Yao, X. B. Location-based services  and GIS in perspective. Computers, Environment and Urban Systems,Vol.30, No.6, 2006, pp. 712-725.

[4] Google.  Google Maps . At http://maps.google.com

[5] Li, C. User preferences, information transactions and location-based services: A  study of urban pedestrian way finding. Computers, Environment and Urban Systems, Vol.30, No. 6, 2004, pp.726–740.

[6] Foursquare.  Foursquare:. At http://www.foursquare.com

[7] Yelp.  Yelp:. At http://www.yelp.com

[8] Gowalla. Gowalla At http://www.gowalla.com

[9] Maria R. Ebling, Ramón Cáceres, "Gaming and Augmented Reality Come to Location-Based Services," IEEE Pervasive Computing, vol. 9, no. 1, pp. 5-6, Jan.-Mar. 2010.