Slides for Scientific Computing Meeting: Benchy and GeoMapper Visualization

Hi all,

Yesterday it happened the XXVI local meeting of the Python Users Group at Pernambuco (PUG-PE).  In the occasion I had the opportunity to present two talks about scientific computing with python.

The first one was the lightweight framework for benchmark analysis on Python Scripts called Benchy, which I developed for about one week to help me on checking performance of several algorithms that I developed in Python.  I covered the framework at my last post which can be found here.

Here are the slides for the presentation:

Benchy: Lightweight framework for Performance Benchmarks from Marcel Caraciolo

The second talk was about a new type of visualization that I developed for social network analysis in order to check the degree of connections between the users at the socialnetwork using their geolocation data to present in a map.

The result was beautiful plots using this new type of visualization.  Amazing!

The slides are available here:

GeoMapper, Python Script for Visualizing Data on Social Networks with Geo-location data from Marcel Caraciolo

I hope you enjoy the slides, any further information feel free to comment!

Regards,

Marcel Caraciolo

Performing runtime benchmarks with Python Monitoring Tool Benchy

Hi all,

I've been working on in the last weeks at a little project that I developed called benchy.  The goal of benchy is answer some trivial questions about which code is faster ?  Or which algorithm consumes more memory ?  I know that there are several tools suitable for this task, but I would like to create some performance reports  by myself using Python.   

Why did I create it ?  Since the beginning of the year I decided to rewrite all the code at Crab, a python framework for building recommender systems.  And one of the main components that required some refactoring was the pairwise metrics such as cosine, pearson, euclidean, etc.  I needed to unit test the performance of several versions of code for those functions. But doing this manually ? It's boring. That's why benchy came for!

What benchy can do ?

Benchy is a lightweight Python library for running performance benchmarks over alternative versions of code.  How can we use it ?

Let's see the cosine function, a popular pairwise function for comparing the similarity between two vectors and matrices in recommender systems.

Let's define the benchmarks to test:



With all benchmarks created, we could test a simple benchmark by calling the method run:


The dict associated to the key memory represents the memory performance results. It gives you the number of calls repeat to the statement, the average consumption usage in units . In addition, the key 'runtime' indicates the runtime performance in timing results. It presents the number of calls repeat following the average time to execute it timing in units.

Do you want see a more presentable output ? It is possible calling the method to_rst with the results as parameter:

Benchmark setup

import numpy
X = numpy.random.uniform(1,5,(1000,))

import scipy.spatial.distance as ssd
X = X.reshape(-1,1)
def cosine_distances(X, Y):
    return 1. - ssd.cdist(X, Y, 'cosine')

Benchmark statement

cosine_distances(X, X)

name	repeat	timing	loops	units
scipy.spatial 0.8.0	3	18.36	10	ms

Now let's check which one is faster and which one consumes less memory. Let's create a BenchmarkSuite. It is referred as a container for benchmarks.:

Finally, let's run all the benchmarks together with the BenchmarkRunner. This class can load all the benchmarks from the suite and run each individual analysis and print out interesting reports:



Next, we will plot the relative timings. It is important to measure how faster the other benchmarks are compared to reference one. By calling the method plot_relative:

As you can see the graph aboe the scipy.spatial.distance function is 2129x slower and the sklearn approach is 19x. The best one is the numpy approach. Let's see the absolute timings. Just call the method plot_absolute:

You may notice besides the bar representing the timings, the line plot representing the memory consumption for each statement. The one who consumes the less memory is the nltk.cluster approach!

Finally, benchy also provides a full repport for all benchmarks by calling the method to_rst:

Performance Benchmarks

These historical benchmark graphs were produced with benchy.

Produced on a machine with

• Intel Core i5 950 processor
• Mac Os 10.6
• Python 2.6.5 64-bit
• NumPy 1.6.1

scipy.spatial 0.8.0

Benchmark setup

import numpy
X = numpy.random.uniform(1,5,(1000,))

import scipy.spatial.distance as ssd
X = X.reshape(-1,1)
def cosine_distances(X, Y):
    return 1. - ssd.cdist(X, Y, 'cosine')

Benchmark statement

cosine_distances(X, X)

name	repeat	timing	loops	units
scipy.spatial 0.8.0	3	19.19	10	ms

sklearn 0.13.1

Benchmark setup

import numpy
X = numpy.random.uniform(1,5,(1000,))

from sklearn.metrics.pairwise import cosine_similarity as cosine_distances

Benchmark statement

cosine_distances(X, X)

name	repeat	timing	loops	units
sklearn 0.13.1	3	0.1812	1000	ms

nltk.cluster

Benchmark setup

import numpy
X = numpy.random.uniform(1,5,(1000,))

from nltk import cluster
def cosine_distances(X, Y):
    return 1. - cluster.util.cosine_distance(X, Y)

Benchmark statement

cosine_distances(X, X)

name	repeat	timing	loops	units
nltk.cluster	3	0.01024	1e+04	ms

numpy

Benchmark setup

import numpy
X = numpy.random.uniform(1,5,(1000,))

import numpy, math
def cosine_distances(X, Y):
    return 1. -  numpy.dot(X, Y) / (math.sqrt(numpy.dot(X, X)) *
                                     math.sqrt(numpy.dot(Y, Y)))

Benchmark statement

cosine_distances(X, X)

name	repeat	timing	loops	units
numpy	3	0.009339	1e+05	ms

Final Results

name	repeat	timing	loops	units	timeBaselines
scipy.spatial 0.8.0	3	19.19	10	ms	2055
sklearn 0.13.1	3	0.1812	1000	ms	19.41
nltk.cluster	3	0.01024	1e+04	ms	1.097
numpy	3	0.009339	1e+05	ms	1

Final code!

I might say this micro-project is still a prototype, however  I tried to build it to be easily extensible. I have several ideas to extend it, but feel free to fork it and send suggestions and bug fixes.  This project was inspired by the open-source project vbench, a framework for performance benchmarks over your source repository's history. I recommend!

For me, benchy will assist me to test several pairwise alternative functions in Crab. :)  Soon I will publish the performance results that we got with the pairwise functions that we built for Crab :)

I hope you enjoyed,

Regards,

Marcel Caraciolo

Crab - Python Framework for Building Recommender Engines Video at Scipy 2011 Conference

Hi all,

My lecture at Scipy Conference 2011 is already available on-line in video, so you can watch me now presenting about our work at Muricoca Labs called "Crab - A Python Framework for Building Recommender Engines"  .  It is a work that I am developing with some machine learning developers as an alternative for python developers that want to work with recommender engines writing Python code.

Further information , it may be found here at the official home project. About my experience at Scipy, you can find at this post.

Here the link for the video,

Cheers,

Marcel Caraciolo