Let's see the
cosine function, a popular pairwise function for comparing the similarity between two vectors and matrices in recommender systems.
Let's define the
benchmarks to test:
With all benchmarks created, we could test a simple benchmark by c
alling the method run:
The dict associated to the key
memory represents the memory performance results. It gives you the number of calls repeat to the statement, the average consumption usage in units . In addition, the key '
runtime' indicates the runtime performance in timing results. It presents the number of calls repeat following the average time to execute it timing in units.
Do you want see a more presentable output ? It is possible calling the method
to_rst with the results as parameter:
Benchmark setup
import numpy
X = numpy.random.uniform(1,5,(1000,))
import scipy.spatial.distance as ssd
X = X.reshape(-1,1)
def cosine_distances(X, Y):
return 1. - ssd.cdist(X, Y, 'cosine')
Benchmark statement
| name | repeat | timing | loops | units |
|---|
| scipy.spatial 0.8.0 | 3 | 18.36 | 10 | ms |
Now let's check which one is faster and which one consumes less memory. Let's create a BenchmarkSuite. It is referred as a container for benchmarks.:
Finally, let's run all the benchmarks together with the
BenchmarkRunner. This class can load all the benchmarks from the suite and
run each individual analysis and print out interesting reports:
Next, we will plot the relative timings. It is important to measure how faster the other benchmarks are compared to reference one. By calling the method
plot_relative:
As you can see the graph aboe the scipy.spatial.distance function is 2129x slower and the sklearn approach is 19x. The best one is the numpy approach. Let's see the absolute timings. Just call the method
plot_absolute:
You may notice besides the bar representing the timings, the line plot representing the memory consumption for each statement. The one who consumes the less memory is the nltk.cluster approach!
Finally, benchy also provides a full repport for all benchmarks by calling the method
to_rst:
Performance Benchmarks
These historical benchmark graphs were produced with
benchy.
Produced on a machine with
- Intel Core i5 950 processor
- Mac Os 10.6
- Python 2.6.5 64-bit
- NumPy 1.6.1
scipy.spatial 0.8.0
Benchmark setup
import numpy
X = numpy.random.uniform(1,5,(1000,))
import scipy.spatial.distance as ssd
X = X.reshape(-1,1)
def cosine_distances(X, Y):
return 1. - ssd.cdist(X, Y, 'cosine')
Benchmark statement
| name | repeat | timing | loops | units |
|---|
| scipy.spatial 0.8.0 | 3 | 19.19 | 10 | ms |
sklearn 0.13.1
Benchmark setup
import numpy
X = numpy.random.uniform(1,5,(1000,))
from sklearn.metrics.pairwise import cosine_similarity as cosine_distances
Benchmark statement
| name | repeat | timing | loops | units |
|---|
| sklearn 0.13.1 | 3 | 0.1812 | 1000 | ms |
nltk.cluster
Benchmark setup
import numpy
X = numpy.random.uniform(1,5,(1000,))
from nltk import cluster
def cosine_distances(X, Y):
return 1. - cluster.util.cosine_distance(X, Y)
Benchmark statement
| name | repeat | timing | loops | units |
|---|
| nltk.cluster | 3 | 0.01024 | 1e+04 | ms |
numpy
Benchmark setup
import numpy
X = numpy.random.uniform(1,5,(1000,))
import numpy, math
def cosine_distances(X, Y):
return 1. - numpy.dot(X, Y) / (math.sqrt(numpy.dot(X, X)) *
math.sqrt(numpy.dot(Y, Y)))
Benchmark statement
| name | repeat | timing | loops | units |
|---|
| numpy | 3 | 0.009339 | 1e+05 | ms |
Final Results
| name | repeat | timing | loops | units | timeBaselines |
|---|
| scipy.spatial 0.8.0 | 3 | 19.19 | 10 | ms | 2055 |
| sklearn 0.13.1 | 3 | 0.1812 | 1000 | ms | 19.41 |
| nltk.cluster | 3 | 0.01024 | 1e+04 | ms | 1.097 |
| numpy | 3 | 0.009339 | 1e+05 | ms | 1 |
Final
code!
I might say this micro-project is still a prototype, however I tried to build it to be easily extensible. I have several ideas to extend it, but feel free to fork it and send suggestions and bug fixes. This project was inspired by the open-source project
vbench, a framework for performance benchmarks over your source repository's history. I recommend!
For me, benchy will assist me to test several pairwise alternative functions in Crab. :) Soon I will publish the performance results that we got with the pairwise functions that we built for Crab :)
I hope you enjoyed,
Regards,
Marcel Caraciolo
![PythonBrasil[9]](https://lh3.googleusercontent.com/blogger_img_proxy/AEn0k_tbpokPcmEDCaVJLi8u0M7PgA3px1Q2rFCiSKWoTrhob1rtTjSIWBbDbuxfXWEWjAk1CEPWu7buFp2lfxI5yM5JftA4_2GZGUuwA9yZQGeUWfmFbKjQE8zya16TC6mI3PJA1OAq4DwhfHfq6bzDyXkaq-oMJHH3mw=s0-d)
Posts
