Pages

High Performance Computation with Python - Part 04

Monday, October 10, 2011


Hi all,

This article is the fourth one of the series about High Computation with Python.  For anyone that missed the first, second and third parts check this link about Python Profiling, this one about Cython and finally this about  Numpy Vectors. The goal is to present approaches to make CPU-demanding tasks in Python run much faster.

The techniques that are being covered:

  1.  Python Profiling - How to find bottlenecks
  2.  Cython -  Annotate your code and compile to C
  3.  Numpy Vectors - Fast vector operations using numpy arrays
  4.  Numpy integration with Cython - fast numerical Python library wrapped by Cython
  5.  PyPy - Python's new Just in Time  Compiler
In this post I will talk about PyPy - the JIT Compiler for Python!

The Problem


In this series we will analyze how to optimize the statistical Spearman Rank's Correlation coefficient,  which it is a particular measure used to compute the similarity between items in recommender systems and assesses how well the relationship between two variables can be described using a monotonic function. The source code for this metric can be found in the first post.


Pypy

PyPy is a Just in TIme compiler for the Python programming language. It is multi-platform and it runs Python 2.7.  With your code running in PyPy, it will make your code (depending on how you write your code) run faster (2 - 10 x speed-ups).   Sometimes some work has to be done in the code because of the use of shortcuts that works in CPython that aren't actually correct in the Python specification.

You can download and install PyPy here. To install it, just place it in your home directory and put a symlink from somewhere to it.  Let's run the spearman.py with PyPy and without Python and see the performance difference;



The difference is about 34.77% faster with PyPy against pure Python  considering the input with 190340 on my Macbook. The amazing part is that I didn't change any line of my code! \m/  
If you aren't using a C library like numpy then you should check PyPy - it might just make your code run several times faster. They are still porting PyPy to support Numpy since it some C libraries required   for running Numpy must be rewritten.   You can see some benchmarks of the porting here.

Although the PyPy team gave us a simple integration with array interfaces that behaves in a numpy-like fashion,  for now it has very few functions and only supports double arithmetic.



I strongly recommend you to take a look at PyPy, it shows a great promise for high performance Python with little effort and specially for the scientific community with the support with existing numpy would be a great advance!

I didn't mention until now by I will write a special post to close this series with High Performance with Python: It is about the module multiprocessing and how you can work with it. I will show some examples and a library called JobLib that wraps it where you can easily use the power of the processors of your machine and do some parallell work.


See you next time,

Regards,

Marcel Caraciolo



9 comments:

  1. you might have an excellent weblog right here! would you like to make some invite posts on my weblog?

    ReplyDelete
  2. Welcome to Wiztech Automation - Embedded System Training in Chennai. We have knowledgeable Team for Embedded Courses handling and we also are after Job Placements offer provide once your Successful Completion of Course. We are Providing on Microcontrollers such as 8051, PIC, AVR, ARM7, ARM9, ARM11 and RTOS. Free Accommodation, Individual Focus, Best Lab facilities, 100% Practical Training and Job opportunities.

    Embedded System Training in chennai
    Embedded System Training Institute in chennai
    Embedded Training in chennai
    Embedded Course in chennai
    Best Embedded System Training in chennai
    Best Embedded System Training Institute in chennai
    Best Embedded System Training Institutes in chennai
    Embedded Training Institute in chennai
    Embedded System Course in chennai
    Best Embedded System Training in chennai

    ReplyDelete
  3. WIZTECH Automation, Anna Nagar, Chennai, has earned reputation offering the best automation training in Chennai in the field of industrial automation. Flexible timings, hands-on-experience, 100% practical. The candidates are given enhanced job oriented practical training in all major brands of PLCs (AB, Keyence, ABB, GE-FANUC, OMRON, DELTA, SIEMENS, MITSUBISHI, SCHNEIDER, and MESSUNG)

    PLC training in chennai
    Automation training in chennai
    Best plc training in chennai
    PLC SCADA training in chennai
    Process automation training in chennai
    Final year eee projects in chennai
    VLSI training in chennai

    ReplyDelete
  4. Embedded system training: Wiztech Automation Provides Excellent training in embedded system training in Chennai - IEEE Projects - Mechanical projects in Chennai. Wiztech provide 100% practical training, Individual focus, Free Accommodation, Placement for top companies. The study also includes standard microcontrollers such as Intel 8051, PIC, AVR, ARM, ARMCotex, Arduino, etc.

    Embedded system training in chennai
    Embedded Course training in chennai
    Matlab training in chennai
    Android training in chennai
    LabVIEW training in chennai
    Robotics training in chennai
    Oracle training in chennai
    Final year projects in chennai
    Mechanical projects in chennai
    ece projects in chennai

    ReplyDelete
  5. Thanks for such an interesting article here. I was searching for something like that for quite a long time and at last I have found it here. best placement engineering college in punjab

    ReplyDelete
  6. Great post. Thank you for sharing such useful information. Please keep sharing

    Best B.Tech College in Noida

    ReplyDelete
  7. أفضل شركه نقل عفش بالرياض
    اليكم افضل النصائح شراء اثاث مستعمل التي تقدم لكم المساعدة في علي نقل العفش بأسهل الطرق المتبعة ودون اي خسائر او تلفيات لعفشك او اجهزتك كما ان تقوم شركات نقل الاثاث بالعديد من الطرق والتنوعة والتي لها اختلاف وطابع خاص عن باقي الشركات في نقل الاثاث فهناك العديد من الشركات التي تقوم بنقل الاثاث بطرق غير صحيحه وخاطئة تضر بالاثاث وتعرضة للتجريح والخدش فأن شراء اثاث مستعمل جدةتعد من افضل الشركات بالرياض التي تقوم بنقل الاثاث بأفضل الطرق المستخدمة والصحيحة في نقل الاثاث ومن تلك الطرق اننا نقوم بتغليف الاثاث اولا بالتغليف الحراري والذي له اهمية كبيرة وهو من افضل طرق التغليف علي الاطلاق ويتم استخدامه في العديد من الاستخدامات المتنوعة والعديدة ، وكما ان لدي اثاث مستعمل
    التغليف الذي يسمي التغليف بالمفرقعات الذي يتم استخدامة في تغليف الاطباق والاواني بأنواعها والزجاج واي شئ يمكن ان يكون قابل للكسر فاعتمادنا الاساسي في تأدية مهمتنا هي ان نقوم بتوصيل اثاثك من دون ان يحدث له اي تلفيات او خدوش او كسرفي نقل اثاث بالرياض كما اننا نعتمد علي اكبر السيارات في نقل العفش واحدث الرافعات التي تقوم برفع عفشك في الاماكن العالية .
    افضل الخطوات المتبعة بشركة نقل اثاث بالرياض
    - الخطوة الاولي التي تهتم بها محلات شراء الاثاث المستعمل بجدة عملية الفك بالرغم من انها تبدو سهلة الا انها تحتاج الي متخصصين وفريق مدرب ولة خبرة طويلة في كيفية الفك والتركيب للحفاظ علية من الخدش او الثني والحفاظ عليه بشكل تام .
    شركة شراء اثاث مستعمل بالرياض

    -الخطوة الثانية وهي تغليف الاثاث بشكل صحيح حتي يتم حمايتة من الاتربة والخدوش التي يتعرض لها اثناء عملية النقل وكذلك نستخدم ممتصات لاي نوع من انواع الصدمات المختلفة التي تؤدي الي تلف الاثاث بأستخدام قطع القماش مع الفلين او بلاستيك الفقاعات او النايلون لبعض من القطع
    شراء الاثاث المستعمل بالرياض
    -الخطوة الثالثة وهي تركيب الاثاث وهي خطوة هامة ايضا عندما يتم نقل الاثاث الي المنزل الجديد بشكل سليم نقوم بترتيب الاثاث داخل المنزل التسليمة الي عميلنا في ابهي صورة كما ان لدينا متخصصين في تركيب الاثاث دون تعرضة لاي تلفيات بأحدث ادوات الفك والتركيب فهم عمالة مدربة بتقنية عالية كما ان شركتنا لها اسعار تنافسية ليس لها مثيل عن باقي شركات نقل الاثاث بالرياض
    حقين شراء الاثاث المستعمل بالرياض


    ReplyDelete