Pages

Tools for Machine Learning Performance Evaluation: ROC Curves in Python

Tuesday, September 21, 2010

Hi all,

Continuing from the my last post about performance analysis on classification machine learning techniques, in this article I will talk about a specific analysis called Discriminatory Power Analysis by using Receiver-Operating Characteristic Curves  (ROC Curves).

First, let's review and introduce some concepts related to the classification problems. When we deal with systems that involves the detection, diagnostics or even the prediction of results, it is really important to check the obtained results in order to validate the discriminative power as good or not of a given analysis.  However only depending on the quantization of hits and misses on a test group does not really tell how good a system is, since it depends on the quality and distribution of the test group data.

For instance, let's consider a fraud detection system, which outputs are 1 or 0, indicating whether a transaction has been classified as a fraudulent or not.  Now suppose that we applied the system on a test group of 100 transactions which we already know which one was a fraud or not, and the system correctly identified 90% of conditions. Do you think it is a good performance, don't you ?

Actually it is not. One information is missing and it is how the test group was distributed.  Let's consider that actually 90% was fraudulent. The system, thus, could have considered  '1' to every input given, and still have achieved a 90% accuracy rate, as the 10% transactions left were not fraudulent. By now, we cannot be sure if the system was really good or worked as a random classifier considering everyone as fraudulent without any previous knowledge or calculations.

In this case, we have to consider another metrics in order to evaluate this unbalance in the test groups. As I said in a previous post in this blog about confusion matrix, it will act as a base for the measures used in the discriminatory power analysis in question.

Confusion Matrix

From this table, I will present some metrics used to help in this analysis.

1. Accuracy
It measures the proportion of correct predictions considering the positive and negative inputs.  It is highly dependant of the data set distribution which can easily lead to wrong conclusions about the system performance.

ACC =  TOTAL HITS/ NUMBER OF ENTRIES IN THE SET
          = (TP + TN) / (P + N)

2. Sensitivity

It measures the proportion of the true positives, that is, the ability of the system on predicting the correct values in the cases presented.

SENS = POSITIVE HITS/ TOTAL POSITIVES
           =  TP/ (TP+FN)

3. Specificity

It measures the proportion of the true negatives, that is, the ability of the system on predicting the 
correct values for the cases that are the opposite to the desired one.

SPEC  =  NEGATIVE HITS/ TOTAL NEGATIVES
           =  TN/ (TN +FP)
4. Efficiency

It is represented by the mean of Sensibility and Specificity.  The perfect decision would be  with 100% of specificity and 100% sensitivity. However this situation is rarely conceived, so a balance between both metrics must be obtained. This metric is a good evaluator for measure the responsiveness in a overall situation to the production of false positives and false negatives. Generally, when the system is too responsive to positive, it tends to produce many false positives and vice-versa.

EFF = (SENS + SPEC) /2

5. Positive Predictive Value
 
This measure indicates the estimation of how good the system is when making a positive affirmation. However it is not recommended to use it alone, causing easily to lead to wrong conclusions about system performance. It is the proportion of the true positives in contrast with all positive predictions.

PPV = POSITIVE HITS / TOTAL POSITIVE PREDICTIONS
         = TP / (TP + FP)
 
 6. Negative Predictive Value

This measure indicates the estimation of how good the system is when making a negative affirmation. However it is not recommended to use it alone, causing easily to lead to wrong conclusions about system performance. It is the proportion of the true negatives in contrast with all negative predictions.

NPV = NEGATIVE HITS / TOTAL NEGATIVE PREDICTIONS
         = TN / (TN + FN)

7. Phi (φ) Coefficient

It is a measure not commonly used and measure the quality of the confusion matrix in a single value which can be compared. For instance, if you have two binary classifications with same classes but with different sizes, it can be used to compare both of them. It returns a value between -1 and +1, where +1 represents a perfect prediction, 0 a random prediction, and -1 an inverse prediction.

φ =  (TP*TN - FP*FN) / sqrt( (TP+FP) * (TP +FN) * (TN+FP) * (TN +FN) )


Source Code

As I presented in the first post of this series, I've updated the confusion matrix by adding the new metrics presented above.

View the script at Gist.Github


The Receiver Operating Characteristic (ROC) Curve

The ROC curve was developed during the World War II and was extremely used by engineers to detect enemy objects in enemy fields. It became famous and widely used in other areas such as medicine, radiology, etc.  More recently, it has been introduced to the area of machine learning and dataming.

The main idea behind the ROC curves is to analyze the output from the classification systems, which are generally continuous.  Because of that, it is necessary to define a cut-off value (or a discriminatory threshold) to classify and count the number of positive and negative predictions (such as the fraudulent or legal transactions in the case of the statuses in bank transactions).  This threshold can be arbitrarily determined, so the best way to compare the performance of different classifiers is to select several cutoff values over the output data and study their effect.

Considering many thresholds, it is possible to calculate a set of pairs (sensitivity, 1-specificity) which can be plotted in a curve.  This curve will be the ROC curve for the system where the y-axis (ordenades) represents the sensitivity and the x-axis (abscissas) represents the complement of specificity (1 - specificity).

Examples of ROC Curves are presentend below.

classifiers
Examples of ROC Curves (From: CezarSouza's Blog)

As you can see in the figure above, higher the ROC curve's area, better the system.  This is a standard measure for classifiers comparison, which is called the area under the ROC Curve (AUC). It is evaluated by a numerical integration, such as, for example, the trapezoidal rule.

Now let's go to the development of a simple ROC curve in Python.  During a data mining project that I worked on last year, I decided to implement a ROC curve in order to analyze the performance of some classifiers.  This work resulted into a open-source library called PyROC which you can use inside your own applications for the creation, visualization and analysis of ROC curves.

Source Code

The project is hosted at my personal repository at GitHub.

Feel free to download the source code and use in your applications.


Using the code


To start using the code in your projects, just create a new ROCData object passing as argument the list of tuples containing the actual data, as measure by the experiment, and the predicted data as given by the prediction classsifier.  The actual data must be a dichotomous variable, with only two possible valid values. Generally it is assumed the values 0 and 1 for represent the two states. For the test data, given by the classifier their values must be continuous and the range between the highest and lowest values considered in the actual data. For instance, if the values are 0 and 1, the test data values must be inside the [0,1] range.

from pyroc import *
random_sample  = random_mixture_Model()  # Generate a custom set randomly
print random_sample[(1, 0.53543926503331496), (1, 0.50937533997469853), (1, 0.58701681878005862), (1, 0.57043399840000497),
(1, 0.56229469766270523), (1, 0.6323079028948545), (1, 0.72283523937059946), (1, 0.55079104791257383),
(1, 0.59841921172330748), (1, 0.63361144887035825)]
 
To compute the area under the curve (AUC) and plot the ROC curve just call the RocData object's methods auc() and plot() . You can also pass the desired number of points to use  for different cutoff values.

#Example instance labels (first index) with the decision function , score (second index)
#-- positive class should be +1 and negative 0.
roc = ROCData(random_sample)  #Create the ROC Object
roc.auc() #get the area under the curve
0.93470000000000053
roc.plot(title='ROC Curve') #Create a plot of the ROC curve
Some metrics are also available evaluated from the confusion matrix such as accuracy, specificity, sensitivity, etc.


#threshold passed as parameter - the cutoff value. (0.5) 
roc.confusion_matrix(0.5)
{'FP': 18, 'TN': 82, 'FN': 18, 'TP': 82}
roc.confusion_matrix(0.5,True)
         Actual class
        +(1)    -(0)
+(1)    82      18      Predicted
-(0)    18      82      class
{'FP': 18, 'TN': 82, 'FN': 18, 'TP': 82}
r1.evaluateMetrics(r1.confusion_matrix(0.5))
{'ACC': 0.81000000000000005, 'PHI': 0.62012403721240439, 'PPV': 0.81632653061224
492, 'SENS': 0.80000000000000004, 'NPV': 0.80392156862745101, 'SPEC': 0.81999999
999999995, 'EFF': 0.81000000000000005}


At least if you wanna plot two curves in the same chart, it is also possible by passing a list of RocData objects. It is useful when you want to compare different classifiers.

x = random_mixture_model()
r1 = ROCData(x)
y = random_mixture_model()
r2 = ROCData(y)
lista = [r1,r2]
plot_multiple_roc(lista,'Multiple ROC Curves',include_baseline=True)
 

There is also another option for you to use the script : stand-alone. Just run at the terminal the command below to see  the available options.

Run at your terminal python pyroc.py


Demonstration


Let's see some ROC curves plotted in action.


 I've finished the second part of the performance analysis on machine learning classifiers. This is a simple introduction, so if you want a better understanding of how ROC curves work and its meaning, please take a look at this website about ROC curves and its applications. It includes many applets for you experiment with the curves. I expect you have enjoyed this post. The next one in this series will be about the F1-Score also known as F-measure.



74 comments:

  1. Thanks a lot. Your post is very instructive and clear. Curiosuly, it's not "confusion" at all :-))

    ReplyDelete
  2. oh so good .... I like your blogger post because you talking about Free Article Directory and i like any thing or any post talking about it as Mixed Economic Systems so i will be happy if your visit my site Articles2Day.ORG

    ReplyDelete
  3. Whats your opinion on Multiclass ROC curves?

    ReplyDelete
  4. Sir, i am not getting what should be the input to ROCData. I am using naive bayes classifier and wnat to plot its ROC curve between TP and FP.
    So the input to ROC curve will be a csv file which contains 3 columns i.e. real class, predicted class and predicted class probability?
    Please help me out.

    ReplyDelete
  5. Here you can sell and buy both new and used products.
    This website is perfect for selling just about anything at all.
    More at free advertising online

    ReplyDelete
  6. WIZTECH Automation, Anna Nagar, Chennai, has earned reputation offering the best automation training in Chennai in the field of industrial automation. Flexible timings, hands-on-experience, 100% practical. The candidates are given enhanced job oriented practical training in all major brands of PLCs (AB, Keyence, ABB, GE-FANUC, OMRON, DELTA, SIEMENS, MITSUBISHI, SCHNEIDER, and MESSUNG)

    PLC training in chennai
    Automation training in chennai
    Best plc training in chennai
    PLC SCADA training in chennai
    Process automation training in chennai
    Final year eee projects in chennai
    VLSI training in chennai

    ReplyDelete
  7. Embedded system training: Wiztech Automation Provides Excellent training in embedded system training in Chennai - IEEE Projects - Mechanical projects in Chennai. Wiztech provide 100% practical training, Individual focus, Free Accommodation, Placement for top companies. The study also includes standard microcontrollers such as Intel 8051, PIC, AVR, ARM, ARMCotex, Arduino, etc.

    Embedded system training in chennai
    Embedded Course training in chennai
    Matlab training in chennai
    Android training in chennai
    LabVIEW training in chennai
    Robotics training in chennai
    Oracle training in chennai
    Final year projects in chennai
    Mechanical projects in chennai
    ece projects in chennai

    ReplyDelete


  8. Excellent blogs!!!!you have for sharing them effect information..we developer very learning to easy


    Function Point Estimation Training

    ReplyDelete
  9. Thanks for sharing the article, its really useful. Keep updating more with us.

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. Great information you are shared, Thanks for sharing most valuable information. keep share on Mulesoft Online Training

    ReplyDelete
  13. the blog is good and Interactive it is about Mulesoft Developer it is useful for students and Mulesoft Developers for more updates on Mulesoft mulesoft Online course hyderabad

    ReplyDelete
  14. Great post dear. It definitely has increased my knowledge on R Programming. Please keep sharing similar write ups of yours. You can check this too for R Programming tutorial as i have recorded this recently on R Programming. and i'm sure it will be helpful to you.https://www.youtube.com/watch?v=rgFVq_Q6VF0

    ReplyDelete
  15. What a fantastic read on Data Science. This has helped me understand a lot in Data Science course. Please keep sharing similar write ups on Data Science. Guys if you are keen to know more on Data Science, must check this wonderful Data Science tutorial and i'm sure you will enjoy learning on Data Science training.:-https://www.youtube.com/watch?v=gXb9ZKwx29U&t=237s

    ReplyDelete
  16. Worthful Data Science tutorial. Appreciate a lot for taking up the pain to write such a quality content on Data Science course. Just now I watched this similar Data Science tutorial and I think this will enhance the knowledge of other visitors for sure. Thanks anyway.:-https://www.youtube.com/watch?v=gXb9ZKwx29U&t=237s

    ReplyDelete
  17. Really good information to show through this blog. I really appreciate you for all the valuable information that you are providing us through your blog. python Online course

    ReplyDelete

  18. Very Impressive ROC Curve Data Science tutorial. The content seems to be pretty exhaustive and excellent and will definitely help in learning ROC Curve Data Science. I'm also a learner taken up ROC Curve Data Science training and I think your content has cleared some concepts of mine. While browsing for ROC Curve tutorials on YouTube i found this fantastic video on ROC Curve. Do check it out if you are interested to know more.:-https://www.youtube.com/watch?v=G_pvQYUm8Ik

    ReplyDelete

  19. Great presentation of ROC Curve Data Science form of blog and ROC Curve Data Science tutorial. Very helpful for beginners like us to understand ROC Curve Data Science course. if you're interested to have an insight on ROC Curve Data Science training do watch this amazing tutorial.:-https://www.youtube.com/watch?v=G_pvQYUm8Ik

    ReplyDelete
  20. Currently Python is the most popular Language in IT. Python adopted as a language of choice for almost all the domain in IT including Web Development, Cloud Computing (AWS, OpenStack, VMware, Google Cloud, etc.. ),Read More

    ReplyDelete
  21. myTectra the Market Leader in Artificial intelligence training in Bangalore
    myTectra offers Artificial intelligence training in Bangalore using Class Room. myTectra offers Live Online Design Patterns Training Globally.Read More

    ReplyDelete
  22. Enjoyed reading the article above, really explains everything in detail, the article is very interesting and effective. Thank you and good luck for the upcoming articles learn python training in Bangalore

    ReplyDelete
  23. Enjoyed reading the article above, really explains everything in detail, the article is very interesting and effective. Thank you and good luck for the upcoming articles Python Programming Course

    ReplyDelete
  24. iSprout Business Center is a home for inspiring workspaces created to match the needs of ever-growing business enterprises. From a single dedicated space for a freelancer to custom-made corners for established corporate, there is room for everyone. To name a few we offer private office spaces, coworking space in Hyderabad, single desks, virtual offices, conference rooms, everything with flexible plans which suit your pockets. Come join the hub of dreamers, workers and achievers! With a suite of amenities to offer, it just gets better!
    Gone are the days when just any office space would do. Today, employee productivity & office ecosystem are greatly influenced by the functional and aesthetic standards of the workstation. office space for rent in hyderabad is here to redefine the art, science, and the future of “plug-n-play office space for rent in Hyderabad, Vijayawada and Chennai.”

    office space in hyderabad
    office space for startups in hyderabad

    office space in vijayawada

    private office space for rent

    coworking space in chennai

    best coworking space in chennai

    business centre in hyderabad

    ReplyDelete
  25. Next from Kia is the KIA SONET, comes with wild design, fantastic build quality, thundering speed & what not! Kia sonet features are truly incredible. Know Kia Sonet on road Price in Hyderabad CarKia 36 @ Jubliee Hills.

    ReplyDelete
  26. This professional hacker is absolutely reliable and I strongly recommend him for any type of hack you require. I know this because I have hired him severally for various hacks and he has never disappointed me nor any of my friends who have hired him too, he can help you with any of the following hacks:

    -Phone hacks (remotely)
    -Credit repair
    -Bitcoin recovery (any cryptocurrency)
    -Make money from home (USA only)
    -Social media hacks
    -Website hacks
    -Erase criminal records (USA & Canada only)
    -Grade change
    -funds recovery

    Email: onlineghosthacker247@ gmail .com

    ReplyDelete
  27. Thanks for sharing good information about Tools for Machine Learning Performance Evaluation. It was Clear,concise and helpful for various issues. Coworking Office Space in Noida

    ReplyDelete

  28. Thanks for sharing good information about The Future of Coworking. It is extremely helpful. Coworking Office in Noida

    ReplyDelete
  29. Thanks for sharing such an amazing content. Really loved to read such content. Keep posting such content in future as well. Well, Qdesq is a platform which enables you to book hastle free coworking space, meeting room, serviced office space, virtual office, and many more.

    ReplyDelete
  30. I havent any word to appreciate this post…..Really i am impressed from this post….the person who create this post it was a great human..thanks for shared this with us. 바카라사이트

    ReplyDelete
  31. 카지노사이트 Wonderful and useful submit. I found this much helpful

    ReplyDelete
  32. 스포츠토토 Outstanding post, keep up with this beneficial perform.

    ReplyDelete
  33. 토토사이트 Read a lovely post from you… This is very helpful for me keep up the good work….

    ReplyDelete
  34. Why couldn't I have the same or similar opinions as you? T^T I hope you also visit my blog and give us a good opinion. 안전놀이터

    ReplyDelete
  35. When did it start? The day I started surfing the Internet to read articles related to . I've been fond of seeing various sites related to 안전놀이터 around the world for over 10 years. Among them, I saw your site writing articles related to and I am very satisfied.

    ReplyDelete
  36. I accidentally searched and visited your site. I still saw several posts during my visit, but the text was neat and readable. I will quote this post and post it on my blog. Would you like to visit my blog later? 메이저놀이터추천

    ReplyDelete
  37. Thank you for sharing this Blog, it really good to read.
    For Hybrid Workplace: Coworking space in Bangalore

    ReplyDelete
  38. This post is quite informative. I love reading your blogs. Quite helpful. Are you also searching for Academic content writing services we are the best solution for you. We are best known for delivering the best services to students without having to break the bank.

    ReplyDelete
  39. Your blogs are great.Are you also searching for nursing pico writing help? we are the best solution for you. We are best known for delivering nursing writing services to students without having to break the bank.

    ReplyDelete
  40. Good blog. Keep sharing. I love them Are you also searching for Cheap assignments? we are the best solution for you. We are best known for delivering writing services to students without having to break the bank

    ReplyDelete
  41. Having a office can boost your productivity because it gives you the positive environment to work and of course when you sit on a chair with a proper working mindset then it feels even more product and excellent to work, well for all those who are looking for the CP Coworking Space you can visit here to check out the best available options for you.

    ReplyDelete
  42. It has fully emerged to crown Singapore's southern shores and undoubtedly placed her on the global map of residential landmarks. I still scored the more points than I ever have in a season for GS. I think you would be hard pressed to find somebody with the same consistency I have had over the years so I am happy with that. 메이저토토사이트

    ReplyDelete
  43. Your article has answered the question I was wondering about! I would like to write a thesis on this subject, but I would like you to give your opinion once :D totosite

    ReplyDelete
  44. This comment has been removed by the author.

    ReplyDelete
  45. This comment has been removed by the author.

    ReplyDelete
  46. Very Nice and Knowledgeable content. Will create a lot among the people. In case you are looking for Best Co Working Space in Noida sector 63 with all basic amenities, visit us.

    ReplyDelete
  47. what i do now not comprehended is absolutely how토토사이트추천

    ReplyDelete
  48. Thanks for sharing this information.
    For co-working space in bangalore . HUSTLEHUB provides a wide range of residential properties in bangalore.

    ReplyDelete