Pages

Tools for Machine Learning Performance Evaluation: Confusion Matrix

Tuesday, August 31, 2010



Hi all, 

I'll start to write some posts starting from now about Supervised and Unsupervised learning, specific related to performance evaluation such as classification accuracy, lift, roc curves, F1-Score and errors.

The Confusion Matrix

Let's start with the one popular tools to evaluate the performance of a model in tasks of classification or prediction:  The confusion matrix (in unsupervised learning it is typically called a matching matrix). Its focus is on the predictive capability of a model  rather than how fast the model takes to perform the classification, scalability, etc.

The confusion matrix is represented by a matrix which each row represents the instances in a predicted class, while each column represents in an actual class. One of the advantages of using this performance evaluation tool is that the data mining analyzer can easily see if the model is confusing two classes (i.e. commonly  mislabeling one as another).

The matrix also shows the accuracy of the classifier as the percentage of correctly classified patterns in a given class divided by the total number of patterns in that class. The overall (average) accuracy of the classifier is also evaluated by using the  confusion matrix.

Let's see a confusion matrix in action by showing an example. Imagine that you have a dataset that consists of 33 patterns that are 'Spam' (S) and 67 patterns that are 'Non-Spam' (NS).  For a classifier trained with this dataset to classify an e-mail as 'Spam' or 'Non-Spam', we can use the confusion matrix to see the classification accuracy based on the training data. In the example confusion matrix below, of the 33 patterns that are 'Spam' (S),  27 were correctly predicted as 'Spams' while 6 were incorrectly predicted as 'Non-Spams' (NB) (achieving an accuracy of 81.8%).  On the other hand, of the 67 patterns that are 'Non-Spams', 57 are correctly predicted as 'Non-Spams' while 10 were incorrectly classified as 'Spams' (an accuracy of 85.1%).  The overall accuracy of the classifier  for predicting both classes given this dataset is evaluated achieving 83%.

Confusion Matrix on spam classification model

However the confusion matrix only tell us how the classifier is behaving for individual classes. When a data set is unbalanced (where the number of samples in one class is significantly more than that in the other class - it happens a lot with Spam/Non-Spam datasets) the accuracy evaluated of a classifier is not representative of the true performance of the classifier. For instance, imagine there are 990 patterns that are 'Non Spam'  and only 10 patterns that are 'Spam' , the classifier can easily be biased towards the class 'Non Spam'.  If the model classifies all the samples as 'Non-Spam', the accuracy will be 99%.  And this is not real indication of the classification's performance. The classifier has a 100% recognition rate for 'Non-Spam'  but a 0% error rate for 'Spam'. Looking at the matrix, the system has trouble in predicting the 'Spam' class, even though the system has to be 99% accurate in its prediction. Given that the prediction of 'Spam' class would be the one of actual interest, only using the confusion matrix to evaluate the model's performance is not enough, but it can give us an insight of how the model is predicting the classes and start to use other metrics that we will explain in the next section.

Confusion Matrix on a unbalanced dataset


The Table of  Confusion

In the Confusion Matrix, for each cell in the matrix we have fields as True Positives, False Positives, False Negatives and True Negatives.  These are defined as:
  • False Positive (FP):  Falsely Predicting a label (or saying that Non-Spam is a Spam).
  • False Negative (FN):  Missing and incoming label (or saying a Spam is Non-Spam).
  • True Positive (TP):  Correctly predicting a label (or saying a Spam is Spam).
  • True Negative (TN): Correctly predicting the other label (or saying Non-Spam is Non-Spam).

Looking at the confusion matrix in a general view is as follows:

Confusion Matrix
 
How can we use those metrics ?  For instance, let's consider the previous model now for predicting if a text message have positive or negative opinion associated (common in sentiment analysis task).  We have a data set with 10.000 text messages where the model correctly predicts 9.700 negative messages, and 100 positive messages. The model still incorrectly predicts 150 messages which are positive to be negative, and 50 messages which are negative to be positive.  The resulting Confusion Matrix is shown below.

Confusion Matrix on Sentiment classification task


For the binary classification problems, which was our case situation , we can derive from those metrics two equations called sensitivity and specificity. They are commonly used for the evaluation of any binary classifier. 

The Specificity (TNR) measures the proportion of messages that are negative (TN) of all the messages that are actually negative (TN+FP). It can be looked at as the probability that the message is classified as negative given that the message does not contain negative words. With higher specificity, fewer positive messages are labeled as negative.

On the other hand, Sensitivity (TPR) is the proportion of messages that are positive (TP) of all the messages that are actually positive (TP+FN).  It can be seen as the probability that the message is positive given that the patient contain positive words. With higher sensitivity, fewer actual messages will be classified as negative.  

Sensitivity can be expressed as :
  • TP / (TP+FN)
and then Specificity which is:
  • TN / (TN+FP)

In general here, Sensitivity means the accuracy on the class Negative, and Specificity means the accuracy on the class Positive. So using these metrics, what is the accuracy on Positive and Negative messages  ?
  • Sensitivity = TP / (TP+FN) = 100/(100+50) = 0.4 = 40% 
  • Specificity = TN / (TN+FP) = 9700/(9700+150) = 0.98 = 98% 

As you can see, if we have a test for sentiment classification with 40% sensitivity and 98% specificity, and we have to check 1000 messages, and 500 of them are positive and 500 are negative. You are likely to get about 200 messages true positives, 300 messages false negatives,  490 true negatives and 10 false positives. You can conclude that the the negative prediction is more confident, specially based on the high value of specificity and the low level of sensitivity. As you can see it's a important metric for analyzing the performance of your classifier only looking both separated.
The relationship between sensitivity and specificity, as well as the performance of the classifier, can be visualized and studied using the ROC curve, which it will be one of the next posts about this topic.

I've developed some code in Python for evaluating the Confusion Matrix, Specificity and Sensitivity of a classifier here.  Please make the necessary changes for adapting for your classifier. 

That's all,

I expect you have enjoyed!

Cheers,

Marcel Caraciolo


References

142 comments:

  1. Great script. We'll be waiting for your comments about ROC curve.

    Muito obrigado Marcel.

    ReplyDelete
    Replies
    1. Hello Everyone !

      USA SSN Leads/Dead Fullz available, along with Driving License/ID Number with good connectivity.

      All SSN's are Tested & Verified.

      **DETAILS IN LEADS/FULLZ**

      ->FULL NAME
      ->SSN
      ->DATE OF BIRTH
      ->DRIVING LICENSE NUMBER
      ->ADDRESS WITH ZIP
      ->PHONE NUMBER, EMAIL
      ->EMPLOYEE DETAILS

      *Price for SSN lead $2
      *You can ask for sample before any deal
      *If you buy in bulk, will give you discount
      *Sampling is just for serious buyers

      ->Hope for the long term business
      ->You can buy for your specific states too

      **Contact 24/7**

      Whatsapp > +923172721122

      Email > leads.sellers1212@gmail.com

      Telegram > @leadsupplier

      ICQ > 752822040

      Delete
  2. hello you made a mistake in your post regarding the example:

    Specificity should be :
    9700/(9700+50) = 0.9948 ( 99.5% )

    Best.

    ReplyDelete
  3. Sensitivity means the accuracy on the class Negative, and Specificity means the accuracy on the class Positive. So using these metrics, what is the accuracy on Positive and Negative messages ?

    is not on the contrary, Positive/Negative?

    ReplyDelete
  4. i think the values in your picture are in the wrong places, the 150 & 50 should swap

    ReplyDelete
  5. Sensitivity = TP / (TP+FN) = 100/(100+50) = 0.4 = 40%
    This calculation is wrong and should be 2/3 or 67%.

    ReplyDelete
  6. Hello,

    You might be interested by my project https://github.com/scls19fr/pandas_confusion and its Pip package https://pypi.python.org/pypi/pandas_confusion

    With this package confusion matrix can be pretty-printed, plot. You can binarize a confusion matrix, get class statistics such as TP, TN, FP, FN, ACC, TPR, FPR, FNR, TNR (SPC), LR+, LR-, DOR, PPV, FDR, FOR, NPV and some overall statistics

    Kind regards

    ReplyDelete
  7. Nice blog with having good information. Its very useful for everyone. Thanks and keep posting this type of blog.

    Data Visualization Training Institutes in Chennai Trichy

    ReplyDelete
  8. Some time I wonder whether I will get the information I am looking for. But you have given complete information which I required.
    MATLAB Training In Noida

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. Machine performance evaluation is an important aspect on determining the output of the machine and tools make the calculation process easier. The writer has explained the tools clearly and in a professional manner and if you require more blog articles that have been written in a professional manner, feel free to hire our writers by clicking on Paraphrasing a Research Proposal.

    ReplyDelete
  11. Hi,

    Aren't sensitivity and specificity are synonym of recall and precision (saw this on udacity course) ?

    Thanks

    ReplyDelete
  12. This concept is a good way to enhance the knowledge.thanks for sharing. please keep it up
    salesforce Online Training Hyderabad

    ReplyDelete
  13. Management Tutors is one of the Assignment Help UK that has been successfully rendering amazing academic assistance to the students.
    Live Chat @ https://www.managementtutors.com/
    Read More @ Assignment Help

    Uk

    Assignment Help Malaysia
    Management

    Assignment Help

    ReplyDelete
  14. Excellent article. Very interesting to read. I really love to read such a nice article. Thanks! keep rocking. Data Science online Training

    ReplyDelete
  15. Live Web Experts is a remarkable academic portal that is known for offering best regulation on Accounting Homework Topic to the scholars at a meagre price.
    https://www.livewebexperts.com/assignment-help/accounting-homework-help

    Assignment Help Florida
    Online Tutoring Help California
    Essay Writing Services Illinois

    ReplyDelete
  16. Dissertation Writing Services Ashford - Help in Dissertation is a remarkable academic portal that is known for offering extensive educational solutions. Parents can even Dissertation Writing Services Ashford.

    Live Chat @ https://www.helpindissertation.co.uk/dissertation-services

    Read More @

    Dissertation Help Aberdeen UK

    Buy Essay Online Belfast

    Dissertation Services Belfast

    ReplyDelete
  17. Nursing Assignment Writer Kansas – The Best Tutors is a reliable academic portal that is known to offer exceptional educational assistance to the nursing assignment services students at a small price. Live Chat @ https://www.besttutors.us/medical-assignment-help

    Read More @ Nursing Assignment Writer Kansas
    Software Engineering Homework Help Alaska

    ReplyDelete
  18. Australia Best Tutor is available providing Help with Assignment Services Melbourne that help the students in preparing a relevant and innovative assignment paper. These experts are highly efficient and well trained.

    Live Chat @ https://www.australiabesttutor.com

    Read More About

    Assignment Help Melbourne
    Help with Assignment Melbourne
    Mathematics Assignment Help Melbourne
    Engineering Assignment Help Melbourne

    ReplyDelete
  19. Nursing Assignment Writer Arkansas – The Best Tutors is a best academic portal that is known for success help in completing assignment services to the students. Live Chat @ https://www.besttutors.us/medical-assignment-help

    Read More @ Forensic Accounting Homework Help Alaska
    Pre- Calculus Homework Help Alabama

    ReplyDelete
  20. My Genius Mind is a reputed academic concern that extends best professional Assignment Help to the students. This service can be availed at any time anywhere.
    Live Chat @ https://www.mygeniusmind.com/my-assignment-help
    Read More Information @ Accounting Assignment Help Brisbane
    Boston South Australia Assignment Help
    Mandurah Western Australia Assignment Help
    NewCastle NSW Assignment Help

    ReplyDelete
  21. Management Tutors, known for offering fantastic coursework help to the students. It is beneficial to the students in the achievement of their goals and getting good grades by the teachers.
    Live Chat @ https://www.managementtutors.com/professional-help-with-assignment-uk
    Read More @ Dissertation Help Services Adelaide
    Business Management Help Melbourne
    Project Management Help Queensland
    Operational Management Assignment Brisbane

    ReplyDelete
  22. Best Assignment Help NSW – Help with Assignments is best My Assignment custom firms make sure to send unique content. They assume severe research and provide sincere work permanently.
    Live Chat @ https://www.helpwithassignments.com/

    Read More @ Business Finance Assignment Help Victoria
    University Nursing Assignment Help Queensland
    Homework Help Services Victoria

    ReplyDelete
  23. The Best Tutors is a remarkable academic portal that is known for offering online dissertation services responses to the subtle students.Live Chat @ Essay Writing Services Arizona

    Read More @ Nursing Assignment Writer Arkansas
    Assignment Tutoring Help Alabama

    ReplyDelete
  24. Australia Best Tutor is offering online assignment help services Australia at affordable price. Here students are joining for best academics grades and good quality content.Services are under below

    Engineering Assignment Help Brisbane
    Management Assignment Help Brisbane
    Assignment Writing Services Brisbane
    Nursing Assignment Help Perth
    Finance Assignment Help Perth

    Live Chat @ https://www.australiabesttutor.com

    ReplyDelete
  25. UK Best Tutor is one of the famous and trustworthy academic portals that is known to provide the students with Assignment Help Scotland directly and successfully.
    Live Chat @ https://www.ukbesttutor.co.uk/

    Read More @ Online Assignment Writing UK
    University Assignment Help UK
    Assignment Help Tutor England

    ReplyDelete
  26. This is a very nice article. thank you for publishing this. i can understand this easily.!!Data Science online Course

    ReplyDelete
  27. Help with Assignments and offers impressive Help Engineering to the students in various subjects. When such tasks are submitted, the students are bound to get high marks or scores.
    Live Chat @ https://www.helpwithassignments.com/engineering-assignment-help

    Read More @ Marketing Management Assignment Help Queensland
    Help with Accounting Assignment Help Australian Territory
    Pay for Finance Assignment Help Queensland

    ReplyDelete
  28. PCB Design Training in Bangalore offered by myTectra. India's No.1 PCB Design Training Institute. Classroom, Online and Corporate training in PCB Design
    pcb design training in bangalore

    ReplyDelete
  29. Thanks for sharing this blog post,Nice written skill machine learning online training

    ReplyDelete
  30. Thanks for sharing this blog post,Nice written skill machine learning online training

    ReplyDelete
  31. Best Tutors, The students can obtain help from the professional and expert authorities with abundant knowledge, which is given by us as a reliable and popular engineering homework help earner.Live Chat @ https://www.besttutors.us/engineering-homework-help

    Read More @ Online Homework Help Colorado
    Geometry Homework Help Colorado
    Finance Homework Help Colorado
    Online Homework Help Arkansas

    ReplyDelete
  32. Thanks a lot for sharing us about this update. Hope you will not get tired on making posts as informative as this. 
    Best Devops training in sholinganallur
    Devops training in velachery
    Devops training in annanagar
    Devops training in tambaram

    ReplyDelete
  33. This is a nice article here with some useful tips for those who are not used-to comment that frequently. Thanks for this helpful information I agree with all points you have given to us. I will follow all of them.
    python Training in Chennai
    python Training in Bangalore
    python Training in Pune

    ReplyDelete
  34. Very Helpful and informative blog! Keep sharing such blogsSoftware Development Company in India

    ReplyDelete
  35. Great post you have there. It has enlightened me and added knowledge about the subject. You can also look at help with SPSS analysis .

    ReplyDelete
  36. Assignment Help UK assistance is provided to UK students. It is a legal and reliable company worldwide.

    ReplyDelete
  37. Machine Learning Online Training Get the knowledge of an in-depth overview of machine learning online training topics with real-time data from the best machine learning course provided by Evantatech.

    ReplyDelete
  38. This is excellent information. It is amazing and wonderful to. visit our site machine learning online training!

    ReplyDelete
  39. This professional hacker is absolutely reliable and I strongly recommend him for any type of hack you require. I know this because I have hired him severally for various hacks and he has never disappointed me nor any of my friends who have hired him too, he can help you with any of the following hacks:

    -Phone hacks (remotely)
    -Credit repair
    -Bitcoin recovery (any cryptocurrency)
    -Make money from home (USA only)
    -Social media hacks
    -Website hacks
    -Erase criminal records (USA & Canada only)
    -Grade change
    -funds recovery

    Email: onlineghosthacker247@ gmail .com

    ReplyDelete
  40. Nero Platinum Suite 2021 Crack is a popular and all-in-one application for burning CD / DVD and Blu-ray discs, as well as burning high-quality videos and movies. It is a huge suite with various tools and features that allow users to edit their images, manage their photo library, extract and burn, stream and convert, create and edit videos and much more. Nero Platinium Suite Crack 2020

    ReplyDelete
  41. iot training in chennai - Iot Training in Chennai - Internet of things is an technology which is basically to connect every product with the Internet.
    Definitely Best IOT Training Institute in Chennai will train you in this technology with an assured placements.

    DevOps training in chennai - It's right time to learn the trending technolgy such as DevOps. Try to find a Best DeVops training Institute in Chennai.

    blue prism training in Chennai - Blue prism is an developing and future technology of IT and start to start for Best Blue prism technology in Chennai.

    uipath training in Chennai - Best uipath course and training in Chennai will also create a better path for your future.

    microsoft azure training in chennai - Best azure training in Chennai offering the courses with guidance for Microsoft azure training.

    ReplyDelete
  42. I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article.
    DIS
    HLA

    ReplyDelete
  43. That's really impressive and helpful information you have given, very valuable content.
    We are also into education and you also can take advantage of Selenium Training in Pune

    ReplyDelete
  44. ****Contact Me****
    *ICQ :748957107
    *Gmail :taimoorh944@gmail.com
    *Telegram :@James307


    SELLING Fresh and valid USA ssn fullz
    99% connectivity with quality
    *If you have any trust issue before any deal you may get few to test
    *Every leads are well checked and available 24 hours
    *Fully cooperate with clients
    *Any invalid info found will be replaced
    *Good credit score above 700 every fullz
    *Payment (BTC&Paypal)
    *Fullz will be available according to demand i.e (format,specific state,specific zip code & specifc name etc..)

    *Format of Fullz/leads/profiles
    °First & last Name
    °SSN
    °DOB
    °(DRIVING LICENSE NUMBER)
    °ADDRESS
    (ZIP CODE,STATE,CITY)
    °PHONE NUMBER
    °EMAIL ADDRESS
    °Relative Details
    °Employment status
    °Previous Address


    $2 for each fullz/lead
    (Price can be negotiable if order in bulk)


    OTHER SERVICES ProvIDING

    *(Dead Fullz)
    *(Email leads with Password)

    *(Dumps track 1 & 2 with pin and without pin)

    *Hacking Tutorials
    *Smtp Linux

    *Contact soon!
    *Hope for a long term Business
    *Thank You!

    ****Contact Me****
    *ICQ :748957107
    *Gmail :taimoorh944@gmail.com
    *Telegram :@James307

    ReplyDelete
  45. Here, we take the time in making genuine and counterfeit Driver License. We do convey this Driver License to the locations of our purchasers. fake australian drivers license generator The conveyance date and time will rely upon the area of the purchasers. We measure phony and genuine Driver permit

    ReplyDelete
  46. Bren Group is pleased to present a new development by Bren Group, called Bren Zahara. This project aims to provide the best in class service apartments for people looking for flats in Lisbon

    ReplyDelete
  47. iMazing Crack Excellent piece of work, and I am in wonder how you manage all of these content and his entry. I would like to say you have superb capabilities related to your work, and lastly, please keep it up because I am looking for the more.

    ReplyDelete
  48. Great beat ! I wish to apprentgice at tthe same time as you amend yor site, 바카라사이트 how can i subscribe for a blog website?
    Thee account aided mee a applicable deal. I have been a little bit familiar of this your broadcast provided bright clear concept

    ReplyDelete
  49. Best Digital Marketing Company in world. Also we provide Web Design and Develpment Service, SEO, Social Media Marketing and Other Service.

    ReplyDelete
  50. Best Digital Marketing Company in world. Also we provide Web Design and Development Service, SEO, Social Media Marketing and Other Service.

    ReplyDelete

  51. This is an informative post. Got a lot of info and details from here. Thank you for sharing this and looking forward to reading more of your post.
    meal kit delivery services market

    ReplyDelete
  52. Studyprovider has experts team are giving the homework help, assignment help, report, thesis, research writing services and project management assignment help available 24/7 seven days a week contact now!

    ReplyDelete
  53. 바카라사이트 A good blog always comes-up with new and exciting information and while reading I have feel that this blog is really have all those quality that qualify a blog to be a one

    ReplyDelete
  54. This comment has been removed by the author.

    ReplyDelete
  55. These tools are pretty helpful for checking machine learning performance. The confusion matrix is the most famous tool used for evaluating the performance of machine learning programs. I am working on an article based on machine learning tools, and your post is quite helpful for me.
    Dissertation writing help

    ReplyDelete
  56. The information you have posted is very useful. The sites you have referred was good. Thanks for sharing.
    cyber security certification malaysia

    ReplyDelete
  57. Nice post. Well what can I say is that these is an interesting and very informative topic on medical marijuna

    ReplyDelete
  58. You really make it look so natural with your exhibition however I see this issue as really something which I figure I could never understand. It appears to be excessively entangled and incredibly expansive for me.

    ReplyDelete
  59. 360DigiTMG, the top-rated organisation among the most prestigious industries around the world, is an educational destination for those looking to pursue their dreams around the globe. The company is changing careers of many people through constant improvement, 360DigiTMG provides an outstanding learning experience and distinguishes itself from the pack. 360DigiTMG is a prominent global presence by offering world-class training. Its main office is in India and subsidiaries across Malaysia, USA, East Asia, Australia, Uk, Netherlands, and the Middle East.

    ReplyDelete
  60. This post is so helpfull and informative.keep updating with more information...
    Benefits Of RPA
    Robotics Process Automation Courses

    ReplyDelete
  61. I got numerous academic tasks to complete in a limited time in my educational period. I did not know how to start my economics homework and manage my time. I had no idea about referencing, university guidelines, and resources. Therefore, I searched for expert guidance online. Numerous websites proposed to provide Economics Homework Help from professionals. But, NeedAssignmentHellp seems to be trustworthy assignment assistance. They had client reviews from the USA, The UK, Australia, New Zealand, and many other countries. I saw various work samples on their website. Their team is supportive and delivered my economics assignments according to the requirements. I got excellent work within a week and scored good grades on my result. I recommend everyone to choose them because they provide 24*7 hours of assistance, original work, and expert guidance at affordable prices.

    ReplyDelete
  62. This comment has been removed by the author.

    ReplyDelete
  63. This is very nice information, Thank you so much for sharing your knowledge. Software License Management in Malta

    ReplyDelete

  64. This is a very well-written website. Keep Up The Good Work! Thanks for the Information.
    https://vstoriginal.com/antares-auto-tune-pro-crack/

    ReplyDelete
  65. Thank you for sharing this enlightening content. It's truly appreciated and highly informative!

    learn more about our ai certification course to get the best knowledge

    ReplyDelete
  66. I found this article on tools for machine learning performance incredibly insightful. In today's data-driven world, optimizing machine learning models is crucial for success, and having the right tools at our disposal can make all the difference.Online finance dissertation help serves as a valuable tool for students pursuing finance degrees. It offers them access to expert guidance, research materials, and a structured approach to crafting top-notch dissertations. Much like machine learning tools help streamline complex data analysis, online finance dissertation help. streamlines the research and writing process, enabling students to deliver high-quality work.

    ReplyDelete
  67. Fantastic Blog ! I thoroughly enjoyed reading your insightful blog post about Best Machine Learning training in Noida. Thank you for sharing your knowledge and expertise in such an approachable manner.

    ReplyDelete
  68. https://support.us.edu/hc/en-us/community/posts/27010015008535-Is-Coinbase-24-7-Live-Support

    ReplyDelete