Pages

Tools for Machine Learning Performance Evaluation: Confusion Matrix

Tuesday, August 31, 2010



Hi all, 

I'll start to write some posts starting from now about Supervised and Unsupervised learning, specific related to performance evaluation such as classification accuracy, lift, roc curves, F1-Score and errors.

The Confusion Matrix

Let's start with the one popular tools to evaluate the performance of a model in tasks of classification or prediction:  The confusion matrix (in unsupervised learning it is typically called a matching matrix). Its focus is on the predictive capability of a model  rather than how fast the model takes to perform the classification, scalability, etc.

The confusion matrix is represented by a matrix which each row represents the instances in a predicted class, while each column represents in an actual class. One of the advantages of using this performance evaluation tool is that the data mining analyzer can easily see if the model is confusing two classes (i.e. commonly  mislabeling one as another).

The matrix also shows the accuracy of the classifier as the percentage of correctly classified patterns in a given class divided by the total number of patterns in that class. The overall (average) accuracy of the classifier is also evaluated by using the  confusion matrix.

Let's see a confusion matrix in action by showing an example. Imagine that you have a dataset that consists of 33 patterns that are 'Spam' (S) and 67 patterns that are 'Non-Spam' (NS).  For a classifier trained with this dataset to classify an e-mail as 'Spam' or 'Non-Spam', we can use the confusion matrix to see the classification accuracy based on the training data. In the example confusion matrix below, of the 33 patterns that are 'Spam' (S),  27 were correctly predicted as 'Spams' while 6 were incorrectly predicted as 'Non-Spams' (NB) (achieving an accuracy of 81.8%).  On the other hand, of the 67 patterns that are 'Non-Spams', 57 are correctly predicted as 'Non-Spams' while 10 were incorrectly classified as 'Spams' (an accuracy of 85.1%).  The overall accuracy of the classifier  for predicting both classes given this dataset is evaluated achieving 83%.

Confusion Matrix on spam classification model

However the confusion matrix only tell us how the classifier is behaving for individual classes. When a data set is unbalanced (where the number of samples in one class is significantly more than that in the other class - it happens a lot with Spam/Non-Spam datasets) the accuracy evaluated of a classifier is not representative of the true performance of the classifier. For instance, imagine there are 990 patterns that are 'Non Spam'  and only 10 patterns that are 'Spam' , the classifier can easily be biased towards the class 'Non Spam'.  If the model classifies all the samples as 'Non-Spam', the accuracy will be 99%.  And this is not real indication of the classification's performance. The classifier has a 100% recognition rate for 'Non-Spam'  but a 0% error rate for 'Spam'. Looking at the matrix, the system has trouble in predicting the 'Spam' class, even though the system has to be 99% accurate in its prediction. Given that the prediction of 'Spam' class would be the one of actual interest, only using the confusion matrix to evaluate the model's performance is not enough, but it can give us an insight of how the model is predicting the classes and start to use other metrics that we will explain in the next section.

Confusion Matrix on a unbalanced dataset


The Table of  Confusion

In the Confusion Matrix, for each cell in the matrix we have fields as True Positives, False Positives, False Negatives and True Negatives.  These are defined as:
  • False Positive (FP):  Falsely Predicting a label (or saying that Non-Spam is a Spam).
  • False Negative (FN):  Missing and incoming label (or saying a Spam is Non-Spam).
  • True Positive (TP):  Correctly predicting a label (or saying a Spam is Spam).
  • True Negative (TN): Correctly predicting the other label (or saying Non-Spam is Non-Spam).

Looking at the confusion matrix in a general view is as follows:

Confusion Matrix
 
How can we use those metrics ?  For instance, let's consider the previous model now for predicting if a text message have positive or negative opinion associated (common in sentiment analysis task).  We have a data set with 10.000 text messages where the model correctly predicts 9.700 negative messages, and 100 positive messages. The model still incorrectly predicts 150 messages which are positive to be negative, and 50 messages which are negative to be positive.  The resulting Confusion Matrix is shown below.

Confusion Matrix on Sentiment classification task


For the binary classification problems, which was our case situation , we can derive from those metrics two equations called sensitivity and specificity. They are commonly used for the evaluation of any binary classifier. 

The Specificity (TNR) measures the proportion of messages that are negative (TN) of all the messages that are actually negative (TN+FP). It can be looked at as the probability that the message is classified as negative given that the message does not contain negative words. With higher specificity, fewer positive messages are labeled as negative.

On the other hand, Sensitivity (TPR) is the proportion of messages that are positive (TP) of all the messages that are actually positive (TP+FN).  It can be seen as the probability that the message is positive given that the patient contain positive words. With higher sensitivity, fewer actual messages will be classified as negative.  

Sensitivity can be expressed as :
  • TP / (TP+FN)
and then Specificity which is:
  • TN / (TN+FP)

In general here, Sensitivity means the accuracy on the class Negative, and Specificity means the accuracy on the class Positive. So using these metrics, what is the accuracy on Positive and Negative messages  ?
  • Sensitivity = TP / (TP+FN) = 100/(100+50) = 0.4 = 40% 
  • Specificity = TN / (TN+FP) = 9700/(9700+150) = 0.98 = 98% 

As you can see, if we have a test for sentiment classification with 40% sensitivity and 98% specificity, and we have to check 1000 messages, and 500 of them are positive and 500 are negative. You are likely to get about 200 messages true positives, 300 messages false negatives,  490 true negatives and 10 false positives. You can conclude that the the negative prediction is more confident, specially based on the high value of specificity and the low level of sensitivity. As you can see it's a important metric for analyzing the performance of your classifier only looking both separated.
The relationship between sensitivity and specificity, as well as the performance of the classifier, can be visualized and studied using the ROC curve, which it will be one of the next posts about this topic.

I've developed some code in Python for evaluating the Confusion Matrix, Specificity and Sensitivity of a classifier here.  Please make the necessary changes for adapting for your classifier. 

That's all,

I expect you have enjoyed!

Cheers,

Marcel Caraciolo


References

98 comments:

  1. Great script. We'll be waiting for your comments about ROC curve.

    Muito obrigado Marcel.

    ReplyDelete
  2. hello you made a mistake in your post regarding the example:

    Specificity should be :
    9700/(9700+50) = 0.9948 ( 99.5% )

    Best.

    ReplyDelete
  3. Sensitivity means the accuracy on the class Negative, and Specificity means the accuracy on the class Positive. So using these metrics, what is the accuracy on Positive and Negative messages ?

    is not on the contrary, Positive/Negative?

    ReplyDelete
  4. i think the values in your picture are in the wrong places, the 150 & 50 should swap

    ReplyDelete
  5. Sensitivity = TP / (TP+FN) = 100/(100+50) = 0.4 = 40%
    This calculation is wrong and should be 2/3 or 67%.

    ReplyDelete
  6. Hello,

    You might be interested by my project https://github.com/scls19fr/pandas_confusion and its Pip package https://pypi.python.org/pypi/pandas_confusion

    With this package confusion matrix can be pretty-printed, plot. You can binarize a confusion matrix, get class statistics such as TP, TN, FP, FN, ACC, TPR, FPR, FNR, TNR (SPC), LR+, LR-, DOR, PPV, FDR, FOR, NPV and some overall statistics

    Kind regards

    ReplyDelete
  7. WIZTECH Automation, Anna Nagar, Chennai, has earned reputation offering the best automation training in Chennai in the field of industrial automation. Flexible timings, hands-on-experience, 100% practical. The candidates are given enhanced job oriented practical training in all major brands of PLCs (AB, Keyence, ABB, GE-FANUC, OMRON, DELTA, SIEMENS, MITSUBISHI, SCHNEIDER, and MESSUNG)

    PLC training in chennai
    Automation training in chennai
    Best plc training in chennai
    PLC SCADA training in chennai
    Process automation training in chennai
    Final year eee projects in chennai
    VLSI training in chennai

    ReplyDelete
  8. Embedded system training: Wiztech Automation Provides Excellent training in embedded system training in Chennai - IEEE Projects - Mechanical projects in Chennai. Wiztech provide 100% practical training, Individual focus, Free Accommodation, Placement for top companies. The study also includes standard microcontrollers such as Intel 8051, PIC, AVR, ARM, ARMCotex, Arduino, etc.

    Embedded system training in chennai
    Embedded Course training in chennai
    Matlab training in chennai
    Android training in chennai
    LabVIEW training in chennai
    Robotics training in chennai
    Oracle training in chennai
    Final year projects in chennai
    Mechanical projects in chennai
    ece projects in chennai

    ReplyDelete
  9. Nice blog with having good information. Its very useful for everyone. Thanks and keep posting this type of blog.

    Data Visualization Training Institutes in Chennai Trichy

    ReplyDelete
  10. Some time I wonder whether I will get the information I am looking for. But you have given complete information which I required.
    MATLAB Training In Noida

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. Machine performance evaluation is an important aspect on determining the output of the machine and tools make the calculation process easier. The writer has explained the tools clearly and in a professional manner and if you require more blog articles that have been written in a professional manner, feel free to hire our writers by clicking on Paraphrasing a Research Proposal.

    ReplyDelete
  13. Hi,

    Aren't sensitivity and specificity are synonym of recall and precision (saw this on udacity course) ?

    Thanks

    ReplyDelete
  14. PLC Training in Delhi Ncr. We DIAC Automation would like to introduce ourselves as leading company providing Placement Program in Advanced Industrial Automation Training and Process Automation Training for industries. Call @9310096831

    ReplyDelete

  15. Thanks, Learned a lot of new things from your post! Good creation and HATS OFF to the creativity of your mind.
    Very interesting and useful blog!
    matlab Training in Gurgaon

    ReplyDelete
  16. This concept is a good way to enhance the knowledge.thanks for sharing. please keep it up
    salesforce Online Training Hyderabad

    ReplyDelete
  17. My Genius Mind is a renowned academic portal that offers impressive academic support to the students.
    Live Chat @ https://www.mygeniusmind.com/
    Read More @
    Western Australia Assignment Help
    Writing Services Melbourne
    Australia Best Tutor
    Hobart Tasmania Assignment Help

    ReplyDelete
  18. Management Tutors is one of the Assignment Help UK that has been successfully rendering amazing academic assistance to the students.
    Live Chat @ https://www.managementtutors.com/
    Read More @ Assignment Help

    Uk

    Assignment Help Malaysia
    Management

    Assignment Help

    ReplyDelete
  19. My Genius Mind is a reputed educational facility providing portal that has been offering high-end Assignment Services Melbourne solutions.
    Live Chat @ https://www.mygeniusmind.com/new-south-wales-assignment-help

    Read More @ Brisbane Queensland Assignment Help
    Instant Assignment Help Tasmania
    Business Management Assignment Help

    ReplyDelete
  20. Excellent article. Very interesting to read. I really love to read such a nice article. Thanks! keep rocking. Data Science online Training

    ReplyDelete
  21. Live Web Experts is a remarkable academic portal that is known for offering best regulation on Accounting Homework Topic to the scholars at a meagre price.
    https://www.livewebexperts.com/assignment-help/accounting-homework-help

    Assignment Help Florida
    Online Tutoring Help California
    Essay Writing Services Illinois

    ReplyDelete
  22. Dissertation Writing Services Ashford - Help in Dissertation is a remarkable academic portal that is known for offering extensive educational solutions. Parents can even Dissertation Writing Services Ashford.

    Live Chat @ https://www.helpindissertation.co.uk/dissertation-services

    Read More @

    Dissertation Help Aberdeen UK

    Buy Essay Online Belfast

    Dissertation Services Belfast

    ReplyDelete
  23. My Genius Mind is a well renowned academic portal that so known to offer amazing Management Assignment Help support to the students.

    Contact Information
    My Genius Mind
    Victoria, Melbourne, Australia
    Live Chat @ https://www.mygeniusmind.com/management-assignment-help

    Read More About @ Assignment Provider Canberra

    Calculus Assignment Help Adelaide

    Medical Assignment Help Perth

    ReplyDelete
  24. Economics assignment answers NSW – Help with Assignments is a well-known academic portal that offers high-end economics answer help to the universities and college students. Live Chat @ https://www.helpwithassignments.com/management-assignment-help

    Read More @ Help with Homework Australia
    Algebra Assignment Solver
    Nursing Homework Assignment Victoria

    ReplyDelete
  25. Amellie Taylor is a renowned academician writing on a series of academic topics. He is associated with the UK Best Tutor, a reliable academic portal. Amellie Taylor is the title of today’s content.
    Live Chat @ https://www.ukbesttutor.co.uk/our-services/dissertation-help-services

    Read More @ Online Homework Help Bristol
    Best Assignment Help Bristol
    Online Essay Writing Bristol

    ReplyDelete
  26. Nursing Assignment Writer Kansas – The Best Tutors is a reliable academic portal that is known to offer exceptional educational assistance to the nursing assignment services students at a small price. Live Chat @ https://www.besttutors.us/medical-assignment-help

    Read More @ Nursing Assignment Writer Kansas
    Software Engineering Homework Help Alaska

    ReplyDelete
  27. Webtrackker is one of the best Institute in Noida. In this Institutes we have offered to a student to get more and more information for your subject with both parts like theory and practise.

    Contact Us Information

    Webtrackker Technology
    C - 67, Sector- 63, Noida - 201301 (Delhi-NCR), INDIA

    Our Services

    Oracle dba training in Noida
    Python Training in Noida
    Salesforce training in Noida
    SAS Training in Noida
    IOT Training in Noida
    Devops training in Noida

    ReplyDelete
  28. The expert Assignment Help Tutor England with the portals prepare a rough draft initially for the clients to consider. At this stage, modifications can be made if required. Once everything has been found to be good, the expert writes the final content.
    Live Chat @ https://www.ukbesttutor.co.uk/

    Read More @ Civil Engineering Assignment Help London
    Best Assignment Help Bristol
    Physics Assignment Help England

    ReplyDelete
  29. Australia Best Tutor is available providing Help with Assignment Services Melbourne that help the students in preparing a relevant and innovative assignment paper. These experts are highly efficient and well trained.

    Live Chat @ https://www.australiabesttutor.com

    Read More About

    Assignment Help Melbourne
    Help with Assignment Melbourne
    Mathematics Assignment Help Melbourne
    Engineering Assignment Help Melbourne

    ReplyDelete
  30. My Homework Help Services Australian Territory – Help with Assignments is a reliable academic company offering high-end direction and help to the students pursuing homework help at a very reasonable price.
    Live Chat @ https://www.helpwithassignments.com/homework-help-services

    Read More @ Economics Help Online Victoria

    Management Assignment Help Victoria

    ReplyDelete
  31. My Homework Help Services Australian Territory – Help with Assignments is a reliable academic company offering high-end direction and help to the students pursuing homework help at a very reasonable price.
    Live Chat @ https://www.helpwithassignments.com/homework-help-services

    Read More @ Economics Help Online Victoria

    Management Assignment Help Victoria

    ReplyDelete
  32. Nursing Assignment Writer Arkansas – The Best Tutors is a best academic portal that is known for success help in completing assignment services to the students. Live Chat @ https://www.besttutors.us/medical-assignment-help

    Read More @ Forensic Accounting Homework Help Alaska
    Pre- Calculus Homework Help Alabama

    ReplyDelete

  33. Artificial Intelligence Companies in bangalore is the field of computer science dedicated to solving cognitive problems commonly associated with human intelligence, such as learning, problem solving, and pattern recognition.

    artificial intelligence companies in indiaMany business certainly expect AI to be disruptive Technology in the coming days or Testing the Waters currently, but why wait when the technology is already transforming every aspect of the way an organization operates?

    ReplyDelete
  34. My Genius Mind is a reputed academic concern that extends best professional Assignment Help to the students. This service can be availed at any time anywhere.
    Live Chat @ https://www.mygeniusmind.com/my-assignment-help
    Read More Information @ Accounting Assignment Help Brisbane
    Boston South Australia Assignment Help
    Mandurah Western Australia Assignment Help
    NewCastle NSW Assignment Help

    ReplyDelete
  35. Management Tutors, known for offering fantastic coursework help to the students. It is beneficial to the students in the achievement of their goals and getting good grades by the teachers.
    Live Chat @ https://www.managementtutors.com/professional-help-with-assignment-uk
    Read More @ Dissertation Help Services Adelaide
    Business Management Help Melbourne
    Project Management Help Queensland
    Operational Management Assignment Brisbane

    ReplyDelete
  36. Best Assignment Help NSW – Help with Assignments is best My Assignment custom firms make sure to send unique content. They assume severe research and provide sincere work permanently.
    Live Chat @ https://www.helpwithassignments.com/

    Read More @ Business Finance Assignment Help Victoria
    University Nursing Assignment Help Queensland
    Homework Help Services Victoria

    ReplyDelete
  37. The Best Tutors is a remarkable academic portal that is known for offering online dissertation services responses to the subtle students.Live Chat @ Essay Writing Services Arizona

    Read More @ Nursing Assignment Writer Arkansas
    Assignment Tutoring Help Alabama

    ReplyDelete
  38. Australia Best Tutor is offering online assignment help services Australia at affordable price. Here students are joining for best academics grades and good quality content.Services are under below

    Engineering Assignment Help Brisbane
    Management Assignment Help Brisbane
    Assignment Writing Services Brisbane
    Nursing Assignment Help Perth
    Finance Assignment Help Perth

    Live Chat @ https://www.australiabesttutor.com

    ReplyDelete
  39. UK Best Tutor is one of the famous and trustworthy academic portals that is known to provide the students with Assignment Help Scotland directly and successfully.
    Live Chat @ https://www.ukbesttutor.co.uk/

    Read More @ Online Assignment Writing UK
    University Assignment Help UK
    Assignment Help Tutor England

    ReplyDelete
  40. This is a very nice article. thank you for publishing this. i can understand this easily.!!Data Science online Course

    ReplyDelete
  41. Help with Assignments and offers impressive Help Engineering to the students in various subjects. When such tasks are submitted, the students are bound to get high marks or scores.
    Live Chat @ https://www.helpwithassignments.com/engineering-assignment-help

    Read More @ Marketing Management Assignment Help Queensland
    Help with Accounting Assignment Help Australian Territory
    Pay for Finance Assignment Help Queensland

    ReplyDelete
  42. The Best Tutors is one of the well-acclaimed and celebrated academic portals that offer the Math Homework Help. These services are the rapid and affordable price.
    Live Chat @ https://www.besttutors.us/mathematics-homework-help
    Read More @ Civil Engineering Homework Help Arkansas
    Internal Auditing Homework Help

    ReplyDelete
  43. The Best Tutors is a reliable and accessible academic portal that is known for offering exclusively detailed Accounting Homework help to the students.Live Chat @ https://www.besttutors.us/accounting-homework-help
    Read More @ Electrical Engineering Homework Help Colorado
    Thesis Writing Services Connecticut
    Thesis Writing Services Arkansas

    ReplyDelete
  44. Gaining Python certifications will validate your skills and advance your career.
    python certification

    ReplyDelete
  45. PCB Design Training in Bangalore offered by myTectra. India's No.1 PCB Design Training Institute. Classroom, Online and Corporate training in PCB Design
    pcb design training in bangalore

    ReplyDelete
  46. I really enjoy the blog.Much thanks again. Really Great salesforce Online Training

    ReplyDelete
  47. Help with Assignments is a renowned academic portal that is known for offering general academic answers, including How to Homework. Live Chat @ https://www.helpwithassignments.com/homework-help-services

    Read More @ Business Finance Assignment Help New South Wales
    Math Answer Online South Australia
    Nursing Assignment Help Queensland

    ReplyDelete
  48. IOT Training in Bangalore - Live Online & Classroom
    Students are made to understand the type of input devices and communications among the devices in a wireless media.
    IOT Training course observes iot as the platform for networking of different devices on the internet and their inter related communication.

    ReplyDelete
  49. Best Tutors, The students can obtain help from the professional and expert authorities with abundant knowledge, which is given by us as a reliable and popular engineering homework help earner.Live Chat @ https://www.besttutors.us/engineering-homework-help

    Read More @ Online Homework Help Colorado
    Geometry Homework Help Colorado
    Finance Homework Help Colorado
    Online Homework Help Arkansas

    ReplyDelete