Confusion Matrix and Cyber Security

Objectives:-

In this article , we will see about confusion matrix and the use of confusion matrix . Also we see how confusion matrix is used in the cyber security world.

What is Confusion Matrix ?

• A Confusion matrix is an N x N matrix used for evaluating the performance of a classification model, where N is the number of target classes. The matrix compares the actual target values with those predicted by the machine learning model. This gives us a holistic view of how well our classification model is performing and what kinds of errors it is making.

• A confusion matrix is a table that is often used to describe the performance of a classification model (or “classifier”) on a set of test data for which the true values are known. The confusion matrix itself is relatively simple to understand, but the related terminology can be confusing.
• In the field of machine learning and specifically, the problem of statistical classification, a confusion matrix, also known as an error matrix, is a specific table layout that allows visualization of the performance of an algorithm, typically a supervised learning one (in unsupervised learning it is usually called a matching matrix).
• There are multiple ways of finding errors in the machine learning model. The Mean Absolute Error(Error/cost) function helps the model to be trained in the correct direction by trying to make the distance between the Actual and predicted value to be 0. We find the error in machine learning model prediction by “y — y^”.
• Mean Square Error(MSE): Points from the data set are taken and they are squared first and then the mean is taken to overcome the error.
• In Classification models, the error is detected with the help of confusion matrix.

Let's understand the structure of confusion matrix

Structure of the Confusion Matrix

A confusion matrix is a good and reliable metric to use with classification problems. It is used to prove that the model is good or bad for different classes and their different impact. For example, if the model needs to catch classes of one particular class more than the other, we can create that measure from the confusion matrix. Let’s understand this by the example of two classes 0 and 1. There are four possible scenarios can happen while prediction:

Class is 1 and our model predicted 1 —> That’s correct!

Class is 1 and our model predicted 0 —> Not good.

Class is 0 and our model predicted 1 —> Again not good.

Class is 0 and our model predicted 0 —> Correct!

We can bind all these scenarios in a matrix-like this :

• The size of the matrix is directly proportional to the number of output classes. It is a square matrix where we assume the column headers as actual values and the row headers as model predictions. The values which are true and predicted true by the model are True Positives (TP), correct negative value predictions are True Negatives (TN), values which were negative but predicted as true are False Positives (FP) and positive values predicted as negative are False Negatives (FN).

True Positive(TP) :- Model predicted positive and actual is positive. So it’s true.

True Negative(TN) :- Model predicted negative and actual is positive. So it’s true.

False Positive(FP) :- Model predicted positive and actual is negative. So it’s false.

False Negative(FN) :- Model predicted negative and actual is negative. So it’s false.

Types of Error in Confusion Matrix

Confusion Matrix gives two type of errors :-

1. Type 1 Error
2. Type 2 Error

a . Type 1 Error

• Type I error, also known as a “False Positive(FP)”:- the error of rejecting a null hypothesis when it is actually true.
• False Positive implies that we are wrongly predicted a negative as positive, How does it imply that we are rejecting a true null hypothesis.

• It means our ML model gives wrong result but in positive way. For example:- Ram result declared by the ML model that he is pass in exam but in reality he failed.
• Type 1 ERROR is very dangerous . In cyber security world , ML model gives that everything is fine , but behind the scene the attackers are continuously attacking the system. So cyber security guys might save their system if they got right information.

b . Type 2 Error

• Type II error, also known as a “False Negative(FN)”: the error of not rejecting a null hypothesis when the alternative hypothesis is the true state of nature.

• It simple words, our ML model gives wrong result but in negative way. For example:- Ram result declared by the ML model that he is failed in the exam but in reality he pass. It is not so much dangerous because in reality he passes the exam.

What can we learn from this?

We learn four things from the confusion matrix :-

1. Precision 
2. Recall
3. F1_score
4. Accuracy

1. Precision :-

It is the portion of values that are identified by the model as correct and are relevant to the problem statement solution. We can also quote this as values, which are a portion of the total positive results given by the model and are positive.

Precision = TP/ (TP + FP)

2. Recall :

It is the portion of values that are correctly identified as positive by the model. It is also termed as True Positive Rate or Sensitivity.

Recall = TP/ (TP+FN)

3. F-1 Score :

It is the harmonic mean of Precision and Recall. It means that if we were to compare two models, then this metric will suppress the extreme values and consider both False Positives and False Negatives at the same time.

F-1 Score = 2*Precision*Recall/ (Precision+Recall)

4. Accuracy :

It is the portion of values that are identified correctly irrespective of whether they are positives or negatives. It means that all True positives and True negatives are included in this

Accurecy =  (TP+TN)/ (TP+TN+FP+FN)

Out of all the terms, precision and recall are most widely used. Their tradeoff is a useful measure of the success of a prediction. The desired model is supposed to have high precision and high recall, but this is only in perfectly separable data. In practical use cases, the data is highly unorganized and imbalanced.

When to use Accuracy / Precision / Recall / F1-Score?

• Accuracy is used when the True Positives and True Negatives are more important. Accuracy is a better metric for Balanced Data.
• Whenever False Positive is much more important use Precision.
• Whenever False Negative is much more important use Recall.
• F1-Score is used when the False Negatives and False Positives are important. F1-Score is a better metric for Imbalanced Data.

Why you we need Confusion matrix?

Here are benefits of using a confusion matrix.

• It shows how any classification model is confused when it makes predictions.
• Confusion matrix not only gives you insight into the errors being made by your classifier but also types of errors that are being made.
• This breakdown helps you to overcomes the limitation of using classification accuracy alone.
• Every column of the confusion matrix represents the instances of that predicted class.
• Each row of the confusion matrix represents the instances of the actual class.
• It provides insight not only the errors which are made by a classifier but also errors that are being made.

Cyber Security

• A criminal activity that uses computer as instrumentality or means for perpetuating further crimes. It is defined as an unlawful act wherein the computer is tool or target or both.
• Cybercrime is vastly growing in the world of tech today. Criminals of the World Wide Web exploit internet users’ personal information for their own gain. They dive deep into the dark web to buy and sell illegal products and services. They even gain access to classified government information.

• The different types of Cyber crimes are:- Crime against government such as cyber terrorism. Crime against persons such as cyber pornography, cyber stalking, cyber defamation. Crime against property such as online gambling, intellectual property infringement, phishing, credit card frauds.
• In the present world, cybercrime offenses are happening at an alarming rate. As the use of the Internet is increasing many offenders, make use of this as a means of communication in order to commit a crime. Cybercrime will cost nearly $6 trillion per annum by 2021 as per the cybersecurity ventures report in 2020. For illegal activities, cybercriminals utilize any network computing devices as a primary means of communication with a victims’ devices, so attackers get profit in terms of finance, publicity and others by exploiting the vulnerabilities over the system.

The framework developed in our work is essential to the creation of a model that can support analytics regarding the identification, detection, and classification of integrated cybercrime offenses (structured and unstructured). The main focus of our work is to find the attacks that take advantage of the security vulnerabilities and analyze these attacks by making use of machine learning techniques.

Cyber Attack Detection and Classification Using Parallel Support Vector Machine

A number of cyber-attack detection and classification methods have been introduced with different levels of success that are used as a countermeasure to preserve data integrity and system availability from attacks. We proposed a Parallel Support Vector Machine (pSVM) algorithm for the detection and classification of cyber attack datasets Basically, cyber attack detection is a classification problem, in which we classify the normal pattern from the abnormal.

• The classification accuracy of PSVM remarkably improve (accuracy for Normal class as well as DOS class is almost 100%) and comparable to false alarm rate and training, testing times.

• The proposed Parallel Support Vector Machine algorithm is evaluated using KDD1999 intrusion detection datasets. The first drawback is that SVM is very sensitive to attacks. The second, SVM designed for the two-class problems it has to be extended for the multiclass problem by choosing a suitable kernel function. Decision-tree-based support vector machine which combines support vector machines and decision tree can be an effective way for solving multi-class problems.
• Improved Support Vector Machine (iSVM) algorithm for classification of cyber attack dataset which gives 100% detection accuracy for Normal and Denial of Service (DOS) classes and comparable to false alarm rate, training, and testing times.

In the confusion matrix above, rows correspond to predicted categories, while columns correspond to actual categories.

• Confusion matrix contains information actual and predicted classifications done by a classifier. The performance of cyber attack detection system is commonly evaluated using the data in a matrix.

This research presents new cyber attack detection and classification system to classify cyber attacks. In this, we developed the performance of IDS using a parallel support vector machine for distributed cyber-attack detection and classification. The new PSVM is shown more efficient for the detection and classification of different types of cyber attacks compared to SDF. The experimental results on the KDD99 benchmark dataset manifest that the proposed algorithm achieved a high detection rate on different types of network attacks.

Conclusion

A confusion matrix is a remarkable approach for evaluating a classification model. It provides accurate insight into how correctly the model has classified the classes depending upon the data fed or how the classes are misclassified.

I hope this help you to clear your confusion about the confusion matrix...

Thanks ...