A Comparative Evaluation of Machine Learning Algorithms for IDS in IoT Network

Seyed Amir Mousavi1, Mostafa Sadeghi2, Mohammad Sadegh Sirjani1

1Department of Computer, Ferdowsi University of Mashhad, Mashhad, Iran

2Department of Computer, Islamic Azad University, Zavareh, Iran

2023 14th International Conference on Information and Knowledge Technology (IKT), pp. 168–174

Paper

Catching Attacks in IoT Traffic

Intrusion detection systems are a second line of defense that watch network traffic for malicious activity. Traditional IDS raise too many false alarms and react slowly; machine learning promises higher detection rates with fewer false alarms.

IDS activity diagram with monitoring, anomaly-based and signature-based checks, filter, analyzer, audit, intelligent, and update agents, ending in dropping packets, blocking the source, and raising an alarm
Figure 1. Activity diagram of an intrusion detection system.

Abstract

With the increasing Internet use, network security has become essential due to the rise in cyber-attacks on network services. To detect these attacks, a robust Intrusion Detection System (IDS) is required. Traditional IDS face challenges like high false alert rates and slow real-time attack detection. Machine learning (ML) can improve this situation, providing a low False Alarm Rate and high detection rates. This research used five ML methods (Logistic Regression, Random Forest, k-Nearest Neighbors, Support Vector Machine, and XGBoost) to classify the UNSW-NB15 dataset. The goal is to evaluate the performance of various machine learning classifiers in detecting attacks for Internet of Things (IoT) network intrusion detection. The study highlighted the importance of further research to reduce false positives and negatives. To evaluate these classifiers, precision, accuracy, recall, and F1 score were used. The results show that XGBoost achieved the highest accuracy and recall. However, only some algorithms performed perfectly in all aspects, suggesting the need for diverse detection strategies. Future research should focus on developing comprehensive systems and ensemble approaches to minimize false alerts and missed detections.

Method

Five classic classifiers are trained on UNSW-NB15, a modern IDS benchmark with real normal traffic and current attack scenarios from the IXIA PerfectStorm tool, described by 49 features. Features are normalized to [0, 1], and the experiments run in Google Colab.

  • Logistic regression. A linear baseline that models the probability of an attack.
  • Random forest. An ensemble of decision trees that votes on each flow.
  • k-nearest neighbors. Labels a flow by its closest neighbors; robust on small or imbalanced data.
  • Support vector machine. Learns non-linear boundaries between normal and attack traffic.
  • XGBoost. Gradient-boosted decision trees.
Pipeline from dataset through window selection, feature extraction, feature selection, classification, and k-fold validation, repeating until accuracy is acceptable
Figure 2. Evaluation pipeline: feature extraction and selection, classification, and k-fold validation until accuracy is acceptable.

Results

Each classifier leads on a different metric. Overall, XGBoost, KNN, and SVM are the top three across accuracy, precision, recall, and F1 score.

Bar chart comparing accuracy, precision, recall, and F1 score of random forest, KNN, XGBoost, logistic regression, and SVM
Figure 3. Comparison of the five algorithms on accuracy, precision, recall, and F1 score.
MetricBestRunner-up
AccuracyXGBoost · 87%KNN, SVM · 84%
PrecisionLogistic regression · 90%KNN, XGBoost, SVM · 79%–82%
RecallXGBoost · 90%SVM, KNN · 85%–86%
F1 scoreKNN · 88%XGBoost · 87%, SVM · 85%
Table 1. Best and runner-up classifiers per metric on UNSW-NB15.

Takeaways

  • XGBoost gives the highest accuracy (87%) and recall (90%) on UNSW-NB15.
  • Logistic regression has the best precision (90%), and KNN the best F1 score (88%).
  • No single classifier wins on every metric, so dependable IoT intrusion detection calls for ensemble or combined approaches.
  • Reducing both false positives and missed detections remains the key open problem.

BibTeX

@INPROCEEDINGS{10433047,
  author={Mousavi, Seyed Amir and Sadeghi, Mostafa and Sirjani, Mohammad Sadegh},
  booktitle={2023 14th International Conference on Information and Knowledge Technology (IKT)},
  title={A Comparative Evaluation of Machine Learning Algorithms for IDS in IoT network},
  year={2023},
  volume={},
  number={},
  pages={168-174},
  keywords={Measurement;Support vector machines;Logistic regression;Machine learning algorithms;Classification algorithms;Internet of Things;Ensemble learning;Network Security;Intrusion Detection System;Artificial Intelligence;Machine Learning},
  doi={10.1109/IKT62039.2023.10433047}
}