Anomaly Detection in Smart Grids Using Machine Learning: A Real-Time Streaming Approach Using Apache Kafka and Kubernetes

Abstract

Smart grids face escalating cyber-physical threats, demanding robust and real-time anomaly detection systems. This paper presents a machine learning (ML) framework for detecting anomalies in smart grid data streams, integrating Apache Kafka and Kubernetes for high-throughput data ingestion and Python-based ML models. We evaluate five algorithms—Random Forest, Neural Network, Support Vector Machine (SVM), Naive Bayes, and K-Nearest Neighbors (KNN)—on a dataset of 100,000 samples with 15 electrical and operational features. The Random Forest model achieves exceptional performance (F1-score: 0.9982, precision: 0.9987, recall: 0.9978), outperforming other models. Kafka integrated in Kubernetes enables real-time data streaming, ensuring timely threat detection. This work bridges scalable data processing with ML-driven security, offering a deployable solution for grid resilience.