An end-to-end machine learning pipeline for detecting and classifying malicious network traffic. The project is built around a simple idea: simulate how a modern Security Operations Center (SOC) might combine predictive modeling, explainability, and automated alerting into one working system.
- Threat Detection — An XGBoost classifier trained on the NSL-KDD dataset, tuned to flag both known and previously unseen attack patterns.
- Explainable AI (XAI) — SHAP (SHapley Additive exPlanations) is used to generate waterfall plots that show which network features actually influenced a given prediction, rather than treating the model as a black box.
- Decoupled Architecture — A FastAPI backend handles inference through a REST API, while a Streamlit frontend provides an interactive UI. Keeping these separate makes the system easier to scale or swap parts of later.
- Automated SOC Alerting — When malicious traffic is detected, the backend fires a webhook notification to a Discord channel in real time, mimicking how an analyst would be paged in a real SOC setup.
- Machine Learning: XGBoost, Scikit-Learn, Pandas, SHAP, Joblib
- Backend API: FastAPI, Uvicorn, Python
requests - Frontend UI: Streamlit
- Version Control: Git & GitHub
- Frontend (Streamlit) — The user uploads a CSV of network traffic (or views a live sample feed). The data is parsed and sent to the backend via an HTTP POST request.
- Backend (FastAPI) — Receives the raw network features, runs them through a pre-fitted Scikit-Learn
ColumnTransformer, and passes the transformed data into the XGBoost model. - Inference & Alerting — The model returns its predictions. If an attack is flagged, a background task sends a JSON payload to a Discord webhook while the predictions are returned to the frontend at the same time.
- Explanation (SHAP) — The frontend displays the results and computes SHAP values so you can see exactly why the model made its call.
Follow these steps to get the full pipeline running locally.
git clone https://github.com/varunlearner/ML-Intrusion-Detection-System.git
cd ML-Intrusion-Detection-Systempython -m venv venv
.\venv\Scripts\activatepip install -r requirements.txtKeep your virtual environment active and start the FastAPI server:
uvicorn api:app --reload --port 8000The API documentation will be available at http://localhost:8000/docs.
Open a second terminal window, activate the virtual environment again, and launch the UI:
python -m streamlit run app.pyThe dashboard will automatically open in your default web browser.
ML-IDS-Project/ ├── .streamlit/ │ └── config.toml # Custom dark-theme UI configuration ├── api.py # FastAPI backend microservice & alerting logic ├── app.py # Streamlit frontend & SHAP visualization ├── ids_model.py # Data pipeline and XGBoost training script ├── ids_preprocessor.joblib # Serialized Scikit-Learn feature scaler/encoder ├── xgb_ids_model.joblib # Trained XGBoost model weights ├── requirements.txt # Python dependencies └── README.md # Project documentation
Varun Singh Computer Science & Artificial Intelligence / Machine Learning