Samsung Innovation Campus — Capstone Project

---

Problem Statement

Indian social media is flooded with hate speech that spans three languages simultaneously — English, Hindi (Devanagari script), and Hinglish (romanized Hindi). Traditional moderation tools fail completely in this multilingual, code-mixed environment.

Standard English-only NLP models cannot understand a sentence like "Ye log bilkul bakwaas hain, unhe yahan se nikalo" — yet this kind of language is pervasive across Indian Twitter, YouTube comments, and WhatsApp. The result: hate speech flourishes unchecked, escalating digital toxicity into real-world discrimination and violence.

This project solves that gap by fine-tuning Google's MuRIL (Multilingual Representations for Indian Languages) transformer on a custom, hand-curated dataset of code-mixed social media posts — enabling automated, accurate detection of hate speech across all three language modes simultaneously.

---

Live Demo:

---

🗣️ What is Hate Speech?

Hate speech is any form of communication — spoken, written, or behavioral — that attacks, demeans, or incites violence/discrimination against a person or group based on protected characteristics such as: Category Examples Race or Ethnicity Slurs targeting Black, Jewish, or Roma communities Religion Calling a faith "barbaric" or its followers "terrorists" Gender & Sexuality Dehumanizing women or LGBTQ+ individuals Caste Labeling lower-caste groups as inferior (highly relevant to this dataset) Nationality / Immigration Status "Send them back" rhetoric

---

Dataset

The datasets used in this project are publicly available on Kaggle:

File	Description
final_cleaned_dataset.csv	Primary training set (4,860 rows)
unified_dataset_raw.csv	Full metadata with LLM reasoning & confidence
combined_hate_speech_dataset.csv	Raw 29,550-row dataset
hard_examples.csv	516 edge-case evaluation examples

Language Composition

Script	Language	Percentage
🔵 Devanagari	Native Hindi (हिंदी)	~33%
🟢 Latin	English	~52%
🟡 Romanized	Hinglish (code-mixed)	~15%

Class Distribution

┌─────────────────────────────────────────────────┐
│  LABEL BREAKDOWN                                │
│                                                 │
│  ✅  Non-Hate  ████████████████████████  3,989  │
│  🚨  Hate      ████████████            1,906    │
│                                                 │
│  Total ──────────────────────────────── 5,895   │
└─────────────────────────────────────────────────┘

Label schema

• corrected_label / hate_label: 0 = Not Hate Speech, 1 = Hate Speech

Train / Validation / Test Split

Split	Size	Purpose
🟦 Train	4,716	Model learning
🟨 Validation	589	Hyperparameter tuning
🟥 Test	590	Final unseen evaluation

Splits are stratified to preserve the hate/non-hate ratio across all three subsets.

---

Project Architecture & Pipeline

Raw Social Media Data
        │
        ▼
┌───────────────────────┐
│   Phase 1: Data       │
│   Collection &        │  ← unified_dataset_raw.csv
│   Merging             │
└───────────┬───────────┘
            │
            ▼
┌───────────────────────┐
│   Phase 2: Text       │
│   Preprocessing &     │  ← final_cleaned_dataset.csv
│   Cleaning            │
└───────────┬───────────┘
            │
            ▼
┌───────────────────────┐
│   Phase 3: Hard       │
│   Example             │  ← combined_training_dataset.csv
│   Augmentation        │
└───────────┬───────────┘
            │
            ▼
┌───────────────────────┐
│   Phase 4: MuRIL      │
│   Fine-tuning &       │  ← final_muril_model_v2/
│   Training            │
└───────────┬───────────┘
            │
            ▼
┌───────────────────────┐
│   Phase 5: Gradio     │
│   Web App             │  ← Live Demo Interface
│   Deployment          │
└───────────────────────┘

---

Data Preprocessing

The preprocessing pipeline was carefully designed to handle multilingual text without destroying Hindi Devanagari matras (diacritics) — a common failure point of naive text cleaning pipelines.

Cleaning Steps

Step	Operation	Reason
1️⃣	Remove URLs	Noise reduction
2️⃣	Remove @username handles	Privacy + noise
3️⃣	Remove RT artifacts	Twitter-specific cleanup
4️⃣	Remove ASCII punctuation (safely)	Preserves Devanagari script
5️⃣	Collapse extra whitespace	Normalization
6️⃣	Lowercase conversion	Normalizes English/Hinglish, safely ignores Hindi

📄 View Preprocessing Code

import pandas as pd
import re
import string

df = pd.read_csv("unified_dataset_raw.csv")

def clean_multilingual_text(text):
    if not isinstance(text, str):
        return ""
    # Remove URLs
    text = re.sub(r"http\S+|www\S+|https\S+", "", text, flags=re.MULTILINE)
    # Remove @handles
    text = re.sub(r"\@\w+", "", text)
    # Remove RT artifacts
    text = re.sub(r"\bRT\b", "", text)
    # Safely remove ASCII punctuation (preserves Hindi matras)
    punctuation_pattern = f"[{re.escape(string.punctuation)}]"
    text = re.sub(punctuation_pattern, " ", text)
    # Collapse whitespace
    text = re.sub(r"\s+", " ", text).strip()
    # Lowercase (safe for Hindi Devanagari)
    text = text.lower()
    return text

df['clean_text'] = df['text'].apply(clean_multilingual_text)
df = df[df['clean_text'].str.strip().astype(bool)]
df_final = df[['clean_text', 'corrected_label', 'language', 'text']].copy()
df_final.to_csv("final_cleaned_dataset.csv", index=False)

---

Hard Example Augmentation

To improve classifier robustness on subtle and implicit hate, the dataset was augmented with carefully curated hard examples that the model might otherwise miss.

Types of Augmented Examples

Type	Description
🔴 Implicit Threats	Hate framed as hypotheticals or warnings
🟡 Coded Hate Speech	Dog-whistle language and in-group slurs
🔵 Ambiguous Hinglish	Phrases with dual interpretations
🟢 Caste-based Insults	Slurs specific to Indian caste discrimination

📄 View Augmentation Code

import pandas as pd

df_original = pd.read_csv("final_cleaned_dataset.csv")
df_hard = pd.read_csv("hard_examples.csv", on_bad_lines='warn')

df_combined = pd.concat([df_original, df_hard], ignore_index=True)
df_combined = df_combined.sample(frac=1, random_state=42).reset_index(drop=True)
df_combined.to_csv("combined_training_dataset.csv", index=False)

print(f"Total rows in combined dataset: {len(df_combined)}")

---

Model: MuRIL Fine-tuning

Why MuRIL?

MuRIL (Multilingual Representations for Indian Languages) is a transformer model developed by Google, pre-trained on 17 Indian languages and their transliterated variants. Unlike standard BERT or mBERT, MuRIL natively understands:

• ✅ Devanagari Hindi script
• ✅ Romanized / transliterated Hindi
• ✅ Hinglish code-mixed text
• ✅ Cross-script language switching within a single sentence

Training Configuration

Hyperparameter	Value
Base Model	google/muril-base-cased
Max Token Length	128
Learning Rate	2e-5
Batch Size	16
Epochs	3
Weight Decay	0.01
Best Model Metric	Macro F1 Score

Evaluation Metric: Macro F1 Score

The model is evaluated using Macro F1 Score, which gives equal weight to both classes regardless of class imbalance — ensuring the hate speech class isn't overshadowed by the larger non-hate class.

Precision = TP / (TP + FP)
Recall    = TP / (TP + FN)
F1        = 2 × (Precision × Recall) / (Precision + Recall)
Macro F1  = (F1_hate + F1_non-hate) / 2

📄 View Training Code

from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer
from transformers import AutoTokenizer
import evaluate, numpy as np

model_name = "google/muril-base-cased"
tokenizer  = AutoTokenizer.from_pretrained(model_name)
model      = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)

# Tokenization
def tokenize_function(examples):
    return tokenizer(examples["clean_text"], padding="max_length",
                     truncation=True, max_length=128)

tokenized_datasets = hf_dataset.map(tokenize_function, batched=True)
tokenized_datasets = tokenized_datasets.remove_columns(["clean_text", "text", "language"])
tokenized_datasets = tokenized_datasets.rename_column("corrected_label", "labels")
tokenized_datasets.set_format("torch")

# Metrics
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return {
        "accuracy":        accuracy_metric.compute(predictions=predictions, references=labels)["accuracy"],
        "f1_macro":        f1_metric.compute(predictions=predictions, references=labels, average="macro")["f1"],
        "precision_macro": precision_metric.compute(predictions=predictions, references=labels, average="macro")["precision"],
        "recall_macro":    recall_metric.compute(predictions=predictions, references=labels, average="macro")["recall"],
    }

# Training
training_args = TrainingArguments(
    output_dir="./muril_hate_speech_v2",
    eval_strategy="epoch", save_strategy="epoch",
    learning_rate=2e-5, per_device_train_batch_size=16,
    num_train_epochs=3, weight_decay=0.01,
    load_best_model_at_end=True, metric_for_best_model="f1_macro"
)
trainer = Trainer(model=model, args=training_args,
                  train_dataset=tokenized_datasets["train"],
                  eval_dataset=tokenized_datasets["validation"],
                  compute_metrics=compute_metrics)
trainer.train()

---

Results & Evaluation

Final Test Set Performance

Class	Precision	Recall	F1-Score	Support
✅ Non-Hate (0)	0.86	0.90	0.88	399
🚨 Hate (1)	0.76	0.70	0.73	191
—	—	—	—	—
🎯 Accuracy			0.83	590
📊 Macro Avg	0.81	0.80	0.80	590
📊 Weighted Avg	0.83	0.83	0.83	590

Confusion Matrix Summary

                    Predicted
                  Non-Hate  │ Hate
                ────────────┼──────────
Actual Non-Hate │   359     │   40     │
       Hate     │    57     │  134     │
                ───────────────────────

---

Gradio Web Application

A fully functional Gradio web interface was built on top of the fine-tuned MuRIL model, enabling real-world hate speech detection across three input modes:

Features

Feature	Description
Raw Text	Paste a comment or social media post directly
Web Article	Enter a URL — the app scrapes and analyzes the full article
Document	Upload .txt or .pdf files for sentence-level analysis
Flag & Report	Users can flag incorrect predictions, saved to Google Drive
Drive Auto-sync	Flagged cases are automatically backed up to Google Drive

Sentence-Level Analysis

For documents and articles, the app splits content into individual sentences and flags each one independently — producing a hate speech density report showing exactly which sentences are toxic and with what confidence.

📄 View App Code (Gradio Interface)

import gradio as gr
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

# Load model from Drive
tokenizer = AutoTokenizer.from_pretrained(drive_path)
model     = AutoModelForSequenceClassification.from_pretrained(drive_path)
model.eval()

def analyze_single_text(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True,
                       padding=True, max_length=128)
    with torch.no_grad():
        logits = model(**inputs).logits
        pred   = logits.argmax(axis=-1).item()
        probs  = torch.nn.functional.softmax(logits, dim=-1)
        conf   = probs[0][pred].item() * 100
    return pred, conf

# Launch app
demo.launch(share=True)

---

Conclusion

Finding	Detail
Multilingual Capability	Successfully interprets English, Devanagari Hindi, and Hinglish code-mixed text
Strong Baseline	83% accuracy and Macro F1-Score of 0.80 on unseen test data
Safe Moderation Bias	High Non-Hate precision — innocent users are rarely flagged incorrectly
Future Work	Improved detection of implicit hate, sarcasm, and subtle toxic expressions

---

Setup & Installation

# Clone the repository
git clone https://github.com/your-username/multilingual-hate-speech-detection.git
cd multilingual-hate-speech-detection

# Install dependencies
pip install transformers datasets evaluate scikit-learn accelerate torch gradio
pip install beautifulsoup4 PyPDF2 pandas numpy matplotlib seaborn

---

🔗 Links & Resources

Resource	Link
Dataset on Kaggle
MuRIL Base Model
Training Notebook	Google Colab (see /notebooks)

---

Built with ❤️ by Vishwajeet Singh