silent-data-corruption

Closed-loop fault localization and checkpoint recovery for LLM Pre-training

Open-source framework for detecting Silent Data Corruption (SDC) in production GPU/accelerator clusters

A PyTorch-based implementation of DrDNA, a post-hoc framework for detecting and mitigating soft errors (SDCs) in deep neural networks. The project profiles layer-wise activation statistics and computes abnormality scores to enable early, high-accuracy SDC detection, validated on ResNet-18 with CIFAR-10 using fault injection.

Empirically calibrated false-positive bound for one-sided checksum ABFT on bfloat16 GEMM, where the classical τ = Ku threshold is undefined. Catches 76% of exponent-bit flips against a pre-registered 90% bar.

Offline, fail-closed verifier for JSONL telemetry event logs. Emits deterministic audit certificates + human summaries with explicit claims/non-claims for bottleneck and integrity review.

[IPDPS'24] Approximately Upper Bounding GPU Kernel SDC Probability across Multiple Inputs

What one flipped bit does to a quantized language model, and where it has to land to matter.

Detects silent data corruption on individual CPU cores via one narrowly-scoped instruction test. Verifies computed results against an independent path, per core, and names the failing byte. Finds faults that all-core stress tests miss.