Silent Data Corruptions at Scale
read the original abstract
Silent Data Corruption (SDC) can have negative impact on large-scale infrastructure services. SDCs are not captured by error reporting mechanisms within a Central Processing Unit (CPU) and hence are not traceable at the hardware level. However, the data corruptions propagate across the stack and manifest as application-level problems. These types of errors can result in data loss and can require months of debug engineering time. In this paper, we describe common defect types observed in silicon manufacturing that leads to SDCs. We discuss a real-world example of silent data corruption within a datacenter application. We provide the debug flow followed to root-cause and triage faulty instructions within a CPU using a case study, as an illustration on how to debug this class of errors. We provide a high-level overview of the mitigations to reduce the risk of silent data corruptions within a large production fleet. In our large-scale infrastructure, we have run a vast library of silent error test scenarios across hundreds of thousands of machines in our fleet. This has resulted in hundreds of CPUs detected for these errors, showing that SDCs are a systemic issue across generations. We have monitored SDCs for a period longer than 18 months. Based on this experience, we determine that reducing silent data corruptions requires not only hardware resiliency and production detection mechanisms, but also robust fault-tolerant software architectures.
This paper has not been read by Pith yet.
Forward citations
Cited by 16 Pith papers
-
Self-Verifying Measurement Records: Hash-Linked Evidence Graphs for Hardware Benchmarking
The paper constructs hash-linked evidence graphs that bind hardware measurement quantities to their verification records, enabling offline auditing with probabilistic matrix checks and security measures against probe ...
-
Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference
A new fault-injection framework enables a systematic empirical study that produces 17 takeaways on error propagation in LLM inference and four software-only mitigation directions.
-
ITHICA: Intra-Thread Instruction Checking Approach for Defect-Induced Silent Data Corruptions
ITHICA generates functional tests via intra-thread instruction duplication and comparison, detecting 39% more defective servers than baseline methods on over 3000 real CPUs while revealing new defect behaviors.
-
FLARE: One-Shot PE-Level Fault Localization in Systolic Arrays via Algebraic Test Vectors
FLARE uses pairwise coprime test vectors to create unique divisibility signatures that localize faulty rows in systolic arrays with one test pass and over 98% probability for 256x256 INT16 arrays.
-
Kernel Contracts: A Specification Language for ML Kernel Correctness Across Heterogeneous Silicon
Kernel Contracts is a specification language that formalizes correctness requirements for ML kernels to ensure consistent results across heterogeneous silicon platforms.
-
Making TransactionIsolation Checking Practical
Boomslang introduces a front-end/back-end pipeline with superpositions in its IR to enable general-purpose checking of arbitrary transaction isolation levels via SMT solving.
-
DRIFT: Harnessing Inherent Fault Tolerance for Efficient and Reliable Diffusion Model Inference
DRIFT uses resilience analysis, targeted DVFS, and adaptive rollback ABFT to deliver 36% average energy savings or 1.7x speedup in diffusion model inference while preserving generation quality.
-
Single-Event Upsets in 3D Gaussian Splatting Rendering: Bit-Level Criticality, Spatial Extent, and a Parallel Support Guard
Bit flips in 3D Gaussian splatting are highly concentrated in effect with certain high-order bits corrupting up to 75.7% of the frame, but a support guard reduces the worst footprint to 11.68% while preserving clean p...
-
Zero knowledge verification for frontier AI training is possible
Proposes zkVM-based protocol for verifiable frontier AI pre-training with committed specs, network observations, Merkle commitments, and FP precompiles, estimating 36-month POC at single-digit overhead.
-
The Anatomy of Silent Data Corruption: GPU Error Pattern Study and Modeling Guidance
Large-scale GPU fault injection shows NaN/inf outcomes are only 1% of SDC, single-bit flips under 40%, and corruption addresses are periodic, supporting distribution-aware modeling.
-
LLM-PRISM: Characterizing Silent Data Corruption from Permanent GPU Faults in LLM Training
LLMs resist low-frequency permanent GPU faults but certain datapaths and precision formats trigger catastrophic training divergence even at moderate fault rates.
-
Gemini: A Family of Highly Capable Multimodal Models
Gemini Ultra reaches human-expert performance on MMLU for the first time and sets new state-of-the-art results on 30 of 32 benchmarks, including all 20 multimodal ones tested.
-
Effective and Memory-Efficient Alternatives to ECC for Reliable Large-Scale DNNs
MSET and CEP deliver higher reliability than SECDED ECC for CNNs and Vision Transformers with zero memory overhead and substantially lower area and delay.
-
Aging Aware Adaptive Voltage Scaling for Reliable and Efficient AI Accelerators
An aging-aware adaptive voltage scaling framework for AI accelerators reduces predicted threshold voltage shifts by ~19% and aging degradation by up to 46% while saving 14% lifetime power by leveraging neural network ...
-
Kwai Keye-VL-2.0 Technical Report
Kwai Keye-VL-2.0-30B-A3B is a 30B MoE model with 3B active parameters using DSA adaptation and MOPD distillation that reports SOTA results on video understanding and agent benchmarks.
-
AIReSim: A Discrete Event Simulator for Large-scale AI Cluster Reliability Modeling
AIReSim is a discrete event simulator for evaluating failure mitigation, recovery, and capacity planning decisions in large AI clusters.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.