Quantizing deep convolutional networks for efficient inference: A whitepaper
read the original abstract
We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks for a wide variety of CNN architectures. Model sizes can be reduced by a factor of 4 by quantizing weights to 8-bits, even when 8-bit arithmetic is not supported. This can be achieved with simple, post training quantization of weights.We benchmark latencies of quantized networks on CPUs and DSPs and observe a speedup of 2x-3x for quantized implementations compared to floating point on CPUs. Speedups of up to 10x are observed on specialized processors with fixed point SIMD capabilities, like the Qualcomm QDSPs with HVX. Quantization-aware training can provide further improvements, reducing the gap to floating point to 1% at 8-bit precision. Quantization-aware training also allows for reducing the precision of weights to four bits with accuracy losses ranging from 2% to 10%, with higher accuracy drop for smaller networks.We introduce tools in TensorFlow and TensorFlowLite for quantizing convolutional networks and review best practices for quantization-aware training to obtain high accuracy with quantized weights and activations. We recommend that per-channel quantization of weights and per-layer quantization of activations be the preferred quantization scheme for hardware acceleration and kernel optimization. We also propose that future processors and hardware accelerators for optimized inference support precisions of 4, 8 and 16 bits.
This paper has not been read by Pith yet.
Forward citations
Cited by 46 Pith papers
-
Neural Network Quantization by Learning Low-Loss Subspaces
Learning quantization-aware linear paths in weight space yields a midpoint whose direct quantization matches quantization-aware training performance without using straight-through estimators.
-
Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin
A geometric local landscape model explains PTQ basin-crossing failure at aggressive bitwidths and proves finite-time QAT recovery via straight-through estimator gradient bias under quantizer-compatibility assumptions.
-
FTerViT: Fully Ternary Vision Transformer
FTerViT introduces fully ternary Vision Transformers with TernaryBitConv2d and TernaryLayerNorm operators, achieving 82.43% ImageNet top-1 at 6.09 MB with 15x compression.
-
Q-ARVD: Quantizing Autoregressive Video Diffusion Models
Q-ARVD introduces final-quality-aware frame weighting and outlier-aware adaptive dual-scale quantization to enable accurate low-bit inference for autoregressive video diffusion models.
-
When Bits Break Recourse: Counterfactual-Faithful Quantization
CFQ trains quantizer parameters and mixed-precision allocation to preserve counterfactual recourse validity, cost, and direction on Adult, German Credit, and COMPAS while matching accuracy of standard quantizers.
-
Amortized-Precision Quantization for Early-Exit Vision Transformers
Amortized-Precision Quantization (APQ) and the MAQEE bi-level framework jointly optimize bit-widths and exit thresholds for early-exit ViTs, cutting BOPs by up to 95% with maintained accuracy across vision tasks.
-
Quantamination: Dynamic Quantization Leaks Your Data Across the Batch
Dynamic quantization creates side channels allowing partial or full recovery of other users' batched data in at least four popular ML frameworks.
-
Privatar: Scalable Privacy-preserving Multi-user VR via Secure Offloading
Privatar uses horizontal frequency partitioning and distribution-aware minimal perturbation to enable private offloading of VR avatar reconstruction, supporting 2.37x more users with modest overhead.
-
QSLM: A Performance- and Memory-aware Quantization Framework with Tiered Search Strategy for Spike-driven Language Models
QSLM automates tiered quantization of spike-driven language models via sensitivity analysis and multi-objective search, delivering up to 86.5% memory reduction and 20% power savings while keeping accuracy close to the...
-
DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Scheduling
DPQuant uses epoch-wise probabilistic layer rotation and DP loss sensitivity to quantize only a changing subset of layers, reducing accuracy degradation from quantization noise in DP-SGD and delivering up to 2.21x thr...
-
Reclaiming Residual Knowledge: A Novel Paradigm to Low-Bit Quantization
CoRa reclaims quantization residuals in pre-trained ConvNets by searching low-rank adapter architectures instead of weights, matching SOTA accuracy on ImageNet in 3-4 bit settings with under 250 iterations on 1600 images.
-
SpinQuant: LLM quantization with learned rotations
SpinQuant learns optimal rotations to enable accurate 4-bit quantization of LLM weights, activations, and KV cache, reducing the zero-shot gap to full precision to 2.9 points on LLaMA-2 7B.
-
Minimizing Quantized Semantic Age of Information (QSAoI) in Foundation Model-Based Semantic Communications
Defines QSAoI metric and develops foundation model-based optimization to minimize its expected value via mixed-precision quantization and blocklength adaptation over fading channels.
-
Privacy-Preserving Federated Autoencoder for ECG Anomaly Detection on Edge Devices
A federated system combining autoencoders, FedAvg, Renyi DP-SGD, and INT8 quantization matches centralized AUROC performance (0.782 for ConvAE) on PTB-XL while halving model size and cutting edge latency by up to 44% ...
-
Nano-U: Efficient Terrain Segmentation for Tiny Robot Navigation
A compact network called Nano-U trained with quantization-aware distillation enables accurate binary terrain segmentation and runs efficiently on ESP32-S3 microcontrollers for tiny robots.
-
OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization
OSAQ suppresses weight outliers in LLMs via a closed-form additive transformation from the Hessian's stable null space, improving 2-bit quantization perplexity by over 40% versus vanilla GPTQ with no inference overhead.
-
OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization
OSAQ uses the low-rank structure of the Hessian to construct a closed-form additive weight transformation that suppresses outliers without changing task loss, enabling better low-bit LLM quantization.
-
EdgeSpike: Spiking Neural Networks for Low-Power Autonomous Sensing in Edge IoT Architectures
EdgeSpike delivers 91.4% mean accuracy on five sensing tasks with 31x lower energy on neuromorphic hardware and 6.3x longer battery life in a seven-month field deployment compared to conventional CNNs.
-
You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations
TAQ estimates per-layer importance from hidden representations and output sensitivity on task calibration data to allocate mixed precision in a training-free PTQ setting, outperforming task-agnostic baselines on accur...
-
Enabling Vibration-Based Gesture Recognition on Everyday Furniture via Energy-Efficient FPGA Implementation of 1D Convolutional Networks
Raw waveform input with lightweight 1D-CNN and 1D-SepCNN models, integer quantization, and hardware-aware search enable 0.95+ accuracy swipe recognition on Spartan-7 FPGAs at under 10 ms latency and 1.2 mJ energy.
-
AHCQ-SAM: Toward Accurate and Hardware-Compatible Post-Training Segment Anything Model Quantization
AHCQ-SAM introduces ACNR, HLUQ, CAG, and LNQ quantization techniques that deliver 15.2% mAP gain on 4-bit SAM-B and 14.01% J&F gain on 4-bit SAM2-Tiny versus prior PTQ methods.
-
Quantization in Federated Learning: Methods, Challenges and Future Directions
This survey introduces a taxonomy for quantization in federated learning organized around client heterogeneity, aggregation consistency, non-IID robustness, privacy integration, and hardware co-optimization, while ana...
-
OffQ: Taming Structured Outliers in LLM Quantization by Offsetting
OffQ mitigates structured activation outliers in LLMs via PCA-based rotation and shared offset absorption to support effective W4A4KV4 uniform quantization.
-
The Thermodynamic Costs of Simple Linear Regression
Thermodynamic lower bounds are approximated for exact and SGD linear regression, producing energy-aware scaling laws for optimal training dataset size given a target generalization error.
-
Bridging the Training-Deployment Gap: Gated Encoding and Multi-Scale Refinement for Efficient Quantization-Aware Image Enhancement
A gated hierarchical image-enhancement network trained with quantization-aware training maintains high visual fidelity after low-precision conversion while keeping low computational cost on mobile devices.
-
A KL Lens on Quantization: Fast, Forward-Only Sensitivity for Mixed-Precision SSM-Transformer Models
KL divergence provides a superior forward-only metric for identifying quantization-sensitive parts in SSM-Transformer hybrids, outperforming MSE and SQNR and supporting practical mixed-precision deployment on edge devices.
-
Weight Group-wise Post-Training Quantization for Medical Foundation Model
Permutation-COMQ is a new post-training quantization algorithm that reorders weights within layers and uses only dot-product and rounding steps to deliver the highest reported accuracy for 2-, 4-, and 8-bit medical fo...
-
MicroBi-ConvLSTM: An Ultra-Lightweight Efficient Model for Human Activity Recognition on Resource Constrained Devices
MicroBi-ConvLSTM achieves competitive accuracy on eight HAR benchmarks with only 11.4K parameters and successful full deployment on two microcontroller platforms where prior models fail.
-
COMET: Co-Optimization of a CNN Model using Efficient-Hardware OBC Techniques
COMET co-optimizes CNN inference via OBC Schemes A/B on inputs/weights, four LUT techniques, and an im2col-based GEMM core to deliver efficient FPGA deployment with negligible accuracy loss on LeNet-5 and All-CNN-C.
-
A 71.2-$\mu$W Speech Recognition Accelerator with Recurrent Spiking Neural Network
A recurrent spiking neural network accelerator achieves 71.2 μW real-time speech recognition on TSMC 28nm CMOS via 96.42% model compression and hardware techniques reducing complexity by 90.49%.
-
Boundary-Aware Quantization: Finite-Scale Decision Geometry of Neural Classifiers
Quantization of neural classifiers produces measurable boundary shifts captured by Jaccard distances and flip rates that correlate between calibration and held-out sets across bit widths.
-
Sub-Billion, Super-Frontier: Small Language Models Rival Zero-Shot Frontier LLMs on General and Literary Relation Extraction
Fine-tuned models under 1B parameters reach micro-F1 of 0.83 on general-domain RE versus 0.69 for GPT-5.4 zero-shot, with similar gains on literary benchmarks.
-
On the Quantization Robustness of Diffusion Language Models in Coding Benchmarks
Diffusion coding model CoDA shows smaller accuracy drops than Qwen3-1.7B under 2-4 bit quantization on HumanEval and MBPP.
-
Energy Efficient LSTM Accelerators for Embedded FPGAs through Parameterised Architecture Design
A parameterized LSTM accelerator for embedded FPGAs achieves 11.89 GOP/s/W energy efficiency at 32873 samples per second during real-time inference.
-
A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits
Combining pruning, quantization, and early exits in CNNs reduces inference latency and memory on real edge devices with minimal accuracy loss.
-
MicroBi-ConvLSTM: An Ultra-Lightweight Efficient Model for Human Activity Recognition on Resource Constrained Devices
MicroBi-ConvLSTM is a convolutional-recurrent model with 11.4K parameters that delivers competitive accuracy on eight HAR benchmarks and full INT8 deployment coverage on Raspberry Pi Pico 2 and ESP32.
-
Quantization Robustness to Input Degradations for Object Detection
Benchmarks of YOLO models from nano to extra-large show that degradation-aware calibration for Static INT8 PTQ yields no broad robustness gains to input degradations over clean-data calibration, with limited exception...
-
Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT
4-bit and 6-bit integer-only quantized Transformers implemented on Spartan-7 FPGA for AIoT time-series forecasting achieve 0.63% higher test loss than 8-bit baselines but up to 132x speedup and 48x lower energy.
-
TwinLiteNet+: An Enhanced Multi-Task Segmentation Model for Autonomous Driving
TwinLiteNet+ is a hybrid-encoder multi-task segmentation model with new UCB, USB, and PCAA modules that reports 92.9% mIoU on drivable area and 34.2% IoU on lane segmentation on BDD100K while using 11x fewer FLOPs tha...
-
Scalable Multi Corpora Neural Language Models for ASR
The authors report scalable training of neural LMs from heterogeneous corpora for ASR second-pass rescoring, delivering 6.2% relative WER reduction with minimal latency increase.
-
W4A4 Quantization for Inference on Wan2.2-I2V-A14B
W4A4 quantization via MixQ-style mixed precision, SmoothQuant-style smoothing, and dual-branch GEMM keeps Wan2.2-I2V-A14B within 2-3.5% of FP16 on VBench I2V metrics while beating a native HiFloat4 baseline.
-
Quantized Machine Learning Models for Medical Imaging in Low-Resource Healthcare Settings
A Float16-quantized MobileNetV2 model for multi-class brain tumor MRI classification reaches 82.37% validation accuracy while shrinking from 35.34 MB to 5.76 MB, a 6.14x reduction with negligible accuracy change.
-
Quantized Probabilistic AI for Gear Fault Diagnosis in Motor Drives
Quantizing weights and activations in a pre-trained probabilistic BNN for gear fault diagnosis yields 30-45% computational efficiency gains with no loss in accuracy or uncertainty estimates.
-
Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities
A survey synthesizing challenges, system architectures, model optimizations, deployment methods, and resource management techniques for large language model inference at the network edge.
-
DEEP-GAP: Deep-learning Evaluation of Execution Parallelism in GPU Architectural Performance
L4 delivers up to 4.4x higher throughput than T4 for ResNet models, peaks at batch sizes 16-32, and INT8 yields up to 58x gains over CPU baselines.
-
Adaptive Quantized Planetary Crater Detection System for Autonomous Space Exploration
Proposes a theoretical blueprint for an adaptive quantized neural network system with multi-sensor fusion and anchor-free detection for planetary crater identification under space hardware constraints.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.