REVIEW 21 cited by
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Once-for-All: Train One Network and Specialize it for Efficient Deployment
read the original abstract
We address the challenging problem of efficient inference across many devices and resource constraints, especially on edge devices. Conventional approaches either manually design or use neural architecture search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationally prohibitive (causing $CO_2$ emission as much as 5 cars' lifetime) thus unscalable. In this work, we propose to train a once-for-all (OFA) network that supports diverse architectural settings by decoupling training and search, to reduce the cost. We can quickly get a specialized sub-network by selecting from the OFA network without additional training. To efficiently train OFA networks, we also propose a novel progressive shrinking algorithm, a generalized pruning method that reduces the model size across many more dimensions than pruning (depth, width, kernel size, and resolution). It can obtain a surprisingly large number of sub-networks ($> 10^{19}$) that can fit different hardware platforms and latency constraints while maintaining the same level of accuracy as training independently. On diverse edge devices, OFA consistently outperforms state-of-the-art (SOTA) NAS methods (up to 4.0% ImageNet top1 accuracy improvement over MobileNetV3, or same accuracy but 1.5x faster than MobileNetV3, 2.6x faster than EfficientNet w.r.t measured latency) while reducing many orders of magnitude GPU hours and $CO_2$ emission. In particular, OFA achieves a new SOTA 80.0% ImageNet top-1 accuracy under the mobile setting ($<$600M MACs). OFA is the winning solution for the 3rd Low Power Computer Vision Challenge (LPCVC), DSP classification track and the 4th LPCVC, both classification track and detection track. Code and 50 pre-trained models (for many devices & many latency constraints) are released at https://github.com/mit-han-lab/once-for-all.
Forward citations
Cited by 21 Pith papers
-
TCP-SSM: Efficient Vision State Space Models with Token-Conditioned Poles
TCP-SSM conditions stable poles on visual tokens to explicitly control memory decay and oscillation in SSMs, cutting computation up to 44% while matching or exceeding accuracy on classification, segmentation, and detection.
-
MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning
MatryoshkaLoRA inserts a crafted diagonal matrix P into LoRA to learn accurate nested low-rank adapters that support dynamic rank selection with minimal performance drop.
-
Privatar: Scalable Privacy-preserving Multi-user VR via Secure Offloading
Privatar uses horizontal frequency partitioning and distribution-aware minimal perturbation to enable private offloading of VR avatar reconstruction, supporting 2.37x more users with modest overhead.
-
Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning
SV-QD-RL couples actor structure with branch-specific value learning via structure-conditioned actor-critic branches to generate diverse high-quality policy repertoires in QD-RL.
-
DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation
DREAM-S combines neural architecture search, target-aware supernet training, and attention-entropy-guided distillation to accelerate speculative decoding in VLMs, reporting up to 3.85x speedup over standard methods.
-
AutoMCU: Feasibility-First MCU Neural Network Customization via LLM-based Multi-Agent Systems
AutoMCU uses feasibility-first LLM multi-agent coordination to automate MCU-constrained neural network design, delivering competitive accuracy on CIFAR-10/100 in 1-2 hours versus hundreds of GPU hours for prior HW-NAS...
-
Surrogate Neural Architecture Codesign Package (SNAC-Pack)
SNAC-Pack automates hardware-aware neural architecture codesign for FPGAs via surrogate-based multi-objective search, QAT/pruning compression, and hls4ml synthesis, yielding compact models with reduced resources on je...
-
Elastic Attention Cores for Scalable Vision Transformers
VECA learns effective visual representations using core-periphery attention where patches interact exclusively via a resolution-invariant set of learned core embeddings, achieving linear O(N) complexity while maintain...
-
SWAN: World-Aware Adaptive Multimodal Networks for Runtime Variations
SWAN is the first adaptive multimodal network that meets variable compute budgets, optimizes layer use by sample complexity, and drops irrelevant features, cutting FLOPs up to 49% in 3D object detection with minimal a...
-
DeepFedNAS: Efficient Hardware-Aware Architecture Adaptation for Heterogeneous IoT Federations via Pareto-Guided Supernet Training
DeepFedNAS delivers up to 1.21% higher accuracy and 61x faster architecture search for federated learning on heterogeneous IoT by replacing random supernet sampling with Pareto-optimal elite architectures and using a ...
-
Inner Monologue: Embodied Reasoning through Planning with Language Models
LLMs form an inner monologue from closed-loop language feedback to improve high-level instruction completion in simulated and real robotic rearrangement and kitchen manipulation tasks.
-
HASA: Subnet Allocation for Compute-Constrained Model-Heterogeneous Federated Learning
HASA computes client heterogeneity scores from local data and assigns wider subnets to less heterogeneous clients, raising mean client test accuracy from 13.82% to 14.32% and improving worst-client accuracy versus uni...
-
JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search
JetViT uses post-training attention search to hybridize full-attention ViTs with linear and window attention blocks, achieving up to 1.79x throughput gains on high-res images while preserving accuracy on DINOv3 and De...
-
Orion: Enabling Self-adaptive Memory Management for On-device Online Continual Learning
Orion is a self-adaptive memory management framework for on-device online continual learning that co-optimizes latency, plasticity, and stability via URGE-based reallocation and prefetching.
-
Surrogate Neural Architecture Codesign Package (SNAC-Pack)
SNAC-Pack is a new framework for hardware-aware neural architecture codesign that uses surrogate models, NSGA-II search, quantization-aware training, and hls4ml synthesis to produce compact FPGA-deployable models.
-
Equinox: Decentralized Scheduling for Hardware-Aware Orbital Intelligence
Equinox uses a barrier-function-derived marginal cost to enable value-based adaptive scheduling and neighbor offloading in energy-constrained satellite constellations, yielding 20-31% throughput gains and higher batte...
-
Running hardware-aware neural architecture search on embedded devices under 512MB of RAM
A HW-NAS framework executable on resource-limited embedded devices generates optimized CNNs for low-end MCUs and reports state-of-the-art human-recognition accuracy on the Visual Wake Word dataset.
-
Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression
The prune-quantize-distill ordering produces a better accuracy-size-latency frontier on CIFAR-10/100 than any single technique or other orderings, with INT8 QAT providing the main runtime gain.
-
Bilevel Optimization for Neural Architecture Search
Reviews NAS methods through bilevel optimization lens, categorizing them into sampling-based and theory-based, and proposes an auxiliary math programming framework for more principled architecture and weight updates.
-
elasticAI.explorer: Towards a Unified End-to-End Framework for Hardware-Aware Neural Architecture Search
elasticAI.explorer is an extensible framework for hardware-aware NAS supporting multiple search space types with YAML specs, code generation, cross-compilation, and on-device benchmarking.
-
Spiking Neural Network Architecture Search: A Survey
A survey of Spiking Neural Network architecture search techniques viewed through a hardware/software co-design lens.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.