REVIEW 9 cited by
Freeze-Thaw Bayesian Optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Freeze-Thaw Bayesian Optimization
read the original abstract
In this paper we develop a dynamic form of Bayesian optimization for machine learning models with the goal of rapidly finding good hyperparameter settings. Our method uses the partial information gained during the training of a machine learning model in order to decide whether to pause training and start a new model, or resume the training of a previously-considered model. We specifically tailor our method to machine learning problems by developing a novel positive-definite covariance kernel to capture a variety of training curves. Furthermore, we develop a Gaussian process prior that scales gracefully with additional temporal observations. Finally, we provide an information-theoretic framework to automate the decision process. Experiments on several common machine learning models show that our approach is extremely effective in practice.
Forward citations
Cited by 9 Pith papers
-
Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning
Successive Halving guided by parametric and non-parametric surrogates finds better loss-compute frontiers for scaling laws while using up to 98.7% less compute than exhaustive uniform allocation.
-
Open-Ended Task Discovery via Bayesian Optimization
Generate-Select-Refine is an open-ended Bayesian optimization method that generates tasks and concentrates evaluations on the best one with only logarithmic regret overhead relative to standard single-task optimization.
-
HARBOR: Automated Harness Optimization
HARBOR formalizes harness optimization as constrained noisy Bayesian optimization over mixed-variable spaces and reports a case study where it outperforms manual tuning on a production coding agent.
-
Adaptive Candidate Point Thompson Sampling for High-Dimensional Bayesian Optimization
ACTS improves Thompson sampling in high-dimensional Bayesian optimization by adaptively reducing the search space using gradients from surrogate samples to produce better maximizer samples.
-
Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
DPOP is a new loss function that prevents DPO from lowering preferred response likelihoods and outperforms standard DPO on diverse datasets, MT-Bench, and enables Smaug-72B to exceed 80% on the Open LLM Leaderboard.
-
Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning
Surrogate-guided Successive Halving yields learning curves with better loss-compute values than uniform or SH-only allocation, enabling accurate scaling laws with up to 98.7% compute savings.
-
Unified Neural Scaling Laws
Presents a single functional form for neural scaling that unifies multiple scaling dimensions and claims higher extrapolation accuracy than prior forms across diverse tasks and architectures.
-
Two-stage Optimization for Machine Learning Workflow
Two-stage optimization for ML workflows that prioritizes data pipeline search over hyperparameter tuning, with time-allocation policies and a specificity metric for pruning.
-
A Tutorial on Bayesian Optimization
Bayesian optimization uses Gaussian process regression to build a surrogate model and acquisition functions to guide sampling for optimizing costly objective functions, including a new formal generalization of expecte...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.