REVIEW 13 cited by
Overcoming catastrophic forgetting in neural networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Overcoming catastrophic forgetting in neural networks
read the original abstract
The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Neural networks are not, in general, capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks which they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on the MNIST hand written digit dataset and by learning several Atari 2600 games sequentially.
Forward citations
Cited by 13 Pith papers
-
Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining
During pretraining, language models exhibit natural ungrokking where learned rules are forgotten based on their support frequency in the corpus, with asymmetric editability of rule survival.
-
Substrate Asymmetry in User-Side Memory: A Diagnostic Framework
User memory in LLMs factors into three orthogonal axes where parametric adapters and retrieval show opposite strengths, with causal evidence from attention interventions and an alignment tax on RLHF models.
-
A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL
Later-domain RL training harms earlier domains via second-order damage concentrated in a low-dimensional shared conflict subspace; brief domain refresh contracts this component to enable selective recovery.
-
GoTTA be Diverse: Rethinking Memory Policies for Test-Time Adaptation
Diversity-aware memory policies improve test-time adaptation performance most under constrained memory budgets and challenging non-i.i.d. streams.
-
Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining
DG-Hard uses Donoho-Gavish hard thresholding on the fine-tuning weight delta to separate task-aligned signal from noise-like residual, recovering damaged capabilities while preserving target-task gains.
-
Improving Layout Representation Learning Across Inconsistently Annotated Datasets via Agentic Harmonization
VLM-based harmonization of inconsistent annotations across two document layout corpora raises detection F-score from 0.860 to 0.883 and table TEDS from 0.750 to 0.814 while tightening embedding clusters.
-
Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs
Machine unlearning in LLMs is often reversible via fine-tuning, indicating suppression not deletion, and a new representation-level framework identifies four forgetting regimes based on reversibility and catastrophicity.
-
Scaling Laws for Task-Specific LLM Distillation
Empirical scaling laws for task-specific LLM distillation in quantitative finance indicate that chain-of-thought supervision recovers general knowledge lost during iterative pruning while in-domain performance degrade...
-
Training Prompt Matters: State-Adaptive Optimization for Robust Fine-Tuning
Paraphrased training prompts induce correlated cross-task differences in forgetting and generalization during LLM fine-tuning; superior prompts can be identified via pre-learning task loss and used in a state-adaptive...
-
MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning
MANGO combines gradient-gating and meta-learned regularization to balance stability and plasticity in single-pass online continual learning, reporting state-of-the-art accuracy on CLEAR-10, CIFAR-100, and Tiny-ImageNet.
-
State Representation and Termination for Recursive Reasoning Systems
Recursive reasoning systems can represent their state via an epistemic state graph and terminate when the linearized order-gap is non-degenerate near the fixed point, providing a local condition for when the stopping ...
-
TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning
TailLoR applies low-rank updates to the singular value matrix of pre-trained weights while using a soft spectral penalty to protect dominant singular directions during continual learning.
-
Foundation-Preserving Adaptation via Generalized Rayleigh-Quotient Optimization
FoLoRA applies generalized Rayleigh-quotient optimization to LoRA updates so that directions are gated by downstream utility divided by a pretraining-proxy forgetting penalty.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.