Pith. sign in

REVIEW 1 major objections 2 minor 46 cited by

Catastrophic forgetting occurs in LLMs during continual instruction tuning and grows more severe as models scale from 1B to 7B parameters.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-05-18 08:22 UTC

load-bearing objection Forgetting gets worse with scale from 1B to 7B but decoder-only models retain more than encoder-decoder ones during continual instruction tuning. the 1 major comments →

arxiv 2308.08747 v5 submitted 2023-08-17 cs.CL

An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

Yun Luo , Zhen Yang , Fandong Meng , Yafu Li , Jie Zhou , Yue Zhang This is my paper
classification cs.CL
keywords catastrophic forgettinglarge language modelscontinual instruction tuningmodel scaleknowledge retentionlanguage biasesfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tests whether large language models lose earlier knowledge when they are fine-tuned sequentially on new instruction sets. Experiments on models from 1 billion to 7 billion parameters show that performance drops on prior tasks involving domain knowledge, reasoning, and reading comprehension. The drop becomes steeper in bigger models, which the authors link to those models starting from higher baseline scores. Decoder-only architectures retain more than encoder-decoder ones, and running general instruction tuning first reduces the amount of forgetting seen in later stages.

Core claim

The authors show through direct measurement that catastrophic forgetting is generally observed in LLMs ranging from 1b to 7b parameters during continual instruction tuning. Severity increases with model scale in this range, possibly because larger models begin with stronger initial performance. BLOOMZ exhibits less forgetting than mT0. LLMs can also reduce language biases such as gender bias during the process, and general instruction tuning beforehand helps limit forgetting in subsequent fine-tuning.

What carries the argument

Continual instruction tuning followed by repeated evaluation of retained accuracy on domain-knowledge, reasoning, and reading-comprehension benchmarks.

Load-bearing premise

The chosen tasks for domain knowledge, reasoning, and reading comprehension give an unbiased picture of retained knowledge that is not heavily shaped by the exact order or content of the new tuning data.

What would settle it

An experiment in which performance on the original tasks stays flat or rises after the model is fine-tuned on new tasks, or in which forgetting lessens rather than increases as model size grows from 1B to 7B.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Forgetting appears across the full 1B–7B size range tested.
  • Within this range, larger models lose more of their prior capabilities.
  • Decoder-only models keep more knowledge than encoder-decoder models.
  • The same tuning process can reduce certain language biases.
  • Running broad instruction tuning before specialized steps limits later forgetting.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the scale trend holds, methods that protect early knowledge may become more important as models grow beyond 7B.
  • The bias-reduction observation raises the possibility that continual tuning can be used deliberately to correct unwanted behaviors learned in pre-training.
  • Interleaving general and domain-specific tuning stages might be tested as a simple way to preserve broad competence.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 2 minor

Summary. The paper empirically examines catastrophic forgetting (CF) during continual instruction tuning of LLMs. It evaluates models ranging from 1B to 7B parameters across tasks in domain knowledge, reasoning, and reading comprehension. The central findings are that CF occurs generally in this scale range, that forgetting severity increases with model scale (attributed to higher initial performance in larger models), that the decoder-only BLOOMZ exhibits less forgetting than the encoder-decoder mT0, that continual fine-tuning can mitigate certain language biases, and that prior general instruction tuning reduces subsequent forgetting.

Significance. If the scale-dependent forgetting claim survives controls for baseline performance and task difficulty, the work would provide useful empirical grounding for continual-learning strategies in LLMs. The multi-perspective evaluation (knowledge, reasoning, comprehension) and the architectural comparison are strengths; the observation that general instruction tuning alleviates later forgetting is practically relevant. The study is purely empirical with no parameter-free derivations or machine-checked proofs.

major comments (1)
  1. [Abstract / Results section] Abstract and experimental results: the claim that 'as the model scale increases, the severity of forgetting intensifies' is load-bearing for the paper's main contribution. The manuscript reports absolute performance deltas on fixed downstream tasks without indicating use of normalized retention metrics (e.g., retained fraction (post-pre)/pre) or regression controls for initial performance. Larger models start with higher baselines, so larger absolute drops can occur even under identical relative forgetting rates; this confound must be ruled out before the scale effect can be asserted.
minor comments (2)
  1. [Experimental setup] The manuscript should report exact dataset names, sizes, and task-difficulty controls for the domain-knowledge, reasoning, and reading-comprehension evaluations, along with error bars or statistical significance tests on all forgetting deltas.
  2. [Method] Clarify the precise continual fine-tuning sequence and data composition used for the 1B–7B models so that readers can assess whether task ordering or content overlap could drive the observed patterns.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive feedback on our manuscript. We address the major comment regarding the interpretation of scale-dependent forgetting below.

read point-by-point responses

  1. Referee: [Abstract / Results section] Abstract and experimental results: the claim that 'as the model scale increases, the severity of forgetting intensifies' is load-bearing for the paper's main contribution. The manuscript reports absolute performance deltas on fixed downstream tasks without indicating use of normalized retention metrics (e.g., retained fraction (post-pre)/pre) or regression controls for initial performance. Larger models start with higher baselines, so larger absolute drops can occur even under identical relative forgetting rates; this confound must be ruled out before the scale effect can be asserted.

    Authors: We acknowledge this valid concern about potential confounding between model scale and baseline performance. While our manuscript already notes that the observed increase in forgetting severity 'may result from the much significant initial performance in the larger LLM', we agree that absolute deltas alone are insufficient to fully substantiate the claim. In the revised version, we will add analyses using normalized retention metrics (e.g., retained fraction = (post - pre)/pre) across the tasks. We will also include regression controls for initial performance to isolate the effect of scale. These additions will clarify whether the severity indeed intensifies with scale beyond what is expected from higher baselines. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical measurements with no derivations or self-referential reductions

full rationale

This paper conducts an empirical study measuring catastrophic forgetting via performance deltas on downstream tasks across model scales (1B-7B). There are no equations, first-principles derivations, fitted parameters renamed as predictions, or load-bearing self-citations that reduce claims to inputs by construction. All reported observations (e.g., scale-dependent forgetting severity, model architecture comparisons) are direct experimental results against external benchmarks, with the analysis self-contained and independent of any internal redefinitions or ansatzes.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This empirical study introduces no free parameters, mathematical axioms, or invented entities; all claims rest on direct experimental measurements of model performance before and after fine-tuning steps.

pith-pipeline@v0.9.0 · 5743 in / 1133 out tokens · 34208 ms · 2026-05-18T08:22:33.106021+00:00 · methodology

0 comments
read the original abstract

Catastrophic forgetting (CF) is a phenomenon that occurs in machine learning when a model forgets previously learned information while acquiring new knowledge for achieving a satisfactory performance in downstream tasks. As large language models (LLMs) have demonstrated remarkable performance, it is intriguing to investigate whether CF exists during the continual instruction tuning of LLMs. This study empirically evaluates the forgetting phenomenon in LLMs' knowledge during continual instruction tuning from the perspectives of domain knowledge, reasoning, and reading comprehension. The experiments reveal that catastrophic forgetting is generally observed in LLMs ranging from 1b to 7b parameters. Surprisingly, as the model scale increases, the severity of forgetting intensifies in such a model sale range which may result from the much significant initial performance in the larger LLM. Comparing the decoder-only model BLOOMZ with the encoder-decoder model mT0, BLOOMZ exhibits less forgetting and retains more knowledge. Interestingly, we also observe that LLMs can mitigate language biases, such as gender bias, during continual fine-tuning. Furthermore, our findings indicate that general instruction tuning can help alleviate the forgetting phenomenon in LLMs during subsequent fine-tuning.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 46 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation

    cs.LG 2026-06 unverdicted novelty 7.0

    In the NTK regime, new-task training induces old-task prediction drift through the cross-task kernel, yielding an exact closed-form forgetting predictor under frozen linear heads and a low-rank concentration result.

  2. Knowledge Editing in Masked Diffusion Language Models

    cs.CL 2026-06 unverdicted novelty 7.0

    Locate-then-edit succeeds at the same early-to-mid MLP locations in masked diffusion models as in autoregressive models, but requires optimization over intermediate partial-mask states to handle multi-token targets.

  3. MeMo: Memory as a Model

    cs.CL 2026-05 unverdicted novelty 7.0

    MeMo encodes new knowledge into a separate memory model for frozen LLMs, achieving strong performance on BrowseComp-Plus, NarrativeQA, and MuSiQue while capturing cross-document relationships and remaining robust to r...

  4. Learning, Fast and Slow: Towards LLMs That Adapt Continually

    cs.LG 2026-05 unverdicted novelty 7.0

    Fast-Slow Training uses context optimization as fast weights alongside parameter updates as slow weights to achieve up to 3x better sample efficiency, higher performance, and less catastrophic forgetting than standard...

  5. HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model

    cs.CL 2026-05 unverdicted novelty 7.0

    Hebatron is the first open-weight Hebrew MoE LLM adapted from Nemotron-3, reaching 73.8% on Hebrew reasoning benchmarks while activating only 3B parameters per pass and supporting 65k-token context.

  6. Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention

    cs.CL 2026-05 unverdicted novelty 7.0

    FLAS learns a multi-step velocity field v_t(h,t,c) to steer activations, outperforming prompting with harmonic means of 1.015 and 1.113 on two Gemma models without per-concept tuning.

  7. SafeAnchor: Preventing Cumulative Safety Erosion in Continual Domain Adaptation of Large Language Models

    cs.LG 2026-04 unverdicted novelty 7.0

    SafeAnchor preserves 93.2% of original safety alignment across sequential domain adaptations by anchoring low-rank safety subspaces and constraining orthogonal updates, while matching unconstrained fine-tuning perform...

  8. Understanding and Enforcing Weight Disentanglement in Task Arithmetic

    cs.AI 2026-04 unverdicted novelty 7.0

    Task-Feature Specialization explains weight disentanglement in task arithmetic and leads to orthogonality, which OrthoReg enforces to enhance performance of model composition methods.

  9. MNAFT: modality neuron-aware fine-tuning of multimodal large language models for image translation

    cs.CL 2026-04 unverdicted novelty 7.0

    MNAFT identifies language-agnostic and language-specific neurons via activation analysis and selectively fine-tunes only relevant ones in MLLMs to close the modality gap and outperform full fine-tuning and other metho...

  10. SLE-FNO: Single-Layer Extensions for Task-Agnostic Continual Learning in Fourier Neural Operators

    cs.LG 2026-03 unverdicted novelty 7.0

    SLE-FNO achieves zero forgetting and strong plasticity-stability balance in continual learning for FNO surrogate models of pulsatile blood flow by adding minimal single-layer extensions across four out-of-distribution tasks.

  11. You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vector

    cs.RO 2026-03 conditional novelty 7.0

    Optimizing a single constant initial noise vector for frozen generative robot policies improves success rates on 38 of 43 tasks by up to 58% relative improvement.

  12. CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training

    cs.LG 2026-02 unverdicted novelty 7.0

    CapTrack shows post-training causes drift beyond facts, with instruction fine-tuning producing stronger behavioral changes than preference optimization across model families.

  13. Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training

    cs.LG 2025-07 unverdicted novelty 7.0

    An RL agent learns domain re-weighting policies from evaluation feedback to improve balanced performance in continual pre-training of LLMs across source and target domains.

  14. Lost in the Flow with Code Talkers: Unveiling the Instruction-Tuning Tax of Large Language Models in Code Tasks

    cs.SE 2026-06 unverdicted novelty 6.0

    Empirical study finds instruction tuning on CodeLLMs improves instruction following at the expense of infilling performance, termed the Instruction-Tuning Tax.

  15. Not Just After One: Sleep-Inspired Replay Prevents Catastrophic Forgetting After Sequential Tasks

    cs.LG 2026-06 unverdicted novelty 6.0

    Sequential training of multiple tasks followed by unsupervised sleep-like replay partially restores performance across all previously learned tasks in neural networks.

  16. Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training

    cs.CL 2026-06 unverdicted novelty 6.0

    Optimal hyperparameters for LLM continued pre-training follow predictable scaling laws derived from proxy models, enabling a two-stage framework that predicts settings from compute budget and checkpoint state to reduc...

  17. Revisiting Parameter-Based Knowledge Editing in Large Language Models: Theoretical Limits and Empirical Evidence

    cs.CL 2026-05 conditional novelty 6.0

    Parameter-based knowledge editing in LLMs induces reasoning collapse via dimensional collapse and is consistently outperformed by a retrieval baseline across varied edit counts, knowledge complexity, and evaluation metrics.

  18. Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining

    cs.LG 2026-05 unverdicted novelty 6.0

    DG-Hard uses Donoho-Gavish hard thresholding on the fine-tuning weight delta to separate task-aligned signal from noise-like residual, recovering damaged capabilities while preserving target-task gains.

  19. MixSD: Mixed Contextual Self-Distillation for Knowledge Injection

    cs.CL 2026-05 unverdicted novelty 6.0

    MixSD achieves superior memorization-retention trade-off in knowledge injection by using mixed self-generated supervision from the base model's conditionals, retaining up to 100% held-out capability versus 1% for stan...

  20. MixSD: Mixed Contextual Self-Distillation for Knowledge Injection

    cs.CL 2026-05 unverdicted novelty 6.0

    MixSD uses dynamic mixing of the model's expert and naive conditionals to create distribution-aligned supervision that improves the memorization-retention tradeoff over standard SFT.

  21. MixSD: Mixed Contextual Self-Distillation for Knowledge Injection

    cs.CL 2026-05 unverdicted novelty 6.0

    MixSD mixes tokens from the base model's expert and naive conditionals to create distribution-aligned supervision for knowledge injection, yielding better memorization-retention trade-offs than SFT across scales and b...

  22. Learning, Fast and Slow: Towards LLMs That Adapt Continually

    cs.LG 2026-05 unverdicted novelty 6.0

    Fast-Slow Training combines slow parameter updates with fast context optimization to achieve up to 3x better sample efficiency, higher performance, less forgetting, and preserved plasticity in continual LLM learning.

  23. Do Self-Evolving Agents Forget? Capability Degradation and Preservation in Lifelong LLM Agent Adaptation

    cs.AI 2026-05 unverdicted novelty 6.0

    Self-evolving LLM agents exhibit capability erosion under continual adaptation, which Capability-Preserving Evolution mitigates by raising retained simple-task performance from 41.8% to 52.8% in workflow evolution und...

  24. PERSA: Reinforcement Learning for Professor-Style Personalized Feedback with LLMs

    cs.AI 2026-05 unverdicted novelty 6.0

    PERSA combines RLHF with selective parameter-efficient updates to top transformer layers, raising style alignment scores from 35% to 96% on code feedback benchmarks while holding correctness near 100%.

  25. $M^2$-VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills

    cs.RO 2026-04 unverdicted novelty 6.0

    M²-VLA shows that generalized VLMs can serve as direct backbones for robotic manipulation by selectively extracting task-critical features via Mixture of Layers and adding Meta Skill Modules for efficient trajectory learning.

  26. Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuning

    cs.RO 2026-02 unverdicted novelty 6.0

    LifeLong-RFT applies chunking-level on-policy reinforcement learning with Quantized Action Consistency Reward, Continuous Trajectory Alignment Reward, and Format Compliance Reward to fine-tune VLA models, achieving a ...

  27. Routing-Based Continual Learning for Multimodal Large Language Models

    cs.LG 2025-11 unverdicted novelty 6.0

    Routing architecture for MLLMs enables continual learning with constant compute, matching multi-task learning performance and supporting cross-modal transfer.

  28. Deep sequence models tend to memorize geometrically; it is unclear why

    cs.LG 2025-10 unverdicted novelty 6.0

    Deep sequence models develop geometric memory in embeddings that encodes novel global relationships, transforming l-fold composition tasks into 1-step navigation via a natural spectral bias connected to Node2Vec.

  29. Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs

    cs.CL 2025-10 unverdicted novelty 6.0

    Masked fine-tuning enables autoregressive LLMs to inject new factual knowledge without paraphrases and with reversal-curse resistance, matching diffusion LLM advantages on QA tasks.

  30. SEAT: Sparse Entity-Aware Tuning for Knowledge Adaptation while Preserving Epistemic Abstention

    cs.AI 2025-06 unverdicted novelty 6.0

    SEAT preserves epistemic abstention in LLMs during knowledge adaptation via sparse tuning and entity-perturbed KL regularization, yielding 18-101% better abstention on unknown queries while retaining near-perfect know...

  31. Hey, That's My Data! Token-Only Dataset Inference in Large Language Models

    cs.CL 2025-06 unverdicted novelty 6.0

    CatShift detects training data membership in LLMs by comparing output shifts induced by fine-tuning on member versus non-member data, relying on catastrophic forgetting without requiring logit access.

  32. Training Prompt Matters: State-Adaptive Optimization for Robust Fine-Tuning

    cs.CL 2026-06 unverdicted novelty 5.0

    Paraphrased training prompts induce correlated cross-task differences in forgetting and generalization during LLM fine-tuning; superior prompts can be identified via pre-learning task loss and used in a state-adaptive...

  33. Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay

    cs.LG 2026-05 unverdicted novelty 5.0

    Self-generated replay from language models nearly eliminates catastrophic forgetting during finetuning except when models are pretrained close to saturation.

  34. MeMo: Memory as a Model

    cs.CL 2026-05 unverdicted novelty 5.0

    MeMo encodes new knowledge into a separate memory model that integrates with frozen LLMs, showing strong performance on QA benchmarks while avoiding catastrophic forgetting and working without access to model weights.

  35. ReCellTy: Domain-Specific Knowledge Graph Retrieval-Augmented LLMs Reasoning Workflow for Single-Cell Annotation

    cs.CL 2025-04 unverdicted novelty 5.0

    ReCellTy constructs a knowledge graph with 18850 nodes and 48944 edges, retrieves relevant entities for differential genes, and applies multi-task LLM reasoning to improve single-cell type annotation over standard LLM...

  36. Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression

    cs.CL 2024-06 unverdicted novelty 5.0

    Introduces Tree Generation (TG-SFT) to generate synthetic instruction-tuning data from LLMs, reducing catastrophic forgetting when fine-tuning MLLMs on domain-specific or multimodal data.

  37. Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation

    cs.SE 2026-06 unverdicted novelty 4.0

    Empirical study on five LLMs finds pretrained-to-aligned paths yield bigger gains over baseline than finetuned-to-aligned paths, though absolute accuracy remains lower for pretrained starts.

  38. RECALL: Recovery Experience Collection for Active Lifelong Learning in Vision-Language-Action Models

    cs.RO 2026-06 unverdicted novelty 4.0

    RECALL introduces uncertainty-guided active data collection for continual fine-tuning of VLAs, showing efficiency gains over passive imitation but requiring replay or regularization to mitigate catastrophic forgetting.

  39. Modular Monolingual Adaptation using Pretrained Language Models

    cs.CL 2026-06 unverdicted novelty 4.0

    Replacing tokens, freezing the corresponding embeddings, and tuning the rest of the model improves NLU performance on low-resource languages compared to full fine-tuning.

  40. CLaaS: Continual learning as a service for sample efficient online learning

    cs.LG 2026-06 unverdicted novelty 4.0

    CLaaS enables sample-efficient online continual learning for agents via replay-buffered parametric updates, outperforming in-context learning in forward transfer and retention on an adversarial task.

  41. Max-Window Scale Estimation for Near-Lossless HiF8 W8A8 Quantization-Aware Training

    cs.LG 2026-05 unverdicted novelty 4.0

    Identifies amax saturation and catastrophic forgetting in HiF8 W8A8 QAT for OpenPangu-Embedded-1B and mitigates them with 64-step max-window DTS and 500-step BF16 warmup at lr=1e-5 to achieve under 0.6% benchmark drops.

  42. Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs

    cs.CL 2026-05 unverdicted novelty 4.0

    Mix-MoE applies separate LM and MT expert groups in two post-pretraining stages with Fourier-enhanced routing to reduce parameter interference and improve multilingual MT over baselines.

  43. Improving Sparse Memory Finetuning

    cs.LG 2026-04 unverdicted novelty 4.0

    Sparse memory modules with KL-based surprising-token selection let retrofitted LLMs acquire new factual knowledge while largely preserving held-out capabilities.

  44. Jailbreak Attacks and Defenses Against Large Language Models: A Survey

    cs.CR 2024-07 accept novelty 4.0

    A survey that creates taxonomies for jailbreak attacks and defenses on LLMs, subdivides them into sub-classes, and compares evaluation approaches.

  45. Towards EnergyGPT: A Large Language Model Specialized for the Energy Sector

    cs.CL 2025-09 unverdicted novelty 3.0

    Fine-tuned LLaMA 3.1-8B variants for the energy sector outperform the base model on domain QA benchmarks, with LoRA delivering similar gains at lower training cost.

  46. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods

    cs.CL 2024-12 accept novelty 3.0

    A survey that organizes LLMs-as-judges research into functionality, methodology, applications, meta-evaluation, and limitations.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages · cited by 42 Pith papers · 7 internal anchors

  1. [1]

    Mathqa: Towards interpretable math word problem solving with operation-based formalisms

    Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Han- naneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    work page 2019

  2. [2]

    Sparks of Artificial General Intelligence: Early experiments with GPT-4

    S´ebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Adaprompt: Adaptive model training for prompt-based nlp

    11 Under review Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, and Yue Zhang. Adaprompt: Adaptive model training for prompt-based nlp. arXiv preprint arXiv:2202.04824,

    work page arXiv

  4. [4]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),...

    work page 2019

  5. [5]

    Measuring Massive Multitask Language Understanding

    URL https: //doi.org/10.5281/zenodo.5371628. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300,

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.5281/zenodo.5371628 2009

  6. [6]

    Continual learning of natural language processing tasks: A survey

    Zixuan Ke and Bing Liu. Continual learning of natural language processing tasks: A survey. arXiv preprint arXiv:2211.12701,

    work page arXiv

  7. [7]

    Continual training of language models for few-shot learning

    Zixuan Ke, Haowei Lin, Yijia Shao, Hu Xu, Lei Shu, and Bing Liu. Continual training of language models for few-shot learning. arXiv preprint arXiv:2210.05549,

    work page arXiv

  8. [8]

    Adam: A Method for Stochastic Optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Race: Large-scale reading comprehension dataset from examinations

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785–794,

    work page 2017

  10. [10]

    Mitigating catastrophic forgetting in task-incremental continual learning with adaptive classification criterion

    Yun Luo, Xiaotian Lin, Zhen Yang, Fandong Meng, Jie Zhou, and Yue Zhang. Mitigating catastrophic forgetting in task-incremental continual learning with adaptive classification criterion. arXiv preprint arXiv:2305.12270, 2023a. Yun Luo, Zhen Yang, Xuefeng Bai, Fandong Meng, Jie Zhou, and Yue Zhang. Investigat- ing forgetting in pre-trained representations ...

    work page arXiv

  11. [11]

    arXiv preprint arXiv:2211.01786 , year=

    Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786,

    work page arXiv

  12. [12]

    Crows-pairs: A chal- lenge dataset for measuring social biases in masked language models

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel Bowman. Crows-pairs: A chal- lenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1953–1967,

    work page 2020

  13. [13]

    Multitask Prompted Training Enables Zero-Shot Task Generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Fine-tuned language models are continual learners

    Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6107–6122,

    work page 2022

  15. [15]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Finetuned Language Models Are Zero-Shot Learners

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652,

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Pretrained language model in continual learning: A comparative study

    13 Under review Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari. Pretrained language model in continual learning: A comparative study. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25- 29,

    work page 2022

  18. [18]

    HellaSwag: Can a Machine Really Finish Your Sentence?

    URL https://openreview.net/forum?id=figzpGMrdD. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    work page internal anchor Pith review Pith/arXiv arXiv 1905