pith. sign in

arxiv: 2605.07053 · v2 · pith:TNLFRSXEnew · submitted 2026-05-08 · 💻 cs.CL · cs.AI

GSM-SEM: Benchmark and Framework for Generating Semantically Variant Augmentations

Jyotika Singh , Fang Tu , Aziza Mirsaidova , Amit Agarwal , Hitesh Laxmichand Patel , Sandip Ghoshal , Miguel Ballesteros , Karan Dua
show 6 more authors
Yassine Benajiba Weiyi Sun Tao Sheng Graham Horwood Sujith Ravi Dan Roth
This is my paper

Pith reviewed 2026-06-30 23:39 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords semantic variancebenchmark robustnessmathematical reasoningLLM evaluationGSM8Ktest set variationproblem perturbation
0
0 comments X

The pith

GSM-SEM generates math benchmark variants by altering entities and relationships while keeping the required calculations and answers fixed.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Benchmarks such as GSM8K measure mathematical reasoning but can be memorized by models when the test sets remain fixed. Most existing variants only change surface features like wording or numbers and leave the core facts unchanged. GSM-SEM instead perturbs entities, attributes, and relationships to change the underlying facts while enforcing that the original arithmetic steps and final answer stay the same. The framework produces new variants on every run without fresh human annotation. When 14 current LLMs are tested on the resulting datasets, accuracy falls consistently and falls further when semantic changes are added to existing symbolic or plus-style variations.

Core claim

GSM-SEM is a reusable stochastic framework that perturbs problem statements by modifying entities, attributes, and relationships, frequently altering underlying facts, while constraining generation to preserve the original calculations, answer, and approximate problem difficulty. The method yields GSM8K-SEM, GSM-Symbolic-SEM, and GSM-Plus-SEM, each with substantially higher semantic variance than prior approaches. Evaluation of 14 SOTA LLMs on these sets shows consistent performance drops, reaching an average 28 percent decline under the maximum-strictness configuration, and the same framework extends to BigBenchHard, LogicBench, and NLR-BIRD.

What carries the argument

The GSM-SEM stochastic perturbation process that modifies entities, attributes, and relationships under explicit constraints that keep original calculations and answers unchanged.

If this is right

  • Performance declines grow larger when semantic perturbations are combined with symbolic or plus-style changes.
  • Fresh benchmark variants can be produced on each evaluation run without requiring new human annotations.
  • The same perturbation approach applies directly to other reasoning benchmarks beyond GSM8K.
  • Static public test sets become less reliable measures once models can be evaluated on on-demand semantic variants.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Models that rely on surface patterns rather than recomputing under changed conditions would be expected to show the largest gaps between original and SEM scores.
  • Repeated application of the framework over time could produce a continuously refreshed evaluation distribution that resists memorization.
  • The constraint mechanism that preserves answers while changing facts could be tested on non-mathematical reasoning tasks to check whether the same variance increase appears.

Load-bearing premise

Changes to entities, attributes, and relationships can be constrained so that the original calculations, answer, and approximate difficulty are preserved even when the underlying facts change.

What would settle it

Human inspection of a sample of generated variants shows that the stated answer no longer solves the altered problem statement, or that LLM accuracy on the variants remains statistically unchanged from the original benchmark.

Figures

Figures reproduced from arXiv: 2605.07053 by Amit Agarwal, Aziza Mirsaidova, Dan Roth, Fang Tu, Graham Horwood, Hitesh Laxmichand Patel, Jyotika Singh, Karan Dua, Miguel Ballesteros, Sandip Ghoshal, Sujith Ravi, Tao Sheng, Weiyi Sun, Yassine Benajiba.

Figure 1
Figure 1. Figure 1: Example perturbations and per-run accu￾racy. Top: original GSM8K problem. Middle: GSM-Symbolic and GSM-Plus rewrites. Bottom: GSM￾SEM variants (orange highlights indicate edited spans). For each panel, accuracy across five independent runs is shown (✓ correct, x incorrect), illustrating higher failures on SEM variants. L3.1 = Llama-3.1-405B-Ins; GPT-5 uses medium/default reasoning effort. often interpreted… view at source ↗
Figure 2
Figure 2. Figure 2: GSM-SEM: Semantic variant generation pipeline. serve the original answer and approximate diffi￾culty. This addresses two practical limitations of many existing benchmarks: (i) released variants are static and can become memorization targets over time; and (ii) extending them with fresh, compa￾rable perturbations is often infeasible or requires re-annotating ground-truth answers, which is hard to do reliabl… view at source ↗
Figure 3
Figure 3. Figure 3: Cosine similarity distribution of GSM8K variants with respect to GSM8K, where GSM8K-SEM shows higher semantic divergence than other variants. show the count-based cosine similarity distribu￾tion between GSM-Symbolic and GSM8K, and be￾tween GSM-Plus and GSM8K. As GSM-Symbolic essentially only swaps entities from the original dataset, its similarity tends to be higher than para￾phrased queries. GSM-Plus show… view at source ↗
Figure 3
Figure 3. Figure 3: Cosine similarity distribution of GSM8K variants with respect to GSM8K, where GSM8K-SEM shows higher semantic divergence than other variants. all unanimously-correct variants) for detailed re￾sults shared in the main paper, with more results in the appendix for other configurations [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of Strictness Filter (Section 3) on PDR% (relative to GSM8K) and statistical significance. Filter settings: none (all samples kept; [α, β] 0-1), min ([α, β] 0.30–0.70), min-med (0.35–0.65), med (0.40–0.60), med￾max (0.45–0.55), and max (all such samples filtered out). Variant dataset sizes across filters are shared in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of Strictness Filter (Section 3) on PDR% (relative to GSM8K) and statistical significance. Filter settings: none (all samples kept; [α, β] 0-1), min ([α, β] 0.30–0.70), min-med (0.35–0.65), med (0.40–0.60), med￾max (0.45–0.55), and max (all such samples filtered out). Variant dataset sizes across filters are shared in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 7
Figure 7. Figure 7: Delta in performance for GSM-variants com [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 7
Figure 7. Figure 7: Delta in performance for GSM-variants com [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 6
Figure 6. Figure 6: Average accuracy across models for GSM8K [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 6
Figure 6. Figure 6: Average accuracy across models for GSM8K [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Strictness filter configurations. Each setting [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Cosine similarity distribution of GSM8K vari [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Cosine similarity distribution using all [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
read the original abstract

Benchmarks like GSM8K are popular measures of mathematical reasoning, but leaderboard gains can overstate true capability due to memorization of fixed test sets. Most robustness variants apply surface-level perturbations (paraphrases, renamings, number swaps, distractors) that largely preserve the underlying facts, and static releases can themselves become memorization targets over time. We introduce GSM-SEM, a reusable and stochastic framework for generating semantically diverse benchmark variants with substantially higher semantic variance than prior approaches. GSM-SEM perturbs problem statements by modifying entities, attributes, and/or relationships, frequently altering underlying facts and requiring models to recompute solutions under new conditions, while constraining generation to preserve the original calculations/answer and approximate problem difficulty. GSM-SEM generates fresh variants on each run without requiring re-annotation, reducing reliance on static public benchmarks for evaluation and thereby lowering the bias of memorization. We apply GSM-SEM on GSM8K and two existing variation suites (GSM-Symbolic and GSM-Plus), producing GSM8K-SEM, GSM-Symbolic-SEM, and GSM-Plus-SEM. Evaluating 14 SOTA LLMs, we observe consistent performance drops with larger decline when semantic perturbations are coupled with symbolic/plus variations (average drop rate 28% in maximum strictness configuration of GSM-SEM). We publicly release the three SEM variants as fully human-validated datasets. Finally, to demonstrate applicability beyond GSM-style math problems, we apply GSM-SEM to additional benchmarks including BigBenchHard, LogicBench, and NLR-BIRD.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces GSM-SEM, a reusable stochastic framework for generating semantically diverse variants of math reasoning benchmarks (GSM8K, GSM-Symbolic, GSM-Plus) by perturbing entities/attributes/relationships while constraining generation to preserve original calculations, answers, and approximate difficulty. It produces three new variants, evaluates 14 SOTA LLMs reporting consistent drops (larger when combined with symbolic/plus variations; 28% average in max-strictness config), releases the variants as fully human-validated datasets, and demonstrates the framework on BigBenchHard, LogicBench, and NLR-BIRD.

Significance. If the preservation constraint is reliably enforced and the variants genuinely increase semantic variance without introducing malformed problems, GSM-SEM could supply a practical, on-demand method for reducing memorization bias in LLM math-reasoning evaluation. The public release of human-validated datasets and the extension beyond GSM-style problems are concrete strengths that would aid reproducibility and broader use.

major comments (2)
  1. [Abstract / framework description] Abstract and framework description: the claim that perturbations of entities, attributes, and relationships can be constrained to preserve the original calculations/answer while frequently altering underlying facts is load-bearing for interpreting the reported performance drops as evidence of semantic robustness rather than generation artifacts. No mechanism (symbolic solver check, template matching, prompt constraints, or post-generation filter) is specified.
  2. [Results / evaluation sections] Results and evaluation sections: the 28% average drop rate is reported specifically for the 'maximum strictness configuration,' yet the manuscript provides no definition or operationalization of strictness levels, making it impossible to assess whether the larger declines under coupled semantic+symbolic variations are robust or configuration-dependent.
minor comments (2)
  1. [Abstract] The abstract asserts 'substantially higher semantic variance than prior approaches' without naming the metric or providing a quantitative comparison table; a brief definition or reference to the relevant table/figure would improve clarity.
  2. [Dataset release paragraph] The human-validation process is described only as 'fully human-validated'; adding a short summary of validation criteria or inter-annotator agreement would strengthen the release claim without altering the central argument.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address each major comment point by point below and commit to revisions that strengthen the clarity of the framework and results sections.

read point-by-point responses

  1. Referee: [Abstract / framework description] Abstract and framework description: the claim that perturbations of entities, attributes, and relationships can be constrained to preserve the original calculations/answer while frequently altering underlying facts is load-bearing for interpreting the reported performance drops as evidence of semantic robustness rather than generation artifacts. No mechanism (symbolic solver check, template matching, prompt constraints, or post-generation filter) is specified.

    Authors: We agree that explicit specification of the enforcement mechanism is essential to support the core claims. The manuscript describes the GSM-SEM framework as using constrained generation via prompt engineering to preserve calculations, answers, and approximate difficulty, followed by human validation of the released datasets. However, we acknowledge that the abstract and initial framework description do not sufficiently detail the specific mechanisms (e.g., prompt constraints and post-generation verification). We will revise the abstract to reference the constraint approach and expand the framework section with a dedicated subsection operationalizing the enforcement steps. revision: yes

  2. Referee: [Results / evaluation sections] Results and evaluation sections: the 28% average drop rate is reported specifically for the 'maximum strictness configuration,' yet the manuscript provides no definition or operationalization of strictness levels, making it impossible to assess whether the larger declines under coupled semantic+symbolic variations are robust or configuration-dependent.

    Authors: We accept this critique. The term 'maximum strictness configuration' is used to denote the most rigorous enforcement of preservation constraints during generation, but the manuscript does not provide an explicit definition or parameterization of the strictness levels. We will add a clear operationalization of the strictness levels (including how they modulate generation parameters) in the results and evaluation sections of the revised manuscript to allow readers to assess robustness. revision: yes

Circularity Check

0 steps flagged

No significant circularity; framework is an independent generation process with external validation

full rationale

The paper describes a stochastic generation framework for benchmark variants without any equations, derivations, fitted parameters, or self-citation chains that reduce claims to inputs by construction. The core process perturbs entities/attributes/relationships while constraining for answer preservation and difficulty, but this is presented as an implemented mechanism (with human validation of outputs) rather than a self-definitional or fitted result. Performance drops are measured empirically on 14 LLMs against the generated sets, and the framework is applied to external benchmarks (BigBenchHard, LogicBench, NLR-BIRD) without reliance on prior author theorems or ansatzes. No load-bearing step matches the enumerated circularity patterns; the work is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The framework rests on the domain assumption that constrained semantic perturbations are feasible while preserving math and difficulty; no free parameters or invented entities are mentioned.

axioms (1)
  • domain assumption Semantic perturbations can be generated to alter facts while preserving calculations and difficulty.
    Core constraint of GSM-SEM as described in the abstract.

pith-pipeline@v0.9.1-grok · 5868 in / 1230 out tokens · 26338 ms · 2026-06-30T23:39:51.884861+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 28 canonical work pages · 8 internal anchors

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Gautam Balakrishnan and Anupam Purwar. 2024. https://doi.org/10.13140/RG.2.2.24790.25926 Evaluating the efficacy of open-source llms in enterprise-specific rag systems: A comparative study of performance and scalability

    work page doi:10.13140/rg.2.2.24790.25926 2024

  4. [4]

    BIG bench authors. 2023. https://openreview.net/forum?id=uyTL5Bvosj Beyond the imitation game: Quantifying and extrapolating the capabilities of language models . Transactions on Machine Learning Research

    2023

  5. [5]

    Neeladri Bhuiya, Viktor Schlegel, and Stefan Winkler. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.147 Seemingly plausible distractors in multi-hop reasoning: Are large language models attentive readers? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2514--2528, Miami, Florida, USA. Association for Co...

    work page doi:10.18653/v1/2024.emnlp-main.147 2024

  6. [6]

    Mikhail Bilenko and Raymond J. Mooney. 2003. https://doi.org/10.1145/956750.956759 Adaptive duplicate detection using learnable string similarity measures . In Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '03, page 39–48, New York, NY, USA. Association for Computing Machinery

    work page doi:10.1145/956750.956759 2003

  7. [7]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  8. [8]

    Giancarlo Crocetti. 2015. https://arxiv.org/abs/1505.03934 Textual spatial cosine similarity . Preprint, arXiv:1505.03934

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  9. [9]

    Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.679 Demystifying prompts in language models via perplexity estimation . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10136--10148, Singapore. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.findings-emnlp.679 2023

  10. [10]

    Aabid Karim, Abdul Karim, Bhoomika Lohana, Matt Keon, Jaswinder Singh, and Abdul Sattar. 2025. https://arxiv.org/abs/2503.18018 Lost in cultural translation: Do llms struggle with math across cultural contexts? Preprint, arXiv:2503.18018

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  11. [11]

    Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2961--2984. Association for Computational Linguistics

    2024

  12. [12]

    Jikai Long, Zijian Hu, Xiaodong Yu, Jianwen Xie, and Zhaozhuo Xu. 2025. https://arxiv.org/abs/2506.17264 Oat-rephrase: Optimization-aware training data rephrasing for zeroth-order llm fine-tuning . Preprint, arXiv:2506.17264

    work page arXiv 2025

  13. [13]

    Riccardo Lunardi, Vincenzo Della Mea, Stefano Mizzaro, and Kevin Roitero. 2025. https://arxiv.org/abs/2509.04013 On robustness and reliability of benchmark-based evaluation of llms . Preprint, arXiv:2509.04013

    work page arXiv 2025

  14. [14]

    Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. 2024. https://doi.org/10.18653/v1/2024.acl-long.757 Rephrasing the web: A recipe for compute and data-efficient language modeling . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14044--14072...

    work page doi:10.18653/v1/2024.acl-long.757 2024

  15. [15]

    Nguyen-Hoang Minh-Cong, Nguyen Van Vinh, and Nguyen Le-Minh. 2023. https://doi.org/10.18653/v1/2023.wmt-1.37 A fast method to filter noisy parallel data WMT 2023 shared task on parallel data curation . In Proceedings of the Eighth Conference on Machine Translation, pages 359--365, Singapore. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.wmt-1.37 2023

  16. [16]

    Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2025. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. In The Thirteenth International Conference on Learning Representations

    2025

  17. [17]

    OpenAI , : , Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, A J Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M a dry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. https://arxiv.org/abs/2410.21276 GPT-4o system card

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. https://doi.org/10.18653/v1/2024.acl-long.739 L ogic B ench: Towards systematic evaluation of logical reasoning ability of large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Li...

    work page doi:10.18653/v1/2024.acl-long.739 2024

  19. [19]

    Ella Rabinovich, Samuel Ackerman, Orna Raz, Eitan Farchi, and Ateret Anaby Tavor. 2023. Predicting question-answering performance of large language models through semantic consistency. In Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 138--154

    2023

  20. [20]

    Harsh Raj, Domenic Rosati, and Subhabrata Majumdar. 2022. Measuring reliability of large language models through semantic consistency. In NeurIPS ML Safety Workshop

    2022

  21. [21]

    Navid Rekabsaz, Mihai Lupu, Allan Hanbury, and Guido Zuccon. 2017. https://doi.org/10.1007/978-3-319-56608-5_31 Exploration of a threshold for similarity based on uncertainty in word embedding . In Advances in Information Retrieval: 39th European Conference on IR Research, ECIR 2017 , pages 396--409. Springer

    work page doi:10.1007/978-3-319-56608-5_31 2017

  22. [22]

    Mohammadtaher Safarzadeh, Afshin Oroojlooyjadid, and Dan Roth. 2025. https://arxiv.org/abs/2509.04657 Evaluating nl2sql via sql2nl . Preprint, arXiv:2509.04657

    work page arXiv 2025

  23. [23]

    Viktor Schlegel, Goran Nenadic, and Riza Batista-Navarro. 2021. https://doi.org/10.1609/aaai.v35i15.17622 Semantics altering modifications for evaluating comprehension in machine reading . Proceedings of the AAAI Conference on Artificial Intelligence, 35:13762--13770

    work page doi:10.1609/aaai.v35i15.17622 2021

  24. [24]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations

    2024

  25. [25]

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Sch\" a rli, and Denny Zhou. 2023 a . Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org

    2023

  26. [26]

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Sch \"a rli, and Denny Zhou. 2023 b . Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210--31227. PMLR

    2023

  27. [27]

    Jyotika Singh, Weiyi Sun, Amit Agarwal, Viji Krishnamurthy, Yassine Benajiba, Sujith Ravi, and Dan Roth. 2025. https://doi.org/10.18653/v1/2025.emnlp-industry.60 Can LLM s narrate tabular data? an evaluation framework for natural language representations of text-to- SQL system outputs . In Proceedings of the 2025 Conference on Empirical Methods in Natural...

    work page doi:10.18653/v1/2025.emnlp-industry.60 2025

  28. [28]

    Jyotika Singh, Fang Tu, Miguel Ballesteros, Weiyi Sun, Sandip Ghoshal, Michelle Yuan, Yassine Benajiba, Sujith Ravi, and Dan Roth. 2026. https://arxiv.org/abs/2604.08782 Mt-osc: Path for llms that get lost in multi-turn conversation . Preprint, arXiv:2604.08782

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  29. [29]

    Kv Aditya Srivatsa and Ekaterina Kochmar. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.72 What makes math word problems challenging for LLM s? In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1138--1148, Mexico City, Mexico. Association for Computational Linguistics

    work page doi:10.18653/v1/2024.findings-naacl.72 2024

  30. [30]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  31. [31]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama: Open and efficient foundation language models . Preprint, arXiv:2302.13971

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  32. [32]

    Yuqing Wang and Yun Zhao. 2024. https://arxiv.org/abs/2406.11020 Rupbench: Benchmarking reasoning under perturbations for robustness evaluation in large language models . Preprint, arXiv:2406.11020

    work page arXiv 2024

  33. [33]

    Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Aky \"u rek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2024. https://doi.org/10.18653/v1/2024.naacl-long.102 Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks . In Proceedings of the 2024 Conference of the North American Chap...

    work page doi:10.18653/v1/2024.naacl-long.102 2024

  34. [34]

    Chuan Xiao, Wei Wang, Xuemin Lin, Jeffrey Xu Yu, and Guoren Wang. 2011. https://doi.org/10.1145/2000824.2000825 Efficient similarity joins for near-duplicate detection . ACM Trans. Database Syst., 36(3)

    work page doi:10.1145/2000824.2000825 2011

  35. [35]

    Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. 2025. On memorization of large language models in logical reasoning. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computati...

    2025

  36. [36]

    Rethinking benchmark and contamination for language models with rephrased samples.arXiv preprint arXiv:2311.04850,

    Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2311.04850 Rethinking benchmark and contamination for language models with rephrased samples . Preprint, arXiv:2311.04850

    work page arXiv 2023

  37. [37]

    Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. 2025. https://openreview.net/forum?id=Tn5B6Udq3E Physics of language models: Part 2.1, grade-school math and the hidden reasoning process . In The Thirteenth International Conference on Learning Representations

    2025

  38. [38]

    Erez Yosef, Oron Anschel, Shunit Haviv Hakimi, Asaf Gendler, Adam Botach, Nimrod Berman, and Igor Kviatkovsky. 2026. https://arxiv.org/abs/2604.22597 Rethinking math reasoning evaluation: A robust llm-as-a-judge framework beyond symbolic rigidity . Preprint, arXiv:2604.22597

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  39. [39]

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024 a . https://openreview.net/forum?id=N8N0hgNDRt Metamath: Bootstrap your own mathematical questions for large language models . In The Twelfth International Conference on Learning Representations

    2024

  40. [40]

    Xfinder: Large language models as auto-mated evaluators for reliable evaluation

    Qingchen Yu, Zifan Zheng, Shichao Song, Zhiyu Li, Feiyu Xiong, Bo Tang, and Ding Chen. Xfinder: Large language models as auto-mated evaluators for reliable evaluation

  41. [41]

    Xiaodong Yu, Ben Zhou, Hao Cheng, and Dan Roth. 2024 b . https://arxiv.org/abs/2410.19056 Reasonagain: Using extractable symbolic programs to evaluate mathematical reasoning . CoRR, arXiv:2410.19056

    work page arXiv 2024

  42. [42]

    Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang, and Jiaya Jia. 2025. https://openreview.net/forum?id=br4H61LOoI MR - GSM 8k: A meta-reasoning benchmark for large language model evaluation . In The Thirteenth International Conference on Learning Representations

    2025

  43. [43]

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, and 1 others. 2024. A careful examination of large language model performance on grade school arithmetic. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pages 46819--46836

    2024

  44. [44]

    Yue Zhou, Yada Zhu, Diego Antognini, Yoon Kim, and Yang Zhang. 2024. https://doi.org/10.18653/v1/2024.naacl-long.153 Paraphrase and solve: Exploring and exploiting the impact of surface form on mathematical reasoning in large language models . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguist...

    work page doi:10.18653/v1/2024.naacl-long.153 2024

  45. [45]

    Wong, Xiaowei Huang, Qiufeng Wang, and Kaizhu Huang

    Zihao Zhou, Shudong Liu, Maizhen Ning, Wei Liu, Jindong Wang, Derek F. Wong, Xiaowei Huang, Qiufeng Wang, and Kaizhu Huang. 2025. https://openreview.net/forum?id=nDvgHIBRxQ Is your model really a good math reasoner? evaluating mathematical reasoning with checklist . In The Thirteenth International Conference on Learning Representations

    2025