Pith. sign in

REVIEW 3 major objections 2 minor 78 references

Foundation models detect refactoring bugs in Java programs at up to 93.8% accuracy using zero-shot prompts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-07-04 01:46 UTC pith:FYCPYWUS

load-bearing objection LLMs hit high accuracy spotting decade-old public refactoring bugs, but memorization is the obvious alternative explanation. the 3 major comments →

arxiv 2605.02096 v2 pith:FYCPYWUS submitted 2026-05-03 cs.SE

Foundation Models as Oracles for Refactoring Correctness Detection

Rohit Gheyi , Rian Melo , Jonhnanthan Oliveira , Marcio Ribeiro , Baldoino Fonseca This is my paper
classification cs.SE
keywords refactoringfoundation modelsbug detectionJavazero-shot promptingIDEcorrectness oraclesoftware maintenance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether foundation models can serve as oracles that judge if an automated refactoring in a Java IDE has preserved program behavior. It gathers 226 actual bugs spanning 47 refactoring types from IntelliJ-IDEA, Eclipse, and NetBeans collected over more than ten years. Models receive only the original code, the refactored code, and a simple prompt asking whether the change is correct, with no task-specific training or rules. Several models reach high accuracy, the strongest proprietary model at 93.8 percent, while also producing short explanations for each judgment. The work positions these models as a lightweight complement to traditional precondition checks and static analyses.

Core claim

Foundation models prompted zero-shot can classify refactoring transformations as correct or buggy across 47 types drawn from real IDE histories, reaching 80.5 percent accuracy for GPT-OSS-20B and 93.8 percent for GPT-5.4 in the first-run setting, while supplying explanations and remaining largely consistent under metamorphic perturbations.

What carries the argument

Zero-shot prompting of foundation models to act as oracles that output a correctness judgment plus explanation for a given pair of original and refactored Java programs.

Load-bearing premise

The 226 collected bugs from three IDEs over a decade stand for the correctness problems that occur in practice, and zero-shot prompting measures model capability without data contamination.

What would settle it

A fresh collection of refactoring bugs gathered after every evaluated model's training cutoff date, on which the models show markedly lower detection rates than on the original 226 cases.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Models handle 47 refactoring types without any refactoring-specific rules being written into the prompt.
  • Model judgments stay largely stable when the code is altered by semantics-preserving changes.
  • Models can return short natural-language explanations that support developer review of flagged cases.
  • Open-weight and proprietary models both show usable accuracy, though results vary by model size and training.
  • Foundation models could act as triage filters that route only suspicious refactorings to human inspection.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Integration into an IDE could let the model run automatically on every proposed refactoring and surface only uncertain cases to the developer.
  • The same prompting approach might be tried on refactorings in languages other than Java to test broader applicability.
  • Collecting bugs that post-date all model training data would give a cleaner test of whether the observed accuracy reflects genuine reasoning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The manuscript claims that foundation models can serve as effective oracles for detecting refactoring correctness issues (behavioral changes or compilation errors) in Java programs via zero-shot prompting, without task-specific training. It evaluates this claim on 226 real refactoring bugs collected over more than a decade from IntelliJ-IDEA, Eclipse, and NetBeans, spanning 47 refactoring types. Reported results include first-run accuracies of 80.5% for GPT-OSS-20B and 93.8% for GPT-5.4, with comparisons to other open-weight and proprietary models, metamorphic testing showing consistency under semantics-preserving perturbations, and discussion of short explanations as potential developer aids.

Significance. If the accuracies reflect semantic analysis of transformations rather than recall of public bug reports, the work shows foundation models can provide adaptable, rule-free oracles that complement handcrafted preconditions and static/dynamic analyses across refactoring types. A strength is the use of authentic, multi-IDE bugs collected over >10 years; the explicit caveat that metamorphic testing offers robustness evidence rather than anti-memorization evidence is also credited as appropriately cautious. This could support lightweight triage in IDE workflows if contamination is addressed.

major comments (3)
  1. [Abstract] Abstract: The abstract reports accuracy numbers (80.5% for GPT-OSS-20B, 93.8% for GPT-5.4) but lacks any description of the prompting strategy, how the dataset was constructed or validated, statistical analysis, or controls for confounding factors, preventing assessment of whether the data support the oracle claim.
  2. [Methods/Evaluation section] Methods/Evaluation section: The central claim requires that high accuracies arise from models detecting behavioral or compilation issues in transformations; however, the 226 bugs are from public IDE reports spanning >10 years and the paper states metamorphic testing 'should be interpreted as robustness evidence rather than as evidence against memorization or data contamination,' so the results do not establish the models as oracles for unseen transformations.
  3. [Results/Discussion] Results/Discussion: The headline claim that foundation models 'can be effective for this task' and 'may serve as lightweight triage aids' is load-bearing on the assumption of no data contamination; without additional experiments (e.g., on synthetic or non-public cases) or explicit contamination controls, the performance variation across models cannot be confidently attributed to oracle-like reasoning.
minor comments (2)
  1. [Abstract] Abstract: Model names such as 'GPT-OSS-20B' and 'GPT-5.4' are unclear without full identifiers, version details, or citations to their sources.
  2. [Results] The manuscript would benefit from a table summarizing per-model accuracies, refactoring-type coverage, and metamorphic test outcomes for easier comparison.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for the constructive feedback. We agree that the abstract would benefit from added context on methods and that the claims must remain qualified given the public dataset and explicit caveats on contamination. We respond point-by-point below and will revise accordingly where feasible.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The abstract reports accuracy numbers (80.5% for GPT-OSS-20B, 93.8% for GPT-5.4) but lacks any description of the prompting strategy, how the dataset was constructed or validated, statistical analysis, or controls for confounding factors, preventing assessment of whether the data support the oracle claim.

    Authors: We agree the abstract is too concise. In revision we will add one sentence describing the zero-shot prompting template, the dataset construction from public multi-IDE issue trackers spanning >10 years and 47 types, and note that no formal statistical tests were applied because the study is exploratory. This improves readability without changing results. revision: yes

  2. Referee: [Methods/Evaluation section] Methods/Evaluation section: The central claim requires that high accuracies arise from models detecting behavioral or compilation issues in transformations; however, the 226 bugs are from public IDE reports spanning >10 years and the paper states metamorphic testing 'should be interpreted as robustness evidence rather than as evidence against memorization or data contamination,' so the results do not establish the models as oracles for unseen transformations.

    Authors: We accept this assessment. The manuscript already includes the stated caveat precisely because we do not claim the results prove oracle behavior on unseen transformations. We will expand the methods section to reiterate the scope (real public bugs only) and to clarify that the work demonstrates potential utility rather than definitive generalization. revision: partial

  3. Referee: [Results/Discussion] Results/Discussion: The headline claim that foundation models 'can be effective for this task' and 'may serve as lightweight triage aids' is load-bearing on the assumption of no data contamination; without additional experiments (e.g., on synthetic or non-public cases) or explicit contamination controls, the performance variation across models cannot be confidently attributed to oracle-like reasoning.

    Authors: The wording 'can be effective' and 'may serve' was chosen to reflect exactly this uncertainty. We will strengthen the discussion by adding an explicit paragraph on the contamination limitation and by listing synthetic or private-case experiments as necessary future work. No new experiments are possible in the current revision cycle. revision: yes

standing simulated objections not resolved
  • Providing experiments on non-public or synthetic refactoring cases that would allow definitive attribution of performance to semantic reasoning rather than memorization.

Circularity Check

0 steps flagged

No circularity: direct empirical evaluation on external bug dataset

full rationale

The paper reports an empirical study that collects 226 real refactoring bugs from public IDE reports (IntelliJ, Eclipse, NetBeans) spanning >10 years and measures zero-shot accuracy of foundation models against that ground-truth set. No equations, derivations, fitted parameters, or self-citation chains appear in the provided text. The central results (e.g., 80.5–93.8 % accuracy) are direct comparisons to an independently assembled external dataset and are therefore self-contained against external benchmarks. The paper explicitly flags that metamorphic testing provides only robustness evidence and does not rule out memorization, which further confirms the absence of any definitional or fitted-input circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The work is an empirical study applying existing foundation models; it introduces no free parameters, axioms, or invented entities.

pith-pipeline@v0.9.1-grok · 5831 in / 1215 out tokens · 33833 ms · 2026-07-04T01:46:19.094255+00:00 · methodology

0 comments
read the original abstract

Refactoring tools in popular Integrated Development Environments (IDEs) can introduce unintended behavioral changes or compilation errors, a persistent challenge that undermines developer trust in automated transformations. Traditional detection approaches rely on handcrafted preconditions, and static and dynamic analyses, yet remain limited in adaptability and can miss subtle correctness issues. This study examines the potential of foundation models to serve as oracles for detecting refactoring bugs in Java programs. We evaluate zero-shot prompting, without task-specific training, across 226 real refactoring bugs collected over more than a decade from widely used Java IDEs (IntelliJ-IDEA, Eclipse, and NetBeans), spanning 47 refactoring types. Our results indicate that foundation models can be effective for this task, although performance varies across models. In the first-run setting, GPT-OSS-20B achieved 80.5% accuracy, while GPT-5.4 reached 93.8%. We also evaluated other open-weight and proprietary models: Gemma-4-31B achieved the strongest result among open-weight models, and Gemini-3.1-Pro-Preview achieved the best overall result among all evaluated models. Metamorphic testing indicates that model predictions remain largely consistent under the tested semantics-preserving perturbations, but these results should be interpreted as robustness evidence rather than as evidence against memorization or data contamination. Beyond detection accuracy, foundation models can provide short explanations that may help support developer inspection, operate across refactoring types without explicitly encoded refactoring-specific rules, and may serve as lightweight triage aids in development workflows. Our findings suggest that foundation models can complement traditional refactoring checks by flagging suspicious transformations for developer inspection.

Figures

Figures reproduced from arXiv: 2605.02096 by Baldoino Fonseca, Jonhnanthan Oliveira, Marcio Ribeiro, Rian Melo, Rohit Gheyi.

Figure 1
Figure 1. Figure 1: Applying Push Down Method B.m to C using Eclipse JDT introduces a behavioral change. As another example, consider applying Extract Method in IntelliJ-IDEA to the conditional inside m. In the original program ( view at source ↗
Figure 2
Figure 2. Figure 2: Applying Extract Method using IntelliJ-IDEA introduces a behavioral change. Although simple, this example mirrors issues reported in real-world systems. Gligoric et al. [28] studied five large open-source Java projects and uncovered 77 refactoring-related bugs in Eclipse, many resembling the problem in view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the research method. Stage 1 summarizes dataset con￾struction, including filtering, reconstruction, and validation of 226 refactoring bugs spanning multiple refactoring types. Stage 2 summarizes the evaluation pipeline, including stability and sensitivity assessment with metamorphic op￾erators, prompt-based model inference with GPT-OSS-20B and GPT-5.4, correctness assessment, and computation of… view at source ↗
Figure 4
Figure 4. Figure 4: GPT-OSS-20B results across five repeated attempts. Acc@ = cumu￾lative overall success rate considering whether an instance is solved in at least one of the first k attempts, tar@ = agreement in binary correctness across the first k attempts, cons@ = correctness of the strict-majority outcome across the first k attempts, BC@ = cumulative success rate on behavioral-change cases, and CE@ = cumulative success … view at source ↗
Figure 5
Figure 5. Figure 5: GPT-5.4 results across five repeated attempts. Acc@ = cumulative overall success rate considering whether an instance is solved in at least one of the first k attempts, tar@ = agreement in correctness and error status across the first k attempts, cons@ = correctness of the strict-majority outcome across the first k attempts, BC@ = cumulative success rate on behavioral-change cases, and CE@ = cumulative suc… view at source ↗
Figure 6
Figure 6. Figure 6: Heatmap of model accuracy across the 10 most frequent refactoring types in the dataset, separated into compilation-error (CE) and behavioral￾change (BC) instances. All remaining refactoring categories are grouped into Others. RQ2 Answer GPT-5.4 is effective and stable at detecting reported refactoring bugs in this benchmark across repeated attempts. Repeated sampling improves cumulative coverage from 93.8%… view at source ↗
Figure 7
Figure 7. Figure 7: Applying Inline Variable to iii using NetBeans introduces a compi￾lation error. 5.2 Temperature Temperature is a key hyperparameter in foundation models that controls the randomness of generated outputs [47]. Lower values make the model view at source ↗
Figure 8
Figure 8. Figure 8: Accuracy across decoding temperatures for GPT-OSS-20B on the full benchmark, behavioral-change cases (BC), and compilation-error cases (CE). We also performed an additional GPT-5.4 run with T = 0, keeping all other parameters unchanged. In this deterministic decoding setting, GPT-5.4 obtained worse results than under the default-temperature configuration: 88.5% view at source ↗
Figure 9
Figure 9. Figure 9: Overall, behavioral-change (BC), and compilation-error (CE) accuracy of the evaluated open models at temperature 0.5. GPT-OSS-20B with 80.5%. These results indicate a clear performance gap between the strongest proprietary model in this evaluation and the remaining models, particularly the smaller open-weight model considered in our study. The per-category results provide a more detailed view of these diff… view at source ↗
Figure 10
Figure 10. Figure 10: Diagram of bugs correctly detected by each model. Numbers indicate the number of bugs in each overlap region. 5.7 Costs Cost is an important consideration when deploying these models at scale. As with the execution-time analysis, our goal is not to provide a definitive or exact cost benchmark. Costs depend on several factors, including hardware availability, cloud provider pricing, API pricing, batching s… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

78 extracted references · 78 canonical work pages · 2 internal anchors

  1. [1]

    Fowler, Refactoring: improving the design of existing code, Addison-Wesley, 1999

    M. Fowler, Refactoring: improving the design of existing code, Addison-Wesley, 1999

    work page 1999

  2. [2]

    Opdyke, Refactoring Object-oriented Frameworks, Ph.D

    W. Opdyke, Refactoring Object-oriented Frameworks, Ph.D. thesis, UIUC (1992)

    work page 1992

  3. [3]

    Roberts, Practical Analysis for Refactoring, Ph.D

    D. Roberts, Practical Analysis for Refactoring, Ph.D. thesis, University of Illinois at Urbana-Champaign (1999)

    work page 1999

  4. [4]

    Golubev, Z

    Y. Golubev, Z. Kurbatova, E. A. AlOmar, T. Bryksin, M. W. Mkaouer, One thousand and one stories: a large-scale survey of software refactoring, in: Proceedings of the Foundations of Software Engineering, ACM, 2021, p. 1303–1313

    work page 2021

  5. [5]

    Soares, R

    G. Soares, R. Gheyi, T. Massoni, Automated Behavioral Testing of Refactoring Engines, IEEE Transactions on Software Engineering 39 (2) (2013) 147–162

    work page 2013

  6. [6]

    Daniel, D

    B. Daniel, D. Dig, K. Garcia, D. Marinov, Automated testing of refactoring engines, in: Proceedings of the Foundations of Software Engineering, ACM, 2007, pp. 185–194

    work page 2007

  7. [7]

    Steimann, A

    F. Steimann, A. Thies, From public to private to absent: Refactoring Java programs under constrained accessibility, in: Proceedings of European Conference on Object-Oriented Programming, Springer, 2009, pp. 419–443

    work page 2009

  8. [8]

    Soares, R

    G. Soares, R. Gheyi, D. Serey, T. Massoni, Making program refactoring safer, IEEE Software 27 (4) (2010) 52–57

    work page 2010

  9. [9]

    Schäfer, O

    M. Schäfer, O. de Moor, Specifying and implementing refactorings, in: Proceedings of the Object-Oriented Programming, Systems, Languages, and Applications, ACM, 2010, pp. 286–301

    work page 2010

  10. [10]

    Schäfer, T

    M. Schäfer, T. Ekman, O. de Moor, Challenge Proposal: Verification of Refactorings, in: Proceedings of the International Conference on Programming Languages Meets Program Verification, ACM, 2008, pp. 67–72

    work page 2008

  11. [11]

    Bloch, N

    J. Bloch, N. Gafter, Java Puzzlers: Traps, Pitfalls, and Corner Cases, Addison-Wesley, 2005

    work page 2005

  12. [12]

    Tempero, T

    E. Tempero, T. Gorschek, L. Angelis, Barriers to Refactoring, Communications of the ACM 60 (10) (2017) 54–61

    work page 2017

  13. [13]

    X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, H. Wang, Large language models for software engineering: A systematic literature review, Transac- tions on Software Engineering and Methodology 33 (8) (2024) 220:1–220:79

    work page 2024

  14. [14]

    J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, Q. Wang, Software testing with large lan- guage models: Survey, landscape, and vision, IEEE Transactions on Software Engineering 50 (2024) 911–936

    work page 2024

  15. [15]

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, J. M. Zhang, Large language models for software engineering: Survey and open problems, in: Proceedings of the International Conference on Software Engineering: Future of Software Engineering, IEEE, 2023, pp. 31–53

    work page 2023

  16. [16]

    F.Steimann,A.Thies,Frombehaviourpreservationtobehaviourmodification:constraint- based mutant generation, in: Proceedings of the International Conference on Software Engineering, ACM, 2010, pp. 425–434

    work page 2010

  17. [17]

    R. A. DeMillo, R. J. Lipton, F. G. Sayward, Hints on test data selection: Help for the practicing programmer, Computer 11 (4) (1978) 34–41

    work page 1978

  18. [18]

    Y. Jia, M. Harman, An analysis and survey of the development of mutation testing, IEEE Transactions on Software Engineering 37 (5) (2011) 649–678

    work page 2011

  19. [19]

    A. V. Aho, R. Sethi, J. D. Ullman, Compilers: Principles, Techniques, and Tools, Addison- Wesley, Reading, Massachusetts, U.S.A., 1986

    work page 1986

  20. [20]

    Google, Antigravity,https://antigravity.google(2026)

    work page 2026

  21. [21]

    Windsurf, Windsurf AI IDE,https://windsurf.com/editor(2026)

    work page 2026

  22. [22]

    Cursor, The AI Code Editor,https://www.cursor.com(2026)

    work page 2026

  23. [23]

    VSCode, Visual studio code,https://code.visualstudio.com(2026)

    work page 2026

  24. [24]

    JetBrains, IntelliJ IDEA,https://www.jetbrains.com/idea/(2026)

    work page 2026

  25. [25]

    Eclipse.org., Eclipse Project,https://www.eclipse.org/topics/ide/(2026)

    work page 2026

  26. [26]

    Apache, Netbeans IDE,http://www.netbeans.org(2026)

    work page 2026

  27. [27]

    Rachatasumrit, M

    N. Rachatasumrit, M. Kim, An empirical investigation into the impact of refactoring on regression testing, in: Proceedings of the International Conference on Software Maintenance, IEEE, 2012, pp. 357–366. 52 Rohit Gheyi et al

    work page 2012

  28. [28]

    Gligoric, F

    M. Gligoric, F. Behrang, Y. Li, J. Overbey, M. Hafiz, D. Marinov, Systematic testing of refactoring engines on real software projects, in: Proceedings of the European Conference on Object-Oriented Programming, Springer Berlin, 2013, pp. 629–653

    work page 2013

  29. [29]

    Mongiovi, R

    M. Mongiovi, R. Gheyi, G. Soares, L. Teixeira, P. Borba, Making refactoring safer through impact analysis, Science of Computer Programming 93 (2014) 39–64

    work page 2014

  30. [30]

    C. Dong, Y. Jiang, Y. Zhang, Y. Zhang, H. Liu, ChatGPT-based test generation for refactoring engines enhanced by feature analysis on examples, in: Proceedings of the International Conference on Software Engineering, ACM, 2025, pp. 746–746

    work page 2025

  31. [31]

    J. B. Goodenough, S. L. Gerhart, Toward a theory of test data selection, Transactions on Software Engineering 1 (2) (1975) 156–173

    work page 1975

  32. [32]

    H. Wang, Z. Xu, H. Zhang, N. Tsantalis, S. H. Tan, Towards understanding refactoring engine bugs, Transactions on Software Engineering and Methodology 35 (5) (2026) 1–55

    work page 2026

  33. [33]

    Zimmermann, R

    T. Zimmermann, R. Premraj, N. Bettenburg, S. Just, A. Schröter, C. Weiss, What makes a good bug report?, IEEE Transactions on Software Engineering 36 (5) (2010) 618–643

    work page 2010

  34. [34]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Ed- wards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. He...

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  35. [35]

    deterministic

    B. Atil, S. Aykent, A. Chittams, L. Fu, R. J. Passonneau, E. Radcliffe, G. R. Ra- jagopal, A. Sloan, T. Tudrej, F. Ture, Z. Wu, L. Xu, B. Baldwin, Non-determinism of “deterministic” LLM settings (2025).arXiv:2408.04667

    work page arXiv 2025

  36. [36]

    Chiang, L

    W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. I. Jordan, J. E. Gonzalez, I. Stoica, Chatbot arena: an open platform for evaluating LLMs by human preference, in: Proceedings of the International Conference on Machine Learning, PMLR, 2024, pp. 8359–8388

    work page 2024

  37. [37]

    LangChain API, OllamaLLM,https://api.python.langchain.com/en/latest/ollama/ llms/langchain_ollama.llms.OllamaLLM.html(2026)

    work page 2026

  38. [38]

    OpenAI, Python API library,https://github.com/openai/openai-python(2026)

    work page 2026

  39. [39]

    P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, G. Neubig, Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Computing Surveys 55 (9) (2023) 1–35

    work page 2023

  40. [40]

    DAIR.AI, Prompt Engineering Guide, https://www.promptingguide.ai/techniques (2026)

    work page 2026

  41. [41]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, Language models are unsupervised multitask learners, https://cdn.openai.com/better-language-models/ language_models_are_unsupervised_multitask_learners.pdf(2019)

    work page 2019

  42. [42]

    Y. Hou, H. Dong, X. Wang, B. Li, W. Che, MetaPrompting: Learning to learn better prompts (2022).arXiv:2209.11486

    work page arXiv 2022

  43. [43]

    Sallou, T

    J. Sallou, T. Durieux, A. Panichella, Breaking the silence: the threats of using LLMs in software engineering, in: Proceedings of the International Conference on Software Engineering: New Ideas and Emerging Results, ACM, 2024, pp. 102–106

    work page 2024

  44. [44]

    Applis, A

    L. Applis, A. Panichella, R. Marang, Searching for quality: Genetic algorithms and metamorphic testing for software engineering ML, in: Proceedings of the Genetic and Evolutionary Computation Conference, ACM, 2023, pp. 1490–1498

    work page 2023

  45. [45]

    T. Y. Chen, F. Kuo, H. Liu, P. Poon, D. Towey, T. H. Tse, Z. Q. Zhou, Metamorphic testing: A review of challenges and opportunities, Computing Surveys 51 (1) (2018) 4:1–4:27

    work page 2018

  46. [46]

    Pawlak, M

    R. Pawlak, M. Monperrus, N. Petitprez, C. Noguera, L. Seinturier, Spoon: A Library for Implementing Analyses and Transformations of Java Source Code, Software: Practice and Experience 46 (9) (2015) 1155–1179

    work page 2015

  47. [47]

    Holtzman, J

    A. Holtzman, J. Buys, L. Du, M. Forbes, Y. Choi, The curious case of neural text degeneration, in: International Conference on Learning Representations, OpenReview.net, 2020. Foundation Models as Oracles for Refactoring Correctness Detection 53

    work page 2020

  48. [48]

    Simões, R

    P. Simões, R. Gheyi, R. Melo, J. Oliveira, M. Ribeiro, W. Assunção, Refmodel: Detecting refactorings using foundation models, in: Proceedings of the Brazilian Symposium on Software Engineering, SBC, 2025, pp. 811–817

    work page 2025

  49. [49]

    Mongiovi, G

    M. Mongiovi, G. Mendes, R. Gheyi, G. Soares, M. Ribeiro, Scaling testing of refactoring engines, in: Proceedings of the International Conference on Software Maintenance and Evolution, IEEE, 2014, pp. 371–380

    work page 2014

  50. [50]

    W. G. Cochran, The comparison of percentages in matched samples, Biometrika 37 (3/4) (1950) 256–266

    work page 1950

  51. [51]

    W. J. Conover, Practical Nonparametric Statistics, Wiley, 1999

    work page 1999

  52. [52]

    McNemar, Note on the sampling error of the difference between correlated proportions or percentages, Psychometrika 12 (2) (1947) 153–157

    Q. McNemar, Note on the sampling error of the difference between correlated proportions or percentages, Psychometrika 12 (2) (1947) 153–157

    work page 1947

  53. [53]

    Holm, A simple sequentially rejective multiple test procedure, Scandinavian Journal of Statistics 6 (2) (1979) 65–70

    S. Holm, A simple sequentially rejective multiple test procedure, Scandinavian Journal of Statistics 6 (2) (1979) 65–70

    work page 1979

  54. [54]

    Batole, A

    F. Batole, A. Bellur, M. Dilhara, M. R. Ullah, Y. Zharov, T. Bryksin, K. Ishikawa, H. Chen, M. Morimoto, S. Motoura, T. Hosomi, T. N. Nguyen, H. Rajan, N. Tsantalis, D. Dig, Leveraging LLMs, IDEs, and semantic embeddings for automated move method refactoring (2025).arXiv:2503.20934

    work page arXiv 2025

  55. [55]

    Opdyke, R

    W. Opdyke, R. Johnson, Refactoring: An Aid in Designing Application Frameworks and Evolving Object-Oriented Systems, in: Proceedings of the Symposium Object-Oriented Programming Emphasizing Practical Applications, Marist College, 1990, pp. 274–282

    work page 1990

  56. [56]

    Tokuda, D

    L. Tokuda, D. Batory, Evolving Object-Oriented Designs with Refactorings, Automated Software Engineering 8 (1) (2001) 89–120

    work page 2001

  57. [57]

    E. A. AlOmar, M. W. Mkaouer, C. D. Newman, A. Ouni, On preserving the behavior in software refactoring: A systematic mapping study, Information and Software Technology 140 (2021) 106675

    work page 2021

  58. [58]

    Drienyovszky, D

    D. Drienyovszky, D. Horpácsi, S. J. Thompson, Quickchecking refactoring tools, in: Proceedings of Workshop on Erlang, ACM, 2010, pp. 75–80

    work page 2010

  59. [59]

    M. Kim, T. Zimmermann, N. Nagappan, An empirical study of refactoring challenges and benefits at microsoft, IEEE Transactions on Software Engineering 40 (7) (2014) 633–649

    work page 2014

  60. [60]

    Murphy-Hill, A

    E. Murphy-Hill, A. P. Black, Breaking the barriers to successful refactoring: Observations and tools for extract method, in: Proceedings of the International Conference on Software Engineering, ACM, 2008, p. 421–430

    work page 2008

  61. [61]

    A. M. Eilertsen, G. C. Murphy, The usability (or not) of refactoring tools, in: Proceedings of the International Conference on Software Analysis, Evolution and Reengineering, IEEE, 2021, pp. 237–248

    work page 2021

  62. [62]

    Agentic refactoring: An empirical study of ai coding agents,

    K. Horikawa, H. Li, Y. Kashiwa, B. Adams, H. Iida, A. E. Hassan, Agentic refactoring: An empirical study of AI coding agents (2025).arXiv:2511.04824

    work page arXiv 2025

  63. [63]

    F. F. Xu, U. Alon, G. Neubig, V. J. Hellendoorn, A systematic evaluation of large language models of code, in: Proceedings of the International Symposium on Machine Programming, ACM, 2022, pp. 1–10

    work page 2022

  64. [64]

    Martinez, L

    S. Martinez, L. Xu, M. Elnaggar, E. A. AlOmar, Software refactoring research with large language models: A systematic literature review, Journal of Systems and Software 235 (2026) 112762

    work page 2026

  65. [65]

    White, S

    J. White, S. Hays, Q. Fu, J. Spencer-Smith, D. C. Schmidt, ChatGPT prompt patterns for improving code quality, refactoring, requirements elicitation, and software design (2023).arXiv:2303.07839

    work page arXiv 2023

  66. [66]

    E. A. AlOmar, A. Venkatakrishnan, M. W. Mkaouer, C. D. Newman, A. Ouni, How to refactor this code? An exploratory study on developer-ChatGPT refactoring conver- sations, in: Proceedings of International Conference on Mining Software Repositories, ACM, 2024, pp. 202–206

    work page 2024

  67. [67]

    B. Liu, Y. Jiang, Y. Zhang, N. Niu, G. Li, H. Liu, An empirical study on the potential of LLMs in automated software refactoring (2024).arXiv:2411.04444

    work page arXiv 2024

  68. [68]

    K.Depalma,I.Miminoshvili,C.Henselder,K.Moss,E.A.AlOmar,ExploringChatGPT’s code refactoring capabilities: An empirical study, Expert Systems with Applications 249 (2024) 123602

    work page 2024

  69. [69]

    Pomian, A

    D. Pomian, A. Bellur, M. Dilhara, Z. Kurbatova, E. Bogomolov, A. Sokolov, T. Bryksin, D. Dig, EM-assist: Safe automated extract method refactoring with LLMs, in: Companion Proceedings of Foundations of Software Engineering, ACM, 2024, pp. 582–586. 54 Rohit Gheyi et al

    work page 2024

  70. [70]

    Cordeiro, S

    J. Cordeiro, S. Noei, Y. Zou, An empirical study on the code refactoring capability of large language models (2024).arXiv:2411.02320

    work page arXiv 2024

  71. [71]

    Shirafuji, Y

    A. Shirafuji, Y. Oda, J. Suzuki, M. Morishita, Y. Watanobe, Refactoring programs using large language models with few-shot examples, in: Proceedings of Asia-Pacific Software Engineering Conference, IEEE, 2023, pp. 151–160

    work page 2023

  72. [72]

    Dilhara, A

    M. Dilhara, A. Bellur, T. Bryksin, D. Dig, Unprecedented code change automation: The fusion of LLMs and transformation by example, Proceedings of the ACM on Software Engineering 1 (FSE) (2024) 631–653

    work page 2024

  73. [73]

    Zhang, Y

    Y. Zhang, Y. Li, G. Meredith, K. Zheng, X. Li, Move method refactoring recommendation based on deep learning and LLM-generated information, Information Sciences 697 (2025) 121753

    work page 2025

  74. [74]

    H. Liu, Y. Wang, Z. Wei, Y. Xu, J. Wang, H. Li, R. Ji, RefBERT: A two-stage pre- trained framework for automatic rename refactoring, in: Proceedings of the International Symposium on Software Testing and Analysis, ACM, 2023, pp. 740–752

    work page 2023

  75. [75]

    Y. C. K. Piao, J. C. Paul, L. D. Silva, A. M. Dakhel, M. Hamdaqa, F. Khomh, Refactoring with LLMs: Bridging human expertise and machine understanding (2025). arXiv: 2510.03914

    work page arXiv 2025

  76. [76]

    Oueslati, M

    K. Oueslati, M. Lamothe, F. Khomh, Refagent: A multi-agent LLM-based framework for automatic software refactoring (2026).arXiv:2511.03153

    work page arXiv 2026

  77. [77]

    Pacheco, S

    C. Pacheco, S. K. Lahiri, M. D. Ernst, T. Ball, Feedback-directed random test generation, in: Proceedings of the International Conference on Software Engineering, IEEE, 2007, pp. 75–84

    work page 2007

  78. [78]

    Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models

    Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Y. Chen, L. Wang, A. T. Luu, W. Bi, F. Shi, S. Shi, Siren’s song in the AI ocean: A survey on hallucination in large language models (2023).arXiv:2309.01219

    work page internal anchor Pith review Pith/arXiv arXiv 2023