Pith. sign in

REVIEW 3 major objections 2 minor 39 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

Protocol-aware evaluation shows LLM CUDA debuggers lose up to 40 percentage points of success once even modest performance preservation is required.

2026-06-30 22:52 UTC pith:65VHDTLI

load-bearing objection CUDABeaver gives a performance-conditioned metric and 213 LLM-derived CUDA tasks that show up to 40pp swings in fixer success, but the tasks' representativeness of real debugging remains the open question. the 3 major comments →

arxiv 2605.08455 v2 pith:65VHDTLI submitted 2026-05-08 cs.LG cs.PLcs.SE

CUDABeaver: Benchmarking LLM-Based Automated CUDA Debugging

Shiyang Li , Haoyang Chen , Mattia Fazzini , Caiwen Ding This is my paper
classification cs.LG cs.PLcs.SE
keywords CUDA debuggingLLM evaluationautomated program repairperformance preservationGPU programmingbenchmark constructionconditional metrics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents CUDABEAVER, a benchmark built from 213 real failing CUDA workspaces generated by LLMs. Each task supplies the broken code, native build and test commands, and raw error output so that repair attempts can be judged on whether they restore both correctness and original performance. Existing evaluations often count a fix as successful when the model simply produces a slower but test-passing version. The authors introduce the conditional metric pass@k(M,C,A) that makes the model, corpus, and protocol axes explicit, including the allowed performance loss. Across seven frontier models the results demonstrate that high tolerance for slowdown inflates reported success while even small tightening of the performance requirement produces large drops in measured ability.

Core claim

CUDABEAVER supplies 213 tasks drawn from LLM-generated CUDA failures together with their native build and test commands and error evidence. It distinguishes genuine repairs from performance-degrading simplifications and reports outcomes by failure category, debugging trajectory, stagnation mode, and performance preservation. The pass@k(M,C,A) metric conditions success on the fixer M, corpus C, and protocol axes A; experiments with seven LLMs show measured success shifts by as much as 40 percentage points when the allowed performance loss is tightened.

What carries the argument

The CUDABEAVER benchmark together with the pass@k(M,C,A) metric that conditions success on model, corpus, and protocol including performance-loss tolerance.

Load-bearing premise

The 213 tasks collected from LLM-generated CUDA failures represent the debugging challenges that matter in practice.

What would settle it

Finding that the distribution of failure modes in independently collected real-world CUDA bugs differs substantially from the 213 tasks would undermine claims about the benchmark's representativeness.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • When performance-loss tolerance is high, fixers register much higher success rates than when even modest preservation is enforced.
  • Many apparent fixes succeed only by replacing optimized CUDA kernels with slower sequential or simplified versions.
  • Reporting must separate outcomes by failure category, trajectory, and stagnation mode to reveal where models actually stall.
  • Native build and test commands allow evaluation that stays faithful to the original compilation and runtime environment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Teams relying on LLMs for CUDA maintenance may need to embed performance regression tests inside the repair loop rather than after it.
  • The same conditional evaluation approach could be applied to debugging tasks in other GPU languages or parallel runtimes.
  • Models trained with explicit objectives for both correctness and performance might narrow the gap observed when tolerance is reduced.
  • Extending the task set to include failures arising from hardware-specific or multi-GPU interactions would test whether the current collection already covers the hardest cases.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper introduces CUDABEAVER, a benchmark of 213 tasks derived from LLM-generated failing CUDA workspaces (each with broken code, native build/test commands, and error evidence), proposes the protocol-aware pass@k(M,C,A) metric, and evaluates seven frontier LLMs. It claims that this metric reveals up to 40pp shifts in measured success rates depending on performance-loss tolerance, arguing that standard evaluations miss cases where models repair by degeneration to slower code.

Significance. If the 213 tasks prove representative of real CUDA failure modes (hardware, async execution, memory hierarchy), the work would usefully demonstrate the sensitivity of LLM debugging evaluations to performance criteria and provide a concrete benchmark for the community. The explicit quantification of protocol effects and the focus on performance preservation are strengths that could inform future code-repair benchmarks.

major comments (3)
  1. [Methods / Task Collection] Task construction (Methods / Benchmark section): the central claim that pass@k(M,C,A) gives a 'more faithful view' rests on the 213 tasks capturing relevant CUDA failure distributions, yet the manuscript supplies no details on generation process, failure categorization, diversity statistics across domains (scientific computing, ML, graphics), or external validation against real-world corpora; without this the 40pp shift cannot be interpreted as general.
  2. [Evaluation Protocol / Results] Performance-loss tolerance (Evaluation and Results sections): the reported up-to-40pp shift is attributed to changes in this threshold, but the manuscript does not define the threshold value(s), provide an equation for the tolerance, or include sensitivity ablations; this free parameter directly controls the headline result and must be specified for the claim to be reproducible.
  3. [Results] Statistical controls (Results): the abstract states concrete results across 213 tasks and seven LLMs, but no mention is made of variance estimates, multiple-run controls, or stratification by failure category; this weakens the reliability of the cross-protocol comparison.
minor comments (2)
  1. [Metric Definition] The notation pass@k(M,C,A) is introduced without an explicit mathematical definition or example computation; adding an equation would improve clarity.
  2. [Abstract] Inconsistent capitalization of the benchmark name (CUDABEAVER vs. CUDABeaver) appears in the abstract and should be standardized.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment below and indicate planned revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Methods / Task Collection] Task construction (Methods / Benchmark section): the central claim that pass@k(M,C,A) gives a 'more faithful view' rests on the 213 tasks capturing relevant CUDA failure distributions, yet the manuscript supplies no details on generation process, failure categorization, diversity statistics across domains (scientific computing, ML, graphics), or external validation against real-world corpora; without this the 40pp shift cannot be interpreted as general.

    Authors: We agree that expanded details on task construction are needed to support interpretability of the 40pp shifts. The tasks originate from LLM-generated failing CUDA workspaces (as noted in the abstract), but the current text provides insufficient elaboration. In revision we will add a dedicated Methods subsection describing the generation pipeline, explicit failure categorization, domain diversity statistics (including counts across scientific computing, ML, and graphics), and an explicit limitations paragraph acknowledging the absence of external validation against independent real-world corpora. revision: yes

  2. Referee: [Evaluation Protocol / Results] Performance-loss tolerance (Evaluation and Results sections): the reported up-to-40pp shift is attributed to changes in this threshold, but the manuscript does not define the threshold value(s), provide an equation for the tolerance, or include sensitivity ablations; this free parameter directly controls the headline result and must be specified for the claim to be reproducible.

    Authors: We concur that the performance-loss tolerance must be formalized for reproducibility. The manuscript references the concept but omits its definition and sensitivity analysis. In the revised version we will (i) state the concrete threshold values employed, (ii) supply the mathematical definition of the tolerance within the pass@k(M,C,A) formulation, and (iii) add sensitivity ablations that vary the tolerance and report resulting score changes. revision: yes

  3. Referee: [Results] Statistical controls (Results): the abstract states concrete results across 213 tasks and seven LLMs, but no mention is made of variance estimates, multiple-run controls, or stratification by failure category; this weakens the reliability of the cross-protocol comparison.

    Authors: The abstract already states that results are reported by failure category. We nevertheless agree that variance estimates and multiple-run controls should be added for rigor. In revision we will include standard-error estimates, clarify the number of evaluation runs performed, and expand the stratification tables. Because certain evaluation components (native compilation and timing) are deterministic, we will also note the practical limits on additional stochastic runs. revision: partial

Circularity Check

0 steps flagged

No significant circularity; empirical benchmark with independent evaluation protocol.

full rationale

The paper is an empirical benchmark study introducing CUDABEAVER with 213 tasks and the pass@k(M,C,A) metric. No equations, derivations, or fitted parameters are present that reduce reported success rates or performance shifts to quantities defined by the paper's own inputs or self-citations. The central claims rest on direct evaluation of LLMs on the collected tasks and explicit conditioning on protocol axes, which are independent of any internal fitting or definitional loop. Representativeness of the task set is an external assumption but does not create circularity in the reported measurements.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The evaluation rests on the domain assumption that the collected tasks and provided build/test commands suffice to distinguish genuine repairs from performance-degrading simplifications; no free parameters or invented entities are introduced in the abstract.

free parameters (1)
  • performance-loss tolerance threshold
    The metric explicitly varies tolerance for performance loss, which directly modulates reported success rates.
axioms (1)
  • domain assumption Native build/test commands and raw error evidence are sufficient to determine whether a candidate fix is both correct and performance-preserving.
    This premise underpins the entire pass@k(M,C,A) protocol and the claim that current evaluations are misleading.

pith-pipeline@v0.9.1-grok · 5797 in / 1350 out tokens · 38272 ms · 2026-06-30T22:52:26.027540+00:00 · methodology

0 comments
read the original abstract

Debugging CUDA programs has long been challenging because failures often arise from subtle interactions among hardware behavior, compiler decisions, memory hierarchy, and asynchronous execution. More importantly, with the rapid expansion of GPU usage across scientific computing, machine learning, graphics, and systems workloads, CUDA debugging has become more challenging than ever. Current evaluations of LLM-based CUDA programming largely miss this setting: a model can pass correctness tests with repair by degeneration, simplifying the CUDA code into a safer but slower program that abandons the original optimization structure. We introduce CUDABEAVER, a benchmark for CUDA debugging from real failing workspaces produced during LLM-based CUDA generation. Each task provides the broken candidate, native build/test commands, raw error evidence, and a single editable file. CUDABEAVER evaluates whether a fixer truly repairs the failing CUDA code or merely finds a slower test-passing replacement, reporting results by failure category, debugging trajectory, stagnation mode, and performance preservation. We further propose pass@k(M,C,A), a protocol-conditional CUDA debugging metric by making the fixer M, corpus C, and protocol axes Aexplicit. Using this metric across 213 tasks and seven frontier LLMs, we show that protocol-aware evaluation gives a more faithful view of CUDA debugging ability: when performance-loss tolerance is high, fixers appear much stronger, but even a minor stricter performance requirement can sharply reduce measured success, shifting scores by up to 40 percentage points.

Figures

Figures reproduced from arXiv: 2605.08455 by Caiwen Ding, Haoyang Chen, Mattia Fazzini, Shiyang Li.

Figure 1
Figure 1. Figure 1: Repair by degeneration. Here, the racecar denotes an optimized GPU kernel and the bicycle a correct-but-slow fallback; a useful repair preserves optimization structure, but LLMs often simplify candidates into slower correct pro￾grams. Existing benchmarks cannot expose repair by degeneration behavior of LLMs. On these benchmarks, eval￾uation starts from specifications and discards the failing intermediate p… view at source ↗
Figure 2
Figure 2. Figure 2: Corpus coverage across workload size (lines of code, log [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Iterative vs. repeated debugging pipelines (axis [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: (Left) Per-model degeneration evidence on the corpus’s eventually-passing tasks: median [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Feedback richness A3 (left) and conversation-history depth A4 (right). Per-model pass@k trajectories; star marks each model’s best setting, dashed grey line marks the default. We restrict to tasks each model solves at p = 0 and compare the model’s own iter-N pass to its own iter-1 pass. Such comparison controls for both reference-baseline strength and cross-model generation gap, leaving within-model tempor… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 20 canonical work pages · 14 internal anchors

  1. [1]

    When benchmarks are targets: Revealing the sensitivity of large language model leaderboards

    Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, et al. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistic...

    2024

  2. [2]

    Kevin: Multi-turn RL for generating CUDA kernels.CoRR, abs/2507.11948, 2025

    Carlo Baronio, Pietro Marsella, Ben Pan, Simon Guo, and Silas Alberti. Kevin: Multi-turn rl for generating cuda kernels.arXiv preprint arXiv:2507.11948, 2025

    work page arXiv 2025

  3. [3]

    An improved monte carlo factorization algorithm.BIT Numerical Mathematics, 20(2):176–184, 1980

    Richard P Brent. An improved monte carlo factorization algorithm.BIT Numerical Mathematics, 20(2):176–184, 1980

    1980

  4. [4]

    Evaluating Large Language Models Trained on Code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  5. [5]

    AVO : Agentic variation operators for autonomous evolutionary search

    Terry Chen, Zhifan Ye, Bing Xu, Zihao Ye, Timmy Liu, Ali Hassani, Tianqi Chen, Andrew Kerr, Haicheng Wu, Yang Xu, et al. Avo: Agentic variation operators for autonomous evolutionary search.arXiv preprint arXiv:2603.24517, 2026

    work page arXiv 2026

  6. [6]

    FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URLhttps://arxiv.org/abs/2307.08691

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  7. [7]

    University of Michigan, 1975

    Kenneth Alan De Jong.An analysis of the behavior of a class of genetic adaptive systems. University of Michigan, 1975

    1975

  8. [8]

    Random mapping statistics

    Philippe Flajolet and Andrew M Odlyzko. Random mapping statistics. InWorkshop on the Theory and Application of of Cryptographic Techniques, pages 329–354. Springer, 1989

    1989

  9. [9]

    Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation

    Mohsen Hariri, Amirhossein Samandar, Michael Hinczewski, and Vipin Chaudhary. Don’t pass@ k: A bayesian framework for large language model evaluation.arXiv preprint arXiv:2510.04265, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [10]

    The Curious Case of Neural Text Degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration.arXiv preprint arXiv:1904.09751, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1904

  11. [11]

    Swe-bench: Can language models resolve real-world github issues? In The twelfth international conference on learning representations, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The twelfth international conference on learning representations, 2023

    2023

  12. [12]

    Addison-Wesley Professional, 2014

    Donald E Knuth.The art of computer programming: Seminumerical algorithms, volume 2. Addison-Wesley Professional, 2014

    2014

  13. [13]

    Tritonbench: Benchmarking large language model capabilities for generating triton operators

    Jianling Li, Shangzhan Li, Zhenye Gao, Qi Shi, Yuxuan Li, Zefan Wang, Jiacheng Huang, WangHaojie WangHaojie, Jianrong Wang, Xu Han, et al. Tritonbench: Benchmarking large language model capabilities for generating triton operators. InFindings of the Association for Computational Linguistics: ACL 2025, pages 23053–23066, 2025

    2025

  14. [14]

    Stitchcuda: An automated multi-agents end-to-end gpu programing framework with rubric- based agentic reinforcement learning, 2026

    Shiyang Li, Zijian Zhang, Winson Chen, Yuebo Luo, Mingyi Hong, and Caiwen Ding. Stitchcuda: An automated multi-agents end-to-end gpu programing framework with rubric- based agentic reinforcement learning, 2026

    2026

  15. [15]

    Wei Liu, Jiawei Xu, Yingru Li, Longtao Zheng, Tianjian Li, Qian Liu, and Junxian He. Dr. kernel: Reinforcement learning done right for triton kernel generations.arXiv preprint arXiv:2602.05885, 2026

    work page arXiv 2026

  16. [16]

    Self-Refine: Iterative Refinement with Self-Feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- refine: Iterative refinement with self-feedback, 2023. URL https://arxiv.org/abs/ 2303.17651. 11

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  17. [17]

    MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention

    MiniMax, :, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, Chengjun Xiao, Chengyu Du, Chi Zhang, Chu Qiao, Chunhao Zhang, Chunhui Du, Congchao Guo, Da Chen, Deming Ding, Dianjun Sun, Dong Li, Enwei Jiao, Haigang Zhou, Haimo Zhang, Han Ding, Haohai Sun, Haoyu Feng, Huaiguang Cai, Haichao Z...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  18. [18]

    State of what art? a call for multi-prompt llm evaluation.Transactions of the Association for Computational Linguistics, 12:933–949, 2024

    Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of what art? a call for multi-prompt llm evaluation.Transactions of the Association for Computational Linguistics, 12:933–949, 2024

    2024

  19. [19]

    Computeeval: Evaluating large language models for cuda code generation

    NVIDIA. Computeeval: Evaluating large language models for cuda code generation. GitHub repository, 2025. URLhttps://github.com/NVIDIA/compute-eval

    2025

  20. [20]

    NVIDIA. Cutlass. GitHub repository, 2026. URL https://github.com/NVIDIA/ cutlass

    2026

  21. [21]

    Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama

    Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar- Lezama. Is self-repair a silver bullet for code generation?arXiv preprint arXiv:2306.09896, 2023

    work page arXiv 2023

  22. [22]

    KernelBench: Can LLMs Write Efficient GPU Kernels?

    Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. Kernelbench: Can llms write efficient gpu kernels?arXiv preprint arXiv:2502.10517, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [23]

    Perfcodegen: Improving performance of llm generated code with execution feedback

    Yun Peng, Akhilesh Deepak Gotmare, Michael R Lyu, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Perfcodegen: Improving performance of llm generated code with execution feedback. In2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge), pages 1–13. IEEE, 2025

    2025

  24. [24]

    John Wiley & Sons, 2008

    Andrea Saltelli, Marco Ratto, Terry Andres, Francesca Campolongo, Jessica Cariboni, Debora Gatelli, Michaela Saisana, and Stefano Tarantola.Global sensitivity analysis: the primer. John Wiley & Sons, 2008

    2008

  25. [25]

    Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.arXiv preprint arXiv:2310.11324, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  26. [26]

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker-Whitcomb, Alex Beutel, Alex Karpenko, Alex Makelov, Alex Neitz, Alex Wei, Alexandra Barr, Alexandre Kirchmeyer, Ale...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  27. [27]

    Hunting cuda bugs at scale with cufuzz

    Mohamed Tarek Ibn Ziad and Christos Kozyrakis. Hunting cuda bugs at scale with cufuzz. Proceedings of the ACM on Programming Languages, 10(OOPSLA1):877–904, 2026

    2026

  28. [28]

    Cucatch: A debugging tool for efficiently catching memory safety violations in cuda applications.Proceedings of the ACM on Programming Languages, 7(PLDI):124–147, 2023

    Mohamed Tarek Ibn Ziad, Sana Damani, Aamer Jaleel, Stephen W Keckler, and Mark Stephen- son. Cucatch: A debugging tool for efficiently catching memory safety violations in cuda applications.Proceedings of the ACM on Programming Languages, 7(PLDI):124–147, 2023

    2023

  29. [29]

    GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models

    5 Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, Bowe...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  30. [30]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Bey...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  31. [31]

    Kimi Team, Yifan Bai, Yiping Bao, Y . Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Chenxiao Gao, Hongcheng Gao, Peizhong G...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  32. [32]

    Thunderkittens

    ThunderKittens Authors. Thunderkittens. GitHub repository, 2026. URL https://github. com/HazyResearch/ThunderKittens

    2026

  33. [33]

    Debugbench: Evaluating debugging capability of large 15 language models

    Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Hui Haotian, Liu Weichuan, Zhiyuan Liu, et al. Debugbench: Evaluating debugging capability of large 15 language models. InFindings of the Association for Computational Linguistics: ACL 2024, pages 4173–4198, 2024

    2024

  34. [34]

    Characterizing and Detecting CUDA Program Bugs

    Mingyuan Wu, Husheng Zhou, Lingming Zhang, Cong Liu, and Yuqun Zhang. Characterizing and detecting cuda program bugs.arXiv preprint arXiv:1905.01833, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1905

  35. [35]

    arXiv preprint arXiv:2304.00385 , year=

    Chunqiu Steven Xia and Lingming Zhang. Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using chatgpt.arXiv preprint arXiv:2304.00385, 2023

    work page arXiv 2023

  36. [36]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  37. [37]

    Cudaforge: An agent framework with hardware feedback for cuda kernel optimization, 2025

    Zijian Zhang, Rong Wang, Shiyang Li, Yuebo Luo, Mingyi Hong, and Caiwen Ding. Cudaforge: An agent framework with hardware feedback for cuda kernel optimization, 2025

    2025

  38. [38]

    Fuzz4cuda: Fuzzing your nvidia gpu libraries through debug interface.Computers & Security, page 104754, 2025

    Yuhao Zhou, Peng Jia, Jiayong Liu, and Ximing Fan. Fuzz4cuda: Fuzzing your nvidia gpu libraries through debug interface.Computers & Security, page 104754, 2025

    2025

  39. [39]

    Cudabench: Benchmarking llms for text-to-cuda generation.arXiv preprint arXiv:2603.02236, 2026

    Jiace Zhu, Wentao Chen, Qi Fan, Zhixing Ren, Junying Wu, Xing Zhe Chai, Chotiwit Run- grueangwutthinon, Yehan Ma, and An Zou. Cudabench: Benchmarking llms for text-to-cuda generation.arXiv preprint arXiv:2603.02236, 2026. 16 A Asymmetric pass@kNormalisation When the fixer is also a broken-start source, its pipeline is structurally one step ahead: the brok...

    work page arXiv 2026