pith. sign in

arxiv: 2409.12186 · v3 · submitted 2024-09-18 · 💻 cs.CL

Qwen2.5-Coder Technical Report

Binyuan Hui , Jian Yang , Zeyu Cui , Jiaxi Yang , Dayiheng Liu , Lei Zhang , Tianyu Liu , Jiajun Zhang
show 16 more authors
Bowen Yu Keming Lu Kai Dang Yang Fan Yichang Zhang An Yang Rui Men Fei Huang Bo Zheng Yibo Miao Shanghaoran Quan Yunlong Feng Xingzhang Ren Xuancheng Ren Jingren Zhou Junyang Lin
This is my paper

Pith reviewed 2026-05-10 12:28 UTC · model grok-4.3

classification 💻 cs.CL
keywords Qwen2.5-Codercode generationlarge language modelspretrainingsynthetic datacode benchmarksmodel evaluationcode repair
0
0 comments X

The pith

Qwen2.5-Coder models reach state-of-the-art code performance across sizes by continued pretraining on over 5.5 trillion tokens.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces the Qwen2.5-Coder series of models sized from 0.5 billion to 32 billion parameters as an upgrade over earlier code-focused versions. It builds on the Qwen2.5 base through continued pretraining on a large code corpus combined with data cleaning, synthetic data creation, and balanced mixing of sources. This process produces strong results on code generation, completion, reasoning, and repair tasks. The models often beat larger models of similar scale while keeping general knowledge and math abilities intact. The work matters because it points to practical ways to build capable coding tools that developers can run and adapt without needing the biggest possible systems.

Core claim

The Qwen2.5-Coder series, built on the Qwen2.5 architecture and continued pretrained on over 5.5 trillion tokens through meticulous data cleaning, scalable synthetic data generation, and balanced data mixing, achieves state-of-the-art performance across more than 10 benchmarks for code generation, completion, reasoning, and repair while retaining general and math skills and consistently outperforming larger models of the same size.

What carries the argument

Continued pretraining on a vast code corpus of over 5.5 trillion tokens using data cleaning, synthetic data generation, and balanced mixing on the Qwen2.5 architecture.

If this is right

  • Code generation and repair tasks become solvable at high quality with models that fit on modest hardware.
  • Specialized training can produce code skills that exceed what raw size alone delivers in competing models.
  • General and math performance stays available, so the models function as versatile assistants rather than narrow tools.
  • Permissive licensing allows direct integration into developer workflows and further research without restrictions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same data preparation steps could transfer to other narrow domains if comparable volumes of clean and synthetic data exist.
  • Smaller models in the series open the door to on-device code completion and debugging features in everyday software.
  • Combining these models with existing general-purpose systems might create hybrid setups that handle mixed coding and non-coding queries efficiently.

Load-bearing premise

The chosen benchmarks and evaluation conditions provide a fair, unbiased measure of real code capabilities that allows direct comparison to other models.

What would settle it

An independent test on a fresh collection of real developer code problems from open repositories where the Qwen2.5-Coder models fail to match or exceed the performance of larger models of the same size.

read the original abstract

In this report, we introduce the Qwen2.5-Coder series, a significant upgrade from its predecessor, CodeQwen1.5. This series includes six models: Qwen2.5-Coder-(0.5B/1.5B/3B/7B/14B/32B). As a code-specific model, Qwen2.5-Coder is built upon the Qwen2.5 architecture and continues pretrained on a vast corpus of over 5.5 trillion tokens. Through meticulous data cleaning, scalable synthetic data generation, and balanced data mixing, Qwen2.5-Coder demonstrates impressive code generation capabilities while retaining general and math skills. These models have been evaluated on a wide range of code-related tasks, achieving state-of-the-art (SOTA) performance across more than 10 benchmarks, including code generation, completion, reasoning, and repair, consistently outperforming larger models of the same model size. We believe that the release of the Qwen2.5-Coder series will advance research in code intelligence and, with its permissive licensing, support wider adoption by developers in real-world applications.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript introduces the Qwen2.5-Coder series of six code-specialized models (0.5B to 32B parameters) built on the Qwen2.5 architecture. These undergo continued pretraining on a 5.5-trillion-token code corpus using data cleaning, scalable synthetic data generation, and balanced mixing. The report claims the resulting models achieve state-of-the-art performance on more than 10 benchmarks spanning code generation, completion, reasoning, and repair, while retaining general and math capabilities, and consistently outperform larger models of equivalent size.

Significance. If the performance claims are substantiated with reproducible details, the work would be significant for releasing a family of strong, permissively licensed code models at multiple scales. The scale of the continued pretraining corpus and the explicit effort to preserve non-code skills via balanced mixing represent a practical contribution to specialized LLM development that could support both research and developer adoption.

major comments (3)
  1. [Abstract] Abstract: The central claim of 'state-of-the-art (SOTA) performance across more than 10 benchmarks' and 'consistently outperforming larger models of the same model size' supplies no benchmark names, baseline models, evaluation methodology (prompting format, few-shot count, decoding parameters, temperature/top-p), error bars, or statistical tests. This absence prevents verification of whether the data support the outperformance assertion.
  2. [Pretraining description] Pretraining description: Continued pretraining on >5.5 trillion tokens creates a material risk of test-set contamination for the cited code benchmarks. The manuscript provides no description of decontamination procedures, overlap checks, or synthetic-data filtering steps that would be required to support the integrity of the SOTA results.
  3. [Evaluation section] Evaluation section: No information is given on whether all compared models (including larger baselines) were evaluated under identical conditions, benchmark versions, or prompting setups. Any deviation would undermine the cross-model size comparison that is load-bearing for the main claim.
minor comments (2)
  1. [Abstract] The model-size notation 'Qwen2.5-Coder-(0.5B/1.5B/3B/7B/14B/32B)' is compact but could be expanded into a clearer bulleted list for readability.
  2. [Abstract] The phrase 'impressive code generation capabilities' is subjective; replacing it with a brief quantitative reference to the claimed benchmark gains would improve precision.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We agree that the current manuscript would benefit from greater specificity in the abstract, pretraining description, and evaluation section to improve verifiability and address potential concerns about contamination and fair comparison. We will incorporate revisions to resolve these issues.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim of 'state-of-the-art (SOTA) performance across more than 10 benchmarks' and 'consistently outperforming larger models of the same model size' supplies no benchmark names, baseline models, evaluation methodology (prompting format, few-shot count, decoding parameters, temperature/top-p), error bars, or statistical tests. This absence prevents verification of whether the data support the outperformance assertion.

    Authors: We agree that the abstract would be strengthened by naming the primary benchmarks and baselines and by briefly indicating the evaluation protocol. In the revised manuscript we will expand the abstract to list the key benchmarks (HumanEval, MBPP, LiveCodeBench, BigCodeBench, etc.), the main comparison models, and a concise statement of the shared prompting and decoding settings. Full tables with per-benchmark scores, error bars, and statistical comparisons will remain in the Evaluation section, but the abstract will now reference them explicitly. revision: yes

  2. Referee: [Pretraining description] Pretraining description: Continued pretraining on >5.5 trillion tokens creates a material risk of test-set contamination for the cited code benchmarks. The manuscript provides no description of decontamination procedures, overlap checks, or synthetic-data filtering steps that would be required to support the integrity of the SOTA results.

    Authors: This is a legitimate concern. The current manuscript does not describe decontamination steps. We will add a new subsection under Data Preparation that details (1) n-gram and embedding-based overlap checks performed against the public versions of the evaluation benchmarks, (2) removal of any detected contaminated samples from the 5.5-trillion-token corpus, and (3) the filtering rules applied during synthetic data generation to prevent benchmark leakage. These procedures were followed during training and will now be documented. revision: yes

  3. Referee: [Evaluation section] Evaluation section: No information is given on whether all compared models (including larger baselines) were evaluated under identical conditions, benchmark versions, or prompting setups. Any deviation would undermine the cross-model size comparison that is load-bearing for the main claim.

    Authors: We confirm that every model—including the larger baselines—was run under a single, fixed evaluation harness using identical benchmark versions, prompt templates, few-shot counts, and decoding parameters (temperature 0.2, top-p 0.95, max tokens 512). The manuscript simply omits an explicit statement of this uniformity. In the revision we will insert a dedicated paragraph at the start of the Evaluation section that enumerates the common protocol, benchmark versions, and hyper-parameters so that the size-comparison claims rest on clearly documented identical conditions. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical SOTA claims rest on external benchmarks

full rationale

The paper reports continued pretraining of Qwen2.5-based models on a 5.5T-token code corpus, followed by data cleaning, synthetic data generation, and balanced mixing, then direct evaluation on public code benchmarks. No equations, fitted parameters, or derivations are present that could reduce to self-definition or self-citation. Performance claims compare against external models under stated conditions; the chain is self-contained against independent benchmarks and does not invoke any of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract provides no explicit free parameters, axioms, or invented entities; the work implicitly relies on standard transformer pretraining assumptions common to LLM literature.

axioms (1)
  • domain assumption Transformer architecture is effective for modeling code sequences
    Models are built upon the Qwen2.5 architecture

pith-pipeline@v0.9.0 · 5578 in / 1094 out tokens · 62237 ms · 2026-05-10T12:28:34.514129+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. UniQL: Towards Dialect-Universal Benchmarking for Text-to-SQL

    cs.AI 2026-06 unverdicted novelty 8.0

    UniQL is a human-verified benchmark providing aligned natural language questions and dialect-specific SQL queries for 16 SQL systems to evaluate cross-dialect generalization.

  2. HKJudge: A Legal Discourse-Annotated Corpus for Interpreting What Courts Find, How They Reason, and What They Rule

    cs.CL 2026-06 accept novelty 8.0

    HKJudge is a new ~290k-sentence expert-annotated corpus of Hong Kong criminal judgments with 26 rhetorical roles and 3 sentencing elements, plus benchmarks on classification and extraction tasks.

  3. HM-Bench: A Comprehensive Benchmark for Multimodal Large Language Models in Hyperspectral Remote Sensing

    cs.CV 2026-04 accept novelty 8.0

    HM-Bench is the first benchmark for MLLMs on hyperspectral images, showing models struggle with complex spatial-spectral reasoning and perform better with visual PCA images than textual reports.

  4. Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing

    cs.CR 2026-04 unverdicted novelty 8.0

    The first SoK on LLM-based AutoPT frameworks provides a six-dimension taxonomy of agent designs and a unified empirical benchmark evaluating 15 frameworks via over 10 billion tokens and 1,500 manually reviewed logs.

  5. Why Do Multi-Agent LLM Systems Fail?

    cs.AI 2025-03 unverdicted novelty 8.0

    The authors create the first large-scale dataset and taxonomy of failure modes in multi-agent LLM systems to explain their limited performance gains.

  6. Training Software Engineering Agents and Verifiers with SWE-Gym

    cs.SE 2024-12 conditional novelty 8.0

    SWE-Gym supplies 2438 executable real-world Python tasks to train SWE agents and verifiers, yielding up to 19% gains and new open-weight SOTA of 32% on SWE-Bench Verified.

  7. Regression Accumulation in Multi-Turn LLM Programming Conversations

    cs.SE 2026-07 conditional novelty 7.0

    Regression accumulation affects 40-73% of 8-turn LLM coding tasks on extended HumanEval+/MBPP+ benchmarks, with verification gates improving final-turn pass rates on prior tests.

  8. The Illusion of Safety: Multi-Tier Verification of AI vs. Human C++ Code

    cs.SE 2026-06 unverdicted novelty 7.0

    Multi-tier verification on VULBENCH-CPP shows AI-generated C++ code triggers confirmed runtime violations roughly twice as often as human code, while static analysis misleadingly indicates parity due to code length.

  9. Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Models

    cs.SE 2026-06 unverdicted novelty 7.0

    Preregistered placebo-controlled decomposition shows external executable counterexamples drive self-repair gains in small code models more than re-exposure or self-critique.

  10. FlipGuard: Defending Large Language Models Against Quantization-Conditioned Backdoor Attacks

    cs.CR 2026-06 unverdicted novelty 7.0

    FlipGuard perturbs LLM weights prior to quantization to neutralize quantization-conditioned backdoor attacks, evaluated via the Defense Effectiveness Ratio on multiple models and quantization schemes.

  11. CodeChat-Eval: Evaluating Large Language Models in Multi-Turn Code Refinement Dialogues

    cs.SE 2026-06 unverdicted novelty 7.0

    CodeChat-Eval shows LLMs lose 19.2% to 69.2% functional correctness over multi-turn refinement dialogues, with largest drops on logic-level and additive changes.

  12. CodeChat-Eval: Evaluating Large Language Models in Multi-Turn Code Refinement Dialogues

    cs.SE 2026-06 unverdicted novelty 7.0

    CodeChat-Eval shows LLMs lose 19-69% functional correctness across multi-turn code refinement dialogues, with largest drops on logic changes and additive requests.

  13. Cliff Tokens: Identifying Single-Token Failure Triggers in LLM Mathematical Reasoning

    cs.AI 2026-06 conditional novelty 7.0

    Cliff tokens are single tokens triggering LLM math reasoning failures, identified via adaptive z-test threshold on token potential; a taxonomy and Cliff-DPO optimization yield up to +6.6 accuracy gains.

  14. Cliff Tokens: Identifying Single-Token Failure Triggers in LLM Mathematical Reasoning

    cs.AI 2026-06 unverdicted novelty 7.0

    Cliff tokens are single-token failure triggers in LLM mathematical reasoning identified via adaptive statistical threshold; intervening at them recovers performance to 1.0 in resampling and yields up to +6.6 accuracy ...

  15. BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language

    cs.CL 2026-06 unverdicted novelty 7.0

    BioMatrix unifies sequences, structures, and language for molecules and proteins inside one decoder-only foundation model via shared discrete tokens and achieves SOTA or competitive results on 77 of 80 downstream tasks.

  16. Agent-Assisted Side-Channel Attacks on Non-Prefix KV Cache in RAG

    cs.CR 2026-06 unverdicted novelty 7.0

    SpliceLeak is the first end-to-end side-channel attack on non-prefix KV cache in RAG, using Step-Wave timing leaks to fingerprint private prompt lengths and extract tokens with up to 100% success using 63 requests per...

  17. The Alignment Problem in Constrained Code Generation

    cs.SE 2026-06 unverdicted novelty 7.0

    Incomplete constrainers in constrained decoding push LLMs into low-probability program regions, making unconstrained decoding outperform constrained decoding on functional correctness across seven models and three benchmarks.

  18. When Do Intrinsic Rewards Work for Code Reasoning? A Comprehensive Study

    cs.AI 2026-06 unverdicted novelty 7.0

    Empirical evaluation on LiveCodeBench shows certainty-based RLIF yields early gains followed by output shortening and reasoning collapse, providing no advantage for RLVR initialization on code tasks.

  19. Repository-Level Solidity Code Generation with Large Language Models: From Prompting to Fine-Tuning

    cs.SE 2026-06 unverdicted novelty 7.0

    Introduces SolidityBench benchmark and SolidityScore metric for repository-level Solidity code generation, finding supervised fine-tuning outperforms prompting, CoT, ICL, and RAG methods on evaluated LLMs.

  20. Bridging Functional Correctness and Runtime Efficiency Gaps in LLM-Based Code Translation

    cs.CL 2026-06 unverdicted novelty 7.0

    SwiftTrans improves both functional correctness and runtime efficiency of LLM code translations via multi-perspective exploration with hierarchical guidance and difference-aware selection with ordinal guidance on exte...

  21. AgentRivet: an automated system for producing Rivet routines from journal publications

    hep-ex 2026-06 unverdicted novelty 7.0

    AgentRivet applies commercial LLMs in an autonomous workflow to extract physics details from ATLAS and CMS papers and generate Rivet routines, achieving few syntax errors but occasional physics implementation issues o...

  22. Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMs

    cs.CL 2026-06 unverdicted novelty 7.0

    ModSleuth reconstructs dependency graphs from public artifacts for four LLM releases, recovering 1,060 source-verified dependencies and exposing license issues, train-evaluation coupling, and documentation gaps.

  23. CODEBLOCK: Learning to Supervise Code at the Right Granularity

    cs.LG 2026-06 unverdicted novelty 7.0

    CodeBlock partitions code responses into syntactically coherent blocks, scores them with generalized cross-entropy and data-flow signals, and applies sparse supervision to achieve higher pass@1 than full SFT using 1.9...

  24. INFRAMIND: Infrastructure-Aware Multi-Agent Orchestration

    cs.AI 2026-06 unverdicted novelty 7.0

    INFRAMIND is an infrastructure-aware multi-agent orchestration framework that uses RL on a hierarchical constrained MDP to jointly optimize topology, model selection, and scheduling under dynamic load.

  25. OpenRTLSet: A Fully Open-Source Dataset for Large Language Model-based Verilog Module Design

    cs.CL 2026-06 unverdicted novelty 7.0

    OpenRTLSet supplies 131k+ Verilog samples with AI-generated descriptions to enable fine-tuning of LLMs for hardware module design.

  26. BenSyc: Benchmarking Conversational Sycophancy and Human Alignment in LLMs for Bengali Contexts

    cs.CL 2026-06 unverdicted novelty 7.0

    BenSyc is the first benchmark for conversational sycophancy in Bengali, with top LLMs achieving only 61.8 Macro-F1 on binary detection and 61.7 on five-class classification while often generating overly validating responses.

  27. PrivCode++: Latent-Conditioned Differentially Private Code Generation for Comprehensive Guarantees

    cs.CR 2026-06 unverdicted novelty 7.0

    PrivCode++ introduces the first DP code generation method protecting both prompts and code via latent-conditioned two-stage training, claiming higher utility and stronger privacy than prior baselines.

  28. The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning

    cs.LG 2026-06 unverdicted novelty 7.0

    PRISM is a contrastive, policy-aware training framework for process reward models that reduces false positives by 22% on PRMBench and boosts downstream accuracy up to 33% in Best-of-N selection by learning reliable re...

  29. When LLMs Invent Rust Crates: An Empirical Study of Hallucination Patterns and Mitigation

    cs.SE 2026-06 unverdicted novelty 7.0

    First empirical study shows crate hallucination in Rust LLMs has consistent rates across models insensitive to parameters and tests prompt-based mitigation.

  30. Towards Unified Song Generation and Singing Voice Conversion with Accompaniment Co-Generation

    cs.SD 2026-06 unverdicted novelty 7.0

    UniSinger unifies speaker-cloned song generation and accompaniment co-generation SVC in one multimodal diffusion transformer model trained with curriculum learning via task-specific modality masking.

  31. SkelDPO: A Skeleton-Guided Direct Preference Optimization Framework for Efficient Code Generation

    cs.SE 2026-06 unverdicted novelty 7.0

    SkelDPO improves code generation efficiency by 2-7% over prior DPO methods via joint preference losses on full code and efficiency-critical skeletons.

  32. ACE-SQL: Adaptive Co-Optimization via Empirical Credit Assignment for Text-to-SQL

    cs.CL 2026-06 unverdicted novelty 7.0

    ACE-SQL jointly optimizes schema linking and SQL generation via RL with empirical credit assignment from execution-correct rollouts, achieving 65.3% greedy execution accuracy on BIRD Dev using 0.93k output tokens.

  33. TeleSWEBench: A Commit-Driven Benchmark for Evaluating LLM-Powered Software Engineering in Telecommunications

    cs.SE 2026-06 unverdicted novelty 7.0

    Presents TeleSWEBench, the first commit-driven benchmark with 734 unit-test cases from srsRAN 5G plus TeleJudge LLM evaluator, showing top ASE tools achieve up to 25% functional success on telecom tasks.

  34. From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression

    cs.CL 2026-06 unverdicted novelty 7.0

    SubFit enables better LLM compression by fitting residual bypasses to non-contiguously selected submodules, outperforming layer-granularity baselines in accuracy-perplexity trade-offs at 12.5-37.5% sparsity.

  35. From Talking Words to Sharing Thoughts: Scalable Multi-LLM Aggregation via Structured Message Passing

    cs.GT 2026-05 unverdicted novelty 7.0

    A bipartite factor graph with message-passing protocol and asymmetric damping aggregates multi-LLM predictions, cutting token use by 97% and API calls by 6X while outperforming baselines on MMLU, MMLU-Pro, GPQA, and MedMCQA.

  36. How Reliable Are AI Attackers Against a Fixed Vulnerable Target? A 400-Run Empirical Study of LLM Penetration Testing Consistency

    cs.CR 2026-05 unverdicted novelty 7.0

    Empirical study of 400 LLM attack runs finds exploitation success rates of 25-85% across four models against a fixed multi-service honeypot, with model-distinctive failure modes and p<0.001 differences.

  37. Inference-Time Alignment of Diffusion Models via Trust-Region Iterative Twisted Sequential Monte Carlo

    cs.LG 2026-05 conditional novelty 7.0

    TRI-TSMC is a trust-region framework for learning twisting functions in SMC-based inference-time alignment of diffusion models that yields zero-variance samplers in theory and better alignment on text and image tasks ...

  38. DepthAgent: Towards Better Universal Depth Estimation via Sample-wise Expert Selection

    cs.CV 2026-05 unverdicted novelty 7.0

    A reinforcement-learned vision-language agent adaptively selects and fuses monocular depth experts per sample for better performance across camera geometries.

  39. Distribution-Aware Reward: Reinforcement Learning over Predictive Distributions for LLM Regression

    cs.LG 2026-05 unverdicted novelty 7.0

    Distribution-Aware Reward optimizes LLM regression by treating rollouts as empirical predictive distributions and rewarding marginal improvements in CRPS quality rather than point accuracy alone.

  40. Code Generation by Differential Test Time Scaling

    cs.SE 2026-05 unverdicted novelty 7.0

    DiffCodeGen clusters code candidates by behavioral similarity from fuzzing-synthesized inputs and selects the largest cluster's medoid, matching or exceeding prior test-time scaling methods with far less token and time cost.

  41. CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning

    cs.CL 2026-05 unverdicted novelty 7.0

    CopT reverses CoT by eliciting a draft answer first then using continuous-embedding contrastive verification and on-policy thinking to reflect and correct, yielding up to 23% higher accuracy and 57% fewer tokens witho...

  42. Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models

    cs.CV 2026-05 conditional novelty 7.0

    SP-CoR is a multimodal LLM framework using dynamics-aware sampling, spectral-physics view fusion, and prompt distillation that outperforms baselines on the new CoopSR benchmark and EgoTeam dataset for multi-robot coop...

  43. Text2CAD-Bench: A Benchmark for LLM-based Text-to-Parametric CAD Generation

    cs.LG 2026-05 unverdicted novelty 7.0

    Text2CAD-Bench supplies 600 dual-prompt examples across four geometric and domain levels to test LLMs on text-to-parametric CAD, finding solid basic performance but sharp drops on complex topology and advanced features.

  44. Constrained Code Generation with Discrete Diffusion

    cs.CL 2026-05 unverdicted novelty 7.0

    Constrained Diffusion for Code (CDC) integrates constraint satisfaction into the reverse denoising process of discrete diffusion models via constraint-aware operators that use optimization and program analysis to stee...

  45. Echo-Forcing: A Scene Memory Framework for Interactive Long Video Generation

    cs.CV 2026-05 unverdicted novelty 7.0

    Echo-Forcing decouples stable anchors, compressed history, and recent dynamics in video diffusion KV caches using hierarchical memory, scene recall frames, and difference-aware decay to support interactive long video ...

  46. Hydra: Efficient, Correct Code Generation via Checkpoint-and-Rollback Support

    cs.SE 2026-05 unverdicted novelty 7.0

    Hydra enables asynchronous static error checking and targeted checkpoint-rollback repair during LLM code generation, cutting latency by up to 71% and token use by up to 70% versus post-hoc repair on C/C++ tasks.

  47. Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench

    cs.AR 2026-05 unverdicted novelty 7.0

    Phoenix-bench shows agentic AI systems lose 37-58% resolved rate when moving from SWE-bench Verified to hardware tasks because bugs spread across parallel modules via signal flow, with testbench feedback lifting perfo...

  48. HLS-Seek: QoR-Aware Code Generation for High-Level Synthesis via Proxy Comparative Reward Reinforcement Learning

    cs.LG 2026-05 unverdicted novelty 7.0

    HLS-Seek replaces full-synthesis RL with a comparative proxy reward model plus uncertainty-triggered real checks, yielding higher correctness and better QoR than larger models at 8.5x lower training cost.

  49. Reward-Weighted On-Policy Distillation with an Open Property-Equivalence Verifier for NL-to-SVA Generation

    cs.AR 2026-05 unverdicted novelty 7.0

    Reward-Weighted On-Policy Distillation with an open property-equivalence verifier produces a 7B model that surpasses prior SOTA on NL-to-SVA generation across pass@1/5/10 metrics.

  50. IndicMedDialog: A Parallel Multi-Turn Medical Dialogue Dataset for Accessible Healthcare in Indic Languages

    cs.CL 2026-05 unverdicted novelty 7.0

    A parallel multi-turn medical dialogue dataset spanning English and nine Indic languages is created from synthetic consultations to enable personalized AI healthcare interactions.

  51. Retrieval is Cheap, Show Me the Code: Executable Multi-Hop Reasoning for Retrieval-Augmented Generation

    cs.AI 2026-05 unverdicted novelty 7.0

    PyRAG turns multi-hop reasoning into executable Python code over retrieval tools for explicit, verifiable step-by-step RAG.

  52. UHR-Micro: Diagnosing and Mitigating the Resolution Illusion in Earth Observation VLMs

    cs.CV 2026-05 unverdicted novelty 7.0

    VLMs show a resolution illusion on UHR Earth observation imagery where higher resolution does not improve micro-target perception; UHR-Micro benchmark and MAP-Agent address this via evidence-centered active inspection.

  53. A Recipe for Long-Context Reasoning in Large Language Models via On-Policy Optimization and Distillation

    cs.CL 2026-05 unverdicted novelty 7.0

    dGRPO merges outcome-based policy optimization with dense teacher guidance from on-policy distillation, yielding more stable long-context reasoning on the new LongBlocks synthetic dataset.

  54. Towards Order Fairness: Mitigating LLMs Order Sensitivity through Dual Group Advantage Optimization

    cs.LG 2026-05 unverdicted novelty 7.0

    DGAO uses reinforcement learning to optimize LLMs for both accuracy and order stability by balancing intra-group accuracy advantages and inter-group stability advantages.

  55. StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning

    cs.SE 2026-05 unverdicted novelty 7.0

    StepCodeReasoner aligns code reasoning with verifiable stepwise execution traces via print anchors and bi-level GRPO reinforcement learning, reaching SOTA results on CRUXEval (91.1%) and LiveCodeBench (86.5%) for a 7B model.

  56. SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation

    cs.CV 2026-05 unverdicted novelty 7.0

    SciVQR is a new benchmark dataset for evaluating multimodal AI models on complex scientific reasoning tasks across six disciplines, including expert solutions for nearly half the items.

  57. PlantMarkerBench: A Multi-Species Benchmark for Evidence-Grounded Plant Marker Reasoning

    cs.CL 2026-05 unverdicted novelty 7.0

    PlantMarkerBench is a new multi-species benchmark with 5,550 evidence instances for evaluating language models on literature-grounded plant marker gene reasoning across expression, localization, function, indirect, an...

  58. PlantMarkerBench: A Multi-Species Benchmark for Evidence-Grounded Plant Marker Reasoning

    cs.CL 2026-05 unverdicted novelty 7.0

    PlantMarkerBench supplies 5,550 literature sentences annotated for plant marker gene evidence validity and type across Arabidopsis, maize, rice and tomato, showing frontier LLMs handle direct expression evidence but s...

  59. Trust Me, Import This: Dependency Steering Attacks via Malicious Agent Skills

    cs.CR 2026-05 unverdicted novelty 7.0

    Malicious Skills induce coding agents to hallucinate and import attacker-controlled packages at high rates while evading detection.

  60. EpiGraph: Building Generalists for Evidence-Intensive Epilepsy Reasoning in the Wild

    cs.AI 2026-05 unverdicted novelty 7.0

    EpiGraph is a new epilepsy knowledge graph with 24,324 entities and 32,009 triplets that improves LLM performance on clinical tasks by up to 41% when used in Graph-RAG.

Reference graph

Works this paper leans on

42 extracted references · 42 canonical work pages · cited by 339 Pith papers · 22 internal anchors

  1. [1]

    GPT-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    arXiv preprint arXiv:2301.03988 , year=

    Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Car- los Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don’t reach for the stars! arXiv preprint arXiv:2301.03988,

    work page arXiv

  3. [3]

    Program Synthesis with Large Language Models

    2024.06.21. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732,

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    Qwen Technical Report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Efficient Training of Language Models to Fill in the Middle

    Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255,

    work page internal anchor Pith review arXiv

  6. [6]

    Language Models are Few-Shot Learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165,

    work page internal anchor Pith review Pith/arXiv arXiv 2005

  7. [7]

    arXiv preprint arXiv:2208.08227 , year=

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feld- man, et al. Multipl-e: A scalable and extensible approach to benchmarking neural code generation. arXiv preprint arXiv:2208.08227,

    work page arXiv

  8. [8]

    arXiv preprint arXiv:2406.07436 , year=

    Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436,

    work page arXiv

  9. [9]

    How to prompt LLMs for text-to-SQL: A study in zero-shot, single- domain, and cross-domain settings

    Shuaichen Chang and Eric Fosler-Lussier. How to prompt llms for text-to-sql: A study in zero-shot, single-domain, and cross-domain settings. arXiv preprint arXiv:2305.11853,

    work page arXiv

  10. [10]

    Evaluating Large Language Models Trained on Code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. arXiv preprint arXiv:2107.03374,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Theoremqa: A theorem-driven question answering dataset

    Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem-driven question answering dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7889–7901,

    work page 2023

  12. [12]

    Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132,

    work page internal anchor Pith review arXiv

  13. [13]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    The Llama 3 Herd of Models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Codebert: A pre-trained model for programming and natural languages

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: A pre-trained model for programming and natural languages. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, v...

    work page 2020

  17. [17]

    Hu, H., Richardson, K., Xu, L., Li, L., K ¨ubler, S., and Moss, L

    doi: 10.18653/V1/2020.FINDINGS-EMNLP .139. URL https://doi.org/10.18653/v1/2020.findings-emnlp.139. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? arXiv preprint arXiv:2406.04127,

    work page doi:10.18653/v1/2020.findings-emnlp 2020

  18. [18]

    arXiv preprint arXiv:2403.04814 , year=

    Linyuan Gong, Sida Wang, Mostafa Elhoushi, and Alvin Cheung. Evaluation of llms on syntax-aware code fill-in-the-middle tasks. arXiv preprint arXiv:2403.04814,

    work page arXiv

  19. [19]

    CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution

    Alex Gu, Baptiste Rozi`ere, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065,

    work page internal anchor Pith review arXiv

  20. [20]

    DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024a. Jiawei Guo, Ziming Li, Xueling Liu, Kaijing Ma, Tianyu Zheng, Zhouliang Yu, Ding Pan, Yizhi Li, Ruibo Liu, ...

    work page internal anchor Pith review arXiv 2009

  21. [21]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874,

    work page internal anchor Pith review arXiv

  22. [22]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contami- nation free evaluation of large language models for code. arXiv preprint arXiv:2403.07974,

    work page internal anchor Pith review arXiv

  23. [23]

    Mistral 7B

    AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, D de las Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b (2023). arXiv preprint arXiv:2310.06825,

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  24. [24]

    StarCoder: may the source be with you!

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024a. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis ...

    work page internal anchor Pith review arXiv

  25. [25]

    Autokaggle: A multi-agent framework for autonomous data science competitions.arXiv preprint arXiv:2410.20424, 2024

    30 Technical Report Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tianyu Zheng, Xinyao Niu, Xiang Yue, Yue Wang, Jian Yang, Jiaheng Liu, et al. Autokaggle: A multi-agent framework for autonomous data science competitions. arXiv preprint arXiv:2410.20424, 2024b. Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human fals...

    work page arXiv

  26. [26]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    J Liu, CS Xia, Y Wang, and L Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arxiv preprint arxiv: 230501210. 2023,

    work page 2023

  27. [27]

    M2rc-eval: Massively multilingual repository-level code completion evaluation

    Jiaheng Liu, Ken Deng, Congnan Liu, Jian Yang, Shukai Liu, He Zhu, Peng Zhao, Linzheng Chai, Yanan Wu, Ke Jin, et al. M2rc-eval: Massively multilingual repository-level code completion evaluation. arXiv preprint arXiv:2410.21157, 2024a. Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, et al. ...

    work page arXiv

  28. [28]

    Reacc: A retrieval-augmented code completion framework

    Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, and Alexey Svy- atkovskiy. Reacc: A retrieval-augmented code completion framework. arXiv preprint arXiv:2203.07722,

    work page arXiv

  29. [29]

    2024.05.29. OpenAI. Gpt-4o. https://openai.com/index/hello-gpt-4o,

    work page 2024

  30. [30]

    YaRN: Efficient Context Window Extension of Large Language Models

    2024.05.13. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071,

    work page internal anchor Pith review arXiv 2024

  31. [31]

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model

    URL https://qwenlm.github.io/blog/ codeqwen1.5/. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290,

    work page internal anchor Pith review arXiv

  32. [32]

    Code Llama: Open Foundation Models for Code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950,

    work page internal anchor Pith review arXiv

  33. [33]

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641,

    work page internal anchor Pith review arXiv 1907

  34. [34]

    Unicoder: Scaling code large language model via universal code

    Tao Sun, Linzheng Chai, Jian Yang, Yuwei Yin, Hongcheng Guo, Jiaheng Liu, Bing Wang, Liqun Yang, and Zhoujun Li. Unicoder: Scaling code large language model via universal code. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, ...

    work page 2024

  35. [35]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    URL https://aclanthology.org/2024.acl-long.100. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  36. [36]

    Magicoder: Em- powering code generation with oss-instruct

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Em- powering code generation with oss-instruct. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net,

    work page 2024

  37. [37]

    U.; Zhang, D.; Ramanathan, M

    URL https://openreview.net/forum?id=XUeoOBid3x. 31 Technical Report Di Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan, and Xiaofei Ma. Repoformer: Selective retrieval for repository-level code completion. arXiv preprint arXiv:2403.10059, 2024a. Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xinrun Du, Di Liang, Daixin Shu, Xia...

    work page arXiv

  38. [38]

    Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task.arXiv preprint arXiv:1809.08887,

    work page Pith review arXiv

  39. [39]

    Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning

    Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume...

    work page 2024

  40. [40]

    SaySelf: Teaching LLMs to express confidence with self-reflective rationales

    doi: 10.18653/V1/2024. ACL-LONG.280. URL https://doi.org/10.18653/v1/2024.acl-long.280. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    work page doi:10.18653/v1/2024 2024

  41. [41]

    (2023).RepoCoder: Repository-level code completion through iterative retrieval and generation

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian- Guang Lou, and Weizhu Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570,

    work page arXiv

  42. [42]

    BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877,

    work page internal anchor Pith review arXiv