Pith. sign in

REVIEW 2 major objections 1 minor 47 references

WeaveBench shows frontier agents achieve at most 41.2 percent success on long-horizon tasks that mix GUI, CLI, and code operations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-27 16:27 UTC pith:UA54UX4B

load-bearing objection WeaveBench supplies a hybrid GUI/CLI/code benchmark with a trajectory judge that exposes overestimation, but task sourcing details are too thin to fully back the saturation claim. the 2 major comments →

arxiv 2606.09426 v2 pith:UA54UX4B submitted 2026-06-08 cs.AI

WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

Wanli Li , Bowen Zhou , Yunyao Yu , Zhou Xu , Yifan Yang , Dongsheng Li , Caihua Shan This is my paper
classification cs.AI
keywords computer-use agentshybrid interfaceslong-horizon tasksbenchmarkGUICLItrajectory evaluationagent orchestration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents WeaveBench as a benchmark of 114 tasks drawn from real user requests across eight work domains, each requiring agents to interleave visual desktop actions with command-line and code operations inside one extended trajectory on an actual Ubuntu desktop. It claims that prior benchmarks evaluate interfaces in isolation and therefore miss the orchestration demands of practical hybrid work. A trajectory-aware judge is introduced that reviews full action traces, files, logs, and screenshots to catch shortcuts such as fabricated evidence, and this judge shows that outcome-only scoring overestimates performance. Evaluations across frontier model-runtime combinations establish a ceiling of 41.2 percent PassRate, indicating the benchmark is not yet saturated.

Core claim

WeaveBench supplies 114 publicly verifiable, long-horizon tasks that force agents to combine GUI observations and actions with CLI and code operations inside single trajectories on a deployed Ubuntu desktop, accompanied by a judge that inspects deliverables, traces, and logs to detect shortcuts, with the result that the highest observed PassRate across frontier pairings is 41.2 percent and outcome-only grading substantially overestimates capability.

What carries the argument

The trajectory-aware judge that examines deliverables, files, screenshots, logs, and action traces while detecting fabricated visual evidence or hard-coded metrics.

Load-bearing premise

The 114 tasks accurately capture the distribution of long-horizon hybrid-interface work that matters in practice.

What would settle it

An agent achieving greater than 70 percent PassRate on the full set of 114 tasks while the trajectory-aware judge still flags no shortcuts would falsify the claim of a persistent critical gap.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agents must acquire integrated orchestration skills across visual, command-line, and code interfaces rather than treating them as separate capabilities.
  • Outcome-only evaluation methods produce inflated performance estimates and should be replaced by trajectory inspection for hybrid tasks.
  • Progress on WeaveBench directly measures readiness for real-world computer-use scenarios that span multiple interfaces over extended sequences.
  • The benchmark supplies a concrete testbed for comparing model-runtime pairings on verifiable, cross-interface deliverables.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If models close the gap on WeaveBench, they may become usable for end-to-end productivity workflows that currently require human switching between applications.
  • Separate training regimes focused on GUI or CLI alone are unlikely to suffice; joint trajectory-level training across interfaces will probably be required.
  • The public verifiability of task artifacts allows future work to add new domains without changing the core evaluation protocol.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper introduces WeaveBench, a benchmark of 114 long-horizon tasks across 8 real-world domains that require agents to orchestrate GUI, CLI, and code operations within single trajectories on a real Ubuntu desktop. Tasks are asserted to be grounded in real user requests and publicly verifiable artifacts. Frontier model-runtime pairings are evaluated, with a maximum PassRate of 41.2%; a companion trajectory-aware judge that inspects deliverables, traces, and shortcuts is shown to reveal substantial overestimation by outcome-only grading.

Significance. If the task set is shown to be representative and the judge validated, the work would usefully document a performance ceiling on hybrid-interface orchestration and supply a reproducible testbed that existing separable benchmarks do not provide.

major comments (2)
  1. [§3] §3 (Task Construction): the manuscript supplies no sampling protocol, curation criteria, inter-rater reliability statistics, or explicit verification that the 114 tasks cannot be solved by single-interface shortcuts. These details are load-bearing for the claim that the benchmark accurately reflects the distribution of long-horizon hybrid work and that the 41.2% ceiling demonstrates an unsaturated gap.
  2. [§4] §4 (Trajectory-aware Judge): no validation of the judge is reported (e.g., agreement with human raters on a held-out subset, or inter-rater reliability). This directly affects the secondary claim that outcome-only grading substantially overestimates performance.
minor comments (1)
  1. The abstract states that tasks are 'grounded in real user requests and publicly verifiable artifacts'; the main text should include a reproducibility statement specifying artifact locations and licensing.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on WeaveBench. We address the two major comments point by point below, indicating where revisions will be made to strengthen the manuscript.

read point-by-point responses

  1. Referee: [§3] §3 (Task Construction): the manuscript supplies no sampling protocol, curation criteria, inter-rater reliability statistics, or explicit verification that the 114 tasks cannot be solved by single-interface shortcuts. These details are load-bearing for the claim that the benchmark accurately reflects the distribution of long-horizon hybrid work and that the 41.2% ceiling demonstrates an unsaturated gap.

    Authors: We agree that the manuscript would benefit from greater transparency on task construction. In the revision we will expand Section 3 with: (1) the sampling protocol, which drew from publicly documented real-user requests across the eight domains; (2) explicit curation criteria requiring each task to necessitate at least two distinct interfaces within a single trajectory; and (3) a description of the verification process used to confirm that no task admits a single-interface shortcut (performed by manual inspection of each task specification against interface capabilities). Inter-rater reliability statistics were not computed during initial curation, as the process was conducted by the authoring team; we will therefore report this as a limitation and, if feasible, add a small-scale agreement check on a subset of tasks. revision: yes

  2. Referee: [§4] §4 (Trajectory-aware Judge): no validation of the judge is reported (e.g., agreement with human raters on a held-out subset, or inter-rater reliability). This directly affects the secondary claim that outcome-only grading substantially overestimates performance.

    Authors: We acknowledge that the current manuscript reports no quantitative validation of the trajectory-aware judge against human raters. In the revised version we will add a validation subsection that measures agreement (Cohen’s kappa or equivalent) between the judge and human raters on a held-out sample of 20–30 trajectories. This will directly support the claim that outcome-only grading overestimates performance. Should resource constraints limit the scale of this study, we will present the validation results as preliminary and note the limitation explicitly. revision: yes

Circularity Check

0 steps flagged

No circularity; performance metrics are direct measurements on new benchmark tasks

full rationale

The paper introduces WeaveBench with 114 tasks and reports PassRate (max 41.2%) and trajectory-judge results as direct empirical outcomes of running frontier models on the tasks. No equations, fitted parameters, predictions derived from prior fits, or self-citation chains appear in the provided text. The central claims rest on measurement rather than any derivation that reduces to its own inputs by construction. Task grounding in real user requests is an empirical premise about data collection, not a definitional or fitted reduction. This is a standard non-circular benchmark paper.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

Abstract-only; the central claim rests on the unverified premise that the constructed tasks and judge faithfully represent real-world hybrid work. No free parameters, invented entities, or additional axioms are stated.

axioms (2)
  • domain assumption The 114 tasks are grounded in real user requests and produce publicly verifiable artifacts.
    Stated in the abstract as the grounding method for task creation.
  • domain assumption The trajectory-aware judge correctly detects shortcut behaviors such as fabricated visual evidence.
    Abstract presents the judge as reliable without further justification.

pith-pipeline@v0.9.1-grok · 5776 in / 1421 out tokens · 19418 ms · 2026-06-27T16:27:51.552891+00:00 · methodology

0 comments
read the original abstract

Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools. Existing benchmarks, however, often evaluate these interfaces as separable capabilities, leaving long-horizon cross-interface orchestration under-tested. Thus, we introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114 tasks across 8 real-world work domains, grounded in real user requests and publicly verifiable artifacts. Each task requires agents to combine GUI observations/actions with CLI/code operations within a single trajectory. We evaluate these tasks on a real Ubuntu desktop inside deployed CLI-agent runtimes, augmented with a minimal desktop-control plugin. We also propose a companion trajectory-aware judge that inspects deliverables, files, screenshots, logs, and action traces, while detecting shortcut behaviors such as fabricated visual evidence or hard-coded metrics. Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated. The trajectory-aware judge further reveals that outcome-only grading substantially overestimates agent performance. Overall, WeaveBench exposes a critical gap in CUA evaluation and provides an effective testbed to measure whether agents can orchestrate GUI, CLI, and code operations across long-horizon real-world tasks.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 3 canonical work pages · 3 internal anchors

  1. [1]

    Introducing ChatGPT agent: bridging research and action.https://openai.com/ index/introducing-chatgpt-agent/, 2025

    OpenAI. Introducing ChatGPT agent: bridging research and action.https://openai.com/ index/introducing-chatgpt-agent/, 2025

    2025

  2. [2]

    Claude Code.https://github.com/anthropics/claude-code, 2026

    Claude Code Team. Claude Code.https://github.com/anthropics/claude-code, 2026

    2026

  3. [3]

    Codex for (almost) everything

    OpenAI. Codex for (almost) everything. https://openai.com/index/ codex-for-almost-everything/, April 2026

    2026

  4. [4]

    Dispatch and computer use in Claude Cowork and Claude Code

    Anthropic. Dispatch and computer use in Claude Cowork and Claude Code. https:// claude.com/blog/dispatch-and-computer-use, 2026

    2026

  5. [5]

    Peekaboo: Mac automation that sees the screen and does the clicks

    OpenClaw Contributors. Peekaboo: Mac automation that sees the screen and does the clicks. https://github.com/openclaw/Peekaboo, 2026

    2026

  6. [6]

    OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. InAdvances in Neural Information P...

    2024

  7. [7]

    Windows Agent Arena: Evaluating multi-modal OS agents at scale, 2024

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows Agent Arena: Evaluating multi-modal OS agents at scale, 2024

    2024

  8. [8]

    AndroidWorld: A dynamic benchmarking environment for autonomous agents, 2025

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. AndroidWorld: A dynamic benchmarking environment for autonomous agents, 2025

    2025

  9. [9]

    VisualWebArena: Evaluating multimodal agents on realistic visual web tasks, 2024

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks, 2024

    2024

  10. [10]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. InInternational Conference on Learning Representations, 2024

    2024

  11. [11]

    Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

    Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprint arXiv:2601.11868, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  12. [12]

    Barr, Mark Harman, Federica Sarro, and He Ye

    Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O’Hearn, Earl T. Barr, Mark Harman, Federica Sarro, and He Ye. TerminalWorld: Benchmarking agents on real-world terminal tasks, 2026. 11

    2026

  13. [13]

    Terminal-World: Scaling terminal-agent environments via agent skills, 2026

    Zihao Cheng et al. Terminal-World: Scaling terminal-agent environments via agent skills, 2026

    2026

  14. [14]

    MCPWorld: A unified benchmarking testbed for API, GUI, and hybrid computer use agents, 2025

    Yunhe Yan, Shihe Wang, Jiajun Du, Yexuan Yang, Yuxuan Shan, Qichen Qiu, Xianqing Jia, Xinge Wang, Xin Yuan, Xu Han, Mao Qin, Yinxiao Chen, Chen Peng, Shangguang Wang, and Mengwei Xu. MCPWorld: A unified benchmarking testbed for API, GUI, and hybrid computer use agents, 2025

    2025

  15. [15]

    OSWorld-MCP: Benchmarking MCP tool invocation in computer-use agents, 2025

    Hongrui Jia, Jitong Liao, Xi Zhang, Haiyang Xu, Tianbao Xie, Chaoya Jiang, Ming Yan, and Si Liu. OSWorld-MCP: Benchmarking MCP tool invocation in computer-use agents, 2025

    2025

  16. [16]

    Programming with pixels: Can computer-use agents do software engineering?, 2025

    Pranjal Aggarwal and Sean Welleck. Programming with pixels: Can computer-use agents do software engineering?, 2025

    2025

  17. [17]

    ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows

    Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, et al. ScienceBoard: Evaluating multi- modal autonomous agents in realistic scientific workflows.arXiv preprint arXiv:2505.19897,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    CocoaBench: Evaluating Unified Digital Agents in the Wild

    CocoaBench Team, Shibo Hao, Zhining Zhang, Zhiqi Liang, Tianyang Liu, Yuheng Zha, Qiyue Gao, Jixuan Chen, Zilong Wang, Zhoujun Cheng, et al. CocoaBench: Evaluating unified digital agents in the wild.arXiv preprint arXiv:2604.11201, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  19. [19]

    SaaS-Bench: Can computer-use agents leverage real-world SaaS to solve professional workflows?, 2026

    Kean Shi, Zihang Li, Tianyi Ma, Zengji Tu, Jialong Wu, Xinbo Xu, Qingyao Yang, Ruoyu Wu, Weichu Xie, Ming Wu, Jason Zeng, Michael Heinrich, Elvis Zhang, Liang Chen, Kuan Li, and Baobao Chang. SaaS-Bench: Can computer-use agents leverage real-world SaaS to solve professional workflows?, 2026

    2026

  20. [20]

    Wildclawbench: A benchmark for real-world, long-horizon agent evaluation, 2026

    Shuangrui Ding, Xuanlang Dai, Long Xing, Shengyuan Ding, Ziyu Liu, Yang JingYi, Penghui Yang, Zhixiong Zhang, Xilin Wei, Xinyu Fang, Yubo Ma, Haodong Duan, Jing Shao, Jiaqi Wang, Dahua Lin, Kai Chen, and Yuhang Zang. Wildclawbench: A benchmark for real-world, long-horizon agent evaluation, 2026

    2026

  21. [21]

    Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Wendong Xu, Yunzhuo Hao, Songcheng Cai, Xiaochen Wang, Huaisong Zhang, Xian Wu, Yi Lu, Minyi Lei, Kai Zou, Huifeng Yin, Ping Nie, Liang Chen, Dongfu Jiang, Wenhu Chen, and Kelsey R. Allen. Clawbench: Can ai agents complete everyday online tasks?, 2026

    2026

  22. [22]

    Claw-Eval: Toward trustworthy evaluation of autonomous agents, 2026

    Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, Qi Liu, Zhifang Sui, and Tong Yang. Claw-Eval: Toward trustworthy evaluation of autonomous agents, 2026

    2026

  23. [23]

    ClawEnvKit: Toward autonomous generation of claw-like agent environments, 2026

    Ming Li et al. ClawEnvKit: Toward autonomous generation of claw-like agent environments, 2026

    2026

  24. [24]

    ClawMark: A living-world benchmark for multi-turn, multi-day, multimodal coworker agents, 2026

    Fanqing Meng, Lingxiao Du, Zijian Wu, Guanzheng Chen, Xiangyan Liu, Jiaqi Liao, Chonghe Jiang, Zhenglin Wan, Jiawei Gu, Pengfei Zhou, et al. ClawMark: A living-world benchmark for multi-turn, multi-day, multimodal coworker agents, 2026

    2026

  25. [25]

    PinchBench: An OpenClaw coding-agent benchmark.https://pinchbench

    Kilo Code Team. PinchBench: An OpenClaw coding-agent benchmark.https://pinchbench. com, GitHub:https://github.com/pinchbench/skill, 2026. Updated April 2026

    2026

  26. [26]

    OpenClaw.https://github.com/openclaw/openclaw, 2026

    OpenClaw Team. OpenClaw.https://github.com/openclaw/openclaw, 2026

    2026

  27. [27]

    Hermes: An open-source agent framework by nous research.https://github

    Nous Research. Hermes: An open-source agent framework by nous research.https://github. com/nousresearch/hermes-agent, 2025. 12

    2025

  28. [28]

    ScreenSpot-Pro: GUI grounding for professional high-resolution computer use, 2025

    Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. ScreenSpot-Pro: GUI grounding for professional high-resolution computer use, 2025

    2025

  29. [29]

    SeeClick: Harnessing GUI grounding for advanced visual GUI agents, 2024

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents, 2024

    2024

  30. [30]

    OS-Atlas: A foundation action model for generalist GUI agents, 2024

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. OS-Atlas: A foundation action model for generalist GUI agents, 2024

    2024

  31. [31]

    Ferret-UI: Grounded mobile UI understanding with multimodal LLMs, 2024

    Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-UI: Grounded mobile UI understanding with multimodal LLMs, 2024

    2024

  32. [32]

    ShowUI: One vision-language-action model for GUI visual agent, 2024

    Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Weix- ian Lei, Lijuan Wang, and Mike Zheng Shou. ShowUI: One vision-language-action model for GUI visual agent, 2024

    2024

  33. [33]

    Aguvis: Unified pure vision agents for autonomous GUI interaction, 2024

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous GUI interaction, 2024

    2024

  34. [34]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InInternational Conference on Learning Representations (ICLR), 2024

    2024

  35. [35]

    Introducing SWE-bench Verified

    OpenAI. Introducing SWE-bench Verified. https://openai.com/index/ introducing-swe-bench-verified/, 2024

    2024

  36. [36]

    CoAct-1: Computer-using multi-agent system with coding actions, 2025

    Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, Ran Xu, and Caiming Xiong. CoAct-1: Computer-using multi-agent system with coding actions, 2025

    2025

  37. [37]

    UFO2: The desktop AgentOS, 2025

    Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Liqun Wang, Fan Yang, Pu Zhao, Chao Du, Lu Li, Yan Kang, Zhao Jiang, Suzhen Zheng, Rujia Wang, Jiaxu Qian, Minghua Ma, Jian-Guang Lou, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. UFO2: The desktop AgentOS, 2025

    2025

  38. [38]

    Concrete problems in AI safety, 2016

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety, 2016

    2016

  39. [39]

    Specification gaming: the flip side of AI ingenuity.DeepMind Blog, 2020

    Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of AI ingenuity.DeepMind Blog, 2020. URLhttps://deepmind.google/discover/blog/ specification-gaming-the-flip-side-of-ai-ingenuity/

    2020

  40. [40]

    Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber

    Mingchen Zhuge, Changsheng Zhao, Dylan R. Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. Agent-as-a-Judge: Evaluate agents with agents, 2024

    2024

  41. [41]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. InAdvances in Neural Information Processing Systems, 2023. 13

    2023

  42. [42]

    G-Eval: NLG evaluation using GPT-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  43. [43]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  44. [44]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. InAdvances in Neural Information Processing Systems, 2023

    2023

  45. [45]

    GPT-5.5 and the GPT-5.x model family.https://openai.com/index/gpt-5-5, 2026

    OpenAI. GPT-5.5 and the GPT-5.x model family.https://openai.com/index/gpt-5-5, 2026

    2026

  46. [46]

    Claude Opus 4.7: Model card and capabilities.https://www.anthropic.com/ news/claude-opus-4-7, 2026

    Anthropic. Claude Opus 4.7: Model card and capabilities.https://www.anthropic.com/ news/claude-opus-4-7, 2026

    2026

  47. [47]

    judging failed

    Google DeepMind. Gemini 3.1 Pro: Technical report. https://deepmind.google/ technologies/gemini, 2026. 14 Appendix A Benchmark Construction ................................................. 16 A.1 Atomic-Capability Decomposition for P1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 A.2 P2/P3 Trajectory Distributions and By-Domain Met...

    2026