pith. sign in

arxiv: 2605.26329 · v1 · pith:BUNL7B6Fnew · submitted 2026-05-25 · 💻 cs.AI

JobBench: Aligning Agent Work With Human Will

Yuetai Li , Yichen Feng , Zhangchen Xu , Zixian Ma , Kaiyuan Zheng , Fengqing Jiang , Xinghua Sun , Rulin Shao
show 16 more authors
Zichen Chen Yue Huang Xinyang Han Brian Lee Kayla Xu Shenglai Zeng Hang Hua Xiangliang Zhang Basel Alomair Ranjay Krishna Luke Zettlemoyer Pang Wei Koh Bhaskar Ramasubramanian Luyao Niu Xiang Yue Radha Poovendran
This is my paper

Pith reviewed 2026-06-29 21:22 UTC · model grok-4.3

classification 💻 cs.AI
keywords AI agentsbenchmarksdelegation tasksoccupational workflowshuman prioritiesagent evaluationfact-anchored rubrics
0
0 comments X

The pith

JobBench shows top AI agents succeed on only 45.9% of tasks experts want delegated.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that existing benchmarks for AI agents emphasize economic replacement over the actual workflows domain experts want to hand off. It introduces JobBench as an alternative with 130 tasks drawn from 35 occupations, each embedded in a workspace of mixed reference files and scored by a chain of fact-anchored binary rubrics that average 35.6 criteria. When 36 models are tested, the strongest result reaches just 45.9 percent. The benchmark is meant to redirect development toward agents that enhance human work by performing the delegations people actually request.

Core claim

JobBench supplies 130 agentic tasks across 35 occupations, each packaged as a workspace of heterogeneous reference files and graded by a fact-anchored chain of rubrics averaging 35.6 binary criteria per task; under this evaluation the strongest model, Claude Opus 4.7 under Claude Code, reaches only 45.9 percent.

What carries the argument

The fact-anchored chain of rubrics that decomposes each task output into binary criteria tied directly to the supplied reference files.

If this is right

  • Development should prioritize agents capable of handling the specific cluttered workflows experts identify for delegation rather than tasks chosen solely by economic scale.
  • Scores on JobBench could serve as a signal for whether an agent aligns with human delegation priorities instead of GDP-driven replacement metrics.
  • The current 45.9 percent ceiling indicates that most professional tasks in the benchmark remain out of reach for existing models.
  • The labor-market target for agent research should shift from replacement stories to enhancement stories.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The benchmark design could be extended by letting experts dynamically update the task list over time instead of fixing it at 130 items.
  • Results might change if the rubrics were allowed to include non-binary criteria that capture partial or creative solutions.
  • The approach could be applied to test whether models fine-tuned specifically on these reference-file workspaces close the performance gap faster than general training.

Load-bearing premise

The 130 chosen tasks and their rubrics correctly represent the high-priority delegation workflows that domain experts would select.

What would settle it

A new survey in which domain experts rate most of the 130 tasks as low priority for delegation, or a model that scores above 80 percent on the full set, would undermine the claim that the benchmark captures what humans want delegated.

Figures

Figures reproduced from arXiv: 2605.26329 by Basel Alomair, Bhaskar Ramasubramanian, Brian Lee, Fengqing Jiang, Hang Hua, Kaiyuan Zheng, Kayla Xu, Luke Zettlemoyer, Luyao Niu, Pang Wei Koh, Radha Poovendran, Ranjay Krishna, Rulin Shao, Shenglai Zeng, Xiangliang Zhang, Xiang Yue, Xinghua Sun, Xinyang Han, Yichen Feng, Yue Huang, Yuetai Li, Zhangchen Xu, Zichen Chen, Zixian Ma.

Figure 1
Figure 1. Figure 1: JobBench overview. Tasks are constructed on top of Workbank [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: JobBench task distribution. Top: the 10 SOC groups covered, with each pie showing the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: JobBench expert onboarding: (1) the domain-expert pool from Prolific, (2) onboard experts [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison between GDPVal and JobBench. (a) Model scores on GDPVal, JobBench [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Cost analysis for JobBench evaluation. (a) Full-run inference cost (USD) on the main set, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: GPT-5.4 under Codex on JobBench-Main as reasoning effort scales from low to xhigh. Performance vs. Cost [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Occupation analysis for research and startup attention over JobBench. Each occupation [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Leaderboard on the JobBench main set (Claude Code for Anthropic models, Codex for [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Task schematic for Reporters. The agent receives an editor’s assignment and a heterogeneous evidence pack of water-quality CSVs, state surveillance reports, regulatory documents, and interview notes, and must deliver a pitch memo, a workbook of cross-source analyses, and a 15-entry source log. Instruction You are an investigative reporter at a regional Connecticut news outlet. Your editor, Sarah Chen, has … view at source ↗
Figure 10
Figure 10. Figure 10: Task schematic for Data Entry Keyers. The agent receives Q3 sales orders across CRM exports, scanned handwritten forms, and ERP records, and must reconcile them under a specified source hierarchy and deliver a discrepancy log, a clean reconciled master, and a findings report. Instruction As Senior Data Entry Keyer at Global Trade Solutions Inc., reconcile Q3 2025 sales order data across the CRM export (So… view at source ↗
Figure 11
Figure 11. Figure 11: Task schematic for Lawyers. The agent receives a town settlement letter, a six-property client database, and a packet of ordinances and case law, and must deliver a constitutional and settlement-risk memo, a jurisdiction-by-jurisdiction regulatory table, and a counter-proposal. Instruction You are a real estate and land use attorney at a mid-sized law firm in North Carolina. Your clients -- Coastal Proper… view at source ↗
Figure 12
Figure 12. Figure 12: Task schematic for Web Administrators. The agent receives access logs, threat indicators, a closed change request, live nginx and iptables configuration, and a network topology, and must reconstruct the kill chain, audit the missed controls, and deliver hardened configurations and a NIST-style incident report. Instruction On November 10, 2025, ShopVault’s monitoring flagged anomalous traffic patterns acro… view at source ↗
read the original abstract

Current benchmarks for occupational AI agents are scoped primarily by economic values, telling a replacement story. We introduce JobBench, which evaluates AI agents on the workflows that experts identify as high-priority for delegation, empowering humans based on their needs instead of replacing them with GDP value. JobBench covers 130 agentic tasks across 35 occupations. Each task is packaged as a workspace of heterogeneous reference files, requiring the agent to reason through the cluttered information streams of real professional work. Outputs are graded by a fact-anchored chain of rubrics, averaging 35.6 binary criteria per task. We evaluate 36 models; the strongest, Claude Opus~4.7 under Claude Code, reaches only 45.9 %. We hope JobBench shifts the community's target labour-market effect from replacement to enhancement: building agents that do what humans actually want delegated, not only what is most economically valuable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper introduces JobBench, a benchmark consisting of 130 agentic tasks across 35 occupations, where tasks are selected via expert nomination as high-priority for delegation to AI agents. Each task includes a workspace of heterogeneous reference files and is evaluated using fact-anchored rubrics averaging 35.6 binary criteria per task. The paper reports results from evaluating 36 models, with the strongest performer (Claude Opus 4.7 under Claude Code) achieving 45.9%, and positions the benchmark as shifting focus from GDP-driven replacement to human-aligned enhancement of workflows.

Significance. If the task selection and rubric construction hold up under scrutiny, JobBench could meaningfully redirect AI agent evaluation toward workflows that domain experts actually want delegated, providing a counterpoint to purely economic benchmarks. The reported performance ceiling of 45.9% supplies a concrete, falsifiable signal about current model limitations on cluttered, multi-file professional tasks.

major comments (2)
  1. [Benchmark construction / evaluation setup] The description of the expert identification method, task nomination process, and any safeguards against selection bias is absent from the provided abstract and evaluation summary. This directly undermines assessment of the central claim that the 130 tasks represent 'high-priority' delegation targets identified by experts (see reader's weakest assumption and soundness rating of 3.0).
  2. [Rubric design] No information is given on rubric validation, inter-rater reliability for the fact-anchored binary criteria, or how the average of 35.6 criteria per task was derived and checked for completeness. This is load-bearing for interpreting the 45.9% result as a reliable performance measure.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. The comments correctly identify gaps in the description of benchmark construction and rubric design. We address each point below and will revise the manuscript to provide the requested details.

read point-by-point responses

  1. Referee: [Benchmark construction / evaluation setup] The description of the expert identification method, task nomination process, and any safeguards against selection bias is absent from the provided abstract and evaluation summary. This directly undermines assessment of the central claim that the 130 tasks represent 'high-priority' delegation targets identified by experts (see reader's weakest assumption and soundness rating of 3.0).

    Authors: The referee is correct that the manuscript does not provide a detailed account of the expert identification method, task nomination process, or safeguards against selection bias. While the abstract and evaluation summary state that tasks were selected via expert nomination, no further methodological information is included. We will add a dedicated subsection in the revised manuscript describing the expert recruitment, nomination criteria, number of experts per occupation, and any bias-mitigation steps such as independent verification or diversity requirements. revision: yes

  2. Referee: [Rubric design] No information is given on rubric validation, inter-rater reliability for the fact-anchored binary criteria, or how the average of 35.6 criteria per task was derived and checked for completeness. This is load-bearing for interpreting the 45.9% result as a reliable performance measure.

    Authors: We agree that the manuscript lacks information on rubric validation, inter-rater reliability, and the derivation of the 35.6 criteria average. The current text only describes the fact-anchored rubrics without validation details. In the revision we will expand the evaluation section to explain rubric construction, how completeness was checked, the process yielding the average criteria count, and any steps taken to ensure consistency. We will also note whether formal inter-rater reliability metrics were computed and, if not, describe the alternative quality controls used. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper presents an empirical benchmark (JobBench) consisting of 130 expert-nominated tasks graded by fact-anchored rubrics. It reports model performance numbers (e.g., 45.9% ceiling) directly from evaluation runs. No derivations, equations, fitted parameters, or predictions appear that could reduce to the inputs by construction. The construction method (expert identification + binary criteria) is stated explicitly as the benchmark definition rather than derived from prior results. No self-citation chains or uniqueness theorems are invoked as load-bearing premises. The central claim is therefore self-contained as a measurement exercise and receives the default non-circularity finding.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is an empirical benchmark introduction paper. No free parameters, axioms, or invented entities are invoked in the central claim.

pith-pipeline@v0.9.1-grok · 5769 in / 1032 out tokens · 35641 ms · 2026-06-29T21:22:25.039476+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Agent Skills Should Go Beyond Text: The Case for Visual Skills

    cs.CV 2026-05 unverdicted novelty 5.0

    The paper proposes that reusable agent skills should incorporate visual elements alongside text, introduces three forms of visual skills and an automatic conversion system, and reports better performance on GUI and vi...

Reference graph

Works this paper leans on

87 extracted references · 19 canonical work pages · cited by 1 Pith paper · 11 internal anchors

  1. [1]

    Prbench: Large-scale expert rubrics for evaluating high-stakes professional reasoning

    Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph, Mohammad Mahmoudi Meymand, et al. PRBench: Large-scale expert rubrics for evaluating high-stakes professional reasoning. arXiv preprint arXiv:2511.11562, 2025. URLhttps://arxiv.org/abs/2511.11562

    work page arXiv 2025

  2. [2]

    System card: Claude Opus 4 & Claude Sonnet 4

    Anthropic. System card: Claude Opus 4 & Claude Sonnet 4. https://www.anthropic. com/claude-4-system-card , May 2025. Covers Claude Opus 4, Sonnet 4, and the broader Claude 4 model family

    2025

  3. [3]

    System card: Claude Opus 4.5

    Anthropic. System card: Claude Opus 4.5. https://www.anthropic.com/ claude-opus-4-5-system-card, November 2025

    2025

  4. [4]

    Claude Code: An agentic coding tool for the terminal

    Anthropic. Claude Code: An agentic coding tool for the terminal. https://docs. claude.com/en/docs/claude-code/overview, 2025. Source: https://github.com/ anthropics/claude-code

    2025

  5. [5]

    System card: Claude Haiku 4.5

    Anthropic. System card: Claude Haiku 4.5. https://www.anthropic.com/ claude-haiku-4-5-system-card, October 2025

    2025

  6. [6]

    System card: Claude Sonnet 4.5

    Anthropic. System card: Claude Sonnet 4.5. https://www.anthropic.com/ claude-sonnet-4-5-system-card, September 2025

    2025

  7. [7]

    System card: Claude Opus 4.6

    Anthropic. System card: Claude Opus 4.6. https://www.anthropic.com/ claude-opus-4-6-system-card, February 2026

    2026

  8. [8]

    Introducing Claude Opus 4.7

    Anthropic. Introducing Claude Opus 4.7. https://www.anthropic.com/news/ claude-opus-4-7 , April 2026. Release announcement; the corresponding system card is listed athttps://www.anthropic.com/system-cards

    2026

  9. [9]

    System card: Claude Sonnet 4.6

    Anthropic. System card: Claude Sonnet 4.6. https://www.anthropic.com/ claude-sonnet-4-6-system-card, February 2026

    2026

  10. [10]

    Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond. Generative AI at work. NBER Working Paper No. 31161, 2023. URLhttps://arxiv.org/abs/2304.11771

    work page arXiv 2023

  11. [11]

    Displacement or complemen- tarity? the labor market impact of generative AI

    Wilbur Xinyuan Chen, Suraj Srinivasan, and Saleh Zakerinia. Displacement or complemen- tarity? the labor market impact of generative AI. Harvard Business School Working Pa- per 25-039, 2024. URL https://www.hbs.edu/ris/Publication%20Files/25-039_ 05fbec84-1f23-459b-8410-e3cd7ab6c88a.pdf

    2024

  12. [12]

    WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. WorkArena: How capable are web agents at solving common knowledge work tasks?, 2024. URLhttps://arxiv.org/abs/2403.07718

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    arXiv preprint arXiv:2303.10130 , year=

    Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. GPTs are GPTs: An early look at the labor market impact potential of large language models, 2023. URL https: //arxiv.org/abs/2303.10130

    work page arXiv 2023

  14. [14]

    Gemini 3 model documentation

    Google DeepMind. Gemini 3 model documentation. https://storage.googleapis. com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf , December 2025. Gemini 3 Pro model card; Gemini 3 Flash documentation is available at https://deepmind. google/models/gemini/flash/

    2025

  15. [15]

    Measuring Massive Multitask Language Understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Con- ference on Learning Representations, 2021. URLhttps://arxiv.org/abs/2009.03300

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  16. [16]

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InInternational Conference on Learning Representations, 2024. URL https://arxiv.org/ abs/2310.06770. 10

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  17. [17]

    AgentBench: Evaluating LLMs as Agents

    Xiao Liu et al. AgentBench: Evaluating LLMs as agents. InInternational Conference on Learning Representations, 2024. URLhttps://arxiv.org/abs/2308.03688

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    Remote labor index: Measuring AI automation of remote work.arXiv preprint arXiv:2510.26787,

    Mantas Mazeika et al. Remote labor index: Measuring ai automation of remote work, 2025. URLhttps://arxiv.org/abs/2510.26787

    work page arXiv 2025

  19. [19]

    GAIA: a benchmark for General AI Assistants

    Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A benchmark for general AI assistants. InInternational Conference on Learning Representations, 2024. URLhttps://arxiv.org/abs/2311.12983

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  20. [20]

    MiniMax M2.5: Built for real-world productivity

    MiniMax. MiniMax M2.5: Built for real-world productivity. https://www.minimax.io/ news/minimax-m25, February 2026

    2026

  21. [21]

    Kimi K2.5.https://github.com/MoonshotAI/Kimi-K2.5, January 2026

    Moonshot AI. Kimi K2.5.https://github.com/MoonshotAI/Kimi-K2.5, January 2026

    2026

  22. [22]

    Codex CLI: A lightweight coding agent for the terminal

    OpenAI. Codex CLI: A lightweight coding agent for the terminal. https://developers. openai.com/codex/cli, 2025. Source:https://github.com/openai/codex

    2025

  23. [23]

    GPT-5 system card

    OpenAI. GPT-5 system card. https://openai.com/index/gpt-5-system-card/ , August

  24. [24]

    Also available as arXiv:2601.03267

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [25]

    GPT-5.1 instant and GPT-5.1 thinking system card addendum

    OpenAI. GPT-5.1 instant and GPT-5.1 thinking system card addendum. https://openai. com/index/gpt-5-system-card-addendum-gpt-5-1/, November 2025

    2025

  26. [26]

    Update to GPT-5 system card: GPT-5.2

    OpenAI. Update to GPT-5 system card: GPT-5.2. https://openai.com/index/ gpt-5-system-card-update-gpt-5-2/, December 2025

    2025

  27. [27]

    GPT-5 Codex system cards

    OpenAI. GPT-5 Codex system cards. https://cdn.openai.com/pdf/ 23eca107-a9b1-4d2c-b156-7deb4fbc697c/GPT-5-3-Codex-System-Card-02. pdf, February 2026. Primary citation for GPT-5.3-Codex; GPT-5-Codex and GPT-5.2-Codex are documented at https://cdn.openai.com/pdf/ 97cc5669-7a25-4e63-b15f-5fd5bdc4d149/gpt-5-codex-system-card.pdf and https://openai.com/index/gp...

    2026

  28. [28]

    GPT-5.4 thinking system card

    OpenAI. GPT-5.4 thinking system card. https://openai.com/index/ gpt-5-4-thinking-system-card/, March 2026

    2026

  29. [29]

    GPT-5.5 system card

    OpenAI. GPT-5.5 system card. https://openai.com/index/gpt-5-5-system-card/ , April 2026

    2026

  30. [30]

    OpenClaw: A self-hosted personal AI assistant platform

    OpenClaw Project. OpenClaw: A self-hosted personal AI assistant platform. https://github. com/openclaw/openclaw, 2026. Version 2026.3.8 used in the experiments

    2026

  31. [31]

    Prolific.ac—a subject pool for online experiments.Journal of Behavioral and Experimental Finance, 17:22–27, 2018

    Stefan Palan and Christian Schitter. Prolific.ac—a subject pool for online experiments.Journal of Behavioral and Experimental Finance, 17:22–27, 2018. doi: 10.1016/j.jbef.2017.12.004

    work page doi:10.1016/j.jbef.2017.12.004 2018

  32. [32]

    GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks

    Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, et al. GDPVal: Evaluating AI model performance on real-world economically valuable tasks.arXiv preprint arXiv:2510.04374, 2025. URLhttps://arxiv.org/abs/2510.04374

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  33. [33]

    Qwen3.5-Plus model release

    Qwen Team. Qwen3.5-Plus model release. https://qwen.ai/blog?id=qwen3.5, February 2026

    2026

  34. [34]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google- proof q&a benchmark. InConference on Language Modeling (COLM), 2024. URL https: //arxiv.org/abs/2311.12022

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    arXiv preprint arXiv:2506.11135 (2025)

    Yijia Shao, Humishka Zope, Yucheng Jiang, Jiaxin Pei, David Nguyen, Erik Brynjolfsson, and Diyi Yang. Future of work with AI agents: Auditing automation and augmentation potential across the U.S. workforce.arXiv preprint arXiv:2506.06576, 2025. doi: 10.48550/arXiv.2506. 06576. URLhttps://arxiv.org/abs/2506.06576. 11

    work page doi:10.48550/arxiv.2506 2025

  36. [36]

    OpenCode: The open source AI coding agent

    SST. OpenCode: The open source AI coding agent. https://opencode.ai, 2025. Source: https://github.com/sst/opencode

    2025

  37. [37]

    Upwork: The world’s work marketplace

    Upwork Inc. Upwork: The world’s work marketplace. https://www.upwork.com/, 2026. Online freelance labor platform; accessed 2026

    2026

  38. [38]

    Bureau of Labor Statistics

    U.S. Bureau of Labor Statistics. Occupational employment and wage statistics, may 2024. https://www.bls.gov/oes/tables.htm, April 2025. National employment and wage estimates by occupation (SOC), released April 2025

    2024

  39. [39]

    MMLU-Pro: A more robust and challenging multi-task language understanding benchmark.Advances in Neural Information Processing Systems, 37:95266–95290, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark.Advances in Neural Information Processing Systems, 37:95266–95290, 2024

    2024

  40. [40]

    Grok 4.2 Fast, March 2026

    xAI. Grok 4.2 Fast, March 2026

    2026

  41. [41]

    OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

    Tianbao Xie et al. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2024. URLhttps://arxiv.org/abs/2404.07972

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  42. [42]

    arXiv preprint arXiv:2603.07980 , year=

    Qianyu Yang, Yang Liu, Jiaqi Li, Jun Bai, Hao Chen, Kaiyuan Chen, Tiliang Duan, Jiayun Dong, Xiaobo Hu, Zixia Jia, Yang Liu, Tao Peng, Yixin Ren, Ran Tian, Zaiyuan Wang, Yanglihong Xiao, Gang Yao, Lingyue Yin, Ge Zhang, Chun Zhang, Jianpeng Jiao, Zilong Zheng, and Yuan Gong. $OneMillion-Bench: How far are language agents from human experts?arXiv preprint ...

    work page arXiv 2026

  43. [43]

    arXiv preprint arXiv:2207.01206

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents. InAdvances in Neural Information Processing Systems, 2022. URLhttps://arxiv.org/abs/2207.01206

    work page arXiv 2022

  44. [44]

    $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL https://arxiv.org/abs/ 2406.12045

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  45. [45]

    MMMU: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert AGI

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert AGI. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9556–9567, 2024

    2024

  46. [46]

    WebArena: A Realistic Web Environment for Building Autonomous Agents

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. InInternational Conference on Learning Representations, 2024. URLhttps://arxiv.org/abs/2307.13854. 12 A Related Work. . . . . . ....

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  47. [47]

    Water Lead Trends

    extends the setting to full desktop environments. GAIA [19], AgentBench [17], WebShop [42], and τ-bench [43] cover assistant tool use, multi-environment generally, grounded shopping, and tool-agent-user interaction. These benchmarks score whether an agent canreacha target state in an environment Workplace and occupational benchmarks.Recent benchmarks targ...

    2024

  48. [48]

    8 pts.Does the response correctly map which of the five CSV water system cities appear on the CT surveillance report’s 2024 high-risk communities list, and which do not? • The response must correctly state that all five water system cities from the CSV (Hartford, New Haven, Bridgeport, Waterbury, Meriden) appear on the CT 2024 Surveillance Report’s top 10...

    2024

  49. [49]

    8 pts.Does the data analysis workbook’s ’Water Lead Trends’ sheet correctly compute the percentage change in 90th-percentile lead levels from 2020 to 2024 for each of the five water systems? • The workbook must compute percentage changes in 90th-percentile lead levels from 2020 to 2024 for all five water systems, with approximately correct values: Hartfor...

    2020

  50. [50]

    • The sheet must present correct figures: 2017 CT had 1,666 tested with 93 (5.6%) at >=5 g/dL vs

    10 pts.Does the ’CT vs National’ sheet correctly extract Connecticut’s data from the CDC dataset and compare it to national totals, noting that CT data is only available for 2017-2019? • The sheet must note that Connecticut data in the CDC dataset is only available for 2017, 2018, and 2019, with years 2020-2022 marked as not submitted. • The sheet must pr...

    2017

  51. [51]

    • The sheet must include the high-risk rankings for each city: New Haven (#1), Bridgeport (#2), Waterbury (#3), Hartford (#4), Meriden (#5)

    6 pts.Does the ’City Cross-Reference’ sheet correctly map each water system city to the town-level blood lead data from the CT surveillance report, including total confirmed tests, and counts at each BLL threshold? • The sheet must include correct town-level data for all five cities: Hartford (3,539 total, 150 at >=3.5, 92 at >=5, 29 at >=10, 12 at >=15, ...

  52. [52]

    • The log must contain at least 15 distinct entries covering key statistics, quotes, and claims used in the pitch memo and data workbook

    8 pts.Does the source verification log contain at least 15 entries and use the correct CSV format with columns Data_Point, Source_File, Page_or_Location, Verified (Yes/No/Partial), and Notes? 19 • The source log must be a CSV file with exactly five columns: Data_Point, Source_File, Page_or_Location, Verified (Yes/No/Partial), and Notes. • The log must con...

  53. [53]

    • The response must state the correct effective date (January 1, 2023) and the correct threshold change (from 5 g/dL to 3.5 g/dL)

    6 pts.Does the response reference the correct legislative basis for Connecticut’s lowered blood lead reference value – Public Act 22-49, effective January 1, 2023, lowering the threshold from 5 g/dL to 3.5 g/dL? • The response must correctly identify Connecticut Public Act 22-49 as the legislative basis for the lowered blood lead reference value. • The re...

    2023

  54. [54]

    • The pitch memo must identify Hartford as having both the most absolute lead service lines (2,500) and the highest ratio of lines to population served (0.63%)

    8 pts.Does the pitch memo discuss the lead service line counts and replacement burden across the five water systems, connecting this to the LCRI’s 10-year replacement mandate? • The pitch memo must include lead service line counts and population served for each system from the 2024 CSV data: Hartford (2,500 lines, 400,000 people, 0.63% ratio), New Haven (...

    2024

  55. [55]

    30% increase

    6 pts.Does the data analysis workbook flag 90th-percentile readings that exceed both the 15 ppb and 10 ppb thresholds for each monitoring period, correctly identifying all exceedance periods? • The workbook must correctly identify the 15 ppb exceedances: Waterbury Jan-Jun 2023 (15.3 ppb) and Jul-Dec 2023 (16.1 ppb). • The workbook must correctly identify ...

    2023

  56. [56]

    • The error type must be classified as ’Calculation’ with the source system identified as ’ERP’

    10 pts.Does the Error Log correctly document the calculation error for ORD-202507-0002 where ERP shows Total_Revenue of $6794.25 instead of the correct $6749.25? • The Error Log must document that for ORD-202507-0002, ERP shows Total_Revenue as $6794.25 while the correct calculation is 75 x $89.99 = $6749.25 (matching CRM and scanned form). • The error ty...

  57. [57]

    • The error type must be classified as ’Typo’ with the source system identified as ’ERP’

    6 pts.Does the Error Log document the customer name typo for ORD-202507-0004 where ERP has ’Pacific Rim Export’ instead of ’Pacific Rim Exports’? • The Error Log must document that for ORD-202507-0004, ERP has Customer_Name as ’Pacific Rim Export’ (missing ’s’) while CRM and scanned form have ’Pacific Rim Exports’. • The error type must be classified as ’...

  58. [58]

    • The error type must be classified as ’Transposition’ (digits 0 and 2 swapped) with the source system identified as ’ERP’

    6 pts.Does the Error Log identify the digit transposition error for ORD-202507-0005 where ERP shows 1020 units sold instead of the correct 1200 units? • The Error Log must document that for ORD-202507-0005, ERP shows Units_Sold as 1020 while CRM and scanned form show 1200. • The error type must be classified as ’Transposition’ (digits 0 and 2 swapped) wit...

  59. [59]

    • The Error Log must reference Customer_Master_List.csv to confirm CUST-006 (Mediterranean Trade Co) is registered with Country ’Italy’

    10 pts.Does the Error Log correctly flag the country naming inconsistency for ORD-202508-0006 where CRM shows ’Italia’ instead of ’Italy’? • The Error Log must document that for ORD-202508-0006, CRM shows Country as ’Italia’ while ERP and scanned form show ’Italy’. • The Error Log must reference Customer_Master_List.csv to confirm CUST-006 (Mediterranean ...

  60. [60]

    • The log should note that ERP Total_Revenue of $4400.00 matches 800 x $5.50, indicating the ERP Unit_Price field is internally inconsistent with its own revenue

    10 pts.Does the Error Log identify the Unit Price discrepancy for ORD-202508-0008 where ERP shows $5.05 instead of $5.50? • The Error Log must document that for ORD-202508-0008, ERP shows Unit_Price as $5.05 while CRM and scanned form show $5.50. • The log should note that ERP Total_Revenue of $4400.00 matches 800 x $5.50, indicating the ERP Unit_Price fi...

  61. [61]

    • Per source priority (scanned highest), the reconciled date must be 09/03/2025

    6 pts.Does the Error Log correctly identify the date error for ORD-202509-0009 where CRM shows 09/30/2025 but ERP and scanned form show 09/03/2025? • The Error Log must document that for ORD-202509-0009, CRM shows Order_Date as ’09/30/2025’ while ERP shows ’09/03/2025’ and scanned form confirms ’09/03/2025’. • Per source priority (scanned highest), the re...

    2025

  62. [62]

    • The Error Log must reference Customer_Master_List.csv to confirm CUST-010 (Canadian Wholesale Inc) is registered with Country ’Canada’

    6 pts.Does the Error Log document the country name typo for ORD-202509-0010 where ERP has ’Canda’ instead of ’Canada’? • The Error Log must document that for ORD-202509-0010, ERP has Country as ’Canda’ while CRM and scanned form show ’Canada’. • The Error Log must reference Customer_Master_List.csv to confirm CUST-010 (Canadian Wholesale Inc) is registere...

  63. [63]

    • The error type must be classified as ’Typo’ with the source system identified as ’ERP’

    6 pts.Does the Error Log identify the Item Type typo for ORD-202508-0018 where ERP has ’Houshold’ instead of ’Household’? • The Error Log must document that for ORD-202508-0018, ERP has Item_Type as ’Houshold’ (missing ’e’) while CRM shows ’Household’. • The error type must be classified as ’Typo’ with the source system identified as ’ERP’. • The reconcil...

  64. [64]

    • The Error Log must use Total_Revenue of $8100.00 to validate the CRM price (45 x $180.00 = $8100.00), not the ERP price (45 x $108.00 = $4860.00)

    6 pts.Does the Error Log identify the Unit Price transposition for ORD-202509-0023 where ERP shows $108.00 instead of $180.00? • The Error Log must document that for ORD-202509-0023, ERP shows Unit_Price as $108.00 while CRM shows $180.00. • The Error Log must use Total_Revenue of $8100.00 to validate the CRM price (45 x $180.00 = $8100.00), not the ERP p...

  65. [65]

    • Since no scanned form exists for this order, CRM takes priority; the source system causing the error must be identified as ’ERP’ with the reconciled value as ’Offline’

    6 pts.Does the Error Log document the Sales Channel discrepancy for ORD-202509-0026 where ERP shows ’Online’ but CRM shows ’Offline’? • The Error Log must document that for ORD-202509-0026, ERP has Sales_Channel as ’Online’ while CRM shows ’Offline’. • Since no scanned form exists for this order, CRM takes priority; the source system causing the error mus...

  66. [66]

    • The calculation must be based on 30 total records, with discrepancies identified in all 17 of the following orders: 0001-0010, 0012, 0015, 0018, 0020, 0023, 0026, 0029

    10 pts.Does the Audit Findings Report include a data quality score calculated as percentage of records with no discrepancies? • The Audit_Findings_Report.txt must include a data quality score calculated as (records with no discrepancies / total records) x 100%. • The calculation must be based on 30 total records, with discrepancies identified in all 17 of...

  67. [67]

    Unknown Supplier Co

    10 pts.Does the Compliance Checklist correctly assess CRT-005 (Total Revenue Calculation Accuracy) as ’Fail’? • The Compliance_Checklist.xlsx must mark CRT-005 ’Total Revenue calculations are accurate (Units x Unit Price)’ as ’Fail’. • The Compliance_Checklist.xlsx must justify the failure by referencing ORD-202507-0002 (ERP: $6794.25 vs correct $6749.25)...

    2025

  68. [68]

    6 pts.Does the analysis correctly identify and quantify the fine exposure for each of the three properties accruing $500/day fines, using the database data showing fines started accruing on November 29, 2025, and correctly project ongoing exposure through the January 31, 2026 fine-waiver deadline? • Identifies the three specific properties accruing fines ...

    2025

  69. [69]

    6 pts.Does the comparative regulatory analysis table include at minimum the seven required provision categories (owner-occupancy requirements, permit caps, night limits, spacing rules, amortization/grand- fathering, fine structures, and zone restrictions) across all three jurisdictions (Millbrook Ordinance 2025-14, Belvidere Township, and New Orleans per ...

    2025

  70. [70]

    $33,600/year gross from LTR)

    6 pts.Does the counter-proposal include a request to extend the transitional period beyond the proposed 3 years (ending December 1, 2028), with economic justification grounded in the financial data? • Counter-proposal argues for extending the transitional period beyond the proposed 3 years (ending December 1, 2028) • Justifies extension with reference to ...

    2028

  71. [71]

    with prejudice,

    6 pts.Does the counter-proposal include a request for a mutual tolling agreement on the statute of limitations during settlement negotiations? 27 • Counter-proposal includes a request for a mutual tolling agreement that pauses the running of applicable statutes of limitations on the clients’ potential constitutional claims (Takings, Due Process, dormant C...

    1983

  72. [72]

    6 pts.Does the analysis apply the Penn Central regulatory takings factors (economic impact, investment- backed expectations, and character of government action) to the clients’ specific circumstances to assess litigation strength? • Applies the economic impact factor: the ordinance eliminates clients’ primary income stream, with historical STR net income ...

    2016

  73. [73]

    virtually per se invalid

    6 pts.Does the analysis identify the dormant Commerce Clause vulnerability in Millbrook’s owner- occupancy requirement by drawing on the Hignell-Stark holding that New Orleans’ similar residen- cy/homestead requirement was facially discriminatory against interstate commerce? • Identifies that Ordinance 2025-14’s owner-occupancy requirement (primary reside...

    2025

  74. [74]

    Mutual Terms

    6 pts.Does the settlement evaluation correctly identify that the Town’s settlement requires dismissal of all claims ’with prejudice’ and analyze the strategic implications of this term? • Flags that under the "Mutual Terms" section, clients would be required to "dismiss any pending or contemplated legal claims against the Town with prejudice" • Explains t...

  75. [75]

    regulatory takings framework

    6 pts.Does the analysis correctly identify that Millbrook’s 180-day amortization period with no grandfathering clause is legally vulnerable when compared to amortization periods upheld in the cases cited by the Town Attorney? • Identifies that the Town Attorney cites A VR, Inc. v. City of St. Louis Park and City of Los Angeles v. Gage as support, but thes...

    2022

  76. [76]

    • In findings_summary.csv, IDOR enumeration of /api/users/1 through /api/users/10 at 08:16:00-09Z must be mapped to a discovery/enumeration phase

    8 pts.Does the findings_summary.csv reconstruct the full kill chain progression for 203.0.113.42 showing the correct sequence: initial probing (SQL injection at 08:13:01-05Z), credential access (brute force at 08:13:08-16Z), enumeration (IDOR user scraping at 08:16:00-09Z), exploitation (XSS at 08:19:00-08Z), data exfiltration (path traversal + config dow...

  77. [77]

    • Item 1f’s risk description must note that /api/v1/debug was accessed and returned 8921 bytes of debug information

    6 pts.Does the cr_compliance_audit.csv include risk assessments that correctly connect non-compliance to specific observed attack outcomes – particularly that the missing dotfile blocking (1c) enabled .env and .git/config exposure, the missing debug endpoint removal (1f) enabled information leakage of 8921 bytes, and the missing admin IP restriction (1e) ...

  78. [78]

    • Port 5432 (PostgreSQL) and port 6379 (Redis) must only accept connections from Application zone 10.100.1.0/24

    6 pts.Does the hardened iptables configuration (iptables_hardened.rules) use zone-based source IP restrictions matching network_topology.json – specifically restricting port 3000 to 10.100.0.0/24 (DMZ), port 3001 to 10.200.0.0/24 (Management), ports 5432 and 6379 to 10.100.1.0/24 (Application zone), and SSH to 10.200.0.0/24? • Port 3000 (Node.js) must onl...

  79. [79]

    • The assessment must identify that admin.shopvault-internal.com certificate expired on 2025-11-01 (9 days before the incident on Nov 10), violating CR item 3a

    6 pts.Does the TLS posture assessment correctly identify all four certificate issues: (1) shop- vault.example.com and api.shopvault.example.com using TLSv1.0 minimum instead of TLSv1.2, (2) admin.shopvault-internal.com certificate expired on 2025-11-01, (3) staging.shopvault.example.com using a self-signed certificate with 1024-bit key on SSLv3, and (4) t...

    2025

  80. [80]

    6 pts.Does the incident report’s executive summary explicitly explain how the unimplemented change request CR-2025-0347 directly enabled the observed attacks – specifically linking missing dotfile protection to .env/.git exposure, missing rate limiting to successful brute force, missing admin restrictions to admin panel accessibility, and missing firewall...

    2025

Showing first 80 references.