Pith. sign in

REVIEW 2 major objections 2 minor 3 references

Binary pass/fail tests for SWE agents treat 10.7 percent of passing trajectories as equivalent to sound solutions when many are lucky passes driven by regression cycles and blind retries.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 21:58 UTC pith:2XD7AD2D

load-bearing objection The paper quantifies messy but passing SWE-agent runs at 10.7% and offers a process framework, but the merged PTA references may weaken the separation they claim. the 2 major comments →

arxiv 2605.12925 v3 pith:2XD7AD2D submitted 2026-05-13 cs.SE cs.AI

AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation

Priyam Sahoo , Gaurav Mittal , Xiaomin Li , Shengjie Ma , Benjamin Steenhoek , Pingping Lin , Yu Hu This is my paper
classification cs.SE cs.AI
keywords SWE-agent evaluationLucky Passprocess-level assessmenttrajectory qualityPrefix Tree AcceptorAgentLenssoftware engineering agentspass rate limitations
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current evaluation of software engineering agents rests on a single binary signal: whether the final patch passes the tests. This paper shows the signal is incomplete because it cannot distinguish a methodical process from a chaotic one that happens to succeed. The authors build AgentLens to score entire trajectories against merged process references and find that 10.7 percent of the passing runs exhibit lucky-pass behaviors such as repeated regressions, missing verification steps, or disordered exploration and implementation. When models are ranked by the new quality score instead of raw pass rate, several shift position by as many as five places.

Core claim

Among 1,815 passing trajectories on 47 SWE-bench Verified tasks, 10.7 percent qualify as Lucky Passes because they contain regression cycles, blind retries, missing verification, or temporally disordered sequences of exploration, implementation, and verification. AgentLens constructs task-level Prefix Tree Acceptor references by merging multiple passing solutions and applies a context-sensitive intent labeler to tag each action as Exploration, Implementation, Verification, or Orchestration. The resulting quality scores partition trajectories into Lucky, Solid, and Ideal tiers and decompose the lucky cases into five recurring mechanisms; across eight model backends the lucky-pass rate ranges

What carries the argument

AgentLens framework that merges multiple passing solutions into task-level Prefix Tree Acceptor (PTA) references and labels actions by context-sensitive intent rather than tool identity alone.

Load-bearing premise

Merging several passing solutions into a single PTA reference produces a reliable standard that correctly flags low-quality trajectories.

What would settle it

A fresh collection of trajectories in which the PTA-derived quality scores show no correlation with independent human ratings of process soundness would falsify the claim that the references separate lucky from solid passes.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Quality scoring separates passing trajectories into Lucky, Solid, and Ideal tiers.
  • Lucky-pass rates differ substantially across model backends, from 0.5 percent to 23.2 percent.
  • Model rankings shift by as many as five positions when quality score replaces pass rate.
  • Lucky passes decompose into five recurring mechanisms that can be targeted for mitigation.
  • Process-level assessment is feasible on 47 tasks that supply enough passing trajectories to build PTA references.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If lucky passes prove common on new tasks, raw pass rate will systematically overstate agent reliability in deployment settings that require consistent reasoning.
  • The same PTA construction could be applied to non-SWE domains where multiple correct solution paths exist, turning outcome-only benchmarks into process benchmarks.
  • Training objectives that penalize the five identified mechanisms might reduce lucky-pass rates without lowering overall pass rate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper claims that outcome-only (pass/fail) evaluation of SWE agents is insufficient, as some passing trajectories exhibit inefficient 'Lucky Pass' behaviors (regression cycles, blind retries, missing verification, or temporally disordered exploration/implementation/verification). On 2,614 OpenHands trajectories across 60 SWE-bench Verified tasks, they retain 1,815 trajectories from 47 tasks with sufficient passes, construct task-level Prefix Tree Acceptor (PTA) references by merging passing solutions, annotate trajectories with a context-sensitive intent labeler (Exploration/Implementation/Verification/Orchestration), and report that 10.7% of passing trajectories are Lucky Passes. Quality scores derived from divergence from the PTA separate trajectories into tiers, alter model rankings across eight backends, and decompose Lucky Passes into five mechanisms. The work introduces the AgentLens framework and AgentLens-Bench dataset, with plans to release artifacts, SDK, and tooling.

Significance. If the PTA references and intent labeling prove robust, the result would demonstrate that binary pass rates mask substantial process-level differences in SWE-agent behavior, motivating a shift toward process-aware evaluation. The explicit plan to release the 1,815-trajectory dataset, PTA references, quality annotations, and analysis tooling is a concrete strength that supports reproducibility and follow-on work.

major comments (2)
  1. [PTA construction] PTA construction (abstract and § on reference building): PTA references are formed by merging all passing trajectories for each task without any described filtering or down-weighting of lucky-pass patterns (regression cycles, disordered E/I/V sequences). Because lucky passes are defined as a subset of the same passing trajectories, this procedure risks embedding the target behaviors inside the reference acceptor, so that divergence scoring no longer isolates the claimed mechanisms.
  2. [Intent labeler validation] Intent labeler validation (abstract and § on annotation): The context-sensitive intent labeler supplies the sole signal for detecting temporal disorder, yet the manuscript reports neither inter-annotator agreement statistics nor any external validation or error analysis of the labeler. Without these, it is impossible to determine whether observed divergences reflect the five listed mechanisms or labeler artifacts.
minor comments (2)
  1. [Abstract] The abstract states that 47 tasks yield the 1,815-trajectory subset but does not give the exact pass-rate threshold used to decide 'enough passing trajectories,' making the selection criterion non-reproducible from the given text.
  2. [Quality score] The quality-score definition and its mapping to Lucky/Solid/Ideal tiers are referenced but not given an explicit formula or pseudocode in the provided abstract; a short algorithmic description would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments highlighting important aspects of our evaluation framework. We respond to each major comment below and indicate the revisions we will incorporate.

read point-by-point responses

  1. Referee: [PTA construction] PTA construction (abstract and § on reference building): PTA references are formed by merging all passing trajectories for each task without any described filtering or down-weighting of lucky-pass patterns (regression cycles, disordered E/I/V sequences). Because lucky passes are defined as a subset of the same passing trajectories, this procedure risks embedding the target behaviors inside the reference acceptor, so that divergence scoring no longer isolates the claimed mechanisms.

    Authors: We acknowledge the validity of this concern. The current PTA construction merges all passing trajectories to capture shared process structure, and lucky passes (10.7%) are identified post hoc via divergence. Because lucky-pass behaviors are infrequent and often involve cycles or disordered sequences not shared across the majority of trajectories, they have limited impact on the merged acceptor. However, to strengthen the method we will revise the manuscript to (1) explicitly document the merging procedure (frequency-based path selection in the prefix tree), (2) report the fraction of lucky-pass trajectories included in each PTA, and (3) add a sensitivity analysis comparing PTAs built with and without the lucky-pass subset. If the analysis shows material change, we will adopt an iterative reference construction that excludes identified lucky passes. revision: yes

  2. Referee: [Intent labeler validation] Intent labeler validation (abstract and § on annotation): The context-sensitive intent labeler supplies the sole signal for detecting temporal disorder, yet the manuscript reports neither inter-annotator agreement statistics nor any external validation or error analysis of the labeler. Without these, it is impossible to determine whether observed divergences reflect the five listed mechanisms or labeler artifacts.

    Authors: We agree that quantitative validation of the intent labeler is necessary. The labeler assigns Exploration/Implementation/Verification/Orchestration labels using both tool identity and preceding trajectory context. In the revision we will add (1) inter-annotator agreement (Cohen’s kappa) computed on a random sample of 200 trajectories labeled independently by two authors, (2) a confusion matrix against a small set of expert-annotated trajectories, and (3) an error analysis of the most frequent disagreement categories. These statistics and the annotation guidelines will be included in a new appendix. revision: yes

Circularity Check

0 steps flagged

No load-bearing circularity; PTA construction and labeling are independent of the reported 10.7% statistic

full rationale

The paper's central empirical claim (10.7% lucky passes among passing trajectories) is obtained by direct counting after new trajectory collection, merging into PTA references, and annotation with a context-sensitive labeler. No equation or derivation reduces the reported percentage to a fitted parameter or self-referential definition; the PTA is built from the same passing trajectories but serves as an external reference tree rather than a tautological input. The intent labeler assigns labels from history rather than tool identity, and no self-citation chain is invoked to justify the uniqueness or correctness of the PTA or labeler. This matches the default case of a self-contained empirical analysis whose result is not forced by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 2 invented entities

Based on abstract only; the central claim rests on the validity of the new intent labeler and PTA merging process without specified free parameters or external benchmarks.

axioms (1)
  • domain assumption The context-sensitive intent labeler correctly assigns actions to Exploration, Implementation, Verification, or Orchestration using trajectory history.
    This labeling is required to build quality scores and identify Lucky Passes.
invented entities (2)
  • Lucky Pass no independent evidence
    purpose: Categorize passing trajectories that exhibit regression cycles, blind retries, or disordered processes.
    Newly introduced category to distinguish from Solid or Ideal trajectories.
  • Prefix Tree Acceptor (PTA) reference no independent evidence
    purpose: Merged model of correct process from multiple passing trajectories for a task.
    Invented construct used as the basis for quality scoring.

pith-pipeline@v0.9.1-grok · 5866 in / 1385 out tokens · 36332 ms · 2026-06-30T21:58:42.459891+00:00 · methodology

0 comments
read the original abstract

Evaluation of software engineering (SWE) agents is dominated by a binary signal: whether the final patch passes the tests. This outcome-only view treats a principled solution and a chaotic trial-and-error process as equivalent. We show that this equivalence is empirically false. We evaluate 2,614 OpenHands trajectories from eight model backends on 60 SWE-bench Verified tasks. Of these, 47 have enough passing trajectories to construct task-level process references, yielding a 1,815-trajectory evaluation subset. Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass: regression cycles, blind retries, missing verification, or temporally disordered exploration, implementation, and verification. We introduce AgentLens, a framework for process-level assessment of SWE-agent trajectories, and define AgentLens-Bench, a dataset of 1,815 trajectories annotated with quality scores, waste signals, divergence points, and 47 task-level Prefix Tree Acceptor (PTA) references. AgentLens builds PTA references by merging multiple passing solutions for the same task, and uses a context-sensitive intent labeler to assign actions to Exploration, Implementation, Verification, or Orchestration based on trajectory history rather than tool identity alone. On AgentLens-Bench, the quality score separates passing trajectories into Lucky, Solid, and Ideal tiers and further decomposes Lucky Passes into five recurring mechanisms. Across the eight model backends, Lucky rates range from 0.5% to 23.2%, and some models move by as many as five rank positions when ranked by quality score instead of pass rate. We plan to release the project repository soon, including AgentLens-Bench artifacts, the AgentLens SDK, and the analysis tooling.

Figures

Figures reproduced from arXiv: 2605.12925 by Benjamin Steenhoek, Gaurav Mittal, Pingping Lin, Priyam Sahoo, Shengjie Ma, Xiaomin Li, Yu Hu.

Figure 1
Figure 1. Figure 1: Passing trajectories are not behaviorally homogeneous. Among 1,136 passing trajectories in AGENTLENS￾Bench, AGENTLENS classifies 229 as Ideal (20.2%), 785 as Solid (69.1%), and 122 as Lucky (10.7%). Binary evaluation treats all of these trajectories as equally successful, while process-aware scoring separates direct, coherent solutions from weak processes that happen to pass. further 69.1% are Solid but im… view at source ↗
Figure 2
Figure 2. Figure 2: AGENTLENS workflow. AGENTLENS starts from raw execution traces and converts them into structured state sequences. Intent-stage labeling assigns each state a process role, such as exploration, implementation, testing, or cleanup. These intent-labeled states become the states used for per-trace PTA construction. Passing trajectories for the same task are then merged into a task-level PTA, which represents th… view at source ↗
Figure 3
Figure 3. Figure 3: PTA construction. Two individual passing traces sharing an early exploration prefix but diverging at imple￾mentation are merged into a single DAG. Shared nodes reflect equivalent actions across agents; branches reflect genuine strategic divergence. During construction, states from different trajectories are merged when they represent equivalent actions. The equivalence engine handles surface variation such… view at source ↗
Figure 4
Figure 4. Figure 4: Intent-stage classification decision tree. Each agent action is classified into one of four intent stages (Exploration, Implementation, Verification, Orchestration) via a priority cascade of seven rules. Rules 1 to 4 (gray diamonds) assign fixed stages based on tool type. Rules 5 to 7 (yellow diamonds) are context-sensitive: the same tool can map to different stages depending on whether a source-file edit … view at source ↗
Figure 5
Figure 5. Figure 5: Scoring signal examples. Each panel contrasts a principled trajectory (left) against a chaotic trajectory (right) on one of the four scoring dimensions: (a) coherence, (b) structural alignment, (c) set coverage, and (d) temporal profile divergence. Together, the four signals capture complementary aspects of process quality. the Ideal-versus-Lucky waste comparison that identifies the blind-retry severity fi… view at source ↗
Figure 6
Figure 6. Figure 6: Failure-mode gallery. Six annotated stage-colored timelines: regression loop, blind-retry cluster, temporal disorder, E/V confusion before and after the context-sensitive fix, unnecessary exploration, and a cyclic pattern. C.5 Heuristic Labeler Validation The main text reports aggregate labeler reliability (Section 5.4): Fleiss’ κ = 0.933 with 96.0% raw agreement across seven annotators. This subsection pr… view at source ↗
Figure 7
Figure 7. Figure 7: Score Density by Outcome. Overlapping density histograms of quality scores for Pass (n=1,136) and Fail (n=679) instances. The vertical dashed line marks the empirically chosen threshold at 46.4. Pass instances concentrate in the upper range while fail instances skew left. This confirms the score’s ability to separate outcomes. C.7 Baseline Results [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Pass Rate vs. Mean Quality Score. Each point represents one of eight LLM coding agents evaluated on AGENTLENS-BENCH. Dot color encodes rank divergence, defined as |PR − QS| where PR is the pass-rate rank and QS is the quality-score rank: • consistent (≤ 1), • moderate (2–3), • divergent (≥ 4). Arrows indicate whether a model’s quality rank improves (▲) or drops (▼) relative to its pass-rate rank. 0 5 10 15… view at source ↗
Figure 9
Figure 9. Figure 9 [PITH_FULL_IMAGE:figures/full_fig_p025_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Lucky Pass category distribution. The 122 Lucky Passes decompose into five categories: C1 Minimal & Unverified (19, 15.6%), C2 Brute-Force Convergence (42, 34.4%), C3 Incomplete Implementation (41, 33.6%), C4 Excessive Exploration (5, 4.1%), and C5 Divergent-but-Valid (15, 12.3%). Categories C2 and C3 together account for 68% of all Lucky Passes. C1: Minimal & Unverified (n = 19, 15.6%). The agent finds a… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

3 extracted references · 1 canonical work pages

  1. [1]

    File-scope matchingvia tree-sitter (Brunsfeld et al., 2024) (confidence 0.90): states targeting the same AST-level scope (function, class, module) are equivalent

    2024

  2. [2]

    Line-range overlapof at least 30% (confidence 0.80–0.95): states reading or editing overlapping regions of the same file are equivalent, with confidence scaled by overlap fraction

  3. [3]

    implementation

    Semantic terminal grouping(confidence 0.70–0.85): terminal commands in the same functional group (e.g., grep, rg, ag are all “search” commands) with Jaccard token similarity above 0.5 are treated as equivalent. An optional LLM fallback, which queries a language model for ambiguous cases, was disabled throughout for reproducibility. The cascade is evaluate...

    work page arXiv 1991