Pith. sign in

REVIEW 2 major objections 2 minor 11 references

LLM tool-calling decisions often diverge from what actually improves task results, but hidden-state estimators can correct them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-07-01 07:56 UTC pith:53B2XUSX

load-bearing objection The paper shows LLMs often misalign their tool calls with optimal need/utility and that hidden-state estimators can build better controllers, but the optimality definition is the key assumption to check. the 2 major comments →

arxiv 2605.00737 v2 pith:53B2XUSX submitted 2026-05-01 cs.AI

To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling

Qinyuan Wu , Soumi Das , Mahsa Amani , Arijit Nag , Seungeon Lee , Krishna P. Gummadi , Abhilasha Ravichander , Muhammad Bilal Zafar This is my paper
classification cs.AI
keywords LLM tool usetool calling decisionsnecessity utility affordabilityhidden state estimatorsweb searchagentic systemsdecision misalignmentperformance improvement
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops a framework for deciding when large language models should use external tools such as web search. It distinguishes between a model's self-perceived need for a tool and the true need based on whether using the tool improves final task results. The analysis reveals consistent misalignment between these two views across several models and experimental setups. The authors then show that lightweight predictors trained on the models' internal hidden states can guide better tool-use decisions than the models make on their own.

Core claim

In every tested setting the models' observed decisions about calling web search tools do not match the decisions that would maximize performance under an optimal allocation. This gap appears between the descriptive view drawn from behavior and the normative view drawn from task outcomes. Estimators built from hidden states can be used to build controllers that raise decision quality and final performance above the self-perceived baseline for most open-source models evaluated.

What carries the argument

A three-factor framework (necessity, utility, affordability) that contrasts normative inference of true need from optimal tool-call allocations with descriptive inference from the model's own behavior.

Load-bearing premise

True need and utility of tool calls can be correctly identified by computing the allocation of calls that produces the best task outcomes.

What would settle it

Running the trained controllers on a held-out task distribution or with a different set of tools and finding no performance gain over the self-perceived baseline would indicate the estimators do not capture general decision improvements.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Controllers based on hidden-state estimators yield stronger task performance than relying on the model's own assessment.
  • The misalignment between perceived and true tool utility holds across open-source and closed models as well as different execution harnesses.
  • Simple controllers driven by these estimators raise overall decision quality in web-search augmented tasks.
  • The approach yields gains for most of the open-source models tested under multiple tools and tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Hidden-state based estimators could be added to agent architectures to override poor self-assessments of tool need.
  • The same misalignment may occur with other tool types such as code interpreters or databases.
  • If the estimators prove robust, they offer a way to reduce redundant or noisy tool calls without changing the base model.
  • Future work might examine whether the framework identifies similar issues in multi-turn agent interactions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper introduces a decision-theoretic framework for assessing LLM decisions on web search tool calls, distinguishing a normative perspective (inferring true necessity, utility, and affordability from optimal tool-call allocations) from a descriptive perspective (inferring self-perceived values from observed behaviors). It evaluates six open-source and one closed-source model across two harnesses, four tools, and three tasks, reporting frequent misalignment between perceived and true values in all settings, and shows that lightweight hidden-state estimators can drive controllers yielding better decision quality and task performance than self-perceived baselines for most open-source models.

Significance. If the normative optimal-allocation lens is shown to be independent of model behavior and robust, the work offers a principled way to diagnose and mitigate suboptimal tool use in agentic systems. The multi-model, multi-harness evaluation and the demonstration of internal-state controllers provide concrete evidence that hidden representations can be leveraged for improved tool-calling policies, which would be a useful contribution to LLM agent research.

major comments (2)
  1. [§3] §3 (Framework, normative perspective): The misalignment claim and the value of the downstream estimators rest on defining 'true' need and utility via an optimal allocation of tool calls. The manuscript must specify the exact procedure used to compute this optimal allocation (e.g., exhaustive enumeration, dynamic programming, or external oracle), confirm that it is computed independently of the models' execution traces or observed behaviors, and demonstrate robustness to the chosen task distribution; any dependence would make the reported misalignment an artifact of the evaluation design rather than a general property of the models.
  2. [§4] §4 (Experiments and results): The claim that the hidden-state controllers 'yield stronger task performance than the self-perceived baseline for most of the open-source models' is load-bearing for the practical contribution. The paper should report per-model, per-harness performance deltas with statistical significance tests, confidence intervals, and controls for multiple comparisons across the six models, two harnesses, four tools, and three tasks; without these, it is unclear whether the gains are reliable or driven by a subset of conditions.
minor comments (2)
  1. [Abstract / §4] The two harnesses (current-turn vs. full-trace conditioning) are central to the evaluation but are only briefly described in the abstract; a dedicated table or subsection summarizing their differences, input formats, and how search results are integrated would improve clarity.
  2. [Abstract] The abstract states results hold 'in every setting' without quantifying the total number of settings or noting any exceptions; adding a summary table of misalignment rates across all model-harness-tool-task combinations would make the scope of the finding precise.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback. We address each major comment below and indicate the planned revisions.

read point-by-point responses

  1. Referee: [§3] §3 (Framework, normative perspective): The misalignment claim and the value of the downstream estimators rest on defining 'true' need and utility via an optimal allocation of tool calls. The manuscript must specify the exact procedure used to compute this optimal allocation (e.g., exhaustive enumeration, dynamic programming, or external oracle), confirm that it is computed independently of the models' execution traces or observed behaviors, and demonstrate robustness to the chosen task distribution; any dependence would make the reported misalignment an artifact of the evaluation design rather than a general property of the models.

    Authors: We agree that the normative perspective requires an explicit description of how the optimal allocation is computed. The current manuscript presents this at a conceptual level without the algorithmic details. In the revised manuscript we will expand §3 with the precise procedure (including pseudocode), explicitly state that the computation relies on task-level ground-truth success metrics rather than any model traces or behaviors, and add a robustness analysis across varied task distributions. revision: yes

  2. Referee: [§4] §4 (Experiments and results): The claim that the hidden-state controllers 'yield stronger task performance than the self-perceived baseline for most of the open-source models' is load-bearing for the practical contribution. The paper should report per-model, per-harness performance deltas with statistical significance tests, confidence intervals, and controls for multiple comparisons across the six models, two harnesses, four tools, and three tasks; without these, it is unclear whether the gains are reliable or driven by a subset of conditions.

    Authors: We acknowledge that the experimental claims would be strengthened by more granular statistical reporting. The manuscript currently summarizes results at an aggregate level. In the revision we will add per-model and per-harness tables in §4 (and the appendix) that include performance deltas, 95% confidence intervals, appropriate significance tests, and corrections for multiple comparisons across the full experimental grid. revision: yes

Circularity Check

0 steps flagged

No circularity: normative vs. descriptive distinction is independent of observed model behavior

full rationale

The paper defines true need/utility via a normative optimal allocation computed separately from model behavior, while self-perceived need/utility is derived from observed behaviors; the misalignment claim and hidden-state estimators follow directly from comparing these two independent lenses without any reduction of one to the other by construction. No equations, fitted parameters renamed as predictions, or load-bearing self-citations appear in the provided text. The evaluation spans multiple models, harnesses, tools, and tasks, supplying external benchmarks that keep the derivation self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only; the framework rests on the unstated domain assumption that an optimal allocation of tool calls can be defined for the chosen tasks and that hidden states contain extractable signals of need/utility. No free parameters or invented entities are mentioned.

axioms (1)
  • domain assumption An optimal allocation of tool calls exists and can be computed to serve as ground truth for true need and utility.
    Invoked when the normative perspective is defined; required for the misalignment claim.

pith-pipeline@v0.9.1-grok · 5818 in / 1389 out tokens · 25893 ms · 2026-07-01T07:56:40.540937+00:00 · methodology

0 comments
read the original abstract

Agentic AI architectures augment LLMs with external tools, unlocking strong capabilities. However, tool use is not always beneficial; some calls may be redundant or even harmful. Effective tool use, therefore, hinges on a core LLM decision: whether to call or not call a tool when performing a task. This decision is particularly challenging for web search tools, where the benefits of external information depend on the model's internal knowledge and its ability to integrate potentially noisy tool responses. We introduce a principled framework inspired by decision-making theory to evaluate web search tool-use decisions along three key factors: necessity, utility, and affordability. Our analysis combines two complementary lenses: a normative perspective that infers true need and utility from an optimal allocation of tool calls, and a descriptive perspective that infers the model's self-perceived need and utility from their observed behaviors. We evaluate six open and one closed-source frontier models under two harnesses, one conditioning on only the current turn and its search results, the other on the full execution traces, across four web-search tools and three tasks. In every setting, we find that a model's perceived need and utility are frequently misaligned with the true need and utility. Building on this framework, we train lightweight estimators of need and utility from the models' hidden states. These estimators drive simple controllers that improve decision quality and yield stronger task performance than the self-perceived baseline for most of the open-source models.

Figures

Figures reproduced from arXiv: 2605.00737 by Abhilasha Ravichander, Arijit Nag, Krishna P. Gummadi, Mahsa Amani, Muhammad Bilal Zafar, Qinyuan Wu, Seungeon Lee, Soumi Das.

Figure 1
Figure 1. Figure 1: Given input x, the model M decides π(x) ∈ {0, 1} to call a tool (response r) or not, producing y = M(x,r) or y = M(x). We compare NO TOOL, ALWAYS TOOL, and SELF-DECISION, and evaluate decisions via need (requires help), utility (performance gain), and affordability (cost vs. gain), distinguishing perceived vs. true quantities. 2023) and improve tool selection during inference (Schick et al., 2023a), while … view at source ↗
Figure 2
Figure 2. Figure 2: True need and true positive utility are correlated, but not perfectly aligned. Rows are grouped by the model’s (GPT￾OSS-120B) factuality scores under NO TOOL (parametric knowledge), while columns show scores under ALWAYS TOOL. Scores are bucketed into low (0–0.1), mid (0.1–0.9), and high (0.9–1). Cells above the diagonal indi￾cate positive utility, while those below indi￾cate negative utility. The bracket … view at source ↗
Figure 3
Figure 3. Figure 3: Perceived need is only partially aligned with tool call (perceived utility). The x-axis shows the model’s perceived need, and the y-axis shows perceived utility / tool-call decisions. Need Need No Need Perceived Need No Need True 205 143 14 138 (a) GPT-OSS Need No Need Perceived Need No Need True 256 69 69 106 (b) Qwen3-A3B Need No Need Perceived Need No Need True 303 14 177 6 (c) Qwen3-IT Need No Need Per… view at source ↗
Figure 5
Figure 5. Figure 5: The perceived need and utility are not aligned with the true need and utility. Entity Task; Top: true vs perceived need. Bottom: true vs. perceived utility across models. Most models are self-consistent: perceived utility (tool calling) correlates with perceived need, with two models showing substantial deviations. In the absence of tool descriptions, model behavior is driven solely by perceived need— i.e.… view at source ↗
Figure 4
Figure 4. Figure 4: Perceived signals only partially align with true utility. Venn diagrams of True Positive Utility, Perceived Need, and Per￾ceived Utility for GPT-OSS-120B on the entity task. Ideally, Per￾ceived Utility ⊆ Perceived Need ⊆ True Positive Utility, but this nesting is violated, explaining the suboptimal performance of SELF￾DECISION. Even as models’ perceptions are self-consistent, their percep￾tions do not alig… view at source ↗
Figure 6
Figure 6. Figure 6: Utility gain over the NO TOOL under varying cost con￾straints. Solid lines show optimal allocation (optimal top-k), dashed lines show model performance with cost information, squares de￾note no cost-awareness, and dot￾ted lines indicate always-calling. Cost information has uneven effects. Without cost descriptions, most models (except Mistral and Llama) achieve slightly higher utility gains than in the cos… view at source ↗
Figure 7
Figure 7. Figure 7: LNE improves true-need prediction across models, with larger gains for smaller models. Confusion matrices appear in view at source ↗
Figure 8
Figure 8. Figure 8: Distribution of the number of Google search results across all entities in the entity dataset. view at source ↗
Figure 9
Figure 9. Figure 9: Entity Task: factuality distribution across different models. s ”The term Sky Blue ¨ primarily refers to Coventry City Football Club, an English professional football team ¨ based in Coventry, which plays in the EFL Championship. The club, originally founded in 1883 as Singers F.C., adopted the nickname The Sky Blues ¨ due to their distinctive sky blue kits. They joined the Football ¨ League in 1898 and ha… view at source ↗
Figure 10
Figure 10. Figure 10: Entity task: No-Tool vs. Force-Tool performance. Rows group entities by the model’s factuality score without a tool (reflecting parametric knowledge), while columns group scores when tool use is forced. Each cell reports the count and the column percentage. Off-diagonal cells indicate performance shifts due to tool use: cells above the diagonal show cases where the tool has positive utility, while cells b… view at source ↗
Figure 11
Figure 11. Figure 11: [Entity Task] Venn diagrams of three sets: view at source ↗
Figure 12
Figure 12. Figure 12: Entity Task: perceived need is only partially aligned with tool use. The x-axis shows perceived utility (number of entities predicted to need or not need external information), and the y-axis shows actual tool-use decisions. Percentages indicate how often the model follows its own prediction (call vs. not call). Results are shown for two prompt variants (v1 and v2). Some responses are excluded due to pars… view at source ↗
Figure 13
Figure 13. Figure 13: [Entity Task] Actual tool calls without budget enforcement. Models do not reliably reduce view at source ↗
Figure 14
Figure 14. Figure 14: [Entity Task] The NDCG rank correlation under different budgets across different models. view at source ↗
Figure 15
Figure 15. Figure 15: Cost-aware tool use on the Entity Task. Left: Utility gain over the no-tool baseline under varying cost constraints. Solid lines show oracle allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotted lines indicate always-calling. Right: Actual tool calls without budget enforcement. Models do not reliably reduce or stop calls as cost… view at source ↗
Figure 16
Figure 16. Figure 16: [Entity Task] The NDCG rank correlation under different budgets across different models. view at source ↗
Figure 17
Figure 17. Figure 17: [Entity Task.] Confusion matrix for the LNE estimator E.2 Descriptive Lens As shown in view at source ↗
Figure 18
Figure 18. Figure 18: The LUEs can predict the True Utility more accurately across most models, especially for small and weaker models. We show the confusion matrixs of the two predictors in view at source ↗
Figure 19
Figure 19. Figure 19: [Entity Task.] Confusion matrix for the LUEx estimator F Results for BFCL task In this section, we show the additional results for the entity task. In view at source ↗
Figure 20
Figure 20. Figure 20: [Entity Task.] Confusion matrix for the LUEx,d estimator 0% 20% 40% 60% 80% 100% % Queries with Tool Calls 0.00 0.05 0.10 0.15 0.20 0.25 Utility Gain over No-Tool Gemma3-27B GPT-OSS-120B Qwen3-30B Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B Ideal curve Cost Description LNE (a) LNE 0% 20% 40% 60% 80% 100% % Queries with Tool Calls 0.00 0.05 0.10 0.15 0.20 0.25 Utility Gain over No-Tool Gemma3-27B GPT-OSS-1… view at source ↗
Figure 21
Figure 21. Figure 21: [Entity Task] Tool-call decisions are guided by the latent need estimator’s predicted view at source ↗
Figure 22
Figure 22. Figure 22: [InvivoQuery Task] factuality distribution across different models. Low Mid High Performance with tool Low Mid High Performance without tool True Need Neutral Positive Negative 53 40 14 62 173 34 40 39 45 (a) GPT-OSS-120B Low Mid High Performance with tool Low Mid High Performance without tool True Need Neutral Positive Negative 20 28 19 24 178 74 15 67 75 (b) Qwen3-30B-A3B Low Mid High Performance with t… view at source ↗
Figure 23
Figure 23. Figure 23: [InvivoQuery Task] NO TOOL vs. ALWAYS TOOL performance. Rows group entities by the model’s factuality score without a tool (reflecting parametric knowledge), while columns group scores when tool use is forced. Each cell reports the count and the column percentage. Off-diagonal cells indicate performance shifts due to tool use: cells above the diagonal show cases where the tool has positive utility, while … view at source ↗
Figure 24
Figure 24. Figure 24: [InvivoQuery Task] Venn diagrams of three sets: view at source ↗
Figure 25
Figure 25. Figure 25: [InvivoQuery Task] perceived need is only partially aligned with tool use. The x-axis shows perceived utility (number of entities predicted to need or not need external information), and the y-axis shows actual tool-use decisions. Percentages indicate how often the model follows its own prediction (call vs. not call). Results are shown for three prompt variants (v2 and v3). Some responses are excluded due… view at source ↗
Figure 26
Figure 26. Figure 26: The perceived need and utility are not aligned with the true need and utility. Invivo￾Query Task; Left: perceived need matrices. Right: true vs. perceived utility across models. 0% 20% 40% 60% 80% 100% % Queries with Tool Calls 0.00 0.05 0.10 0.15 Utility Gain over No-Tool Gemma3-27B GPT-OSS-120B Qwen3-30B Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B Cost Description No Cost Description (a) Utility gain wi… view at source ↗
Figure 27
Figure 27. Figure 27: Cost-aware tool use on the Invivo Task with implicit budget notification. Left: Utility gain over the no-tool baseline under varying cost constraints. Solid lines show oracle allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotted lines indicate always-calling. Right: Actual tool calls without budget enforcement. Models do not rel… view at source ↗
Figure 28
Figure 28. Figure 28: [InvivoQuery Task] The NDCG rank correlation under different budgets across different view at source ↗
Figure 29
Figure 29. Figure 29: Cost-aware tool use on the Invivo Task with explicit budget notification. Left: Utility gain over the no-tool baseline under varying cost constraints. Solid lines show oracle allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotted lines indicate always-calling. Right: Actual tool calls without budget enforcement. Models do not rel… view at source ↗
Figure 30
Figure 30. Figure 30: [InvivoQuery Task] The NDCG rank correlation under different budgets across different view at source ↗
Figure 31
Figure 31. Figure 31: InvivoQuery task: The LNE can predict the True Need more accurately across most models, especially for small and weaker models. 38 view at source ↗
Figure 32
Figure 32. Figure 32: [InvivoQuery Task.] Confusion matrix for the view at source ↗
Figure 33
Figure 33. Figure 33: The LUE can predict the True Utility more accurately across most models, especially for small and weaker models. In the view at source ↗
Figure 34
Figure 34. Figure 34: [InvivoQuery Task.] Confusion matrix for the view at source ↗
Figure 35
Figure 35. Figure 35: [InvivoQuery Task.] Confusion matrix for the view at source ↗
Figure 36
Figure 36. Figure 36: [InvivoQuery Task] Tool-call decisions are guided by the latent need estimator’s predicted view at source ↗
Figure 37
Figure 37. Figure 37: [BFCL Task] factuality distribution across different models. Incorrect Correct Performance with tool Incorrect Correct Performance without tool True Need Neutral Positive Negative 70 127 7 110 (a) GPT-OSS-120B Incorrect Correct Performance with tool Incorrect Correct Performance without tool True Need Neutral Positive Negative 108 144 12 50 (b) Qwen3-30B-A3B Incorrect Correct Performance with tool Incorre… view at source ↗
Figure 38
Figure 38. Figure 38: [BFCL task]: No-Tool vs. Force-Tool performance. Rows group entities by the model’s factuality score without a tool (reflecting parametric knowledge), while columns group scores when tool use is forced. Each cell reports the count and the column percentage. Off-diagonal cells indicate performance shifts due to tool use: cells above the diagonal show cases where the tool has positive utility, while cells b… view at source ↗
Figure 39
Figure 39. Figure 39: [BFCL Task] Venn diagrams of three sets: view at source ↗
Figure 40
Figure 40. Figure 40: [BFCL Task]: perceived need is only partially aligned with tool use. The x-axis shows perceived utility (number of entities predicted to need or not need external information), and the y-axis shows actual tool-use decisions. Percentages indicate how often the model follows its own prediction (call vs. not call). Results are shown for three prompt variants. Some responses are excluded due to parsing failur… view at source ↗
Figure 41
Figure 41. Figure 41: [BFCL Task] The perceived need and utility are not aligned with the true need and utility. Left: perceived need matrices. Right: true vs. perceived utility across models. 0% 20% 40% 60% 80% 100% % Queries with Tool Calls 0.0 0.1 0.2 0.3 0.4 0.5 Utility Gain over No-Tool Gemma3-27B GPT-OSS-120B Qwen3-30B Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B Cost Description No Cost Description (a) Utility gain with … view at source ↗
Figure 42
Figure 42. Figure 42: Cost-aware tool use on the BFCL Task with implicit budget notification.. Left: Utility gain over the no-tool baseline under varying cost constraints. Solid lines show oracle allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotted lines indicate always-calling. Right: Actual tool calls without budget enforcement. Models do not reli… view at source ↗
Figure 43
Figure 43. Figure 43: [BFCL Task] The NDCG rank correlation under different budgets across different models. view at source ↗
Figure 44
Figure 44. Figure 44: Cost-aware tool use on the BFCL Task with explicit budget notification. Left: Utility gain over the no-tool baseline under varying cost constraints. Solid lines show oracle allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotted lines indicate always-calling. Right: Actual tool calls without budget enforcement. Models do not relia… view at source ↗
Figure 45
Figure 45. Figure 45: [BFCL Task] The NDCG rank correlation under different budgets across different models. view at source ↗
Figure 46
Figure 46. Figure 46: BFCL task: The LNE can predict the True Need more accurately across most models, especially for small and weaker models. Need No Need Predicted Need No Need Actual 162 (78%) 35 (33%) 45 (22%) 72 (67%) (a) GPT-OSS￾120B Need No Need Predicted Need No Need Actual 221 (89%) 31 (47%) 27 (11%) 35 (53%) (b) Qwen3-30B￾A3B Need No Need Predicted Need No Need Actual 189 (84%) 30 (34%) 37 (16%) 58 (66%) (c) Qwen3-30… view at source ↗
Figure 47
Figure 47. Figure 47: [BFCL Task.] Confusion matrix for the LNE estimator Gemma3-27B GPT-OSS-120B Qwen3-30B Qwen3-30B-IT Mistral3.1-24B Llama3.2-3B 0.00 0.20 0.40 0.60 0.80 1.00 Accuracy Perceived Utility LUEx, d LUEx view at source ↗
Figure 48
Figure 48. Figure 48: The LUE can predict the True Utility more accurately across most models, especially for small and weaker models.. 45 view at source ↗
Figure 49
Figure 49. Figure 49: [BFCL Task.] Confusion matrix for the LUEx estimator Help No Help Predicted Help No Help Actual 132 (68%) 55 (46%) 63 (32%) 64 (54%) (a) GPT-OSS￾120B Help No Help Predicted Help No Help Actual 115 (72%) 55 (36%) 45 (28%) 99 (64%) (b) Qwen3-30B￾A3B Help No Help Predicted Help No Help Actual 131 (75%) 39 (28%) 43 (25%) 101 (72%) (c) Qwen3-30B￾A3B-Instruct Help No Help Predicted Help No Help Actual 176 (71%)… view at source ↗
Figure 50
Figure 50. Figure 50: [BFCL Task.] Confusion matrix for the LUEx,d estimator 0% 20% 40% 60% 80% 100% % Queries with Tool Calls 0.0 0.1 0.2 0.3 0.4 0.5 Utility Gain over No-Tool Gemma3-27B GPT-OSS-120B Qwen3-30B Qwen-3-30B-IT Mistral3.1-24B-IT Llama3.2-3B Ideal curve Cost Description LNE (a) LNE 0% 20% 40% 60% 80% 100% % Queries with Tool Calls 0.0 0.1 0.2 0.3 0.4 0.5 Utility Gain over No-Tool Gemma3-27B GPT-OSS-120B Qwen3-30B … view at source ↗
Figure 51
Figure 51. Figure 51: [BFCL Task] Tool-call decisions are guided by the latent need estimator’s predicted view at source ↗
Figure 52
Figure 52. Figure 52: [Entity Task, Perplexity Search] True need and positive utility are correlated, but not perfectly aligned. Rows group by the model’s (GPT-OSS-120B) factuality scores under NO TOOL (parametric knowledge), while columns show scores under ALWAYS TOOL. Scores are bucketed into low (0–0.1), mid (0.1–0.9), and high (0.9–1). Cells above the diagonal indicate positive utility, while those below indicate negative … view at source ↗
Figure 53
Figure 53. Figure 53: [Entity Task; Perplexity Search] Perceived need is only partially aligned with tool call (perceived utility). Model: GPT-OSS-120B. The x-axis shows the model’s perceived need, and the y-axis shows perceived utility / tool-call decisions. 64 52 30 3 1 62 78 True Positive Utility Perceived Need Perceived Utility view at source ↗
Figure 54
Figure 54. Figure 54: [Entity Task; Perplexity Search] Perceived signals only partially align with true utility. Venn diagrams of True Positive Utility, Perceived Need, and Perceived Utility for GPT-OSS-120B on the entity task illustrate this misalignment. Ideally, Perceived Utility ⊆ Perceived Need ⊆ True Positive Utility. However, this nesting is violated, indicating misalignment with true utility and helping to explain the … view at source ↗
Figure 55
Figure 55. Figure 55: [Entity Task; Perplexity Search] The perceived need and utility are not aligned with the true need and utility. Entity Task; Top: perceived need matrices. Bottom: true vs. perceived utility across models. 47 view at source ↗
Figure 56
Figure 56. Figure 56: [Entity Task; Perplexity Search] Cost-aware tool use with explicit budget notification. Left: Utility gain over the no-tool baseline under varying cost constraints. Solid lines show oracle allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotted lines indicate always-calling. Right: Actual tool calls without budget enforcement. Mod… view at source ↗
Figure 57
Figure 57. Figure 57: [Entity Task; Perplexity Search] Cost-aware tool use with implicit budget notification. Left: Utility gain over the no-tool baseline under varying cost constraints. Solid lines show oracle allocation (optimal top-k), dashed lines show model performance with cost information, squares denote no cost-awareness, and dotted lines indicate always-calling. Right: Actual tool calls without budget enforcement. Mod… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

11 extracted references

  1. [1]

    Claim extraction.An extraction model (default: GPT-4o) is prompted with a structured JSON schema to decompose the model’s response into atomic, checkable claims (factual, numerical,historical,definition,other)

  2. [2]

    type": "object

    Claim verification.A verification model (default: GPT-4o) is given the extracted claims and uses its built-in web search tool to assess each claim against live web sources. It returns a booleanis correctflag and one-to-three source URLs per claim. 16 Preprint. Under review. The factuality score for a response is s= correct claims total claims ∈[0, 1]. Bot...

    2023

  3. [3]

    Source: https://leviton.com/products/cda01-k12

    Current Transformer — Submetering — Submeter — CDA01-K12 Current Transformer, Solid Core, 100A, 100mA, 0.72” Opening, 48” Leads, +/-0.3% Accuracy, Black, For Submetering. Source: https://leviton.com/products/cda01-k12

  4. [4]

    The Leviton CDA01-K12 solid core current transformer (CT) is cost effective and less susceptible to damage during installation

    Leviton CDA01-K12 Current Transformer, 100:0.1A, Solid ... The Leviton CDA01-K12 solid core current transformer (CT) is cost effective and less susceptible to damage during installation. Source: https://www.powermeterstore.com/product/

  5. [5]

    Designed for accurately capturing measurements of power consumption, CTs are easy to specify and install

    VerifEye™, Sub-Metering Current Transformer, 100A, Solid ... Designed for accurately capturing measurements of power consumption, CTs are easy to specify and install. Source: https://www.graybar.com/

  6. [6]

    Source: https://www.bulbspro.com/media/pdf/CDA01-K12.pdf

    CDA01-K12.pdf Leviton solid core CTs are cost effective and less susceptible to damage during installation. Source: https://www.bulbspro.com/media/pdf/CDA01-K12.pdf

  7. [7]

    In a paragraph, could you tell me what you know about CDA01?

    Leviton® CDA01-R12 Solid Core Sub-Metering Current ... 100:0.1 current ratio, 100 A primary, 0.1 A secondary, 0.3% accuracy. Source: https://www.steinerelectric.com/... Question: In a paragraph, could you tell me what you know about CDA01? <|im end|> <|im start|>assistant The no-tool setup.We skip the tool decision stage for the no-search setup. The promp...

    2024

  8. [8]

    The club plays in the EFL Championship and is nicknamed the Sky Blues after its sky blue colours

    Coventry City F.C. The club plays in the EFL Championship and is nicknamed the Sky Blues after its sky blue colours. Source: https://en.wikipedia.org/wiki/Coventry City F.C

  9. [9]

    Source: https://www.ccfc.co.uk/

    Club History The Sky Blues played their first game at the Ricoh Arena in 2005, winning 3-0. Source: https://www.ccfc.co.uk/

    2005

  10. [10]

    Source: https://www.planetsport.com/

    Sky Blue FC Profile Sky Blue FC was founded in New Jersey in 2007, with its inaugural season in 2009. Source: https://www.planetsport.com/

    2007

  11. [11]

    The Sky Blues

    Founded in 1883 as Singers F.C. Coventry City F.C., now known as "The Sky Blues", was founded in 1883. Source: https://www.facebook.com/... Question: In a paragraph, could you tell me what you know about Sky Blue?<|im end|> <|im start|>assistant <|im end|> The response that gets the factuality score of 0.58. 25 Preprint. Under review. 0.0 0.2 0.4 0.6 0.8 ...

    2005