Pith. sign in

REVIEW 1 major objections 1 minor 77 references

PageGuide browser extension uses visual overlays to ground LLM answers on web page elements, raising accuracy and cutting time in user tests for find, guide, and hide tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-07-01 09:06 UTC pith:BYTMOF6F

load-bearing objection PageGuide gives a concrete browser extension for visually grounding LLM answers on the page via DOM overlays, but the N=94 study gains cannot be assessed without any methodology details. the 1 major comments →

arxiv 2604.23772 v3 pith:BYTMOF6F submitted 2026-04-26 cs.HC

PageGuide: Browser extension to assist users in navigating a webpage and locating information

Tin Nguyen , Thang T. Truong , Runtao Zhou , Trung Bui , Chirag Agarwal , Anh Totti Nguyen This is my paper
classification cs.HC
keywords browser extensionweb navigationLLM groundingvisual overlaysuser studyinformation findingtask guidancecontent hiding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces PageGuide, a browser extension that connects AI-generated answers directly to the source content on a webpage through visual overlays instead of leaving users to search or trust results blindly. It tackles three needs with specific modes: Find highlights relevant evidence in place for quick verification, Guide delivers one instruction at a time so users can follow along themselves, and Hide lets people remove distracting sections after deciding. In a study with 94 participants, these features produced measurable gains over regular browsing, such as higher success rates and shorter completion times across the modes. The approach keeps the user in the loop while leveraging AI, addressing the common problem of cluttered pages and ungrounded assistance.

Core claim

PageGuide is a browser extension that grounds LLM answers directly in the HTML DOM via visual overlays. It supports three modes: Find for locating and highlighting relevant evidence in-situ, Guide for presenting step-by-step instructions one at a time, and Hide for allowing users to decide whether to remove distracting elements. In a user study with 94 participants, PageGuide outperformed unaided browsing with Hide accuracy improving by 26 percentage points and task time dropping by 70 percent, Guide completion rate increasing by 30 percentage points, and Find reducing Ctrl+F usage by 80 percent along with 19 percent less task time.

What carries the argument

Visual overlays that map LLM outputs to specific HTML DOM elements for in-place highlighting, guidance, and hiding.

Load-bearing premise

The performance gains measured in the N=94 user study are caused by the PageGuide features rather than by task selection, participant assignment, or other aspects of the study design.

What would settle it

A follow-up study using different tasks, a larger or more diverse participant pool, and tighter controls that finds no significant difference in accuracy or time between PageGuide and unaided browsing would show the reported benefits do not hold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Users can verify AI answers on the actual page without separate manual searches.
  • Step-by-step guidance enables users to complete multi-step tasks themselves rather than relying on full automation.
  • Selective hiding of content improves focus and accuracy when locating information on cluttered pages.
  • Reduced use of shortcuts like Ctrl+F indicates lower manual search effort across tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The grounding method could be adapted to non-browser interfaces such as document viewers or mobile apps to provide similar source-linked assistance.
  • Repeated interactions might let the system learn and suggest personalized hiding rules based on past user choices.
  • Pairing the overlay approach with more advanced browser agents could create systems that offer guidance alongside partial automation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The paper presents PageGuide, a browser extension that grounds LLM outputs via visual DOM overlays to support three modes: Find (locating and highlighting evidence in-situ), Guide (step-by-step instructions for multi-step tasks), and Hide (selectively hiding distracting content). It claims that in a user study (N=94), PageGuide outperforms unaided browsing with a 26 percentage point accuracy gain (86.7% relative) and 70% time reduction in Hide, a 30 percentage point completion rate increase in Guide, and an 80% drop in Ctrl+F usage plus 19% time reduction in Find.

Significance. If the empirical claims hold after methodological details are supplied, the work would be significant for HCI by showing how visual grounding can address verification and trust problems with ungrounded LLM assistants and browser agents. The public release of code and demo at pageguide.github.io is a clear strength for reproducibility and extension by others.

major comments (1)
  1. [Abstract] Abstract (user study paragraph): The headline quantitative claims (Hide +26pp accuracy, Guide +30pp completion, Find -80% Ctrl+F) rest entirely on the N=94 study, yet the manuscript supplies no information on design (between- vs. within-subjects, counterbalancing, task selection criteria), controls (prior familiarity, blinding), or analysis (statistical tests, effect sizes, confidence intervals). This is load-bearing for the central claim that gains are caused by the visual-grounding, step-by-step, and hiding mechanisms rather than confounds.
minor comments (1)
  1. [Abstract] Abstract: grammatical and agreement errors ('PageGuide outperform', 'Hide accuracy improve', 'Code and demo is at') should be corrected for clarity.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for highlighting the need for greater methodological transparency in the user study. We agree that the reported performance gains require detailed justification of the experimental design to rule out confounds, and we will revise the manuscript to supply this information.

read point-by-point responses

  1. Referee: [Abstract] Abstract (user study paragraph): The headline quantitative claims (Hide +26pp accuracy, Guide +30pp completion, Find -80% Ctrl+F) rest entirely on the N=94 study, yet the manuscript supplies no information on design (between- vs. within-subjects, counterbalancing, task selection criteria), controls (prior familiarity, blinding), or analysis (statistical tests, effect sizes, confidence intervals). This is load-bearing for the central claim that gains are caused by the visual-grounding, step-by-step, and hiding mechanisms rather than confounds.

    Authors: We acknowledge that the manuscript currently provides only high-level results without the requested methodological details on study design, controls, or analysis. This omission weakens the ability to attribute the gains specifically to PageGuide's mechanisms. In the revision we will expand the User Study section with a full description of the experimental protocol, including design type and counterbalancing, task selection criteria, controls for prior familiarity and blinding, and the statistical tests, effect sizes, and confidence intervals used. These additions will directly address the concern that the claims rest on unverified assumptions. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical user study with no derivations or self-referential claims

full rationale

The paper describes a browser extension (PageGuide) and reports results from an N=94 user study comparing it to unaided browsing. No equations, fitted parameters, predictions derived from inputs, or self-citations appear in the abstract or described claims. The central performance deltas (e.g., +26pp accuracy, -70% time) are presented as direct empirical outcomes rather than reductions of any prior result by construction. This matches the default case of a self-contained empirical system paper with no load-bearing derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is an applied HCI systems paper. No free parameters, mathematical axioms, or invented entities are involved; claims rest on the unreviewed user study whose design details are absent from the abstract.

pith-pipeline@v0.9.1-grok · 5808 in / 1080 out tokens · 25503 ms · 2026-07-01T09:06:24.917968+00:00 · methodology

0 comments
read the original abstract

Users browsing the web daily struggle to quickly locate relevant information in cluttered pages, complete unfamiliar multi-step tasks, and stay focused amid distracting content. State-of-the-art AI assistants (e.g., ChatGPT, Gemini, Claude) and browser agents (e.g., OpenAI Operator, Browser Use) can answer questions and automate actions, yet they return answers without showing where the information comes from on the page, forcing users to manually verify results and blindly trust every automated steps. We present PageGuide, a browser extension that grounds LLM answers directly in the HTML DOM via visual overlays, addressing three core user needs: (a) Find-locating and highlighting relevant evidence in-situ so users can instantly verify answers on the page; (b) Guide-showing step-by-step instructions (e.g. how to change password) one at a time so users can follow and perform actions by themselves; and (c) Hide-hiding distracting content-giving users a chance to decide to hide an element or not. In a user study (N=94), PageGuide outperform unaided browsing across all modes: Hide accuracy improve by 26 percentage points (86.7% relative gain) and task completion time drops by 70%; Guide completion rate increases by 30 percentage points; and Find reduces manual search effort, with Ctrl+F usage falling by 80% and task time decreasing by 19%. Code and demo is at: pageguide.github.io.

Figures

Figures reproduced from arXiv: 2604.23772 by Anh Totti Nguyen, Chirag Agarwal, Runtao Zhou, Thang T. Truong, Tin Nguyen, Trung Bui.

Figure 1
Figure 1. Figure 1: (a) Existing web agents can fail in three ways: by providing answers that cannot be verified from the page, failing view at source ↗
Figure 2
Figure 2. Figure 2: Given the query “How to add ABC to this GitHub project?”, PageGuide (powered by Gemini-3-Flash ) generates a step-by-step plan and delivers it one step at a time. The target UI element is highlighted directly on the page (e.g., Settings , Collaborators )), while the sidebar panel shows the current instruction, the outcome hint (), and Next / Stop controls. The user always drives the pace: each step only a… view at source ↗
Figure 3
Figure 3. Figure 3: On social platforms such as X.com, users often encounter repetitive or distracting content. Given the query view at source ↗
Figure 4
Figure 4. Figure 4: Given a user query, the Router assigns it to one of view at source ↗
Figure 5
Figure 5. Figure 5: Task performance comparing control and extension conditions across all three features. view at source ↗
Figure 6
Figure 6. Figure 6: Task completion time (seconds) for the control and extension conditions, restricted to correctly completed tasks. view at source ↗
Figure 7
Figure 7. Figure 7: Behavioral signals (mean ± SE) comparing the control and extension conditions. Each bar shows the average count or distance per task for five metrics: Ctrl+F presses, text selections, mouse clicks, scroll count, and mouse movement distance. All five metrics decrease substantially with PageGuide, indicating that users rely less on manual search and perform fewer interactions to complete the same tasks. unai… view at source ↗
Figure 12
Figure 12. Figure 12: Using these responses, we address RQ3: do users per￾ceive completing tasks with PageGuide as easier and less effortful compared to completing the same tasks without it? RQ3) Users rate PageGuide as easier to use and effective in helping them locate answers. Participants’ subjective ratings view at source ↗
Figure 8
Figure 8. Figure 8: Behavioral metrics (mean ± SE) broken down by task type (Find, Guide, Hide) and condition. While Find and Hide show consistent reductions across all signals, Guide shows a different pattern: page visits and mouse movement distance increase with PageGuide, reflecting that the extension actively guides users to navigate to new pages as part of the task. F1: PageGuide can correctly locate the information need… view at source ↗
Figure 9
Figure 9. Figure 9: Post-study Likert ratings (1 = Strongly Disagree, 7 = Strongly Agree) for each interaction mode. Each mode includes view at source ↗
Figure 10
Figure 10. Figure 10: Task outcome distributions for Guide and Hide tasks, based on participants’ response to “Did you com￾plete the task?” after each trial (completed / partial / failed), shown as the percentage of participants per category. With PageGuide, completed outcomes increase and failures de￾crease in both features. Notably, Guide shows a rise in partial completions under the extension condition, indicating that user… view at source ↗
Figure 11
Figure 11. Figure 11: Task interfaces displayed to participants during the user study, covering each of view at source ↗
Figure 12
Figure 12. Figure 12: Post-Study Questionnaire view at source ↗
Figure 13
Figure 13. Figure 13: Routing prompt used to classify user queries into appropriate handlers ( view at source ↗
Figure 14
Figure 14. Figure 14: Prompt used to answer user queries grounded in webpage content, producing responses with interleaved references view at source ↗
Figure 15
Figure 15. Figure 15: Hide mode prompt. Given the page element index and the user’s natural language description of content to suppress, the LLM returns a ranked list of up to 15 matching element indices, each paired with a one-sentence hiding reason. The result is used to apply display:none to the identified elements and populate the inline placeholder shown to the user view at source ↗
Figure 16
Figure 16. Figure 16: Guide mode prompt. The LLM receives the current page element index, the user query, the current step number, and any previously completed steps. It returns one action at a time as a JSON object specifying the instruction text, the SoM index and text of the target element, the action type (click, input, scroll), and a next-step hint view at source ↗
Figure 17
Figure 17. Figure 17: Data generation pipelines for the three subsets of PageGuide. (A) The Find pipeline collects webpages, filters for view at source ↗
Figure 18
Figure 18. Figure 18: Users can upload a PDF and ask questions about its contents through the PDF upload feature. In this example, the view at source ↗
Figure 19
Figure 19. Figure 19: PageGuide allows users to upload an image and ask questions about it, while grounding its textual response in evidence from the webpage and specific areas of the user’s image. In this example, the user asks whether the tiger in the uploaded image is the same species as the tiger described in the webpage, and the agent responds with evidence-backed information. Note: The yellow bounding box highlighting th… view at source ↗
Figure 20
Figure 20. Figure 20: With the Page Off feature enabled, users can ask questions that are not related to the currently active tab. The view at source ↗
Figure 21
Figure 21. Figure 21: Example of Gemini Chatbot on the query “How many female casts in Stranger Things?”. The chatbot provides an answer with partial references to text spans, such as “Stranger Things,” but does not link evidence to individual cast names, making verification difficult. In addition, the relevant evidence is embedded within dense text sections, and the highlights include unrelated information, such as male cast … view at source ↗
Figure 22
Figure 22. Figure 22: Example of Browser Use on https://en.wikipedia.org/wiki/Stranger_Things. The agent generates a response that view at source ↗
Figure 23
Figure 23. Figure 23: Example of MolmoWeb [14] on https://en.wikipedia.org/wiki/Elon_Musk. The agent generates a response that correctly answers the query is unable to highlight corresponding evidence, such as through bounding boxes, which makes it difficult for the users to verify the response view at source ↗
Figure 24
Figure 24. Figure 24: Example of Gemini Agent on https://us.megabus.com/. The agent generates a response that correctly answers the view at source ↗
Figure 25
Figure 25. Figure 25: Given the query “How to find the time frame for finding a lost item?” on https://us.megabus.com/, PageGuide (powered by Gemini-3-Flash ) not only generates a step-by-step plan but also highlights the relevant text spans on the webpage, making it easier for users to locate and verify the information view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 26 canonical work pages · 7 internal anchors

  1. [1]

    Chirag Agarwal, Sree Harsha Tanneru, and Himabindu Lakkaraju. 2024. Faithful- ness vs. plausibility: On the (un) reliability of explanations from large language models.arXiv preprint arXiv:2402.04614(2024)

    work page arXiv 2024

  2. [2]

    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, et al. 2019. Guidelines for Human-AI Interaction. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 3:1–3:13. doi:10.1145/ 3290605.3300233

    work page arXiv 2019

  3. [3]

    Anthropic. 2024. 3.5 Sonnet, Computer Use, and the API updates. Anthropic announcement. https://www.anthropic.com/news/3-5-models-and-computer- use Accessed: 2026-03-03

    2024

  4. [4]

    Anthropic. 2026. Get started with Claude in Chrome. Anthropic Help Cen- ter. https://support.claude.com/en/articles/12012173-get-started-with-claude- in-chrome Accessed: 2026-03-31

    work page arXiv 2026

  5. [5]

    Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Carbune, Jason Lin, Jindong Chen, and Abhanshu Sharma

  6. [6]

    Screenai: A vision-language model for ui and infographics understanding.arXiv preprint arXiv:2402.04615, 2024

    ScreenAI: A vision-language model for UI and infographics understanding. arXiv preprint arXiv:2402.04615

    work page arXiv

  7. [7]

    Longformer: The Long-Document Transformer

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long- Document Transformer. arXiv preprint arXiv:2004.05150

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  8. [8]

    browser-use. 2025. browser-use: Make websites accessible for AI agents. GitHub repository. https://github.com/browser-use/browser-use Accessed: 2026-03-02

    2025

  9. [9]

    Phil Cuvin, Hao Zhu, and Diyi Yang. 2025. DECEPTICON: How Dark Patterns Manipulate Web Agents. arXiv preprint arXiv:2512.22894

    work page arXiv 2025

  10. [10]

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. InProceedings of the 2021 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies. 4599–4610

    2021

  11. [11]

    DataReportal, We Are Social, and Meltwater. 2025. Digital 2025: Global Overview Report. Annual global digital report. https://datareportal.com/reports/digital- 2025-global-overview-report

    2025

  12. [12]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2Web: Towards a Generalist Agent for the Web. InAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 28091–28114. https://proceedings.neurips.cc/pa...

    2023

  13. [13]

    Google. 2025. Bring Gemini to Chrome. Google Blog. https://blog.google/ products-and-platforms/products/chrome/gemini-3-auto-browse/ Accessed: 2026-03-02

    2025

  14. [14]

    Ananya Gubbi Mohanbabu, Yotam Sechayk, and Amy Pavel. 2025. Task Mode: Dynamic Filtering for Task-Specific Web Navigation using LLMs. InProceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, New York, NY, USA, 1–18

    2025

  15. [15]

    Tanmay Gupta, Piper Wolters, Zixian Ma, Peter Sushko, Rock Yuren Pang, Diego Llanes, Yue Yang, Taira Anderson, Boyuan Zheng, Zhongzheng Ren, Harsh Trivedi, Taylor Blanton, Caleb Ouellette, Winson Han, Ali Farhadi, and Ranjay Krishna. 2026. MolmoWeb: Open Visual Web Agent and Open Data for the Open Web. https://allenai.org/papers/molmoweb

    2026

  16. [16]

    GWI. 2025. Ad blockers in 2025: Key trends and what they mean for advertisers. GWI insights article. https://www.gwi.com/blog/ad-blockers

    2025

  17. [17]

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Assoc...

    work page doi:10.18653/v1/2024.acl-long.371 2024

  18. [18]

    Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Pittsburgh, Pennsylvania, USA)(CHI ’99). Association for Computing Machinery, New York, NY, USA, 159–166. doi:10.1145/302979.303030

    work page doi:10.1145/302979.303030 1999

  19. [19]

    Shagun Jhaver, Alice Qian Zhang, Quan Ze Chen, Nikhila Natarajan, Ruotong Wang, and Amy X Zhang. 2023. Personalizing content moderation on social media: User perspectives on moderation choices, interface design, and labor. Proceedings of the ACM on Human-Computer Interaction7, CSCW2 (2023), 1–33

    2023

  20. [20]

    Jaylen Jones, Zhehao Zhang, Yuting Ning, Eric Fosler-Lussier, Pierre-Luc St- Charles, Yoshua Bengio, Dawn Song, Yu Su, and Huan Sun. 2026. When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer- Use Agents. arXiv preprint arXiv:2602.08235

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  21. [21]

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. Vi- sualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), Lun-Wei Ku, ...

    work page doi:10.18653/v1/2024.acl-long.50 2024

  22. [22]

    Thomas Kuntz, Agatha Duzan, Hao Zhao, Francesco Croce, Zico Kolter, Nicolas Flammarion, and Maksym Andriushchenko. 2025. OS-Harm: A benchmark for measuring safety of computer use agents. arXiv preprint arXiv:2506.14866

    work page arXiv 2025

  23. [23]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: A Benchmark for Question Answering Research. I...

    work page doi:10.1162/tacl_a_00276 2019

  24. [24]

    Hao-Ping (Hank) Lee, Yi-Shyuan Chiang, Lan Gao, Stephanie Yang, Philipp Winter, and Sauvik Das. 2025. Purpose Mode: Reducing Distraction through Toggling Attention Capture Damaging Patterns on Social Media Web Sites.ACM Trans. Comput.-Hum. Interact.32, 1, Article 10 (April 2025), 41 pages. doi:10.1145/ 3711841

    2025

  25. [25]

    Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov

  26. [26]

    ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents

    ST-WebAgentBench: A benchmark for evaluating safety and trustworthi- ness in web agents. arXiv preprint arXiv:2410.06703

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [27]

    Ping Liu, Karthik Shivaram, Aron Culotta, Matthew Shapiro, and Mustafa Bilgic

  28. [28]

    InProceedings of the International AAAI Conference on Web and Social Media, Vol

    How Does Empowering Users with Greater System Control Affect News Filter Bubbles?. InProceedings of the International AAAI Conference on Web and Social Media, Vol. 18. 943–957

  29. [29]

    Elita Lobo, Chirag Agarwal, and Himabindu Lakkaraju. 2025. On the impact of fine-tuning on chain-of-thought reasoning. InProceedings of the 2025 Confer- ence of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 11679–11698

    2025

  30. [30]

    Kai Mei, Jiang Guo, Shuaichen Chang, Mingwen Dong, Dongkyu Lee, Xing Niu, and Jiarong Jiang. 2026. R-WoM: Retrieval-augmented World Model For Computer-use Agents. arXiv:2510.11892 [cs.CL] https://arxiv.org/abs/2510.11892

    work page arXiv 2026

  31. [31]

    Yohei Nakajima. 2023. BabyAGI. GitHub repository. https://github.com/ yoheinakajima/babyagi Accessed: 2026-03-03

    2023

  32. [32]

    Tin Nguyen, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen

  33. [33]

    HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputs

    HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputs. arXiv preprint arXiv:2503.02003. doi:10.48550/arXiv.2503.02003 Version 5

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2503.02003

  34. [34]

    OpenAI. 2025. ChatGPT Agent: Browsing and completing tasks on the web (Atlas). OpenAI product page. https://chatgpt.com/features/agent Accessed: 2026-03-02

    2025

  35. [35]

    OpenAI. 2025. Introducing Operator. OpenAI product announcement. https: //openai.com/index/introducing-operator/ Accessed: 2026-03-03

    2025

  36. [36]

    Significant Gravitas. 2023. Auto-GPT. GitHub repository. https://github.com/ Significant-Gravitas/AutoGPT Accessed: 2026-03-03

    2023

  37. [37]

    Stanford News. 2025. Social media research tool lowers the political temperature. (November 2025). https://news.stanford.edu/stories/2025/11/social-media-tool- polarization-user-control-research Accessed: 2026-03-26

    2025

  38. [38]

    John Sweller. 1988. Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science12, 2 (1988), 257–285. doi:10.1207/s15516709cog1202_4

    work page doi:10.1207/s15516709cog1202_4 1988

  39. [39]

    Sree Harsha Tanneru, Dan Ley, Chirag Agarwal, and Himabindu Lakkaraju. 2024. On the Difficulty of Faithful Chain-of-Thought Reasoning in Large Language Models. InTrustworthy Multi-modal Foundation Models and AI Agents (TiFA). https://openreview.net/forum?id=3h0kZdPhAC

    2024

  40. [40]

    The Browser Company. 2025. Dia Browser. Official product page. https: //www.diabrowser.com/ Accessed: 2026-03-02

    2025

  41. [41]

    Ada Defne Tur, Nicholas Meade, Xing Han Lu, Alejandra Zambrano, Arkil Pa- tel, Esin Durmus, Spandana Gella, Karolina Stanczak, and Siva Reddy. 2025. SafeArena: Evaluating the safety of autonomous web agents. arXiv preprint arXiv:2503.04957

    work page arXiv 2025

  42. [42]

    Maria Wang, Srinivas Sunkara, Gilles Baechler, Jason Lin, Yun Zhu, Fedir Zubach, Lei Shu, and Jindong Chen. 2024. WebQuest: A benchmark for multimodal QA on web page sequences. arXiv preprint arXiv:2409.13711

    work page arXiv 2024

  43. [43]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. InAdvances in , , Nguyen et al., Tin Nguyen, Thang T. Truong, Runtao Zhou, Trung Bui, Chirag Agarwal, and Anh Totti Nguyen Neural Information Processing Systems, S. K...

    2022

  44. [44]

    Wikipedia contributors. 2026. List of most-visited websites — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/wiki/List_of_most-visited_websites. Accessed: 2026-04-01

    2026

  45. [45]

    Chen Xiang, Yuchen Zeng, Xiaofei Wang, et al. 2025. CoW Pilot: A Framework for Human-in-the-Loop Web Agents. InProceedings of the 2025 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). Association for Computational Linguistics, Albuquerque, New Mexico, 61–72. doi:10.18653/v1/2025.naacl-demo.6

    work page doi:10.18653/v1/2025.naacl-demo.6 2025

  46. [46]

    Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. 2025. An Illusion of Progress? Assessing the Current State of Web Agents. arXiv:2504.01382 [cs.AI] https://arxiv.org/abs/2504.01382

    work page arXiv 2025

  47. [47]

    Roosta, and Tianmin Shu

    Suyu Ye, Haojun Shi, Darren Shih, Hyokun Yun, Tanya G. Roosta, and Tianmin Shu. 2026. RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users.Proceedings of the AAAI Conference on Artificial Intelligence40, 40 (Mar. 2026), 34441–34449. doi:10.1609/aaai.v40i40.40742

    work page doi:10.1609/aaai.v40i40.40742 2026

  48. [48]

    Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. 2024. Agent-SafetyBench: Evaluating the safety of LLM agents. arXiv preprint arXiv:2412.14470

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  49. [49]

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  50. [50]

    Runtao Zhou, Giang Nguyen, Nikita Kharya, Anh Nguyen, and Chirag Agarwal

  51. [51]

    InProceedings of the 31st International Conference on Intelligent User Interfaces (IUI ’26)

    Improving Human Verification of LLM Reasoning through Interactive Explanation Interfaces. InProceedings of the 31st International Conference on Intelligent User Interfaces (IUI ’26). Association for Computing Machinery, New York, NY, USA, 456–473. doi:10.1145/3742413.3789134

    work page doi:10.1145/3742413.3789134

  52. [52]

    WebArena: A Realistic Web Environment for Building Autonomous Agents

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv:2307.13854 [cs.AI] https://arxiv.org/abs/2307.13854 PageGuide: Browser extension to assist users in navigating a webp...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  53. [53]

    guide" - For step-by-step

    "guide" - For step-by-step "how to" questions that need interactive guidance

  54. [54]

    "hide" - For requests to hide, remove, or suppress distracting/annoying content (ads, banners, popups, cookie notices, sidebars, recommendations, etc.)

  55. [55]

    image_find

    "image_find" - For questions about an UPLOADED IMAGE (finding similar items, comparing with page content)

  56. [56]

    pdf_find

    "pdf_find" - For questions about PDF documents (summarize, find specific content, extract info from PDFs)

  57. [57]

    find" - For questions, information lookup, finding content, highlighting elements (DEFAULT) ROUTING RULES: -

    "find" - For questions, information lookup, finding content, highlighting elements (DEFAULT) ROUTING RULES: - "guide": User wants to LEARN how to do something in steps (e.g.,"how do I report this video?", "where can I find settings?", "help me delete my account") - "hide": User wants to hide or remove something on the page (e.g.,"hide the ads", "remove th...

  58. [58]

    Answer the question based on the page content if possible

  59. [59]

    text"] citations inline to reference specific elements from the PAGE INDEX - N is the index number from PAGE INDEX -

    If the page content has the answer, use [N:"text"] citations inline to reference specific elements from the PAGE INDEX - N is the index number from PAGE INDEX - "text" is the EXACT text snippet to highlight (copy from the page content)

  60. [60]

    Each citation should point to an element that supports that part of your answer

  61. [61]

    For lists of items, cite each one with the specific text to highlight

  62. [62]

    Use ONE citation per item (if same text has multiple indices, pick the link)

  63. [63]

    The "text" should be a short, specific phrase (not the entire element text)

  64. [64]

    Consider conversation history for context, but always answer based on CURRENT page content

  65. [65]

    Wikipedia’s [1], [2], [3]) — only use [N:"text"] format where N comes from the PAGE INDEX above

    NEVER reproduce existing footnote markers from the webpage itself (e.g. Wikipedia’s [1], [2], [3]) — only use [N:"text"] format where N comes from the PAGE INDEX above

  66. [66]

    The information is not provided on this page

    **CRITICAL**: If the information is NOT provided on this page: - State exactly: "The information is not provided on this page. " - Then, providing the answer using your own general knowledge base is HIGHLY ENCOURAGED. Do not simply stop after stating it is not on the page. - You MUST include citations to real, valid source URLs using STANDARD MARKDOWN LIN...

  67. [67]

    PAGE INDEX - Visible elements on the page

  68. [68]

    USER QUESTION - What the user wants to do

  69. [69]

    STEP NUMBER - Current step (1 = first step)

  70. [70]

    " or

    PREVIOUS STEPS - What was done before (if any) Your job: Guide the user ONE STEP at a time. IMPORTANT CONCEPTS: - Some buttons/options are HIDDEN in menus (like "... " or " ..." three-dot menus) - If the target isn’t visible, guide user to open the menu FIRST - Common hidden locations: dropdown menus, "More" buttons, three-dot menus, right-click menus, se...

  71. [71]

    ONE step at a time - don’t overwhelm the user

  72. [72]

    If target is likely hidden in a menu, first step should open that menu

  73. [73]

    waitFor":

    Use "waitFor": "click" when user needs to click something

  74. [74]

    isLastStep

    Set "isLastStep": true only when the goal is achieved

  75. [75]

    Make instructions clear and specific

  76. [76]

    Highlight the element user needs to interact with EXAMPLES: PAGE INDEX:

  77. [77]

    How do I report this video?

    (button) Save Q: "How do I report this video?" (Step 1) → { "step":1, "instruction":"Click the three-dot menu ( ...) to see more options", "highlight":"index":5, "text":" ...", "wait- For":"click", "isLastStep":false, "nextStepHint":"The menu will open with Report option" } Q: "How do I report this video?" (Step 2, after menu opened) PAGE INDEX now shows:...

    work page arXiv