pith. sign in

arxiv: 2606.30697 · v1 · pith:UU7FIMJFnew · submitted 2026-06-29 · 💻 cs.OS · cs.AI· cs.CV

LUMOS: A Semantic Operating-System Layer for Accessibility-Grounded AI Agents

Yogeswar Reddy Thota This is my paper

Pith reviewed 2026-07-01 01:52 UTC · model grok-4.3

classification 💻 cs.OS cs.AIcs.CV
keywords accessibility metadataAI agentssemantic UIoperating systemsLLM interactionUI automationmachine-readable semanticsobserve-act loop
0
0 comments X

The pith

LUMOS converts accessibility metadata into semantic blueprints so AI agents can observe and act on UI elements without screenshots.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current operating systems present interfaces through pixels and visuals that suit humans but create high costs and ambiguity for AI agents. LUMOS addresses this by turning native accessibility metadata and browser structures into compact, machine-readable semantic blueprints that list stable identifiers, roles, names, values, bounds, and action options. Language models then follow an observe-act loop that issues grounded commands on these visible UI primitives. The approach matters because it lowers token usage, removes coordinate uncertainty, and enables consistent operation across applications without writing per-app scripts. It frames existing accessibility support as the basis for more direct machine interaction with computers.

Core claim

LUMOS converts native accessibility metadata and browser UI structures into machine readable semantic blueprints with stable identifiers, roles, names, values, bounds, and action affordances. It also supports live semantic pointer grounding by querying the UI element under or near the cursor through operating-system automation APIs. An LLM then acts through an accessibility grounded observe act loop using constrained visible-UI primitives rather than application-specific scripts. LUMOS does not claim to replace visual agents; instead, it reduces dependence on screenshots when operating systems already provide semantic structure.

What carries the argument

LUMOS semantic interaction layer that converts accessibility metadata into machine-readable blueprints carrying stable identifiers and action affordances.

If this is right

  • AI agents incur lower token costs because they process structured semantic data instead of image pixels or OCR output.
  • Action reliability increases through explicit element roles, bounds, and affordances rather than visual interpretation.
  • Agents achieve cross-application consistency by using OS-provided primitives instead of writing application-specific code.
  • The layer supplies a concrete route toward operating systems that expose machine-readable interaction as a native feature.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar semantic layers could be built for mobile platforms that already expose accessibility APIs.
  • A hybrid system could fall back to visual methods only when accessibility metadata is absent for a given element.
  • Wider use of the layer would create market pressure for developers to make accessibility metadata more complete.
  • The observe-act pattern could transfer to other domains such as robotic control where sensor data needs semantic grounding.

Load-bearing premise

Operating systems provide sufficient, complete, and reliable accessibility metadata across applications and browsers to support reliable AI agent operation without fallback to visual methods.

What would settle it

Implement LUMOS on a standard desktop and run agents on common applications such as web browsers and office software; count the fraction of tasks that fail due to missing, incomplete, or inconsistent accessibility data.

Figures

Figures reproduced from arXiv: 2606.30697 by Yogeswar Reddy Thota.

Figure 1
Figure 1. Figure 1: Conceptual motivation. LUMOS converts a human-facing visual [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: End-to-end LUMOS data-flow pipeline. A user command is converted into semantic observations from UIA, DOM/accessibility trees, system state, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Live semantic pointer grounding. LUMOS maps a physical pointer [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Notepad blueprint extraction from the debug traces. The visible [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Diagnostic evidence extracted from the Notepad debug logs. Early runs showed literal instruction copying, append-style corrections, repeated fragments, [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Diagnostic counts extracted from the pasted Notepad development [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
read the original abstract

Current operating systems expose interfaces optimized for human users but not for AI agents. Humans benefit from pixels, icons, windows, visual grouping, mouse movement, and keyboard shortcuts; AI agents instead need compact semantic state, grounded actions, and reliable feedback. As a result, many computer-use agents are forced to interpret screenshots, OCR output, and visual crops, introducing high token costs, visual ambiguity, latency, and coordinate uncertainty. This paper introduces LUMOS (Language Model Unified Machine-Readable Operating-System Semantics), a semantic interaction layer between AI agents and operating systems. LUMOS converts native accessibility metadata and browser UI structures into machine readable semantic blueprints with stable identifiers, roles, names, values, bounds, and action affordances. It also supports live semantic pointer grounding by querying the UI element under or near the cursor through operating-system automation APIs. An LLM then acts through an accessibility grounded observe act loop using constrained visible-UI primitives rather than application-specific scripts. LUMOS does not claim to replace visual agents; instead, it reduces dependence on screenshots when operating systems already provide semantic structure. These results suggest a path toward AI-native operating systems and machine-readable interaction layers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper introduces LUMOS, a semantic interaction layer that converts native OS accessibility metadata and browser UI structures into machine-readable semantic blueprints containing stable identifiers, roles, names, values, bounds, and action affordances. It enables an LLM-driven observe-act loop using constrained visible-UI primitives and live semantic pointer grounding via OS automation APIs, with the goal of reducing token costs, latency, visual ambiguity, and coordinate uncertainty compared to screenshot/OCR-based agents. The approach is positioned as complementary to visual methods, applying only where semantic structure is already present, and is claimed to suggest a path toward AI-native operating systems.

Significance. If the architectural claims hold and the accessibility metadata proves sufficiently complete and stable, LUMOS could meaningfully lower the cost and error rate of computer-use agents by replacing pixel-based interpretation with grounded semantic primitives. The proposal is a clean systems-level idea that directly addresses a practical pain point in current LLM agents, but its significance remains prospective because the manuscript contains no empirical validation, coverage statistics, or cross-application measurements.

major comments (3)
  1. [Abstract] Abstract: The statements that LUMOS 'reduces dependence on screenshots,' 'reduces token costs, latency, and ambiguity,' and enables reliable operation 'when operating systems already provide semantic structure' are presented as results, yet the manuscript supplies no quantitative evaluation, token-count comparisons, latency measurements, success-rate metrics, or coverage analysis across applications to support these claims.
  2. [Abstract] Abstract and introduction: The central premise that native accessibility trees are 'sufficiently complete, consistent, and stable' to support an observe-act loop without fallback is asserted but left unquantified; no data on failure modes, fraction of UI elements missing semantic metadata, or cross-app reliability is provided, rendering the practical scope of the proposal indeterminate.
  3. [Abstract] Abstract: The claim that the system 'supports live semantic pointer grounding by querying the UI element under or near the cursor' is described at a high level but lacks any specification of the OS APIs used, error-handling strategy when the query returns incomplete data, or how the resulting blueprint is serialized for the LLM.
minor comments (1)
  1. The abstract refers to 'these results' but the manuscript appears to be an architectural proposal without an evaluation section; clarifying whether the work is intended as a position paper or a systems contribution with forthcoming experiments would help readers.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive review. The comments correctly identify that the abstract presents design objectives in language that could be read as reporting completed results. We address each point below and will revise the manuscript accordingly to ensure claims accurately reflect the conceptual nature of the proposal.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The statements that LUMOS 'reduces dependence on screenshots,' 'reduces token costs, latency, and ambiguity,' and enables reliable operation 'when operating systems already provide semantic structure' are presented as results, yet the manuscript supplies no quantitative evaluation, token-count comparisons, latency measurements, success-rate metrics, or coverage analysis across applications to support these claims.

    Authors: We agree that the abstract phrasing implies empirical outcomes. The manuscript is a systems proposal describing an architecture and does not contain evaluations. We will revise the abstract to use prospective language (e.g., 'is designed to reduce dependence on screenshots' and 'suggests a path toward AI-native operating systems') and remove any implication of measured results. revision: yes

  2. Referee: [Abstract] Abstract and introduction: The central premise that native accessibility trees are 'sufficiently complete, consistent, and stable' to support an observe-act loop without fallback is asserted but left unquantified; no data on failure modes, fraction of UI elements missing semantic metadata, or cross-app reliability is provided, rendering the practical scope of the proposal indeterminate.

    Authors: The manuscript already qualifies applicability to settings 'when operating systems already provide semantic structure' and states that LUMOS is complementary to visual methods. We acknowledge that the abstract does not sufficiently foreground variability in accessibility metadata. We will add an explicit limitations paragraph discussing known incompleteness of accessibility trees and the need for fallback mechanisms, drawing on existing platform documentation. revision: partial

  3. Referee: [Abstract] Abstract: The claim that the system 'supports live semantic pointer grounding by querying the UI element under or near the cursor' is described at a high level but lacks any specification of the OS APIs used, error-handling strategy when the query returns incomplete data, or how the resulting blueprint is serialized for the LLM.

    Authors: The abstract is intentionally high-level. The full manuscript describes the observe-act loop; we will expand the abstract with a brief reference to the relevant platform APIs (UI Automation, Accessibility frameworks) and add a short implementation subsection covering error handling for incomplete queries and the JSON serialization format used for blueprints. revision: yes

Circularity Check

0 steps flagged

No circularity: purely architectural proposal with no derivations or self-referential fits

full rationale

The paper contains no equations, fitted parameters, predictions, or mathematical derivations. Its central claim is an architectural description of converting existing OS accessibility metadata into semantic blueprints, explicitly conditioned on cases where such metadata is already present. No self-citation chains, uniqueness theorems, or ansatzes are invoked to justify the core mechanism. The reader's assessment of score 0.0 is confirmed by direct inspection of the provided text.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The proposal relies on the existence and completeness of OS accessibility APIs but introduces no new mathematical axioms, free parameters, or invented entities.

axioms (1)
  • domain assumption Operating systems provide accessible metadata for UI elements that is sufficiently complete and accurate for AI agent use
    Central to the conversion into semantic blueprints and the observe-act loop.

pith-pipeline@v0.9.1-grok · 5737 in / 1168 out tokens · 43966 ms · 2026-07-01T01:52:35.177252+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments,

    T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y . Liu, Y . Xu, S. Zhou, S. Savarese, C. Xiong, V . Zhong, and T. Yu, “OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments,” 2024

    2024

  2. [2]

    WebArena: A realistic web environment for building autonomous agents,

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Fried, U. Alon, and G. Neubig, “WebArena: A realistic web environment for building autonomous agents,” 2023

    2023

  3. [3]

    WindowsWorld: A process-centric benchmark of autonomous gui agents in professional cross-application environments,

    J. Li, Y . Li, C. Zhao, Z. Xu, B. Hu, and M. Zhang, “WindowsWorld: A process-centric benchmark of autonomous gui agents in professional cross-application environments,” 2026

    2026

  4. [4]

    UI Automation Overview - Win32 apps,

    Microsoft, “UI Automation Overview - Win32 apps,” https://learn.microsoft.com/en-us/windows/win32/winauto/ uiauto-uiautomationoverview, accessed: 2026-06-16

    2026

  5. [5]

    UI Automation Tree Overview - Win32 apps,

    Microsoft, “UI Automation Tree Overview - Win32 apps,” https://learn. microsoft.com/en-us/windows/win32/winauto/uiauto-treeoverview, ac- cessed: 2026-06-16

    2026

  6. [6]

    UI Automation - Win32 apps,

    Microsoft, “UI Automation - Win32 apps,” https://learn.microsoft.com/ en-us/windows/win32/winauto/entry-uiauto-win32, accessed: 2026-06- 16

    2026

  7. [7]

    ReAct: Synergizing Reasoning and Acting in Language Models

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “ReAct: Synergizing reasoning and acting in language mod- els,” inInternational Conference on Learning Representations, 2023, arXiv:2210.03629

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    Mind2Web: Towards a generalist agent for the web,

    X. Deng, Y . Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y . Su, “Mind2Web: Towards a generalist agent for the web,” 2023

    2023

  9. [9]

    Introducing operator,

    OpenAI, “Introducing operator,” https://openai.com/index/ introducing-operator/, 2025, accessed: 2026-06-16

    2025

  10. [10]

    Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku,

    Anthropic, “Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku,” https://www.anthropic.com/news/ 3-5-models-and-computer-use, 2024, accessed: 2026-06-16

    2024

  11. [11]

    OmniParser for pure vision based GUI agent,

    Y . Lu, J. Yang, Y . Shen, and A. Awadallah, “OmniParser for pure vision based GUI agent,” 2024

    2024

  12. [12]

    ScreenAI: A vision- language model for UI and infographics understanding,

    G. Baechler, S. Sunkara, M. Wang, F. Zubach, H. Mansoor, V . Etter, V . Carbune, J. Lin, J. Chen, and A. Sharma, “ScreenAI: A vision- language model for UI and infographics understanding,” 2024

    2024

  13. [13]

    UI-TARS: Pioneering automated GUI interaction with native agents,

    Y . Qin, Y . Ye, J. Fang, H. Wang, S. Liang, S. Tian, J. Zhang, J. Li, Y . Li, S. Huang, W. Zhong, K. Li, J. Yang, Y . Miao, W. Lin, L. Liu, X. Jiang, Q. Ma, J. Li, X. Xiao, K. Cai, C. Li, Y . Zheng, C. Jin, C. Li, X. Zhou, M. Wang, H. Chen, Z. Li, H. Yang, H. Liu, F. Lin, T. Peng, X. Liu, and G. Shi, “UI-TARS: Pioneering automated GUI interaction with na...

    2025

  14. [14]

    Agent S: An open agentic framework that uses computers like a human,

    S. Agashe, J. Han, S. Gan, J. Yang, A. Li, and X. E. Wang, “Agent S: An open agentic framework that uses computers like a human,” 2024

    2024

  15. [15]

    AutoGen: Enabling next-gen LLM applications via multi- agent conversation,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang, “AutoGen: Enabling next-gen LLM applications via multi- agent conversation,” 2023

    2023

  16. [16]

    Robotic process automation,

    W. M. P. van der Aalst, M. Bichler, and A. Heinzl, “Robotic process automation,”Business & Information Systems Engineering, vol. 60, no. 4, pp. 269–272, 2018

    2018