Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining

Bowen Ye; Feifan Song; Hao Tian; Lei Li; Shuhao Gu; Sujian Li; Weimin Xiong; Zihao Yue

arxiv: 2605.14747 · v1 · pith:WUMGFHF7new · submitted 2026-05-14 · 💻 cs.CL · cs.AI· cs.CV· cs.LG

Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining

Weimin Xiong , Shuhao Gu , Bowen Ye , Zihao Yue , Lei Li , Feifan Song , Sujian Li , Hao Tian This is my paper

Pith reviewed 2026-06-30 20:58 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.CVcs.LG

keywords GUI agentsvideo-to-trajectory synthesismultimodal pretrainingWildGUI datasetautomated data generationgraphical user interfacesinteraction trajectoriesagent generalization

0 comments

The pith

Video2GUI extracts 12 million GUI interaction trajectories from unlabeled videos to pretrain agents.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Video2GUI, an automated framework that identifies GUI tutorial videos from internet sources and converts them into structured interaction trajectories. Processing 500 million video metadata entries produces the WildGUI dataset spanning over 1,500 applications and websites. Pre-training multimodal models on this data delivers 5-20% gains on GUI grounding and action benchmarks. The approach targets the scarcity of large, diverse training data that currently limits how well GUI agents generalize beyond narrow domains.

Core claim

Video2GUI is a fully automated framework that employs a coarse-to-fine filtering strategy to identify high-quality GUI tutorial videos from unlabeled internet videos and converts their visual content into structured agent trajectories. Applying the pipeline to 500 million video metadata entries yields the WildGUI dataset containing 12 million interaction trajectories across more than 1,500 applications and websites. Pre-training Qwen2.5-VL and Mimo-VL on WildGUI produces consistent improvements of 5-20% across multiple GUI grounding and action benchmarks, matching or surpassing state-of-the-art performance.

What carries the argument

The coarse-to-fine filtering strategy within Video2GUI that selects GUI tutorial videos and structures their content into accurate interaction trajectories for large-scale pretraining.

If this is right

GUI agents can be pretrained on trajectories from thousands of diverse real-world applications without manual annotation.
Existing multimodal models gain measurable improvements on grounding and action tasks after exposure to the dataset.
The released dataset and pipeline support continued scaling of training data for generalized agents.
Performance gains appear consistent across the two model families tested in the experiments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same video-to-trajectory approach could supply training data for agents in related interactive domains such as web navigation or mobile robotics.
Mixing the automatically generated trajectories with smaller manually annotated sets might produce further gains.
Future refinements could add verification steps to reduce any residual noise in the extracted sequences.

Load-bearing premise

The filtering strategy can reliably select high-quality videos and convert their content into accurate, structured trajectories without introducing substantial noise or systematic errors.

What would settle it

Pre-training the same models on WildGUI and measuring no improvement or a performance drop on the GUI benchmarks, or manual inspection revealing high rates of incorrect action sequences in the extracted trajectories.

Figures

Figures reproduced from arXiv: 2605.14747 by Bowen Ye, Feifan Song, Hao Tian, Lei Li, Shuhao Gu, Sujian Li, Weimin Xiong, Zihao Yue.

**Figure 1.** Figure 1: Overview of the VideoGUI pipeline. VideoGUI consists of three stages: (A) coarse-to-fine video filtering for selecting high-quality tutorial videos, (B) trajectory extraction that converts video segments into instruction–trajectories sequences, and (C) action spatial grounding that maps low-level instructions to precise UI targets to produce grounded trajectories. (e.g., click, type, scroll), and bt denote… view at source ↗

**Figure 3.** Figure 3: Impact of scaling pre-training data on performance. OSWorld-G, comparing against the ”Stage 2 Only” baseline post-trained exclusively on open-source data. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Figure 4: Average scores from human evaluation. language instructions. Building on these foundations, recent works (Qin et al., 2025; Yan et al., 2025; Zeng et al., 2025) have achieved state-of-the-art performance through innovations in training strategies and agent design. UI-TARS (Qin et al., 2025) develops a native end-to-end GUI agent via multi-stage post-training, enhancing perception, action and reasoning cap… view at source ↗

**Figure 5.** Figure 5: Performance of the video scoring model on the test set across three quality dimensions. When training Qwen2.5-Omni as the video scoring model, we similarly prioritize inference efficiency. We attach three regression heads to the final hidden layer of Qwen2.5-Omni, corresponding to the three scoring dimensions. The model is 12 [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗

**Figure 6.** Figure 6: Dataset statistics. Distribution of (a) platforms, (b) software categories, and (c) website categories in WildGUI. 0m 2m 4m 6m 8m 10m 12m Duration (minutes) 0 100k 200k 300k 400k 500k 600k 700k 800k Number of Videos (a) Video Duration Distribution 0 10 20 30 40 50 60 Steps per Trajectory 0 200k 400k 600k 800k 1.0M 1.2M Count (b) Trajectory Length Distribution 0 10.0M 20.0M 30.0M 40.0M 50.0M Count wait doub… view at source ↗

**Figure 7.** Figure 7: Dataset statistics of WildGUI: (a) video duration distribution; (b) distribution of steps per trajectory; and the action types for (c) desktop and (d) mobile environments. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗

**Figure 8.** Figure 8: WildGUI example. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_8.png] view at source ↗

**Figure 9.** Figure 9: WildGUI example. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_9.png] view at source ↗

**Figure 10.** Figure 10: WildGUI example. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_10.png] view at source ↗

**Figure 11.** Figure 11: WildGUI example. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_11.png] view at source ↗

read the original abstract

Recent advances in multimodal large language models have driven growing interest in graphical user interface (GUI) agents, yet their generalization remains constrained by the scarcity of large-scale training data spanning diverse real-world applications. Existing datasets rely heavily on costly manual annotations and are typically confined to narrow domains. To address this challenge, we propose Video2GUI, a fully automated framework that extracts grounded GUI interaction trajectories directly from unlabeled Internet videos. Video2GUI employs a coarse-to-fine filtering strategy to identify high-quality GUI tutorial videos and convert them into structured agent trajectories. Applying this pipeline to 500 million video metadata entries, we construct WildGUI, a large-scale dataset containing 12 million interaction trajectories spanning over 1,500 applications and websites. Pre-training Qwen2.5-VL and Mimo-VL on WildGUI yields consistent improvements of 5-20% across multiple GUI grounding and action benchmarks, matching or surpassing state-of-the-art performance. We will release both the WildGUI dataset and the Video2GUI pipeline to support future research of GUI agents.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper gives a scalable video-mining pipeline for GUI trajectories and a 12M-example dataset that lifts two VL models on agent benchmarks, but the accuracy of the extracted data is not shown in enough detail.

read the letter

The core new piece is Video2GUI, which runs a coarse-to-fine filter over 500 million video metadata entries to pull out GUI tutorial clips and convert them into structured trajectories. The result is WildGUI, 12 million trajectories covering more than 1,500 applications and sites. That scale is bigger than the manually labeled sets people have been using, and the authors release both the data and the pipeline.

They then pretrain Qwen2.5-VL and Mimo-VL on it and report 5-20% gains on several grounding and action benchmarks, sometimes matching or beating prior numbers. The automation itself is the practical advance here; it sidesteps the cost of hand annotation and opens a route to more diverse real-world coverage.

The main gap is evidence on trajectory quality. The abstract describes the filtering steps but gives no error rates, no human agreement numbers on the extracted actions, and no ablation that removes low-confidence samples to check whether the gains survive cleaner data. Without those checks it is hard to know how much noise is in the set or whether the reported lifts are robust. The citation pattern looks standard for the area, but the experimental section will need to carry more weight than the abstract suggests.

This is for groups working on GUI agents who are bottlenecked by data volume rather than by model architecture. A reader who needs large-scale interaction examples will get concrete value from the released dataset even if the validation story needs tightening. The work is coherent on its own terms and shows clear thinking about the data problem, so it deserves a serious referee.

I would send it to review and ask specifically for quantitative validation of the trajectories and for ablations on data quality.

Referee Report

2 major / 2 minor

Summary. The paper introduces Video2GUI, an automated coarse-to-fine pipeline that mines unlabeled internet videos to extract structured GUI interaction trajectories, yielding the WildGUI dataset of 12 million trajectories across >1,500 applications. It reports that pre-training Qwen2.5-VL and Mimo-VL on WildGUI produces 5-20% gains on GUI grounding and action benchmarks, reaching or exceeding prior state-of-the-art.

Significance. A scalable, annotation-free source of diverse real-world GUI trajectories would address a central bottleneck for GUI agents. If the extracted trajectories prove sufficiently accurate, the reported gains would indicate a practical route to improved generalization; the release of both dataset and pipeline would further amplify impact.

major comments (2)

[Abstract, §4] Abstract and §4 (dataset construction): the claim that the coarse-to-fine filtering produces high-quality trajectories rests on an unquantified assumption; no human agreement rates, extraction error rates, or ablation removing low-confidence samples are supplied to anchor the 5-20% downstream gains.
[§5] §5 (experiments): baseline comparisons and statistical tests for the reported improvements are not described; it is therefore unclear whether the gains are robust to different random seeds, trajectory noise levels, or alternative filtering thresholds.

minor comments (2)

Notation for trajectory elements (e.g., action types, grounding coordinates) should be defined once in a table or figure caption rather than repeated inline.
[§3] The 500-million metadata entry figure would benefit from a brief breakdown of how many videos survive each filtering stage.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and commit to revisions that strengthen the validation of trajectory quality and experimental reporting.

read point-by-point responses

Referee: [Abstract, §4] Abstract and §4 (dataset construction): the claim that the coarse-to-fine filtering produces high-quality trajectories rests on an unquantified assumption; no human agreement rates, extraction error rates, or ablation removing low-confidence samples are supplied to anchor the 5-20% downstream gains.

Authors: We agree that the manuscript would benefit from direct quantification of trajectory quality. The original submission did not include human evaluation metrics or ablations on the filtering stages. In the revision we will add a human study on a sampled subset of trajectories reporting inter-annotator agreement and extraction error rates, together with an ablation that removes low-confidence samples and measures the resulting change in downstream pre-training gains. revision: yes
Referee: [§5] §5 (experiments): baseline comparisons and statistical tests for the reported improvements are not described; it is therefore unclear whether the gains are robust to different random seeds, trajectory noise levels, or alternative filtering thresholds.

Authors: We acknowledge the need for more detailed experimental reporting. The revised §5 will expand the description of all baselines, report results averaged across multiple random seeds with standard deviations, and include statistical significance tests. We will also add supplementary experiments that vary trajectory noise levels and filtering thresholds to demonstrate robustness of the observed gains. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper constructs WildGUI via an automated Video2GUI pipeline applied to external video metadata, then measures downstream pretraining gains on separate GUI grounding and action benchmarks. No equations, fitted parameters, or claims reduce to self-definition or self-citation by construction; the central result is an empirical delta on held-out test sets after dataset creation.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that unlabeled internet videos contain extractable, high-quality GUI interactions that can be automatically converted into usable agent trajectories at scale.

axioms (1)

domain assumption Internet videos contain sufficient high-quality GUI interaction examples that can be automatically identified and structured into agent trajectories.
This assumption underpins the entire coarse-to-fine filtering and conversion process described in the abstract.

pith-pipeline@v0.9.1-grok · 5742 in / 1182 out tokens · 31760 ms · 2026-06-30T20:58:42.285536+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories
cs.CL 2026-06 conditional novelty 7.0

ISE creates 23,132 execution-grounded multi-turn OS agent trajectories via intent simulation and live execution, improving agent performance on ClawEval from 19.3 to 37.7 pass@1 with Qwen3-8B.

Reference graph

Works this paper leans on

20 extracted references · 1 canonical work pages · cited by 1 Pith paper · 1 internal anchor

[1]

GTA1: GUI Test-time Scaling Agent

URL https://api.semanticscholar. org/CorpusID:274859421. Yang, Y ., Li, D., Dai, Y ., Yang, Y ., Luo, Z., Zhao, Z., Hu, Z., Huang, J., Saha, A., Chen, Z., et al. Gta1: Gui test-time scaling agent.arXiv preprint arXiv:2507.05791, 2025. Ye, J., Zhang, X., Xu, H., Liu, H., Wang, J., Zhu, Z., Zheng, Z., Gao, F., Cao, J., Lu, Z., et al. Mobile-agent-v3: Fundam...

work page internal anchor Pith review Pith/arXiv arXiv 2025
[2]

**Screen Recordings: ** Direct capture of a computer, tablet, or smartphone screen
[3]

**Software Interaction: ** Users interacting with graphical user interfaces (e.g ., clicking, typing, navigating menus, using tools)
[4]

Not Relevant

**Application Demos/Tutorials: ** Walkthroughs of software (e.g., Photoshop, Excel, VS Code, Web Browsers, OS settings). 13 14A video is considered "Not Relevant" if it is: 15- Real-world camera footage (Vlogs, IRL, nature, people talking to camera without screen share). 16- Gaming content (unless it is a specific tutorial on UI/Settings). 17- Static slid...

2024
[5]

Shot Splitting

mm:ss - mm:ss 24... 25(Split as granularly as possible.) 26 27You must strictly confine your analysis to the provided video segment ONLY. Do NOT generate any content for timestamps beyond the end of this provided segment. 28 29## Part 2: Task Annotations 30Second, after the "Shot Splitting" section, you must output a single JSON list [{...}, {...}]. 31Eac...
[6]

Shot Segmentation

**Use Absolute Timestamps: ** All timestamps in your output (e.g., "Shot Segmentation") must be **absolute** (relative to the *full* video), not relative to the current clip
[7]

Annotation History

**Maintain Consistency: ** Your new analysis must be fully **consistent** with the "Annotation History". Ensure that your interpretations, character analyses, and narrative arcs logically continue from the previous annotations
[8]

Annotation History

**Use Global Context: ** You must use the "Annotation History" to understand the **global context ** (e.g., established narrative, characters, themes) to better interpret the events within the current clip. You are encouraged to cite specific evidence from the history (e.g., "which mirrors the action at 04:30") to enhance your interpretation
[9]

Annotation History

**Do Not Mention Text in History: ** The "Annotation History" text is provided * only* for your internal context. Your output must **never** mention the existence of this text (e.g., do not write "Based on the history provided..." or "In the previous annotation..."). Your analysis must read as a single, seamless continuation of the history, as if you were...
[10]

If the last sub-task is already complete, start a fresh sub-task with the next available index

**Continue unfinished tasks: ** If the last sub-task in the preceding part is incomplete, retain its task index and finish every remaining clip within that sub-task. If the last sub-task is already complete, start a fresh sub-task with the next available index. 22 23A more detailed task instruction for the current **[Current Start Time] - [Current End Tim...
[11]

10 * Determine if the target element mentioned in the instruction (e.g., button, text field, icon) is **actually present and visible ** in the current ‘ screenshot‘

**Feasibility Check: ** 8 9 * Analyze the ‘screenshot‘ and ‘grounding_instruction‘. 10 * Determine if the target element mentioned in the instruction (e.g., button, text field, icon) is **actually present and visible ** in the current ‘ screenshot‘. 11
[12]

feasible

**Grounding & Prediction: ** 13 14 * ** If not feasible ** (e.g., the ’Submit’ button mentioned in the instruction does not exist in the screenshot), set ‘feasible‘ to ‘false‘ in the 24 Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining output and provide a reason. 15 * ** If feasible ** (the target element ...
[13]

The center points must use relative coordinates (0-1000) and be formatted as follows: ‘<point>y x</point>‘
[14]

Prompt 6: Prompt for GUI Grounding

The bounding boxes must use relative coordinates (0-1000) and be formatted as follows: ‘<bbox>y1 x1 y2 x2</bbox>‘. Prompt 6: Prompt for GUI Grounding
[15]

{}\". Output a JSON in the format [{{\

Locate UI components that match the command: \"{}\". Output a JSON in the format [{{\"point\": [...], \"label\": \"{{the_whole_command}}\"}}, ...]. 2
[16]

{}\". Output a JSON in the format [{{\

Locate UI components that match the command: \"{}\". Output a JSON in the format [{{\"bbox_2d\": [...], \"label\": \"{{the_whole_command}}\"}}, ...]. Prompt 7: Prompt for GUI Action Prediction
[17]

action\": \

You are a GUI agent. You will be provided with a screenshot, a goal, and your action history. You need to perform the next action to complete the task.\n\n## Action Space\n{}\n\n## Goal\n{}\n\n## Previous Actions\n{}\n\nNow, output the next action in json format [{{\"action\": \"{{action_name}}\"}}, ...]. 2
[18]

action\": \

You are a GUI agent. You will be provided with a screenshot, a goal, and your action history. You need to analyze the current situation and perform the next action to complete the task.\n\n## Action Space\n{}\n\n## Goal\n{}\n\n## Previous Actions\n{}\n\nPlease output your response in the following format:\n\ nThought: <your reasoning about what to do next...
[19]

{}\", perform the next action to complete the task. Output a JSON in the format [{{\

Based on the screenshot and the goal: \"{}\", perform the next action to complete the task. Output a JSON in the format [{{\"action\": \"{{action_name }}\"}}, ...]. 2
[20]

action\": \

Based on the screenshot and the goal: \"{}\", analyze the current situation and perform the next action to complete the task.\n\nPlease output your response in the following format:\n\nThought: <your reasoning about what to do next>\ nAction: <the action you will take>\n\n‘‘‘json\n[{{\"action\": \"{{action_name }}\"}}, ...]\n‘‘‘ 26 Video2GUI: Synthesizing...

[1] [1]

GTA1: GUI Test-time Scaling Agent

URL https://api.semanticscholar. org/CorpusID:274859421. Yang, Y ., Li, D., Dai, Y ., Yang, Y ., Luo, Z., Zhao, Z., Hu, Z., Huang, J., Saha, A., Chen, Z., et al. Gta1: Gui test-time scaling agent.arXiv preprint arXiv:2507.05791, 2025. Ye, J., Zhang, X., Xu, H., Liu, H., Wang, J., Zhu, Z., Zheng, Z., Gao, F., Cao, J., Lu, Z., et al. Mobile-agent-v3: Fundam...

work page internal anchor Pith review Pith/arXiv arXiv 2025

[2] [2]

**Screen Recordings: ** Direct capture of a computer, tablet, or smartphone screen

[3] [3]

**Software Interaction: ** Users interacting with graphical user interfaces (e.g ., clicking, typing, navigating menus, using tools)

[4] [4]

Not Relevant

**Application Demos/Tutorials: ** Walkthroughs of software (e.g., Photoshop, Excel, VS Code, Web Browsers, OS settings). 13 14A video is considered "Not Relevant" if it is: 15- Real-world camera footage (Vlogs, IRL, nature, people talking to camera without screen share). 16- Gaming content (unless it is a specific tutorial on UI/Settings). 17- Static slid...

2024

[5] [5]

Shot Splitting

mm:ss - mm:ss 24... 25(Split as granularly as possible.) 26 27You must strictly confine your analysis to the provided video segment ONLY. Do NOT generate any content for timestamps beyond the end of this provided segment. 28 29## Part 2: Task Annotations 30Second, after the "Shot Splitting" section, you must output a single JSON list [{...}, {...}]. 31Eac...

[6] [6]

Shot Segmentation

**Use Absolute Timestamps: ** All timestamps in your output (e.g., "Shot Segmentation") must be **absolute** (relative to the *full* video), not relative to the current clip

[7] [7]

Annotation History

**Maintain Consistency: ** Your new analysis must be fully **consistent** with the "Annotation History". Ensure that your interpretations, character analyses, and narrative arcs logically continue from the previous annotations

[8] [8]

Annotation History

**Use Global Context: ** You must use the "Annotation History" to understand the **global context ** (e.g., established narrative, characters, themes) to better interpret the events within the current clip. You are encouraged to cite specific evidence from the history (e.g., "which mirrors the action at 04:30") to enhance your interpretation

[9] [9]

Annotation History

**Do Not Mention Text in History: ** The "Annotation History" text is provided * only* for your internal context. Your output must **never** mention the existence of this text (e.g., do not write "Based on the history provided..." or "In the previous annotation..."). Your analysis must read as a single, seamless continuation of the history, as if you were...

[10] [10]

If the last sub-task is already complete, start a fresh sub-task with the next available index

**Continue unfinished tasks: ** If the last sub-task in the preceding part is incomplete, retain its task index and finish every remaining clip within that sub-task. If the last sub-task is already complete, start a fresh sub-task with the next available index. 22 23A more detailed task instruction for the current **[Current Start Time] - [Current End Tim...

[11] [11]

10 * Determine if the target element mentioned in the instruction (e.g., button, text field, icon) is **actually present and visible ** in the current ‘ screenshot‘

**Feasibility Check: ** 8 9 * Analyze the ‘screenshot‘ and ‘grounding_instruction‘. 10 * Determine if the target element mentioned in the instruction (e.g., button, text field, icon) is **actually present and visible ** in the current ‘ screenshot‘. 11

[12] [12]

feasible

**Grounding & Prediction: ** 13 14 * ** If not feasible ** (e.g., the ’Submit’ button mentioned in the instruction does not exist in the screenshot), set ‘feasible‘ to ‘false‘ in the 24 Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining output and provide a reason. 15 * ** If feasible ** (the target element ...

[13] [13]

The center points must use relative coordinates (0-1000) and be formatted as follows: ‘<point>y x</point>‘

[14] [14]

Prompt 6: Prompt for GUI Grounding

The bounding boxes must use relative coordinates (0-1000) and be formatted as follows: ‘<bbox>y1 x1 y2 x2</bbox>‘. Prompt 6: Prompt for GUI Grounding

[15] [15]

{}\". Output a JSON in the format [{{\

Locate UI components that match the command: \"{}\". Output a JSON in the format [{{\"point\": [...], \"label\": \"{{the_whole_command}}\"}}, ...]. 2

[16] [16]

{}\". Output a JSON in the format [{{\

Locate UI components that match the command: \"{}\". Output a JSON in the format [{{\"bbox_2d\": [...], \"label\": \"{{the_whole_command}}\"}}, ...]. Prompt 7: Prompt for GUI Action Prediction

[17] [17]

action\": \

You are a GUI agent. You will be provided with a screenshot, a goal, and your action history. You need to perform the next action to complete the task.\n\n## Action Space\n{}\n\n## Goal\n{}\n\n## Previous Actions\n{}\n\nNow, output the next action in json format [{{\"action\": \"{{action_name}}\"}}, ...]. 2

[18] [18]

action\": \

You are a GUI agent. You will be provided with a screenshot, a goal, and your action history. You need to analyze the current situation and perform the next action to complete the task.\n\n## Action Space\n{}\n\n## Goal\n{}\n\n## Previous Actions\n{}\n\nPlease output your response in the following format:\n\ nThought: <your reasoning about what to do next...

[19] [19]

{}\", perform the next action to complete the task. Output a JSON in the format [{{\

Based on the screenshot and the goal: \"{}\", perform the next action to complete the task. Output a JSON in the format [{{\"action\": \"{{action_name }}\"}}, ...]. 2

[20] [20]

action\": \

Based on the screenshot and the goal: \"{}\", analyze the current situation and perform the next action to complete the task.\n\nPlease output your response in the following format:\n\nThought: <your reasoning about what to do next>\ nAction: <the action you will take>\n\n‘‘‘json\n[{{\"action\": \"{{action_name }}\"}}, ...]\n‘‘‘ 26 Video2GUI: Synthesizing...