ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

UGround supports resolutions up to 1344 × 1344, while QwenVL operates at 448 × 448.

“For instance, UGround (Gou et al., 2024) supports resolutions up to 1344 × 1344, while QwenVL (Bai et al., 2023) operates at 448 × 448.”

• cites Gou et al., 2024 sourceSupported
• cites Bai et al., 2023 sourceSupported

May combine two claims.

Professional applications remain largely unexplored in GUI grounding, with only a few works featuring specialized tasks such as coding in VSCode.

“In contrast, professional applications remain largely unexplored, with only a few works featuring specialized tasks such as coding in VSCode (Xie et al., 2024).”

• cites Xie et al., 2024 sourceSupported

May combine two claims.

Increased resolution results in smaller relative target sizes in the screenshot, and GUI grounding models generally exhibit worse performance as target bounding box size becomes smaller.

“the increased resolution results in smaller relative target sizes in the screenshot, where GUI grounding models generally exhibit worse performance, as demonstrated in Figure 2”

• Also stated in: § 1 Introduction · p. 2 · boosted

  “There is a universal decrease in accuracy as the target bounding box size becomes smaller.”

• cites Cheng et al., 2024 sourceNot checked — no full text of the source
• cites Wu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Gou et al., 2024 sourcePartly supported

  Appendix C.2 (p. 20) reports that higher model input resolution generally improves grounding performance, particularly for web and desktop interfaces with small links and icons. It does not establish that higher screenshot resolution makes target bounding boxes relatively smaller or measure performance by bounding-box size, so the claimed relationship is not fully supported. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

To enable fair comparison, the number of iterations in Iterative Zooming and Iterative Narrowing were both set to 3 following Nguyen (2024), and GPT-4o was used as the planner for ScreenSeekeR.

“To enable a fair comparison, the number of iterations in Iterative Zooming and Iterative Narrowing are both set to 3 following Nguyen (2024). For ScreenSeekeR, we utilize GPT-4o (OpenAI, 2024) as the planner.”

• cites Nguyen (2024) sourceSupported
• cites OpenAI, 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Visual search methods depend on predefined splitting strategies, which constrain search flexibility and may result in missing contextual information in GUI environments.

“However, these methods depend on predefined splitting strategies, which constrain search flexibility and may result in missing contextual information in GUI environments.”

• cites Wu and Xie, 2023 sourcePartly supported

  Section 3.3.3 (page unavailable) describes a fixed, recursive patch-splitting strategy, and a later passage notes that targets at patch boundaries may be missed. The paper does not establish that this constrains search flexibility or causes missing contextual information in GUI environments; its discussion concerns general images, not GUIs. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Wang et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims. The quote only partly states this claim.

ScreenSeekeR boosts OS-Atlas7B's performance from 18.9% to 48.1% on ScreenSpot-Pro, a 254% relative improvement, achieving state-of-the-art performance without additional training.

“Based on this insight, we propose ScreenSeekeR , a visual search method that utilizes the GUI knowledge of a strong planner to guide a cascaded search, achieving state-ofthe-art performance with 48.1% without any additional training.”

• Also stated in: § 1 Introduction · p. 2 · plain

  “With this approach, ScreenSeekeR boosts the OS-Atlas7B model's performance from 18.9% to 48.1%, achieving a 254% relative improvement.”

May combine two claims.

Existing GUI grounding models perform poorly on ScreenSpot-Pro, with the best model achieving only 18.9%.

“Existing GUI grounding models perform poorly on this dataset, with the best model achieving only 18.9%.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the example task 'delete file or folder,' ReGround was misled into grounding the file tab in the background VSCode window.

“given the task of”

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-4o and QwenVL-7B performance improved with Chinese instructions.

“the performance of GPT-4o and QwenVL-7B improved”

May combine two claims.

Both OS-Atlas-7B and UGround (7B) exhibit peak performance within specific resolution ranges, with performance declining as image sizes deviate.

“Both models exhibit peak performance within specific resolution ranges, with performance declining as image sizes deviate.”

May combine two claims.

OSAtlas-7B leads performance on ScreenSpot-Pro with an accuracy of 18.9%, closely followed by UGround and AriaUI.

“OSAtlas-7B leads the performance with an accuracy of 18.9%, closely followed by UGround and AriaUI.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In ScreenSpot-Pro, text constitutes 62.6% of elements, with the remainder being icons.

“The text constitute 62.6% of the elements, with the remainder being icons.”

May combine two claims.

In the same example, ScreenSeekeR successfully grounds the target UI and generates a natural search trajectory.

“In contrast, ScreenSeekeR not only successfully grounds the target UI but also generates a natural search trajectory.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ScreenSpot-Pro includes six distinct application genres, with a primary focus on four types of professional applications, plus office productivity software and common operating system tasks.

“ScreenSpot-Pro includes six distinct application genres, with a primary focus on four types of professional applications. Additionally, it features office productivity software and common operating system tasks.”

May combine two claims.

Targets in ScreenSpot-Pro occupy 0.07% of the screenshot area on average, compared to 2.01% in ScreenSpot.

“Notably, targets in ScreenSpot-Pro occupy 0.07% of the screenshot area on average, a significant reduction compared to 2.01% of ScreenSpot (Cheng et al., 2024).”

• cites Cheng et al., 2024 sourceNot checked — no full text of the source

May combine two claims.

Previous benchmarks such as ScreenSpot neglect high-resolution professional factors and only evaluate GUI grounding in cropped screenshot areas in easy usages.

“However, previous benchmarks such as ScreenSpot (Cheng et al., 2024) neglect these factors and only evaluate GUI grounding in cropped screenshot areas in easy usages, as shown in Figure 3.”

• Also stated in: § 2.1 GUI Grounding · p. 3 · plain

  “For instance, ScreenSpot (Cheng et al., 2024) facilitates artificial screenshots by cropping regions from full-screen images”

• Also stated in: § Add a new pattern · p. 3 · plain

  “ScreenSpot-Pro features screenshots of the entire screen, while ScreenSpot contains unrealistic screenshots cropped to local areas.”

• cites Cheng et al., 2024 sourceNot checked — no full text of the source

May combine two claims.

Annotators precisely verified interactable regions of GUI elements, excluding irrelevant areas, ensuring bounding boxes closely match each element.

“Annotators precisely verify the interactable regions of the GUI elements, excluding irrelevant areas. This ensures bounding boxes closely match each element for accurate representation and minimal ambiguity.”

Classified as probably not a substantial claim. The quote only partly states this claim.

Experts with at least five years of experience using the relevant applications recorded the data for ScreenSpot-Pro.

“Experts with at least five years of experience using the relevant applications were invited to record the data.”

May combine two claims.

ScreenSeekeR is a search-based method that enhances accuracy by refining the search space, achieving a substantial performance boost without additional training.

“we proposed ScreenSeekeR, a search-based method that enhances accuracy by refining the search space, achieving a substantial performance boost without additional training.”

Classified as probably not a substantial claim. May combine two claims.

To obtain authentic high-resolution images, screens with resolution greater than 1080p (1920 × 1080) were prioritized, monitor scaling was disabled, and dual-monitor setups were captured spanning both displays.

“To obtain authentic high-resolution images, we prioritized screens with a resolution greater than 1080p (1920 × 1080), a configuration commonly found among annotators. Monitor scaling was disabled. In dual-monitor setups, images were captured to span both displays.”

Probably not a substantial claim: may not pass S4 (based).

The UI of applications typically follows a well-defined hierarchy, with menus, tools, and properties organized within sub-panels or child windows.

“Unlike natural images, the UI of applications typically follows a well-defined hierarchy. For example, menus, tools, and properties are often organized within sub-panels or child windows, providing potential cues on where to search for a UI target.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The centrality-based scoring approach emulates human visual attention and mitigates bias towards large areas, which would otherwise slow down the search process.

“This centrality-based approach emulates human visual attention, and mitigates the bias towards large areas, which would otherwise slow down the search process.”

May combine two claims.

The screen capture tool allowed experts to label bounding boxes and provide instructions in real-time, enhancing consistency and quality of annotations.

“When activated, this tool takes a screenshot and overlays it on the screen, allowing experts to label the bounding boxes and provide instructions directly. This method enhances the consistency and quality of the annotations, as experts can label tasks in real-time without the need to recall the purposes and context of their actions in hindsight.”

Classified as probably not a substantial claim.

Every task in ScreenSpot-Pro includes a Chinese instruction translated by GPT-4 and reviewed by authors fluent in both languages.

“every task in the benchmark also includes a Chinese instruction translated by GPT-4 and reviewed by the authors who are fluent in both languages.”

• AssumptionRests on an assumption: In professional scenarios, it is common for non-English speakers to operate in both their native languages and English.

Probably not a substantial claim: may not pass S4 (based).

Iterative Narrowing's superior performance is likely due to its superior image-splitting strategy when the target is positioned near the center of the x or y axes.

“likely due to its superior image-splitting strategy when the target is positioned near the center of the x or y axes.”

Classified as probably not a substantial claim.

The varying formats of location outputs were handled to ensure a fair comparison across models.

“We handle the varying formats of the location outputs to ensure a fair comparison across models.”

May combine two claims.

Each ScreenSpot-Pro instance was reviewed by at least two annotators to ensure correct instructions and target bounding boxes, and ambiguous instructions were resolved to guarantee only one target.

“Each instance is reviewed by at least two annotators to ensure correct instructions and target bounding boxes. Ambiguous instructions are resolved, guaranteeing only one target.”