CoChat
Sign Up
PreprintCited by 41Multimodal Machine Learning ApplicationsUsability and User Interface DesignAdvanced Neural Network Applications

Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

Open access
You, Keen, Zhang, Haotian, Schoop, Eldon, Weers, Floris, Swearngin, Amanda, Nichols, Jeffrey, Yang, Yinfei, Gan, Zhe·2024·arXiv
DOI 10.48550/arxiv.2404.05719arXiv 2404.05719
Open this paper in CoChat — free accountOpen PDF

Abstract

We present Ferret-UI, a multimodal large language model tailored for mobile UI screens with referring, grounding, and reasoning capabilities.

AI snapshot

Written from the paper’s full text.

TL;DR

Ferret-UI is a multimodal LLM adapted from Ferret for mobile UI screens, adding “any resolution” sub-image encoding and instruction data for referring, grounding, and reasoning tasks. It outperforms GPT-4V on all elementary UI tasks and surpasses Fuyu and CogAgent on advanced tasks, though it depends on a UI detection model and shows some cross-platform generalization limits.

Key findings

  • Ferret-UI-anyres beats GPT-4V on all elementary UI task averages: iPhone referring 82.4 vs 61.3, Android referring 82.4 vs 37.7, iPhone grounding 81.4 vs 70.3, Android grounding 83.8 vs 4.7.
  • Adding anyres improves iPhone referring and grounding by 2 points and raises iPhone advanced-task average from 73.4 (base) to 93.9, while Android advanced average drops from 80.5 (base) to 71.7 in Table 2.
  • Elementary tasks improve advanced tasks: advanced-only training scores 64.6 on iPhone and 64.3 on Android; adding both iPhone and Android elementary tasks raises these to 73.4 and 80.5.
  • On Spotlight benchmarks, Ferret-UI scores 115.6 CIDEr on screen2words and 140.3 on widget captions, beating Spotlight’s 106.7 and 141.8 respectively, but falls short on taperception (72.9 vs 88.4).

Methods

  • Ferret-UI builds on Ferret with any-resolution handling: screens are divided into 1x2 or 2x1 sub-images based on aspect ratio, encoded separately alongside full-image features; the vision encoder is frozen while the decoder and projection layer are trained.
  • Training data include Android screens from RICO/Spotlight (26,527 train, 3,080 test) and iPhone screens from AMP (84,685 train, 9,410 test), with 7 elementary tasks at 40k samples each per platform, 4 advanced tasks at 10k each per platform, and a total training mixture of 250K samples.
  • Evaluation covers 14 mobile UI tasks: 3 Spotlight tasks plus dual iPhone/Android versions of 11 UI tasks; metrics include exact match/accuracy, IoU > 0.5 for grounding, CIDEr for screen2words/widget captions, F1 for taperception, and GPT-4 scoring for advanced open-ended tasks.

Limitations

  • The paper states the UI detection model is a bottleneck: Ferret-UI cannot learn undetected screen aspects such as colors, design, usability, or missed elements like time, WiFi, and battery.
  • The paper states taperception labels are noisy; human annotation correlates with GPT-4V predictions 76% of the time but with taperception labels only 8%.
  • The paper notes GPT-4V’s Set-of-Mark prompting struggles with many small UI elements and restricts free region reference; evident from the text, advanced-task evaluation uses small test sets (25 iPhone and 5 Android screens, 56 and 13 unique images) and GPT-4 scoring may favor GPT-4V’s more verbose answers.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2404.05719

Cites

62 of 62 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 9 cited sources checked · 0 supported · 0 flagged

  • Findingnone given
    § 5.3 Result Analysis: Elementary UI Tasks · p. 14

    When a Tab or an Icon is seated on top of a row of tabs, it is highly likely to be considered part of the tabs.

    “In line with the first observation, when a Tab or an Icon is seated on top of a row of tabs, it is highly likely to be considered part of the tabs.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 5.2 Ablation Studies · p. 12

    With only advanced task data, the performance is 64% for both platforms.

    “We see that with only advanced task data, the performance is 64% for both platforms.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 16 · p. 17

    The effectiveness of Set-of-Mark prompting of GPT-4V diminishes in scenarios involving a multitude of small UI elements, a common occurrence in Android detection tasks.

    “First, its effectiveness diminishes in scenarios involving a multitude of small UI elements, a common occurrence in Android detection tasks.”
    Quote in the paperSubstantial claim
  • Findingnone given
    § 5.4 Result Analysis: Advanced UI Tasks · p. 15

    These detailed answers are more favored in scoring than Ferret-UI's concise answers.

    “However, these detailed answers are more favored in scoring than Ferret-UI's concise answers.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The GPT-4 model evaluator's preferences are a valid measure of response quality.
  • Findingown data
    § 5.3 Result Analysis: Elementary UI Tasks · p. 14

    The model struggles when it needs to understand relationships among widgets.

    “First, the model struggles when it needs to understand relationships among widgets.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 24
  • Classified as probably not a substantial claim. 20

  • May combine two claims.

    Findingreasoning
    § 5.1 Results · p. 10

    Ferret-UI performance falls short on taperception but is still competitive.

    “Ferret-UI performance falls short on TaP but is still competitive”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § 5.1 Results · p. 11

    GPT-4V demonstrates decent performance on iPhone tasks, but its performances on Android tasks, especially grounding tasks, are significantly worse.

    “While GPT-4V demonstrates decent performance on iPhone tasks, its performances on Android tasks, especially grounding tasks, are significantly worse.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.1 Results · p. 11

    Ferret-UI outperforms Ferret and GPT-4V in most elementary tasks except for iPhone find text.

    “Ferret-UI outperforms Ferret and GPT-4V in most elementary tasks except for iPhone fi nd text .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.1 Results · p. 10

    Ferret-UI demonstrates superior performance in screen2words and widget captions compared to Spotlight, even though Spotlight uses 80M web page screenshots and 2.69M mobile screenshots for pre-training.

    “Compared to Spotlight, Ferret-UI demonstrates superior performance in S2W and WiC , even though Spotlight uses 80M web page screenshots and 2.69M mobile screenshots for pre-training.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Spotlight [30]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § B Advanced Task Data Quality Analysis · p. 22

    The vocabulary size for detailed description is 30,866.

    “The vocabulary size for each task is as follows: 30,866 for detailed description , 15,666 for conversation perception , 12,092 for conversation interaction , and 14,896 for function inference .”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.3 Result Analysis: Elementary UI Tasks · p. 14

    The model predicts a neighboring text instead of the text in the targeted region, common for smaller texts and texts very close to other texts.

    “First, the model predicts a neighboring text instead of the text in the targeted region. This is common for smaller texts and texts very close to other texts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation
    § 5.1 Results · p. 12

    Fuyu tends to generate answers that are generally relevant, but its responses lack the detail and precision exhibited by Ferret-UI.

    “While Fuyu [6] tends to generate answers that are generally relevant, its responses lack the detail and precision exhibited by Ferret-UI.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] 6. Bavishi, R., Elsen, E., Hawthorne, C., Nye, M., Odena, A., Somani, A., Taşırlar, S.: Introducing our multimodal models (2023), https://www.adept.ai/blog/fuyu-8b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data
    § 1 Introduction · p. 4

    Ferret-UI significantly surpasses the base Ferret model, illustrating the importance of domain-specific model training.

    “We observe that Ferret-UI significantly surpasses the base Ferret model, illustrating the importance of domainspecific model training.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § B Advanced Task Data Quality Analysis · p. 22

    The vocabulary size for conversation perception is 15,666.

    “The vocabulary size for each task is as follows: 30,866 for detailed description , 15,666 for conversation perception , 12,092 for conversation interaction , and 14,896 for function inference .”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Understanding with Multimodal LLMs · p. 1

    Ferret-UI excels beyond most open-source UI multimodal large language models and surpasses GPT-4V on all elementary UI tasks.

    “Ferret-UI excels not only beyond most open-source UI MLLMs, but also surpasses GPT-4V on all the elementary UI tasks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § 16 · p. 16

    GPT-4V is capable of noting overall design aesthetic, a level of insight Ferret-UI is not trained to offer due to its reliance on detected elements alone.

    “For example, in generating detailed descriptions, GPT-4V is capable of noting 'The overall design conforms to Apple's aesthetic with a minimalistic, clean, dark theme', a level of insight Ferret-UI is not trained to offer due to its reliance on detected elements alone.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The UI detection model's detected elements are sufficient for the UI tasks Ferret-UI is trained and evaluated on.

  • May combine two claims.

    Findingreasoning
    § 5.1 Results · p. 12

    GPT-4V secures higher scores across all tasks by consistently delivering more detailed responses than Ferret-UI.

    “Conversely, GPT4V secures higher scores across all tasks by consistently delivering more detailed responses than Ferret-UI, a characteristic that aligns with the preferences of the model evaluator (GPT-4).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The GPT-4 model evaluator's preferences are a valid measure of response quality.

  • May combine two claims.

    Findingown data
    § 5.4 Result Analysis: Advanced UI Tasks · p. 15

    The accuracies for Ferret-UI and GPT-4V on grounded conversation output boxes are 91.7% and 93.4%, respectively.

    “The accuracies for Ferret-UI and GPT-4V are 91.7% and 93.4%, respectively.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The GPT-4V evaluation scores are comparable to Ferret-UI scores despite different output formats (raw coordinates vs pre-defined boxes).

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § B Advanced Task Data Quality Analysis · p. 22

    The vocabulary size for conversation interaction is 12,092.

    “The vocabulary size for each task is as follows: 30,866 for detailed description , 15,666 for conversation perception , 12,092 for conversation interaction , and 14,896 for function inference .”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Understanding with Multimodal LLMs · p. 1

    UI screens typically exhibit a more elongated aspect ratio and contain smaller objects of interest than natural images.

    “Given that UI screens typically exhibit a more elongated aspect ratio and contain smaller objects of interest ( e.g. , icons, texts) than natural images, we incorporate 'any resolution' on top of Ferret to magnify details and leverage enhanced visual features.”
    • Also stated in: § 1 Introduction · p. 3 · plain
      “UI screens typically exhibit more elongated aspect ratios and contain smaller objects of interests ( e.g. , icons and texts) than natural images.”
    • Also stated in: § 3 Method · p. 6 · plain
      “UI screens are predominantly characterized by aspect ratios that are more extended compared to those found in natural images, as evidenced in Tab. 1a”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § B Advanced Task Data Quality Analysis · p. 22

    From the interaction conversation answers, tap emerges as the predominant action.

    “Notably, from the interaction conversation answers, we observe that tap emerges as the predominant action.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § B Advanced Task Data Quality Analysis · p. 22

    In conversation perception questions, the top tri-grams include phrases like 'are there any', 'where is the', and 'what is the'.

    “Notably, in conversation perception questions, the top tri-grams include phrases like are there any' , where is the' , and what is the' , while those for interactions comprise phrases like How can I' , I want to' , and Can I do' .”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § C Taperception Label Analysis · p. 23

    Our labels exhibit a high correlation with GPT-4V predictions, but differ significantly from the taperception dataset labels.

    “Our labels exhibit a high correlation with GPT-4V predictions, but differing significantly from the taperception dataset labels.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § B Advanced Task Data Quality Analysis · p. 22

    Among conversation interaction turns, 15 question turns include bounding boxes, whereas all answer turns include bounding boxes.

    “Among these, 15 question turns include bounding boxes, whereas all answer turns include bounding boxes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § B Advanced Task Data Quality Analysis · p. 22

    In perception answers, prominent tri-grams consist of expressions such as 'bottom of the', 'at the top', and 'there is a'.

    “Similarly, in perception answers, prominent tri-grams consist of expressions such as 'bottom of the' , 'at the top' , and 'there is a' , while interaction answers primarily feature tri-grams like 'by tapping on' , 'tapping on the' , and 'can tap on' .”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § B Advanced Task Data Quality Analysis · p. 22

    In conversation interaction, there are 33,649 question turns and 29,933 answer turns.

    “In the realm of conversation interaction , we observe 33,649 question turns and 29,933 answer turns.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § B Advanced Task Data Quality Analysis · p. 22

    The vocabulary size for function inference is 14,896.

    “The vocabulary size for each task is as follows: 30,866 for detailed description , 15,666 for conversation perception , 12,092 for conversation interaction , and 14,896 for function inference .”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § B Advanced Task Data Quality Analysis · p. 22

    In conversation interaction questions, the top tri-grams include phrases like 'How can I', 'I want to', and 'Can I do'.

    “Notably, in conversation perception questions, the top tri-grams include phrases like are there any' , where is the' , and what is the' , while those for interactions comprise phrases like How can I' , I want to' , and Can I do' .”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § B Advanced Task Data Quality Analysis · p. 22

    In interaction answers, prominent tri-grams consist of expressions such as 'by tapping on', 'tapping on the', and 'can tap on'.

    “Similarly, in perception answers, prominent tri-grams consist of expressions such as 'bottom of the' , 'at the top' , and 'there is a' , while interaction answers primarily feature tri-grams like 'by tapping on' , 'tapping on the' , and 'can tap on' .”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning
    § 1 Introduction · p. 3

    Ferret's capabilities provide a solid foundation in interactive UI-centric tasks.

    “We posit that such capabilities provide a solid foundation in interactive UI-centric tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § 5.1 Results · p. 10

    The lower taperception performance could be due to the noisiness of the taperception labels.

    “our studies further suggest that this could be due to the noisiness of the taperception labels.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The taperception labels are noisy.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning
    § 6 Conclusion · p. 17

    Ferret-UI demonstrates remarkable proficiency in referring, grounding, and reasoning.

    “FerretUI demonstrates remarkable proficiency in referring, grounding, and reasoning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given
    § Understanding with Multimodal LLMs · p. 1

    After training on the curated datasets, Ferret-UI exhibits outstanding comprehension of UI screens and the capability to execute open-ended instructions.

    “After training on the curated datasets, Ferret-UI exhibits outstanding comprehension of UI screens and the capability to execute open-ended instructions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 16 · p. 17

    The small size of some UI components means that the addition of labels may obscure original content or even extend beyond the intended areas.

    “The small size of some UI components means that the addition of labels may obscure original content or even extend beyond the intended areas.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given
    § 1 Introduction · p. 3

    Advanced tasks prepare the model to engage in more nuanced discussions about visual components, formulate action plans with specific goals, and interpret the general purpose of a screen.

    “These advanced tasks prepare the model to engage in more nuanced discussions about visual components, formulate action plans with specific goals in mind, and interpret the general purpose of a screen.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § 1 Introduction · p. 3

    Other works targeting UI tasks have primarily focused on processing entire screens as singular inputs, only support referring tasks with one bounding box in the input, and leverage GPT-4V to navigate UI screens.

    “On the other hand, other works targeting directly at UI tasks have primarily focused on processing entire screens as singular inputs ( e.g. , Pix2Struct [27], ILuvUI [23], CogAgent [20]), only supports referring tasks with one bounding box in the input ( e.g. , Spotlight [30]), and leveraging GPT-4V [51] to navigate UI screens, as seen in MM-Navigator [49], AppAgent [55], and MobileAgent [47].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Pix2Struct [27]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites ILuvUI [23]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites CogAgent [20]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning
    § 3 Method · p. 6

    This approach facilitates advanced single-screen interactions and paves the way for new applications, such as improving accessibility.

    “This approach not only facilitates advanced single-screen interactions, but also paves the way for new applications, such as improving accessibility.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 1 Introduction · p. 3

    Ferret-UI is the first multimodal large language model designed to execute precise referring and grounding tasks specific to UI screens, while adeptly interpreting and acting upon open-ended language instructions.

    “In this paper, we present Ferret-UI, the first MLLM designed to execute precise referring and grounding tasks specific to UI screens, while adeptly interpreting and acting upon open-ended language instructions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § Understanding with Multimodal LLMs · p. 1

    General-domain multimodal large language models often fall short in their ability to comprehend and interact effectively with user interface screens.

    “Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with user interface (UI) screens.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given
    § 1 Introduction · p. 3

    Multimodal large language models like Ferret, Shikra, and Kosmos2 demonstrate strong referring and grounding capabilities, but their scope is mainly restricted to natural images.

    “On one hand, while Multimodal Large Language Models (MLLMs) like Ferret [53], Shikra [8], and Kosmos2 [41] demonstrate strong referring and grounding capabilities, their scope is mainly restricted to natural images.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Ferret [53]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Shikra [8]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Kosmos2 [41]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 5.2 Ablation Studies · p. 12

    The skill set honed through advanced tasks would be advantageous for tackling Spotlight tasks, which occupy a middle ground in complexity between elementary and advanced tasks.

    “Thus, it stands to reason that the skill set honed through advanced tasks would be advantageous for tackling Spotlight tasks, which occupy a middle ground in complexity between elementary and advanced tasks.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given
    § 5.2 Ablation Studies · p. 12

    Executing advanced tasks requires more sophisticated skills than understanding one specific UI element on the screen as in elementary tasks.

    “Moreover, executing advanced tasks requires more sophisticated skills than understanding one specific UI element on the screen as in elementary tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data
    § B Advanced Task Data Quality Analysis · p. 22

    The observation of tri-grams is consistent with the intended objectives for each conversation category, with perception focusing on visual elements and interaction emphasizing actions.

    “This observation is consistent with our intended objectives for each conversation category, with perception focusing on visual elements and interaction emphasizing actions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § C Taperception Label Analysis · p. 23

    This discrepancy underscores the complexity of predicting tappability solely based on single images, highlighting the inherent challenges in obtaining clear-cut labels for this task.

    “This discrepancy underscores the complexity of predicting tappability solely based on single images, highlighting the inherent challenges in obtaining clear-cut labels for this task.”
    Quote in the paperSubstantial claim
39 of 39

  • The paper relies on this without support.

    GPT-4 scores for open-ended responses are valid and reliable for evaluating advanced UI tasks.

    “As the advanced tasks require open-ended responses, we use GPT-4 to score both the label and the prediction.”

  • The paper relies on this without support.

    The taperception labels are noisy.

    “our studies further suggest that this could be due to the noisiness of the taperception labels.”

  • The paper relies on this without support.

    The GPT-4 model evaluator's preferences are a valid measure of response quality.

    “a characteristic that aligns with the preferences of the model evaluator (GPT-4).”

  • The paper relies on this without support.

    The UI detection model's detected elements are sufficient for the UI tasks Ferret-UI is trained and evaluated on.

    “Given that both our elementary and advanced tasks are predicated upon the detection of UI elements, Ferret-UI is not able to learn aspects of screens that are not detected, such as colors, design, usability, and UI elements that the detection model misses ( e.g. , topmost time, WIFI, battery).”

  • The paper relies on this without support.

    The GPT-4V evaluation scores are comparable to Ferret-UI scores despite different output formats (raw coordinates vs pre-defined boxes).

    “Considering FerretUI generates raw coordinates whereas GPT-4V chooses from a set of pre-defined boxes, Ferret-UI's grounding ability on UI screens is noteworthy.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior work reports frequent misrecognition of UI structure, limited sensitivity to design intent, and weakly actionable critiques in current AI UI evaluation systems.

    “Prior work reports frequent misrecognition of UI structure [34, 94], limited sensitivity to design intent [17, 57], and weakly actionable critiques [48].”

    From Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation (2026) · cites this paper as [94]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Two tables give different Android anyres advanced-task averages

    Major

    Table 2 reports 71.7 for Ferret-UI-anyres on Android advanced tasks, whereas Table 3 reports 70.1 for the same model and category.

  • Widget captions is described both with and without an input box

    Minor

    The task prompt identifies the element by a bounding box, but the GPT-4V evaluation details classify widget captions as a task without bounding boxes and say it receives the original image.

  • Claim of beating GPT-4V on all elementary tasks has an exception

    Minor

    The abstract says Ferret-UI surpasses GPT-4V on all elementary tasks, but the results section explicitly identifies iPhone find text as an exception.

  • Training-data table lists Android advanced samples despite their stated absence

    Minor

    Table 1 lists 10k × 4 advanced-task training samples in both platform columns, while the results section says Android advanced-task data is not included in the training mix.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. [44]44. Shi, T., Karpathy, A., Fan, L., Hernandez, J., Liang, P.: World of bits: An opendomain platform for web-based agents. In: International Conference on Machine Learning. pp. 3135-3144. PMLR (2017)

Cited by

Sorted by citations

Related

Authors
You, K., Zhang, H., Schoop, E., Weers, F., Swearngin, A., Nichols, J., Yang, Y., & Gan, Z.
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
240-255
ISSN
0302-9743 (ISSN-L)
DOI
10.48550/arxiv.2404.05719
arXiv
2404.05719
OpenAlex
W4403888362
Semantic Scholar
3687e55cf21c4d0041d9bf0b74988319bfe3402b
Cited by
40
References
20
Citations by year
2026: 13 · 2025: 25 · 2024: 2
FWCI
8.42
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
multimodal large language models, visual grounding, aspect ratio, icon recognition, instruction following

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-20.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-20.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review