PreprintCited by 6Information Retrieval and Search BehaviorDigital Marketing and Social MediaConsumer Market Behavior and Pricing

Does Rank Still Matter? Position Bias When AI Agents Shop on Our Behalf

Open access
Wadi, Davood, Ma, Yu·2026·arXiv

Abstract

When shopping is delegated to AI agents, it is unclear whether the ranking advantage documented for humans persists. Across 7,000 sessions with five large language models (LLMs) and varying reasoning effort, we compare AI agents with human field data. AI agents search more extensively than human consumers. As with humans, lower-ranked listings are less likely to be inspected, although the effect is smaller for AI agents. Unlike humans, AI agents show a pattern consistent with the lost-in-the-middle effect, whereby middle listings have the lowest probability of inspection. At the choice stage, position effects are concentrated at lower reasoning effort, but higher effort reduces the middle penalty for every LLM that exhibits it. These findings suggest that, when search is delegated, displayed attributes may matter more than placement, and that exposure to position bias depends on how the AI agent is configured.

AI snapshot

Written from the paper’s full text.

TL;DR

AI agents searching hotel listings inspect more deeply than humans and never decline to book, yet randomized position still predicts inspection—weakly and often nonmonotonically. Position effects on final choice vary by LLM and shrink with higher reasoning effort, while all models converge on the same undominated listing.

Key findings

  • AI agents searched more than humans: inspections per session ranged from 1.63 to 5.83 versus 1.12 for humans; sessions with only one inspection occurred in 0.8%–37.4% of AI sessions versus 93% of human sessions. All AI agents booked in 100% of sessions, versus 66% human conversion and 34% outside option.
  • Position predicted inspection for every LLM, with coefficients from -0.0002 to -0.0005 (all ps<0.001), four to ten times weaker than the human benchmark of -0.0019. Moving a listing down ten ranks cost humans about 1.9 percentage points of inspection probability but AI agents only 0.2–0.5 points. For three of four LLMs, inspection fell from the top to a minimum around ranks 68–74 and then rose again; Pro was mostly a primacy effect.
  • Position effects on final choice were heterogeneous: not significant for Flash (-0.000014, p=0.338) or Pro (-0.000008, p=0.606), but significant for Sonnet (-0.000035, p=0.018) and Flash Lite (-0.000080, p<0.001). Average chosen rank ranged from 44.0 to 49.7 against the position-neutral value of 50.5. All LLMs converged on the same undominated listing, citizenM New York Times Square, which captured 78.2% of all bookings pooled across LLMs (1,563 of 2,000 sessions).
  • Higher reasoning effort reduced position effects to insignificance at the highest level for both tested LLMs. Search depth moved in opposite directions (Flash Lite fell from 3.12 to 1.91 inspections; Pro rose from 0.39 to 5.83), and average chosen rank converged toward 50.5 (Flash Lite 43.97 to 51.20; Pro 40.96 to 49.72).

Methods

  • The experiment randomized the presentation order of 100 hotel listings in each session. LLM agents could call an inspect tool for more hotel details and a submit_choice tool to book; they could also terminate without booking. The context replicated Ursu (2018), with trip parameters for two adults, no children, one room, and a two-night stay including a Saturday night.
  • Main sample: four proprietary LLMs from Google (Gemini 3.1 Pro, Gemini 3.7 Flash, Gemini 3.1 Flash Lite) and Anthropic (Claude Sonnet 5), with 500 independent sessions per model (2,000 total) under default reasoning and sampling parameters. Follow-ups manipulated reasoning effort across available levels for Gemini 3.1 Flash Lite (four levels) and Gemini 3.1 Pro (three levels), with 500 replications per cell (3,500 total), and a reduced-prompt variation with 500 additional Flash Lite sessions.
  • Analysis used linear probability models with standard errors clustered at the session level. Position was indexed 1–100; controls included price, review score, chain, and promotion. Quadratic position models tested nonlinearity.

Limitations

  • Stated: the study fixed the query and destination in a hotel booking task; other tasks with differing attribute complexity could produce different results.
  • Stated: the pattern is consistent with lost-in-the-middle, but the paper does not identify it as the mechanism; doing so would require attention-level access the authors did not have. The LLMs tested are also moving targets, and their positional biases may be reduced or eliminated in future releases.
  • Evident in the text: the designs differ in list length—100 listings in this study versus roughly seven impressions on average in the human benchmark—so the per-rank slope is estimated over a much longer span in the AI data.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2608.22697

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 19 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Interpretationreasoning

    Lost-in-the-middle is an interpretation consistent with the observed U-shaped inspection pattern rather than a mechanism the study identifies.

    “Since we do not observe attention directly, lost-in-the-middle is an interpretation consistent with this pattern rather than a mechanism we identify.”
    • Also stated in: § 4. Conclusion · plain
      “we observe a pattern consistent with lost-in-the-middle, but we do not identify it as the mechanism.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Liu et al. 2024 source
  • Findingown data

    For the tested LLMs except Gemini 3.1 Pro, the position effect on inspection is U-shaped: inspection declines from the top of the listing page to a minimum around ranks 68 to 74 and then rises again.

    “Rather than declining monotonically, for the LLMs tested (except Gemini 3.1 Pro) inspection falls from the top of the page to a minimum around ranks 68 to 74 and then rises again, so the bottom of the listing page is a better place to be than the middle (a phenomenon called the lost-in-the-middle effect; Liu et al. 2024).”
    • Also stated in: § 3.4.3. Nonlinear position effects · plain
      “Inspection declines from the top of the listing page to a minimum between ranks 68 and 74 before increasing for the three LLMs with significant quadratic terms (Figure W2, left panel).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Liu et al. 2024 source
    • Rests on an assumption: AI agents can serve as valid surrogates for human consumers in the hotel search environment, such that their behavior is comparable to human field data.
    • Rests on an assumption: The inspect tool call is a valid analogue of a human click/inspection.
  • Interpretationown data

    Higher reasoning effort can help mitigate the lost-in-the-middle effect in LLMs.

    “This provides evidence that higher reasoning effort can help mitigate the lost-in-themiddle effect in LLMs.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Liu et al. 2024 source

      Section 2.3 documents the lost-in-the-middle effect, and Section 4.3 reports that instruction fine-tuning slightly reduced positional bias in some models. The paper does not examine higher reasoning effort or show that it mitigates the effect; that mechanism is the citing authors’ inference, not a finding of this source. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation

    LLMs retrieve information from the beginning and end of a long context more reliably than information placed in the middle (lost-in-the-middle).

    “Information placed in the middle of a long context is retrieved less reliably than information at either end (Liu et al. 2024).”
    • Also stated in: § 2. Background · plain
      “LLMs retrieve information from the beginning and end of a long context more reliably than information placed in the middle (Liu et al. 2024).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Liu et al. 2024 source
  • Findingown data

    Increasing reasoning effort reduces the position effect on final choice to insignificance in both the least and most capable LLMs tested.

    “Increasing reasoning effort reduces the position effect to insignificance in both the least and most capable LLMs tested.”
    • Also stated in: § 3.4.4. Reasoning effort · plain
      “For the final choice, increasing the reasoning effort reduces the position effects for both LLMs and neither is significant at the highest effort level.”
    Quote in the paperSubstantial claim
  • May combine two claims. 19
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 4
  • May combine two claims.

    Findingown data

    Under the alternative prompt, search depth is marginally lower (M=2.92 vs. M=3.12) but remains far above the human benchmark (M=1.12).

    “Search depth is marginally lower ( M Alternative = 2.92 vs. M Original = 3.12 ) but remains far above the human benchmark ( M Human = 1.12 ).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ursu 2018 source
    • Rests on an assumption: The human benchmark from Ursu (2018) is comparable to the AI agent experiment despite differences in list length (100 vs ~7) and platform.
    • Rests on an assumption: The inspect tool call is a valid analogue of a human click/inspection.
  • May combine two claims.

    Findingown data

    The nonlinearity in position effect is confined to inspection and does not systematically transmit to the final choice.

    “The nonlinearity is confined to inspection and does not systematically transmit to the final choice.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The position coefficient on inspection is negative and significant for all tested LLMs, ranging from -0.0002 to -0.0005, which is four to ten times smaller than the human benchmark of -0.0019.

    “The position coefficient on inspection is negative and significant for all LLMs, ranging from -0.0002 to -0.0005 (all ps<0.001;Table 2, left panel), whereas the human benchmark is -0.0019 (p<0.001).”
    • Also stated in: § 1. Introduction · plain
      “rank nonetheless continues to predict inspection for every LLM tested, although four to ten times more weakly than for humans.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Ursu 2018 source
    • Rests on an assumption: AI agents can serve as valid surrogates for human consumers in the hotel search environment, such that their behavior is comparable to human field data.
    • Rests on an assumption: The human benchmark from Ursu (2018) is comparable to the AI agent experiment despite differences in list length (100 vs ~7) and platform.
    • Rests on an assumption: The inspect tool call is a valid analogue of a human click/inspection.
  • May combine two claims.

    Findingown data

    All tested LLMs converge on the same undominated listing, citizenM New York Times Square, which captures 78.2% of all bookings (1,563 of 2,000 sessions).

    “All models nonetheless converge on the same undominated listing.”
    • Also stated in: § Additional information · plain
      “The modal choice, citizenM New York Times Square, is undominated on price and review score, with the highest review score in the listing (4.7) at the lowest nightly price for that score ($220).Its choice shares range from 55.4% to 100%, and pooled across LLMs this single listing captures 78.2% of all choices (1,563 of 2,000 sessions).”
    • Also stated in: § 4. Conclusion · plain
      “All four LLMs nonetheless converged on the same undominated listing, which captured 78.2% of all bookings.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: AI agents can serve as valid surrogates for human consumers in the hotel search environment, such that their behavior is comparable to human field data.
    • Rests on an assumption: The four tested LLMs are representative of AI agents used for shopping.
  • May combine two claims.

    Findingown data

    Under the alternative prompt (without recommendation or inspection instructions), conversion remains at 100% and the outside option is never selected.

    “Conversion remains at 100% and the outside option is never selected (Table 3).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The submit_choice tool and outside option in AI sessions are equivalent to human booking and outside option.
  • May combine two claims. The quote only partly states this claim.

    Findingown data

    In the hotel search environment, AI agents never declined to book a hotel, booking in 100% of sessions and selecting the outside option in 0% of sessions.

    “AI agents search more deeply than humans and never decline to buy.”
    • Also stated in: § 3.4. Results · plain
      “all AI agents booked in 100% of sessions.”
    • Also stated in: § 4. Conclusion · plain
      “Outside option selected | 0.0% | 0.0% | 0.0% | 0.0% | 34.0% |”
    Quote in the paperSubstantial claim
    • Rests on an assumption: AI agents can serve as valid surrogates for human consumers in the hotel search environment, such that their behavior is comparable to human field data.
    • Rests on an assumption: The submit_choice tool and outside option in AI sessions are equivalent to human booking and outside option.
  • May combine two claims.

    Findingown data

    With reasoning effort manipulation, conversion rate remains at 100% and the outside option is never selected.

    “With reasoning effort manipulation, conversion rate remains at 100% and the outside option is never selected (Table 3).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The submit_choice tool and outside option in AI sessions are equivalent to human booking and outside option.
  • May combine two claims.

    Findingown data

    Search depth changes in opposite directions with reasoning effort across the two tested LLMs: it falls from 3.12 to 1.91 inspections per session for Flash Lite and rises from 0.39 to 5.83 for Pro.

    “Search depth falls from 3.12 to 1.91 inspections per session for Flash Lite and rises from 0.39 to 5.83 for Pro (Table 3; Table W5).”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The modal choice, citizenM New York Times Square, is undominated on price and review score, with the highest review score in the listing (4.7) at the lowest nightly price for that score ($220).

    “The modal choice, citizenM New York Times Square, is undominated on price and review score, with the highest review score in the listing (4.7) at the lowest nightly price for that score ($220).”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    Every LLM concentrates between 89.6% and 100% of its choices in the same five hotels out of 100, and shares the same modal choice.

    “Every LLM concentrates between 89.6% and 100% of its choices in the same five hotels out of 100, and shares the same modal choice (Table W3).”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The effect of position on the final choice varies by LLM: it is indistinguishable from zero for Flash and Pro, significant but smaller than humans for Sonnet, and significant and similar magnitude to humans for Flash Lite.

    “For Flash and Pro, it is indistinguishable from zero (-0.000014, p=0.338 and -0.000008, p=0.606, respectively).”
    • Also stated in: § Additional information · plain
      “Sonnet, it is significant (-0.000035, p=0.018) but smaller than humans (-0.0001, p<0.001).For Flash Lite, it is significant (-0.000080, p<0.001) and similar magnitude to humans (Table 2,”
    • Also stated in: § 4. Conclusion · plain
      “At the booking stage, rank mattered for two of the four LLMs (Sonnet and Flash Lite) but not for the other two (Flash and Pro).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Ursu 2018 source
    • Rests on an assumption: AI agents can serve as valid surrogates for human consumers in the hotel search environment, such that their behavior is comparable to human field data.
    • Rests on an assumption: The human benchmark from Ursu (2018) is comparable to the AI agent experiment despite differences in list length (100 vs ~7) and platform.
  • May combine two claims.

    Findingown data

    Increasing reasoning effort moves the average rank of the chosen hotel toward the position-neutral value of 50.5 (from 43.97 to 51.20 for Flash Lite and from 40.96 to 49.72 for Pro) and raises the modal choice proportion (from 61.4% to 96.6% and from 48.0% to 55.4%).

    “Furthermore, the average rank of the chosen hotel converges toward the positionneutral value of 50.5 (from 43.97 to 51.20 for Flash Lite and from 40.96 to 49.72 for Pro), and the modal choice proportion rises from 61.4% to 96.6% and from 48.0% to 55.4% (Table 3).”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data

    The study's fixed query and destination in a hotel booking task limit generalizability; other tasks with differing attribute complexity could produce different results.

    “Our study has several limitations.We fixed the query and destination in a hotel booking task.Other tasks with differing levels of attribute complexity could produce different results.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Premisecitation

    LLMs display heuristics and biases in consumption settings.

    “although the same LLMs have been shown to display heuristics and biases in consumption settings (Wadi et al. 2026b).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wadi et al. 2026b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Sampling parameters can no longer be manipulated for most modern LLMs; Google and Anthropic APIs return errors if non-default sampling parameters are used.

    “Sampling parameters can no longer be manipulated for most modern LLMs.Google and Anthropic APIs return errors if non-default sampling parameters are used (Anthropic 2026; Google 2026).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Anthropic 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Google 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductreasoning

    The presentation order of the 100 hotels was fully randomized for each session, eliminating endogeneity bias inherent in search engine ranking algorithms.

    “To eliminate the endogeneity bias inherent in search engine ranking algorithms, where higher-quality options are systematically placed at the top, the presentation order of the 100 hotels was fully randomized for each session.”
    • Also stated in: § 3.4.1. Position effect on inspection · boosted
      “Because we randomize the order of the 100 hotels in every session, any relationship between position and behavior is causal by construction.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Premisecitation

    Review scores remain a primary heuristic and trust indicator in online decision environments, driving choice even when rankings shift.

    “review scores, which remain a primary heuristic and trust indicator in online decision environments (Wadi et al. 2026a), driving choice even when rankings shift.”
    • Also stated in: § 4. Conclusion · plain
      “review scores, which remain a primary heuristic and trust indicator in online decision environments (Wadi et al. 2026c), driving choice even when rankings shift.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Wadi et al. 2026a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Wadi et al. 2026c

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Standard errors in the reduced-form models were clustered at the session level.

    “Because the 100 hotel observations within a session were not independent, standard errors in the reduced-form models were clustered at the session level.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Premisecitation

    Human consumers who click at least once book 66% of the time and take the outside option 34% of the time.

    “human consumers who click at least once book 66% of the time and take the outside option 34% of the time”
    • Also stated in: § 4. Conclusion · plain
      “Outside option selected | 0.0% | 0.0% | 0.0% | 0.0% | 34.0% |”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Ursu 2018 source
  • May combine two claims.

    Premisecitation

    Human consumers inspect 1.12 hotels per session, and 93% of their sessions contain a single inspection.

    “Research shows that with hotel search, humans make on average 1.12 clicks and 93% of sessions contain only one click (Ursu 2018).”
    • Also stated in: § 3.4. Results · plain
      “Human consumers inspect 1.12 hotels per session, and 93% of their sessions contain a single inspection.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Ursu 2018 source
  • Classified as probably not a substantial claim.

    Premisecitation

    Rank determines which alternatives enter the consideration set rather than what they are worth after they enter.

    “The canonical explanation is that ranking determines which alternatives enter the consideration set while leaving the consumer's valuation of an inspected alternative untouched (Ursu 2018).”
    • Also stated in: § 2. Background · plain
      “Rank, or position, determines which alternatives enter the consideration set rather than what they are worth after they enter.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Ursu 2018 source
  • May combine two claims.

    Study design and conductcitation

    The designs differ in list length: our page shows 100 listings, while impressions in the human benchmark average roughly seven, so the per-rank slope is estimated over a much longer span in our data.

    “The designs differ in list length: our page shows 100 listings, while impressions in the human benchmark average roughly seven, so the per-rank slope is estimated over a much longer span in our data.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Ursu 2018 source
22 of 22
  • The paper relies on this without support.

    AI agents can serve as valid surrogates for human consumers in the hotel search environment, such that their behavior is comparable to human field data.

    “We study this by placing LLM agents in a hotel search environment, comparable to a human field benchmark.”
    “To ensure comparability with an established human benchmark, we replicate the context of the randomized field experiment conducted by Ursu (2018).”
  • The paper relies on this without support.

    The human benchmark from Ursu (2018) is comparable to the AI agent experiment despite differences in list length (100 vs ~7) and platform.

    “The designs differ in list length: our page shows 100 listings, while impressions in the human benchmark average roughly seven, so the per-rank slope is estimated over a much longer span in our data.”
  • The paper relies on this without support.

    The inspect tool call is a valid analogue of a human click/inspection.

    “Inspected was coded 1 when the agent invoked the inspect tool for a given hotel and 0 otherwise (an analogue of a click in the human benchmark).”
  • The paper relies on this without support.

    The submit_choice tool and outside option in AI sessions are equivalent to human booking and outside option.

    “At any point, the LLM could use the submit_choice tool to book a hotel.It could also terminate the session without making a booking, representing the outside option.”
  • The paper relies on this without support.

    The four tested LLMs are representative of AI agents used for shopping.

    “The main experiment comprises 2,000 sessions across four major LLMs from two providers (i.e., Google and Anthropic).”
  • The paper relies on this without support.

    AI agents receive the entire results page at once and do not scroll, so they pay no marginal cost to read later listings.

    “An AI agent can receive the entire results page at once, in its context window.It does not scroll and pays no marginal cost to read the hundredth listing compared to the first.”
  • The paper relies on this without support.

    The Google LLMs represent distinct levels of computational capability, with Gemini 3.1 Pro as highest-capability and Gemini 3.1 Flash Lite as lowest-capability.

    “The LLMs from Google represent distinct levels of computational capability.Gemini 3.1 Pro is Google's highest-capability LLM and is expected to facilitate complex, multi-attribute calculations.In contrast, Gemini 3.1 Flash Lite is the lowest-capability LLM, designed for lightweight and efficient tasks.”
7 of 7

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Introduction says rank does not affect booking, but results say it does for two models

    Major

    The introduction states that position does not affect final choice for AI agents. The conclusion reports that rank mattered at the booking stage for Sonnet and Flash Lite.

1 of 1

Cites

Flagged first, then by citations

2 of the 45 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

  1. Wadi D, Legoux R, Fredette M, Sénécal S (2026c) The interplay of altruism and financial incentives: Maximizing online reviews through effective messaging. Journal of Electronic Commerce Research 27:

Cited by

Sorted by citations

Authors
Wadi, D., & Ma, Y.
Affiliations
McGill University (CA)
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
OpenAlex
W7204189884
Cited by
0
References
45
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University) (cc-by-nc-sa)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Social Sciences › Sociology and Political Science › Digital Marketing and Social Media
  • Social Sciences › Business, Management and Accounting › Marketing › Consumer Market Behavior and Pricing
Keywords
search engine ranking, position bias, AI agents, large language models, agentic search, consumer choice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review