Does Rank Still Matter? Position Bias When AI Agents Shop on Our Behalf
Open accessAbstract
When shopping is delegated to AI agents, it is unclear whether the ranking advantage documented for humans persists. Across 7,000 sessions with five large language models (LLMs) and varying reasoning effort, we compare AI agents with human field data. AI agents search more extensively than human consumers. As with humans, lower-ranked listings are less likely to be inspected, although the effect is smaller for AI agents. Unlike humans, AI agents show a pattern consistent with the lost-in-the-middle effect, whereby middle listings have the lowest probability of inspection. At the choice stage, position effects are concentrated at lower reasoning effort, but higher effort reduces the middle penalty for every LLM that exhibits it. These findings suggest that, when search is delegated, displayed attributes may matter more than placement, and that exposure to position bias depends on how the AI agent is configured.
AI snapshot
Written from the paper’s full text.
TL;DR
AI agents searching hotel listings inspect more deeply than humans and never decline to book, yet randomized position still predicts inspection—weakly and often nonmonotonically. Position effects on final choice vary by LLM and shrink with higher reasoning effort, while all models converge on the same undominated listing.
Key findings
- AI agents searched more than humans: inspections per session ranged from 1.63 to 5.83 versus 1.12 for humans; sessions with only one inspection occurred in 0.8%–37.4% of AI sessions versus 93% of human sessions. All AI agents booked in 100% of sessions, versus 66% human conversion and 34% outside option.
- Position predicted inspection for every LLM, with coefficients from -0.0002 to -0.0005 (all ps<0.001), four to ten times weaker than the human benchmark of -0.0019. Moving a listing down ten ranks cost humans about 1.9 percentage points of inspection probability but AI agents only 0.2–0.5 points. For three of four LLMs, inspection fell from the top to a minimum around ranks 68–74 and then rose again; Pro was mostly a primacy effect.
- Position effects on final choice were heterogeneous: not significant for Flash (-0.000014, p=0.338) or Pro (-0.000008, p=0.606), but significant for Sonnet (-0.000035, p=0.018) and Flash Lite (-0.000080, p<0.001). Average chosen rank ranged from 44.0 to 49.7 against the position-neutral value of 50.5. All LLMs converged on the same undominated listing, citizenM New York Times Square, which captured 78.2% of all bookings pooled across LLMs (1,563 of 2,000 sessions).
- Higher reasoning effort reduced position effects to insignificance at the highest level for both tested LLMs. Search depth moved in opposite directions (Flash Lite fell from 3.12 to 1.91 inspections; Pro rose from 0.39 to 5.83), and average chosen rank converged toward 50.5 (Flash Lite 43.97 to 51.20; Pro 40.96 to 49.72).
Methods
- The experiment randomized the presentation order of 100 hotel listings in each session. LLM agents could call an inspect tool for more hotel details and a submit_choice tool to book; they could also terminate without booking. The context replicated Ursu (2018), with trip parameters for two adults, no children, one room, and a two-night stay including a Saturday night.
- Main sample: four proprietary LLMs from Google (Gemini 3.1 Pro, Gemini 3.7 Flash, Gemini 3.1 Flash Lite) and Anthropic (Claude Sonnet 5), with 500 independent sessions per model (2,000 total) under default reasoning and sampling parameters. Follow-ups manipulated reasoning effort across available levels for Gemini 3.1 Flash Lite (four levels) and Gemini 3.1 Pro (three levels), with 500 replications per cell (3,500 total), and a reduced-prompt variation with 500 additional Flash Lite sessions.
- Analysis used linear probability models with standard errors clustered at the session level. Position was indexed 1–100; controls included price, review score, chain, and promotion. Quadratic position models tested nonlinearity.
Limitations
- Stated: the study fixed the query and destination in a hotel booking task; other tasks with differing attribute complexity could produce different results.
- Stated: the pattern is consistent with lost-in-the-middle, but the paper does not identify it as the mechanism; doing so would require attention-level access the authors did not have. The LLMs tested are also moving targets, and their positional biases may be reduced or eliminated in future releases.
- Evident in the text: the designs differ in list length—100 listings in this study versus roughly seven impressions on average in the human benchmark—so the per-rank slope is estimated over a much longer span in the AI data.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
4 of 19 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured
- Interpretationreasoning
Lost-in-the-middle is an interpretation consistent with the observed U-shaped inspection pattern rather than a mechanism the study identifies.
“Since we do not observe attention directly, lost-in-the-middle is an interpretation consistent with this pattern rather than a mechanism we identify.”
- Also stated in: § 4. Conclusion · plain
“we observe a pattern consistent with lost-in-the-middle, but we do not identify it as the mechanism.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Liu et al. 2024 source
- Also stated in: § 4. Conclusion · plain
- Findingown data
For the tested LLMs except Gemini 3.1 Pro, the position effect on inspection is U-shaped: inspection declines from the top of the listing page to a minimum around ranks 68 to 74 and then rises again.
“Rather than declining monotonically, for the LLMs tested (except Gemini 3.1 Pro) inspection falls from the top of the page to a minimum around ranks 68 to 74 and then rises again, so the bottom of the listing page is a better place to be than the middle (a phenomenon called the lost-in-the-middle effect; Liu et al. 2024).”
- Also stated in: § 3.4.3. Nonlinear position effects · plain
“Inspection declines from the top of the listing page to a minimum between ranks 68 and 74 before increasing for the three LLMs with significant quadratic terms (Figure W2, left panel).”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Liu et al. 2024 source
- Rests on an assumption: AI agents can serve as valid surrogates for human consumers in the hotel search environment, such that their behavior is comparable to human field data.
- Rests on an assumption: The inspect tool call is a valid analogue of a human click/inspection.
- Also stated in: § 3.4.3. Nonlinear position effects · plain
- Interpretationown data
Higher reasoning effort can help mitigate the lost-in-the-middle effect in LLMs.
“This provides evidence that higher reasoning effort can help mitigate the lost-in-themiddle effect in LLMs.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Liu et al. 2024 source
Section 2.3 documents the lost-in-the-middle effect, and Section 4.3 reports that instruction fine-tuning slightly reduced positional bias in some models. The paper does not examine higher reasoning effort or show that it mitigates the effect; that mechanism is the citing authors’ inference, not a finding of this source. (LLM, conf=0.86)
Experimental — accuracy not yet measured
- Premisecitation
LLMs retrieve information from the beginning and end of a long context more reliably than information placed in the middle (lost-in-the-middle).
“Information placed in the middle of a long context is retrieved less reliably than information at either end (Liu et al. 2024).”
- Also stated in: § 2. Background · plain
“LLMs retrieve information from the beginning and end of a long context more reliably than information placed in the middle (Liu et al. 2024).”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Liu et al. 2024 source
- Also stated in: § 2. Background · plain
- Findingown data
Increasing reasoning effort reduces the position effect on final choice to insignificance in both the least and most capable LLMs tested.
“Increasing reasoning effort reduces the position effect to insignificance in both the least and most capable LLMs tested.”
- Also stated in: § 3.4.4. Reasoning effort · plain
“For the final choice, increasing the reasoning effort reduces the position effects for both LLMs and neither is significant at the highest effort level.”
Quote in the paperSubstantial claim - Also stated in: § 3.4.4. Reasoning effort · plain
- Findingown data
Under the alternative prompt, the position effect retains the same shape, magnitude, and significance as under the original prompt.
“The position effect retains the same shape, magnitude, and significance.”
- Also stated in: § 3.4.5. Prompt variation · plain
“The linear term is negative and the quadratic term positive for both inspection (-0.1514, p<0.001 and 0.1086, p<0.001) and the final choice (-0.0626, p<0.001 and 0.0524, p<0.001), matching the original prompt (Table W6).”
Quote in the paperSubstantial claim - Also stated in: § 3.4.5. Prompt variation · plain
- Findingown data
In the hotel search environment, AI agents search more deeply than human consumers, with mean inspections per session ranging from 1.63 to 5.83 across the tested LLMs versus 1.12 for humans.
“AI agents search more deeply than humans and never decline to buy.”
- Also stated in: § 1. Introduction · plain
“AI agents search substantially more (between 1.63 and 5.83 inspections per session) than humans (1.12 inspections per session).”
- Also stated in: § 3.4. Results · plain
“Human consumers inspect 1.12 hotels per session, and 93% of their sessions contain a single inspection.Every AI agent searches more than this, with means ranging from 1.63 to 5.83 inspections per session (RQ1).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ursu 2018 source
- Rests on an assumption: AI agents can serve as valid surrogates for human consumers in the hotel search environment, such that their behavior is comparable to human field data.
- Rests on an assumption: The human benchmark from Ursu (2018) is comparable to the AI agent experiment despite differences in list length (100 vs ~7) and platform.
- Rests on an assumption: The inspect tool call is a valid analogue of a human click/inspection.
- Also stated in: § 1. Introduction · plain
- Findingown data
The average rank of the chosen hotel ranges from 44.0 to 49.7 across LLMs, against the position-neutral value of 50.5.
“The average rank of the chosen hotel ranges from 44.0 to 49.7 (against the position-neutral value of 50.5).”
- Also stated in: § 4. Conclusion · plain
“Position of the chosen hotel | 47.58 | 43.97 | 49.46 | 49.72 | - |”
Quote in the paperSubstantial claim - Also stated in: § 4. Conclusion · plain
- Findingown data
The U-shape for the inspection curve loses its curvature as reasoning effort increases.
“The U-shape for the inspection curve accordingly loses its curvature as effort increases (Fig. 2, left panels).”
Quote in the paperSubstantial claim - Interpretationown data
The recommendation and inspection instructions do not account for the reported findings.
“Therefore, the recommendation and inspection instructions do not account for the reported findings.”
Quote in the paperSubstantial claim - Interpretationreasoning
Out-of-the-box exposure to position depends on how an LLM is configured and not on capability alone.
“out-of-the-box exposure to position depends on how an LLM is configured and not on capability alone.”
- Also stated in: § 4. Conclusion · plain
“out-of-the-box exposure to ranking depends on configuration and not on capability alone.”
Quote in the paperSubstantial claim - Also stated in: § 4. Conclusion · plain
- Interpretationreasoning
Delegating to AI agents breaks the link between inspection and choice that makes rankings economically valuable for human consumers.
“we show that delegating to AI agents breaks the link between inspection and choice, which used to make rankings economically valuable.”
- Also stated in: § 4. Conclusion · plain
“Position therefore shapes which alternatives enter the consideration set for every LLM, while its influence on the final booking varies across LLMs.”
Quote in the paperSubstantial claim - Also stated in: § 4. Conclusion · plain
- Premisecitation
The human benchmark position coefficient on inspection is -0.0019.
“whereas the human benchmark is -0.0019 (p<0.001).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ursu 2018 source
- Interpretationown data
For agentic search, the attributes displayed on a results page matter more than placement within it.
“For agentic search, the attributes displayed on a results page matter more than placement within it.”
Quote in the paperSubstantial claim - Interpretationreasoning
Reasoning effort accounts for the curvature observed for Flash Lite because Flash Lite's default reasoning effort is set to the lowest level (minimal) by Google.
“It accounts for the curvature observed for Flash Lite in the previous section, because Flash Lite's default reasoning effort is set to the lowest level (minimal) by Google.”
Quote in the paperSubstantial claim - Premisecitation
For human consumers, an effect on the consideration set is an effect on demand because the consideration set mostly holds one alternative.
“For human consumers, an effect on the consideration set is an effect on demand, because the consideration set mostly holds one alternative.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ursu 2018 source
- May combine two claims. 19
- The quote only partly states this claim. 1
- Classified as probably not a substantial claim. 4
May combine two claims.
Findingown dataUnder the alternative prompt, search depth is marginally lower (M=2.92 vs. M=3.12) but remains far above the human benchmark (M=1.12).
“Search depth is marginally lower ( M Alternative = 2.92 vs. M Original = 3.12 ) but remains far above the human benchmark ( M Human = 1.12 ).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Ursu 2018 source
- Rests on an assumption: The human benchmark from Ursu (2018) is comparable to the AI agent experiment despite differences in list length (100 vs ~7) and platform.
- Rests on an assumption: The inspect tool call is a valid analogue of a human click/inspection.
May combine two claims.
Findingown dataThe nonlinearity in position effect is confined to inspection and does not systematically transmit to the final choice.
“The nonlinearity is confined to inspection and does not systematically transmit to the final choice.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe position coefficient on inspection is negative and significant for all tested LLMs, ranging from -0.0002 to -0.0005, which is four to ten times smaller than the human benchmark of -0.0019.
“The position coefficient on inspection is negative and significant for all LLMs, ranging from -0.0002 to -0.0005 (all ps<0.001;Table 2, left panel), whereas the human benchmark is -0.0019 (p<0.001).”
- Also stated in: § 1. Introduction · plain
“rank nonetheless continues to predict inspection for every LLM tested, although four to ten times more weakly than for humans.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ursu 2018 source
- Rests on an assumption: AI agents can serve as valid surrogates for human consumers in the hotel search environment, such that their behavior is comparable to human field data.
- Rests on an assumption: The human benchmark from Ursu (2018) is comparable to the AI agent experiment despite differences in list length (100 vs ~7) and platform.
- Rests on an assumption: The inspect tool call is a valid analogue of a human click/inspection.
- Also stated in: § 1. Introduction · plain
May combine two claims.
Findingown dataAll tested LLMs converge on the same undominated listing, citizenM New York Times Square, which captures 78.2% of all bookings (1,563 of 2,000 sessions).
“All models nonetheless converge on the same undominated listing.”
- Also stated in: § Additional information · plain
“The modal choice, citizenM New York Times Square, is undominated on price and review score, with the highest review score in the listing (4.7) at the lowest nightly price for that score ($220).Its choice shares range from 55.4% to 100%, and pooled across LLMs this single listing captures 78.2% of all choices (1,563 of 2,000 sessions).”
- Also stated in: § 4. Conclusion · plain
“All four LLMs nonetheless converged on the same undominated listing, which captured 78.2% of all bookings.”
Quote in the paperSubstantial claim- Rests on an assumption: AI agents can serve as valid surrogates for human consumers in the hotel search environment, such that their behavior is comparable to human field data.
- Rests on an assumption: The four tested LLMs are representative of AI agents used for shopping.
- Also stated in: § Additional information · plain
May combine two claims.
Findingown dataUnder the alternative prompt (without recommendation or inspection instructions), conversion remains at 100% and the outside option is never selected.
“Conversion remains at 100% and the outside option is never selected (Table 3).”
Quote in the paperSubstantial claim- Rests on an assumption: The submit_choice tool and outside option in AI sessions are equivalent to human booking and outside option.
May combine two claims. The quote only partly states this claim.
Findingown dataIn the hotel search environment, AI agents never declined to book a hotel, booking in 100% of sessions and selecting the outside option in 0% of sessions.
“AI agents search more deeply than humans and never decline to buy.”
- Also stated in: § 3.4. Results · plain
“all AI agents booked in 100% of sessions.”
- Also stated in: § 4. Conclusion · plain
“Outside option selected | 0.0% | 0.0% | 0.0% | 0.0% | 34.0% |”
Quote in the paperSubstantial claim- Rests on an assumption: AI agents can serve as valid surrogates for human consumers in the hotel search environment, such that their behavior is comparable to human field data.
- Rests on an assumption: The submit_choice tool and outside option in AI sessions are equivalent to human booking and outside option.
- Also stated in: § 3.4. Results · plain
May combine two claims.
Findingown dataWith reasoning effort manipulation, conversion rate remains at 100% and the outside option is never selected.
“With reasoning effort manipulation, conversion rate remains at 100% and the outside option is never selected (Table 3).”
Quote in the paperNot a substantial claim- Rests on an assumption: The submit_choice tool and outside option in AI sessions are equivalent to human booking and outside option.
May combine two claims.
Findingown dataSearch depth changes in opposite directions with reasoning effort across the two tested LLMs: it falls from 3.12 to 1.91 inspections per session for Flash Lite and rises from 0.39 to 5.83 for Pro.
“Search depth falls from 3.12 to 1.91 inspections per session for Flash Lite and rises from 0.39 to 5.83 for Pro (Table 3; Table W5).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe modal choice, citizenM New York Times Square, is undominated on price and review score, with the highest review score in the listing (4.7) at the lowest nightly price for that score ($220).
“The modal choice, citizenM New York Times Square, is undominated on price and review score, with the highest review score in the listing (4.7) at the lowest nightly price for that score ($220).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataEvery LLM concentrates between 89.6% and 100% of its choices in the same five hotels out of 100, and shares the same modal choice.
“Every LLM concentrates between 89.6% and 100% of its choices in the same five hotels out of 100, and shares the same modal choice (Table W3).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe effect of position on the final choice varies by LLM: it is indistinguishable from zero for Flash and Pro, significant but smaller than humans for Sonnet, and significant and similar magnitude to humans for Flash Lite.
“For Flash and Pro, it is indistinguishable from zero (-0.000014, p=0.338 and -0.000008, p=0.606, respectively).”
- Also stated in: § Additional information · plain
“Sonnet, it is significant (-0.000035, p=0.018) but smaller than humans (-0.0001, p<0.001).For Flash Lite, it is significant (-0.000080, p<0.001) and similar magnitude to humans (Table 2,”
- Also stated in: § 4. Conclusion · plain
“At the booking stage, rank mattered for two of the four LLMs (Sonnet and Flash Lite) but not for the other two (Flash and Pro).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ursu 2018 source
- Rests on an assumption: AI agents can serve as valid surrogates for human consumers in the hotel search environment, such that their behavior is comparable to human field data.
- Rests on an assumption: The human benchmark from Ursu (2018) is comparable to the AI agent experiment despite differences in list length (100 vs ~7) and platform.
- Also stated in: § Additional information · plain
May combine two claims.
Findingown dataIncreasing reasoning effort moves the average rank of the chosen hotel toward the position-neutral value of 50.5 (from 43.97 to 51.20 for Flash Lite and from 40.96 to 49.72 for Pro) and raises the modal choice proportion (from 61.4% to 96.6% and from 48.0% to 55.4%).
“Furthermore, the average rank of the chosen hotel converges toward the positionneutral value of 50.5 (from 43.97 to 51.20 for Flash Lite and from 40.96 to 49.72 for Pro), and the modal choice proportion rises from 61.4% to 96.6% and from 48.0% to 55.4% (Table 3).”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationown dataThe study's fixed query and destination in a hotel booking task limit generalizability; other tasks with differing attribute complexity could produce different results.
“Our study has several limitations.We fixed the query and destination in a hotel booking task.Other tasks with differing levels of attribute complexity could produce different results.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationLLMs display heuristics and biases in consumption settings.
“although the same LLMs have been shown to display heuristics and biases in consumption settings (Wadi et al. 2026b).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Wadi et al. 2026b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
PremisecitationSampling parameters can no longer be manipulated for most modern LLMs; Google and Anthropic APIs return errors if non-default sampling parameters are used.
“Sampling parameters can no longer be manipulated for most modern LLMs.Google and Anthropic APIs return errors if non-default sampling parameters are used (Anthropic 2026; Google 2026).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Anthropic 2026
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Google 2026
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
Study design and conductreasoningThe presentation order of the 100 hotels was fully randomized for each session, eliminating endogeneity bias inherent in search engine ranking algorithms.
“To eliminate the endogeneity bias inherent in search engine ranking algorithms, where higher-quality options are systematically placed at the top, the presentation order of the 100 hotels was fully randomized for each session.”
- Also stated in: § 3.4.1. Position effect on inspection · boosted
“Because we randomize the order of the 100 hotels in every session, any relationship between position and behavior is causal by construction.”
Quote in the paperSubstantial claim- Also stated in: § 3.4.1. Position effect on inspection · boosted
May combine two claims.
PremisecitationReview scores remain a primary heuristic and trust indicator in online decision environments, driving choice even when rankings shift.
“review scores, which remain a primary heuristic and trust indicator in online decision environments (Wadi et al. 2026a), driving choice even when rankings shift.”
- Also stated in: § 4. Conclusion · plain
“review scores, which remain a primary heuristic and trust indicator in online decision environments (Wadi et al. 2026c), driving choice even when rankings shift.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Wadi et al. 2026a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Wadi et al. 2026c
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Also stated in: § 4. Conclusion · plain
Classified as probably not a substantial claim.
Study design and conductreasoningStandard errors in the reduced-form models were clustered at the session level.
“Because the 100 hotel observations within a session were not independent, standard errors in the reduced-form models were clustered at the session level.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationHuman consumers who click at least once book 66% of the time and take the outside option 34% of the time.
“human consumers who click at least once book 66% of the time and take the outside option 34% of the time”
- Also stated in: § 4. Conclusion · plain
“Outside option selected | 0.0% | 0.0% | 0.0% | 0.0% | 34.0% |”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ursu 2018 source
- Also stated in: § 4. Conclusion · plain
May combine two claims.
PremisecitationHuman consumers inspect 1.12 hotels per session, and 93% of their sessions contain a single inspection.
“Research shows that with hotel search, humans make on average 1.12 clicks and 93% of sessions contain only one click (Ursu 2018).”
- Also stated in: § 3.4. Results · plain
“Human consumers inspect 1.12 hotels per session, and 93% of their sessions contain a single inspection.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ursu 2018 source
- Also stated in: § 3.4. Results · plain
Classified as probably not a substantial claim.
PremisecitationRank determines which alternatives enter the consideration set rather than what they are worth after they enter.
“The canonical explanation is that ranking determines which alternatives enter the consideration set while leaving the consumer's valuation of an inspected alternative untouched (Ursu 2018).”
- Also stated in: § 2. Background · plain
“Rank, or position, determines which alternatives enter the consideration set rather than what they are worth after they enter.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ursu 2018 source
- Also stated in: § 2. Background · plain
May combine two claims.
Study design and conductcitationThe designs differ in list length: our page shows 100 listings, while impressions in the human benchmark average roughly seven, so the per-rank slope is estimated over a much longer span in our data.
“The designs differ in list length: our page shows 100 listings, while impressions in the human benchmark average roughly seven, so the per-rank slope is estimated over a much longer span in our data.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ursu 2018 source
The paper relies on this without support.
AI agents can serve as valid surrogates for human consumers in the hotel search environment, such that their behavior is comparable to human field data.
“We study this by placing LLM agents in a hotel search environment, comparable to a human field benchmark.”
“To ensure comparability with an established human benchmark, we replicate the context of the randomized field experiment conducted by Ursu (2018).”
The paper relies on this without support.
The human benchmark from Ursu (2018) is comparable to the AI agent experiment despite differences in list length (100 vs ~7) and platform.
“The designs differ in list length: our page shows 100 listings, while impressions in the human benchmark average roughly seven, so the per-rank slope is estimated over a much longer span in our data.”
The paper relies on this without support.
The inspect tool call is a valid analogue of a human click/inspection.
“Inspected was coded 1 when the agent invoked the inspect tool for a given hotel and 0 otherwise (an analogue of a click in the human benchmark).”
The paper relies on this without support.
The submit_choice tool and outside option in AI sessions are equivalent to human booking and outside option.
“At any point, the LLM could use the submit_choice tool to book a hotel.It could also terminate the session without making a booking, representing the outside option.”
The paper relies on this without support.
The four tested LLMs are representative of AI agents used for shopping.
“The main experiment comprises 2,000 sessions across four major LLMs from two providers (i.e., Google and Anthropic).”
The paper relies on this without support.
AI agents receive the entire results page at once and do not scroll, so they pay no marginal cost to read later listings.
“An AI agent can receive the entire results page at once, in its context window.It does not scroll and pays no marginal cost to read the hundredth listing compared to the first.”
The paper relies on this without support.
The Google LLMs represent distinct levels of computational capability, with Gemini 3.1 Pro as highest-capability and Gemini 3.1 Flash Lite as lowest-capability.
“The LLMs from Google represent distinct levels of computational capability.Gemini 3.1 Pro is Google's highest-capability LLM and is expected to facilitate complex, multi-attribute calculations.In contrast, Gemini 3.1 Flash Lite is the lowest-capability LLM, designed for lightweight and efficient tasks.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Introduction says rank does not affect booking, but results say it does for two models
MajorThe introduction states that position does not affect final choice for AI agents. The conclusion reports that rank mattered at the booking stage for Sonnet and Flash Lite.
“For AI agents, rank increases inspection but does not affect the final choice.”
“At the booking stage, rank mattered for two of the four LLMs (Sonnet and Flash Lite) but not for the other two (Flash and Pro).”
Cites
The Effect of Word of Mouth on Sales: Online Book Reviews
Chevalier, J. A., & Mayzlin, D. (2006) · Journal of Marketing Research · doi:10.1509/jmkr.43.3.345 · Cited by 6,003 · cited 1× · see citing works
“AI agents concentrated their bookings on listings with high review scores and low prices (Table W4), which mirrors human consumers (Chevalier and Mayzlin, 2006; Floyd et al., 2014) and is consistent with other studies of AI shopping agents (Allouah et al., 2026; Cherep et al., 2026; Wadi and Ma,…”
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957 · cited 1× · see citing works
“A I] 8 O c t 2026 information at either end (Liu et al., 2024; Su et al., 2024).”
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · cited 4× · see citing works
“AI agents’ position bias is concentrated in the middle rather than at the bottom of the page (Ursu, 2018; Liu et al., 2024).”
InfluentialFirm-Created Word-of-Mouth Communication: Evidence from a Field Test
Godes, D., & Mayzlin, D. (2009) · Marketing Science · doi:10.1287/mksc.1080.0444 · Cited by 837 · cited 1× · see citing works
“Sellers may therefore benefit more from strengthening these signals (e.g., by soliciting reviews; Godes and Mayzlin, 2009; Wadi et al., 2026c,a; Burtch et al., 2018) than from improving their placement.”
Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search
Joachims, T., Granka, L. A., Pan, B., Hembrooke, H. A., Radlinski, F., & Gay, G. K. (2007) · ACM Transactions on Information Systems · doi:10.1145/1229179.1229181 · Cited by 678 · cited 2× · see citing works
“This is because scanning a listing page is sequential, scrolling takes effort (Joachims et al., 2007; Greminger, 2026), multi-attribute comparison is cognitively demanding (Punj and Moore, 2009), and attention is depleted over the course of a search episode (Ursu et al., 2023).”
How Online Product Reviews Affect Retail Sales: A Meta-analysis
Floyd, K., Freling, R. E., Alhoqail, S. A., Cho, H. Y., & Freling, T. H. (2014) · Journal of Retailing · doi:10.1016/j.jretai.2014.04.004 · Cited by 631 · cited 1× · see citing works
“…their bookings on listings with high review scores and low prices (Table W4), which mirrors human consumers (Chevalier and Mayzlin, 2006; Floyd et al., 2014) and is consistent with other studies of AI shopping agents (Allouah et al., 2026; Cherep et al., 2026; Wadi and Ma, 2026b;…”
Stimulating Online Reviews by Combining Financial Incentives and Social Norms
Burtch, G., Hong, Y., Bapna, R., & Griskevicius, V. (2016) · Management Science · doi:10.1287/mnsc.2016.2715 · Cited by 399 · cited 1× · see citing works
“Sellers may therefore benefit more from strengthening these signals (e.g., by soliciting reviews; Godes and Mayzlin, 2009; Wadi et al., 2026c,a; Burtch et al., 2018) than from improving their placement.”
Ursu, R. M. (2018) · Marketing Science · doi:10.1287/mksc.2017.1072 · Cited by 328 · cited 14× · see citing works
“Human benchmark data adapted from Ursu (2018), Table 1; the human sample includes only impressions with one or more clicks.”
InfluentialGemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05530 · Cited by 306 · cited 1× · see citing works
“However, the large language models (LLMs) that power the AI agents display heuristics and biases of their own (Brady et al., 2025; Wadi et al., 2026b; Macmillan-Scott and Musolesi, 2024; Wadi and Ma, 2026a; Allouah et al., 2026), including a tendency to retrieve information placed in the middle of…”
Consideration-set heuristics ☆
Hauser, J. (2014) · Journal of Business Research · doi:10.1016/j.jbusres.2014.02.015 · Cited by 250 · cited 1× · see citing works
“The canonical explanation is that ranking determines which alternatives enter the consideration set while leaving the consumer’s valuation of an inspected alternative unchanged (Hauser, 2014; Akchen and Mitrofanov, 2026; Ursu, 2018).”
Using LLMs for Market Research
Brand, J., Israeli, A., & Ngwe, D. (2023) · doi:10.2139/ssrn.4395751 · Cited by 190 · see citing works
Toward large reasoning models: A survey of reinforced reasoning with large language models
Xu, F., Hao, Q., Shao, C., Zong, Z., Li, Y., Wang, J., Zhang, Y., Wang, J., Lan, X., Gong, J., et al. (2025) · Patterns · doi:10.1016/j.patter.2025.101370 · Cited by 186 · cited 1× · see citing works
“Users can set the reasoning effort that the model expends before acting (Xu et al., 2025; Alomrani et al., 2025), which raises the question of whether position effects can be mitigated through configuration of the LLM.”
Frontiers: Determining the Validity of Large Language Models for Automated Perceptual Analysis
Li, P., Castelo, N., Katona, Z., & Sárváry, M. (2024) · Marketing Science · doi:10.1287/mksc.2023.0454 · Cited by 165 · cited 1× · see citing works
“This included the hotel’s star rating, specific amenities, granular sub-ratings for cleanliness and service, and detailed room availability and room price.”
Position Effects in Search Advertising and their Moderators: A Regression Discontinuity Approach
Narayanan, S., & Kalyanam, K. (2015) · Marketing Science · doi:10.1287/mksc.2014.0893 · Cited by 157 · cited 1× · see citing works
“Search engine optimization budgets and the premium paid for top slots rely on the well-documented finding that people examine and purchase higher-placed alternatives more often (Narayanan and Kalyanam, 2015; Joachims et al., 2007; Donnelly et al., 2024).”
Consumer Click Behavior at a Search Engine: The Role of Keyword Popularity
Jerath, K., Ma, L.-Y., & Park, Y.-H. (2014) · Journal of Marketing Research · doi:10.1509/jmr.13.0099 · Cited by 156 · see citing works
Mogaji, E., & Jain, V. (2024) · Journal of Consumer Behaviour · doi:10.1002/cb.2345 · Cited by 120 · cited 1× · see citing works
“Rather than sifting a results page themselves, consumers increasingly rely on conversational agents that retrieve, compare, and transact on their behalf (Mogaji and Jain, 2024; Lee et al., 2026; Kumar et al., 2026; Hasselwander et al., 2026).”
Frontiers: Can Large Language Models Capture Human Preferences?
Goli, A., & Singh, A. (2024) · Marketing Science · doi:10.1287/mksc.2023.0306 · Cited by 115 · cited 1× · see citing works
“To eliminate the endogeneity bias inherent in search engine ranking algorithms, where higher-quality options are systematically placed at the top, the presentation order of the 100 hotels was fully randomized for each session.”
Information search and consideration set formation in a web-based store environment
Punj, G., & Moore, R. (2008) · Journal of Business Research · doi:10.1016/j.jbusres.2007.04.013 · Cited by 84 · cited 1× · see citing works
“This is because scanning a listing page is sequential, scrolling takes effort (Joachims et al., 2007; Greminger, 2026), multi-attribute comparison is cognitively demanding (Punj and Moore, 2009), and attention is depleted over the course of a search episode (Ursu et al., 2023).”
(Ir)rationality and cognitive biases in large language models
Macmillan-Scott, O., & Musolesi, M. (2024) · Royal Society Open Science · doi:10.1098/rsos.240255 · Cited by 55 · see citing works
Search Gaps and Consumer Fatigue
Ursu, R. M., Zhang, Q., & Honka, E. (2022) · Marketing Science · doi:10.1287/mksc.2022.1359 · Cited by 49 · cited 1× · see citing works
“This is because scanning a listing page is sequential, scrolling takes effort (Joachims et al., 2007; Greminger, 2026), multi-attribute comparison is cognitively demanding (Punj and Moore, 2009), and attention is depleted over the course of a search episode (Ursu et al., 2023).”
Welfare Effects of Personalized Rankings
Donnelly, R., Kanodia, A., & Morozov, I. (2023) · Marketing Science · doi:10.1287/mksc.2023.1441 · Cited by 41 · cited 1× · see citing works
“Search engine optimization budgets and the premium paid for top slots rely on the well-documented finding that people examine and purchase higher-placed alternatives more often (Narayanan and Kalyanam, 2015; Joachims et al., 2007; Donnelly et al., 2024).”
Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
Alomrani, M. A., Zhang, Y., Li, D., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2507.02076 · Cited by 34 · cited 1×
“Users can set the reasoning effort that the model expends before acting (Xu et al., 2025; Alomrani et al., 2025), which raises the question of whether position effects can be mitigated through configuration of the LLM.”
Consumer Search: What Can We Learn from Pre-Purchase Data?
Honka, E., Seiler, S., & Ursu, R. (2024) · Social Science Research Network · doi:10.2139/ssrn.4649002 · Cited by 32 · cited 1×
“Moreover, because human consumers typically click on a single alternative (Honka et al., 2024), an effect of rank on clicking is effectively an effect on choice.”
Toward agentic AI: User acceptance of a deeply personalized AI super assistant (AISA)
Hasselwander, M., Sunio, V., Lah, O., & Mogaji, E. (2025) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2025.104620 · Cited by 25 · cited 1× · see citing works
“Rather than sifting a results page themselves, consumers increasingly rely on conversational agents that retrieve, compare, and transact on their behalf (Mogaji and Jain, 2024; Lee et al., 2026; Kumar et al., 2026; Hasselwander et al., 2026).”
Dual-process theory and decision-making in large language models
Brady, O., Nulty, P., Zhang, L., Ward, T. E., & McGovern, D. P. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-025-00506-1 · Cited by 16 · cited 1× · see citing works
“However, the large language models (LLMs) that power the AI agents display heuristics and biases of their own (Brady et al., 2025; Wadi et al., 2026b; Macmillan-Scott and Musolesi, 2024; Wadi and Ma, 2026a; Allouah et al., 2026), including a tendency to retrieve information placed in the middle of…”
The sequential search model: A framework for empirical research
Ursu, R. M., Seiler, S., & Honka, E. (2024) · Quantitative Marketing and Economics · doi:10.1007/s11129-024-09291-2 · Cited by 14 · see citing works
Kumar, V., Kotler, P., & Kumar, A. (2025) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-025-01120-6 · Cited by 9 · cited 1× · see citing works
“Rather than sifting a results page themselves, consumers increasingly rely on conversational agents that retrieve, compare, and transact on their behalf (Mogaji and Jain, 2024; Lee et al., 2026; Kumar et al., 2026; Hasselwander et al., 2026).”
Balaskas, S. (2026) · Information · doi:10.3390/info17030222 · Cited by 8 · cited 1× · see citing works
“Consumers are now delegating parts of this search to AI agents (Balaskas, 2026; Wadi and Ma, 2026a).”
Wadi, D., & Fredette, M. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.findings-emnlp.500 · Cited by 7
Self-citationPrior Information and Consumer Search: Evidence from Eye-trackingPreprint
Zhang, Q. P., Ursu, R., & Erdem, T. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3717885 · Cited by 6 · see citing works
Whom Do AI Agents Serve? Role-Induced Sponsorship Bias in Decision Models and Generative LLMsPreprint
Wadi, D., & Ma, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.17989 · Cited by 4 · cited 2× · see citing works
“…their bookings on listings with high review scores and low prices (Table W4), which mirrors human consumers (Chevalier and Mayzlin, 2006; Floyd et al., 2014) and is consistent with other studies of AI shopping agents (Allouah et al., 2026; Cherep et al., 2026; Wadi and Ma, 2026b; Kireyev, 2026).”
Self-citationTrade-Offs Between Ranking Objectives: Descriptive Evidence and Structural Estimation
Greminger, R. (2022) · doi:10.48550/arxiv.2210.16408 · Cited by 4 · cited 1×
“This is because scanning a listing page is sequential, scrolling takes effort (Joachims et al., 2007; Greminger, 2026), multi-attribute comparison is cognitively demanding (Punj and Moore, 2009), and attention is depleted over the course of a search episode (Ursu et al., 2023).”
Consider or Choose? The Role and Power of Consideration Sets
Akchen, Y.-C., & Mitrofanov, D. (2025) · Management Science · doi:10.1287/mnsc.2023.00442 · Cited by 4 · see citing works
Consider or Choose? The Role and Power of Consideration Sets
Akchen, Y.-C., & Mitrofanov, D. (2023) · doi:10.48550/arxiv.2302.04354 · Cited by 3 · cited 1×
“The canonical explanation is that ranking determines which alternatives enter the consideration set while leaving the consumer’s valuation of an inspected alternative unchanged (Hauser, 2014; Akchen and Mitrofanov, 2026; Ursu, 2018).”
Lee, W.-C., Lee, J.-S., & Suh, J. (2026) · Applied Sciences · doi:10.3390/app16062816 · Cited by 2 · cited 1× · see citing works
“Rather than sifting a results page themselves, consumers increasingly rely on conversational agents that retrieve, compare, and transact on their behalf (Mogaji and Jain, 2024; Lee et al., 2026; Kumar et al., 2026; Hasselwander et al., 2026).”
Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases
Wadi, D., Ghodrat, M., & Philp, M. (2026) · doi:10.48550/arxiv.2608.10503 · Cited by 2 · cited 1×
“This included the hotel’s star rating, specific amenities, granular sub-ratings for cleanliness and service, and detailed room availability and room price.”
Self-citationPriceBench: A Diagnostic Benchmark for Price, Quality, and Brand Preferences in LLM Booking Agents
Kireyev, P. (2026) · doi:10.48550/arxiv.2609.31468 · Cited by 1 · cited 1×
“…their bookings on listings with high review scores and low prices (Table W4), which mirrors human consumers (Chevalier and Mayzlin, 2006; Floyd et al., 2014) and is consistent with other studies of AI shopping agents (Allouah et al., 2026; Cherep et al., 2026; Wadi and Ma, 2026b; Kireyev, 2026).”
Hengle, A., Bajpai, P., Dan, S., et al. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.290 · Cited by 1 · cited 1×
“Unlike a human consumer, an AI agent receives the entire results page at once within its context window (Hengle et al., 2026), so inspecting a listing at the bottom of the page requires no scrolling.”
Allouah, A., Besbes, O., Figueroa, J. D., Kanoria, Y., & Kumar, A. (2025) · arXiv · cited 2×
“…their bookings on listings with high review scores and low prices (Table W4), which mirrors human consumers (Chevalier and Mayzlin, 2006; Floyd et al., 2014) and is consistent with other studies of AI shopping agents (Allouah et al., 2026; Cherep et al., 2026; Wadi and Ma, 2026b; Kireyev, 2026).”
A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments
Cherep, M., Ma, C., Xu, A., Shaked, M., Maes, P., & Singh, N. (2025) · arXiv · cited 1×
“…their bookings on listings with high review scores and low prices (Table W4), which mirrors human consumers (Chevalier and Mayzlin, 2006; Floyd et al., 2014) and is consistent with other studies of AI shopping agents (Allouah et al., 2026; Cherep et al., 2026; Wadi and Ma, 2026b; Kireyev, 2026).”
Shopping by algorithm: How agentic AI deploys human heuristics as a surrogate consumer
Wadi, D., & Ma, Y. (2026) · arXiv · doi:10.48550/arxiv.2609.28372 · cited 2×
“Consumers are now delegating parts of this search to AI agents (Balaskas, 2026; Wadi and Ma, 2026a).”
Self-citationWadi, D., Frédette, M., Sénécal, S., & Legoux, R. (2026) · Journal of Research in Interactive Marketing · doi:10.1108/jrim-04-2026-0253 · cited 1×
“Position effects among human consumers are attributed to the costs of scrolling, which are absent for an AI agent that receives the entire listing page at once, yet position continues to shape inspection.”
Self-citationShopping By Algorithm: How Agentic AI Deploys Human Heuristics as a Surrogate ConsumerPreprint
Wadi, D., & Ma, Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6938899
Self-citation
Flagged first, then by citations
2 of the 45 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Wadi D, Legoux R, Fredette M, Sénécal S (2026c) The interplay of altruism and financial incentives: Maximizing online reviews through effective messaging. Journal of Electronic Commerce Research 27:
Cited by
Sorted by citations
Whom Do AI Agents Serve? Role-Induced Sponsorship Bias in Decision Models and Generative LLMsPreprint
Wadi, D., & Ma, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.17989 · Cited by 4 · see citing works
“Research has shown that LLMs can show position effects (Liu et al., 2024; Pezeshkpour and Hruschka, 2024; Yin et al., 2026; Wadi and Ma, 2026a; Wang et al., 2024).”
Self-citationEvery Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases
Wadi, D., Ghodrat, M., & Philp, M. (2026) · doi:10.48550/arxiv.2608.10503 · Cited by 2
“Recent works have utilized psychometric tests to measure a wide array of constructs (Jiang et al., 2024; Li et al., 2024; Helwe et al., 2025).”
Self-citationLiu, D. (2026) · arXiv.org · doi:10.48550/arxiv.2607.05363 · Cited by 1
“We use SovScore as a reporting index rather than a universal user-utility function: all component metrics are reported separately, paired comparisons are computed on the same scenario-model units, and the appendix checks that conclusions are stable under task-heavy, privacy-heavy, and burden-heavy…”
Huang, H.-N., & Xiao, J. (2026) · doi:10.48550/arxiv.2610.05823
“Benchmarks of shopping-agent behaviour [51, 52] anchor our assistant-side measurements.”
The Default Trap: Rethinking Plan Evaluation in Tool-Using LLM Agents
Li, X.-Q., Ning, J.-J., & Kong, Y.-B. (2026) · doi:10.48550/arxiv.2609.36829
“Shopping agents show distinct positional responses in inspection and final choice (Wadi & Ma, 2026); attention interventions examine tool-selection errors (Chen, 2026).”
Shopping by algorithm: How agentic AI deploys human heuristics as a surrogate consumer
Wadi, D., & Ma, Y. (2026) · arXiv · doi:10.48550/arxiv.2609.28372
Self-citation
- Authors
- Wadi, D., & Ma, Y.
- Affiliations
- McGill University (CA)
- ORCID
- Wadi, Davood
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2608.22697
- OpenAlex
- W7204189884
- Cited by
- 0
- References
- 45
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (cc-by-nc-sa)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Sociology and Political Science › Digital Marketing and Social Media
- Social Sciences › Business, Management and Accounting › Marketing › Consumer Market Behavior and Pricing
- Keywords
- search engine ranking, position bias, AI agents, large language models, agentic search, consumer choice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review