GEO: Generative Engine Optimization

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The experimental setup closely mimics the workflow used in previous works and the general design adopted by commercial GEs such as you.com and perplexity.ai.

“The setup closely mimics the workflow used in previous works and the general design adopted by commercial GEs such as you.com and perplexity.ai.”

• cites [14] 14. Nelson F. Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating Verifiabil- ity in Generative Search Engines. ArXiv abs/2304.09848 (2023). https://api. semanticscholar.org/CorpusID:258212854Supported

Probably not a substantial claim: may not pass S3 (load-bearing).

G-Eval is the current state-of-the-art for evaluation with LLMs.

“We use G-Eval [15], the current state-of-the-art for evaluation with LLMs, to measure each of these sub-metrics.”

• cites [15] 15. Yang Liu, Dan Iter, Yichong Xu, Shuo Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. ArXiv abs/2303.16634 (2023). https://api.semanticscholar.org/CorpusID:257804696 sourcePartly supported

  The Introduction and Conclusion say G-EVAL outperformed existing NLG evaluators in human alignment on the benchmarks studied. The Limitations section specifies that those tests covered text summarization and dialogue generation, so the paper does not establish the broader, time-dependent claim that G-Eval is the current state of the art for evaluation with LLMs generally. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Citations are important given the tendency of LLMs to hallucinate information.

“Citations are important given the tendency of LLMs to hallucinate information [10].”

• cites [10] 10. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. Comput. Surveys 55, 12 (2023), 1-38. sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The answer was generated by gpt3.5-turbo using the same prompt as prior work.

“The answer is then generated by the gpt3.5-turbomodel [20] using the same prompt as prior work [14].”

• cites [20] 20. OpenAI. 2022. Introducing ChatGPT. https://openai.com/index/chatgpt/Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [14] 14. Nelson F. Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating Verifiabil- ity in Generative Search Engines. ArXiv abs/2304.09848 (2023). https://api. semanticscholar.org/CorpusID:258212854Could not verify

  The citation to this paper is attached to “using the same prompt as prior work.” Section 3.1 describes evaluating four commercial generative search engines, but our copy does not identify a prompt for generating answers with gpt-3.5-turbo. Searches for the same prompt, a generation prompt, and gpt-3.5-turbo found no passage establishing that the citing authors used this paper’s prompt. (LLM, conf=0.75)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Strategic text sequences can manipulate LLM recommendations to enhance product visibility in generative engines.

“In a recent working draft, Kumar and Lakkaraju [11] showed that strategic text sequences can manipulate LLM recommendations to enhance product visibility in generative engines.”

• cites [11] 11. Aounon Kumar and Himabindu Lakkaraju. 2024. Manipulating Large Language Models to Increase Product Visibility. arXiv:2404.07981 [cs.IR] sourceSupported

May combine two claims.

The Cite Sources method led to a 115.1% increase in visibility for websites ranked fifth in SERP, while on average the visibility of the top-ranked website decreased by 30.3%.

“For example, the Cite Sources method led to a substantial 115.1% increase in visibility for websites ranked fifth in SERP, while on average, the visibility of the top-ranked website decreased by 30.3%.”

May combine two claims.

GEO-bench contains 80% informational queries and 10% each for transactional and navigational queries.

“We preserve the real-world query distribution, with our benchmark containing 80% informational queries and 10% each for transactional and navigational queries.”

May combine two claims.

GEO-bench consists of queries from 25 diverse domains, includes 9 different types of queries, and encompasses 7 different categorizations.

“Overall, GEO-bench consists of queries from 25 diverse domains such as Arts, Health, and Games; it features a range of query difficulties from simple to multi-faceted; includes 9 different types of queries such as informational and transactional; and encompasses 7 different categorizations.”

May combine two claims.

Domains such as 'Law & Government' and question types like 'Opinion' benefit significantly from the addition of relevant statistics in the website content.

“For example, as shown in row 5 of Table 3, domains such as 'Law & Government' and question types like 'Opinion' benefit significantly from the addition of relevant statistics in the website content, as implemented by Statistics Addition.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The benchmark includes queries from nine different sources, each further categorized based on their target domain, difficulty, query intent, and other dimensions.

“The benchmark includes queries from nine different sources, each further categorized based on their target domain, difficulty, query intent, and other dimensions.”

May combine two claims.

On Perplexity.ai, Cite Sources and Statistics Addition show improvements of up to 9% and 37% on the two metrics.

“Methods that performed well in our generative engine such as Cite Sources, Statistics Addition show improvements of up to 9% and 37% on the two metrics.”

May combine two claims.

The best performing GEO methods on Perplexity.ai improve upon baseline by 22% on Position-Adjusted Word Count and 37% on Subjective Impression.

“The best performing methods improve upon baseline by 22% on Position-Adjusted Word Count and 37% on Subjective Impression.”

May combine two claims.

The best combination of GEO methods is Fluency Optimization and Statistics Addition, and it outperforms any single GEO strategy by more than 5.5%.

“with the best combination (Fluency Optimization and Statistics Addition) outperforming any single GEO strategy by more than 5.5%”

• Also stated in: § Deployed Generative Engine · plain

  “Using Fluency Optimization and Statistics Addition in conjunction results in maximum performance.”

May combine two claims.

The best GEO methods improve upon baseline by 41% on Position-Adjusted Word Count and 28% on Subjective Impression.

“The best methods improve upon baseline by 41% and 28% on Position-Adjusted Word Count and Subjective Impression respectively.”

Classified as probably not a substantial claim.

Each query is augmented with the cleaned text content of the top 5 search results from the Google search engine.

“Each query is augmented with the cleaned text content of the top 5 search results from the Google search engine.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Generative Engines provide rich, structured responses and embed websites as inline citations in the response, often embedding them with different lengths, at varying positions, and with diverse styles.

“Generative Engines provide rich, structured responses and embed websites as inline citations in the response, often embedding them with different lengths, at varying positions, and with diverse styles.”

Probably not a substantial claim: may not pass S2 (checkable).

The notion of visibility in generative engines is more nuanced and multi-faceted than traditional search engines.

“Further, GEO introduces a flexible framework for defining visibility metrics tailor-made for generative engines as the notion of visibility in generative engines is more nuanced and multi-faceted than traditional search engines (Figure 3).”

May combine two claims.

By enhancing their content with GEO, smaller content creators can reach a wider audience, leveling the playing field and allowing them to compete more effectively with larger corporations.

“By enhancing their content with GEO, they can reach a wider audience, leveling the playing field and allowing them to compete more effectively with larger corporations.”

May combine two claims.

Generative engines remove the need to navigate to websites by directly providing a precise and comprehensive response, potentially reducing organic traffic to websites and impacting their visibility.

“Generative Engines, in contrast to traditional search engines, remove the need to navigate to websites by directly providing a precise and comprehensive response, potentially reducing organic traffic to websites and impacting their visibility [16].”

• cites [16] 16. G. D. Maayan. 2023. How Google SGE will impact your traffic -and 3 SGE recovery case studies. Search Engine Land (5 Sep 2023). https://searchengineland.com/how-google-sge-will-impact-your-traffic- and-3-sge-recovery-case-studies-431430Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

GEO methods require minimal changes but significantly improve visibility in GE responses, enhancing both the credibility and richness of the content.

“These methods, involving adding relevant statistics (Statistics Addition), incorporating credible quotes (Quotation Addition), and including citations from reliable sources (Cite Sources) in the website content, require minimal changes but significantly improve visibility in GE responses, enhancing both the credibility and richness of the content.”

Classified as probably not a substantial claim. May combine two claims.

GEO-bench is a comprehensive benchmark for evaluating Generative Engines and serves as a standard testbed for assessing them.

“Owing to its specially designed high diversity, the size of the benchmark, and its real-world nature, GEObench is a comprehensive benchmark for evaluating Generative Engines and serves as a standard testbed for assessing them for various purposes in this and future works.”

Probably not a substantial claim: may not pass S2 (checkable).

Generative engines will significantly disrupt the creator economy.

“With millions of small businesses and individuals relying on online traffic and visibility for their livelihood, generative engines will significantly disrupt the creator economy.”

• AssumptionRests on an assumption: Millions of small businesses and individuals rely on online traffic and visibility for their livelihood.

May combine two claims.

Average ranking on the response page is a good measure of visibility in traditional search engines, but does not apply to generative engines.

“While average ranking on the response page is a good measure of visibility in traditional search engines, which present a linear list of websites, this does not apply to generative engines.”

Probably not a substantial claim: may not pass S2 (checkable).

GEO has the potential to democratize the digital space.

“This finding highlights GEO's potential as a tool to democratize the digital space.”

May combine two claims.

The same GEO methods were evaluated on Perplexity.ai, a commercially deployed generative engine, highlighting the generalizability of the proposed methods.

“Further in Section C.1, we evaluate the same Generative Engine Optimization methods on Perplexity.ai, which is a commercially deployed generative engine, highlighting the generalizability of our proposed Generative Engine Optimization methods.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The generative model in generative engines is not limited to keyword matching, and the use of language models in ingesting source documents and response generation results in a more nuanced understanding of text documents and user query.

“This is because, unlike traditional search engines, the generative model in generative engines is not limited to keyword matching, and the use of language models in ingesting source documents and response generation results in a more nuanced understanding of text documents and user query.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

A more persuasive form of writing is likely to hold more value in debates.

“This aligns with our intuition, as a more persuasive form of writing is likely to hold more value in debates.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Changes made by GEO methods are targeted changes in textual content, do not affect other metadata such as domain name and backlinks, and are less likely to affect search engine rankings.

“However, we note that changes made by GEO methods are targeted changes in textual content, bearing some resemblance with SEO methods, while not affecting other metadata such as domain name, backlinks, etc, and thus, they are less likely to affect search engine rankings.”

Classified as probably not a substantial claim. May combine two claims.

The experiments were run on five different random seeds and the average is reported.

“Further, to reduce variance in results, we run our experiments on five different random seeds and report the average.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Every query in GEO-bench is tagged and manually inspected.

“Lastly, while every query in our proposed GEO-bench is tagged and manually inspected, there may be discrepancies due to subjective interpretations or errors in labeling.”

Classified as probably not a substantial claim.

All answers on Perplexity.ai were generated only using the provided file sources.

“Since perplexity.aidoes not allow the user to specify source URLs, we instead provide source text as file uploads to perplexity.ai while ensuring all answers are generated only using the file sources provided.”

Classified as probably not a substantial claim. May combine two claims.

Experiments were run on five different random seeds and the average is reported.

“we run our experiments on five different random seeds and report the average.”

• Also stated in: § C RESULTS · plain

  “We perform experiments on 5 random seeds and present results with statistical deviations in Table 6”

Classified as probably not a substantial claim.

G-Eval scores were normalized to have the same mean and variance as Position-Adjusted Word Count to enable a fair and meaningful comparison.

“However, since G-Eval scores are poorly calibrated, we normalize them to have the same mean and variance as Position-Adjusted Word Count to enable a fair and meaningful comparison.”

• AssumptionRests on an assumption: G-Eval scores are poorly calibrated.

Probably not a substantial claim: may not pass S2 (checkable).

GEO methods are likely to be used by content creators in the real world.

“The findings underscore the importance of studying GEO methods in combination, as they are likely to be used by content creators in the real world.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The combination analysis was conducted on a subset of 200 examples from the test split due to cost constraints.

“Due to cost constraints, the analysis was conducted on a subset of 200 examples from the test split, and therefore the numbers presented here differ from those in Table 1”

• AssumptionRests on an assumption: The 200-example subsets used for the combination and Perplexity.ai analyses are representative of the full test split.

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

The Perplexity.ai evaluation used a subset of 200 samples of the test set.

“We evaluate all our methods on a subset of 200 samples of our test set.”

• AssumptionRests on an assumption: The 200-example subsets used for the combination and Perplexity.ai analyses are representative of the full test split.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

With generative engines rapidly emerging as the primary information delivery paradigm and SEO is not directly applicable, new techniques are needed.

“With generative engines rapidly emerging as the primary information delivery paradigm and SEO is not directly applicable; new techniques are needed.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Query tagging used GPT-4 and high recall and precision were manually verified on the test split.

“For tagging, we employ the GPT-4 model and manually verify high recall and precision on the test split.”

Probably not a substantial claim: may not pass S4 (based).

This work is the first to formalize the GEO paradigm.

“In summary, our work is the first to formalize the important and timely GEO paradigm, releasing algorithms and infrastructure (benchmarks, datasets, and metrics) to facilitate rapid progress in generative engines by the community.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The advent of Generative Engines might initially seem disadvantageous to smaller entities.

“The advent of Generative Engines might initially seem disadvantageous to these smaller entities.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The automated tagging system can produce noisy tags that should not be considered carefully.

“However, owing to such an automated system, the tags can be noisy and should not be considered carefully.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The queries in GEO-bench closely resemble real-world queries.

“Additionally, despite our efforts to ensure the queries in our GEObench closely resemble real-world queries, the nature of queries can change over time, necessitating continuous updates.”

May combine two claims.

The GEO methods show robustness to varying queries, yielding significant improvements despite query diversity.

“This shows the robustness of these methods to varying queries, yielding significant improvements despite query diversity.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Five different responses were sampled at temperature=0.7 to reduce statistical deviations.

“We sample 5 different responses at temperature=0.7, to reduce statistical deviations.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The paper's work unifies prior approaches and provides a common benchmark for improving these systems.

“Our work unifies these approaches and provides a common benchmark for improving these systems in the future.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There may be discrepancies in GEO-bench tags due to subjective interpretations or errors in labeling.

“Lastly, while every query in our proposed GEO-bench is tagged and manually inspected, there may be discrepancies due to subjective interpretations or errors in labeling.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Many lower-ranked websites are created by small content creators or independent businesses, who traditionally struggle to compete with larger corporations in top search engine results.

“Many lower-ranked websites are created by small content creators or independent businesses, who traditionally struggle to compete with larger corporations in top search engine results.”

Probably not a substantial claim: may not pass S3 (load-bearing).

GEO-bench contains queries from nine datasets.

“GEO-bench contains queries from nine datasets.”

Classified as probably not a substantial claim.

All impression metrics were normalized by multiplying them with a constant factor so that the sum of the impressions of all citations in a response equals 1.

“Furthermore, all impression metrics are normalized by multiplying them with a constant factor so that the sum of the impressions of all citations in a response equals 1.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper manually confirmed high recall and precision in tagging queries using GPT-4.

“For tagging, we use the GPT-4 model and manually confirm high recall and precision in tagging.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The source selected for optimization is chosen randomly but remains constant for a particular query across all GEO methods.

“The source 𝑠 𝑖 selected for optimization is chosen randomly but remains constant for a particular query across all GEO methods.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Traditional search engines rely on multiple factors, such as the number of backlinks and domain presence, which are challenging for small creators to achieve.

“This is because traditional search engines rely on multiple factors, such as the number of backlinks and domain presence, which are challenging for small creators to achieve.”