CoChat
Sign Up
PreprintCited by 34Data Visualization and AnalyticsExplainable Artificial Intelligence (XAI)Software Engineering Research

LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models

Open access
Kahng, Minsuk, Tenney, Ian, Pushkarna, Mahima, Liu, Michael Xieyang, Wexler, James, Reif, Emily, Kallarackal, Krystal, Chang, Minsuk +2 more·2024·arXiv
DOI 10.48550/arxiv.2402.10524arXiv 2402.10524
Open this paper in CoChat — free accountOpen PDF

Abstract

Automatic side-by-side evaluation has emerged as a promising approach to evaluating the quality of responses from large language models (LLMs). However, analyzing the results from this evaluation approach raises scalability and interpretability challenges. In this paper, we present LLM Comparator, a novel visual analytics tool for interactively analyzing results from automatic side-by-side evaluation. The tool supports interactive workflows for users to understand when and why a model performs better or worse than a baseline model, and how the responses from two models are qualitatively different. We iteratively designed and developed the tool by closely working with researchers and engineers at a large technology company. This paper details the user challenges we identified, the design and development of the tool, and an observational study with participants who regularly evaluate their models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.10524

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Users want data visualization plots between the original prompt and its revision, akin to LLM Comparator and EvalLM.

    “This suggests users want data visualization plots between the original prompt and its revision, akin to LLM Comparator and EvalLM [24, 27].”

    From Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (2024) · cites this paper as [24]

    The Visualization Design and Development section describes LLM Comparator as a side-by-side comparison tool with a visualization summary, supporting its use as an example of visual comparison. But Section 3.1 says it compares two models’ responses to a prompt, not an original prompt with its revision. The cited paper therefore supports the analogy, not that specific comparison. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 20

  1. Saleema Amershi, Max Chickering, Steven M Drucker, Bongshin Lee, Patrice Simard, and Jina Suh. 2015. ModelTracker: Redesigning performance analysis tools for machine learning. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI). 337-346. https://doi.org/10.1145/ 2702123.2702509 · DOI
  2. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 (2016). https://arxiv.org/abs/1606.06565 · arXiv
  3. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. PaLM 2 technical report. arXiv preprint arXiv:2305.10403 (2023). https://arxiv.org/abs/2305.10403 · arXiv
  4. Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena Glassman. 2023. ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing. arXiv preprint arXiv:2309.09128 (2023). https: //arxiv.org/abs/2309.09128 · arXiv
  5. Angie Boggust, Brandon Carter, and Arvind Satyanarayan. 2022. Embedding Comparator: Visualizing differences in global structure and local neighborhoods via small multiples. In 27th International Conference on Intelligent User Interfaces (IUI). 746-766. https://doi.org/10.1145/3490099.3511122 · DOI
  6. Richard Brath, Daniel Keim, Johannes Knittel, Shimei Pan, Pia Sommerauer, and Hendrik Strobelt. 2023. The Role of Interactive Visualization in Explaining (Large) NLP Models: from Data to Inference. arXiv preprint arXiv:2301.04528 (2023). https://arxiv.org/abs/2301.04528 · arXiv
  7. Adam Coscia and Alex Endert. 2023. KnowledgeVIS: Interpreting Language Mod- els by Comparing Fill-in-the-Blank Prompts. IEEE Transactions on Visualization and Computer Graphics (2023).
  8. Michael Gleicher, Aditya Barve, Xinyi Yu, and Florian Heimerl. 2020. Boxer: Interactive comparison of classifier results. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 181-193. https://arxiv.org/abs/2004.07964 · arXiv
  9. Minsuk Kahng, Pierre Y Andrews, Aditya Kalro, and Duen Horng Chau. 2017. ActiVis: Visual exploration of industry-scale deep neural network models. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 88-97. https: //doi.org/10.1109/TVCG.2017.2744718 · DOI
  10. Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2023. EvalLM: Interactive Evaluation of Large Language Model Prompts on User- Defined Criteria. arXiv preprint arXiv:2309.13633 (2023). https://arxiv.org/abs/ 2309.13633 · arXiv
  11. Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, et al. 2024. LiPO: Listwise Preference Optimization through Learning-to-Rank. arXiv preprint arXiv:2402.01878 (2024). https://arxiv.org/abs/2402.01878 · arXiv
  12. Yao Ming, Huamin Qu, and Enrico Bertini. 2018. RuleMatrix: Visualizing and understanding classifiers with rules. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2018), 342-352. https://doi.org/10.1109/TVCG.2018. 2864812 · DOI · Paper page
  13. Hendrik Strobelt, Benjamin Hoover, Arvind Satyanarayan, and Sebastian Gehrmann. 2021. LMdiff: A visual diff tool to compare language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations. https://arxiv.org/abs/2111.01582 · arXiv
  14. Hendrik Strobelt, Albert Webson, Victor Sanh, Benjamin Hoover, Johanna Beyer, Hanspeter Pfister, and Alexander M Rush. 2022. Interactive and visual prompt engineering for ad-hoc task adaptation with large language models. IEEE Transactions on Visualization and Computer Graphics 29, 1 (2022), 1146-1156. https://arxiv.org/abs/2208.07852 · arXiv
  15. Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and Ann Yuan. 2020. The language interpretability tool: Extensible, in- teractive visualizations and analysis for NLP models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations. https://arxiv.org/abs/2008.05122 · arXiv
  16. Junpeng Wang, Liang Wang, Yan Zheng, Chin-Chia Michael Yeh, Shubham Jain, and Wei Zhang. 2022. Learning-from-disagreement: A model comparison and visual analytics framework. IEEE Transactions on Visualization and Computer Graphics (2022). https://arxiv.org/abs/2201.07849 · arXiv
  17. Zihan Wang, Jingbo Shang, and Ruiqi Zhong. 2023. Goal-Driven Explainable Clustering via Language Descriptions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). 10626-10649. https://doi.org/10.18653/v1/2023.emnlp-main.657 · DOI
  18. Jie M Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering 48, 1 (2020), 1-36. https://arxiv.org/abs/1906.10742 · arXiv
  19. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Neural Information Processing Systems (NeurIPS): Datasets and Benchmarks Track. https://arxiv.org/abs/2306.05685 · arXiv
  20. Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Steinhardt. 2022. Describ- ing differences between text distributions with natural language. In Interna- tional Conference on Machine Learning (ICML). PMLR, 27099-27116. https: //proceedings.mlr.press/v162/zhong22a.html

Cited by

Sorted by citations

Authors
Kahng, M., Tenney, I., Pushkarna, M., Liu, M. X., Wexler, J., Reif, E., Kallarackal, K., Chang, M., Terry, M., & Dixon, L.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
1-7
DOI
10.48550/arxiv.2402.10524
arXiv
2402.10524
OpenAlex
W4396833346
Cited by
34
References
0
Citations by year
2026: 9 · 2025: 19 · 2024: 6
FWCI
7.04
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
large language models, visual analytics, interactive data analysis, model comparison, interpretability, baseline model, user challenges

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review