CoChat
Sign Up
Cited by 78Generative Adversarial Networks and Image SynthesisMultimodal Machine Learning ApplicationsImage and Video Quality Assessment

Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation

Open access
Otani, Mayu, Togashi, Riku, Sawai, Yu, Ishigami, Ryosuke, Nakashima, Yuta, Rahtu, Esa, Heikkilä, Janne, Satoh, Shin'ichi·2023·2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr52729.2023.01372
Open this paper in CoChat — free accountOpen PDF

Abstract

Human evaluation is critical for validating the performance of text-to-image generative models, as this highly cognitive process requires deep comprehension of text and images. However, our survey of 37 recent papers reveals that many works rely solely on automatic measures (e.g., FID) or perform poorly described human evaluations that are not reliable or repeatable. This paper proposes a standardized and well-defined human evaluation protocol to facilitate verifiable and reproducible human evaluation in future works. In our pilot data collection, we experimentally show that the current automatic measures are incompatible with human perception in evaluating the performance of the text-to-image generation results. Furthermore, we provide insights for designing human evaluation experiments reliably and conclusively. Finally, we make several resources publicly available to the community to facilitate easy and fast implementations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52729.2023.01372

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The most dominant approach to gathering human annotations is to 'crowd-source' either anonymous people or community members as 'data workers', who are given simple, tokenistic tasks with little or no context to the data.

    “The most dominant approach to gathering human annotations is to 'crowd-source' [7, 53, 65] either anonymous people or community members as 'data workers', who are given simple, tokenistic tasks 'that even inexperienced annotators can finish in an instance' [31, 85], with little or no context to the data.”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [85]

  • Where humans annotate data for AI image evaluation, their role commonly involves identifying objects or features in images, image comparisons, or gamified approaches.

    “Where humans annotate data for AI image evaluation, their role commonly involves: identifying objects or features in images [52]; image comparisons to indicate similarities or preferences across two or more samples (e.g., [52, 85]); or gamified approaches (e.g., describing artifacts characteristics for another annotator to guess [99]).”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [85]

    The annotation-interface passage (Section 2; page unavailable) and Figure A describe annotators rating images for fidelity and caption alignment, not identifying objects or features. I searched for object-identification tasks and annotator comparisons of image samples; the paper’s pairwise comparisons (Section 3; page unavailable) are statistical tests of ratings, not comparisons performed by annotators. The gamified approach is cited to a different source in the citing sentence. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 68

  1. Herman Aguinis, Isabel Villamor, and Ravi S. Ramani. Mturk research: Review and recommendations. Journal of Management, 47(4):823-837, 2021. 2, 8
  2. Ahmed Alaa, Boris van Breugel, Evgeny S. Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? Sample-level metrics for evaluating and auditing generative models. In ICML, volume 162, pages 290-306, 2022. 2
  3. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018. 7
  4. Ching-Yuan Bai, Hsuan-Tien Lin, Colin Raffel, and Wendy Chi-wen Kan. On training sample memorization: Lessons from benchmarking generative modeling with a large-scale competition. In ACM SIGKDD Conference on Knowledge Discovery & Data Mining, page 2534-2542, 2021. 2
  5. Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshi- aki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2020 conference on machine translation (WMT20). In Conference on Machine Translation, pages 1-55, 2020. 3
  6. Mikolaj Binkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In ICLR, 2018. 2
  7. Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. CogView: Mastering text-to- image generation via transformers. In NeurIPS, volume 34, pages 19822-19835, 2021.
  8. Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers, 2022. arXiv:2204.14217 [cs]. 1, 2, 3, 5, 6 · arXiv
  9. Tan M. Dinh, Rang Nguyen, and Binh-Son Hua. TISE: Bag of metrics for text-to-image synthesis evaluation. In ECCV, volume 13696, pages 594-609, 2022. 3
  10. Larry V Hedges. Distribution theory for glass's estimator of effect size and related estimators. Journal of Educational Statistics, 6(2):107-128, 1981. 7
  11. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In EMNLP, pages 7514-7528, 2021. 1, 2, 6
  12. Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis. In CVPR, pages 7986-7994, 2018. 3
  13. Toni Kaplan, Susumu Saito, Kotaro Hara, and Jeffrey Bigham. Striving to earn more: A survey of work strategies and tool use among crowd workers. AAAI Conference on Human Computation and Crowdsourcing, 6(1):70-78, 2018.
  14. Ryan Kennedy, Scott Clifford, Tyler Burleigh, Philip D. Waggoner, Ryan Jewell, and Nicholas J. G. Winter. The shape of and solutions to the mturk quality crisis. Political Science Research and Methods, 8(4):614-629, 2020. 2, 4
  15. Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, and Daniel S. Weld. GENIE: A leaderboard for human-in- the-loop evaluation of text generation. In EMNLP, 2022. 1, 2, 3, 4
  16. Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall met- ric for assessing generative models. In NeurIPS, 2019. 2
  17. Zhiheng Li, Martin Renqiang Min, Kai Li, and Chenliang Xu. StyleT2I: Toward compositional and high-fidelity text- to-image synthesis. In CVPR, pages 18197-18207, 2022. 3
  18. Jiadong Liang, Wenjie Pei, and Feng Lu. CPGAN: Content- parsing generative adversarial networks for text-to-image synthesis. In ECCV, page 18, 2020. 3
  19. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, pages 740-755, 2014. 2, 4, 5
  20. Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In ECCV, 2022. 2
  21. Shaohui Liu, Yi Wei, Jiwen Lu, and Jie Zhou. An improved evaluation framework for generative adversarial networks. arXiv preprint arXiv:1803.07474, 2018. 2, 6 · arXiv
  22. Adyasha Maharana, Darryl Hannan, and Mohit Bansal. StoryDALL-E: Adapting pretrained text-to-image trans- formers for story continuation. In ECCV, page 18, 2022. 2, 3
  23. Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealis- tic image generation and editing with text-guided diffusion models. In ICML, volume 162, pages 16784-16804, 2022. 1, 5, 6, 7
  24. Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in GAN evaluation. In CVPR, pages 11410-11420, 2022. 1, 2, 6
  25. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, pages 10684- 10695, 2022. 4, 5, 6
  26. Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In NeurIPS, page 2234-2242, 2016. 2
  27. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In ACL, 2018. 5
  28. M. S. Silberman, B. Tomlinson, R. LaPlante, J. Ross, L. Irani, and A. Zaldivar. Responsible research with crowds: Pay crowdworkers at least minimum wage. Commun. ACM, 61(3):39-41, 2018. 3, 5
  29. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception ar- chitecture for computer vision. In CVPR, pages 2818-2826, 2016.
  30. Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. AttnGAN: Fine- grained text to image generation with attentional generative adversarial networks. In CVPR, 2018. 2
  31. Kun Yan, Lei Ji, Chenfei Wu, Ming Zhou, Nan Duan, and Shuai Ma. Trace controlled text to image generation. In ECCV, page 17, 2022. 2, 3
  32. Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling Autoregressive Models for Content- Rich Text-to-Image Generation, 2022. arXiv:2206.10789 [cs]. 2, 3, 5, 6 · arXiv
  33. Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiao- gang Wang, Xiaolei Huang, and Dimitris Metaxas. Stack- GAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, pages 5908-5916, 2017. 2, 3
  34. Sharon Zhou, Mitchell L. Gordon, Ranjay Krishna, Austin Narcomey, Li Fei-Fei, and Michael S. Bernstein. HYPE: A benchmark for human eye perceptual evaluation of genera- tive models. In NeurIPS, pages 3444-3456, 2019. 2, 4
  35. Jun Cheng, Fuxiang Wu, Yanling Tian, Lei Wang, and Dapeng Tao. RiFeGAN: Rich Feature Generation for Text-to-Image Synthesis From Prior Knowledge. In CVPR, pages 10911-10920, 2020.
  36. Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. CogView: Mastering text-to-image generation via transformers. In NeurIPS, volume 34, pages 19822-19835, 2021.
  37. Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers, 2022. arXiv:2204.14217 [cs]. · arXiv
  38. Tan M. Dinh, Rang Nguyen, and Binh-Son Hua. TISE: Bag of metrics for text-to-image synthesis evaluation. In ECCV, volume 13696, pages 594-609, 2022. 2
  39. Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-A-Scene: Scene-Based Text-to- Image Generation with Human Priors. In ECCV, page 19, 2022. 2
  40. Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022. arXiv:2208.01618 [cs]. · arXiv
  41. Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector Quantized Diffusion Model for Text-to-Image Synthesis. In CVPR, pages 10696-10706, 2022. 2
  42. Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis. In CVPR, pages 7986-7994, 2018. 2
  43. Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-Based Real Image Editing with Diffusion Models, 2022. arXiv:2210.09276 [cs]. · arXiv
  44. Qicheng Lao, Mohammad Havaei, Ahmad Pesaranghader, Francis Dutil, Lisa Di Jorio, and Thomas Fevens. Dual Adversarial Inference for Text-to-Image Synthesis. In ICCV, pages 7567-7576, 2019. 2
  45. Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable Text-to-Image Generation. In NeurIPS, volume 32, 2019. 2
  46. Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao. Object-Driven Text-To-Image Synthesis via Adversarial Training. In CVPR, pages 12174-12182, 2019. 2
  47. Zhiheng Li, Martin Renqiang Min, Kai Li, and Chenliang Xu. StyleT2I: Toward compositional and high-fidelity text-to-image synthesis. In CVPR, pages 18197-18207, 2022. 2
  48. Jiadong Liang, Wenjie Pei, and Feng Lu. CPGAN: Content-parsing generative adversarial networks for text-to-image synthesis. In ECCV, page 18, 2020. 2
  49. Wentong Liao, Kai Hu, Michael Ying Yang, and Bodo Rosenhahn. Text to Image Generation With Semantic-Spatial Aware GAN. In CVPR, pages 18187-18196, 2022. 2
  50. Adyasha Maharana, Darryl Hannan, and Mohit Bansal. StoryDALL-E: Adapting pretrained text-to-image transformers for story continuation. In ECCV, page 18, 2022. 2
  51. Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In ICML, volume 162, pages 16784-16804, 2022. 2
  52. Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. MirrorGAN: Learning Text-To-Image Generation by Redescription. In CVPR, pages 1505-1514, 2019. 2
  53. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-Image Generation, 2021. arXiv:2102.12092 [cs]. · arXiv
  54. Shulan Ruan, Yong Zhang, Kun Zhang, Yanbo Fan, Fan Tang, Qi Liu, and Enhong Chen. DAE-GAN: Dynamic Aspect-Aware GAN for Text-to-Image Synthesis. In ICCV, pages 13960-13969, 2021. 2
  55. Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine Tuning Text- to-Image Diffusion Models for Subject-Driven Generation, 2022. arXiv:2208.12242 [cs]. · arXiv
  56. Hongchen Tan, Xiuping Liu, Xin Li, Yi Zhang, and Baocai Yin. Semantics-Enhanced Adversarial Nets for Text-to-Image Synthesis. In ICCV, pages 10501-10510, 2019. 2
  57. Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu. DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis. In CVPR, pages 16515-16525, 2022. 2
  58. Fuxiang Wu, Liu Liu, Fusheng Hao, Fengxiang He, and Jun Cheng. Text-to-Image Synthesis Based on Object-Guided Joint-Decoding Transformer. In CVPR, pages 18113-18122, 2022. 2
  59. Kun Yan, Lei Ji, Chenfei Wu, Ming Zhou, Nan Duan, and Shuai Ma. Trace controlled text to image generation. In ECCV, page 17, 2022. 2
  60. Guojun Yin, Bin Liu, Lu Sheng, Nenghai Yu, Xiaogang Wang, and Jing Shao. Semantics Disentangling for Text-To-Image Genera- tion. In CVPR, pages 2327-2336, 2019. 2
  61. Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation, 2022. arXiv:2206.10789 [cs]. · arXiv
  62. Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-Modal Contrastive Learning for Text-to-Image Generation. In CVPR, pages 833-842, 2021. 2
  63. Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In CVPR, pages 5908-5916, Venice, 2017. 2
  64. Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks, 2018. arXiv:1710.10916 [cs, stat]. · arXiv
  65. Zizhao Zhang, Yuanpu Xie, and Lin Yang. Photographic Text-to-Image Synthesis With a Hierarchically-Nested Adversarial Network. In CVPR, pages 6199-6208, 2018. 2
  66. Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Towards Language-Free Training for Text-to-Image Generation. In CVPR, pages 17907-17917, 2022. 2
  67. Bin Zhu and Chong-Wah Ngo. CookGAN: Causality Based Text-to-Image Synthesis. In CVPR, pages 5519-5527, 2020. 2
  68. Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To- Image Synthesis. In CVPR, pages 5802-5810, 2019. 2

Cited by

Sorted by citations

Authors
Otani, M., Togashi, R., Sawai, Y., Ishigami, R., Nakashima, Y., Rahtu, E., Heikkilä, J., & Satoh, S.
Affiliations
  • CyberAgent (Japan) (JP)
  • The University of Osaka (JP)
  • Tampere University (FI)
  • University of Oulu (FI)
ORCID
Venue
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2023-06
Pages
14277-14286
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr52729.2023.01372
OpenAlex
W4386071621
Cited by
78
References
6
Citations by year
2026: 23 · 2025: 29 · 2024: 23 · 2023: 3
FWCI
8.36
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Academy of Finland (324346,353139)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
Keywords
text-to-image generation, human evaluation, reproducibility, evaluation protocol, automatic evaluation metrics, FID, human perception, generative models, assessment resources

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review