CoChat
Sign Up
PreprintCited by 42Machine Learning and Data ClassificationArtificial Intelligence ApplicationsScientific Computing and Data Management

The DOI matches its DataCite record; 4 of the 4 references it cites were found in a registry or an index.

EvalAI: Towards Better Evaluation Systems for AI Agents

Open access
Yadav, Deshraj, Jain, Rishabh, Agrawal, Harsh, Chattopadhyay, Prithvijit, Singh, Taranjeet, Jain, Akash, Singh, Shiv Baran, Lee, Stefan, Batra, Dhruv·2019·arXiv
DOI 10.48550/arxiv.1902.03570arXiv 1902.03570
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce EvalAI, an open source platform for evaluating and comparing machine learning (ML) and artificial intelligence algorithms (AI) at scale. EvalAI is built to provide a scalable solution to the research community to fulfill the critical need of evaluating machine learning models and agents acting in an environment against annotations or with a human-in-the-loop. This will help researchers, students, and data scientists to create, collaborate, and participate in AI challenges organized around the globe. By simplifying and standardizing the process of benchmarking these models, EvalAI seeks to lower the barrier to entry for participating in the global scientific effort to push the frontiers of machine learning and artificial intelligence, thereby increasing the rate of measurable progress in this domain.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1902.03570

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Stan use the Mahalanobis distance for matching, significantly outperforming the strongest baseline (+40% sAMOTA) and setting a new state-of-the-art on the nuScenes tracking benchmark.

    “Stan [16] use the Mahalanobis distance for matching, significantly outperforming the strongest baseline ( +40% sAMOTA) and setting a new state-of-the-art on the nuScenes tracking benchmark.”

    From nuScenes: A Multimodal Dataset for Autonomous Driving (2020) · cites this paper as [16]

    The supplied passages discuss EvalAI’s evaluation platform and challenges, including fastMRI (Section 7.4, p. 9), rather than Stan’s tracking method. Searches for Mahalanobis distance, nuScenes, and +40% sAMOTA found no passage in our copy addressing the claimed matching method or benchmark result. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 26

  1. [2][2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV) , 2015. 1, 3, 8 -
  2. [3][3] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. CoRR , abs/1207.4708, 2012. 4 - · arXiv
  3. [4][4] Antoine Bordes and Jason Weston. Learning end-to-end goal-oriented dialog. CoRR , abs/1605.07683, 2016. 2 - · arXiv
  4. [5][5] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR , abs/1606.01540, 2016. 4 - · arXiv
  5. [6][6] Prithvijit Chattopadhyay, Deshraj Yadav, Viraj Prabhu, Arjun Chandrasekaran, Abhishek Das, Stefan Lee, Dhruv Batra, and Devi Parikh. Evaluating visual conversational agents via cooperative human-ai games. In Proceedings of the Fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP) , 2017. 2 -
  6. [8][8] Bo Dai, Dahua Lin, Raquel Urtasun, and Sanja Fidler. Towards diverse and natural image descriptions via a conditional gan. 2017 IEEE International Conference on Computer Vision (ICCV) , pages 2989-2998, 2017. 4, 8 -
  7. [9][9] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1-10, 2018. 2, 3, 9 -
  8. [10][10] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, and Dhruv Batra. Visual dialog. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1080-1089, 2017. 3, 4, 8 -
  9. [11][11] Abhishek Das, Satwik Kottur, José M. F. Moura, Stefan Lee, and Dhruv Batra. Learning cooperative visual dialog agents with deep reinforcement learning. 2017 IEEE International Conference on Computer Vision (ICCV) , pages 2970-2979, 2017. 4, 8 -
  10. [14][14] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. CoRR , abs/1604.06778, 2016. 4 - · arXiv
  11. [16][16] Donald Geman, Stuart Geman, Neil Hallonquist, and Laurent Younes. Visual turing test for computer vision systems. Proceedings of the National Academy of Sciences , page 201422953, 2015. 5 -
  12. [17][17] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS) , 2014. 4 -
  13. [18][18] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017. 8 -
  14. [19][19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Computer Vision (ICCV), 2017 IEEE International Conference on , pages 2980-2988. IEEE, 2017. 4 -
  15. [23][23] Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems (NIPS) , 2012. 1 -
  16. [24][24] Dianqi Li, Xiaodong He, Qiuyuan Huang, Ming-Ting Sun, and Lei Zhang. Generating diverse and accurate visual captions by comparative adversarial learning. arXiv preprint arXiv:1804.00861 , 2018. 4, 8 - · arXiv
  17. [25][25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755. Springer, 2014. 3 -
  18. [26][26] Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023 , 2016. 8 - · arXiv
  19. [27][27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015. 4 -
  20. [28][28] Alexander H Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. Parlai: A dialog research software platform. arXiv preprint arXiv:1705.06476 , 2017. 4 - · arXiv
  21. [30][30] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 779-788, 2016. 4 -
  22. [31][31] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations (ICLR) , 2015. 4 -
  23. [33][33] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4631-4640, 2016. 2 -
  24. [34][34] Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. Openml: Networked science in machine learning. SIGKDD Explorations , 15(2):49-60, 2013. 4 -
  25. [35][35] Jure Zbontar, Florian Knoll, Anuroop Sriram, Matthew J Muckley, Mary Bruno, Aaron Defazio, Marc Parente, Krzysztof J Geras, Joe Katsnelson, Hersh Chandarana, et al. fastmri: An open dataset and benchmarks for accelerated mri. arXiv preprint arXiv:1811.08839 , 2018. 9 - · arXiv
  26. [36][36] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA) , 2017. 2

Cited by

Sorted by citations

Metadata

Authors
Yadav, D., Jain, R., Agrawal, H., Chattopadhyay, P., Singh, T., Jain, A., Singh, S. B., Lee, S., & Batra, D.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1902.03570
arXiv
1902.03570
OpenAlex
W2920151874
Cited by
42
References
4
Citations by year
2025: 4 · 2024: 8 · 2023: 4 · 2022: 8 · 2021: 8 · 2020: 9 · 2019: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
Keywords
AI agent evaluation, benchmarking, open-source platform, human-in-the-loop evaluation, AI challenges, machine learning model evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review