Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its DataCite record; 47 of the 47 references it cites were found in a registry or an index; 7 of the 13 sources its claims cite were checked for support.

Humanity's Last Exam

Open access
Phan, Long, Gatti, Alice, Han, Ziwen, Li, Nathaniel, Hu, Josephina, Zhang, Hugh, Zhang, Chen Bo Calvin, Shaaban, Mohamed, Ling, John, Shi, Sean, Choi, Michael, Agrawal, Anish, Chopra, Arnav, Khoja, Adam, Kim, Ryan, Ren, Richard, Hausenloy, Jason, Zhang, Oliver, Mazeika, Mantas, Dodonov, Dmitry, Nguyen, Tung, Lee, Jaeho, Anderson, Daron, Doroshenko, Mikhail, Stokes, Alun Cennyth, Mahmood, Mobeen, Pokutnyi, Oleksandr, Iskra, Oleg, Wang, Jessica P., Levin, John-Clark, Kazakov, Mstyslav, Feng, Fiona, Feng, Steven Y., Zhao, Haoran, Yu, Michael, Gangal, Varun, Zou, Chelsea, Wang, Zihan, Popov, Serguei, Gerbicz, Robert, Galgon, Geoff, Schmitt, Johannes, Yeadon, Will, Lee, Yongki, Sauers, Scott, Sanchez, Alvaro, Giska, Fabian, Roth, Marc, Riis, Søren, Utpala, Saiteja, Burns, Noah, Goshu, Gashaw M., Naiya, Mohinder Maheshbhai, Agu, Chidozie, Giboney, Zachary, Cheatom, Antrell, Fournier-Facio, Francesco, Crowson, Sarah-Jane, Finke, Lennart, Cheng, Zerui, Zampese, Jennifer, Hoerr, Ryan G., Nandor, Mark, Park, Hyunwoo, Gehrunger, Tim, Cai, Jiaqi, McCarty, Ben, Garretson, Alexis C, Taylor, Edwin, Sileo, Damien, Ren, Qiuyu, Qazi, Usman, Li, Lianghui, Nam, Jungbae, Wydallis, John B., Arkhipov, Pavel, Shi, Jack Wei Lun, Bacho, Aras, Willcocks, Chris G., Cao, Hangrui, Motwani, Sumeet, Santos, Emily de Oliveira, Veith, Johannes, Vendrow, Edward, Cojoc, Doru, Zenitani, Kengo, Robinson, Joshua, Tang, Longke, Li, Yuqi, Vendrow, Joshua, Fraga, Natanael Wildner, Kuchkin, Vladyslav, Maksimov, Andrey Pupasov, Marion, Pierre, Efremov, Denis, Lynch, Jayson, Liang, Kaiqu, Mikov, Aleksandar, Gritsevskiy, Andrew, Guillod, Julien, Demir, Gözdenur, Martinez, Dakotah, Pageler, Ben, Zhou, Kevin, Soori, Saeed, Press, Ori, Tang, Henry, Rissone, Paolo, Green, Sean R., Brüssel, Lina, Twayana, Moon, Dieuleveut, Aymeric, Imperial, Joseph Marvin, Prabhu, Ameya, Yang, Jinzhou, Crispino, Nick, Rao, Arun, Zvonkine, Dimitri, Loiseau, Gabriel, Kalinin, Mikhail, Lukas, Marco, Manolescu, Ciprian, Stambaugh, Nate, Mishra, Subrata, Hogg, Tad, Bosio, Carlo, Coppola, Brian P, Salazar, Julian, Jin, Jaehyeok, Sayous, Rafael, Ivanov, Stefan, Schwaller, Philippe, Senthilkuma, Shaipranesh, Bran, Andres M, Algaba, Andres, Houte, Kelsey Van den, Van Der Sypt, Lynn, Verbeken, Brecht F., Noever, David, Kopylov, Alexei, Myklebust, Benjamin, Li, Bikun, Schut, Lisa, Zheltonozhskii, Evgenii, Yuan, Qiaochu, Lim, Derek, Stanley, Richard, Yang, Tong, Maar, John, Wykowski, Julian, Oller, Martí, Sahu, Anmol, Ardito, Cesare Giulio, Hu, Yuzheng, Kamdoum, Ariel Ghislain Kemogne, Jin, Alvin, Vilchis, Tobias Garcia, Zu, Yuexuan, Lackner, Martin, Koppel, James, Sun, Gongbo, Antonenko, Daniil S., Chern, Steffi, Zhao, Bingchen, Arsene, Pierrot, Cavanagh, Joseph M, Li, Daofeng, Shen, Jiawei, Crisostomi, Donato, Zhang, Wenjin, Dehghan, Ali, Ivanov, Sergey, Perrella, David, Kaparov, Nurdin, Zang, Allen, Sucholutsky, Ilia, Kharlamova, Arina, Orel, Daniil, Poritski, Vladislav, Ben-David, Shalev, Berger, Zachary, Whitfill, Parker, Foster, Michael, Munro, Daniel, Ho, Linh, Sivarajan, Shankar, Hava, Dan Bar, Kuchkin, Aleksey, Holmes, David, Rodriguez-Romero, Alexandra, Sommerhage, Frank, Zhang, Anji, Moat, Richard, Schneider, Keith, Kazibwe, Zakayo, Clarke, Don, Kim, Dae Hyun, Dias, Felipe Meneguitti, Fish, Sara, Elser, Veit, Kreiman, Tobias, Vilchis, Victor Efren Guadarrama, Klose, Immo, Anantheswaran, Ujjwala, Zweiger, Adam, Rawal, Kaivalya, Li, Jeffery, Nguyen, Jeremy, Daans, Nicolas, Heidinger, Haline, Radionov, Maksim, Rozhoň, Václav, Ginis, Vincent, Stump, Christian, Cohen, Niv, Poświata, Rafał, Tkadlec, Josef, Goldfarb, Alan, Wang, Chenguang, Padlewski, Piotr, Barzowski, Stanislaw, Montgomery, Kyle, Stendall, Ryan, Tucker-Foltz, Jamie, Stade, Jack, Rogers, T. Ryan, Goertzen, Tom, Grabb, Declan, Shukla, Abhishek, Givré, Alan, Ambay, John Arnold, Sen, Archan, Aziz, Muhammad Fayez, Inlow, Mark H, He, Hao, Zhang, Ling, Kaddar, Younesse, Ängquist, Ivar, Chen, Yanxu, Wang, Harrison K, Ramakrishnan, Kalyan, Thornley, Elliott, Terpin, Antonio, Schoelkopf, Hailey, Zheng, Eric, Carmi, Avishy, Brown, Ethan D. L., Zhu, Kelin, Bartolo, Max, Wheeler, Richard, Stehberger, Martin, Bradshaw, Peter, Heimonen, JP, Sridhar, Kaustubh, Akov, Ido, Sandlin, Jennifer, Makarychev, Yury, Tam, Joanna, Hoang, Hieu, Cunningham, David M., Goryachev, Vladimir, Patramanis, Demosthenes, Krause, Michael, Redenti, Andrew, Aldous, David, Lai, Jesyin, Coleman, Shannon, Xu, Jiangnan, Lee, Sangwon, Magoulas, Ilias, Zhao, Sandy, Tang, Ning, Cohen, Michael K., Paradise, Orr, Kirchner, Jan Hendrik, Ovchynnikov, Maksym, Matos, Jason O., Shenoy, Adithya, Wang, Michael, Nie, Yuzhou, Sztyber-Betley, Anna, Faraboschi, Paolo, Riblet, Robin, Crozier, Jonathan, Halasyamani, Shiv, Verma, Shreyas, Joshi, Prashant, Meril, Eli, Ma, Ziqiao, Andréoletti, Jérémy, Singhal, Raghav, Platnick, Jacob, Nevirkovets, Volodymyr, Basler, Luke, Ivanov, Alexander, Khoury, Seri, Gustafsson, Nils, Piccardo, Marco, Mostaghimi, Hamid, Chen, Qijia, Singh, Virendra, Khánh, Tran Quoc, Rosu, Paul, Szlyk, Hannah, Brown, Zachary, Narayan, Himanshu, Menezes, Aline, Roberts, Jonathan, Alley, William, Sun, Kunyang, Patel, Arkil, Lamparth, Max, Reuel, Anka, Xin, Linwei, Xu, Hanmeng, Loader, Jacob, Martin, Freddie, Wang, Zixuan, Achilleos, Andrea, Preu, Thomas, Korbak, Tomek, Bosio, Ida, Kazemi, Fereshteh, Chen, Ziye, Bálint, Biró, Lo, Eve J. Y., Wang, Jiaqi, Nunes, Maria Inês S., Milbauer, Jeremiah, Bari, M Saiful, Wang, Zihao, Ansarinejad, Behzad, Sun, Yewen, Durand, Stephane, Elgnainy, Hossam, Douville, Guillaume, Tordera, Daniel, Balabanian, George, Wolff, Hew, Kvistad, Lynna, Milliron, Hsiaoyun, Sakor, Ahmad, Eron, Murat, O., Andrew Favre D., Shah, Shailesh, Zhou, Xiaoxiang, Kamalov, Firuz, Abdoli, Sherwin, Santens, Tim, Barkan, Shaul, Tee, Allison, Zhang, Robin, Tomasiello, Alessandro, De Luca, G. Bruno, Looi, Shi-Zhuo, Le, Vinh-Kha, Kolt, Noam, Pan, Jiayi, Rodman, Emma, Drori, Jacob, Fossum, Carl J, Muennighoff, Niklas, Jagota, Milind, Pradeep, Ronak, Fan, Honglu, Eicher, Jonathan, Chen, Michael, Thaman, Kushal, Merrill, William, Firsching, Moritz, Harris, Carter, Ciobâcă, Stefan, Gross, Jason, Pandey, Rohan, Gusev, Ilya, Jones, Adam, Agnihotri, Shashank, Zhelnov, Pavel, Mofayezi, Mohammadreza, Piperski, Alexander, Zhang, David K., Dobarskyi, Kostiantyn, Leventov, Roman, Soroko, Ignat, Duersch, Joshua, Taamazyan, Vage, Ho, Andrew, Ma, Wenjie, Held, William, Xian, Ruicheng, Zebaze, Armel Randy, Mohamed, Mohanad, Leser, Julian Noah, Yuan, Michelle X, Yacar, Laila, Lengler, Johannes, Olszewska, Katarzyna, Di Fratta, Claudio, Oliveira, Edson, Jackson, Joseph W., Zou, Andy, Chidambaram, Muthu, Manik, Timothy, Haffenden, Hector, Stander, Dashiell, Dasouqi, Ali, Shen, Alexander, Golshani, Bita, Stap, David, Kretov, Egor, Uzhou, Mikalai, Zhidkovskaya, Alina Borisovna, Winter, Nick, Rodriguez, Miguel Orbegozo, Lauff, Robert, Wehr, Dustin, Tang, Colin, Hossain, Zaki, Phillips, Shaun, Samuele, Fortuna, Ekström, Fredrik, Hammon, Angela, Patel, Oam, Farhidi, Faraz, Medley, George, Mohammadzadeh, Forough, Peñaflor, Madellene, Kassahun, Haile, Friedrich, Alena, Perez, Rayner Hernandez, Pyda, Daniel, Sakal, Taom, Dhamane, Omkar, Mirabadi, Ali Khajegili, Hallman, Eric, Okutsu, Kenchi, Battaglia, Mike, Maghsoudimehrabani, Mohammad, Amit, Alon, Hulbert, Dave, Pereira, Roberto, Weber, Simon, Handoko, Peristyy, Anton, Malina, Stephen, Mehkary, Mustafa, Aly, Rami, Reidegeld, Frank, Dick, Anna-Katharina, Friday, Cary, Singh, Mukhwinder, Shapourian, Hassan, Kim, Wanyoung, Costa, Mariana, Gurdogan, Hubeyb, Kumar, Harsh, Ceconello, Chiara, Zhuang, Chao, Park, Haon, Carroll, Micah, Tawfeek, Andrew R., Steinerberger, Stefan, Aggarwal, Daattavya, Kirchhof, Michael, Dai, Linjie, Kim, Evan, Ferret, Johan, Shah, Jainam, Wang, Yuzhou, Yan, Minghao, Burdzy, Krzysztof, Zhang, Lixin, Franca, Antonio, Pham, Diana T., Loh, Kang Yong, Robinson, Joshua, Jackson, Abram, Giordano, Paolo, Petersen, Philipp, Cosma, Adrian, Colino, Jesus, White, Colin, Votava, Jacob, Vinnikov, Vladimir, Delaney, Ethan, Spelda, Petr, Stritecky, Vit, Shahid, Syed M., Mourrat, Jean-Christophe, Vetoshkin, Lavr, Sponselee, Koen, Bacho, Renas, Yong, Zheng-Xin, de la Rosa, Florencia, Cho, Nathan, Li, Xiuyu, Malod, Guillaume, Weller, Orion, Albani, Guglielmo, Lang, Leon, Laurendeau, Julien, Kazakov, Dmitry, Adesanya, Fatimah, Portier, Julien, Hollom, Lawrence, Souza, Victor, Zhou, Yuchen Anna, Degorre, Julien, Yalın, Yiğit, Obikoya, Gbenga Daniel, Rai, Bigi, Filippo, Boscá, M. C., Shumar, Oleg, Bacho, Kaniuar, Recchia, Gabriel, Popescu, Mara, Shulga, Nikita, Tanwie, Ngefor Mildred, Lux, Thomas C. H., Rank, Ben, Ni, Colin, Brooks, Matthew, Yakimchyk, Alesia, Huanxu, Liu, Cavalleri, Stefano, Häggström, Olle, Verkama, Emil, Newbould, Joshua, Gundlach, Hans, Brito-Santana, Leonor, Amaro, Brian, Vajipey, Vivek, Grover, Rynaa, Wang, Ting, Kratish, Yosi, Li, Wen-Ding, Gopi, Sivakanth, Caciolai, Andrea, de Witt, Christian Schroeder, Hernández-Cámara, Pablo, Rodolà, Emanuele, Robins, Jules, Williamson, Dominic, Cheng, Vincent, Raynor, Brad, Qi, Hao, Segev, Ben, Fan, Jingxuan, Martinson, Sarah, Wang, Erik Y., Hausknecht, Kaylie, Brenner, Michael P., Mao, Mao, Demian, Christoph, Kassani, Peyman, Zhang, Xinyu, Avagian, David, Scipio, Eshawn Jessica, Ragoler, Alon, Tan, Justin, Sims, Blake, Plecnik, Rebeka, Kirtland, Aaron, Bodur, Omer Faruk, Shinde, D. P., Labrador, Yan Carlos Leyva, Adoul, Zahra, Zekry, Mohamed, Karakoc, Ali, Santos, Tania C. B., Shamseldeen, Samir, Karim, Loukmane, Liakhovitskaia, Anna, Resman, Nate, Farina, Nicholas, Gonzalez, Juan Carlos, Maayan, Gabe, Anderson, Earth, Pena, Rodrigo De Oliveira, Kelley, Elizabeth, Mariji, Hodjat, Pouriamanesh, Rasoul, Wu, Wentao, Finocchio, Ross, Alarab, Ismail, Cole, Joshua, Ferreira, Danyelle, Johnson, Bryan, Safdari, Mohammad, Dai, Liangti, Arthornthurasuk, Siriphan, McAlister, Isaac C., Moyano, Alejandro José, Pronin, Alexey, Fan, Jing, Ramirez-Trinidad, Angel, Malysheva, Yana, Pottmaier, Daphiny, Taheri, Omid, Stepanic, Stanley, Perry, Samuel, Askew, Luke, Rodríguez, Raúl Adrián Huerta, Minissi, Ali M. R., Lorena, Ricardo, Iyer, Krishnamurthy, Fasiludeen, Arshad Anil, Clark, Ronald, Ducey, Josh, Piza, Matheus, Somrak, Maja, Vergo, Eric, Qin, Juehang, Borbás, Benjámin, Chu, Eric, Lindsey, Jack, Jallon, Antoine, McInnis, I. M. J., Chen, Evan, Semler, Avi, Gloor, Luk, Shah, Tej, Carauleanu, Marc, Lauer, Pascal, Huy, Tran Đuc, Shahrtash, Hossein, Duc, Emilien, Lewark, Lukas, Brown, Assaf, Albanie, Samuel, Weber, Brian, Vaz, Warren S., Clavier, Pierre, Fan, Yiyang, Silva, Gabriel Poesia Reis e, Long, Lian, Abramovitch, Marcus, Jiang, Xi, Mendoza, Sandra, Islam, Murat, Gonzalez, Juan, Mavroudis, Vasilios, Xu, Justin, Kumar, Pawan, Goswami, Laxman Prasad, Bugas, Daniel, Heydari, Nasser, Jeanplong, Ferenc, Jansen, Thorben, Pinto, Antonella, Apronti, Archimedes, Galal, Abdallah, Ze-An, Ng, Singh, Ankit, Jiang, Tong, Xavier, Joan of Arc, Agarwal, Kanu Priya, Berkani, Mohammed, Zhang, Gang, Du, Zhehang, Junior, Benedito Alves de Oliveira, Malishev, Dmitry, Remy, Nicolas, Hartman, Taylor D., Tarver, Tim, Mensah, Stephen, Loume, Gautier Abou, Morak, Wiktor, Habibi, Farzad, Hoback, Sarah, Cai, Will, Gimenez, Javier, Montecillo, Roselynn Grace, Łucki, Jakub, Campbell, Russell, Sharma, Asankhaya, Meer, Khalida, Gul, Shreen, Gonzalez, Daniel Espinosa, Alapont, Xavier, Hoover, Alex, Chhablani, Gunjan, Vargus, Freddie, Agarwal, Arunim, Jiang, Yibo, Patil, Deepakkumar, Outevsky, David, Scaria, Kevin Joseph, Maheshwari, Rajat, Dendane, Abdelkader, Shukla, Priti, Cartwright, Ashley, Bogdanov, Sergei, Mündler, Niels, Möller, Sören, Arnaboldi, Luca, Thaman, Kunvar, Siddiqi, Muhammad Rehan, Saxena, Prajvi, Gupta, Himanshu, Fruhauff, Tony, Sherman, Glen, Vincze, Mátyás, Usawasutsakorn, Siranut, Ler, Dylan, Radhakrishnan, Anil, Enyekwe, Innocent, Salauddin, Sk Md, Muzhen, Jiang, Maksapetyan, Aleksandr, Rossbach, Vivien, Harjadi, Chris, Bahaloohoreh, Mohsen, Sparrow, Claire, Sidhu, Jasdeep, Ali, Sam, Bian, Song, Lai, John, Singer, Eric, Uro, Justine Leon, Bateman, Greg, Sayed, Mohamed, Menshawy, Ahmed, Duclosel, Darling, Bezzi, Dario, Jain, Yashaswini, Aaron, Ashley, Tiryakioglu, Murat, Siddh, Sheeshram, Krenek, Keith, Shah, Imad Ali, Jin, Jun, Creighton, Scott, Peskoff, Denis, EL-Wasif, Zienab, P, Ragavendran, Richmond, Michael, McGowan, Joseph, Patwardhan, Tejal, Sun, Hao-Yu, Sun, Ting, Zubić, Nikola, Sala, Samuele, Ebert, Stephen, Kaddour, Jean, Schottdorf, Manuel, Wang, Dianzhuo, Petruzella, Gerol, Meiburg, Alex, Medved, Tilen, ElSheikh, Ali, Hebbar, S Ashwin, Vaquero, Lorenzo, Yang, Xianjun, Poulos, Jason, Zouhar, Vilém, Bogdanik, Sergey, Zhang, Mingfang, Sanz-Ros, Jorge, Anugraha, David, Dai, Yinwei, Nhu, Anh N., Wang, Xue, Demircali, Ali Anil, Jia, Zhibai, Zhou, Yuyin, Wu, Juncheng, He, Mike, Chandok, Nitin, Sinha, Aarush, Luo, Gaoxiang, Le, Long, Noyé, Mickaël, Perełkiewicz, Michał, Pantidis, Ioannis, Qi, Tianbo, Purohit, Soham Sachin, Parcalabescu, Letitia, Nguyen, Thai-Hoa, Winata, Genta Indra, Ponti, Edoardo M., Li, Hanchen, Dhole, Kaustubh, Park, Jongee, Abbondanza, Dario, Wang, Yuanli, Nayak, Anupam, Caetano, Diogo M., Wong, Antonio A. W. L., del Rio-Chanona, Maria, Kondor, Dániel, Francois, Pieter, Chalstrey, Ed, Zsambok, Jakob, Hoyer, Dan, Reddish, Jenny, Hauser, Jakob, Rodrigo-Ginés, Francisco-Javier, Datta, Suchandra, Shepherd, Maxwell, Kamphuis, Thom, Zhang, Qizheng, Kim, Hyunjun, Sun, Ruiji, Yao, Jianzhu, Dernoncourt, Franck, Krishna, Satyapriya, Rismanchian, Sina, Pu, Bonan, Pinto, Francesco, Wang, Yingheng, Shridhar, Kumar, Overholt, Kalon J., Briia, Glib, Nguyen, Hieu, David, Bartomeu, Soler, Pang, Tony CY, Wecker, Adam, Xiong, Yifan, Li, Fanfei, Huber, Lukas S., Jaeger, Joshua, De Maddalena, Romano, Lù, Xing Han, Zhang, Yuhui, Beger, Claas, Kon, Patrick Tser Jern, Li, Sean, Sanker, Vivek, Yin, Ming, Liang, Yihao, Zhang, Xinlu, Agrawal, Ankit, Yifei, Li S., Zhang, Zechen, Cai, Mu, Sonmez, Yasin, Cozianu, Costin, Li, Changhao, Slen, Alex, Yu, Shoubin, Park, Hyun Kyu, Sarti, Gabriele, Briański, Marcin, Stolfo, Alessandro, Nguyen, Truong An, Zhang, Mike, Perlitz, Yotam, Hernandez-Orallo, Jose, Li, Runjia, Shabani, Amin, Juefei-Xu, Felix, Dhingra, Shikhar, Zohar, Orr, Nguyen, My Chiffon, Pondaven, Alexander, Yilmaz, Abdurrahim, Zhao, Xuandong, Jin, Chuanyang, Jiang, Muyan, Todoran, Stefan, Han, Xinyao, Kreuer, Jules, Rabern, Brian, Plassart, Anna, Maggetti, Martino, Yap, Luther, Geirhos, Robert, Kean, Jonathon, Wang, Dingsu, Mollaei, Sina, Sun, Chenkai, Yin, Yifan, Wang, Shiqi, Li, Rui, Chang, Yaowen, Wei, Anjiang, Bizeul, Alice, Wang, Xiaohan, Arrais, Alexandre Oliveira, Mukherjee, Kushin, Chamorro-Padial, Jorge, Liu, Jiachen, Qu, Xingyu, Guan, Junyi, Bouyamourn, Adam, Wu, Shuyu, Plomecka, Martyna, Chen, Junda, Tang, Mengze, Deng, Jiaqi, Subramanian, Shreyas, Xi, Haocheng, Chen, Haoxuan, Zhang, Weizhi, Ren, Yinuo, Tu, Haoqin, Kim, Sejong, Chen, Yushun, Marjanović, Sara Vera, Ha, Junwoo, Luczyna, Grzegorz, Ma, Jeff J., Shen, Zewen, Song, Dawn, Zhang, Cedegao E., Wang, Zhun, Gendron, Gaël, Xiao, Yunze, Smucker, Leo, Weng, Erica, Lee, Kwok Hao, Ye, Zhe, Ermon, Stefano, Lopez-Miguel, Ignacio D., Knights, Theo, Gitter, Anthony, Park, Namkyu, Wei, Boyi, Chen, Hongzheng, Pai, Kunal, Elkhanany, Ahmed, Lin, Han, Siedler, Philipp D., Fang, Jichao, Mishra, Ritwik, Zsolnai-Fehér, Károly, Jiang, Xilin, Khan, Shadab, Yuan, Jun, Jain, Rishab Kumar, Lin, Xi, Peterson, Mike, Wang, Zhe, Malusare, Aditya, Tang, Maosen, Gupta, Isha, Fosin, Ivan, Kang, Timothy, Dworakowska, Barbara, Matsumoto, Kazuki, Zheng, Guangyao, Sewuster, Gerben, Villanueva, Jorge Pretel, Rannev, Ivan, Chernyavsky, Igor, Chen, Jiale, Banik, Deepayan, Racz, Ben, Dong, Wenchao, Wang, Jianxin, Bashmal, Laila, Gonçalves, Duarte V., Hu, Wei, Bar, Kaushik, Bohdal, Ondrej, Patlan, Atharv Singh, Dhuliawala, Shehzaad, Geirhos, Caroline, Wist, Julien, Kansal, Yuval, Chen, Bingsen, Tire, Kutay, Yücel, Atak Talay, Christof, Brandon, Singla, Veerupaksh, Song, Zijian, Chen, Sanxing, Ge, Jiaxin, Ponkshe, Kaustubh, Park, Isaac, Shi, Tianneng, Ma, Martin Q., Mak, Joshua, Lai, Sherwin, Moulin, Antoine, Cheng, Zhuo, Zhu, Zhanda, Zhang, Ziyi, Patil, Vaidehi, Jha, Ketan, Men, Qiutong, Wu, Jiaxuan, Zhang, Tianchi, Vieira, Bruno Hebling, Aji, Alham Fikri, Chung, Jae-Won, Mahfoud, Mohammed, Hoang, Ha Thi, Sperzel, Marc, Hao, Wei, Meding, Kristof, Xu, Sihan, Kostakos, Vassilis, Manini, Davide, Liu, Yueying, Toukmaji, Christopher, Paek, Jay, Yu, Eunmi, Demircali, Arif Engin, Sun, Zhiyi, Dewerpe, Ivan, Qin, Hongsen, Pflugfelder, Roman, Bailey, James, Morris, Johnathan, Heilala, Ville, Rosset, Sybille, Yu, Zishun, Chen, Peter E., Yeo, Woongyeong, Jain, Eeshaan, Yang, Ryan, Chigurupati, Sreekar, Chernyavsky, Julia, Reddy, Sai Prajwal, Venugopalan, Subhashini, Batra, Hunar, Park, Core Francisco, Tran, Hieu, Maximiano, Guilherme, Zhang, Genghan, Liang, Yizhuo, Shiyu, Hu, Xu, Rongwu, Pan, Rui, Suresh, Siddharth, Liu, Ziqi, Gulati, Samaksh, Zhang, Songyang, Turchin, Peter, Bartlett, Christopher W., Scotese, Christopher R., Cao, Phuong M., Wu, Ben, Karwowski, Jacek, Scaramuzza, Davide, Nattanmai, Aakaash, McKellips, Gordon, Cheraku, Anish, Suhail, Asim, Luo, Ethan, Deng, Marvin, Luo, Jason, Zhang, Ashley, Jindel, Kavin, Paek, Jay, Halevy, Kasper, Baranov, Allen, Liu, Michael, Avadhanam, Advaith, Zhang, David, Cheng, Vincent, Ma, Brad, Fu, Evan, Do, Liam, Lass, Joshua, Yang, Hubert, Sunkari, Surya, Bharath, Vishruth, Ai, Violet, Leung, James, Agrawal, Rishit, Zhou, Alan, Chen, Kevin, Kalpathi, Tejas, Xu, Ziqi, Wang, Gavin, Xiao, Tyler, Maung, Erik, Lee, Sam, Yang, Ryan, Yue, Roy, Zhao, Ben, Yoon, Julia, Sun, Sunny, Singh, Aryan, Luo, Ethan, Peng, Clark, Osbey, Tyler, Wang, Taozhi, Echeazu, Daryl, Yang, Hubert, Wu, Timothy, Patel, Spandan, Kulkarni, Vidhi, Sundarapandiyan, Vijaykaarti, Zhang, Ashley, Le, Andrew, Nasim, Zafir, Yalam, Srikar, Kasamsetty, Ritesh, Samal, Soham, Yang, Hubert, Sun, David, Shah, Nihar, Saha, Abhijeet, Zhang, Alex, Nguyen, Leon, Nagumalli, Laasya, Wang, Kaixin, Zhou, Alan, Wu, Aidan, Luo, Jason, Telluri, Anwith, Dillmann, Steven, Wang, Zhengxiang, Luo, Junyu, Lunn, Hugo, Gazizov, Artem, Borde, Haitz Sáez de Ocáriz, Trus, Ivan, Hervault, Morgan, Zhang, Zheyu, Chen, Bo, Wu, Yuchen, Cordier, Christopher J., Kaynar, Gün, Ayvaz, Cansin, Avdiunina, Polina, Brust, Johannes, Diao, Xingjian, Meaney, K. D., Gu, Yifan, Wang, Chenyu, Dong, Chenzhuo, Wright, William, Brave, Simon, Root, Owen, Liu, Jiayuan, Lok, Chow Chun, Li, Tianqin, Du, Shiyi, He, Dailan, Liu, Lufeiya, Jamalzadegan, Sina, Ramakrishna, Anil, Xu, Xuanqing, Qing, Xin, Luo, Xin, Li, Wenkai, Bo, Shi, Gusev, Filipp, Skandalis, Maximos, Ma, Desheng, Zhang, Chunhui, Qiu, Haoran, Hart, Allen G, Gabrielsson, Rickard Brüel, Akov, Ido, Lukoianov, Artem, Hu, Pengchong, Li, Siting, Lyu, Weimin, Zhao, Xingchen, Yang, Tianyu, Wu, Weiyi, Lei, Deren, Baber, Robert W., Yuan, Xiangchi, Wang, Lanyi, Cui, Yanjun, David, Johannes-Rudolf, Potlapelly, Sujay, Kalfus, Jeremy, Xu, Haotian, Bendahi, Abderrahim, Bao, Qiming, Jia, Yingxin, Gui, Jiang, Hu, Jiaming, Liu, Zheyuan, Şık, Furkan, Handoko, Brandon, Yue, Summer, Wang, Alexandr, Hendrycks, Dan · 2025 · arXiv

Published version of a preprint

Abstract

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.

Overview

Integrity signals

Registry

DataCite · 10.48550/arxiv.2501.14249

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Performance of frontier LLMs on popular benchmarks and HLE. Compared with the saturation of other popular capability benchmarks, HLE accuracy remains low across several frontier models, demonstrating its effectiveness for measuring advanced, closed-ended, academic capabilities.
Fig. 1 Performance of frontier LLMs on popular benchmarks and HLE. Compared with the saturation of other popular capability benchmarks, HLE accuracy remains low across several frontier models, demonstrating its effectiveness for measuring advanced, closed-ended, academic capabilities.CC BY-NC-ND · PubMed Central
Fig. 2 Distribution of HLE questions across categories. HLE consists of 2,500 exam questions in over a hundred subjects, grouped into eight high-level categories.
Fig. 2 Distribution of HLE questions across categories. HLE consists of 2,500 exam questions in over a hundred subjects, grouped into eight high-level categories.CC BY-NC-ND · PubMed Central
Fig. 3 Example questions from HLE. Samples of the diverse and challenging questions submitted to HLE.
Fig. 3 Example questions from HLE. Samples of the diverse and challenging questions submitted to HLE.CC BY-NC-ND · PubMed Central
Fig. 4 HLE dataset creation pipeline. We accept questions that make frontier LLMs fail, then iteratively refine them with the help of expert peer reviewers. Each question is then manually approved by organizers or expert reviewers trained by organizers. A private held-out set is kept apart from the public set to assess model overfitting and gaming on the public benchmark.
Fig. 4 HLE dataset creation pipeline. We accept questions that make frontier LLMs fail, then iteratively refine them with the help of expert peer reviewers. Each question is then manually approved by organizers or expert reviewers trained by organizers. A private held-out set is kept apart from the public set to assess model overfitting and gaming on the public benchmark.CC BY-NC-ND · PubMed Central
Fig. 5 Accuracy compared with reasoning token budget. Accuracy binned by the total number of generated output tokens, showing a log-linear increase in accuracy peaking around 214 tokens before reversing.
Fig. 5 Accuracy compared with reasoning token budget. Accuracy binned by the total number of generated output tokens, showing a log-linear increase in accuracy peaking around 214 tokens before reversing.CC BY-NC-ND · PubMed Central
Extended Data Fig. 1 Example of a structured response using an LLM judge. Exact-match answers in HLE sometimes require several reasoning steps to compare the AI’s final answer with the correct answer; therefore, a capable LLM judge with reasoning capabilities is necessary.
Extended Data Fig. 1 Example of a structured response using an LLM judge. Exact-match answers in HLE sometimes require several reasoning steps to compare the AI’s final answer with the correct answer; therefore, a capable LLM judge with reasoning capabilities is necessary.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

2 Valid · 5 Not valid · 0 Not checked · 33 Own results and interpretations

14 more claims

Create a free account to see all 40 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 18
  • It may state two separate claims as one. 12

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 14 references

Research works printed but not linked: 16

  1. [1]Hendrycks, D. et al. Measuring massive multitask language understanding. In Proc. International Conference on Learning Representations (ICLR) https://openreview.net/forum?id=d7KBjmI3GmQ (ICLR, 2021).
  2. [9]Rein, D. et al. GPQA: A graduate-level Google-proof Q&A benchmark. In Proc. First Conference on Language Modeling (COLM) https://openreview.net/forum?id=Ti67584b98 (COLM, 2024).
  3. [12]Li, N. et al. The WMDP benchmark: measuring and reducing malicious use with unlearning. In Proc. 41st International Conference on Machine Learning (ICML), 28713–28738 (PMLR, 2024).
  4. [14]Srivastava, A. et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Trans. Mach. Learn. Res. https://openreview.net/forum?id=uyTL5Bvosj (2023).
  5. [16]Hendrycks, D. et al. Measuring mathematical problem solving with the MATH dataset. In Proc. 35th Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track https://openreview.net/forum?id=7Bywt2mQsCe (NeurIPS, 2021).
  6. [17]Lu, P. et al. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In Proc. International Conference on Learning Representations (ICLR) https://openreview.net/forum?id=KUNzEQMWU7 (ICLR, 2024).
  7. [21]Gao, B. et al. Omni-MATH: A universal Olympiad level mathematic benchmark for large language models. In Proc. International Conference on Learning Representations (ICLR) https://openreview.net/forum?id=yaqPf0KAlN (ICLR, 2025).
  8. [22]Chan, J. S. et al. MLE-bench: Evaluating machine learning agents on machine learning engineering. In Proc. International Conference on Learning Representations (ICLR) https://openreview.net/forum?id=6s5uXNWGIh (ICLR, 2025).
  9. [23]Zhang, A. K. et al. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. In Proc. International Conference on Learning Representations (ICLR) https://openreview.net/forum?id=tc90LV0yRL (ICLR, 2025).
  10. [24]Jimenez, C. E. et al. SWE-bench: Can language models resolve real-world Github issues? In Proc. International Conference on Learning Representations (ICLR) https://openreview.net/forum?id=VTF8yNQM66 (ICLR, 2024).
  11. [30]Perez, E. et al. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics: ACL 2023, 13387–13434 (ACL, 2023).
  12. [34]Hendrycks, D. et al. What would Jiminy Cricket do? Towards agents that behave morally. In Proc. 35th Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track https://openreview.net/forum?id=G1muTb5zuO7 (NeurIPS, 2021).
  13. [37]Wang, A. et al. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Vol. 32, 3261–3275 (NeurIPS, 2019).
  14. [46]Yao, S. et al. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. In Proc. International Conference on Learning Representations (ICLR) https://openreview.net/forum?id=roNSXZpUDN (ICLR, 2025).
  15. [47]Andriushchenko, M. et al. AgentHarm: A benchmark for measuring harmfulness of LLM agents. In Proc. International Conference on Learning Representations (ICLR) https://openreview.net/forum?id=AC5n7xHuR1 (ICLR, 2025).
  16. [50]Srinivasan, V. K. et al. NexusRaven: a commercially-permissive language model for function calling. In NeurIPS 2023 Foundation Models for Decision Making Workshop (NeurIPS, 2023).

Cited by

See all 1,003

Metadata

Authors
Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Zhang, C. B. C., Shaaban, M., Ling, J., Shi, S., Choi, M., Agrawal, A., Chopra, A., Khoja, A., Kim, R., Ren, R., Hausenloy, J., Zhang, O., Mazeika, M., Dodonov, D., Nguyen, T., Lee, J., Anderson, D., Doroshenko, M., Stokes, A. C., Mahmood, M., Pokutnyi, O., Iskra, O., Wang, J. P., Levin, J.-C., Kazakov, M., Feng, F., Feng, S. Y., Zhao, H., Yu, M., Gangal, V., Zou, C., Wang, Z., Popov, S., Gerbicz, R., Galgon, G., Schmitt, J., Yeadon, W., Lee, Y., Sauers, S., Sanchez, A., Giska, F., Roth, M., Riis, S., Utpala, S., Burns, N., Goshu, G. M., Naiya, M. M., Agu, C., Giboney, Z., Cheatom, A., Fournier-Facio, F., Crowson, S.-J., Finke, L., Cheng, Z., Zampese, J., Hoerr, R. G., Nandor, M., Park, H., Gehrunger, T., Cai, J., McCarty, B., Garretson, A. C., Taylor, E., Sileo, D., Ren, Q., Qazi, U., Li, L., Nam, J., Wydallis, J. B., Arkhipov, P., Shi, J. W. L., Bacho, A., Willcocks, C. G., Cao, H., Motwani, S., Santos, E. D. O., Veith, J., Vendrow, E., Cojoc, D., Zenitani, K., Robinson, J., Tang, L., Li, Y., Vendrow, J., Fraga, N. W., Kuchkin, V., Maksimov, A. P., Marion, P., Efremov, D., Lynch, J., Liang, K., Mikov, A., Gritsevskiy, A., Guillod, J., Demir, G., Martinez, D., Pageler, B., Zhou, K., Soori, S., Press, O., Tang, H., Rissone, P., Green, S. R., Brüssel, L., Twayana, M., Dieuleveut, A., Imperial, J. M., Prabhu, A., Yang, J., Crispino, N., Rao, A., Zvonkine, D., Loiseau, G., Kalinin, M., Lukas, M., Manolescu, C., Stambaugh, N., Mishra, S., Hogg, T., Bosio, C., Coppola, B. P., Salazar, J., Jin, J., Sayous, R., Ivanov, S., Schwaller, P., Senthilkuma, S., Bran, A. M., Algaba, A., Houte, K. V. D., Van Der Sypt, L., Verbeken, B. F., Noever, D., Kopylov, A., Myklebust, B., Li, B., Schut, L., Zheltonozhskii, E., Yuan, Q., Lim, D., Stanley, R., Yang, T., Maar, J., Wykowski, J., Oller, M., Sahu, A., Ardito, C. G., Hu, Y., Kamdoum, A. G. K., Jin, A., Vilchis, T. G., Zu, Y., Lackner, M., Koppel, J., Sun, G., Antonenko, D. S., Chern, S., Zhao, B., Arsene, P., Cavanagh, J. M., Li, D., Shen, J., Crisostomi, D., Zhang, W., Dehghan, A., Ivanov, S., Perrella, D., Kaparov, N., Zang, A., Sucholutsky, I., Kharlamova, A., Orel, D., Poritski, V., Ben-David, S., Berger, Z., Whitfill, P., Foster, M., Munro, D., Ho, L., Sivarajan, S., Hava, D. B., Kuchkin, A., Holmes, D., Rodriguez-Romero, A., Sommerhage, F., Zhang, A., Moat, R., Schneider, K., Kazibwe, Z., Clarke, D., Kim, D. H., Dias, F. M., Fish, S., Elser, V., Kreiman, T., Vilchis, V. E. G., Klose, I., Anantheswaran, U., Zweiger, A., Rawal, K., Li, J., Nguyen, J., Daans, N., Heidinger, H., Radionov, M., Rozhoň, V., Ginis, V., Stump, C., Cohen, N., Poświata, R., Tkadlec, J., Goldfarb, A., Wang, C., Padlewski, P., Barzowski, S., Montgomery, K., Stendall, R., Tucker-Foltz, J., Stade, J., Rogers, T. R., Goertzen, T., Grabb, D., Shukla, A., Givré, A., Ambay, J. A., Sen, A., Aziz, M. F., Inlow, M. H., He, H., Zhang, L., Kaddar, Y., Ängquist, I., Chen, Y., Wang, H. K., Ramakrishnan, K., Thornley, E., Terpin, A., Schoelkopf, H., Zheng, E., Carmi, A., Brown, E. D. L., Zhu, K., Bartolo, M., Wheeler, R., Stehberger, M., Bradshaw, P., Heimonen, J., Sridhar, K., Akov, I., Sandlin, J., Makarychev, Y., Tam, J., Hoang, H., Cunningham, D. M., Goryachev, V., Patramanis, D., Krause, M., Redenti, A., Aldous, D., Lai, J., Coleman, S., Xu, J., Lee, S., Magoulas, I., Zhao, S., Tang, N., Cohen, M. K., Paradise, O., Kirchner, J. H., Ovchynnikov, M., Matos, J. O., Shenoy, A., Wang, M., Nie, Y., Sztyber-Betley, A., Faraboschi, P., Riblet, R., Crozier, J., Halasyamani, S., Verma, S., Joshi, P., Meril, E., Ma, Z., Andréoletti, J., Singhal, R., Platnick, J., Nevirkovets, V., Basler, L., Ivanov, A., Khoury, S., Gustafsson, N., Piccardo, M., Mostaghimi, H., Chen, Q., Singh, V., Khánh, T. Q., Rosu, P., Szlyk, H., Brown, Z., Narayan, H., Menezes, A., Roberts, J., Alley, W., Sun, K., Patel, A., Lamparth, M., Reuel, A., Xin, L., Xu, H., Loader, J., Martin, F., Wang, Z., Achilleos, A., Preu, T., Korbak, T., Bosio, I., Kazemi, F., Chen, Z., Bálint, B., Lo, E. J. Y., Wang, J., Nunes, M. I. S., Milbauer, J., Bari, M. S., Wang, Z., Ansarinejad, B., Sun, Y., Durand, S., Elgnainy, H., Douville, G., Tordera, D., Balabanian, G., Wolff, H., Kvistad, L., Milliron, H., Sakor, A., Eron, M., O., A. F. D., Shah, S., Zhou, X., Kamalov, F., Abdoli, S., Santens, T., Barkan, S., Tee, A., Zhang, R., Tomasiello, A., De Luca, G. B., Looi, S.-Z., Le, V.-K., Kolt, N., Pan, J., Rodman, E., Drori, J., Fossum, C. J., Muennighoff, N., Jagota, M., Pradeep, R., Fan, H., Eicher, J., Chen, M., Thaman, K., Merrill, W., Firsching, M., Harris, C., Ciobâcă, S., Gross, J., Pandey, R., Gusev, I., Jones, A., Agnihotri, S., Zhelnov, P., Mofayezi, M., Piperski, A., Zhang, D. K., Dobarskyi, K., Leventov, R., Soroko, I., Duersch, J., Taamazyan, V., Ho, A., Ma, W., Held, W., Xian, R., Zebaze, A. R., Mohamed, M., Leser, J. N., Yuan, M. X., Yacar, L., Lengler, J., Olszewska, K., Di Fratta, C., Oliveira, E., Jackson, J. W., Zou, A., Chidambaram, M., Manik, T., Haffenden, H., Stander, D., Dasouqi, A., Shen, A., Golshani, B., Stap, D., Kretov, E., Uzhou, M., Zhidkovskaya, A. B., Winter, N., Rodriguez, M. O., Lauff, R., Wehr, D., Tang, C., Hossain, Z., Phillips, S., Samuele, F., Ekström, F., Hammon, A., Patel, O., Farhidi, F., Medley, G., Mohammadzadeh, F., Peñaflor, M., Kassahun, H., Friedrich, A., Perez, R. H., Pyda, D., Sakal, T., Dhamane, O., Mirabadi, A. K., Hallman, E., Okutsu, K., Battaglia, M., Maghsoudimehrabani, M., Amit, A., Hulbert, D., Pereira, R., Weber, S., Handoko, Peristyy, A., Malina, S., Mehkary, M., Aly, R., Reidegeld, F., Dick, A.-K., Friday, C., Singh, M., Shapourian, H., Kim, W., Costa, M., Gurdogan, H., Kumar, H., Ceconello, C., Zhuang, C., Park, H., Carroll, M., Tawfeek, A. R., Steinerberger, S., Aggarwal, D., Kirchhof, M., Dai, L., Kim, E., Ferret, J., Shah, J., Wang, Y., Yan, M., Burdzy, K., Zhang, L., Franca, A., Pham, D. T., Loh, K. Y., Robinson, J., Jackson, A., Giordano, P., Petersen, P., Cosma, A., Colino, J., White, C., Votava, J., Vinnikov, V., Delaney, E., Spelda, P., Stritecky, V., Shahid, S. M., Mourrat, J.-C., Vetoshkin, L., Sponselee, K., Bacho, R., Yong, Z.-X., de la Rosa, F., Cho, N., Li, X., Malod, G., Weller, O., Albani, G., Lang, L., Laurendeau, J., Kazakov, D., Adesanya, F., Portier, J., Hollom, L., Souza, V., Zhou, Y. A., Degorre, J., Yalın, Y., Obikoya, G. D., Rai, Bigi, F., Boscá, M. C., Shumar, O., Bacho, K., Recchia, G., Popescu, M., Shulga, N., Tanwie, N. M., Lux, T. C. H., Rank, B., Ni, C., Brooks, M., Yakimchyk, A., Huanxu, Liu, Cavalleri, S., Häggström, O., Verkama, E., Newbould, J., Gundlach, H., Brito-Santana, L., Amaro, B., Vajipey, V., Grover, R., Wang, T., Kratish, Y., Li, W.-D., Gopi, S., Caciolai, A., de Witt, C. S., Hernández-Cámara, P., Rodolà, E., Robins, J., Williamson, D., Cheng, V., Raynor, B., Qi, H., Segev, B., Fan, J., Martinson, S., Wang, E. Y., Hausknecht, K., Brenner, M. P., Mao, M., Demian, C., Kassani, P., Zhang, X., Avagian, D., Scipio, E. J., Ragoler, A., Tan, J., Sims, B., Plecnik, R., Kirtland, A., Bodur, O. F., Shinde, D. P., Labrador, Y. C. L., Adoul, Z., Zekry, M., Karakoc, A., Santos, T. C. B., Shamseldeen, S., Karim, L., Liakhovitskaia, A., Resman, N., Farina, N., Gonzalez, J. C., Maayan, G., Anderson, E., Pena, R. D. O., Kelley, E., Mariji, H., Pouriamanesh, R., Wu, W., Finocchio, R., Alarab, I., Cole, J., Ferreira, D., Johnson, B., Safdari, M., Dai, L., Arthornthurasuk, S., McAlister, I. C., Moyano, A. J., Pronin, A., Fan, J., Ramirez-Trinidad, A., Malysheva, Y., Pottmaier, D., Taheri, O., Stepanic, S., Perry, S., Askew, L., Rodríguez, R. A. H., Minissi, A. M. R., Lorena, R., Iyer, K., Fasiludeen, A. A., Clark, R., Ducey, J., Piza, M., Somrak, M., Vergo, E., Qin, J., Borbás, B., Chu, E., Lindsey, J., Jallon, A., McInnis, I. M. J., Chen, E., Semler, A., Gloor, L., Shah, T., Carauleanu, M., Lauer, P., Huy, T. Đ., Shahrtash, H., Duc, E., Lewark, L., Brown, A., Albanie, S., Weber, B., Vaz, W. S., Clavier, P., Fan, Y., Silva, G. P. R. E., Long, Lian, Abramovitch, M., Jiang, X., Mendoza, S., Islam, M., Gonzalez, J., Mavroudis, V., Xu, J., Kumar, P., Goswami, L. P., Bugas, D., Heydari, N., Jeanplong, F., Jansen, T., Pinto, A., Apronti, A., Galal, A., Ze-An, N., Singh, A., Jiang, T., Xavier, J. O. A., Agarwal, K. P., Berkani, M., Zhang, G., Du, Z., Junior, B. A. D. O., Malishev, D., Remy, N., Hartman, T. D., Tarver, T., Mensah, S., Loume, G. A., Morak, W., Habibi, F., Hoback, S., Cai, W., Gimenez, J., Montecillo, R. G., Łucki, J., Campbell, R., Sharma, A., Meer, K., Gul, S., Gonzalez, D. E., Alapont, X., Hoover, A., Chhablani, G., Vargus, F., Agarwal, A., Jiang, Y., Patil, D., Outevsky, D., Scaria, K. J., Maheshwari, R., Dendane, A., Shukla, P., Cartwright, A., Bogdanov, S., Mündler, N., Möller, S., Arnaboldi, L., Thaman, K., Siddiqi, M. R., Saxena, P., Gupta, H., Fruhauff, T., Sherman, G., Vincze, M., Usawasutsakorn, S., Ler, D., Radhakrishnan, A., Enyekwe, I., Salauddin, S. M., Muzhen, J., Maksapetyan, A., Rossbach, V., Harjadi, C., Bahaloohoreh, M., Sparrow, C., Sidhu, J., Ali, S., Bian, S., Lai, J., Singer, E., Uro, J. L., Bateman, G., Sayed, M., Menshawy, A., Duclosel, D., Bezzi, D., Jain, Y., Aaron, A., Tiryakioglu, M., Siddh, S., Krenek, K., Shah, I. A., Jin, J., Creighton, S., Peskoff, D., EL-Wasif, Z., P, R., Richmond, M., McGowan, J., Patwardhan, T., Sun, H.-Y., Sun, T., Zubić, N., Sala, S., Ebert, S., Kaddour, J., Schottdorf, M., Wang, D., Petruzella, G., Meiburg, A., Medved, T., ElSheikh, A., Hebbar, S. A., Vaquero, L., Yang, X., Poulos, J., Zouhar, V., Bogdanik, S., Zhang, M., Sanz-Ros, J., Anugraha, D., Dai, Y., Nhu, A. N., Wang, X., Demircali, A. A., Jia, Z., Zhou, Y., Wu, J., He, M., Chandok, N., Sinha, A., Luo, G., Le, L., Noyé, M., Perełkiewicz, M., Pantidis, I., Qi, T., Purohit, S. S., Parcalabescu, L., Nguyen, T.-H., Winata, G. I., Ponti, E. M., Li, H., Dhole, K., Park, J., Abbondanza, D., Wang, Y., Nayak, A., Caetano, D. M., Wong, A. A. W. L., del Rio-Chanona, M., Kondor, D., Francois, P., Chalstrey, E., Zsambok, J., Hoyer, D., Reddish, J., Hauser, J., Rodrigo-Ginés, F.-J., Datta, S., Shepherd, M., Kamphuis, T., Zhang, Q., Kim, H., Sun, R., Yao, J., Dernoncourt, F., Krishna, S., Rismanchian, S., Pu, B., Pinto, F., Wang, Y., Shridhar, K., Overholt, K. J., Briia, G., Nguyen, H., David, Bartomeu, S., Pang, T. C., Wecker, A., Xiong, Y., Li, F., Huber, L. S., Jaeger, J., De Maddalena, R., Lù, X. H., Zhang, Y., Beger, C., Kon, P. T. J., Li, S., Sanker, V., Yin, M., Liang, Y., Zhang, X., Agrawal, A., Yifei, L. S., Zhang, Z., Cai, M., Sonmez, Y., Cozianu, C., Li, C., Slen, A., Yu, S., Park, H. K., Sarti, G., Briański, M., Stolfo, A., Nguyen, T. A., Zhang, M., Perlitz, Y., Hernandez-Orallo, J., Li, R., Shabani, A., Juefei-Xu, F., Dhingra, S., Zohar, O., Nguyen, M. C., Pondaven, A., Yilmaz, A., Zhao, X., Jin, C., Jiang, M., Todoran, S., Han, X., Kreuer, J., Rabern, B., Plassart, A., Maggetti, M., Yap, L., Geirhos, R., Kean, J., Wang, D., Mollaei, S., Sun, C., Yin, Y., Wang, S., Li, R., Chang, Y., Wei, A., Bizeul, A., Wang, X., Arrais, A. O., Mukherjee, K., Chamorro-Padial, J., Liu, J., Qu, X., Guan, J., Bouyamourn, A., Wu, S., Plomecka, M., Chen, J., Tang, M., Deng, J., Subramanian, S., Xi, H., Chen, H., Zhang, W., Ren, Y., Tu, H., Kim, S., Chen, Y., Marjanović, S. V., Ha, J., Luczyna, G., Ma, J. J., Shen, Z., Song, D., Zhang, C. E., Wang, Z., Gendron, G., Xiao, Y., Smucker, L., Weng, E., Lee, K. H., Ye, Z., Ermon, S., Lopez-Miguel, I. D., Knights, T., Gitter, A., Park, N., Wei, B., Chen, H., Pai, K., Elkhanany, A., Lin, H., Siedler, P. D., Fang, J., Mishra, R., Zsolnai-Fehér, K., Jiang, X., Khan, S., Yuan, J., Jain, R. K., Lin, X., Peterson, M., Wang, Z., Malusare, A., Tang, M., Gupta, I., Fosin, I., Kang, T., Dworakowska, B., Matsumoto, K., Zheng, G., Sewuster, G., Villanueva, J. P., Rannev, I., Chernyavsky, I., Chen, J., Banik, D., Racz, B., Dong, W., Wang, J., Bashmal, L., Gonçalves, D. V., Hu, W., Bar, K., Bohdal, O., Patlan, A. S., Dhuliawala, S., Geirhos, C., Wist, J., Kansal, Y., Chen, B., Tire, K., Yücel, A. T., Christof, B., Singla, V., Song, Z., Chen, S., Ge, J., Ponkshe, K., Park, I., Shi, T., Ma, M. Q., Mak, J., Lai, S., Moulin, A., Cheng, Z., Zhu, Z., Zhang, Z., Patil, V., Jha, K., Men, Q., Wu, J., Zhang, T., Vieira, B. H., Aji, A. F., Chung, J.-W., Mahfoud, M., Hoang, H. T., Sperzel, M., Hao, W., Meding, K., Xu, S., Kostakos, V., Manini, D., Liu, Y., Toukmaji, C., Paek, J., Yu, E., Demircali, A. E., Sun, Z., Dewerpe, I., Qin, H., Pflugfelder, R., Bailey, J., Morris, J., Heilala, V., Rosset, S., Yu, Z., Chen, P. E., Yeo, W., Jain, E., Yang, R., Chigurupati, S., Chernyavsky, J., Reddy, S. P., Venugopalan, S., Batra, H., Park, C. F., Tran, H., Maximiano, G., Zhang, G., Liang, Y., Shiyu, H., Xu, R., Pan, R., Suresh, S., Liu, Z., Gulati, S., Zhang, S., Turchin, P., Bartlett, C. W., Scotese, C. R., Cao, P. M., Wu, B., Karwowski, J., Scaramuzza, D., Nattanmai, A., McKellips, G., Cheraku, A., Suhail, A., Luo, E., Deng, M., Luo, J., Zhang, A., Jindel, K., Paek, J., Halevy, K., Baranov, A., Liu, M., Avadhanam, A., Zhang, D., Cheng, V., Ma, B., Fu, E., Do, L., Lass, J., Yang, H., Sunkari, S., Bharath, V., Ai, V., Leung, J., Agrawal, R., Zhou, A., Chen, K., Kalpathi, T., Xu, Z., Wang, G., Xiao, T., Maung, E., Lee, S., Yang, R., Yue, R., Zhao, B., Yoon, J., Sun, S., Singh, A., Luo, E., Peng, C., Osbey, T., Wang, T., Echeazu, D., Yang, H., Wu, T., Patel, S., Kulkarni, V., Sundarapandiyan, V., Zhang, A., Le, A., Nasim, Z., Yalam, S., Kasamsetty, R., Samal, S., Yang, H., Sun, D., Shah, N., Saha, A., Zhang, A., Nguyen, L., Nagumalli, L., Wang, K., Zhou, A., Wu, A., Luo, J., Telluri, A., Dillmann, S., Wang, Z., Luo, J., Lunn, H., Gazizov, A., Borde, H. S. D. O., Trus, I., Hervault, M., Zhang, Z., Chen, B., Wu, Y., Cordier, C. J., Kaynar, G., Ayvaz, C., Avdiunina, P., Brust, J., Diao, X., Meaney, K. D., Gu, Y., Wang, C., Dong, C., Wright, W., Brave, S., Root, O., Liu, J., Lok, C. C., Li, T., Du, S., He, D., Liu, L., Jamalzadegan, S., Ramakrishna, A., Xu, X., Qing, X., Luo, X., Li, W., Bo, S., Gusev, F., Skandalis, M., Ma, D., Zhang, C., Qiu, H., Hart, A. G., Gabrielsson, R. B., Akov, I., Lukoianov, A., Hu, P., Li, S., Lyu, W., Zhao, X., Yang, T., Wu, W., Lei, D., Baber, R. W., Yuan, X., Wang, L., Cui, Y., David, J.-R., Potlapelly, S., Kalfus, J., Xu, H., Bendahi, A., Bao, Q., Jia, Y., Gui, J., Hu, J., Liu, Z., Şık, F., Handoko, B., Yue, S., Wang, A., & Hendrycks, D.
Affiliations
  • Michael Pokorny
  • Massachusetts Institute of Technology (US)
  • George Mason University (US)
  • Brown University (US)
  • McGill University (CA)
  • Carnegie Mellon University (US)
  • University of Washington (US)
  • University of Cambridge (GB)
  • Queen's University (CA)
  • Stanford University (US)
  • Novo Nordisk (Denmark) (DK)
  • Universidade do Porto (PT)
  • Eötvös Loránd University (HU)
  • ETH Zurich (CH)
  • Durham University (GB)
  • University of Minnesota (US)
  • Queen Mary University of London (GB)
  • Microsoft (United States) (US)
  • Auckland University of Technology (NZ)
  • Alberta Health Services (CA)
  • University of Illinois Chicago (US)
  • MAGNET: Machine Learning in Information Networks (FR)
  • Princeton University (US)
  • University of Canterbury (NZ)
  • Metropolitan State University of Denver (US)
  • Seoul National University (KR)
  • Accenture (United States) (US)
  • Tufts University (US)
  • Institut national de recherche en sciences et technologies du numérique (FR)
  • University of California, Berkeley (US)
  • University of British Columbia (CA)
  • École Polytechnique Fédérale de Lausanne (CH)
  • Concordia University (CA)
  • Institute of Science and Technology Austria (AT)
  • National University of Singapore (SG)
  • California Institute of Technology (US)
  • University of Oxford (GB)
  • Humboldt-Universität zu Berlin (DE)
  • Columbia University (US)
  • Stony Brook University (US)
  • University of Luxembourg (LU)
  • Universidade Federal de Juiz de Fora (BR)
  • Rockwell Automation (United States) (US)
  • École Normale Supérieure - PSL (FR)
  • University of Toronto (CA)
  • University of Tübingen (DE)
  • Sapienza University of Rome (IT)
  • École Normale Supérieure Paris-Saclay (FR)
  • University of North Texas (US)
  • Institut Polytechnique de Paris (FR)
  • National University (PH)
  • Maastricht University (NL)
  • Washington University in St. Louis (US)
  • University of California, Los Angeles (US)
  • Centre National de la Recherche Scientifique (FR)
  • Martin Luther University Halle-Wittenberg (DE)
  • Leibniz University Hannover (DE)
  • Indian Institute of Technology Bombay (IN)
  • Institute for Molecular Manufacturing (US)
  • University of Michigan (US)
  • Université Paris-Saclay (FR)
  • Vrije Universiteit Brussel (BE)
  • PeopleTec (United States) (US)
  • University of Chicago (US)
  • Technion – Israel Institute of Technology (IL)
  • University of Miami (US)
  • Technische Universität Berlin (DE)
  • University of Manchester (GB)
  • University of Illinois Urbana-Champaign (US)
  • University of Calgary (CA)
  • Universidad Iberoamericana (MX)
  • TU Wien (AT)
  • University of Wisconsin–Madison (US)
  • Yale University (US)
  • University of Edinburgh (GB)
  • The University of Western Australia (AU)
  • New York University (US)
  • Mohamed bin Zayed University of Artificial Intelligence (AE)
  • University of Waterloo (CA)
  • University of California San Diego (US)
  • University of Maryland, College Park (US)
  • Manhattan School of Music (US)
  • Leiden University (NL)
  • The Open University (GB)
  • Corteva (United States) (US)
  • Sanford Burnham Prebys Medical Discovery Institute (US)
  • Yonsei University (KR)
  • Universidade de São Paulo (BR)
  • Harvard University (US)
  • Cornell University (US)
  • University of Leeds (GB)
  • Arizona State University (US)
  • Swinburne University of Technology (AU)
  • KU Leuven (BE)
  • Universidad Nacional Agraria La Molina (PE)
  • Brandenburg University of Technology Cottbus-Senftenberg (DE)
  • Ruhr University Bochum (DE)
  • National Information Processing Institute (PL)
  • Charles University (CZ)
  • Cranfield University (GB)
  • University of Copenhagen (DK)
  • The University of Sydney (AU)
  • Indian Institute of Technology Delhi (IN)
  • Universidad de Buenos Aires (AR)
  • University of Technology Sydney (AU)
  • Indiana State University (US)
  • Australian National University (AU)
  • KTH Royal Institute of Technology (SE)
  • DELL (United States) (US)
  • Ben-Gurion University of the Negev (IL)
  • Donald & Barbara Zucker School of Medicine at Hofstra/Northwell (US)
  • Cohere (Canada) (US)
  • University of Pennsylvania (US)
  • Toyota Technological Institute at Chicago (US)
  • Northeastern University (US)
  • Case Western Reserve University (US)
  • University of Windsor (CA)
  • St. Jude Children's Research Hospital (US)
  • Rochester Institute of Technology (US)
  • Emory University (US)
  • Anthropic (United States) (US)
  • European Organization for Nuclear Research (CH)
  • University of California, Santa Barbara (US)
  • Hewlett Packard Enterprise (United States) (US)
  • North Carolina State University (US)
  • All India Institute of Medical Sciences (IN)
  • Tel Aviv University (IL)
  • Georgia Institute of Technology (US)
  • Northwestern University (US)
  • University of Arizona (US)
  • University of Lisbon (PT)
  • Indian Institute of Technology Kharagpur (IN)
  • Posts and Telecommunications Institute of Technology (VN)
  • Duke University (US)
  • Mila - Quebec Artificial Intelligence Institute (CA)
  • University College London (GB)
  • University of Zurich (CH)
  • University of Padua (IT)
  • Boston University (US)
  • Royal Veterinary College (GB)
  • Instituto Superior Técnico (PT)
  • The Ohio State University (US)
  • Cairo University (EG)
  • Universitat de València (ES)
  • Monash University (AU)
  • The University of Texas at Dallas (US)
  • Canadian University of Dubai (AE)
  • Hebrew University of Jerusalem (IL)
  • University of Milano-Bicocca (IT)
  • University of Massachusetts Lowell (US)
  • Virginia Tech (US)
  • University of Geneva (CH)
  • Laboratoire d'Informatique, de Robotique et de Microélectronique de Montpellier (FR)
  • Google (United States) (US)
  • California Polytechnic State University (US)
  • University of Mannheim (DE)
  • Stockholm University (SE)
  • Eastern Idaho Technical College (US)
  • University of Tennessee at Knoxville (US)
  • Johns Hopkins University (US)
  • University of Amsterdam (NL)
  • Fraunhofer-Einrichtung für Individualisierte Medizintechnik (DE)
  • University of Pisa (IT)
  • Georgia State University (US)
  • Polytechnic University of the Philippines (PH)
  • University of Oregon (US)
  • Drexel University (US)
  • University of Mumbai (IN)
  • Gakushuin University (JP)
  • University of Guelph (CA)
  • Intuit (United States) (US)
  • Centre Tecnologic de Telecomunicacions de Catalunya (ES)
  • Temple University (US)
  • École Normale Supérieure de Lyon (FR)
  • Cisco Systems (United States) (US)
  • Indian Institute of Technology BHU (IN)
  • Google DeepMind (United Kingdom) (GB)
  • The University of Texas at Arlington (US)
  • University of Vienna (AT)
  • Universitatea Națională de Știință și Tehnologie Politehnica București (RO)
  • Ollscoil na Gaillimhe – University of Galway (IE)
  • Eastern Institute of Technology (NZ)
  • Czech Technical University in Prague (CZ)
  • Universität Hamburg (DE)
  • Helmholtz Center for Information Security (DE)
  • Universidad de Morón (AR)
  • Université Paris Cité (FR)
  • Politecnico di Milano (IT)
  • The New School (US)
  • Universidad de Granada (ES)
  • Quinn
  • Heidelberg University (DE)
  • La Trobe University (AU)
  • Université de Yaoundé I (CM)
  • Universität Innsbruck (AT)
  • Tony
  • Chalmers University of Technology (SE)
  • Children's Hospital of Orange County (US)
  • Centro de Investigación Científica y de Educación Superior de Ensenada (MX)
  • University of Bradford (GB)
  • Beni-Suef University (EG)
  • Boğaziçi University (TR)
  • Mansoura University (EG)
  • Hospital for Sick Children (CA)
  • University of Bristol (GB)
  • University of Oklahoma (US)
  • University of Arkansas at Fayetteville (US)
  • Florida Atlantic University (US)
  • Bournemouth University (GB)
  • University of Warwick (GB)
  • University of Alabama in Huntsville (US)
  • University of Hertfordshire (GB)
  • OncoPrecision (United States) (US)
  • Central College (US)
  • Nottingham Trent University (GB)
  • Max Planck Institute for Intelligent Systems (DE)
  • University of Virginia (US)
  • Dartmouth College (US)
  • Instituto de Engenharia de Sistemas e Computadores Microsistemas e Nanotecnologias (PT)
  • James Madison University (US)
  • Instituto Gonçalo Moniz (BR)
  • Rice University (US)
  • Rutgers, The State University of New Jersey (US)
  • École Polytechnique (FR)
  • Quod
  • National Technological University (AR)
  • John Crane (United States) (US)
  • The Alan Turing Institute (GB)
  • Puducherry Technological University (IN)
  • Leibniz Institute for Science and Mathematics Education (DE)
  • Royal Holloway University of London (GB)
  • Tanta University (EG)
  • University of Malaya (MY)
  • Hemwati Nandan Bahuguna Garhwal University (IN)
  • Mohamed I University (MA)
  • Northern Illinois University (US)
  • Bethune-Cookman University (US)
  • University of California, Irvine (US)
  • Central Mindanao University (PH)
  • University of the Fraser Valley (CA)
  • Missouri University of Science and Technology (US)
  • CSMSS Chh. Shahu College of Engineering (IN)
  • Sheffield Teaching Hospitals NHS Foundation Trust (GB)
  • RMIT University (AU)
  • German Research Centre for Artificial Intelligence (DE)
  • University of Trento (IT)
  • Chulalongkorn University (TH)
  • Aligarh Muslim University (IN)
  • University of Southern California (US)
  • Menoufia University (EG)
  • Instituto Politécnico Nacional (MX)
  • University of Bologna (IT)
  • Manipal University Jaipur (IN)
  • OpenAI (United States) (US)
  • The University of Texas at Austin (US)
  • Murdoch University (AU)
  • University of Delaware (US)
  • Williams College (US)
  • University of Maribor (SI)
  • Fondazione Bruno Kessler (IT)
  • Brigham and Women's Hospital (US)
  • The University of Tokyo (JP)
  • Vellore Institute of Technology University (IN)
  • Centre Hospitalier Régional et Universitaire de Nancy (FR)
  • Delft University of Technology (NL)
  • Scripps Research Institute (US)
  • Atilim University (TR)
  • Leonardo (Italy) (IT)
  • Complexity Science Hub (AT)
  • Universidad Nacional de Educación a Distancia (ES)
  • Saxion (NL)
  • Korea Advanced Institute of Science and Technology (KR)
  • Adobe Systems (United States) (US)
  • National Aerospace University – Kharkiv Aviation Institute (UA)
  • University of Bern (CH)
  • University of North Carolina at Chapel Hill (US)
  • University of Groningen (NL)
  • Jagiellonian University (PL)
  • Universitat Politècnica de València (ES)
  • Royal Bank of Canada (CA)
  • Mayo Clinic (US)
  • Imperial College London (GB)
  • University of Lausanne (CH)
  • Dalhousie University (CA)
  • Universitat de Lleida (ES)
  • Amazon (United States) (US)
  • University of California, Santa Cruz (US)
  • University of Seoul (KR)
  • University of Auckland (NZ)
  • Korea University of Technology and Education (KR)
  • University of California, Davis (US)
  • Baylor College of Medicine (US)
  • Indraprastha Institute of Information Technology Delhi (IN)
  • New Jersey Institute of Technology (US)
  • Purdue University West Lafayette (US)
  • Utrecht University (NL)
  • University of Klagenfurt (AT)
  • Max Planck Institute for Security and Privacy (DE)
  • Goethe University Frankfurt (DE)
  • Universidad del Valle (CO)
  • Bilkent University (TR)
  • Universitat Pompeu Fabra (ES)
  • The University of Melbourne (AU)
  • Ankara University (TR)
  • AIT Austrian Institute of Technology GmbH (AT)
  • Providence College (US)
  • University of Jyväskylä (FI)
  • Weizmann Institute of Science (IL)
  • Indiana University Bloomington (US)
  • Nanyang Technological University (SG)
  • University of Sheffield (GB)
  • Centre Inria de l'Université de Lille (FR)
Corresponding author
  • Phan, Long
  • Hendrycks, Dan
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Volume
649
Issue
8099
Pages
1139-1146
ISSN
0028-0836 (ISSN-L)
OpenAlex
W4406840771
Cited by
65
References
14
Citations by year
2026: 51 · 2025: 14
FWCI
174.28
Type
journal article
Language
en
Open access
Locations
  • Nature (publishedVersion, cc-by-nc-nd)
  • Research Square (acceptedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • RiuNet (Universitat Politècnica de València) (publishedVersion, cc-by-nc-nd)
  • Repository for Publications and Research Data (ETH Zurich) (publishedVersion, cc-by-nc-nd)
  • SHURA (Sheffield Hallam University Research Archive) (Sheffield Hallam University) (acceptedVersion)
  • Open Research Online (The Open University) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • HAL (submittedVersion, other-oa)
  • CERN Document Server (European Organization for Nuclear Research) (submittedVersion, cc-by-nc-nd)
  • Jyväskylä University Digital Archive (University of Jyväskylä) (submittedVersion, cc-by-nc-nd)
  • Lirias (KU Leuven) (submittedVersion, cc-by-nc-nd)
  • Tuwhera (Auckland University of Technology) (submittedVersion, cc-by-nc-nd)
  • Open Repository and Bibliography (University of Luxembourg) (submittedVersion, other-oa)
  • Research Explorer (The University of Manchester) (submittedVersion)
  • VUBIR (Vrije Universiteit Brussel) (submittedVersion, cc-by)
  • KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
  • Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) (submittedVersion, cc-by)
  • Research Explorer (The University of Manchester) (submittedVersion, cc-by-nc-nd)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, other-oa)
  • Virtual Community of Pathological Anatomy (University of Castilla La Mancha) (submittedVersion)
  • Repositori digital de la UPF (Universitat Pompeu Fabra) (submittedVersion, cc-by-nc-nd)
  • VUBIR (Vrije Universiteit Brussel) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Office of Science (DE-AC02-06CH11357)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, benchmarking, automated grading, question difficulty
MeSH terms
Artificial Intelligence, Educational Measurement, Humans, Language, Benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Humanity's Last Exam. https://app.cochat.ai/papers/10.48550/arxiv.2501.14249
The paper
Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Zhang, C. B. C., Shaaban, M., Ling, J., Shi, S., Choi, M., Agrawal, A., Chopra, A., Khoja, A., Kim, R., Ren, R., Hausenloy, J., Zhang, O., Mazeika, M., . . . Hendrycks, D. (2025). Humanity's Last Exam. arXiv. https://doi.org/10.48550/arxiv.2501.14249

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.