The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Measuring the Occupational Impact of AI: Tasks, Cognitive Abilities and AI Benchmarks
Open accessAbstract
In this paper we develop a framework for analysing the impact of Artificial Intelligence (AI) on occupations. This framework maps 59 generic tasks from worker surveys and an occupational database to 14 cognitive abilities (that we extract from the cognitive science literature) and these to a comprehensive list of 328 AI benchmarks used to evaluate research intensity across a broad range of different AI areas. The use of cognitive abilities as an intermediate layer, instead of mapping work tasks to AI benchmarks directly, allows for an identification of potential AI exposure for tasks for which AI applications have not been explicitly created. An application of our framework to occupational databases gives insights into the abilities through which AI is most likely to affect jobs and allows for a ranking of occupations with respect to AI exposure. Moreover, we show that some jobs that were not known to be affected by previous waves of automation may now be subject to higher AI exposure. Finally, we find that some of the abilities where AI research is currently very intense are linked to tasks with comparatively limited labour input in the labour markets of advanced economies (e.g., visual and auditory processing using deep learning, and sensorimotor interaction through (deep) reinforcement learning). This article appears in the special track on AI and Society.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Third, occupational exposure is typically measured using a single LLM or expert-elicitation framework (Eloundou et al., 2024;Tolan et al., 2021;Acemoglu and Restrepo, 2022;Gmyrek et al., 2023), which embeds model-specific bias.”
From Generative AI impacts on intra-urban inequality and skill premium in Beijing (2026) · cites this paper as Tolan et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 3×
“How to compare between different benchmarks for the same task (e.g., COCO vs. MNIST (Bottou et al., 1994) vs. ImageNet (Deng et al., 2009)) or different tasks for the same benchmark?”
BackgroundMethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
The future of employment: How susceptible are jobs to computerisation?
Frey, C. B., & Osborne, M. A. (2017) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2016.08.019 · Cited by 8,727
The Skill Content of Recent Technological Change: An Empirical Exploration
Autor, D. H., Levy, F., & Murnane, R. J. (2003) · The Quarterly Journal of Economics · doi:10.1162/003355303322552801 · Cited by 6,205
An Experimental Application of the DELPHI Method to the Use of Experts
Dalkey, N., & Helmer, O. (1963) · Management Science · doi:10.1287/mnsc.9.3.458 · Cited by 6,061
Why Are There Still So Many Jobs? The History and Future of Workplace Automation
Autor, D. H. (2015) · Journal of Economic Perspectives · doi:10.1257/jep.29.3.3 · Cited by 3,917
The growth of low-skill service jobs and the polarization of the US labor market
Autor, D. H., & Dorn, D. (2013) · American Economic Review · doi:10.1257/aer.103.5.1553 · Cited by 3,824 · cited 1×
“Previous waves of technological progress have also had a sustained impact on labour markets (Autor and Dorn, 2013), yet the notion prevails that the impact of ML will be different (Brynjolfsson et al., 2018).”
BackgroundAcemoglu, D., & Restrepo, P. (2018) · American Economic Review · doi:10.1257/aer.20160696 · Cited by 3,121 · cited 3×
“For instance, replacing labour by AI would only be economically feasible if the effective costs of performing the task with machines are lower (Acemoglu and Restrepo, 2018).”
BackgroundKnow What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Skills, Tasks and Technologies: Implications for Employment and Earnings
Acemoglu, D., & Autor, D. (2010) · doi:10.3386/w16082 · Cited by 2,180
Explaining Job Polarization: Routine-Biased Technological Change and Offshoring
Goos, M., Manning, A., & Salomons, A. (2014) · American Economic Review · doi:10.1257/aer.104.8.2509 · Cited by 2,111 · cited 1×
“According to some studies, previous waves of technological progress led to a polarisation on the labour market where the automation of middle-skill occupations pushed middle-skill workers to either low- or high-skill occupations (Autor et al., 2003; Goos et al., 2014).”
ResultTechnical change, inequality, and the labor market
Acemoglu, D. (2002) · Journal of Economic Literature · doi:10.1257/jel.40.1.7 · Cited by 1,767 · cited 2×
“…a labour-enhancing one, it could imply a significant expansion of productivity for high-skilled occupations, potentially leading to occupational upgrading effects and an expansion of income inequality (very much like the traditional hypothesis of skills-biased technical change; see Acemoglu (2002)).”
BackgroundThe Growing Importance of Social Skills in the Labor Market*
Deming, D. J. (2017) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx022 · Cited by 1,727
What can machine learning do? Workforce implications
Brynjolfsson, E., & Mitchell, T. (2017) · Science · doi:10.1126/science.aap8062 · Cited by 1,131 · cited 2×
“For instance, some complementary conditions and restructuring of business processes may be necessary to enable the integration of AI in the workplace (Brynjolfsson et al., 2018; Brynjolfsson and Mitchell, 2017).”
BackgroundThe use of google trends in health care research: a systematic review
Nuti, S. V., Wayda, B., Ranasinghe, I., Wang, S., Dreyer, R. P., Chen, S. I., & Murugiah, K. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109583 · Cited by 1,086
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051
The Productivity J-Curve: How Intangibles Complement General Purpose Technologies
Brynjolfsson, E., Rock, D., & Syverson, C. (2021) · American Economic Journal: Macroeconomics · doi:10.1257/mac.20180386 · Cited by 1,003
The History of Technological Anxiety and the Future of Economic Growth: Is This Time Different?
Mokyr, J., Vickers, C., & Ziebarth, N. L. (2015) · Journal of Economic Perspectives · doi:10.1257/jep.29.3.31 · Cited by 698
What Can Machines Learn and What Does It Mean for Occupations and the Economy?
Brynjolfsson, E., Mitchell, T., & Rock, D. (2018) · AEA Papers and Proceedings · doi:10.1257/pandp.20181019 · Cited by 695
Future Progress in Artificial Intelligence: A Survey of Expert Opinion
Müller, V. C., & Bostrom, N. (2016) · Synthese Library · doi:10.1007/978-3-319-26485-1_33 · Cited by 655
Superhuman AI for multiplayer poker
Brown, N., & Sandholm, T. (2019) · Science · doi:10.1126/science.aay2400 · Cited by 592
Viewpoint: when will AI exceed human performance? evidence from AI experts
Grace, K., Salvatier, J., Dafoe, A., Zhang, B., & Evans, O. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11222 · Cited by 579
The Impact of Artificial Intelligence on the Labor Market
Webb, M. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3482150 · Cited by 531
Commonsense reasoning and commonsense knowledge in artificial intelligence
Davis, E., & Marcus, G. (2015) · Communications of the ACM · doi:10.1145/2701413 · Cited by 434
Automation and jobs: when technology boosts employment*
Bessen, J. (2019) · Economic Policy · doi:10.1093/epolic/eiaa001 · Cited by 409
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5699 · Cited by 359
Polanyi's Paradox and the Shape of Employment Growth
Autor, D. (2014) · doi:10.3386/w20485 · Cited by 323
When Will AI Exceed Human Performance? Evidence from AI Experts
Grace, K., Salvatier, J., Dafoe, A., Zhang, B., & Evans, O. (2017) · arXiv · doi:10.48550/arxiv.1705.08807 · Cited by 285
Mapping the Landscape of Human-Level Artificial General Intelligence
Adams, S. S., Arel, I., Bach, J., Coop, R., Furlan, R., Goertzel, B., Hall, J. S., Samsonovich, A., Scheutz, M., Schlesinger, M., et al. (2012) · AI Magazine · doi:10.1609/aimag.v33i1.2322 · Cited by 213
Scaling Laws for Autoregressive Generative Modeling
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.14701 · Cited by 150
Protecting against evaluation overfitting in empirical reinforcement learning
Whiteson, S., Tanner, B., Taylor, M. E., & Stone, P. (2011) · 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL) · doi:10.1109/adprl.2011.5967363 · Cited by 86 · cited 1×
“Furthermore, the specialisation of many metrics to the domain, the evaluation overfitting (Whiteson et al., 2011), and the lack of continuity in some evaluation procedures can also be recognised as limitations and constraints (Hernández-Orallo, 2017b) when evaluating the progress of AI.”
Background
Research works printed but not linked: 42
- Adams, S. S., G. Banavar, and M. Campbell (2016). I-athlon: Towards a multi-dimensional Turing test. AI Magazine 37(1), 78-84.
- Agrawal, A., J. Horton, N. Lacetera, and E. Lyons (2015). Digitization and the contract labor mar- ket: A research agenda. In Economic Analysis of the Digital Economy, pp. 219-250. University of Chicago Press.
- Archibugi, D. (1992). Patenting as an Indicator of Technological Innovation: A Review. Science and Public Policy 19(6), 357-368.
- Arntz, M., T. Gregory, and U. Zierahn (2016). The Risk of Automation for Jobs in OECD Countries: A Comparative Analysis. OECD Social, Employment and Migration Working Papers 2(189), 47- 54.
- Autor, D. H. and M. J. Handel (2013). Putting Tasks to the Test: Human Capital, Job Tasks, and Wages. Journal of Labor Economics 31(S1), S59-S96.
- Bogliacino, F., M. Piva, and M. Vivarelli (2012). R&D and Employment: An Application of the LSDVC Estimator Using European Microdata. Economics Letters 116(1), 56-59.
- Bottou, L., C. Cortes, J. S. Denker, H. Drucker, I. Guyon, L. D. Jackel, Y. LeCun, U. A. Müller, E. Säckinger, P. Y. Simard, et al. (1994). Comparison of Classifier Methods: A Case Study in Handwritten Digit Recognition. In International Conference on Pattern Recognition, pp. 77- 77. IEEE Computer Society Press.
- Brundage, M. (2016). Modeling Progress in AI. In AI, Ethics, and Society, Papers from the 2016 AAAI Workshop, Phoenix, Arizona, USA, February 13, 2016.
- Brynjolfsson, E., D. Rock, and C. Syverson (2018). Artificial Intelligence and the Modern Produc- tivity Paradox: A Clash of Expectations and Statistics. In The Economics of Artificial Intelli- gence: An Agenda. University of Chicago Press.
- Buchanan, B. G., J. Eckroth, and R. Smith (2013). A Virtual Archive for the History of AI. AI Magazine 34(2), 86-86.
- Calvino, F. and M. E. Virgillito (2018). The Innovation-Employment Nexus: A Critical Survey of Theory and Empirics. Journal of Economic Surveys 32(1), 83-117.
- Campbell, M., A. J. Hoane, and F. Hsu (2002). Deep Blue. Artificial Intelligence 134(1-2), 57 -83.
- Carroll, J. B. et al. (1993). Human Cognitive Abilities: A Survey of Factor-Analytic Studies. Cam- bridge University Press.
- Coad, A. and R. Rao (2011). The Firm-Level Employment Effects of Innovations in High-Tech US Manufacturing Industries. Journal of Evolutionary Economics 21(2), 255-283.
- Coxeter, H. S. M. (1961). Introduction to Geometry. John Wiley & Sons, New York, London.
- Dosi, G. (1988). Sources, Procedures, and Microeconomic Effects of Innovation. Journal of Eco- nomic Literature, 1120-1171.
- Felten, E. W., M. Raj, and R. Seamans (2018). A Method to Link Advances in Artificial Intelligence to Occupational Abilities. In AEA Papers and Proceedings, Volume 108, pp. 54-57.
- Fernández-Macías, E., E. Gómez, J. Hernández-Orallo, B. S. Loe, B. Martens, F. Martínez-Plumed, and S. Tolan (2018). A Multidisciplinary Task-Based Perspective for Evaluating the Impact of AI Autonomy and Generality on the Future of Work. arXiv preprint arXiv:1807.02416. · arXiv
- Ferrucci, D. A. (2012). Introduction to "This is Watson". IBM Journal of Research and Develop- ment 56(3.4), 1-1.
- Fiske, D. W. (1949). Consistency of the Factorial Structures of Personality Ratings from Different Sources. The Journal of Abnormal and Social Psychology 44(3), 329.
- Goos, M., A. Manning, and A. Salomons (2009). Job Polarization in Europe. American Economic Review 99(2), 58-63.
- Griliches, Z. (1998). Patent Statistics as Economic Indicators: A Survey. In R&D and Productivity: The Econometric Evidence, pp. 287-343. University of Chicago Press.
- Hendrycks, D., C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2020). Mea- suring Massive Multitask Language Understanding. arXiv preprint arXiv:2009.03300. · arXiv
- Hernández-Orallo, J. (2017a). Evaluation in Artificial Intelligence: From Task-Oriented to Ability- Oriented Measurement. Artificial Intelligence Review 48(3), 397-447.
- Hernández-Orallo, J. (2017b). The Measure of All Minds: Evaluating Natural and Artificial Intel- ligence. Cambridge University Press.
- Hernández-Orallo, J. and K. Vold (2019). AI Extenders: The Ethical and Societal Implications of Humans Cognitively Extended by AI. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 507-513.
- Jaffe, A. and G. de Rassenfosse (2017). Patent Citation Data in Social Science Research: Overview and Best Practices. Journal of the Association for Information Science and Technology 68(6), 1360-1374.
- Keith, T. Z. and M. R. Reynolds (2010). Cattell-Horn-Carroll Abilities and Cognitive Tests: What We've Learned from 20 Years of Research. Psychology in the Schools 47(7), 635-650.
- Lin, T.-Y., M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014). Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision, pp. 740-755. Springer.
- Manning, C. D., C. D. Manning, and H. Schütze (1999). Foundations of Statistical Natural Lan- guage Processing. MIT press.
- Martínez-Plumed, F., S. Avin, M. Brundage, A. Dafoe, S. Ó. hÉigeartaigh, and J. Hernández- Orallo (2018). Accounting for the Neglected Dimensions of AI Progress. arXiv preprint arXiv:1806.00610. · arXiv
- Martínez-Plumed, F., E. Gómez, and J. Hernández-Orallo (2020a). Tracking AI: The Capability is (Not) Near. In Proceedings of the Twenty-six European Conference on Artificial Intelligence. IOS Press.
- Martínez-Plumed, F., E. Gómez, and J. Hernández-Orallo (2020b). Tracking the Evolution of AI: The AIcollaboratory. In Proceedings of the 1st International Workshop: Evaluating Progress in Artificial Intelligence (EPAI 2020).
- Martinez-Plumed, F. and J. Hernandez-Orallo (2018). Dual Indicators to Analyse AI Benchmarks: Difficulty, Discrimination, Ability and Generality. IEEE Transactions on Games, 1-1.
- Martínez-Plumed, F., S. Tolan, A. Pesole, J. Hernández-Orallo, E. Fernández-Macías, and E. Gómez (2020). Does AI Qualify for the Job? A Bidirectional Model Mapping Labour and AI Intensities. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 94-100.
- Müller, V. C. and N. Bostrom (2014). Future Progress in Artificial Intelligence: A Poll Among Experts. AI Matters 1(1), 9-11.
- Polanyi, M. (1966). The Logic of Tacit Inference. Philosophy 41(155), 1-18.
- Purves, C., C. Cangea, and P. Veličković (2019). The PlayStation Reinforcement Learning Envi- ronment (PSXLE). arXiv preprint arXiv:1912.06101. · arXiv
- Saari, D. G. (2008). Complexity and the Geometry of Voting. Mathematical and Computer Mod- elling 48(9-10), 1335-1356.
- Schaie, K. W. (2010). Primary Mental Abilities. In I. B. Weiner and W. E. Craighead (Eds.), Corsini Encyclopedia of Psychology, pp. 1286-1288. Wiley.
- Vinyals, O. et al. (2017). Starcraft II: A New Challenge for Reinforcement Learning. arXiv preprint arXiv:1708.04782. · arXiv
- Wasserman, E. A. and T. R. Zentall (2006). Comparative Cognition: Experimental Explorations of Animal Intelligence. Oxford University Press.
Cited by
Zhai, C., Wibowo, S., & Li, L. D. (2024) · Smart Learning Environments · doi:10.1186/s40561-024-00316-7 · Cited by 1,567
Felten, E., Raj, M., & Seamans, R. (2021) · Strategic Management Journal · doi:10.1002/smj.3286 · Cited by 574
GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023) · arXiv · doi:10.48550/arxiv.2303.10130 · Cited by 2,330
“(Tolan et al., 2021) • Expert labeling of automation potential for a set of O*NET occupations where experts had high confidence, combined with a probabilistic classifier to estimate automation potential for the remainder of O*NET occupations.”
BackgroundSix Human-Centered Artificial Intelligence Grand Challenges
Ozmen Garibay, O., Winslow, B., Andolina, S., Antona, M., Bodenschatz, A., Coursaris, C., Falco, G., Fiore, S. M., Garibay, I., Grieman, K., et al. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2153320 · Cited by 501
Mannuru, N. R., Shahriar, S., Teel, Z. A., Wang, T., Lund, B. D., Tijani, S., Pohboon, C. O., Agbaji, D., Alhassan, J., Galley, J., et al. (2025) · Information Development · doi:10.1177/02666669231200628 · Cited by 314
Metadata
- Authors
- Tolan, S., Pesole, A., Martínez-Plumed, F., Fernández-Macías, E., Hernández-Orallo, J., & Gómez, E.
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2021-06-09
- Volume
- 71
- Pages
- 191-236
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W3082947733
- Cited by
- 165
- References
- 34
- FWCI
- 19.95
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.