The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Measuring the Occupational Impact of AI: Tasks, Cognitive Abilities and AI Benchmarks
Open accessAbstract
In this paper we develop a framework for analysing the impact of Artificial Intelligence (AI) on occupations. This framework maps 59 generic tasks from worker surveys and an occupational database to 14 cognitive abilities (that we extract from the cognitive science literature) and these to a comprehensive list of 328 AI benchmarks used to evaluate research intensity across a broad range of different AI areas. The use of cognitive abilities as an intermediate layer, instead of mapping work tasks to AI benchmarks directly, allows for an identification of potential AI exposure for tasks for which AI applications have not been explicitly created. An application of our framework to occupational databases gives insights into the abilities through which AI is most likely to affect jobs and allows for a ranking of occupations with respect to AI exposure. Moreover, we show that some jobs that were not known to be affected by previous waves of automation may now be subject to higher AI exposure. Finally, we find that some of the abilities where AI research is currently very intense are linked to tasks with comparatively limited labour input in the labour markets of advanced economies (e.g., visual and auditory processing using deep learning, and sensorimotor interaction through (deep) reinforcement learning). This article appears in the special track on AI and Society.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Third, occupational exposure is typically measured using a single LLM or expert-elicitation framework (Eloundou et al., 2024;Tolan et al., 2021;Acemoglu and Restrepo, 2022;Gmyrek et al., 2023), which embeds model-specific bias.”
From Generative AI impacts on intra-urban inequality and skill premium in Beijing (2026) · cites this paper as Tolan et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 3×
“How to compare between different benchmarks for the same task (e.g., COCO vs. MNIST (Bottou et al., 1994) vs. ImageNet (Deng et al., 2009)) or different tasks for the same benchmark?”
BackgroundMethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
The future of employment: How susceptible are jobs to computerisation?
Frey, C. B., & Osborne, M. A. (2017) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2016.08.019 · Cited by 8,727
The Skill Content of Recent Technological Change: An Empirical Exploration
Autor, D. H., Levy, F., & Murnane, R. J. (2003) · The Quarterly Journal of Economics · doi:10.1162/003355303322552801 · Cited by 6,205
An Experimental Application of the DELPHI Method to the Use of Experts
Dalkey, N., & Helmer, O. (1963) · Management Science · doi:10.1287/mnsc.9.3.458 · Cited by 6,061
Why Are There Still So Many Jobs? The History and Future of Workplace Automation
Autor, D. H. (2015) · Journal of Economic Perspectives · doi:10.1257/jep.29.3.3 · Cited by 3,917
The growth of low-skill service jobs and the polarization of the US labor market
Autor, D. H., & Dorn, D. (2013) · American Economic Review · doi:10.1257/aer.103.5.1553 · Cited by 3,824 · cited 1×
“Previous waves of technological progress have also had a sustained impact on labour markets (Autor and Dorn, 2013), yet the notion prevails that the impact of ML will be different (Brynjolfsson et al., 2018).”
BackgroundAcemoglu, D., & Restrepo, P. (2018) · American Economic Review · doi:10.1257/aer.20160696 · Cited by 3,121 · cited 3×
“For instance, replacing labour by AI would only be economically feasible if the effective costs of performing the task with machines are lower (Acemoglu and Restrepo, 2018).”
Background
Research works printed but not linked: 42
- Adams, S. S., G. Banavar, and M. Campbell (2016). I-athlon: Towards a multi-dimensional Turing test. AI Magazine 37(1), 78-84.
- Agrawal, A., J. Horton, N. Lacetera, and E. Lyons (2015). Digitization and the contract labor mar- ket: A research agenda. In Economic Analysis of the Digital Economy, pp. 219-250. University of Chicago Press.
- Archibugi, D. (1992). Patenting as an Indicator of Technological Innovation: A Review. Science and Public Policy 19(6), 357-368.
- Arntz, M., T. Gregory, and U. Zierahn (2016). The Risk of Automation for Jobs in OECD Countries: A Comparative Analysis. OECD Social, Employment and Migration Working Papers 2(189), 47- 54.
- Autor, D. H. and M. J. Handel (2013). Putting Tasks to the Test: Human Capital, Job Tasks, and Wages. Journal of Labor Economics 31(S1), S59-S96.
- Bogliacino, F., M. Piva, and M. Vivarelli (2012). R&D and Employment: An Application of the LSDVC Estimator Using European Microdata. Economics Letters 116(1), 56-59.
- Bottou, L., C. Cortes, J. S. Denker, H. Drucker, I. Guyon, L. D. Jackel, Y. LeCun, U. A. Müller, E. Säckinger, P. Y. Simard, et al. (1994). Comparison of Classifier Methods: A Case Study in Handwritten Digit Recognition. In International Conference on Pattern Recognition, pp. 77- 77. IEEE Computer Society Press.
- Brundage, M. (2016). Modeling Progress in AI. In AI, Ethics, and Society, Papers from the 2016 AAAI Workshop, Phoenix, Arizona, USA, February 13, 2016.
- Brynjolfsson, E., D. Rock, and C. Syverson (2018). Artificial Intelligence and the Modern Produc- tivity Paradox: A Clash of Expectations and Statistics. In The Economics of Artificial Intelli- gence: An Agenda. University of Chicago Press.
- Buchanan, B. G., J. Eckroth, and R. Smith (2013). A Virtual Archive for the History of AI. AI Magazine 34(2), 86-86.
- Calvino, F. and M. E. Virgillito (2018). The Innovation-Employment Nexus: A Critical Survey of Theory and Empirics. Journal of Economic Surveys 32(1), 83-117.
- Campbell, M., A. J. Hoane, and F. Hsu (2002). Deep Blue. Artificial Intelligence 134(1-2), 57 -83.
- Carroll, J. B. et al. (1993). Human Cognitive Abilities: A Survey of Factor-Analytic Studies. Cam- bridge University Press.
- Coad, A. and R. Rao (2011). The Firm-Level Employment Effects of Innovations in High-Tech US Manufacturing Industries. Journal of Evolutionary Economics 21(2), 255-283.
- Coxeter, H. S. M. (1961). Introduction to Geometry. John Wiley & Sons, New York, London.
- Dosi, G. (1988). Sources, Procedures, and Microeconomic Effects of Innovation. Journal of Eco- nomic Literature, 1120-1171.
- Felten, E. W., M. Raj, and R. Seamans (2018). A Method to Link Advances in Artificial Intelligence to Occupational Abilities. In AEA Papers and Proceedings, Volume 108, pp. 54-57.
- Fernández-Macías, E., E. Gómez, J. Hernández-Orallo, B. S. Loe, B. Martens, F. Martínez-Plumed, and S. Tolan (2018). A Multidisciplinary Task-Based Perspective for Evaluating the Impact of AI Autonomy and Generality on the Future of Work. arXiv preprint arXiv:1807.02416. · arXiv
- Ferrucci, D. A. (2012). Introduction to "This is Watson". IBM Journal of Research and Develop- ment 56(3.4), 1-1.
- Fiske, D. W. (1949). Consistency of the Factorial Structures of Personality Ratings from Different Sources. The Journal of Abnormal and Social Psychology 44(3), 329.
- Goos, M., A. Manning, and A. Salomons (2009). Job Polarization in Europe. American Economic Review 99(2), 58-63.
- Griliches, Z. (1998). Patent Statistics as Economic Indicators: A Survey. In R&D and Productivity: The Econometric Evidence, pp. 287-343. University of Chicago Press.
- Hendrycks, D., C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2020). Mea- suring Massive Multitask Language Understanding. arXiv preprint arXiv:2009.03300. · arXiv
- Hernández-Orallo, J. (2017a). Evaluation in Artificial Intelligence: From Task-Oriented to Ability- Oriented Measurement. Artificial Intelligence Review 48(3), 397-447.
- Hernández-Orallo, J. (2017b). The Measure of All Minds: Evaluating Natural and Artificial Intel- ligence. Cambridge University Press.
- Hernández-Orallo, J. and K. Vold (2019). AI Extenders: The Ethical and Societal Implications of Humans Cognitively Extended by AI. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 507-513.
- Jaffe, A. and G. de Rassenfosse (2017). Patent Citation Data in Social Science Research: Overview and Best Practices. Journal of the Association for Information Science and Technology 68(6), 1360-1374.
- Keith, T. Z. and M. R. Reynolds (2010). Cattell-Horn-Carroll Abilities and Cognitive Tests: What We've Learned from 20 Years of Research. Psychology in the Schools 47(7), 635-650.
- Lin, T.-Y., M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014). Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision, pp. 740-755. Springer.
- Manning, C. D., C. D. Manning, and H. Schütze (1999). Foundations of Statistical Natural Lan- guage Processing. MIT press.
- Martínez-Plumed, F., S. Avin, M. Brundage, A. Dafoe, S. Ó. hÉigeartaigh, and J. Hernández- Orallo (2018). Accounting for the Neglected Dimensions of AI Progress. arXiv preprint arXiv:1806.00610. · arXiv
- Martínez-Plumed, F., E. Gómez, and J. Hernández-Orallo (2020a). Tracking AI: The Capability is (Not) Near. In Proceedings of the Twenty-six European Conference on Artificial Intelligence. IOS Press.
- Martínez-Plumed, F., E. Gómez, and J. Hernández-Orallo (2020b). Tracking the Evolution of AI: The AIcollaboratory. In Proceedings of the 1st International Workshop: Evaluating Progress in Artificial Intelligence (EPAI 2020).
- Martinez-Plumed, F. and J. Hernandez-Orallo (2018). Dual Indicators to Analyse AI Benchmarks: Difficulty, Discrimination, Ability and Generality. IEEE Transactions on Games, 1-1.
- Martínez-Plumed, F., S. Tolan, A. Pesole, J. Hernández-Orallo, E. Fernández-Macías, and E. Gómez (2020). Does AI Qualify for the Job? A Bidirectional Model Mapping Labour and AI Intensities. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 94-100.
- Müller, V. C. and N. Bostrom (2014). Future Progress in Artificial Intelligence: A Poll Among Experts. AI Matters 1(1), 9-11.
- Polanyi, M. (1966). The Logic of Tacit Inference. Philosophy 41(155), 1-18.
- Purves, C., C. Cangea, and P. Veličković (2019). The PlayStation Reinforcement Learning Envi- ronment (PSXLE). arXiv preprint arXiv:1912.06101. · arXiv
- Saari, D. G. (2008). Complexity and the Geometry of Voting. Mathematical and Computer Mod- elling 48(9-10), 1335-1356.
- Schaie, K. W. (2010). Primary Mental Abilities. In I. B. Weiner and W. E. Craighead (Eds.), Corsini Encyclopedia of Psychology, pp. 1286-1288. Wiley.
- Vinyals, O. et al. (2017). Starcraft II: A New Challenge for Reinforcement Learning. arXiv preprint arXiv:1708.04782. · arXiv
- Wasserman, E. A. and T. R. Zentall (2006). Comparative Cognition: Experimental Explorations of Animal Intelligence. Oxford University Press.
Cited by
Zhai, C., Wibowo, S., & Li, L. D. (2024) · Smart Learning Environments · doi:10.1186/s40561-024-00316-7 · Cited by 1,567
Felten, E., Raj, M., & Seamans, R. (2021) · Strategic Management Journal · doi:10.1002/smj.3286 · Cited by 574
GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023) · arXiv · doi:10.48550/arxiv.2303.10130 · Cited by 2,330
“(Tolan et al., 2021) • Expert labeling of automation potential for a set of O*NET occupations where experts had high confidence, combined with a probabilistic classifier to estimate automation potential for the remainder of O*NET occupations.”
BackgroundSix Human-Centered Artificial Intelligence Grand Challenges
Ozmen Garibay, O., Winslow, B., Andolina, S., Antona, M., Bodenschatz, A., Coursaris, C., Falco, G., Fiore, S. M., Garibay, I., Grieman, K., et al. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2153320 · Cited by 501
Mannuru, N. R., Shahriar, S., Teel, Z. A., Wang, T., Lund, B. D., Tijani, S., Pohboon, C. O., Agbaji, D., Alhassan, J., Galley, J., et al. (2025) · Information Development · doi:10.1177/02666669231200628 · Cited by 314
Metadata
- Authors
- Tolan, S., Pesole, A., Martínez-Plumed, F., Fernández-Macías, E., Hernández-Orallo, J., & Gómez, E.
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2021-06-09
- Volume
- 71
- Pages
- 191-236
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W3082947733
- Cited by
- 165
- References
- 34
- FWCI
- 19.95
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Measuring the Occupational Impact of AI: Tasks, Cognitive Abilities and AI Benchmarks. https://app.cochat.ai/papers/10.1613/jair.1.12647
- The paper
- Tolan, S., Pesole, A., Martínez-Plumed, F., Fernández-Macías, E., Hernández-Orallo, J., & Gómez, E. (2021). Measuring the Occupational Impact of AI: Tasks, Cognitive Abilities and AI Benchmarks. Journal of Artificial Intelligence Research. https://doi.org/10.1613/jair.1.12647
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.