CoChat
Cited by 165

The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.

Measuring the Occupational Impact of AI: Tasks, Cognitive Abilities and AI Benchmarks

Open access
Tolan, Songül, Pesole, Annarosa, Martínez-Plumed, Fernando, Fernández-Macías, Enrique, Hernández-Orallo, José, Gómez, Emilia · 2021 · Journal of Artificial Intelligence Research

Abstract

In this paper we develop a framework for analysing the impact of Artificial Intelligence (AI) on occupations. This framework maps 59 generic tasks from worker surveys and an occupational database to 14 cognitive abilities (that we extract from the cognitive science literature) and these to a comprehensive list of 328 AI benchmarks used to evaluate research intensity across a broad range of different AI areas. The use of cognitive abilities as an intermediate layer, instead of mapping work tasks to AI benchmarks directly, allows for an identification of potential AI exposure for tasks for which AI applications have not been explicitly created. An application of our framework to occupational databases gives insights into the abilities through which AI is most likely to affect jobs and allows for a ranking of occupations with respect to AI exposure. Moreover, we show that some jobs that were not known to be affected by previous waves of automation may now be subject to higher AI exposure. Finally, we find that some of the abilities where AI research is currently very intense are linked to tasks with comparatively limited labour input in the labour markets of advanced economies (e.g., visual and auditory processing using deep learning, and sensorimotor interaction through (deep) reinforcement learning). This article appears in the special track on AI and Society.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1613/jair.1.12647

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Illustrative example of the bidirectional and indirect mapping between job market and Artificial Intelligence (abilities described in Appendix A). The full list of tasks is presented in Table 6 of Appendix C. The full list of AI benchmarks is presented in Table 7 of Appendix D.The notation we use is t for the tasks, a for the abilities and b for the benchmarks. The arrows are represented by correspondence matrices U (task-ability correspondence) and R (ability-benchmark correspondence).
Figure 1 Illustrative example of the bidirectional and indirect mapping between job market and Artificial Intelligence (abilities described in Appendix A). The full list of tasks is presented in Table 6 of Appendix C. The full list of AI benchmarks is presented in Table 7 of Appendix D.The notation we use is t for the tasks, a for the abilities and b for the benchmarks. The arrows are represented by correspondence matrices U (task-ability correspondence) and R (ability-benchmark correspondence).CC-BY · From the PDF
Figure 2 Cattell-Horn-Carroll's three stratum model. The broad abilities are Crystallised Intelligence (Gc), Fluid Intelligence (Gf), Quantitative Reasoning (Gq), Reading and Writing Ability (Grw), Short-Term Memory (Gsm), Long-Term Storage and Retrieval (Glr), Visual Processing (Gv), Auditory Processing (Ga), Processing Speed (Gs) and Decision/Reaction Time/Speed (Gt).
Figure 2 Cattell-Horn-Carroll's three stratum model. The broad abilities are Crystallised Intelligence (Gc), Fluid Intelligence (Gf), Quantitative Reasoning (Gq), Reading and Writing Ability (Grw), Short-Term Memory (Gsm), Long-Term Storage and Retrieval (Glr), Visual Processing (Gv), Auditory Processing (Ga), Processing Speed (Gs) and Decision/Reaction Time/Speed (Gt).CC-BY · From the PDF
Figure 3 Progress (trends represented with dashed coloured lines) across different evaluation metrics for COCOobject recognition benchmark (Krizhevsky et al., 2009).
Figure 3 Progress (trends represented with dashed coloured lines) across different evaluation metrics for COCOobject recognition benchmark (Krizhevsky et al., 2009).CC-BY · From the PDF
Figure 4 Average rate of activity level or intensity (dark blue dashed line) and linear fit (red dashed line) for a set of illustrative AI benchmarks over the period 2008-2018.
Figure 4 Average rate of activity level or intensity (dark blue dashed line) and linear fit (red dashed line) for a set of illustrative AI benchmarks over the period 2008-2018.CC-BY · From the PDF
Figure 5 Distribution of ability-specific scores across occupations.
Figure 5 Distribution of ability-specific scores across occupations.CC-BY · From the PDF
Figure 6 Relevance per cognitive ability weighted by (average) rate intensity for different periods of years over the last decade (2008-2018).
Figure 6 Relevance per cognitive ability weighted by (average) rate intensity for different periods of years over the last decade (2008-2018).CC-BY · From the PDF
Figure 7 Ability-specific AI exposure scores for selected occupations. Left: grouped by people, ideas or things abilities. Right: detailed for the 14 abilities.
Figure 7 Ability-specific AI exposure scores for selected occupations. Left: grouped by people, ideas or things abilities. Right: detailed for the 14 abilities.CC-BY · From the PDF
Figure 8 Scatterplot and best fit line, AI exposure score (percentiles) against wage percentiles Source: Structure of Earnings Survey 2014
Figure 8 Scatterplot and best fit line, AI exposure score (percentiles) against wage percentiles Source: Structure of Earnings Survey 2014CC-BY · From the PDF
Figure 9 Ability-specific scores of cognitive abilities for selected occupations. Coloured markers represent the highlighted occupation. Coloured dashes represent intervals due to perturbation of ability-task assignments. Grey lines visualise the relevance scores for the other eight occupations.
Figure 9 Ability-specific scores of cognitive abilities for selected occupations. Coloured markers represent the highlighted occupation. Coloured dashes represent intervals due to perturbation of ability-task assignments. Grey lines visualise the relevance scores for the other eight occupations.CC-BY · From the PDF
Figure 10 Correlation plots with other AI exposure scores. Black line represents the 45 degree line ( ρ = 1).
Figure 10 Correlation plots with other AI exposure scores. Black line represents the 45 degree line ( ρ = 1).CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 42

  1. Adams, S. S., G. Banavar, and M. Campbell (2016). I-athlon: Towards a multi-dimensional Turing test. AI Magazine 37(1), 78-84.
  2. Agrawal, A., J. Horton, N. Lacetera, and E. Lyons (2015). Digitization and the contract labor mar- ket: A research agenda. In Economic Analysis of the Digital Economy, pp. 219-250. University of Chicago Press.
  3. Archibugi, D. (1992). Patenting as an Indicator of Technological Innovation: A Review. Science and Public Policy 19(6), 357-368.
  4. Arntz, M., T. Gregory, and U. Zierahn (2016). The Risk of Automation for Jobs in OECD Countries: A Comparative Analysis. OECD Social, Employment and Migration Working Papers 2(189), 47- 54.
  5. Autor, D. H. and M. J. Handel (2013). Putting Tasks to the Test: Human Capital, Job Tasks, and Wages. Journal of Labor Economics 31(S1), S59-S96.
  6. Bogliacino, F., M. Piva, and M. Vivarelli (2012). R&D and Employment: An Application of the LSDVC Estimator Using European Microdata. Economics Letters 116(1), 56-59.
  7. Bottou, L., C. Cortes, J. S. Denker, H. Drucker, I. Guyon, L. D. Jackel, Y. LeCun, U. A. Müller, E. Säckinger, P. Y. Simard, et al. (1994). Comparison of Classifier Methods: A Case Study in Handwritten Digit Recognition. In International Conference on Pattern Recognition, pp. 77- 77. IEEE Computer Society Press.
  8. Brundage, M. (2016). Modeling Progress in AI. In AI, Ethics, and Society, Papers from the 2016 AAAI Workshop, Phoenix, Arizona, USA, February 13, 2016.
  9. Brynjolfsson, E., D. Rock, and C. Syverson (2018). Artificial Intelligence and the Modern Produc- tivity Paradox: A Clash of Expectations and Statistics. In The Economics of Artificial Intelli- gence: An Agenda. University of Chicago Press.
  10. Buchanan, B. G., J. Eckroth, and R. Smith (2013). A Virtual Archive for the History of AI. AI Magazine 34(2), 86-86.
  11. Calvino, F. and M. E. Virgillito (2018). The Innovation-Employment Nexus: A Critical Survey of Theory and Empirics. Journal of Economic Surveys 32(1), 83-117.
  12. Campbell, M., A. J. Hoane, and F. Hsu (2002). Deep Blue. Artificial Intelligence 134(1-2), 57 -83.
  13. Carroll, J. B. et al. (1993). Human Cognitive Abilities: A Survey of Factor-Analytic Studies. Cam- bridge University Press.
  14. Coad, A. and R. Rao (2011). The Firm-Level Employment Effects of Innovations in High-Tech US Manufacturing Industries. Journal of Evolutionary Economics 21(2), 255-283.
  15. Coxeter, H. S. M. (1961). Introduction to Geometry. John Wiley & Sons, New York, London.
  16. Dosi, G. (1988). Sources, Procedures, and Microeconomic Effects of Innovation. Journal of Eco- nomic Literature, 1120-1171.
  17. Felten, E. W., M. Raj, and R. Seamans (2018). A Method to Link Advances in Artificial Intelligence to Occupational Abilities. In AEA Papers and Proceedings, Volume 108, pp. 54-57.
  18. Fernández-Macías, E., E. Gómez, J. Hernández-Orallo, B. S. Loe, B. Martens, F. Martínez-Plumed, and S. Tolan (2018). A Multidisciplinary Task-Based Perspective for Evaluating the Impact of AI Autonomy and Generality on the Future of Work. arXiv preprint arXiv:1807.02416. · arXiv
  19. Ferrucci, D. A. (2012). Introduction to "This is Watson". IBM Journal of Research and Develop- ment 56(3.4), 1-1.
  20. Fiske, D. W. (1949). Consistency of the Factorial Structures of Personality Ratings from Different Sources. The Journal of Abnormal and Social Psychology 44(3), 329.
  21. Goos, M., A. Manning, and A. Salomons (2009). Job Polarization in Europe. American Economic Review 99(2), 58-63.
  22. Griliches, Z. (1998). Patent Statistics as Economic Indicators: A Survey. In R&D and Productivity: The Econometric Evidence, pp. 287-343. University of Chicago Press.
  23. Hendrycks, D., C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2020). Mea- suring Massive Multitask Language Understanding. arXiv preprint arXiv:2009.03300. · arXiv
  24. Hernández-Orallo, J. (2017a). Evaluation in Artificial Intelligence: From Task-Oriented to Ability- Oriented Measurement. Artificial Intelligence Review 48(3), 397-447.
  25. Hernández-Orallo, J. (2017b). The Measure of All Minds: Evaluating Natural and Artificial Intel- ligence. Cambridge University Press.
  26. Hernández-Orallo, J. and K. Vold (2019). AI Extenders: The Ethical and Societal Implications of Humans Cognitively Extended by AI. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 507-513.
  27. Jaffe, A. and G. de Rassenfosse (2017). Patent Citation Data in Social Science Research: Overview and Best Practices. Journal of the Association for Information Science and Technology 68(6), 1360-1374.
  28. Keith, T. Z. and M. R. Reynolds (2010). Cattell-Horn-Carroll Abilities and Cognitive Tests: What We've Learned from 20 Years of Research. Psychology in the Schools 47(7), 635-650.
  29. Lin, T.-Y., M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014). Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision, pp. 740-755. Springer.
  30. Manning, C. D., C. D. Manning, and H. Schütze (1999). Foundations of Statistical Natural Lan- guage Processing. MIT press.
  31. Martínez-Plumed, F., S. Avin, M. Brundage, A. Dafoe, S. Ó. hÉigeartaigh, and J. Hernández- Orallo (2018). Accounting for the Neglected Dimensions of AI Progress. arXiv preprint arXiv:1806.00610. · arXiv
  32. Martínez-Plumed, F., E. Gómez, and J. Hernández-Orallo (2020a). Tracking AI: The Capability is (Not) Near. In Proceedings of the Twenty-six European Conference on Artificial Intelligence. IOS Press.
  33. Martínez-Plumed, F., E. Gómez, and J. Hernández-Orallo (2020b). Tracking the Evolution of AI: The AIcollaboratory. In Proceedings of the 1st International Workshop: Evaluating Progress in Artificial Intelligence (EPAI 2020).
  34. Martinez-Plumed, F. and J. Hernandez-Orallo (2018). Dual Indicators to Analyse AI Benchmarks: Difficulty, Discrimination, Ability and Generality. IEEE Transactions on Games, 1-1.
  35. Martínez-Plumed, F., S. Tolan, A. Pesole, J. Hernández-Orallo, E. Fernández-Macías, and E. Gómez (2020). Does AI Qualify for the Job? A Bidirectional Model Mapping Labour and AI Intensities. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 94-100.
  36. Müller, V. C. and N. Bostrom (2014). Future Progress in Artificial Intelligence: A Poll Among Experts. AI Matters 1(1), 9-11.
  37. Polanyi, M. (1966). The Logic of Tacit Inference. Philosophy 41(155), 1-18.
  38. Purves, C., C. Cangea, and P. Veličković (2019). The PlayStation Reinforcement Learning Envi- ronment (PSXLE). arXiv preprint arXiv:1912.06101. · arXiv
  39. Saari, D. G. (2008). Complexity and the Geometry of Voting. Mathematical and Computer Mod- elling 48(9-10), 1335-1356.
  40. Schaie, K. W. (2010). Primary Mental Abilities. In I. B. Weiner and W. E. Craighead (Eds.), Corsini Encyclopedia of Psychology, pp. 1286-1288. Wiley.
  41. Vinyals, O. et al. (2017). Starcraft II: A New Challenge for Reinforcement Learning. arXiv preprint arXiv:1708.04782. · arXiv
  42. Wasserman, E. A. and T. R. Zentall (2006). Comparative Cognition: Experimental Explorations of Animal Intelligence. Oxford University Press.

Cited by

See all 165

Metadata

Authors
Tolan, S., Pesole, A., Martínez-Plumed, F., Fernández-Macías, E., Hernández-Orallo, J., & Gómez, E.
Venue
Journal of Artificial Intelligence Research
Published
2021-06-09
Volume
71
Pages
191-236
ISSN
1076-9757 (ISSN-L)
OpenAlex
W3082947733
Cited by
165
References
34
FWCI
19.95
Type
journal article
Language
en
Open access
diamond

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.