Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its DataCite record; 45 of the 45 references it cites were found in a registry or an index; none of the 57 sources its claims cite is checked for support yet.

Working with AI: Measuring the Applicability of Generative AI to Occupations

Open access

Abstract

With generative AI emerging as a general-purpose technology, understanding its economic effects is among society's most pressing questions. Earlier work on AI impact largely relied on predictions of AI capabilities or focused narrowly on individual firms. Drawing instead on real-world AI usage, we analyze a dataset of 200k anonymized conversations with Microsoft Bing Copilot to measure AI applicability to occupations. We use an LLM-based pipeline to classify the O*NET work activities assisted or performed by AI in each conversation. We find that the most common and successful AI-assisted work activities involve information work--the creation, processing, and communication of information. At the occupation level, we find widespread AI applicability cutting across sectors, as most occupations have information work components. Our methodology also allows us to predict which occupations are more likely to delegate tasks to AI and which are more likely to use AI to assist existing workflows. Finally, we see an increase in occupation-level AI applicability over our 9-month data period, driven primarily by people using AI for more varied tasks. AI applicability scores and component usage and success metrics are available at https://github.com/microsoft/working-with-ai.

Overview

CoChat Summary

TL;DR

Analyzing 200k anonymized Microsoft Bing Copilot conversations from January–September 2024, the paper maps real-world generative AI usage to O*NET work activities and computes occupation-level AI applicability scores. It finds AI is most often and most successfully used for information work, with broad but uneven applicability across occupations and increasing task variety over the nine-month period.

Key findings

  • Information work dominates: the most common and successful AI-assisted activities involve creating, gathering, and disseminating information. Top occupations by AI applicability score include Interpreters and Translators (0.492), Historians (0.462), Writers and Authors (0.454), Sales Representatives of Services (0.449), and Customer Service Representatives (0.408); top SOC major groups are Computer & Mathematical (0.29), Sales & Related (0.29), and Office & Administrative Support (0.26).
  • AI applicability is widespread but weakly related to wage: most occupations have at least some AI applicability, and the employment-weighted correlation between AI applicability and occupational wage is r = 0.13. The score correlates with Eloundou et al.’s exposure predictions at employment-weighted r = 0.73, but the authors suggest prior work underestimated relevance to some information work occupations like proofreaders and software developers.
  • User goals and AI actions differ: in 40% of conversations the user-goal and AI-action IWA sets are disjoint, and in 96% there are more IWAs unique to one side than in common; for every work activity matched to a user goal, the AI performs an average of two work activities in service of the request. Media and financial operations occupations are relatively likelier to delegate tasks to AI, while physical occupations like food preparation are likelier to use AI assistively.
  • Over the 9-month period, the number of user goals covering 75% of activity increases from 20 to 28, and IWA distribution entropy increases by about 0.5 bits (≈40% increase in perplexity). Coverage is the primary contributor to AI applicability scores: a coverage-only ranking has Kendall tau 0.88 with the full score, adding scope raises it to 0.95, and completion provides the last 0.05.

Methods

  • Data: 200k anonymized, privacy-scrubbed U.S. Bing Copilot conversations from January 1 to September 30, 2024, split into COPILOT-UNIFORM (~100k uniform sample) and COPILOT-THUMBS (100k conversations with at least one thumbs up/down). The analysis uses O*NET 29.0, 2023 BLS OEWS data, and 2024 CPS data, covering 785 SOC codes and 149.8 million workers.
  • Classification pipeline: two-stage GPT-4o prompts summarize each conversation’s user goal and AI action in the style of O*NET Intermediate Work Activities (IWAs), generate four rewordings, rank all 332 IWAs by embedding cosine similarity, then binary-classify each IWA as matching the user goal and/or AI action. Validation against human annotations on 195 conversations found human-human κ ≈ 0.51 and human-LLM pipeline κ ≈ 0.40 on a held-out test set, with overall agreement well over 90%.
  • Metrics: activity share is computed with a 0.05% coverage threshold; success is measured via thumbs positive feedback share, an LLM completion classifier, and a six-point scope rating. These are aggregated into occupation-level AI applicability scores using O*NET importance weights, separately for user goals and AI actions, then averaged.

Limitations

  • Paper-stated: the data are from 2024 and models have improved since; the data represent only one AI platform (Bing Copilot), while other platforms differ (e.g., ChatGPT skews toward writing, Claude toward coding and mathematics); completion and scope measure AI utility toward an IWA, not productivity impact on workers; O*NET is U.S.-centric, may lag current workplace activities, and does not capture non-occupational work or the “connecting glue” between tasks.
  • Paper-stated: the 0.05% coverage threshold means absolute scores depend strongly on the threshold, so relative comparisons are more meaningful; thumbs feedback may suffer selection bias; IWA classification has moderate κ values due to unclear boundaries between IWAs; scope validation shows substantial variance and a larger human-human vs. human-LLM gap than IWA classification.
  • Evident from the text: the study is observational and usage-based, so it does not establish causal effects of AI on employment or wages; the authors explicitly caution that automation vs. augmentation conclusions are hard to predict. Reproducibility is limited because raw conversations cannot be released, though aggregated metrics are public; employment weighting materially affects wage/education correlations (unweighted r = 0.17 for user goals and 0.32 for AI actions vs. weighted r = 0.13).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2507.07935

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Frequency of work activities in Bing Copilot usage. ( A ) The 15 IWAs that occur most frequently as user goals. ( B ) The 15 IWAs that occur most frequently as AI actions. ( C ) The frequencies of GWAs as user goals and AI actions, compared to a rough estimate of GWA workforce frequency (described in Section B). All panels show researcher-generated IWA groupings.
Figure 1 Frequency of work activities in Bing Copilot usage. ( A ) The 15 IWAs that occur most frequently as user goals. ( B ) The 15 IWAs that occur most frequently as AI actions. ( C ) The frequencies of GWAs as user goals and AI actions, compared to a rough estimate of GWA workforce frequency (described in Section B). All panels show researcher-generated IWA groupings.CC-BY · From the PDF
Figure 2 Success of Bing Copilot work activities. The top and bottom 15 IWAs (with user goal or AI action activity share at least 0.05%) by the product of their completion rate and the share of conversations where Bing Copilot demonstrated at least a moderate scope of capability.
Figure 2 Success of Bing Copilot work activities. The top and bottom 15 IWAs (with user goal or AI action activity share at least 0.05%) by the product of their completion rate and the share of conversations where Bing Copilot demonstrated at least a moderate scope of capability.CC-BY · From the PDF
Figure 3 Top occupations by AI applicability score and their contributing IWAs. The 10 SOC occupations with the highest AI applicability scores (right) along with the 10 IWAs that provide the greatest contributions to those scores (left). Asterisks indicate 2023 U.S. occupational employment (none: up to 50k, one: 50k to 100k, two: 100k to 500k, three: more than 500k); the heights of each stratum indicate the same. Occupations' applicability scores are in parentheses, and occupations are sorted by their score in decreasing order. Portions of the occupational strata not connected to an IWA by a colored flow represent IWAs not present in the Figure that still contribute to the occupation's applicability score. Both occupation and IWA titles have been shortened for space.
Figure 3 Top occupations by AI applicability score and their contributing IWAs. The 10 SOC occupations with the highest AI applicability scores (right) along with the 10 IWAs that provide the greatest contributions to those scores (left). Asterisks indicate 2023 U.S. occupational employment (none: up to 50k, one: 50k to 100k, two: 100k to 500k, three: more than 500k); the heights of each stratum indicate the same. Occupations' applicability scores are in parentheses, and occupations are sorted by their score in decreasing order. Portions of the occupational strata not connected to an IWA by a colored flow represent IWAs not present in the Figure that still contribute to the occupation's applicability score. Both occupation and IWA titles have been shortened for space.CC-BY · From the PDF
Image 4
CC-BY · From the PDF
Figure 5 Monthly trends for occupation-level metrics. ( A ) Average AI applicability scores. ( B ) Average occupational coverage. ( C ) Average occupation-aggregated completion. ( D ) Average occupation-aggregated scope share. All four metrics are employment-weighted averages over occupations, with IWA-level metrics in B-D aggregated to occupations via weights w ij . The shaded regions show 95% confidence intervals for the weighted mean.
Figure 5 Monthly trends for occupation-level metrics. ( A ) Average AI applicability scores. ( B ) Average occupational coverage. ( C ) Average occupation-aggregated completion. ( D ) Average occupation-aggregated scope share. All four metrics are employment-weighted averages over occupations, with IWA-level metrics in B-D aggregated to occupations via weights w ij . The shaded regions show 95% confidence intervals for the weighted mean.CC-BY · From the PDF
Figure K.6: Distribution of IWAs per conversation. ( A ) The distribution of the number of IWAs to which a conversation is matched for user goal in both the uniform and thumbs Copilot datasets. ( B ) The same distributions for AI action IWAs.
Figure K.6: Distribution of IWAs per conversation. ( A ) The distribution of the number of IWAs to which a conversation is matched for user goal in both the uniform and thumbs Copilot datasets. ( B ) The same distributions for AI action IWAs.CC-BY · From the PDF
Figure K.7: IWA frequency in AI actions and user goals. For each IWA, this Figure plots the fraction of user intents ( x axis) and AI actions ( y axis) described by that IWA. Outliers show which IWAs are more likely to be performed (blue) or assisted (red) by Bing Copilot. When a conversation is labeled with multiple IWAs, that share of Copilot activity is evenly distributed among the IWAs so that the sum of all IWA fractions is 1. Table L.2 lists the relative frequencies for the most off-diagonal IWAs.
Figure K.7: IWA frequency in AI actions and user goals. For each IWA, this Figure plots the fraction of user intents ( x axis) and AI actions ( y axis) described by that IWA. Outliers show which IWAs are more likely to be performed (blue) or assisted (red) by Bing Copilot. When a conversation is labeled with multiple IWAs, that share of Copilot activity is evenly distributed among the IWAs so that the sum of all IWA fractions is 1. Table L.2 lists the relative frequencies for the most off-diagonal IWAs.CC-BY · From the PDF
Figure K.8: IWAs with the highest and lowest shares of positive feedback. This Figure shows the top and bottom 15 IWAs by the share of positive feedback, filtered to common IWAs matched in at least 1% of conversations in our feedback dataset, with bootstrapped 95% confidence intervals. The common IWAs with highest positive feedback share include two about writing and editing; four about evaluating or purchasing goods and services; and six about researching information about health, culture, law, policy, and society. Meanwhile, the common IWAs with the lowest positive feedback share include five visual design and five data analysis IWAs.
Figure K.8: IWAs with the highest and lowest shares of positive feedback. This Figure shows the top and bottom 15 IWAs by the share of positive feedback, filtered to common IWAs matched in at least 1% of conversations in our feedback dataset, with bootstrapped 95% confidence intervals. The common IWAs with highest positive feedback share include two about writing and editing; four about evaluating or purchasing goods and services; and six about researching information about health, culture, law, policy, and society. Meanwhile, the common IWAs with the lowest positive feedback share include five visual design and five data analysis IWAs.CC-BY · From the PDF
Figure K.9: Share of positive feedback by GWA. This Figure plots the positive feedback share for each GWA, aggregating common IWAs into their GWAs and with bootstrapped 95% confidence intervals; any IWA appearing in less than 1% of our feedback data is ignored. 14 GWAs have no common IWAs and are thus excluded from this plot, including those relating to operating vehicles, repairing equipment, and management tasks like hiring, negotiation, and guiding subordinates.
Figure K.9: Share of positive feedback by GWA. This Figure plots the positive feedback share for each GWA, aggregating common IWAs into their GWAs and with bootstrapped 95% confidence intervals; any IWA appearing in less than 1% of our feedback data is ignored. 14 GWAs have no common IWAs and are thus excluded from this plot, including those relating to operating vehicles, repairing equipment, and management tasks like hiring, negotiation, and guiding subordinates.CC-BY · From the PDF
Figure K.10: Relationship between thumbs feedback and task completion rate for each IWA. ( A ) Scatterplot of user goal IWA positive feedback share (measured in COPILOT-THUMBS) and completion rate (measured in COPILOT-UNIFORM). ( B ) The same scatterplot for AI action IWAs. There is a strong correlation between thumbs feedback rate and GPT-4o-mini task completion rate for each IWA, indicating that both are capturing real signal about AI success in assisting or performing an IWA. Point size proportional to square root of IWA match count in COPILOT-UNIFORM. Weighted correlations also weighted by match count in COPILOT-UNIFORM.
Figure K.10: Relationship between thumbs feedback and task completion rate for each IWA. ( A ) Scatterplot of user goal IWA positive feedback share (measured in COPILOT-THUMBS) and completion rate (measured in COPILOT-UNIFORM). ( B ) The same scatterplot for AI action IWAs. There is a strong correlation between thumbs feedback rate and GPT-4o-mini task completion rate for each IWA, indicating that both are capturing real signal about AI success in assisting or performing an IWA. Point size proportional to square root of IWA match count in COPILOT-UNIFORM. Weighted correlations also weighted by match count in COPILOT-UNIFORM.CC-BY · From the PDF
Figure K.11: Correlation between IWA completion rates in COPILOT-UNIFORM and COPILOTTHUMBS. ( A ) Scatterplot of user goal IWA completion rate in our two datasets. ( B ) The same scatterplot for AI action IWAs. IWA-level completion rates are consistent between COPILOTUNIFORM and COPILOT-THUMBS. Point size proportional to square root of IWA match count in COPILOT-UNIFORM. Weighted correlations also weighted by match count in COPILOT-UNIFORM.
Figure K.11: Correlation between IWA completion rates in COPILOT-UNIFORM and COPILOTTHUMBS. ( A ) Scatterplot of user goal IWA completion rate in our two datasets. ( B ) The same scatterplot for AI action IWAs. IWA-level completion rates are consistent between COPILOTUNIFORM and COPILOT-THUMBS. Point size proportional to square root of IWA match count in COPILOT-UNIFORM. Weighted correlations also weighted by match count in COPILOT-UNIFORM.CC-BY · From the PDF
Figure K.12: IWAs with the highest and lowest completion rates. This Figure shows the top and bottom 15 IWAs by completion rate, filtered to 'common' IWAs with activity share at least 0.1% in COPILOT-UNIFORM, with 95% confidence intervals (using the normal approximation to binomial confidence intervals). Task completion shows some of the same patterns as positive feedback fraction (Figure K.8), with visual design and data analysis on the low end and writing on the high end. One notable difference is that the highest completion rate IWAs include 7 about explaining, advising, or training.
Figure K.12: IWAs with the highest and lowest completion rates. This Figure shows the top and bottom 15 IWAs by completion rate, filtered to 'common' IWAs with activity share at least 0.1% in COPILOT-UNIFORM, with 95% confidence intervals (using the normal approximation to binomial confidence intervals). Task completion shows some of the same patterns as positive feedback fraction (Figure K.8), with visual design and data analysis on the low end and writing on the high end. One notable difference is that the highest completion rate IWAs include 7 about explaining, advising, or training.CC-BY · From the PDF
Figure K.13: IWAs with the highest and lowest fraction of conversations at moderate or higher scope. This Figure shows the top and bottom 15 IWAs by how often they are assigned scope of impact at least moderate, filtered to 'common' IWAs with activity share at least 0.1% in COPILOT-UNIFORM, with 95% confidence intervals (using the normal approximation to binomial confidence intervals). Some patterns mirror those of thumbs feedback and completion, with research and writing IWAs having high scope, and data analysis and visual design IWAs having low scope. AI performance consistently has lower impact scope than user assistance.
Figure K.13: IWAs with the highest and lowest fraction of conversations at moderate or higher scope. This Figure shows the top and bottom 15 IWAs by how often they are assigned scope of impact at least moderate, filtered to 'common' IWAs with activity share at least 0.1% in COPILOT-UNIFORM, with 95% confidence intervals (using the normal approximation to binomial confidence intervals). Some patterns mirror those of thumbs feedback and completion, with research and writing IWAs having high scope, and data analysis and visual design IWAs having low scope. AI performance consistently has lower impact scope than user assistance.CC-BY · From the PDF
Figure K.14: Relationship between scope and task completion rate for each IWA. ( A ) Scatterplot of user goal IWA moderate or higher scope share and completion rate. ( B ) The same scatter plot for AI action IWAs. The relationship between scope and completion is much weaker than between completion and thumbs feedback, as they are capturing different questions. Point size proportional to square root of IWA match count in COPILOT-UNIFORM. Weighted correlations also weighted by match count in COPILOT-UNIFORM.
Figure K.14: Relationship between scope and task completion rate for each IWA. ( A ) Scatterplot of user goal IWA moderate or higher scope share and completion rate. ( B ) The same scatter plot for AI action IWAs. The relationship between scope and completion is much weaker than between completion and thumbs feedback, as they are capturing different questions. Point size proportional to square root of IWA match count in COPILOT-UNIFORM. Weighted correlations also weighted by match count in COPILOT-UNIFORM.CC-BY · From the PDF
Figure K.15: Correlation between scope and activity share in COPILOT-UNIFORM for each IWA. ( A ) Scatterplot of user goal IWA activity share and the fraction of conversations in which the scope is classified moderate or higher. ( B ) The same scatterplot for AI action IWAs. On both sides, impact scope is a good predictor of what activities people seek AI assistance with (better than completion or satisfaction).
Figure K.15: Correlation between scope and activity share in COPILOT-UNIFORM for each IWA. ( A ) Scatterplot of user goal IWA activity share and the fraction of conversations in which the scope is classified moderate or higher. ( B ) The same scatterplot for AI action IWAs. On both sides, impact scope is a good predictor of what activities people seek AI assistance with (better than completion or satisfaction).CC-BY · From the PDF
Figure K.16: AI applicability score for major occupation groups. The employment-weighted mean, median, and inter-quartile range of the AI applicability score for each major occupation group. Highlighted occupational groups have a majority of workers in occupations classified as information work.
Figure K.16: AI applicability score for major occupation groups. The employment-weighted mean, median, and inter-quartile range of the AI applicability score for each major occupation group. Highlighted occupational groups have a majority of workers in occupations classified as information work.CC-BY · From the PDF
Figure K.17: Top 25 occupations by AI applicability score and their contributing IWAs. Similar to Figure 3, the 25 SOC occupations with the highest AI applicability scores (right) along with the 20 IWAs that provide the greatest contributions to those scores (left). Asterisks indicate 2023 U.S. occupational employment (none: up to 50k, one: 50k to 100k, two: 100k to 500k, three: more than 500k); the heights of each stratum indicate the same. Occupations' applicability scores are in parentheses, and occupations are sorted by their score in decreasing order. Portions of the occupational strata not connected to an IWA by a colored flow represent IWAs not present in the Figure that still contribute to the occupation's applicability score. Both occupation and IWA titles have been shortened for space.
Figure K.17: Top 25 occupations by AI applicability score and their contributing IWAs. Similar to Figure 3, the 25 SOC occupations with the highest AI applicability scores (right) along with the 20 IWAs that provide the greatest contributions to those scores (left). Asterisks indicate 2023 U.S. occupational employment (none: up to 50k, one: 50k to 100k, two: 100k to 500k, three: more than 500k); the heights of each stratum indicate the same. Occupations' applicability scores are in parentheses, and occupations are sorted by their score in decreasing order. Portions of the occupational strata not connected to an IWA by a colored flow represent IWAs not present in the Figure that still contribute to the occupation's applicability score. Both occupation and IWA titles have been shortened for space.CC-BY · From the PDF
Figure K.18: AI applicability scores by predicted exposure AI applicability score versus E1 exposure from Eloundou et al. [3], the fraction of an occupation's tasks for which human raters predicted LLMs could reduce completion time by 50%. Occupations are colored by their distance from the employment-weighted regression line: blue points have higher AI applicability scores than would be expected from their E1 exposure.
Figure K.18: AI applicability scores by predicted exposure AI applicability score versus E1 exposure from Eloundou et al. [3], the fraction of an occupation's tasks for which human raters predicted LLMs could reduce completion time by 50%. Occupations are colored by their distance from the employment-weighted regression line: blue points have higher AI applicability scores than would be expected from their E1 exposure.CC-BY · From the PDF
Figure K.19: AI applicability scores by average wage and educational requirement. (A) AI applicability score by average occupational wage. There is a point for each occupation and darker points for the employment-weighted average of each ventile. Binscatter conditional means and least squares fits are weighted by employment. (B) AI applicability score by modal education requirement reported to O*NET. Boxplot medians and quartiles are weighted by employment (e.g., they show the score of the median worker rather than that of the median occupation).
Figure K.19: AI applicability scores by average wage and educational requirement. (A) AI applicability score by average occupational wage. There is a point for each occupation and darker points for the employment-weighted average of each ventile. Binscatter conditional means and least squares fits are weighted by employment. (B) AI applicability score by modal education requirement reported to O*NET. Boxplot medians and quartiles are weighted by employment (e.g., they show the score of the median worker rather than that of the median occupation).CC-BY · From the PDF
Figure K.20: AI applicability score by wage, separating out user goals and AI actions. ( A ) Wage plot as in Figure K.19A, but showing user goal AI applicability score only. ( B ) AI action applicability score only. The correlation is higher on the AI side.
Figure K.20: AI applicability score by wage, separating out user goals and AI actions. ( A ) Wage plot as in Figure K.19A, but showing user goal AI applicability score only. ( B ) AI action applicability score only. The correlation is higher on the AI side.CC-BY · From the PDF
Figure K.21: AI applicability score by educational requirement, separating out user goals and AI actions. ( A ) Education plot from Figure K.19B, but showing only user goal AI applicability score. ( B ) Using AI action scores. As with wage, the relationship with education is stronger on the AI side.
Figure K.21: AI applicability score by educational requirement, separating out user goals and AI actions. ( A ) Education plot from Figure K.19B, but showing only user goal AI applicability score. ( B ) Using AI action scores. As with wage, the relationship with education is stronger on the AI side.CC-BY · From the PDF
Figure K.22: AI applicability score by wage, unweighted . This Figure is the same as Figure K.20 without weighting the binscatters by employment. ( A ) Scatterplot with unweighted binscatter of each occupation's AI applicability score calculated over user goals only against the occupation's average wage. ( B ) Using AI action AI applicability score. The relationship is much less noisy without employment weighting, likely because weighting by employment causes the binscatters to be influenced dramatically by a small number of high-employment occupations, increasing the variance due to noise in the coverage metric and in the mapping between occupations and IWAs. Excluding the top 10% of highest-paid workers strengthens the correlation.
Figure K.22: AI applicability score by wage, unweighted . This Figure is the same as Figure K.20 without weighting the binscatters by employment. ( A ) Scatterplot with unweighted binscatter of each occupation's AI applicability score calculated over user goals only against the occupation's average wage. ( B ) Using AI action AI applicability score. The relationship is much less noisy without employment weighting, likely because weighting by employment causes the binscatters to be influenced dramatically by a small number of high-employment occupations, increasing the variance due to noise in the coverage metric and in the mapping between occupations and IWAs. Excluding the top 10% of highest-paid workers strengthens the correlation.CC-BY · From the PDF
Figure K.23: AI applicability score by educational requirement, unweighted. This Figure is the same as Figure K.21 without weighting by employment. ( A ) User goal AI applicability scores over occupations with different educational requirements, with boxplots over occupations (not weighting by employment). ( B ) Using AI action scores. As in Figure K.22, this reduces the noise in the relationship.
Figure K.23: AI applicability score by educational requirement, unweighted. This Figure is the same as Figure K.21 without weighting by employment. ( A ) User goal AI applicability scores over occupations with different educational requirements, with boxplots over occupations (not weighting by employment). ( B ) Using AI action scores. As in Figure K.22, this reduces the noise in the relationship.CC-BY · From the PDF
Figure K.24: Correlations between AI applicability score and 2024 Current Population Survey occupational demographics, weighted by employment. ( A ) Correlation with percent women. ( B ) Correlation with percent white. ( C ) Correlation with percent black or African American. ( D ) Correlation with percent Asian. ( E ) Correlation with percent Hispanic or Latino. ( F ) Correlation with median age. For each panel, we drop occupation codes with missing demographic data in the 2024 CPS Annual Averages.
Figure K.24: Correlations between AI applicability score and 2024 Current Population Survey occupational demographics, weighted by employment. ( A ) Correlation with percent women. ( B ) Correlation with percent white. ( C ) Correlation with percent black or African American. ( D ) Correlation with percent Asian. ( E ) Correlation with percent Hispanic or Latino. ( F ) Correlation with median age. For each panel, we drop occupation codes with missing demographic data in the 2024 CPS Annual Averages.CC-BY · From the PDF
Figure K.25: Rationale for our chosen coverage threshold. (A) Effects of coverage threshold on number of occupations with user goal coverage 0 and 1. Our threshold of 0.05% approximately minimizes the number of occupations assigned user goal coverage 0 or 1. (B) The correlation between AI applicability scores defined using different coverage thresholds and the one we report with threshold 0.05%. Across thresholds spanning three orders of magnitude, we get strongly correlated rankings of occupations by AI applicability. Contrast this robustness of relative applicability score with the absolute measure in Figure K.26, which is highly sensitive to arbitrary choice of threshold.
Figure K.25: Rationale for our chosen coverage threshold. (A) Effects of coverage threshold on number of occupations with user goal coverage 0 and 1. Our threshold of 0.05% approximately minimizes the number of occupations assigned user goal coverage 0 or 1. (B) The correlation between AI applicability scores defined using different coverage thresholds and the one we report with threshold 0.05%. Across thresholds spanning three orders of magnitude, we get strongly correlated rankings of occupations by AI applicability. Contrast this robustness of relative applicability score with the absolute measure in Figure K.26, which is highly sensitive to arbitrary choice of threshold.CC-BY · From the PDF
Image 26
CC-BY · From the PDF
Figure K.27: Mean and standard deviation of occupation coverage by threshold. ( A ) The averages and standard deviations of occupation user goal coverage for different thresholds for the share of chat activity an IWA must have to 'be done' with the LLM. ( B ) The same means and standard deviations for coverage derived from AI action IWAs. We use a threshold of 0.0005, which results in 127 and 87 IWAs being considered covered (of 332) on the user goal and AI action sides, respectively.
Figure K.27: Mean and standard deviation of occupation coverage by threshold. ( A ) The averages and standard deviations of occupation user goal coverage for different thresholds for the share of chat activity an IWA must have to 'be done' with the LLM. ( B ) The same means and standard deviations for coverage derived from AI action IWAs. We use a threshold of 0.0005, which results in 127 and 87 IWAs being considered covered (of 332) on the user goal and AI action sides, respectively.CC-BY · From the PDF
Figure K.28: Distribution of occupation coverage. ( A ) The distribution of occupation user goal coverage scores (fraction of importance-weighted work in an occupation with user goal activity share at least 0.05% in COPILOT-UNIFORM). ( B ) The same distribution for AI action coverage.
Figure K.28: Distribution of occupation coverage. ( A ) The distribution of occupation user goal coverage scores (fraction of importance-weighted work in an occupation with user goal activity share at least 0.05% in COPILOT-UNIFORM). ( B ) The same distribution for AI action coverage.CC-BY · From the PDF
1 Figure K.29: IWAs contributing to user goal vs. AI action asymmetry. This tripartite graph shows six prominent occupation groups that are skewed toward user goal or AI action applicability (left), the three IWAs contributing the most to each group's skew (middle) ordered by the extent to which those IWAs skew more to the AI action side, and the GWAs those IWAs belong to (right). Blue indicates a larger skew toward AI action applicability, while red indicates a larger skew toward user goal applicability. White is neutral. To identify which IWAs contribute most to an occupation group's skew, we take the second component magnitudes in an IWA-level PCA (where we perform PCA on IWA-level scope × completion × coverage indicator), multiply them by IWA weights w ij for each occupation, and then take an employment-weighted average to get a score for each occupational-group-IWA pair. IWA-level scores are averaged to get GWA-level scores.
1 Figure K.29: IWAs contributing to user goal vs. AI action asymmetry. This tripartite graph shows six prominent occupation groups that are skewed toward user goal or AI action applicability (left), the three IWAs contributing the most to each group's skew (middle) ordered by the extent to which those IWAs skew more to the AI action side, and the GWAs those IWAs belong to (right). Blue indicates a larger skew toward AI action applicability, while red indicates a larger skew toward user goal applicability. White is neutral. To identify which IWAs contribute most to an occupation group's skew, we take the second component magnitudes in an IWA-level PCA (where we perform PCA on IWA-level scope × completion × coverage indicator), multiply them by IWA weights w ij for each occupation, and then take an employment-weighted average to get a score for each occupational-group-IWA pair. IWA-level scores are averaged to get GWA-level scores.CC-BY · From the PDF
Figure K.30: Monthly IWA breadth metrics. ( A ) Number of IWAs whose activity share crossed the 0.05% coverage threshold within each month. ( B ) Entropy of the IWA distribution in each month. Both metrics show an increasing variety in user goals and AI actions over the data period, indicating users performing more varied tasks over time.
Figure K.30: Monthly IWA breadth metrics. ( A ) Number of IWAs whose activity share crossed the 0.05% coverage threshold within each month. ( B ) Entropy of the IWA distribution in each month. Both metrics show an increasing variety in user goals and AI actions over the data period, indicating users performing more varied tasks over time.CC-BY · From the PDF
Figure K.31: Ratio of IWA activity share by rank between last and first three months. Users performed tail IWAs (ranks ∼ 20 -250 ) more often at the end of the data period than at the beginning, reinforcing the trends from Figure K.30.
Figure K.31: Ratio of IWA activity share by rank between last and first three months. Users performed tail IWAs (ranks ∼ 20 -250 ) more often at the end of the data period than at the beginning, reinforcing the trends from Figure K.30.CC-BY · From the PDF
Figure K.32: Monthly conversation-level metrics. ( A ) Fraction of conversations in each month labeled as task completed. ( B ) Fraction of IWA matches on each side labeled with scope moderate or higher. ( C ) Share of thumbs feedback that was positive by month.
Figure K.32: Monthly conversation-level metrics. ( A ) Fraction of conversations in each month labeled as task completed. ( B ) Fraction of IWA matches on each side labeled with scope moderate or higher. ( C ) Share of thumbs feedback that was positive by month.CC-BY · From the PDF

Verification

Claims the paper makes

0 Valid · 4 Not valid · 32 Not checked · 156 Own results and interpretations

73 more claims

Create a free account to see all 192 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 78
  • Probably not a substantial claim. 51

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 40 references

Research works printed but not linked: 5

  1. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/forum?id=VTF8yNQM66.
  2. Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. SWE-Lancer: Can frontier LLMs earn $1 million from real-world freelance software engineering? In Forty- second International Conference on Machine Learning, 2025. URL https://openreview.net/ forum?id=xZXhFg43EI.
  3. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. WildChat: 1M ChatGPT interaction logs in the wild. In International Conference on Learning Representa- tions, pages 34590-34605, 2024. URL https://proceedings.iclr.cc/paper_files/paper/ 2024/file/9421261e06f1a63a352b068f1ac90609-Paper-Conference.pdf.
  4. Peter Ferdinand Drucker. Landmarks of tomorrow: a report on the new "post-modern" world. Harper, New York, 1st edition, 1959.
  5. Scott Counts, Yan Chen, Jing Dong, Himanshu Sharma, Andrey Zaikin, Rui Hu, Alperen Kok, Gorkem Ozer Yilmaz, Siddharth Suri, Kiran Tomlinson, Sonia Jaffe, and Will Wang. AI in the enterprise: How people use M365 Copilot Chat, 2026. URL https://arxiv.org/abs/2605. 23958.

Cited by

See all 155

Metadata

Authors
Tomlinson, K., Jaffe, S., Wang, W., Counts, S., & Suri, S.
Affiliations
Microsoft (United States) (US)
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
OpenAlex
W4417063314
Cited by
152
References
40
Citations by year
2026: 127 · 2025: 25
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
Microsoft Research
Topics
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Labor market dynamics and wage inequality
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
generative AI, occupation, information work, AI-assisted workflows, large language models, economic impact of AI, general purpose technologies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-17, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-17 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Working with AI: Measuring the Applicability of Generative AI to Occupations. https://app.cochat.ai/papers/10.48550/arxiv.2507.07935
The paper
Tomlinson, K., Jaffe, S., Wang, W., Counts, S., & Suri, S. (2025). Working with AI: Measuring the Applicability of Generative AI to Occupations. arXiv. https://doi.org/10.48550/arxiv.2507.07935

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.