AI in Software Engineering at Facebook
Open accessAbstract
How can artificial intelligence help software engineers better do their jobs and advance the state of the practice? We describe three productivity tools that learn patterns from software artifacts: code search using natural language, code recommendation, and automatic bug fixing.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior research has shown improvements in coding speed with AI tools.
“While some research has shown improvements in coding speed [12], developer throughput [13], and perceived productivity [14], more work must be done to validate these assertions across, for example, tasks, developer contexts, user groups, and more.”
From How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial (2025) · cites this paper as [12]
The Abstract describes using ML to help developers work more efficiently, and the Takeaways say these methods can remove day-to-day inefficiencies. Those passages support a general efficiency claim, but they do not report a demonstrated improvement in coding speed. (LLM, conf=0.82)
Experimental — accuracy not yet measuredCurrent studies have limited tool and task complexity in experimental settings.
“Other shortcomings include missing nuance or understanding of the impact of AI tools by developer- or task-level contexts (e.g., [13]), limited tool and task complexity in experimental settings (e.g., using GitHub Classroom rather than a more naturalistic developer environment [12]), and small sample sizes (e.g., only 32 participants in [15], 24 participants in [16], or 21 participants in [17]).”
From How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial (2025) · cites this paper as [12]
Peng et al.'s randomized controlled trial (n=95) found a 56% speed increase for developers using GitHub Copilot compared to those not using it.
“One much-discussed study was a randomized controlled trial by Peng et al. [12] ( n = 95 ), which found a 56% speed increase for developers using GitHub Copilot-an AI code assistant-, compared to those not using it.”
From How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial (2025) · cites this paper as [12]
The Takeaways and Developer Feedback sections discuss Facebook’s developer tools, but neither reports a GitHub Copilot trial. Searches for Peng, the 95-person randomized trial, and a 56% speed increase found no passage addressing the claim in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measuredThis estimate (21%) is significantly smaller than the 56% estimate shared by Peng and colleagues about GitHub Copilot [12], but aligns with the 26% productivity increase attributed to Copilot by Cui and colleagues [13].
“This estimate is significantly smaller than the 56% estimate shared by Peng and colleagues about GitHub Copilot [12], but aligns with the 26% productivity increase attributed to Copilot by Cui and colleagues [13].”
From How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial (2025) · cites this paper as [12]
The Takeaways section discusses Facebook’s ML tools and developer efficiency, but gives no 21% productivity estimate. Searches of our copy for GitHub Copilot, Peng, Cui, and the 56% and 26% estimates found no passage addressing the comparison. Page numbers were not provided. (LLM, conf=0.76)
Experimental — accuracy not yet measuredThe difference between our estimate and that published in [12] is likely attributable to differences in the underlying populations recruited from, with Peng et al. recruiting from Upwork and this study sampling full-time Googlers.
“More likely, however, the difference is caused by differences in the underlying populations we recruited from. Indeed, Peng et al. [12] recruited from Upwork (a freelancing platform), while we only sampled full-time Googlers.”
From How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial (2025) · cites this paper as [12]
The Developer Feedback and Evaluation sections discuss code search at Facebook, not recruitment from Upwork or sampling of full-time Googlers. Searches for Peng et al., Upwork, Googlers, and differences between study populations found no passage addressing the attributed claim. (LLM, conf=0.77)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228 · see citing works
Flagged first, then by citations
Research works printed but not linked: 16
- M. Machalica, A. Samylkin, M. Porth, and S. Chandra, "Predictive test selection," in Proc. 41st Int. Conf. Softw. Eng. Softw. Eng. Pract. (ICSE-SEIP 19), 2019, pp. 91-100. doi: 10.1109/ICSE-SEIP.2019.00018. · DOI
- R. Qian, Y. Yu, W. Park, V. Mu- rali, S. Fink, and S. Chandra, "Debugging crashes using continuous contrast set mining," in ICSE-SEIP '20: Proc. ACM/IEEE 42nd Int. Conf. Software Engineering: Soft- ware Engineering in Practice, June 2020, pp. 61-70. https://doi .org/10.1145/3377813.3381369. · DOI
- R. Sirres et al., "Augmenting and structuring user queries to support efficient free-form code search," in Proc. 40th Int. Conf. Softw. Eng. (ICSE'18), 2018, p. 945. doi: 10.1145/3180155.3182513. · DOI
- H. Sajnani, V. Saini, J. Svajlenko, C. K. Roy, and C. V. Lopes, "Sour- cererCC: Scaling code clone de- tection to bigcode," in Proc. 38th Int. Conf. Softw. Eng. (ICSE'16), 2016, pp. 1157-1168. doi: 10.1145/2884781.2884877. · DOI
- S. Sachdev, H. Li, S. Luan, S. Kim, K. Seohyun, and S. Chandra, "Retrieval on source code: A neural code search," in Proc. 2nd ACM SIGPLAN Int. Workshop on Mach. Learn. Program. Languages, 2018, pp. 31-41.
- P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, "Enriching word vectors with subword information," 2016, arXiv: 1607.04606. · arXiv
- T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, "Distributed representations of words and phrases and their compositionality," 2013, arXiv: 1310.4546. · arXiv
- H. Li, S. Kim, and S. Chandra, "Neu- ral code search evaluation dataset," 2019, arXiv: 1908.09804 [cs.SE]. · arXiv
- J. Cambronero, H. Li, S. Kim, K. Sen, and S. Chandra, "When deep learn- ing met code search," in Proc. 27th ACM Joint Meeting European Softw. Eng. Conf. Symp. Found. Softw. Eng. (ESEC/FSE 2019), 2019, pp. 964- 974. doi: 10.1145/3338906.3340458. · DOI
- S Luan, D. Yang, C. Barnaby, K. Sen, and S. Chandra, "Aroma: Code recommendation via structural code search," in Proc. ACM Program. Languages, Oct. 2019, vol. 3, no. OOPSLA, pp. 152:1-152:28. doi: 10.1145/3360578. · DOI
- C. Barnaby, K. Sen, T. Zhang, E. Glass- man, and S. Chandra, Exempla Gratis (E.G.): Code Examples for Free. New York: Association for Computing Ma- chinery, 2020, pp. 1353-1364.
- X. B. D. Le, D. Lo, and C. Le Goues. "History Driven Program Repair," in Proc. IEEE 23rd Int. Conf. Softw. Anal., Evolution, Reeng. (SANER), 2016, vol. 1, pp. 213-224.
- C. Calcagno et al., "Moving fast with software verification," in Proc. NASA Formal Method Symp., 2015.
- J.-R. Falleri, F. Morandat, X. Blanc, M. Martinez, and M. Monperrus, "Fine-grained and accurate source code differencing," in Proc. ACM/ IEEE Int. Conf. Automat. Softw. Eng. (ASE'14), 2014, pp. 313-324. doi: 10.1145/2642937.2642982. · DOI
- J. Bader, A. Scott, M. Pradel, and S. Chandra, "Getafix: Learning to fix bugs automatically," in Proc. ACM Program. Languages, Oct. 2019, vol. 3, no. OOPSLA. doi: 10.1145/3360585. · DOI
- X. Ren, F. Shah, F. Tip, B. G Ry- der, and O. C. Chesley, "Chianti: A tool for change impact analy- sis of Java programs," in Proc. 19th Annu. ACM SIGPLAN Conf. Object-Oriented Program- ming, Syst., Languages, Appl., Oct. 2004, pp. 432-448. doi: 10.1145/1035292.1029012. · DOI
Cited by
Sorted by citations
AI-driven innovations in software engineering: a review of current practices and future directions
Alenezi, M., & Akour, M. (2025) · Applied Sciences · doi:10.3390/app15031344 · Cited by 121 · see citing works
Identifying the Factors That Influence Trust in AI Code Completion
Brown, A., D'Angelo, S., Murillo, A., Jaspan, C., & Green, C. (2024) · Proceedings of the 1st ACM International Conference on AI-Powered Software · doi:10.1145/3664646.3664757 · Cited by 20 · see citing works
How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00060 · Cited by 20 · see citing works
“Furthermore, substantial effort has been invested in building tools for internal use, such as at Meta [12] and Google [13], [14], [15].”
Self-citation
- Authors
- Bader, J., Seohyun Kim, S., Sifei Luan, F., Chandra, S., & Meijer, E.
- Affiliations
- Meta (United States) (US)
- Facebook, Facebook Inc., Menlo Park, California 94025 United States
- Venue
- IEEE Software
- Published
- 2021-07
- Volume
- 38
- Issue
- 4
- Pages
- 52-61
- ISSN
- 0740-7459 (ISSN-L)
- 1937-4194
- OpenAlex
- W3132150767
- Cited by
- 16
- References
- 1
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 2 · 2023: 3 · 2022: 3 · 2021: 1
- FWCI
- 2.03
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE Software (publishedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- artificial intelligence in software development, code recommendation, Facebook
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
