CoChat
Sign Up
PreprintCited by 70Software Engineering ResearchSoftware Testing and Debugging TechniquesSoftware Engineering Techniques and Practices

The DOI matches its DataCite record.

Bugs in Large Language Models Generated Code: An Empirical Study

Open access
Tambon, Florian, Dakhel, Arghavan Moradi, Nikanjam, Amin, Khomh, Foutse, Desmarais, Michel C., Antoniol, Giuliano·2024·arXiv
DOI 10.48550/arxiv.2403.08937arXiv 2403.08937
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) for code have gained significant attention recently. They can generate code in different programming languages based on provided prompts, fulfilling a long-lasting dream in Software Engineering (SE), i.e., automatic code generation. Similar to human-written code, LLM-generated code is prone to bugs, and these bugs have not yet been thoroughly examined by the community. Given the increasing adoption of LLM-based code generation tools (e.g., GitHub Copilot) in SE activities, it is critical to understand the characteristics of bugs contained in code generated by LLMs. This paper examines a sample of 333 bugs collected from code generated using three leading LLMs (i.e., CodeGen, PanGu-Coder, and Codex) and identifies the following 10 distinctive bug patterns: Misinterpretations, Syntax Error, Silly Mistake, Prompt-biased code, Missing Corner Case, Wrong Input Type, Hallucinated Object, Wrong Attribute, Incomplete Generation, and Non-Prompted Consideration. The bug patterns are presented in the form of a taxonomy. The identified bug patterns are validated using an online survey with 34 LLM practitioners and researchers. The surveyed participants generally asserted the significance and prevalence of the bug patterns. Researchers and practitioners can leverage these findings to develop effective quality assurance techniques for LLM-generated code. This study sheds light on the distinctive characteristics of LLM-generated code.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.08937

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Tambon et al. (2024) identified 10 unique bug patterns among 333 collected errors, confirmed by 34 LLM practitioners and researchers.

    “The research identifies 10 unique bug patterns among 333 collected errors, and these patterns are confirmed by 34 LLM practitioners and researchers.”

    From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Tambon et al. (2024)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 59

  1. D. Wong, A. Kothig, and P. Lam, "Exploring the verifiability of code generated by github copilot," arXiv preprint arXiv:2209.01766, 2022. · arXiv
  2. S. Imai, "Is github copilot a substitute for human pair-programming? an empirical study," in Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, 2022, pp. 319-321.
  3. F. A. Sakib, S. H. Khan, and A. Karim, "Extending the frontier of chatgpt: Code generation and debugging," arXiv preprint arXiv:2307.08260, 2023. · arXiv
  4. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03374, 2021. · arXiv
  5. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288, 2023. · arXiv
  6. H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, T. Xie, and Q. Wang, "Codereval: A benchmark of pragmatic code generation with generative pre-trained models," arXiv preprint arXiv:2302.00288v1, 2023. · arXiv
  7. M. Jin, S. Shahriar, M. Tufano, X. Shi, S. Lu, N. Sundaresan, and A. Svyatkovskiy, "Inferfix: End-to-end program repair with llms," in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 1646-1656. [Online]. Available: https://doi.org/10.1145/3611643.3613892 · DOI · Paper page
  8. J. Liu, C. S. Xia, Y. Wang, and L. Zhang, "Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation," arXiv preprint arXiv:2305.01210, 2023. · arXiv
  9. A. Moradi Dakhel, V. Majdinasab, A. Nikanjam, F. Khomh, M. C. Desmarais, and Z. M. J. Jiang, "Github copilot ai pair programmer: Asset or liability?" Journal of Systems and Software, vol. 203, p. 111734, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/ pii/S0164121223001292
  10. C. Bird, D. Ford, T. Zimmermann, N. Forsgren, E. Kalliamvakou, T. Low- dermilk, and I. Gazit, "Taking flight with copilot: Early insights and op- portunities of ai-powered pair-programming tools," Queue, vol. 20, no. 6, pp. 35-57, 2022.
  11. O. Asare, M. Nagappan, and N. Asokan, "Is github's copilot as bad as humans at introducing vulnerabilities in code?" Empirical Softw. Engg., vol. 28, no. 6, sep 2023. [Online]. Available: https: //doi.org/10.1007/s10664-023-10380-1 · DOI · Paper page
  12. J. Li, Y. Zhang, Z. Karas, C. McMillan, K. Leach, and Y. Huang, "Do machines and humans focus on similar code? exploring explain- ability of large language models in code summarization," arXiv preprint arXiv:2402.14182, 2024. · arXiv
  13. R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser, "Are mutants a valid substitute for real faults in software testing?" in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, ser. FSE 2014. New York, NY, USA: Association for Computing Machinery, 2014, p. 654-665. [Online]. Available: https://doi.org/10.1145/2635868.2635929 · DOI
  14. Z. Fan, X. Gao, M. Mirchev, A. Roychoudhury, and S. H. Tan, "Automated repair of programs from large language models," in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1469-1481.
  15. R. Pan, A. R. Ibrahimzada, R. Krishna, D. Sankar, L. P. Wassi, M. Merler, B. Sobolev, R. Pavuluri, S. Sinha, and R. Jabbarvand, "Lost in translation: A study of bugs introduced by large language models while translating code," arXiv preprint arXiv:2308.03109v3, 2023. · arXiv
  16. N. Nguyen and S. Nadi, "An empirical evaluation of github copilot's code suggestions," in Proceedings of the 19th International Conference on Min- ing Software Repositories, 2022, pp. 1-5.
  17. C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, "Codamosa: Escap- ing coverage plateaus in test generation with pre-trained large language models," in Accepted by 45th International Conference on Software Engi- neering (ICSE), 2023.
  18. X. Chen, M. Lin, N. Schärli, and D. Zhou, "Teaching large language models to self-debug," arXiv preprint arXiv:2304.05128, 2023. · arXiv
  19. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
  20. F. Christopoulou, G. Lampouras, M. Gritta, G. Zhang, Y. Guo, Z. Li, Q. Zhang, M. Xiao, B. Shen, L. Li et al., "Pangu-coder: Program synthesis with function-level language modeling," arXiv preprint arXiv:2207.11280, 2022. · arXiv
  21. W. Zeng, X. Ren, T. Su, H. Wang, Y. Liao, Z. Wang, X. Jiang, Z. Yang, K. Wang, X. Zhang et al., "Pangu-α: Large-scale autoregressive pretrained chinese language models with auto-parallel computation," arXiv preprint arXiv:2104.12369, 2021. · arXiv
  22. D. Zan, B. Chen, F. Zhang, D. Lu, B. Wu, B. Guan, Y. Wang, and J.- G. Lou, "When neural model meets nl2code: A survey," arXiv preprint arXiv:2212.09420, 2022. · arXiv
  23. E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, "Codegen: An open large language model for code with multi-turn program synthesis," arXiv preprint arXiv:2203.13474, 2022. · arXiv
  24. B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, "Codet: Code generation with generated tests," arXiv preprint arXiv:2207.10397, 2022. · arXiv
  25. C. B. Seaman, "Qualitative methods in empirical studies of software en- gineering," IEEE Transactions on software engineering, vol. 25, no. 4, pp. 557-572, 1999.
  26. N. Humbatova, G. Jahangirova, G. Bavota, V. Riccio, A. Stocco, and P. Tonella, "Taxonomy of real faults in deep learning systems," in Pro- ceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 2020, pp. 1110-1121.
  27. M. X. Liu, A. Sarkar, C. Negreanu, B. Zorn, J. Williams, N. Toronto, and A. D. Gordon, ""what it wants me to say": Bridging the abstraction gap between end-user programmers and code-generating large language models," in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023, pp. 1-31.
  28. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Ka- plan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, "Evaluating large language models trained on code," 2021.
  29. F. Tambon, A. Nikanjam, L. An, F. Khomh, and G. Antoniol, "Silent bugs in deep learning frameworks: an empirical study of keras and tensorflow," Empirical Software Engineering, vol. 29, no. 1, p. 10, 2024.
  30. Y. Fu, P. Liang, A. Tahir, Z. Li, M. Shahin, and J. Yu, "Secu- rity weaknesses of copilot generated code in github," arXiv preprint arXiv:2310.02059, 2023. · arXiv
  31. D. Spadini, M. Aniche, and A. Bacchelli, "Pydriller: Python framework for mining software repositories," in Proceedings of the 2018 26th ACM Joint meeting on european software engineering conference and symposium on the foundations of software engineering, 2018, pp. 908-911.
  32. A. Nikanjam, M. M. Morovati, F. Khomh, and H. Ben Braiek, "Faults in deep reinforcement learning programs: a taxonomy and a detection approach," Automated Software Engineering, vol. 29, 2021.
  33. V. Nardone, B. Muse, M. Abidi, F. Khomh, and M. Di Penta, "Video game bad smells: What they are and how developers perceive them," ACM Trans. Softw. Eng. Methodol., vol. 32, no. 4, may 2023. [Online]. Available: https://doi.org/10.1145/3563214 · DOI
  34. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al., "Palm: Scal- ing language modeling with pathways," Journal of Machine Learning Re- search, vol. 24, no. 240, pp. 1-113, 2023.
  35. M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, "Adaptive test generation using a large language model," arXiv preprint arXiv:2302.06527, 2023. · arXiv
  36. J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou, "Chain-of-thought prompting elicits reasoning in large language models," in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 24 824-24 837. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf
  37. J. Li, G. Li, Y. Li, and Z. Jin, "Structured chain-of-thought prompting for code generation," arXiv preprint arXiv:2305.06599, 2023. · arXiv
  38. V. Guilherme and A. Vincenzi, "An initial investigation of chatgpt unit test generation capability," in Proceedings of the 8th Brazilian Symposium on Systematic and Automated Software Testing, ser. SAST '23. New York, NY, USA: Association for Computing Machinery, 2023, p. 15-24. [Online]. Available: https://doi.org/10.1145/3624032.3624035 · DOI
  39. G. Fraser and A. Arcuri, "A large-scale evaluation of automated unit test generation using evosuite," ACM Trans. Softw. Eng. Methodol., vol. 24, no. 2, dec 2014. [Online]. Available: https://doi.org/10.1145/2685612 · DOI
  40. G. L. Scoccia, "Exploring early adopters' perceptions of chatgpt as a code generation tool," in 2023 38th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW), 2023, pp. 88-93.
  41. M. Kazemitabaar, X. Hou, A. Henley, B. J. Ericson, D. Weintrop, and T. Grossman, "How novices use llm-based code generators to solve cs1 coding tasks in a self-paced learning environment," arXiv preprint arXiv:2309.14049, 2023. · arXiv
  42. F. Fischer, K. Böttinger, H. Xiao, C. Stransky, Y. Acar, M. Backes, and S. Fahl, "Stack overflow considered harmful? the impact of copy&paste on android application security," in 2017 IEEE Symposium on Security and Privacy (SP), 2017, pp. 121-136.
  43. T. Zhang, G. Upadhyaya, A. Reinhardt, H. Rajan, and M. Kim, "Are code examples on an online q&a forum reliable?: A study of api misuse on stack overflow," in 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE), 2018, pp. 886-896.
  44. M. Verdi, A. Sami, J. Akhondali, F. Khomh, G. Uddin, and A. K. Motlagh, "An empirical study of c++ vulnerabilities in crowd-sourced code examples," IEEE Trans. Softw. Eng., vol. 48, no. 5, p. 1497-1514, may 2022. [Online]. Available: https://doi.org/10.1109/TSE.2020.3023664 · DOI
  45. P. Vaithilingam, T. Zhang, and E. L. Glassman, "Expectation vs. experi- ence: Evaluating the usability of code generation tools powered by large language models," in CHI Conference on Human Factors in Computing Systems Extended Abstracts, 2022, pp. 1-7.
  46. M. Monperrus, "Automatic software repair: A bibliography," ACM Comput. Surv., vol. 51, no. 1, jan 2018. [Online]. Available: https: //doi.org/10.1145/3105906 · DOI
  47. Z. Li, C. Wang, Z. Liu, H. Wang, D. Chen, S. Wang, and C. Gao, "Cctest: Testing and repairing code completion systems," in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023, pp. 1238-1250.
  48. S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang et al., "Codexglue: A machine learning bench- mark dataset for code understanding and generation," arXiv preprint arXiv:2102.04664, 2021. · arXiv
  49. N. Muennighoff, Q. Liu, A. Zebaze, Q. Zheng, B. Hui, T. Y. Zhuo, S. Singh, X. Tang, L. von Werra, and S. Longpre, "Octopack: Instruction tuning code large language models," arXiv preprint arXiv:2308.07124, 2023. · arXiv
  50. Y. Liu, C. Tantithamthavorn, Y. Liu, and L. Li, "On the reliability and explainability of automated code generation approaches," arXiv preprint arXiv:2302.09587, 2023. · arXiv
  51. A. Mastropaolo, L. Pascarella, E. Guglielmi, M. Ciniselli, S. Scal- abrino, R. Oliveto, and G. Bavota, "On the robustness of code genera- tion techniques: An empirical study on github copilot," arXiv preprint arXiv:2302.00438, 2023. · arXiv
  52. S. Honarvar, M. van der Wilk, and A. Donaldson, "Turbulence: Systemat- ically and automatically testing instruction-tuned large language models for code," arXiv preprint arXiv:2312.14856, 2023. · arXiv
  53. S. H. Tan, J. Yi, S. Mechtaev, A. Roychoudhury et al., "Codeflaws: a programming competition benchmark for evaluating automated program repair tools," in 2017 IEEE/ACM 39th International Conference on Soft- ware Engineering Companion (ICSE-C). IEEE, 2017, pp. 180-182.
  54. K. Jesse, T. Ahmed, P. T. Devanbu, and E. Morgan, "Large language models and simple, stupid bugs," in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR). Los Alamitos, CA, USA: IEEE Computer Society, may 2023, pp. 563-575. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/MSR59073.2023.00082 · DOI
  55. T. Zhang, C. Gao, L. Ma, M. Lyu, and M. Kim, "An empirical study of common challenges in developing deep learning applications," in 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE), 2019, pp. 104-115.
  56. M. M. Morovati, A. Nikanjam, F. Tambon, F. Khomh, and Z. M. Jiang, "Bug characterization in machine learning-based systems," Empirical Soft- ware Engineering, vol. 29, no. 1, p. 14, 2024.
  57. A. Cater-Steel, M. Toleman, and T. Rout, "Addressing the challenges of replications of surveys in software engineering research," in 2005 Interna- tional Symposium on Empirical Software Engineering, 2005. IEEE, 2005, pp. 10-pp.
  58. J. Bogatinovski and O. Kao, "Auto-logging: AI-centred logging instrumen- tation," in 2023 IEEE/ACM 45th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). IEEE, 2023, pp. 95-100.
  59. A. N. Oppenheim, Questionnaire design, interviewing and attitude mea- surement. Bloomsbury Publishing, 2000. 20.6% -11.8% -29.4% -32.4% -5.9% 73.5% -14.7% -2.9% -8.8% -0% 41.2% -38.2% -8.8% -11.8% -0% 26.5% -20.6% -11.8% -29.4% -11.8% Generation Misinterpretation 5.9% -35.3% -26.5% -26.5% -5.9% 29.4% -17.6% -32.4% -17.6% -2.9% 14.7% -23.5% -26.5% -32.4% -2.9% 5.9% -11.8% -38.2% -35.3% -8.8% NPC 17.6% -23.5% -35.3% -20.6% -2.9% 23.5% -23.5% -26.5% -23.5% -2.9% 17.6% -32.4% -29.4% -14.7% -5.9% 29.4% -29.4% -23.5% -11.8% -5.9% Silly 29.4% -29.4% -20.6% -17.6% -2.9% 58.8% -26.5% -14.7% -0% -0% 67.6% -29.4% -2.9% -0% -0% 55.9% -26.5% -8.8% -8.8% -0% Mistake Wrong 17.6% -38.2% -14.7% -20.6% -8.8% 35.3% -23.5% -20.6% -17.6% -2.9% 17.6% -50% -14.7% -17.6% -0% 26.5% -38.2% -17.6% -14.7% -2.9% Input Type Missing 11.8% -17.6% -20.6% -35.3% -14.7% 5.9% -14.7% -26.5% -44.1% -8.8% 14.7% -14.7% -38.2% -23.5% -8.8% 17.6% -29.4% -17.6% -29.4% -5.9% Corner Cases Wrong 0% -17.6% -26.5% -47.1% -8.8% 20.6% -44.1% -26.5% -5.9% -2.9% 17.6% -50% -17.6% -11.8% -2.9% 8.8% -38.2% -35.3% -11.8% -5.9% Attribute Syntax 47.1% -20.6% -20.6% -5.9% -5.9% 85.3% -8.8% -5.9% -0% -0% 82.4% -8.8% -8.8% -0% -0% 70.6% -14.7% -2.9% -11.8% -0% Error Prompt-biased 11.8% -23.5% -11.8% -44.1% -8.8% 50.0% -26.5% -17.6% -5.9% -0% 11.8% -47.1% -26.5% -11.8% -2.9% 5.9% -23.5% -26.5% -29.4% -14.

Cited by

Sorted by citations

Metadata

Authors
Tambon, F., Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Antoniol, G.
Affiliations
Polytechnique Montréal (CA)
Corresponding author
  • Tambon, Florian
  • Dakhel, Arghavan Moradi
  • Nikanjam, Amin
  • Khomh, Foutse
  • Desmarais, Michel C.
  • Antoniol, Giuliano
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
30
Issue
3
ISSN
1382-3256 (ISSN-L)
DOI
10.48550/arxiv.2403.08937
arXiv
2403.08937
OpenAlex
W4407551092
Cited by
70
References
0
Citations by year
2026: 43 · 2025: 24 · 2024: 3
FWCI
69.67
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Empirical Software Engineering (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • PolyPublie (École Polytechnique de Montréal) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Canadian Institute for Advanced Research (CRDPJ 537462-18)
  • Consortium de Recherche et d’innovation en Aérospatiale au Québec (CRDPJ 537462-18)
  • Fonds de recherche du Québec
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
LLM-generated code, bug patterns, bug taxonomy, automatic code generation, CodeGen, codices, hallucinations, syntactic errors

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review