The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Empirical review of automated analysis tools on 47,587 Ethereum smart contracts
Open accessAbstract
Over the last few years, there has been substantial research on automated analysis, testing, and debugging of Ethereum smart contracts. However, it is not trivial to compare and reproduce that research. To address this, we present an empirical evaluation of 9 state-of-the-art automated analysis tools using two new datasets: i) a dataset of 69 annotated vulnerable smart contracts that can be used to evaluate the precision of analysis tools; and ii) a dataset with all the smart contracts in the Ethereum Blockchain that have Solidity source code available on Etherscan (a total of 47,518 contracts). The datasets are part of SmartBugs, a new extendable execution framework that we created to facilitate the integration and comparison between multiple analysis tools and the analysis of Ethereum smart contracts. We used SmartBugs to execute the 9 automated analysis tools on the two datasets. In total, we ran 428,337 analyses that took approximately 564 days and 3 hours, being the largest experimental setup to date both in the number of tools and in execution time. We found that only 42% of the vulnerabilities from our annotated dataset are detected by all the tools, with the tool Mythril having the higher accuracy (27%). When considering the largest dataset, we observed that 97% of contracts are tagged as vulnerable, thus suggesting a considerable number of false positives. Indeed, only a small number of vulnerabilities (and of only two categories) were detected simultaneously by four or more tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Software engineers rely on mature and well-integrated tools within integrated development environments to automate many repetitive and tedious activities [55, 218].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [55]
Citations
Cites
Making Smart Contracts Smarter
Luu, L., Chu, D.-H., Olickel, H., Saxena, P., & Hobor, A. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978309 · Cited by 2,249
Securify: Practical Security Analysis of Smart Contracts
Tsankov, P., Dan, A., Drachsler-Cohen, D., Gervais, A., Bünzli, F., & Vechev, M. (2018) · Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3243734.3243780 · Cited by 1,043
Slither: A Static Analysis Framework for Smart Contracts
Feist, J., Grieco, G., & Groce, A. (2019) · 2019 IEEE/ACM 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB) · doi:10.1109/wetseb.2019.00008 · Cited by 872
ZEUS: Analyzing Safety of Smart Contracts
Kalra, S., Goel, S., Dhawan, M., & Sharma, S. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23082 · Cited by 702
SmartCheck: static analysis of ethereum smart contracts
Tikhomirov, S., Voskresenskaya, E., Ivanitskiy, I., Takhaviev, R., Marchenko, E., & Alexandrov, Y. (2018) · Proceedings of the 1st International Workshop on Emerging Trends in Software Engineering for Blockchain · doi:10.1145/3194113.3194115 · Cited by 672
Finding The Greedy, Prodigal, and Suicidal Contracts at Scale
Nikolić, I., Kolluri, A., Sergey, I., Saxena, P., & Hobor, A. (2018) · Proceedings of the 34th Annual Computer Security Applications Conference · doi:10.1145/3274694.3274743 · Cited by 660
Formal Verification of Smart Contracts: Short Paper
Bhargavan, K., Delignat-Lavaud, A., Fournet, C., Gollamudi, A., Gonthier, G., Kobeissi, N., Kulatova, N., Rastogi, A., Sibut-Pinote, T., Swamy, N., et al. (2016) · Proceedings of the 2016 ACM Workshop on Programming Languages and Analysis for Security · doi:10.1145/2993600.2993611 · Cited by 626
A Survey on Ethereum Systems Security: Vulnerabilities, Attacks, and Defenses
Chen, H., Pendleton, M., Njilla, L., & Xu, S. (2021) · ACM Computing Surveys · doi:10.1145/3391195 · Cited by 480
Osiris: Hunting for Integer Bugs in Ethereum Smart Contracts
Torres, C. F., Schütte, J., & State, R. (2018) · Proceedings of the 34th Annual Computer Security Applications Conference · doi:10.1145/3274694.3274737 · Cited by 440
KEVM: A Complete Formal Semantics of the Ethereum Virtual Machine
Hildenbrandt, E., Saxena, M., Rodrigues, N., Zhu, X., Daian, P., Guth, D., Moore, B., Park, D., Zhang, Y., Stefanescu, A., et al. (2018) · 2018 IEEE 31st Computer Security Foundations Symposium (CSF) · doi:10.1109/csf.2018.00022 · Cited by 418
Research works printed but not linked: 4
- Ilya Grishchenko, Matteo Maffei, and Clara Schneidewind. 2018. A Semantic Framework for the Security Analysis of Ethereum Smart Contracts. In Principles of Security and Trust, Lujo Bauer and Ralf Küsters (Eds.). Springer International Publishing, Cham, 243--269.
- Bernhard Mueller. 2018. Smashing ethereum smart contracts for fun and real profit. In 9th Annual HITB Security Conference (HITBSecConf). HITB, Amsterdam, Netherlands, 54.
- 10.5555/3291291.3291303 · doi:10.5555/3291291.3291303
- Matt Suiche. 2017. Porosity: A decompiler for blockchain-based smart contracts bytecode. DEF con 25 (2017), 11.
Cited by
A Survey on Blockchain Interoperability: Past, Present, and Future Trends
Belchior, R., Vasconcelos, A., Guerreiro, S., & Correia, M. (2022) · ACM Computing Surveys · doi:10.1145/3471140 · Cited by 757
A Survey of Smart Contract Formal Specification and Verification
Tolmach, P., Li, Y., Lin, S.-W., Liu, Y., & Li, Z. (2022) · ACM Computing Surveys · doi:10.1145/3464421 · Cited by 236
Ghaleb, A., & Pattabiraman, K. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397385 · Cited by 263
SmartBugs: a framework to analyze solidity smart contracts
Ferreira, J. F., Cruz, P., Durieux, T., & Abreu, R. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3415298 · Cited by 161
Self-citationSMARTIAN: Enhancing Smart Contract Fuzzing with Static and Dynamic Data-Flow Analyses
Choi, J., Kim, D., Kim, S., Grieco, G., Groce, A., & Cha, S. K. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678888 · Cited by 192
Metadata
- Authors
- Durieux, T., Ferreira, J. F., Abreu, R., & Cruz, P.
- Affiliations
- University of Lisbon (PT)
- Venue
- Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2020-06-27
- Pages
- 530-541
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W2982275082
- Cited by
- 409
- References
- 37
- Citations by year
- 2026: 54 · 2025: 101 · 2024: 87 · 2023: 78 · 2022: 42 · 2021: 35 · 2020: 12
- FWCI
- 31.86
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Fundação para a Ciência e a Tecnologia (PTDC/CCI-COM/29300/2017)
- Horizon 2020 (822404)
- Topics
- Physical Sciences › Computer Science › Information Systems › Blockchain Technology Applications and Security
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- Ethereum smart contracts, automated analysis tool, vulnerability detection, Solidity, Mythril, false positives, tool comparison, Etherscan, smart contract security
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Empirical review of automated analysis tools on 47,587 Ethereum smart contracts. https://app.cochat.ai/papers/10.1145/3377811.3380364
- The paper
- Durieux, T., Ferreira, J. F., Abreu, R., & Cruz, P. (2020). Empirical review of automated analysis tools on 47,587 Ethereum smart contracts. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. https://doi.org/10.1145/3377811.3380364
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.