The DOI matches its Crossref record; 61 of the 62 references it cites were found in a registry or an index.
Empirical assessment and comparison of neuro-evolutionary methods for the automatic off-line design of robot swarms
Open accessAbstract
Neuro-evolution is an appealing approach to generating collective behaviors for robot swarms. In its typical application, known as off-line automatic design, the neural networks controlling the robots are optimized in simulation. It is understood that the so-called reality gap, the unavoidable differences between simulation and reality, typically causes neural network to be less effective on real robots than what is predicted by simulation. In this paper, we present an empirical study on the extent to which the reality gap impacts the most popular and advanced neuro-evolutionary methods for the off-line design of robot swarms. The results show that the neural networks produced by the methods under analysis performed well in simulation, but not in real-robot experiments. Further, the ranking that could be observed in simulation between the methods eventually disappeared. We find compelling evidence that real-robot experiments are needed to reliably assess the performance of neuro-evolutionary methods and that the robustness to the reality gap is the main issue to be addressed to advance the application of neuro-evolution to robot swarms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“EvoStick is the only neuroevolutionary method that has been tested in the automatic design of robot swarms for several missions, without undergoing any mission-specific modification 48 .”
From Automatic design of stigmergy-based behaviours for robot swarms (2024) · cites this paper as [48]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It has been observed that the control software produced by AutoMoDe crosses the reality gap [44][45][46][47] better than traditional approaches based on neuroevolution 43,48 , in which robots are controlled by a neural network that is optimised using an evolutionary algorithm [49][50][51] .”
From Automatic design of stigmergy-based behaviours for robot swarms (2024) · cites this paper as [48]
The cited source says what the claim attributes to it.
In the citing paper
“We do not consider more advanced neuroevolutionary methods (e.g., CMA-ES 70 , xNES 71 , and NEAT 72 ) as previous research has shown that they do not provide any performance advantage over EvoStick when applied off the shelf 48 .”
From Automatic design of stigmergy-based behaviours for robot swarms (2024) · cites this paper as [48]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
(2016) · Cited by 11,679
Practical Nonparametric Statistics
Groggel, D. J. (2000) · Technometrics · doi:10.1080/00401706.2000.10486067 · Cited by 11,041
Completely Derandomized Self-Adaptation in Evolution Strategies
Hansen, N., & Ostermeier, A. (2001) · Evolutionary Computation · doi:10.1162/106365601750190398 · Cited by 4,397 · cited 13×
“The data indicate that, at least on the missions considered, neither the effective search of CMA-ES and xNES , nor the advanced abilities of NEAT to shape the topology of networks have the potential to improve the performance of the neuro-evolutionary approach.”
MethodInfluentialNeural Networks and the Bias/Variance Dilemma
Geman, S., Bienenstock, E., & Doursat, R. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.1 · Cited by 3,578 · cited 1×
“The de fi nition of the approach was inspired by the notion of bias/variance trade-off of machine learning 45 .”
Evolving Neural Networks through Augmenting Topologies
Stanley, K. O., & Miikkulainen, R. (2002) · Evolutionary Computation · doi:10.1162/106365602320169811 · Cited by 3,258 · cited 13×
“The data indicate that, at least on the missions considered, neither the effective search of CMA-ES and xNES , nor the advanced abilities of NEAT to shape the topology of networks have the potential to improve the performance of the neuro-evolutionary approach.”
BackgroundMethodInfluentialGraphical Methods for Data Analysis.
Healy, M. J. R., Chambers, J. M., Cleveland, W. S., Kleiner, B., & Tukey, P. A. (1984) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981587 · Cited by 2,315
Graphical Methods for Data Analysis.
Fisher, N. I., Chambers, J. M., Cleveland, W. S., Kleiner, B., & Tukey, P. A. (1984) · Biometrics · doi:10.2307/2531418 · Cited by 1,982
Swarm robotics: a review from the swarm engineering perspective
Brambilla, M., Ferrante, E., Birattari, M., & Dorigo, M. (2013) · Swarm Intelligence · doi:10.1007/s11721-012-0075-2 · Cited by 1,762
Self-citationThe irace package: Iterated racing for automatic algorithm configuration
López-Ibáñez, M., Dubois-Lacoste, J., Pérez Cáceres, L., Birattari, M., & Stützle, T. (2016) · Operations Research Perspectives · doi:10.1016/j.orp.2016.09.002 · Cited by 1,761 · cited 1×
“The space of the possible combinations of the aforementioned modules is explored using Iterated F-race 69 .”
MethodSelf-citation
Research works printed but not linked: 8
- Berman, S., Kumar, V. & Nagpal, R. Design of control policies for spatially inhomogeneous robot swarms with application to commercial pollination in IEEE International Conference on Robotics and Automation, ICRA (pp. 378–385. IEEE, Piscataway, NJ, USA, 2011).
- Brambilla, M., Brutschy, A., Dorigo, M. & Birattari, M. Property-driven design for swarm robotics: a design method based on prescriptive modeling and model checking. ACM Tran. Auton. Adap. 9, 17:1–17:28 (2014).
- Gauci, M, Chen, J, Li, W, Dodd, T. J & Groß, R. Clustering Objects with Robots That Do Not Compute in Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems - AAMAS2014. (International Foundation for Autonomous Agents and Multiagent Systems: Richland, SC, USA, 2014) 421–428.
- Brooks, R. A. Artificial life and real robots in Towards a Practice of Autonomous Systems. Proceedings of the First European Conference on Artificial Life (eds. Varela, F. J. & Bourgine, P.) 3–10 (MIT Press, Cambridge, MA, USA, 1992).
- Caruana, R., Lawrence, S. & Giles, C. L. Overfitting in neural nets: backpropagation, conjugate gradient, and early stopping in Advances in Neural Information Processing Systems 13 (eds. Leen, T. K., Dietterich, T. G. & Tresp, V.) 402–408 (MIT Press, Cambridge, MA, USA, 2001).
- Morgan, N. & Bourlard, H. Generalization and Parameter Estimation in Feedforward Nets: Some Experiments in Proceedings of the 2nd International Conference on Neural Information Processing Systems, NIPS (ed. Touretzky, D.) 630–637 (MIT Press, Cambridge, MA, USA, 1989).
- Mondada, F. et al. The e-puck, a robot designed for education in engineering in Proceedings of the 9th Conference on Autonomous Robot Systems and Competitions (eds. Gonçalves, P., Torres, P. & Alves, C.) 59–65 (Instituto Politécnico de Castelo Branco, Castelo Branco, Portugal, 2009).
- Lunacek, M & Whitley, L. D. The Dispersion Metric and the CMA Evolution Strategy in Proceedings of the 8th annual conference on Genetic and Evolutionary Computation, GECCO. (ACM: New York, NY, USA, 2006) 477–484.
Cited by
Balancing Collective Exploration and Exploitation in Multi-Agent and Multi-Robot Systems: A Review
Kwa, H. L., Kit, J. L., & Bouffanais, R. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2021.771520 · Cited by 65
“This highlights the need for more extensive experimentation involving larger systems operating in more complex environments (Hasselmann et al., 2021).”
BackgroundSocial learning in swarm robotics
Bredèche, N., & Fontbonne, N. (2021) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2020.0309 · Cited by 61
“Despite the various problems of using controlled environments during learning [10], this form of prior-to-deployment learning has been successfully applied to several problems, from training builder robots for construction [7] to autonomous high sea boats for surveillance [17] and outdoor UAVs…”
Recent trends in robot learning and evolution for swarm robotics
Kuckling, J. (2023) · Frontiers in Robotics and AI · doi:10.3389/frobt.2023.1134841 · Cited by 40
“, 2009) robots in five different missions such as aggregation, homing, shelter, foraging, and gate passing (Hasselmann et al., 2021).”
BackgroundSemantic-Driven Autonomous Visual Navigation for Unmanned Aerial Vehicles
Yue, P., Xin, J. L., Zhang, Y. M., Lu, Y., & Shan, M. (2024) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2024.3363761 · Cited by 32
“This situation is known as reality gap [15].”
BackgroundTowards applied swarm robotics: current limitations and enablers
Kegeleirs, M., & Birattari, M. (2025) · Frontiers in Robotics and AI · doi:10.3389/frobt.2025.1607978 · Cited by 33
Self-citation
Metadata
- Authors
- Hasselmann, K., Ligot, A., Ruddick, J., & Birattari, M.
- Affiliations
- Université Libre de Bruxelles (BE)
- Corresponding author
- Birattari, Mauro
- Venue
- Nature Communications
- Published
- 2021-07-16
- Volume
- 12
- Issue
- 1
- Pages
- 4345-4345
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC8285396
- OpenAlex
- W3178918501
- Cited by
- 53
- References
- 62
- Citations by year
- 2026: 10 · 2025: 11 · 2024: 14 · 2023: 5 · 2022: 8 · 2021: 5
- FWCI
- 3.42
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature Communications (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- European Commission (681872, HORIZON2020)
- Fonds De La Recherche Scientifique - FNRS
- Topics
- Physical Sciences › Engineering › Mechanical Engineering › Modular Robots and Swarm Intelligence
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Metaheuristic Optimization Algorithms Research
- Keywords
- neuroevolution, swarm robotics, reality gap, sim-to-real transfer, collective behavior, neural network controller, evolutionary algorithm comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Empirical assessment and comparison of neuro-evolutionary methods for the automatic off-line design of robot swarms. https://app.cochat.ai/papers/10.1038/s41467-021-24642-3
- The paper
- Hasselmann, K., Ligot, A., Ruddick, J., & Birattari, M. (2021). Empirical assessment and comparison of neuro-evolutionary methods for the automatic off-line design of robot swarms. Nature Communications. https://doi.org/10.1038/s41467-021-24642-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.