A benchmark for comparison of cell tracking algorithms
Open accessAbstract
Motivation: Automatic tracking of cells in multidimensional time-lapse fluorescence microscopy is an important task in many biomedical applications. A novel framework for objective evaluation of cell tracking algorithms has been established under the auspices of the IEEE International Symposium on Biomedical Imaging 2013 Cell Tracking Challenge. In this article, we present the logistics, datasets, methods and results of the challenge and lay down the principles for future uses of this benchmark. Results: The main contributions of the challenge include the creation of a comprehensive video dataset repository and the definition of objective measures for comparison and ranking of the algorithms. With this benchmark, six algorithms covering a variety of segmentation and tracking paradigms have been compared and ranked based on their performance on both synthetic and real datasets. Given the diversity of the datasets, we do not declare a single winner of the challenge. Instead, we present and discuss the results for each individual dataset separately. Availability and implementation: The challenge Web site (http://www.codesolorzano.com/celltrackingchallenge) provides access to the training and competition datasets, along with the ground truth of the training videos. It also provides access to Windows and Linux executable files of the evaluation software and most of the algorithms that competed in the challenge. Contact: codesolorzano@unav.es Supplementary information: Supplementary data are available at Bioinformatics online.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Collective cell migration in morphogenesis, regeneration and cancer
Friedl, P., & Gilmour, D. T. (2009) · Nature Reviews Molecular Cell Biology · doi:10.1038/nrm2720 · Cited by 2,668 · see citing works
Cancer Invasion and the Microenvironment: Plasticity and Reciprocity
Friedl, P., & Alexander, S. (2011) · Cell · doi:10.1016/j.cell.2011.11.016 · Cited by 1,940 · see citing works
Measurement of mechanical tractions exerted by cells in three-dimensional matrices
Legant, W. R., Miller, J. S., Blakely, B. L., Cohen, D. M., Genin, G. M., & Chen, C. S. (2010) · Nature Methods · doi:10.1038/nmeth.1531 · Cited by 647 · see citing works
Cell population tracking and lineage construction with spatiotemporal context
Li, K., Miller, E. D., Chen, M., et al. (2008) · Medical Image Analysis · doi:10.1016/j.media.2008.06.001 · Cited by 352 · see citing works
Segmenting and tracking fluorescent cells in dynamic 3-D microscopy with coupled active surfaces
Dufour, A., Shinin, V., Tajbakhsh, S., et al. (2005) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.852790 · Cited by 314 · see citing works
Zimmer, C., Labruyère, E., Meas‐Yedid, V., Guillén, N., & Olivo‐Marín, J. (2002) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2002.806292 · Cited by 284 · see citing works
Advanced Level-Set-Based Cell Tracking in Time-Lapse Fluorescence Microscopy
Dzyubachyk, O., van Cappellen, W. A., Essers, J., Niessen, W. J., & Meijering, E. H. W. (2010) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2009.2038693 · Cited by 255 · see citing works
Self-citationMultiple Hypothesis Tracking for Cluttered Biological Image Sequences
Chenouard, N., Bloch, I., & Olivo‐Marín, J. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.97 · Cited by 246 · see citing works
Tracking in cell and developmental biology
Meijering, E. H. W., Dzyubachyk, O., Smal, I., et al. (2009) · Seminars in Cell and Developmental Biology · doi:10.1016/j.semcdb.2009.07.004 · Cited by 244 · see citing works
Self-citationBenchmarking HEp-2 Cells Classification Methods
Foggia, P., Percannella, G., Soda, P., et al. (2013) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2013.2268163 · Cited by 232 · see citing works
Al-Kofahi, O., Radke, R. J., Goderie, S. K., Shen, Q., Temple, S., & Roysam, B. (2006) · Cell Cycle · doi:10.4161/cc.5.3.2426 · Cited by 172 · see citing works
3-D Active Meshes: Fast Discrete Deformable Models for Cell Tracking in 3-D Time-Lapse Microscopy
Dufour, A., Thibeaux, R., Labruyère, E., et al. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2099125 · Cited by 163 · see citing works
Segmentation of confocal microscope images of cell nuclei in thick tissue sections
Ortíz-de-Solórzano, C., Rodriguez, E. G., Jones, A. V., Pinkel, D., Gray, J. W., Sudar, D., & Lockett, S. J. (1999) · Journal of Microscopy · doi:10.1046/j.1365-2818.1999.00463.x · Cited by 156 · see citing works
Coupled minimum-cost flow cell tracking for high-throughput quantitative analysis
Padfield, D. R., Rittscher, J., & Roysam, B. (2010) · Medical Image Analysis · doi:10.1016/j.media.2010.07.006 · Cited by 142 · see citing works
Svoboda, D., Kozubek, M., & Stejskal, S. (2009) · Cytometry Part A · doi:10.1002/cyto.a.20714 · Cited by 130 · see citing works
Self-citationA call for bioimaging software usability
Carpenter, A. E., Kamentsky, L. D., & Eliceiri, K. W. (2012) · Nature Methods · doi:10.1038/nmeth.2073 · Cited by 122 · see citing works
Multiple Nuclei Tracking Using Integer Programming for Quantitative Cancer Cell Cycle Analysis
Li, F., Zhou, X., Ma, J., & Wong, S. T. C. (2009) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2009.2027813 · Cited by 119 · see citing works
Comparison of segmentation algorithms for fluorescence microscopy images of cells
Dima, A. A., Elliott, J. T., Filliben, J. J., Halter, M. W., Peskin, A. P., Bernal, J., Kociołek, M., Brady, M. C., Tang, H. C., & Plant, A. L. (2011) · Cytometry Part A · doi:10.1002/cyto.a.21079 · Cited by 109 · see citing works
Reliable cell tracking by global data association
Bise, R., Yin, Z., & Takeo Kanade, A. (2011) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2011.5872571 · Cited by 108 · see citing works
Segmentation and Shape Tracking of Whole Fluorescent Cells Based on the Chan–Vese Model
Maška, M., Daněk, O., Garasa, S., et al. (2013) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2013.2243463 · Cited by 102 · see citing works
Self-citationLin, G., Chawla, M. K., Olson, K., Guzowski, J. F., Barnes, C. A., & Roysam, B. (2004) · Cytometry Part A · doi:10.1002/cyto.a.20099 · Cited by 99 · see citing works
On the digital trail of mobile cells
Zimmer, C., Zhang, B., Dufour, A., et al. (2006) · IEEE Signal Processing Magazine · doi:10.1109/msp.2006.1628878 · Cited by 92 · see citing works
Rapoport, D. H., Becker, T., Mamlouk, A. M., Schicktanz, S., & Kruse, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0027315 · Cited by 62 · see citing works
Magnusson, K. E. G., & Jaldén, J. (2012) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2012.6235564 · Cited by 61 · see citing works
Self-citationAn automatic segmentation algorithm for 3D cell cluster splitting using volumetric confocal images
Indhumathi, C., Cai, Y., Guan, F. Y., et al. (2011) · Journal of Microscopy · doi:10.1111/j.1365-2818.2010.03482.x · Cited by 35 · see citing works
AUTOMATIC SEGMENTATION OF NUCLEI IN 3D MICROSCOPY IMAGES OF C.ELEGANS
Long, F., Peng, H., & Myers, E. (2007) · IEEE International Symposium on Biomedical Imaging: From Nano to Macro (ISBI) · doi:10.1109/isbi.2007.356907 · Cited by 35 · see citing works
Comparison of parameter‐adapted segmentation methods for fluorescence micrographs
Held, C., Palmisano, R., Häberle, L., et al. (2011) · Cytometry Part A · doi:10.1002/cyto.a.21122 · Cited by 34 · see citing works
Automated and semi-automated cell tracking: addressing portability challenges
Kan, A., Chakravorty, R., Bailey, J. A., et al. (2011) · Journal of Microscopy · doi:10.1111/j.1365-2818.2011.03529.x · Cited by 23 · see citing works
Tracking and Quantitative Analysis of Dynamic Movements of Cells and ParticlesPreprint
Rohr, K., Godinez, W. J., Harder, N., Wörz, S., Mattes, J., Tvaruskó, W., & Eils, R. (2010) · Cold Spring Harbor Protocols · doi:10.1101/pdb.top80 · Cited by 22 · see citing works
Self-citationGeneration of Synthetic Image Datasets for Time-Lapse Fluorescence Microscopy
Svoboda, D., & Ulman, V. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-31298-4_56 · Cited by 20 · see citing works
Self-citationQuantitative in vivo microscopy: the return from the ‘omics’
Fernández‐González, R., Muñoz‐Barrutia, A., Barcellos‐Hoff, M. H., & Ortíz-de-Solórzano, C. (2006) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2006.07.005 · Cited by 16 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021) · Nature Methods · doi:10.1038/s41592-020-01008-z · Cited by 10,477 · see citing works
Deep learning for cellular image analysis
Moen, E., Bannon, D., Kudo, T., Graf, W., Covert, M., & Van Valen, D. (2019) · Nature Methods · doi:10.1038/s41592-019-0403-1 · Cited by 1,225 · see citing works
Deep learning in microscopy image analysis: a survey
Xing, F., Xie, Y., Su, H., Liu, F., & Yang, L. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2766168 · Cited by 460 · see citing works
- Authors
- Maška, M., Ulman, V., Svoboda, D., Matula, P., Matula, P., Ederra, C., Urbiola, A., España, T., Venkatesan, S., Balak, D. M. W., Karas, P., Bolcková, T., Štreitová, M., Carthel, C., Coraluppi, S., Harder, N., Rohr, K., Magnusson, K. E. G., Jaldén, J., Blau, H. M., Dzyubachyk, O., Křížek, P., Hagen, G. M., Pastor-Escuredo, D., Jimenez-Carretero, D., Ledesma-Carbayo, M. J., Muñoz-Barrutia, A., Meijering, E., Kozubek, M., & Ortiz-de-Solorzano, C.
- Affiliations
- Masaryk University (CZ)
- 1 Center for Biomedical Image Analysis, Masaryk University, 602 00 Brno, Czech Republic, 2Cancer Imaging Laboratory, Oncology Division, Center for Applied Medical Research, University of Navarra, 31008 Pamplona, Spain, 3Biomedical Imaging Group Rotterdam, Erasmus University Medical Center, 3015 GE Rotterdam, The Netherlands, 4Fusion Technology and Systems Department, Compunetix Inc., Monroeville, PA 15146, USA, 5Biomedical Computer Vision Group, Department of Bioinformatics and Functional Genomics, University of Heidelberg, BIOQUANT, IPMB and DKFZ, 69120 Heidelberg, Germany, 6KTH Royal Institute of Technology, ACCESS Linnaeus Center, Department of Signal Processing, 100 44 Stockholm, Sweden, 7Baxter Laboratory for Stem Cell Biology, Department of Microbiology and Immunology, Institute for Stem Cell Biology and Regenerative Medicine, Stanford University School of Medicine, Stanford, CA 94305, USA, 8Division of Image Processing, Leiden University Medical Center, 2300 RC Leiden, The Netherlands, 9Institute of C
- ORCID
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-02-12
- Volume
- 30
- Issue
- 11
- Pages
- 1609-1617
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- PMC
- PMC4029039
- OpenAlex
- W2141273399
- Cited by
- 415
- References
- 31
- Citations by year
- 2026: 25 · 2025: 19 · 2024: 24 · 2023: 22 · 2022: 46 · 2021: 32 · 2020: 37 · 2019: 47 · 2018: 39 · 2017: 40 · 2016: 26 · 2015: 46 · 2014: 12
- FWCI
- 24.53
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Bioinformatics (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- Funders
- California Institute for Regenerative Medicine (RT1-01001-1)
- Univerzita Karlova v Praze (UNCE204022, 204022)
- Grantová Agentura České Republiky (302/12/G157, GPP205/12/P392)
- Centro para el Desarrollo Tecnológico Industrial
- Bundesministerium für Bildung und Forschung
- Vetenskapsrådet
- National Institutes of Health (5r21ar062359-02)
- National Institute on Aging (R01AG009521, R01AG020961)
- National Heart, Lung, and Blood Institute (R01HL096113)
- National Institute of Arthritis and Musculoskeletal and Skin Diseases (R21-AR062359)
- European Social Fund
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Advanced Fluorescence Microscopy Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Keywords
- cell tracking, benchmark, Cell Tracking Challenge, time-lapse fluorescence microscopy, algorithm evaluation, segmentation, synthetic dataset, ground truth, multi-target tracking
- MeSH terms
- Algorithms, Microscopy, Fluorescence, Benchmarking, Cell Tracking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
