Convolutional Networks Can Learn to Generate Affinity Graphs for Image Segmentation
Open accessAbstract
Many image segmentation algorithms first generate an affinity graph and then partition it. We present a machine learning approach to computing an affinity graph using a convolutional network (CN) trained using ground truth provided by human experts. The CN affinity graph can be paired with any standard partitioning algorithm and improves segmentation accuracy significantly compared to standard hand-designed affinity functions. We apply our algorithm to the challenging 3D segmentation problem of reconstructing neuronal processes from volumetric electron microscopy (EM) and show that we are able to learn a good affinity graph directly from the raw EM images. Further, we show that our affinity graph improves the segmentation accuracy of both simple and sophisticated graph partitioning algorithms. In contrast to previous work, we do not rely on prior knowledge in the form of hand-designed image features or image preprocessing. Thus, we expect our algorithm to generalize effectively to arbitrary image types.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The hemibrain images were automatically segmented using flood-filling convolutional nets, whereas FlyWire used boundary-detecting convolutional nets.
“The hemibrain images were automatically segmented using flood-filling convolutional nets109, whereas FlyWire used the older, less computationally expensive approach of boundary-detecting convolutional nets110,111.”
From Neuronal wiring diagram of an adult brain (2024) · cites this paper as [111]
Sections 1 and 3 describe a convolutional network that predicts affinities between neighboring voxels, allowing segmentation along object boundaries. The paper’s dataset is retinal tissue (section 4.1); it does not say that FlyWire used this approach or compare its computational cost with flood-filling networks. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891 · see citing works
Fast approximate energy minimization via graph cuts
Boykov, Y., Veksler, O., & Zabih, R. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.969114 · Cited by 6,988 · see citing works
The structure of the nervous system of the nematode <i>Caenorhabditis elegans</i>
White, J. G., Southgate, E., Thomson, J. N., & Brenner, S. (1986) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1986.0056 · Cited by 5,589 · cited 1× · see citing works
“White et al. (1986) took over 10 years to complete the manual reconstruction of the entire nervous system of a nematode worm C. elegans, which contains only 302 neurons, underscoring the need for significant automation.”
BackgroundNeural Networks: Tricks of the Trade: Second Edition
(2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8 · Cited by 2,116 · see citing works
Serial Block-Face Scanning Electron Microscopy to Reconstruct Three-Dimensional Tissue Nanostructure
Denk, W., & Horstmann, H. (2004) · PLoS Biology · doi:10.1371/journal.pbio.0020329 · Cited by 1,644 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 24
- Al-Kofahi, K., Lasek, S., Szarowski, D., Pace, C., Nagy, G., Turner, J., et al. (2002). Rapid automated three-dimensional tracing of neurons from confocal image stacks. IEEE Trans. Info. Tech. Biomed., 6(2), 171-187.
- Andres, B., K öthe, U., Helmstaedter, M., Denk, W., & Hamprecht, F. A. (2008). Seg- mentation of SBFSEM volume data of neural tissue by hierarchical classification. In G. Rigoll (Ed.), Pattern recognition (vol. 5096 of LNCS, pp. 142-152). Berlin: Springer.
- Briggman, K. L., & Denk, W. (2006). Towards neural circuit reconstruction with volume electron microscopy techniques. Curr. Opin. Neurobiol., 16(5), 562- 570.
- Carlbom, I., Terzopoulos, D., & Harris, K. M. (1994). Computer-assisted registration, segmentation, and 3D reconstruction from images of neuronal tissue sections. IEEE Trans. Medical Imaging, 13(2), 351-362.
- Cormen, T. H., Leiserson, C. E., Rivest, R. L., & Stein, C. (2000). Introduction to algorithms. Cambridge, MA: MIT Press.
- Cour, T., Benezit, F., & Shi, J. (2005). Spectral segmentation with multiscale graph decomposition. IEEE Conf. Computer Vision and Pattern Rec. (Vol. 2, 1124-1131). Piscataway, NJ: IEEE.
- Dhillon, I., Guan, Y., & Kulis, B. (2005). A fast kernel-based multilevel algorithm for graph clustering. In Proc. ACM SIGKDD Intl. Conf. Knowledge Discovery in Data Mining (pp. 629-634). New York: ACM Press.
- Egmont-Petersen, M., de Ridder, D., & Handels, H. (2002). Image processing with neural networks: A review. Pattern Recognition, 35(10), 2279-2301.
- Felzenszwalb, P., & Huttenlocher, D. (2004). Efficient graph-based image segmenta- tion. International Journal of Computer Vision, 59(2), 167-181.
- Fiala, J. C. (2005). Reconstruct: A free editor for serial section microscopy. Journal of Microscopy, 218(1), 52-61.
- Fowlkes, C., Martin, D., & Malik, J. (2003). Learning affinity functions for image seg- mentation: Combining patch-based and gradient-based approaches. . Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recog- nition. Washington, DC: IEEE Computer Society.
- Gdalyahu, Y., Weinshall, D., & Werman, M. (1999). Stochastic image segmentation by typical cuts. In Proceedings of the IEEE Computer Society Conference on Com- puter Vision and Pattern Recognition (Vol. 2, pp. 596-601). Washington, DC: IEEE Computer Society.
- Helmstaedter, M. N., Briggman, K. L., & Denk, W. (2007). Segmentation of SBFSEM volume data of neural tissue by hierarchical classification. Society for Neuroscience Abstracts, pp. 142-152.
- Jain, V., Murray, J., Roth, F., Turaga, S., Zhigulin, V., Briggman, K., et al. (2007). Supervised learning of image restoration with convolutional networks. In Proc. Intl. Conf. Computer Vision. Piscataway, NJ: IEEE.
- LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324.
- LeCun, Y., Bottou, L., Orr, G., & Muller, K. (1998). Efficient backprop. In G. Orr & K. Muller (Eds.), Neural networks: Tricks of the trade. Berlin: Springer.
- Liang, J., McInerney, T., & Terzopoulos, D. (2006). United Snakes. Medical Image Analysis, 10(2), 215-233.
- Martin, D. R., Fowlkes, C. C., & Malik, J. (2004). Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE Trans. on Pattern Analysis and Machine Intelligence, 26(1), 1-20.
- Nekovei, R., & Sun, Y. (1995). Back-propagation network and its configuration for blood vessel detection in angiograms. IEEE Trans. Neural Networks, 6(1), 64-72.
- Ning, F., Delhomme, D., LeCun, Y., Piano, F., Bottou, L., & Barbano, P. E. (2005). Toward automatic phenotyping of developing embryos from videos. IEEE Trans- actions on Image Processing, 14(9), 1360-1371.
- Ricci, E., & Perfetti, R. (2007). Retinal blood vessel segmentation using line operators and support vector classification. IEEE Transactions on Medical Imaging, 26(10), 1357-1365.
- Shi, J., & Malik, J. (2000). Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8), 888-905.
- Sinthanayothin, C., Boyce, J. F., Cook, H. L., & Williamson, T. H. (1999). Automated localisation of the optic disc, fovea, and retinal blood vessels from digital colour fundus images. British Journal of Opthalmology, 83, 902-910.
- Smith, S. (2007). Circuit reconstruction tools today. Current Opinion in Neurobiology, 17(5), 601-608.
Cited by
Sorted by citations
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503 · see citing works
Connectomic reconstruction of the inner plexiform layer in the mouse retina
Helmstaedter, M., Briggman, K. L., Turaga, S. C., Jain, V., Seung, H. S., & Denk, W. (2013) · Nature · doi:10.1038/nature12346 · Cited by 997 · see citing works
Self-citationDeep convolutional neural networks for multi-modality isointense infant brain image segmentation
Zhang, W., Li, R., Deng, H., Wang, L., Lin, W., Ji, S., & Shen, D. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2014.12.061 · Cited by 817 · see citing works
Jing, L., Zhao, M., Li, P., & Xu, X. (2017) · Measurement · doi:10.1016/j.measurement.2017.07.017 · Cited by 698 · see citing works
Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks
Dou, Q., Chen, H., Yu, L., Zhao, L., Qin, J., Wang, D., Mok, V. C., Shi, L., & Heng, P.-A. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528129 · Cited by 679 · see citing works
Neuronal wiring diagram of an adult brain
Dorkenwald, S., Matsliah, A., Sterling, A. R., Schlegel, P., Yu, S.-C., McKellar, C. E., Lin, A., Costa, M., Eichler, K., Yin, Y., et al. (2024) · Nature · doi:10.1038/s41586-024-07558-y · Cited by 570 · see citing works
“The hemibrain images were automatically segmented using flood-filling convolutional nets 109 , whereas FlyWire used the older, less computationally expensive approach of boundary-detecting convolutional nets 110,111 .”
MethodSelf-citationDeep Feature Learning for Knee Cartilage Segmentation Using a Triplanar Convolutional Neural Network
Prasoon, A., Petersen, K., Igel, C., Lauze, F., Dam, E., & Nielsen, M. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_31 · Cited by 582 · see citing works
Dong, H., Song, K., He, Y., Xu, J., Yan, Y., & Meng, Q. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2958826 · Cited by 559 · see citing works
Improving Computer-Aided Detection UsingConvolutional Neural Networks and Random View Aggregation
Roth, H. R., Lu, L., Liu, J., Yao, J., Seff, A., Cherry, K., Kim, L., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2482920 · Cited by 608 · see citing works
“Extensions of ConvNets to 3D have been proposed, but computational cost and memory consumption can be still too high to efficiently implement them on current computer graphics hardware units [23].”
BackgroundInfluentialHigh-precision automated reconstruction of neurons with flood-filling networks
Januszewski, M., Kornfeld, J., Li, P. H., Pope, A., Blakely, T., Lindsey, L., Maitin-Shepard, J., Tyka, M., Denk, W., & Jain, V. (2018) · Nature Methods · doi:10.1038/s41592-018-0049-4 · Cited by 380 · see citing works
Self-citationJing, L., Wang, T., Zhao, M., & Wang, P. (2017) · Sensors · doi:10.3390/s17020414 · Cited by 379 · see citing works
Machine learning for medical ultrasound: status, methods, and future opportunities
Brattain, L. J., Telfer, B. A., Dhyani, M., Grajo, J. R., & Samir, A. E. (2018) · Abdominal Radiology · doi:10.1007/s00261-018-1517-0 · Cited by 262 · see citing works
Reconstruction of neocortex: Organelles, compartments, cells, circuits, and activity
Turner, N. L., Macrina, T., Bae, J. A., Yang, R., Wilson, A. M., Schneider-Mizell, C., Lee, K., Lu, R., Wu, J., Bodor, A. L., et al. (2022) · Cell · doi:10.1016/j.cell.2022.01.023 · Cited by 206 · see citing works
Self-citationUltrasound image-based thyroid nodule automatic segmentation using convolutional neural networks
Ma, J., Wu, F., Jiang, T., Zhao, Q., & Kong, D. (2017) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-017-1649-7 · Cited by 187 · see citing works
Neuronal wiring diagram of an adult brain
Dorkenwald, S., Matsliah, A., Sterling, A. R., Schlegel, P., Yu, S.-C., McKellar, C. E., Lin, A., Costa, M., Eichler, K., Yin, Y., et al. (2023) · doi:10.1101/2023.06.27.546656 · Cited by 149 · see citing works
Self-citationSteel Surface Defect Detection Using a New Haar–Weibull-Variance Model in Unsupervised Manner
Liu, K., Wang, H., Chen, H., Qu, E., Tian, Y., & Sun, H. (2017) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2017.2712838 · Cited by 161 · see citing works
Automated synaptic connectivity inference for volume electron microscopy
Dorkenwald, S., Schubert, P. J., Killinger, M. F., Urban, G., Mikula, S., Svara, F., & Kornfeld, J. (2017) · Nature Methods · doi:10.1038/nmeth.4206 · Cited by 132 · see citing works
Binary and analog variation of synapses between cortical pyramidal neurons
Dorkenwald, S., Turner, N. L., Macrina, T., Lee, K., Lu, R., Wu, J., Bodor, A. L., Bleckert, A. A., Brittain, D., Kemnitz, N., et al. (2019) · doi:10.1101/2019.12.29.890319 · Cited by 130 · see citing works
“We trained a 3D convolutional network to generate three nearest neighbor (Turaga et al. 2010) and 13 long-range affinity maps (K.”
MethodSelf-citationLee, K., Turner, N., Macrina, T., Wu, J., Lu, R., & Seung, H. S. (2019) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2019.04.001 · Cited by 52 · see citing works
Self-citationCellular Anatomy of the Mouse Primary Motor Cortex
Muñoz-Castañeda, R., Zingg, B., Matho, K. S., Wang, Q., Chen, X., Foster, N. N., Narasimhan, A., Li, A., Hirokawa, K. E., Huo, B., et al. (2020) · doi:10.1101/2020.10.02.323154 · Cited by 270 · see citing works
ShuTu: Open-Source Software for Efficient and Accurate Reconstruction of Dendritic Morphology
Jin, D. Z., Zhao, T., Hunt, D. L., Tillage, R. P., Hsu, C.-L., & Spruston, N. (2019) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2019.00068 · Cited by 13 · see citing works
- Authors
- Turaga, S. C., Murray, J. F., Jain, V., Roth, F., Helmstaedter, M., Briggman, K., Denk, W., & Seung, H. S.
- Affiliations
- Massachusetts Institute of Technology (US)
- Howard Hughes Medical Institute (US)
- Max Planck Institute for Medical Research (DE)
- Corresponding author
- Turaga, Srinivas C.
- Murray, Joseph F.
- Jain, Viren
- Roth, Fabian
- Helmstaedter, Moritz
- Briggman, Kevin
- Denk, Winfried
- Seung, H. Sebastian
- ORCID
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2010-02
- Volume
- 22
- Issue
- 2
- Pages
- 511-538
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2169805405
- Cited by
- 376
- References
- 5
- Citations by year
- 2026: 5 · 2025: 13 · 2024: 13 · 2023: 25 · 2022: 23 · 2021: 34 · 2020: 34 · 2019: 39 · 2018: 39 · 2017: 33 · 2016: 23 · 2015: 23 · 2014: 18 · 2013: 21 · 2012: 12
- FWCI
- 12.13
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion)
- Max Planck Institute for Plasma Physics (submittedVersion)
- Max Planck Digital Library (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Funders
- Howard Hughes Medical Institute
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
- Keywords
- image segmentation, graph partitioning, convolutional neural networks, 3D segmentation, graph-based segmentation
- MeSH terms
- Algorithms, Artificial Intelligence, Image Processing, Computer-Assisted, Mathematical Computing, Mathematics, Microscopy, Electron, Pattern Recognition, Automated, Neural Networks, Computer, Mathematical Concepts
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
