CoChat
Sign Up
Cited by 39Advanced Electron Microscopy Techniques and ApplicationsCell Image Analysis TechniquesMedical Image Segmentation Techniques

Petascale pipeline for precise alignment of images from serial section electron microscopy

Open access
Popovych, Sergiy, Macrina, Thomas, Kemnitz, Nico, Castro, Manuel, Nehoran, Barak, Jia, Zhen, Bae, J. Alexander, Mitchell, Eric, Mu, Shang, Trautman, Eric T., Saalfeld, Stephan, Li, Kai, Seung, H. Sebastian·2024·Nature Communications
DOI 10.1038/s41467-023-44354-0PMC PMC10767115

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

The reconstruction of neural circuits from serial section electron microscopy (ssEM) images is being accelerated by automatic image segmentation methods. Segmentation accuracy is often limited by the preceding step of aligning 2D section images to create a 3D image stack. Precise and robust alignment in the presence of image artifacts is challenging, especially as datasets are attaining the petascale. We present a computational pipeline for aligning ssEM images with several key elements. Self-supervised convolutional nets are trained via metric learning to encode and align image pairs, and they are used to initialize iterative fine-tuning of alignment. A procedure called vector voting increases robustness to image artifacts or missing image data. For speedup the series is divided into blocks that are distributed to computational workers for alignment. The blocks are aligned to each other by composing transformations with decay, which achieves a global alignment without resorting to a time-consuming global optimization. We apply our pipeline to a whole fly brain dataset, and show improved accuracy relative to prior state of the art. We also demonstrate that our pipeline scales to a cubic millimeter of mouse visual cortex. Our pipeline is publicly available through two open source Python packages.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41467-023-44354-0

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Computational advances have now closed this gap and FAFB images were accurately aligned with a new approach that leverages convolutional nets.

    “Computational advances have now closed this gap107 and FAFB images were accurately aligned with a new approach that leverages convolutional nets108.”

    From Neuronal wiring diagram of an adult brain (2024) · cites this paper as [108]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 36

  1. [1]Harris KM Uniform serial sectioning for transmission electron microscopy J. Neurosci. 2006 26 12101 12103 10.1523/JNEUROSCI.3994-06.2006 17122034 PMC6675417 · DOI
  2. [2]Briggman KL Bock DD Volume electron microscopy for neuronal circuit reconstruction Curr. Opin. Neurobiol. 2012 22 154 161 10.1016/j.conb.2011.10.022 22119321 · DOI
  3. [3]Zheng Z A complete electron microscopy volume of the brain of adult drosophila melanogaster Cell 2018 174 730–743.e22 10.1016/j.cell.2018.06.019 30033368 PMC6063995 · DOI · Paper page
  4. [4]Yin W A petascale automated imaging pipeline for mapping neuronal circuits with high-throughput transmission electron microscopy Nat. Commun. 2020 11 4949 10.1038/s41467-020-18659-3 33009388 PMC7532165 · DOI
  5. [6]Shapson-Coe, A. et al. A connectomic study of a petascale fragment of human cerebral cortex. bioRxiv10.1101/2021.05.29.446289 (2021).
  6. [7]Lee K Convolutional nets for reconstructing neural circuits from brain images acquired by serial section electron microscopy Curr. Opin. Neurobiol. 2019 55 188 198 10.1016/j.conb.2019.04.001 31071619 PMC6559369 · DOI · Paper page
  7. [8]Khairy, K., Denisov, G. & Saalfeld, S. Joint deformable registration of large EM image volumes: a matrix solver approach. arXiv https://arxiv.org/abs/1804.10019 (2018). · arXiv
  8. [9]Saalfeld S Fetter R Cardona A Tomancak P Elastic volume reconstruction from series of ultra-thin microscopy sections Nat. Methods 2012 9 717 10.1038/nmeth.2072 22688414 · DOI · Paper page
  9. [10]Bock DD Network anatomy and in vivo physiology of visual cortical neurons Nature 2011 471 177 182 10.1038/nature09802 21390124 PMC3095821 · DOI
  10. [11]Wetzel, A. W. et al. Registering large volume serial-section electron microscopy image sets for neural circuit reconstruction using FFT signal whitening. In 2016 IEEE Applied Imagery Pattern Recognition Workshop (AIPR) 1–10 (IEEE, 2016).
  11. [12]Scheffer, L. K., Karsh, B. & Vitaladevun, S. Automated alignment of imperfect EM images for neural reconstruction. arXiv https://arxiv.org/abs/1304.6034 (2013). · arXiv
  12. [13]Lowe, D. G. Object recognition from local scale-invariant features. In Proceedings of the Seventh IEEE International Conference on Computer Vision 1150–1157 (IEEE, 1999).
  13. [14]Kazhdan, M. et al. Gradient-domain fusion for color correction in large EM image stacks. arXiv https://arxiv.org/abs/1506.02079 (2015). · arXiv
  14. [16]Mitchell, E., Keselj, S., Popovych, S., Buniatyan, D. & Seung, H. S. Siamese encoding and alignment by multiscale learning with self-supervision. arXiv https://arxiv.org/abs/1904.02643 (2019). · arXiv
  15. [17]Yoo, I. et al. ssEMnet: Serial-section electron microscopy image registration using a spatial transformer network with learned features. Deep Learn.10.1007/978-3-319-67558-9_29 (2017). · DOI
  16. [18]Balakrishnan, G., Zhao, A., Sabuncu, M. R., Guttag, J. & Dalca, A. V. VoxelMorph: A learning framework for deformable medical image registration. IEEE Trans. Med. Imag. https://ieeexplore.ieee.org/document/8633930 (2019). 30716034
  17. [19]Zhou, S. et al. Fast and accurate electron microscopy image registration with 3D convolution. In Medical Image Computing and Computer Assisted Intervention (MICCAI) 478–486 (Springer International Publishing, 2019).
  18. [20]Shu, C., Xin, T., Zhou, F., Chen, X. & Han, H. Dual networks for High-Precision and High-Speed registration of brain electron microscopy images. Brain Sci.10, 453312 (2020). PMC7071614 32045982
  19. [21]Nguyen-Duc, T. et al. Weakly supervised learning in deformable EM image registration using slice interpolation. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI) 670–673 (ISBI, 2019).
  20. [22]Jain, V. Adversarial image alignment and interpolation. arXiv https://arxiv.org/abs/1707.00067 (2017). · arXiv
  21. [23]Popovych, S., Alexander Bae, J. & Seung, H. S. Caesar: Segment-wise alignment method for solving discontinuous deformations. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI) 1214–1218 (IEEE, 2020).
  22. [24]Bajcsy R Kovačič S Multiresolution elastic matching Comp. Vision Graph. Image Process. 1989 46 1 21 10.1016/S0734-189X(89)80014-3 · DOI
  23. [25]Ranjan, A. & Black, M. J. Optical flow estimation using a spatial pyramid network. arXiv https://arxiv.org/abs/1611.00850 (2016). · arXiv
  24. [26]Bromley, J., Guyon, I., LeCun, Y., Sackinger, E. & Shah, R. Signature verification using a “siamese” time delay neural network. In Proceedings of the 6th International Conference on Neural Information Processing Systems 737–744 (Morgan Kaufmann Publishers Inc. USA, 1993).
  25. [27]Broit, C. Optimal Registration of Deformed Images. Ph.D. Thesis (University of Pennsylvania, 1981).
  26. [28]Tasdizen T Automatic mosaicking and volume assembly for high-throughput serial-section transmission electron microscopy J. Neurosci. Methods 2010 193 132 144 10.1016/j.jneumeth.2010.08.001 20713087 PMC2952705 · DOI
  27. [29]Möller, B., Garcia, R. & Posch, S. Towards objective quality assessment of image registration results. In VISAPP Proceedings of the Second International Conference on Computer Vision Theory and Applications 233–242 (VISAPP, 2007).
  28. [30]Li, P. H. et al. Automated reconstruction of a serial-section em drosophila brain with flood-filling networks and local realignment. bioRxiv10.1101/605634 (2020).
  29. [31]Dorkenwald S FlyWire: online community for whole-brain connectomics Nat. Methods 2022 19 119 128 10.1038/s41592-021-01330-0 34949809 PMC8903166 · DOI · Paper page
  30. [33]Hui, T.W., Tang, X. & Loy, C.C. LiteFlowNet: A lightweight convolutional neural network for optical flow estimation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 8981–8989 (IEEE, 2018).
  31. [35]Mahalingam G A scalable and modular automated pipeline for stitching of large electron microscopy datasets eLife 2022 11 e76534 10.7554/eLife.76534 35880860 PMC9427110 · DOI
  32. [36]Buniatyan, D. et al. Weakly supervised deep metric learning for template matching. In Advances in Computer Vision 3rd edn, Vol. 2 (Springer International Publishing, 2020).
  33. [37]Jaderberg, M., Simonyan, K., Zisserman, A. & Kavukcuoglu, K. Spatial transformer networks. In Advances in Neural Information Processing Systems 5th edn, Vol. 6 (eds. Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M. & Garnett, R.) Ch. 2017–2025 (Curran Associates, Inc., 2015).
  34. [38]Williams, L. Pyramidal parametrics. In Proceedings of the 10th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’83 1–11 (Association for Computing Machinery USA, 1983).
  35. [39]Wu, J., Silversmith, W. M., Lee, K. & Seung, H. S. Chunkflow: hybrid cloud processing of large 3D images by convolutional nets. Nat. Methods10.1038/s41592-021-01088-5 (2021). 33750934
  36. [40]Hider, R. et al. The brain observatory storage service and database (bossdb): a cloud-native approach for petascale neuroscience discovery. Front. Neuroinform.10.3389/fninf.2022.828787 (2022). PMC8885591 35242021 · DOI

Cited by

Sorted by citations

Authors
Popovych, S., Macrina, T., Kemnitz, N., Castro, M., Nehoran, B., Jia, Z., Bae, J. A., Mitchell, E., Mu, S., Trautman, E. T., Saalfeld, S., Li, K., & Seung, H. S.
Affiliations
  • Princeton University (US)
  • Howard Hughes Medical Institute (US)
Corresponding author
Seung, H. Sebastian
ORCID
Venue
Nature Communications
Publisher
Springer Science and Business Media LLC
Published
2024-01-04
Volume
15
Issue
1
Pages
289-289
ISSN
2041-1723 (ISSN-L)
DOI
10.1038/s41467-023-44354-0
PMC
PMC10767115
OpenAlex
W4390578327
Cited by
39
References
4
Citations by year
2026: 10 · 2025: 17 · 2024: 10 · 2023: 1 · 2022: 1
FWCI
6.76
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • ODNI | Intelligence Advanced Research Projects Activity (D16PC00004, D16PC0005, D16PC00003)
  • U.S. Department of Health & Human Services | NIH | National Institute of Neurological Disorders and Stroke (U19 NS104648, R01NS104926)
  • U.S. Department of Health & Human Services | NIH | National Eye Institute (R01 EY027036)
  • U.S. Department of Health & Human Services | NIH | National Institute of Mental Health (U01 MH114824, RF1MH123400)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Structural Biology › Advanced Electron Microscopy Techniques and Applications
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
Keywords
serial section electron microscopy, image registration, 3D image reconstruction, neural circuit reconstruction, metric learning, composition of transformations, fly brain, mouse visual cortex, image artifacts
MeSH terms
Animals, Brain, Image Processing, Computer-Assisted, Microscopy, Electron, Imaging, Three-Dimensional, Mice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review