Self-supervised similarity search for large scientific datasets
Open accessAbstract
We present the use of self-supervised learning to explore and exploit large unlabeled datasets. Focusing on 42 million galaxy images from the latest data release of the Dark Energy Spectroscopic Instrument (DESI) Legacy Imaging Surveys, we first train a self-supervised model to distill low-dimensional representations that are robust to symmetries, uncertainties, and noise in each image. We then use the representations to construct and publicly release an interactive semantic similarity search tool. We demonstrate how our tool can be used to rapidly discover rare objects given only a single example, increase the speed of crowd-sourcing campaigns, and construct and improve training sets for supervised applications. While we focus on images from sky surveys, the technique is straightforward to apply to any scientific dataset of any dimensionality. The similarity search web app can be found at https://github.com/georgestein/galaxy_search
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
Schlegel, D. J., Finkbeiner, D. P., & Davis, M. (1998) · The Astrophysical Journal · doi:10.1086/305772 · Cited by 15,126 · see citing works
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · see citing works
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
McInnes, L., Healy, J., & Melville, J. (2018) · arXiv · doi:10.48550/arxiv.1802.03426 · Cited by 7,101 · see citing works
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · see citing works
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.131 · Cited by 4,451 · see citing works
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · see citing works
Improved Baselines with Momentum Contrastive LearningPreprint
Chen, X., Fan, H., Girshick, R., & He, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04297 · Cited by 2,579 · see citing works
Overview of the DESI Legacy Imaging Surveys
Dey, A., Schlegel, D. J., Lang, D., Blum, R., Burleigh, K., Fan, X., Findlay, J. R., Finkbeiner, D. P., Herrera, D., Juneau, S., et al. (2019) · The Astronomical Journal · doi:10.3847/1538-3881/ab089d · Cited by 1,881 · see citing works
THE POPULATION OF GALAXY–GALAXY STRONG LENSES IN FORTHCOMING OPTICAL IMAGING SURVEYS
Collett, T. E. (2015) · The Astrophysical Journal · doi:10.1088/0004-637x/811/1/20 · Cited by 354 · see citing works
The strong gravitational lens finding challenge
Metcalf, R. B., Meneghetti, M., Avestruz, C., Bellagamba, F., Bom, C. R., Bertin, E., Cabanac, R., Courbin, F., Davies, A., Decencière, E., et al. (2019) · Astronomy & Astrophysics · doi:10.1051/0004-6361/201832797 · Cited by 152 · see citing works
Self-supervised Pretraining of Visual Features in the Wild
Goyal, P., Caron, M., Lefaudeux, B., Xu, M., Wang, P., Pai, V., Singh, M., Liptchinsky, V., Misra, I., Joulin, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.01988 · Cited by 141 · see citing works
Self-supervised Representation Learning for Astronomical Images
Hayat, M. A., Stein, G., Harrington, P., Lukić, Z., & Mustafa, M. (2021) · The Astrophysical Journal Letters · doi:10.3847/2041-8213/abf2c7 · Cited by 79 · see citing works
Self-citationMining for Strong Gravitational Lenses with Self-supervised Learning
Stein, G., Blaum, J., Harrington, P., Medan, T., & Lukić, Z. (2022) · The Astrophysical Journal · doi:10.3847/1538-4357/ac6d63 · Cited by 52 · see citing works
Self-citationBuckley‐Geer, E., Lin, H., Drabek-Maunder, E., Allam, S., Tucker, D. L., Armstrong, R., Barkhouse, W., Bertin, E., Brodwin, M., Desai, S., et al. (2011) · The Astrophysical Journal · doi:10.1088/0004-637x/742/1/48 · Cited by 20 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
AstroCLIP: a cross-modal foundation model for galaxies
Parker, L., Lanusse, F., Golkar, S., Sarra, L., Cranmer, M., Bietti, A., Eickenberg, M., Krawezik, G. P., McCabe, M., Morel, R., et al. (2024) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stae1450 · Cited by 75 · see citing works
A review of unsupervised learning in astronomy
Fotopoulou, S. (2024) · Astronomy and Computing · doi:10.1016/j.ascom.2024.100851 · Cited by 45 · see citing works
Photometric redshifts from SDSS images with an interpretable deep capsule network
Dey, B., Andrews, B. H., Newman, J. A., Mao, Y.-Y., Rau, M. M., & Zhou, R. (2022) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stac2105 · Cited by 36 · see citing works
Anomalies, representations, and self-supervision
Dillon, B. M., Favaro, L., Feiden, F., Modak, T., & Plehn, T. (2024) · SciPost Physics Core · doi:10.21468/scipostphyscore.7.3.056 · Cited by 22 · see citing works
Data compression and inference in cosmology with self-supervised machine learning
Akhmetzhanova, A., Mishra-Sharma, S., & Dvorkin, C. (2023) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stad3646 · Cited by 18 · see citing works
Enabling unsupervised discovery in astronomical images through self-supervised representations
Mohale, K., & Lochner, M. (2024) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stae926 · Cited by 18 · see citing works
Detecting galaxy tidal features using self-supervised representation learning
Desmons, A., Brough, S., & Lanusse, F. (2024) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stae1402 · Cited by 15 · see citing works
A deep neural network based reverse radio spectrogram search algorithm
Ma, P. X., Croft, S., Lintott, C. J., & Siemion, A. P. V. (2023) · RAS Techniques and Instruments · doi:10.1093/rasti/rzad056 · Cited by 6 · see citing works
Radio Galaxy Zoo: Leveraging latent space representations from variational autoencoder
Andrianomena, S., & Tang, H. (2024) · Journal of Cosmology and Astroparticle Physics · doi:10.1088/1475-7516/2024/06/034 · Cited by 6 · see citing works
Optimizing machine learning methods to discover strong gravitational lenses in the deep lens survey
C., K. V. G., Sheng, S., Jones, T. A., Choi, C. P., & Sharpnack, J. (2023) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stad1709 · Cited by 4 · see citing works
Scaling Laws for Galaxy Images
Walmsley, M., Bowles, M., Scaife, A. M. M., Makechemu, J. S., Gordon, A. J., Ferguson, A. M. N., Mann, R. G., Pearson, J., Popp, J. J., Bovy, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.02973 · Cited by 2 · see citing works
“We hope this paper will provide a comparison point for future work investigating self-supervised alternatives [39, 26, 40] or hybrid approaches [41].”
BackgroundAstroMAE: Redshift Prediction Using a Masked Autoencoder with a Novel Fine-Tuning Architecture
Fathkouhi, A. D., & Fox, G. (2024) · IEEE International Conference on eScience · doi:10.1109/e-science62913.2024.10678724 · Cited by 2 · see citing works
Augmenting X-ray astronomical representations with text
Vago, N. O. P., Tella, R. D., Cuesta-Lazaro, C., Smith, M. J., Garraffo, C., & Martínez-Galarza, J. R. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12343-1
- Authors
- Stein, G., Harrington, P., Blaum, J., Medan, T., & Lukic, Z.
- Affiliations
- Lawrence Berkeley National Laboratory (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2110.13151
- OpenAlex
- W3210518412
- Cited by
- 13
- References
- 15
- Citations by year
- 2026: 1 · 2024: 7 · 2023: 4 · 2022: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- U.S. Department of Energy (DE‐ AC02‐06CH11357, -AC02-05CH11231, AC02‐06CH11357, 05CH11231, 06CH11357, DE-AC02-)
- National Energy Research Scientific Computing Center (DE–AC02–05CH11231, 05CH11231, DE-AC02-06CH11357, AC02-05CH11231)
- Office of Science (AC02‐06CH11357, AC02-05CH11231, DE-AC02, 05CH11231, 06CH11357, DE-AC02-06CH11357)
- Advanced Scientific Computing Research (DE-AC02-05CH11231, AC02-05CH11231, AC02-06CH11357, DE-AC02-06CH11357)
- High Energy Physics (AC02-06CH11357, DEAC02--05CH11231, DE-AC02-06CH11357)
- Argonne National Laboratory (DE-AC02, AC02–06CH11357, DE-AC02-05CH11231, 06CH11357)
- Lawrence Berkeley National Laboratory (AC02‐05CH11231, DE-AC02-06CH11357, 05CH11231)
- Topics
- Physical Sciences › Physics and Astronomy › Astronomy and Astrophysics › Galaxies: Formation, Evolution, Phenomena
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Engineering › Computational Mechanics › Astronomical Observations and Instrumentation
- Keywords
- self-supervised learning, semantic similarity search, galaxy images, DESI Legacy Imaging Surveys, low-dimensional representation, crowdsourcing, training set construction, sky survey, interactive web tool
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
