Investigating interoperability of the LSST data management software stack with Astropy
Open accessAbstract
The Large Synoptic Survey Telescope (LSST) will be an 8.4m optical survey telescope sited in Chile and capable of imaging the entire sky twice a week. The data rate of approximately 15TB per night and the requirements to both issue alerts on transient sources within 60 seconds of observing and create annual data releases means that automated data management systems and data processing pipelines are a key deliverable of the LSST construction project. The LSST data management software has been in development since 2004 and is based on a C++ core with a Python control layer. The software consists of nearly a quarter of a million lines of code covering the system from fundamental WCS and table libraries to pipeline environments and distributed process execution. The Astropy project began in 2011 as an attempt to bring together disparate open source Python projects and build a core standard infrastructure that can be used and built upon by the astronomy community. This project has been phenomenally successful in the years since it has begun and has grown to be the de facto standard for Python software in astronomy. Astropy brings with it considerable expectations from the community on how astronomy Python software should be developed and it is clear that by the time LSST is fully operational in the 2020s many of the prospective users of the LSST software stack will expect it to be fully interoperable with Astropy. In this paper we describe the overlap between the LSST science pipeline software and Astropy software and investigate areas where the LSST software provides new functionality. We also discuss the possibilities of re-engineering the LSST science pipeline software to build upon Astropy, including the option of contributing affliated packages.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The LSST Science Platform builds on recent trends in remote data analysis and experiences from JHU SciServer, Gaia GAVIP, and NOAO Datalab.
“It builds on recent trends in remote data analysis and practical experiences in the astronomical context gathered by projects such as the JHU SciServer ( Raddick et al. 2017 ) , Gaia GAVIP ( Vagg et al. 2016 ) , or NOAO Datalab ( Fitzpatrick et al. 2016 ) .”
From LSST: From Science Drivers to Reference Design and Anticipated Data Products (2019) · cites this paper as Fitzpatrick et al. 2016
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Astropy: A community Python package for astronomy
The Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., Greenfield, P., Droettboom, M., Bray, E., Aldcroft, T., Davis, M., Ginsburg, A., Price-Whelan, A. M., et al. (2013) · Astronomy & Astrophysics · doi:10.1051/0004-6361/201322068 · Cited by 14,620 · see citing works
Self-citationData Structures for Statistical Computing in Python
McKinney, W. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-00a · Cited by 11,511 · see citing works
LSST: from Science Drivers to Reference Design and Anticipated Data Products
Ivezić, Ž., Kahn, S. M., Tyson, J. A., Abel, B., Acosta, E., Allsman, R., Alonso, D., AlSayyad, Y., Anderson, S. F., Andrew, J., et al. (2008) · arXiv · doi:10.48550/arxiv.0805.2366 · Cited by 4,357 · see citing works
Self-citationLSST: From Science Drivers to Reference Design and Anticipated Data Products
Ivezić, Ž., Kahn, S. M., Tyson, J. A., Abel, B., Acosta, E., Allsman, R., Alonso, D., AlSayyad, Y., Anderson, S. F., Andrew, J., et al. (2019) · The Astrophysical Journal · doi:10.3847/1538-4357/ab042c · Cited by 4,357 · see citing works
Self-citationThe Iraf Data Reduction And Analysis System
Tody, D. (1986) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.968154 · Cited by 1,824 · see citing works
Cython: The Best of Both Worlds
Behnel, S., Bradshaw, R., Citro, C., Dalcín, L., Seljebotn, D. S., & Smith, K. (2010) · Computing in Science & Engineering · doi:10.1109/mcse.2010.118 · Cited by 1,233 · see citing works
The James Webb Space Telescope
Gardner, J. P., Mather, J. C., Clampin, M., Doyon, R., Greenhouse, M., Hammel, H. B., Hutchings, J. B., Jakobsen, P., Lilly, S., Long, K. S., et al. (2006) · Space Science Reviews · doi:10.1007/s11214-006-8315-7 · Cited by 629 · see citing works
Representations of celestial coordinates in FITS
Calabretta, M. R., & Greisen, E. W. (2002) · Astronomy & Astrophysics · doi:10.1051/0004-6361:20021327 · Cited by 431 · see citing works
Miyazaki, S., Komiyama, Y., Nakaya, H., Kamata, Y., Doi, Y., Hamana, T., Karoji, H., Furusawa, H., Kawanomoto, S., Morokuma, T., et al. (2012) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.926844 · Cited by 332 · see citing works
Self-citationMatsuoka, Y., Onoue, M., Kashikawa, N., Iwasawa, K., Strauss, M. A., Nagao, T., Imanishi, M., Niida, M., Toba, Y., Akiyama, M., et al. (2016) · The Astrophysical Journal · doi:10.3847/0004-637x/828/1/26 · Cited by 228 · see citing works
Self-citationCoding Together at Scale: GitHub as a Collaborative Social Network
Lima, A., Rossi, L., & Musolesi, M. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14552 · Cited by 153 · see citing works
Automated scientific software scripting with SWIG
Beazley, D. M. (2003) · Future Generation Computer Systems · doi:10.1016/s0167-739x(02)00171-1 · Cited by 118 · see citing works
An open source application framework for astronomical imaging pipelines
Axelrod, T. S., Kantor, J. P., Lupton, R. H., & Pierfederici, F. (2010) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.857297 · Cited by 117 · see citing works
Self-citationThe LSST Data Management SystemPreprint
Jurić, M., Kantor, J. P., Lim, K.-T., Lupton, R. H., Dubois-Felsmann, G. P., Jenness, T., Axelrod, T. S., Aleksić, J., Allsman, R. A., AlSayyad, Y., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.07914 · Cited by 96 · see citing works
Self-citationThe Hierarchical Triangular Mesh
Kunszt, P., Szalay, A. S., & Thakar, A. R. (2006) · Springer eBooks · doi:10.1007/10849171_83 · Cited by 76 · see citing works
ASDF: A new data format for astronomy
Greenfield, P., Droettboom, M., & Bray, E. M. (2015) · Astronomy and Computing · doi:10.1016/j.ascom.2015.06.004 · Cited by 60 · see citing works
Collective (2001) · ESO astrophysics symposia · doi:10.1007/b82674 · Cited by 44 · see citing works
LSST Telescope and site status
Gressler, W., DeVries, J., Hileman, E. A., Neill, D., Sebag, J., Wiecha, O., Andrew, J., Lotz, P. J., & Schoening, W. E. (2014) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2056711 · Cited by 43 · see citing works
Knight, S. (2005) · Computing in Science & Engineering · doi:10.1109/mcse.2005.11 · Cited by 26 · see citing works
CFITSIO, v2.0: A New Full-Featured Data Interface
Pence, W. D. (1999) · ASPC · Cited by 26 · see citing works
Learning from FITS: Limitations in use in modern astronomical research
Thomas, B., Jenness, T., Economou, F., Greenfield, P., Hirst, P., Berry, D., Bray, E. M., Gray, N., Muna, D., Turner, J., et al. (2015) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · Cited by 24 · see citing works
Self-citationLearning from FITS: Limitations in use in modern astronomical research
Thomas, B., Jenness, T., Economou, F., Greenfield, P., Hirst, P., Berry, D. S., Bray, E., Gray, N., Muna, D., Turner, J., et al. (2015) · Astronomy and Computing · doi:10.1016/j.ascom.2015.01.009 · Cited by 22 · see citing works
Self-citationLearning from 25 years of the extensible N-Dimensional Data Format
Jenness, T., Berry, D., Currie, M. J., Draper, P. W., Economou, F., Gray, N., McIlwrath, B., Shortridge, K., Taylor, M. B., Wallace, P. T., et al. (2014) · Astronomy and Computing · doi:10.1016/j.ascom.2014.11.001 · Cited by 21 · see citing works
Self-citationKoposov, S. E., & Bartunov, O. S. (2006) · Cited by 18 · see citing works
ParselTongue: AIPS Talking Python
Kettenis, M., van Langevelde, H. J., Reynolds, C., & Cotton, W. D. (2006) · Cited by 18 · see citing works
More flexibility in representing geometric distortion in astronomical images
Shupe, D. L., Laher, R. R., Storrie‐Lombardi, L. J., Surace, J., Grillmair, C. J., Levitan, D., & Sesar, B. (2012) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.925460 · Cited by 18 · see citing works
Standards of Fundamental Astronomy
Hohenkerk, C. Y. (2011) · Scholarpedia · doi:10.4249/scholarpedia.11404 · Cited by 16 · see citing works
Space-Time Coordinate Metadata for the Virtual Observatory Version 1.33
Rots, A. H. (2007) · doi:10.5479/ads/bib/2007ivoa.spec.1030r · Cited by 15 · see citing works
The LSST Data Management System
Jurić, M., Dubois-Felsmann, G. P., Ciardi, D. R., & Dyk, S. D. V. (2015) · CaltechAUTHORS (California Institute of Technology) · Cited by 13 · see citing works
Self-citationAnalysing ALMA data with CASAPreprint
Petry, D. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1201.3454 · Cited by 12 · see citing works
AST: A library for modelling and manipulating coordinate systems
Berry, D., Warren-Smith, R. F., & Jenness, T. (2016) · Astronomy and Computing · doi:10.1016/j.ascom.2016.02.003 · Cited by 11 · see citing works
Self-citationAstrOmatic Software in the Era of Large Stellar Photometric Surveys
Bertin, E., Delorme, P., & Bouy, H. (2011) · Astrophysics and space science proceedings · doi:10.1007/978-3-642-22113-2_9 · Cited by 10 · see citing works
AIPS Developments in the Nineties
van Moorsel, G., Kemball, A., & Greisen, E. W. (1996) · Cited by 8 · see citing works
What Python Can Do for Astronomy
Greenfield, P. (2011) · ASPC · Cited by 8 · see citing works
Economou, F., Jenness, T., & Rees, N. P. (2002) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.460642 · Cited by 5 · see citing works
Self-citationORAC-DR: Pipelining With Other People's Code
Economou, F., Bridger, A., Wright, G., Jenness, T., Currie, M. J., & Adamson, A. J. (1999) · ASPC · Cited by 5 · see citing works
Self-citationThe LSST Data Processing Pipeline
Axelrod, T. S., Connolly, A. J., Ivezić, Ž., Kantor, J., Lupton, R. H., Plante, R., Stubbs, C. W., & Wittman, D. (2004) · AAS · Cited by 5 · see citing works
Self-citationKantor, J. (2014) · Cited by 5 · see citing works
Efficient data structures for masks on 2D grids
Reinecke, M., & Hivon, E. F. (2015) · Astronomy and Astrophysics · doi:10.1051/0004-6361/201526549 · Cited by 4 · see citing works
The James Webb Space Telescope Data Calibration Pipeline
Bushouse, H., Droettboom, M., & Greenfield, P. (2015) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-7b98e3ed-007 · Cited by 4 · see citing works
DrizzlePac: Managing Multi-component WCS Solutions for HST Data
Hack, W., Dencheva, N., & Fruchter, A. S. (2013) · ASPC · Cited by 4 · see citing works
Using the AST Library to Create and Use STC-S Region Descriptions
Berry, D., & Draper, P. W. (2010) · Durham Research Online (Durham University) · Cited by 4 · see citing works
The LSST Data Processing Software Stack: Summer 2015 ReleasePreprint
Jenness, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06790 · Cited by 4 · see citing works
Self-citationOrganizations and Strategies in Astronomy II
Heck, A. (2001) · Astrophysics and space science library · doi:10.1007/978-94-010-0666-8 · Cited by 3 · see citing works
A New CL for IRAF Based On Python
Greenfield, P., & White, R. L. (2000) · Cited by 2 · see citing works
The Development of SAOImage DS9: Lessons Learned from a Small but Successful Software Project
Joye, W. A., & Mandel, E. (2005) · Cited by 2 · see citing works
A next-generation open-source toolkit for FITS file image viewing
Jeschke, E., Inagaki, T. J., & Kackley, R. D. (2012) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.924930 · Cited by 2 · see citing works
Calabretta, M. R. (2011) · Astrophysics Source Code Library · Cited by 2 · see citing works
GWCS: A Library for Managing World Coordinate Systems
Dencheva, N., Greenfield, P., & Droettboom, M. (2017) · ASPC · Cited by 2 · see citing works
The JWST Data Calibration Pipeline
Bushouse, H., Droettboom, M., & Greenfield, P. (2017) · Astronomical Data Analysis Software and Systems XXVII · Cited by 2 · see citing works
Lsst Classes, As Astropy Spin-Off Candidates
Bosch, J. (2016) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.48435 · Cited by 1 · see citing works
Self-citationStreamlining an IRAF data reduction process with AstroPy and NDMapper
Turner, J. E. H. (2016) · doi:10.5281/zenodo.50888 · Cited by 1 · see citing works
PyMidas--A Python Interface to ESO-MIDAS
Hook, R. N., Maisala, S., Oittinen, T., Ullgrén, M., Vasko, K., Savolainen, V., Lindroos, J., Anttila, M., Solin, O., Møller, P. L., et al. (2006) · Cited by 1 · see citing works
Mandel, E., & Murray, S. S. (1998) · Cited by 1 · see citing works
Easy Comparison of Dataset Footprints with MOC
Fernique, P., & Boch, T. (2012) · ASPC · Cited by 1 · see citing works
Streamlining An Iraf Data Reduction Process With Astropy And Ndmapper
Turner, J. E. H. (2016) · doi:10.5281/zenodo.50991
MOC - HEALPix Multi-Order Coverage map Version 1.0Preprint
Fernique, P., Boch, T., Donaldson, T., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.02937
Flagged first, then by citations
5 of the 62 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Kahn, S., "Final design of the Large Synoptic Survey Telescope," in [Ground-Based and Airborne Telescopes IV], Hall, H. J., Gilmozzi, R., and Marshall, H. K., eds., Proc. SPIE 9906, in press (2016).
- DiBona, C., Cooper, D., and Stone, M., eds., [Open Sources 2.0: The Continuing Evolution ], O'Reilly (2006). ISBN 0-596-00802-3.
- Draper, P. W., Berry, D. S., Jenness, T., and Economou, F., "GAIA -Version 4," in [Astronomical Data Analysis Software and Systems XVIII], Bohlender, D. A., Durand, D., and Dowler, P., eds., ASP Conf. Ser. 411, 575 (Sept. 2009).
- Currie, M. J., Berry, D. S., Jenness, T., Gibb, A. G., Bell, G. S., and Draper, P. W., "Starlink Software in 2013," in [Astronomical Data Analysis Software and Systems XXIII ], Manset, N. and Forshay, P., eds., ASP Conf. Ser. 485, 391 (May 2014).
Cited by
Sorted by citations
The Astropy Project: Building an Open-science Project and Status of the v2.0 Core Package
Price-Whelan, A., Sipőcz, B., Günther, H. M., Lim, P., Crawford, S., Conseil, S., Shupe, D., Craig, M., Dencheva, N., Ginsburg, A., et al. (2018) · Astronomical Journal · doi:10.3847/1538-3881/aabc4f · Cited by 6,858 · see citing works
Self-citationLSST: From Science Drivers to Reference Design and Anticipated Data Products
Ivezić, Ž., Kahn, S. M., Tyson, J. A., Abel, B., Acosta, E., Allsman, R., Alonso, D., AlSayyad, Y., Anderson, S. F., Andrew, J., et al. (2019) · The Astrophysical Journal · doi:10.3847/1538-4357/ab042c · Cited by 4,357 · see citing works
Self-citationGammapy - A prototype for the CTA science tools
Lefaucheur, J., Deil, C., Zanin, R., Boisson, C., Khelifi, B., Terrier, R., Wood, M., Mohrmann, L., Chakraborty, N., Watson, J. J., et al. (2017) · International Cosmic Ray Conference · doi:10.22323/1.301.0766 · Cited by 94 · see citing works
Gammapy - A prototype for the CTA science toolsPreprint
Deil, C., Zanin, R., Lefaucheur, J., Boisson, C., Khelifi, B., Terrier, R., Wood, M., Mohrmann, L., Chakraborty, N., Watson, J. J., et al. (2017) · arXiv (Cornell University) · doi:10.3204/pubdb-2018-00836 · Cited by 33 · see citing works
Developers, R. O. S. P., AlSayyad, Y., Armstrong, R., Axelrod, T. S., Banovetz, J., Bechtol, K., Bellm, E. C., Bernardinelli, P. H., Bosch, J., Bradshaw, A., et al. (2025) · DOE SLAC National Accelerator Laboratory Repository · doi:10.71929/rubin/2570545 · Cited by 20 · see citing works
Self-citationSun, Q.-B., Qian, S. ‐., Zhū, L., Dong, A., Zhi, Q., Liao, W.-P., Zhao, E.-G., Han, Z.-T., Liu, W., Zang, L., et al. (2022) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stac3272 · Cited by 13 · see citing works
LSST data management software development practices and tools
Jenness, T., Economou, F., O’Mullane, W., Lim, K.-T., Hoblitt, J., Sick, J., Thornton, A., Hernandez, F., Findeisen, K. P., Pietrowicz, S., et al. (2018) · SPIE Software and Cyberinfrastructure for Astronomy · doi:10.1117/12.2312157 · Cited by 12 · see citing works
Self-citationLSST active optics system software architecture
Thomas, S. J., Chandrasekharan, S., Lotz, P. J., Xin, B., Claver, C. F., Angeli, G. Z., Sebag, J., & Dubois-Felsmann, G. P. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2231798 · Cited by 7 · see citing works
Self-citationBratt, S. (2023) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.769 · Cited by 5 · see citing works
NGTS-33b: a young super-Jupiter hosted by a fast-rotating massive hot star
Alves, D. R., Jenkins, J. S., Vines, J. I., Battley, M. P., Lendl, M., Bouchy, F., Nielsen, L. D., Gill, S., Moyano, M., Anderson, D. R., et al. (2024) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stae2582 · Cited by 5 · see citing works
The LSST Real-Time Active Optics System
Thomas, S. J. (2017) · Adaptive Optics for Extremely Large Telescopes (AO4ELT) · doi:10.26698/ao4elt5.0137 · Cited by 3 · see citing works
Multifilter observations of the complex periodic variations in eight pre-main sequence stars
Koen, C. (2022) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stac3276 · Cited by 2 · see citing works
Ivezić, Ž. (2016) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.44697
Large Synoptic Survey Telescope Town Hall
Willman, B. (2016) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.44768
The Lsst Data Processing Software Stack
Jenness, T. (2016) · American Astronomical Society Meeting Abstracts #227 · doi:10.5281/zenodo.44634
Self-citation
- Authors
- Jenness, T., Bosch, J., Owen, R., Parejko, J., Sick, J., Swinbank, J., de Val-Borro, M., Dubois-Felsmann, G., Lim, K.-T., Lupton, R. H., Schellart, P., Krughoff, K. S., & Tollerud, E. J.
- Affiliations
- LSST (United States)
- Princeton University (US)
- University of Washington (US)
- Princeton Univ. (United States)
- California Institute of Technology (US)
- SLAC National Accelerator Laboratory (US)
- Space Telescope Science Institute (US)
- ORCID
- Venue
- SPIE Proceedings
- Publisher
- SPIE
- Published
- 2016-07-26
- Volume
- 9913
- Pages
- 99130G
- ISSN
- 0277-786X (ISSN-L)
- OpenAlex
- W2496031908
- Cited by
- 14
- References
- 62
- Citations by year
- 2024: 1 · 2023: 1 · 2022: 2 · 2019: 1 · 2018: 2 · 2017: 3 · 2016: 4
- FWCI
- 2.75
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE (publishedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- Zenodo (CERN European Organization for Nuclear Research) (cc-by)
- Funders
- National Science Foundation (1227061)
- U.S. Department of Energy (76SF00515, DE–AC02–76SF00515, AC02‐76SF00515, DE-AC02-)
- University of Washington
- SLAC National Accelerator Laboratory (DEAC02−76SF00515, AC02‐76SF00515)
- Topics
- Physical Sciences › Physics and Astronomy › Instrumentation › Astronomy and Astrophysical Research
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Engineering › Computational Mechanics › Astronomical Observations and Instrumentation
- Keywords
- LSST, astropy, software interoperability, software reengineering, transient alerts, data release
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
