CoChat
Sign Up
Cited by 6Scientific Computing and Data ManagementDistributed and Parallel Computing SystemsCloud Computing and Resource Management

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

An accessible infrastructure for artificial intelligence using a Docker-based JupyterLab in Galaxy

Open access
Kumar, Anup, Cuccuru, Gianmauro, Grüning, Björn, Backofen, Rolf·2022·GigaScience
DOI 10.1093/gigascience/giad028PMC PMC10132306

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Artificial intelligence (AI) programs that train on large datasets require powerful compute infrastructure consisting of several CPU cores and GPUs. JupyterLab provides an excellent framework for developing AI programs, but it needs to be hosted on such an infrastructure to enable faster training of AI programs using parallel computing. Findings: An open-source, docker-based, and GPU-enabled JupyterLab infrastructure is developed that runs on the public compute infrastructure of Galaxy Europe consisting of thousands of CPU cores, many GPUs, and several petabytes of storage to rapidly prototype and develop end-to-end AI projects. Using a JupyterLab notebook, long-running AI model training programs can also be executed remotely to create trained models, represented in open neural network exchange (ONNX) format, and other output datasets in Galaxy. Other features include Git integration for version control, the option of creating and executing pipelines of notebooks, and multiple dashboards and packages for monitoring compute resources and visualization, respectively. Conclusions: These features make JupyterLab in Galaxy Europe highly suitable for creating and managing AI projects. A recent scientific publication that predicts infected regions in COVID-19 computed tomography scan images is reproduced using various features of JupyterLab on Galaxy Europe. In addition, ColabFold, a faster implementation of AlphaFold2, is accessed in JupyterLab to predict the 3-dimensional structure of protein sequences. JupyterLab is accessible in 2 ways—one as an interactive Galaxy tool and the other by running the underlying Docker container. In both ways, long-running training can be executed on Galaxy’s compute infrastructure. Scripts to create the Docker container are available under MIT license at https://github.com/usegalaxy-eu/gpu-jupyterlab-docker.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/gigascience/giad028

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 45

  1. Pearson, 2016, The FASTA package—protein and DNA sequence similarity searching and alignment programs
  2. Kumar, 2022, Bioinformatics, 443, 10.1016/B978-0-323-89775-4.00020-1 · DOI
  3. Kluyver, 2016, Jupyter Notebooks—A Publishing Format for Reproducible Computational Workflows, 87
  4. The Galaxy Community, 2022, The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2022 update, Nucleic Acids Res, 50, W345, 10.1093/nar/gkac247 · DOI
  5. Kumar, 2021, Container for machine learning and deep learning in Jupyter notebook
  6. Galaxy Europe, 2019, Live instance of the European Galaxy server
  7. Kumar, 2021, GPU enabled interactive Jupyter notebook for machine learning
  8. Compute resources in Galaxy Europe, 2023, GitHub
  9. Merkel, 2014, Docker: lightweight linux containers for consistent development and deployment, Linux J, 2014, 2
  10. Pedregosa, 2011, Scikit-learn: machine learning in Python, J Machine Learn Res, 12, 2825
  11. Bai, 2019, ONNX: Open Neural Network Exchange
  12. Harris, 2020, Array programming with NumPy, Nature, 585, 357, 10.1038/s41586-020-2649-2 · DOI · Paper page
  13. Virtanen, 2020, SciPy 1 0 Contributors. SciPy 1.0: fundamental algorithms for scientific computing in Python, Nat Methods, 17, 261, 10.1038/s41592-019-0686-2 · DOI · Paper page
  14. NVIDIA Corporation, 2014, CUDA and cuDNN images from gitlab.com/nvidia/cuda
  15. Kumar, 2021, Jupyter container used for Data Science and Tensorflow
  16. Docker Hub, 2013, Docker
  17. Waskom, 2021, Seaborn: statistical data visualization, J Open Source Softw, 6, 3021, 10.21105/joss.03021 · DOI · Paper page
  18. Hunter, 2007, Matplotlib: a 2D graphics environment, IEEE Comput Soc, 9, 90
  19. Bokeh Development Team, 2018, Bokeh: Python library for interactive visualization
  20. Alekhin, 2012., OpenCV: Open Source Computer Vision Library.
  21. Van der Walt, 2014, scikit-image: image processing in Python, PeerJ, 2, e453, 10.7717/peerj.453 · DOI
  22. Sloggett, 2013, BioBlend: automating pipeline analyses within Galaxy and CloudMan, Bioinformatics, 29, 1685, 10.1093/bioinformatics/btt199 · DOI
  23. Corlay, 2015, 2-D plotting library for Project Jupyter
  24. Tuloup, 2018, Rendering of live Jupyter notebooks with interactive widgets
  25. Tomlinson, 2021, A JupyterLab extension for displaying GPU usage dashboards
  26. Mirdita, 2022, ColabFold: making protein folding accessible to all, Nat Methods, 19, 679, 10.1038/s41592-022-01488-1 · DOI · Paper page
  27. Bisong, 2019, Google Colaboratory., 10.1007/978-1-4842-4470-8_7 · DOI
  28. Amazon SageMaker, 2017, Amazon SageMaker
  29. Boileau, 2020, Exploring high-dimensional biological data with sparse contrastive principal component analysis, Bioinformatics, 36, 3422, 10.1093/bioinformatics/btaa176 · DOI
  30. Batut, 2018, Community-driven data analysis training for biology, Cell Systems, 6, 752, 10.1016/j.cels.2018.05.012 · DOI
  31. Kumar, 2022, A Docker-based interactive Jupyterlab powered by GPU for artificial intelligence in Galaxy (Galaxy Training Materials).
  32. Saeedizadeh, 2021, COVID TV-Unet: segmenting COVID-19 chest CT images using connectivity imposed Unet, Comput Methods Programs Biomed Update, 1, 100007, 10.1016/j.cmpbup.2021.100007 · DOI
  33. Kumar, 2022, Jupyterlab notebooks, GitHub
  34. Kumar, 2022, COVID Image segmentation datasets and trained model, Zenodo.
  35. Saeedizadeh, 2021, COVID TV-Unet: Segmenting COVID-19 chest CT images using connectivity imposed Unet, GitHub
  36. Steinegger, 2017, MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets, Nat Biotechnol, 35, 1026, 10.1038/nbt.3988 · DOI · Paper page
  37. Chen, 1992, 4 Oxalocrotonate tautomerase, an enzyme composed of 62 amino acid residues per monomer, J Biol Chem, 267, 17716, 10.1016/S0021-9258(19)37101-7 · DOI
  38. Mirdita, 2022, ColabFold: making protein folding accessible to all, GitHub
  39. Kumar, 2022, Run long-running jupyterlab script, Github.
  40. Kumar, 2021, Custom jupyterlab notebook function to start model training job in Galaxy, Github
  41. Kumar, 2022, Remotely trained image segmentation model, Galaxy
  42. 2016, Galaxy’s Interactive Environments
  43. Kubeflow, 2017, GitHub
  44. Apache Airflow, 2019, GitHub
  45. Kumar A, Cuccuru G, Gruening B, et al. Supporting data for “An Accessible Infrastructure for Artificial Intelligence Using a Docker-Based JupyterLab in Galaxy.” GigaScience Database. 2023. 10.5524/102381. · DOI · Paper page

Metadata

Authors
Kumar, A., Cuccuru, G., Grüning, B., & Backofen, R.
Affiliations
University of Freiburg (DE)
Corresponding author
Kumar, Anup
ORCID
Venue
GigaScience
Publisher
Oxford University Press (OUP)
Published
2022-12-28
Volume
12
ISSN
2047-217X (ISSN-L)
DOI
10.1093/gigascience/giad028
PMC
PMC10132306
OpenAlex
W4367048100
Cited by
6
References
3
Citations by year
2025: 5 · 2022: 1
FWCI
1.25
Type
journal article
Language
en
Open access
Licence
Locations
  • GigaScience (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • DFG
  • Bundesministerium für Bildung und Frauen (031A538A)
Topics
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
  • Physical Sciences › Computer Science › Computer Networks and Communications › Distributed and Parallel Computing Systems
  • Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
Keywords
JupyterLab, Docker containers, GPU computing, AI training, ONNX, Galaxy tool, ColabFold, protein structure prediction
MeSH terms
COVID-19, Amino Acid Sequence, Artificial Intelligence, Humans, Software, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review