BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

ALiBi has been adopted by BLOOM for achieving strong extrapolation capabilities.

“For example, ALiBi has been adopted by BLOOM [108] for achieving strong extrapolation capabilities.”

Systems like AllenAI's OLMo Instruct, BloomZ and LLM360's AmberChat are approaching full openness status and top the openness leaderboard.

“One is the broad open source approach seen in systems like AllenAI's OLMo Instruct [20], BloomZ [61] and LLM360's AmberChat [33], which are approaching full openness status and top the openness leaderboard.”

BloomZ's multiple preprints document data curation and finetuning in great detail; in contrast, Llama's single preprint offers fewer details and appears strategically vague on crucial details such as training datasets and instruction tuning.

“BloomZ's multiple preprints document data curation and finetuning [50, 61] in great detail; in contrast, Llama's single preprint offers fewer details and appears strategically vague on crucial details (for instance, training datasets and instruction tuning).”

True openness of this kind, and not just open weight models, can speed up innovation and afford inclusion and diversity.

“It is true openness of this kind, and not just open weight models, that can speed up innovation and afford inclusion and diversity [20, 61].”

BLOOM-176B was trained on 1,161,261 A100-hours at a throughput of 150 TFLOP/GPU/s at 48% model FLOPs utilization and a cost of $3 million (including experiments).

“BLOOM-176B was trained on 1,161,261 A100-hours at a throughput of 150 TFLOP/GPU/s at 48% model FLOPs utilization and a cost of $3 million (including experiments) [25].”

Training costs and compute resources are independently known for BLOOM-176B and OPT-175B.

“Training costs and compute resources are independently known for BLOOM-176B and OPT-175B, so we compare our estimates with these.”

Reciprocal and share-alike licensing, such as the Responsible AI License under which BLOOM was released, is the architectural attempt to convert appropriation into provision, and its enforceability is among the open problems.

“Reciprocal and share-alike licensing, the Responsible AI License under which BLOOM was released (BigScience Workshop et al., 2022) being one instance, is the architectural attempt to convert appropriation into provision, and its enforceability is among the open problems of Section 14.”

FP16 can compromise computational accuracy.

“However, research has demonstrated that FP16 can compromise computational accuracy [108].”

Open generative AI systems are on the rise, with small players and academic initiatives leading the way in open innovation and scientific documentation and several larger corporations joining the fray by releasing models billed as open.

“Open generative AI systems are on the rise, with small players and academic initiatives leading the way in open innovation and scientific documentation [20, 32, 61] and several larger corporations joining the fray by releasing models billed as 'open'.”

The LLM data used to train the base model is documented in great detail by Bloom, while for Llama only the vaguest details are provided in a corporate preprint.

“The LLM data that was used to train the base model is documented in great detail by Bloom [61], while for Llama only the vaguest details are provided in a corporate preprint: 'a new mix of data from publicly available sources, which does not include data from Meta's products or services' [57].”

An average of 1 to 2 failures per week occurred when training the BLOOM model on a cluster of 384 NVIDIA A100 GPUs.

“an average of 1 to 2 failures per week occurred when training the BLOOM model on a cluster of 384 NVIDIA A100 GPUs [25].”

The model layer acquires its commons exemplar in the openly built BLOOM (BigScience Workshop et al., 2022), its accountability machinery in the Foundation Model Transparency Index (Bommasani et al., 2023), and its critical self-correction in the openwashing diagnosis of Widder et al. (2024) and the Open Source AI Definition (Open Source Initiative, 2024).

“The model layer acquires its commons exemplar in the openly built BLOOM (BigScience Workshop et al., 2022), its accountability machinery in the Foundation Model Transparency Index (Bommasani et al., 2023), and its critical self-correction in the openwashing diagnosis of Widder et al. (2024) and the Open Source AI Definition (Open Source Initiative, 2024).”

BLOOM is a 176-billion-parameter multilingual model built by an open scientific collaboration on a publicly granted national supercomputer and released under a Responsible AI License.

“The collaborative exemplar of the layer is the BigScience workshop's BLOOM, a 176-billion-parameter multilingual model built by an open scientific collaboration on a publicly granted national supercomputer and released under a Responsible AI License (BigScience Workshop et al., 2022).”

The open-weight model collaboration of the model layer is, in its governance, an extension of scientific peer production to artifacts that were previously proprietary, as BLOOM demonstrates.

“the open-weight model collaboration of the model layer is, in its governance, an extension of scientific peer production to artifacts that were previously proprietary, as the BigScience workshop's BLOOM demonstrates (BigScience Workshop et al., 2022).”

Many educators and educational institutions may not have the knowledge or expertise to effectively integrate new technologies in their teaching.

“Many educators and educational institutions may not have the knowledge or expertise to effectively integrate new technologies in their teaching [56].”