CoChat
Sign Up
Cited by 15Topic ModelingNatural Language Processing TechniquesSpeech and dialogue systems

BIELIK 7B V0.1: POLISH LANGUAGE MODEL - DEVELOPMENT, INSIGHTS, AND EVALUATION

Open access
Ociepa, Krzysztof, Flis, Łukasz, Wróbel, Krzysztof, Gwoździej, Adrian, Kinas, Remigiusz·2025·Computer Science
DOI 10.7494/csci.2025.26.4.7689
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce Bielik 7B v0.1, a 7-billion-parameter generative text model for Polish language processing. Trained on curated Polish corpora, this model addresses key challenges in language model development through innovative techniques. These include Weighted Instruction Cross-Entropy Loss, which balances the learning of different instruction types, and Adaptive Learning Rate, which dynamically adjusts the learning rate based on training progress. To evaluate performance, we created the Open PL LLM Leaderboard and Polish MT-Bench, novel frameworks assessing various NLP tasks and conversational abilities. Bielik 7B v0.1 demonstrates significant improvements, achieving a 9 percentage point increase in average score compared to Mistral-7B-v0.1 on the RAG Reader task. It also excels in the Polish MT-Bench, particularly in Reasoning (6.15/10) and Role-playing (7.83/10) categories. This model represents a substantial advancement in Polish language AI, offering a powerful tool for diverse linguistic applications and setting new benchmarks in the field.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.7494/csci.2025.26.4.7689

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 26

  1. Beltagy I., Peters M.E., Cohan A.: Longformer: The Long-Document Trans- former, ArXiv preprint, vol. abs/2004.05150, 2020. https://arxiv.org/abs/ 2004.05150. · arXiv
  2. Chen L., Li S., Yan J., Wang H., Gunaratna K., Yadav V., Tang Z., et al.: Alpa- Gasus: Training a Better Alpaca with Fewer Data. In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, 2024. https://openreview.net/forum?id=FdVXgSJhvz.
  3. Child R., Gray S., Radford A., Sutskever I.: Generating Long Sequences with Sparse Transformers, ArXiv preprint, vol. abs/1904.10509, 2019. https: //arxiv.org/abs/1904.10509. · arXiv
  4. Dadas S.: Training Effective Neural Sentence Encoders from Automati- cally Mined Paraphrases. In: 2022 IEEE International Conference on Sys- tems, Man, and Cybernetics (SMC), pp. 371-378, 2022. doi: 10.1109 / SMC53654.2022.9945218. · DOI
  5. Dadas S., Perełkiewicz M., Poświata R.: Evaluation of Sentence Representations in Polish. In: N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 1674-1680, European Language Resources Association, Marseille, France, 2020. https://aclanthology.org/2020.lrec-1.207.
  6. Dao T.: FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. In: The Twelfth International Conference on Learning Represen- tations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, 2024. https://openreview.net/forum?id=mZn2Xyh9Ec.
  7. Dauphin Y.N., Fan A., Auli M., Grangier D.: Language Modeling with Gated Convolutional Networks. In: D. Precup, Y.W. Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, Proceedings of Machine Learning Research, vol. 70, pp. 933-941, PMLR, 2017. http://proceedings.mlr.press/v70/dauphin17a.html.
  8. Frantar E., Ashkboos S., Hoefler T., Alistarh D.: GPTQ: Accurate Post- training Compression for Generative Pretrained Transformers, ArXiv preprint, vol. abs/2210.17323, 2022. https://arxiv.org/abs/2210.17323. · arXiv
  9. Granziol D., Zohren S., Roberts S.J.: Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training, J Mach Learn Res, vol. 23, pp. 173:1-173:65, 2020. https://api.semanticscholar.org/CorpusID: 226281826.
  10. Ibrahim A., Thérien B., Gupta K., Richter M.L., Anthony Q., Lesort T., Belilovsky E., Rish I.: Simple and Scalable Strategies to Continually Pre-train Large Language Models, ArXiv preprint, vol. abs/2403.08763, 2024. https: //arxiv.org/abs/2403.08763. · arXiv
  11. Imamura K., Sumita E.: Extending the Subwording Model of Multilingual Pre- trained Models for New Languages, ArXiv preprint, vol. abs/2211.15965, 2022. https://arxiv.org/abs/2211.15965. · arXiv
  12. Jiang A.Q., Sablayrolles A., Mensch A., Bamford C., Chaplot D.S., de las Casas D., Bressand F., Lengyel G., Lample G., Saulnier L., Lavaud L.R., Lachaux M.A., Stock P., Scao T.L., Lavril T., Wang T., Lacroix T., Sayed W.E.: Mistral 7B, ArXiv preprint, vol. abs/2310.06825, 2023. https://arxiv.org/abs/2310.06825. · arXiv
  13. Jiang Z., Gu J., Zhu H., Pan D.Z.: Pre-RMSNorm and Pre-CRMSNorm Trans- formers: Equivalent and Efficient Pre-LN Transformers. In: A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, S. Levine (eds.), Advances in Neu- ral Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, Decem- ber 10 -16, 2023, 2023. http://papers.nips.cc/paper_files/paper/2023/hash/ 8f1bacee31caf990a4f08d84f0ccb322-Abstract-Conference.html.
  14. Loshchilov I., Hutter F.: Decoupled Weight Decay Regularization. In: 7th Inter- national Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019. https://openreview.net/forum?id= Bkg6RiCqY7.
  15. Marcinczuk M., Ptak M., Radziszewski A., Piasecki M.: Open dataset for de- velopment of Polish Question Answering systems. In: Proceedings of the 6th Language & Technology Conference: Human Language Technologies as a Chal- lenge for Computer Science and Linguistics, Wydawnictwo Poznanskie, Fundacja Uniwersytetu im. Adama Mickiewicza, 2013.
  16. Ogrodniczuk M., Kopeć M.: The Polish Summaries Corpus. In: N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, S. Piperidis (eds.), Proceedings of the Ninth International Confer- ence on Language Resources and Evaluation (LREC'14), pp. 3712-3715, Eu- ropean Language Resources Association (ELRA), Reykjavik, Iceland, 2014. http: //www.lrec-conf.org/proceedings/lrec2014/pdf/1211_Paper.pdf.
  17. Okulska I., Stetsenko D., Kołos A., Karlińska A., Głąbińska K., Nowakowski A.: StyloMetrix: An Open-Source Multilingual Tool for Representing Stylomet- ric Vectors, ArXiv preprint, vol. abs/2309.12810, 2023. https://arxiv.org/abs/ 2309.12810. · arXiv
  18. Ostapenko O., Lesort T., Rodriguez P., Arefin M.R., Douillard A., Rish I., Char- lin L.: Continual Learning with Foundation Models: An Empirical Study of Latent Replay. In: S. Chandar, R. Pascanu, D. Precup (eds.), Proceedings of The 1st Conference on Lifelong Learning Agents, Proceedings of Machine Learning Research, vol. 199, pp. 60-91, PMLR, 2022. https://proceedings.mlr.press/v199/ ostapenko22a.html.
  19. Rybak P., Przybyła P., Ogrodniczuk M.: PolQA: Polish Question Answering Dataset. In: N. Calzolari, M.Y. Kan, V. Hoste, A. Lenci, S. Sakti, N. Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Lin- guistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 12846- 12855, ELRA and ICCL, Torino, Italia, 2024. https://aclanthology.org/2024.lrec- main.1125.
  20. Shazeer N.: GLU Variants Improve Transformer, ArXiv preprint, vol. abs/2002.05202, 2020. https://arxiv.org/abs/2002.05202. · arXiv
  21. Tuora R., Zwierzchowska A., Zawadzka-Paluektau N., Klamra C., Kobyliński Ł.: PoQuAD-The Polish Question Answering Dataset-Description and Analysis. In: Proceedings of the 12th Knowledge Capture Conference 2023, pp. 105-113, 2023. doi: 10.1145/3587259.3627548. · DOI
  22. Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser L., Polosukhin I.: Attention is All you Need. In: I. Guyon, U. von Luxburg, S. Bengio, H.M. Wallach, R. Fergus, S.V.N. Vishwanathan, R. Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017. doi: 10.65215/pc26a033. · DOI
  23. Wang G., Cheng S., Zhan X., Li X., Song S., Liu Y.: OpenChat: Advancing Open-source Language Models with Mixed-Quality Data. In: The Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Aus- tria, May 7-11, 2024, OpenReview.net, 2024. https://openreview.net/forum?id= AOJyfhWYHf.
  24. Xu H., Zhan X., Yin H., Qin H.: Discriminator-Weighted Offline Imitation Learn- ing from Suboptimal Demonstrations. In: K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, S. Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, Proceed- ings of Machine Learning Research, vol. 162, pp. 24725-24742, PMLR, 2022. https://proceedings.mlr.press/v162/xu22l.html.
  25. Zheng L., Chiang W., Sheng Y., Zhuang S., Wu Z., Zhuang Y., Lin Z., et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In: A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, S. Levine (eds.), Advances in Neu- ral Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, De- cember 10-16, 2023, 2023. http://papers.nips.cc/paper_files/paper/2023/hash/ 91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html.
  26. Zhou C., Liu P., Xu P., Iyer S., Sun J., Mao Y., Ma X., et al.: LIMA: Less Is More for Alignment. In: A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, S. Levine (eds.), Advances in Neural Information Processing Systems 36: An- nual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10-16, 2023, 2023. http://papers.nips.cc/ paper_files / paper / 2023 / hash / ac662d74829e4407ce1d126477f4a03a -Abstract - Conference.html.

Cited by

Sorted by citations

Related

Authors
Ociepa, K., Flis, Ł., Wróbel, K., Gwoździej, A., & Kinas, R.
Affiliations
  • Akademickie Centrum Komputerowe Cyfronet AGH (PL)
  • Jagiellonian University (PL)
ORCID
Venue
Computer Science
Publisher
AGH University Press
Published
2025-12-28
Volume
26
Issue
4
ISSN
  • 1508-2806 (ISSN-L)
  • 2300-7036
DOI
10.7494/csci.2025.26.4.7689
OpenAlex
W4404306905
Cited by
7
References
0
Citations by year
2026: 7
FWCI
2.81
Type
journal article
Language
en
Open access
Licence
Locations
  • Computer Science (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Jagiellonian University Repository (Jagiellonian University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (cc-by)
Funders
  • Infrastruktura PL-Grid
  • Academic Computer Centre Cyfronet, AGH University of Science and Technology
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
adaptive learning rate, conversational AI, low-resource language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review