CoChat

Automatic Classification of Learning Objectives Based on Bloom's Taxonomy

Open access
Yuheng Li, Rakovic, Mladen, Boon Xin Poh, Gasevic, Dragan, Guanliang Chen · 2022 · Zenodo

The DOI matches its DataCite record; 4 of the 4 references it cites were found in a registry or an index; none of the 51 sources its claims cite is checked for support yet.

Abstract

Learning objectives, especially those well defined by applying Bloom's taxonomy for Cognitive Objectives, have been widely recognized as important in various teaching and learning practices. However, many educators have difficulties developing learning objectives appropriate to the levels in Bloom's taxonomy, as they need to consider the progression of learners' skills with learning content as well as dependencies between different learning objectives. To remedy this challenge, we aimed to apply state-of-the-art computational techniques to automate the classification of learning objectives based on Bloom's taxonomy. Specifically, we collected 21,380 learning objectives from 5,558 different courses at an Australian university and manually labeled them according to the six cognitive levels of Bloom's taxonomy. Based on the labeled dataset, we applied five conventional machine learning approaches (i.e., naive Bayes, logistic regression, support vector machine, random forest, and XGBoost) and one deep learning approach based on pre-trained language model BERT to construct classifiers to automatically determine a learning objective's cognitive levels. In particular, we adopted and compared two methods in constructing the classifiers, i.e., constructing multiple binary classifiers (one for each cognitive level in Bloom's taxonomy) and constructing only one multi-class multi-label classifier to simultaneously identify all the corresponding cognitive levels. Through extensive evaluations, we demonstrated that: (i) BERT-based classifiers outperformed the others in all cognitive levels (Cohen's Kappa up to 0.93 and F1 score up to 0.95); (ii) three machine learning models – support vector machine, random forest, and XGBoost — delivered performance comparable to the BERT-based classifiers; and (iii) most of the binary BERT-based classifiers (5 out of 6) slightly outperformed the multi-class multi-label BERT-based classifier, suggesting that separating the characterization of different cognitive levels seemed to be a better choice than building only one model to identify all cognitive levels at one time.

Overview

CoChat Summary

## TL;DR

The paper builds machine learning and BERT-based classifiers to automatically assign Bloom’s taxonomy cognitive levels to learning objectives using 21,380 objectives from 5,558 courses at an Australian university. It finds that BERT, SVM, random forest, and XGBoost perform strongly, with the best models reaching Cohen’s κ up to 0.93 and F1 up to 0.95.

## Key findings

  • The dataset contained 21,380 learning objectives from 5,558 courses across 10 faculties; some objectives had multiple labels, including 2,325 with two cognitive levels, 280 with three, and 2 with four.
  • High-performing models included SVM, random forest, XGBoost, and BERT; across these models, Cohen’s κ ranged from 0.79 to 0.93, accuracy from 0.92 to 0.99, and F1 from 0.83 to 0.95.
  • The paper reports that binary BERT models outperformed all binary machine learning models and achieved Cohen’s κ of 0.87 to 0.93, accuracy of 0.96 to 0.99, and F1 of 0.88 to 0.95.
  • In the comparison of binary vs. multi-class multi-label approaches, binary classifiers generally performed better; for example, the paper notes binary BERT outperformed multi-class multi-label BERT in most cases, with evaluate as the main exception.

## Methods

  • The study used 21,380 publicly available learning objectives scraped from course web pages at an Australian university in 2021, then manually labeled them with Bloom’s taxonomy cognitive levels by a trained human coder; a second coder independently checked 30% of the labels.
  • The authors trained six model families: five traditional classifiers (SVM, logistic regression, naive Bayes, random forest, XGBoost) and one deep learning approach (BERT), implemented both as binary classifiers and as multi-class multi-label classifiers where appropriate.
  • Data were split 80:20 into training and test sets; traditional models used 3-fold cross-validation with grid search for tuning, and evaluation used accuracy, Cohen’s κ, AUC, and F1.

## Limitations

  • The paper states that some learning objectives might belong to other domains, such as the affective domain, rather than Bloom’s cognitive domain; it suggests future datasets should be labeled for other domains as well.
  • The paper states that the supervised machine learning and deep learning methods require extensive labeled data, and that preparing such a dataset is costly and time-consuming.
  • Evident from the text: the data come from one Australian university and were labeled by coders trained on Bloom’s taxonomy, so generalizability to other institutions or labeling practices is not tested here.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.5281/zenodo.6853191

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 2 Not valid · 34 Not checked · 40 Own results and interpretations

33 more claims

Create a free account to see all 76 claims and to verify the 51 references still open.

See all claims — free account
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • It may state two separate claims as one. 12
  • Probably not a substantial claim. 32

Create a free account to see these claims.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 29

  1. [2][2] H. L. Andrade. A critical review of research on student self-assessment. In Frontiers in Education, volume 4, page 87. Frontiers, 2019. -
  2. [4][4] B. S. Bloom et al. Taxonomy of educational objectives. vol. 1: Cognitive domain. New York: McKay, 20(24):1, 1956. -
  3. [5][5] W.-C. Chang and M.-S. Chung. Automatic applying bloom's taxonomy to classify and analysis the cognition level of english question items. In 2009 Joint Conferences on Pervasive Computing (JCPC), pages 727–734, 2009. -
  4. [6][6] G. Chen, J. Yang, C. Hauff, and G.-J. Houben. Learningq: a large-scale dataset for educational question generation. In Twelfth International AAAI Conference on Web and Social Media, 2018. -
  5. [7][7] J. Chen, J. Feng, X. Sun, and Y. Liu. Co-training semi-supervised deep learning for sentiment classification of mooc forum posts. Symmetry, 12(1):8, 2020. -
  6. [8][8] T. Chen and C. Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pages 785–794, New York, NY, USA, 2016. ACM. -
  7. [11][11] V. Echeverría, J. C. Gomez, and M.-F. Moens. Automatic labeling of forums using bloom's taxonomy. In International Conference on Advanced Data Mining and Applications, pages 517–528. Springer, 2013. -
  8. [13][13] R. Gluga, J. Kay, R. Lister, Simon, and S. Kleitman. Mastering cognitive development theory in computer science education. Computer Science Education, 23(1):24–57, 2013. -
  9. [14][14] S. X. Guo, X. Sun, S. X. Wang, Y. Gao, and J. Feng. Attention-based character-word hybrid neural networks with semantic and structural information for identifying of urgent posts in mooc discussion forums. IEEE Access, 7:120522–120532, 2019. -
  10. [15][15] K. Jayakodi, M. Bandara, I. Perera, and D. Meedeniya. Wordnet and cosine similarity based classifier of exam questions using bloom's taxonomy. International Journal of Emerging Technologies in Learning, 11(4), 2016. -
  11. [18][18] M. Laddha, V. Lokare, A. Kiwelekar, and L. Netak. Classifications of the summative assessment for revised bloom's taxonomy by using deep learning. International Journal of Engineering Trends and Technology, 69:211–218, 03 2021. -
  12. [19][19] M. B. Larson and B. B. Lockee. Streamlined ID: A practical guide to instructional design. Routledge, 2019. -
  13. [20][20] C. Leistner, A. Saffari, J. Santner, and H. Bischof. Semi-supervised random forests. In 2009 IEEE 12th international conference on computer vision, pages 506–513. IEEE, 2009. -
  14. [21][21] S. Masapanta-Carrión and J. Á. Velázquez-Iturbide. A systematic review of the use of bloom's taxonomy in computer science education. In Proceedings of the 49th acm technical symposium on computer science education, pages 441–446, 2018. -
  15. [22][22] M. L. McHugh. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276–282, 2012. -
  16. [23][23] A. Miaschi and F. Dell'Orletta. Contextual and non-contextual word embeddings: an in-depth linguistic investigation. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 110–119, Online, July 2020. Association for Computational Linguistics. -
  17. [24][24] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119, 2013. -
  18. [25][25] M. Mohammed and N. Omar. Question classification based on bloom's taxonomy cognitive domain using modified tf-idf and word2vec. PloS one, 15(3):e0230442, 2020. -
  19. [26][26] R. W. Morshead. Taxonomy of educational objectives handbook ii: Affective domain. Studies in Philosophy and Education, 4(1):164–170, 1965. -
  20. [27][27] V. Nikolovski, D. Kitanovski, D. Trajanov, and I. Chorbev. Case study: Predicting students objectivity in self-evaluation responses using bert single-label and multi-label fine-tuned deep-learning models. In V. Dimitrova and I. Dimitrovski, editors, ICT Innovations 2020. Machine Learning and Applications, pages 98–110, Cham, 2020. Springer International Publishing. -
  21. [28][28] A. Osadi, N. Fernando, and V. Welgama. Ensemble classifier based approach for classification of examination questions into bloom's taxonomy cognitive levels. International Journal of Computer Applications, 162:975–8887, 04 2017. -
  22. [29][29] A. Osman and A. A. Yahya. Classifications of exam questions using natural language syntatic features: A case study based on bloom's taxonomy. In Proc. 6th Int. Arab Conf. Quality Assurance Higher Edu., 2016. -
  23. [30][30] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. -
  24. [32][32] C. Sammut and G. I. Webb. Encyclopedia of machine learning. Springer Science & Business Media, 2011. -
  25. [36][36] S. Shaikh, S. M. Daudpotta, and A. S. Imran. Bloom's learning outcomes' automatic classification using lstm and pretrained word embeddings. IEEE Access, 9:117887–117909, 2021. -
  26. [37][37] S. Sharma and D. Mehrotra. Comparative analysis of multi-label classification algorithms. In 2018 First International Conference on Secure Cyber Computing and Communication (ICSCCC), pages 35–38, 2018. -
  27. [39][39] T. Tang, X. Tang, and T. Yuan. Fine-tuning bert for multi-label sentiment analysis in unbalanced code-switching text. IEEE Access, 8:193248–193256, 2020. -
  28. [42][42] R. Yarullin and P. Serdyukov. Bert for sequence-to-sequence multi-label text classification. In W. M. P. van der Aalst, V. Batagelj, D. I. Ignatov, M. Khachay, O. Koltsova, A. Kutuzov, S. O. Kuznetsov, I. A. Lomazova, N. Loukachevitch, A. Napoli, A. Panchenko, P. M. Pardalos, M. Pelillo, A. V. Savchenko, and E. Tutubalina, editors, Analysis of Images, Social Networks and Texts, pages 187–198, Cham, 2021. Springer International Publishing. -
  29. [44][44] J. Zhang, C. Wong, N. Giacaman, and A. Luxton-Reilly. Automated classification of computing education questions using bloom's taxonomy. In Australasian Computing Education Conference, ACE '21, page 58–65, New York, NY, USA, 2021. Association for Computing Machinery.

Cited by

Metadata

Authors
Li, Y., Rakovic, M., Poh, B. X., Gasevic, D., & Chen, G.
Venue
Zenodo
Publisher
Zenodo
Published
2022-07-18
OpenAlex
W4285778212
Cited by
22
References
0
Citations by year
2026: 3 · 2025: 8 · 2024: 6 · 2023: 5
FWCI
37.42
Type
conference paper
Language
en
Open access
Locations
  • Monash University Research Portal (Monash University) (submittedVersion, other-oa)
  • HAL (submittedVersion)
  • Zenodo (CERN European Organization for Nuclear Research) (cc-by)
Topics
  • Social Sciences › Education › Educational Assessment and Pedagogy
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
Keywords
learning objectives, Bloom's taxonomy, cognitive levels, automatic classification, BERT, multi-label classification, binary classifiers, course syllabus analysis, educational data mining, machine learning classifier comparison

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.