CoChat
Sign Up
Cited by 23Topic ModelingComputational and Text Analysis MethodsAdvanced Text Analysis Techniques

Quality indices for topic model selection and evaluation: a literature review and case study

Open access
Meaney, Christopher, Stukel, Therese A., Austin, Peter C., Moineddin, Rahim, Greiver, Michelle, Escobar, Michael·2023·BMC Medical Informatics and Decision Making
DOI 10.1186/s12911-023-02216-1PMC PMC10362613
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Topic models are a class of unsupervised machine learning models, which facilitate summarization, browsing and retrieval from large unstructured document collections. This study reviews several methods for assessing the quality of unsupervised topic models estimated using non-negative matrix factorization. Techniques for topic model validation have been developed across disparate fields. We synthesize this literature, discuss the advantages and disadvantages of different techniques for topic model validation, and illustrate their usefulness for guiding model selection on a large clinical text corpus. Design, setting and data: Using a retrospective cohort design, we curated a text corpus containing 382,666 clinical notes collected between 01/01/2017 through 12/31/2020 from primary care electronic medical records in Toronto Canada. Methods: Several topic model quality metrics have been proposed to assess different aspects of model fit. We explored the following metrics: reconstruction error, topic coherence, rank biased overlap, Kendall’s weighted tau, partition coefficient, partition entropy and the Xie-Beni statistic. Depending on context, cross-validation and/or bootstrap stability analysis were used to estimate these metrics on our corpus. Results: Cross-validated reconstruction error favored large topic models (K ≥ 100 topics) on our corpus. Stability analysis using topic coherence and the Xie-Beni statistic also favored large models (K = 100 topics). Rank biased overlap and Kendall’s weighted tau favored small models (K = 5 topics). Few model evaluation metrics suggested mid-sized topic models (25 ≤ K ≤ 75) as being optimal. However, human judgement suggested that mid-sized topic models produced expressive low-dimensional summarizations of the corpus. Conclusions: Topic model quality indices are transparent quantitative tools for guiding model selection and evaluation. Our empirical illustration demonstrated that different topic model quality indices favor models of different complexity; and may not select models aligning with human judgment. This suggests that different metrics capture different aspects of model goodness of fit. A combination of topic model quality indices, coupled with human validation, may be useful in appraising unsupervised topic models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12911-023-02216-1

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 43

  1. [1]Gentzkow M Kelly B Taddy M Text as Data Journal of Economic Literature 2019 57 535 574 10.1257/jel.20181020 · DOI
  2. [2]Deerwester S Dumais S Furnas G Indexing by Latent Semantic Analysis Journal of the American Society for Information Science 1990 41 391 408 10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9 · DOI
  3. [3]Berry M Dumais S O’Brien G Using Linear Algebra for Intelligent Information Retrieval SIAM Rev 1995 37 573 595 10.1137/1037127 · DOI
  4. [4]Landauer T Dumais S A Solution to Plato’s Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge Psychological Reviews 1997 104 211 240 10.1037/0033-295X.104.2.211 · DOI · Paper page
  5. [5]Lee D Seung S Learning the Parts of an Object by Non-Negative Matrix Factorization Nature 1999 401 788 791 10.1038/44565 10548103 · DOI
  6. [6]Lee D, Seung HS. Algorithms for non-negative matrix factorization. Advances in neural information processing systems. 2000;13.
  7. [7]Xu W, Liu X, Gong Y. Document clustering based on non-negative matrix factorization. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval; 2003. pp. 267–73.
  8. [8]Hofmann T. Probabilistic latent semantic indexing. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval; 1999. pp. 50–7.
  9. [11]Griffiths, T, Steyvers, M. Probabilistic Topic Models. In Handbook of Latent Semantic Analysis. Chapter 21. 2007.
  10. [12]Blei D Probabilistic Topic Models Commun ACM 2012 55 77 84 10.1145/2133806.2133826 · DOI · Paper page
  11. [13]Udell M Horn C Zadeh R Generalized Low Rank Models Foundations and Trends in Machine Learning 2016 9 1 118 10.1561/2200000055 · DOI
  12. [14]Paatero P Tapper U Positive Matrix Factorization: A Non-Negative Factor Model with Optimal Utilization of Error Estimates of Data Values Environmetrics 1994 5 111 126 10.1002/env.3170050203 · DOI
  13. [15]Berry M Browne M Langville A Algorithms and Applications for Approximate Non-Negative Matrix Factorization Comput Stat Data Anal 2007 52 155 173 10.1016/j.csda.2006.11.006 · DOI
  14. [16]Chang J, Gerrish S, Wang C, Boyd-Graber J, Blei D. Reading tea leaves: How humans interpret topic models. Advances in neural information processing systems. 2009;22.
  15. [17]Doogan C, Buntine W. Topic model or topic twaddle? re-evaluating semantic interpretability measures. In Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies; 2021. pp. 3824–48.
  16. [19]Palacio-Nino, J, Berzal, F. Evaluation Metrics for Unsupervised Learning Algorithms. Arxiv. 2019; 1–9. URL: https://arxiv.org/pdf/1905.05667.pdf · arXiv
  17. [20]Matthews P. Human-In-The-Loop Topic Modelling: Assessing topic labelling and genre-topic relations with a movie plot summary corpus. In The Human Position in an Artificial World: Creativity, Ethics and AI in Knowledge Organization. Ergon-Verlag; 2019. pp. 181–207.
  18. [21]Krippendorff K Content Analysis: An Introduction to its Methodology 2008 2 Thousand Oaks, California Sage Publications
  19. [22]Hastie T, Tibshirani R, Friedman JH, Friedman JH. The elements of statistical learning: data mining, inference, and prediction. New York: Springer; 2009. Vol. 2, pp. 1–758.
  20. [23]Wold S Cross-Validatory Estimation of the Number of Components in Factor Analysis and Principal Component Analysis Technometrics 1978 20 387 405 10.1080/00401706.1978.10489693 · DOI
  21. [24]Owen A Perry P Bi-Cross Validation of the SVD and Non-Negative Matrix Factorization Annals of Applied Statistics 2009 3 564 594 10.1214/08-AOAS227 · DOI
  22. [25]Bro R Khejdahl K Smilde A Cross-Validation of Component Model: A Critical Look at Current Methods Annals of Bioanalytic Chemistry 2008 390 1241 1251 10.1007/s00216-007-1790-1 18214448 · DOI
  23. [26]Greene D, O’Callaghan D, Cunningham P. How many topics? stability analysis for topic models. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part I 14. Springer Berlin Heidelberg; 2014. pp. 498–513.
  24. [27]Lange T Roth V Braun M Stability Based Validation of Clustering Solutions Neural Comput 2004 16 1299 1323 10.1162/089976604773717621 15130251 · DOI
  25. [28]AlSumait L, Barbará D, Gentle J, Domeniconi C. Topic significance ranking of LDA generative models. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2009, Bled, Slovenia, September 7-11, 2009, Proceedings, Part I 20. Springer Berlin Heidelberg; 2009. pp. 67–82.
  26. [29]Newman D, Lau JH, Grieser K, Baldwin T. Automatic evaluation of topic coherence. In Human language technologies: The 2010 annual conference of the North American chapter of the association for computational linguistics; 2010. pp. 100–8.
  27. [30]Mimno D, Wallach H, Talley E, Leenders M, McCallum A. Optimizing semantic coherence in topic models. In Proceedings of the 2011 conference on empirical methods in natural language processing; 2011. pp. 262–72.
  28. [31]Röder M, Both A, Hinneburg A. Exploring the space of topic coherence measures. In Proceedings of the eighth ACM international conference on Web search and data mining; 2015. pp. 399–408.
  29. [32]Fagin R Kumar R Sivakumar D Comparing Top-K Lists SIAM Journal of Discrete Mathematics 2003 17 134 160 10.1137/S0895480102412856 · DOI
  30. [33]Webber W Moffat A Zobel J A Similarity Measure for Indefinite Rankings ACM Transactions on Information Systems 2010 28 1 34 10.1145/1852102.1852106 · DOI
  31. [34]Hurley J Cattell R Producing Direct Rotation to Test a Hypothesized Factor Solution Behavioural Science 1962 7 258 262 10.1002/bs.3830070216 · DOI
  32. [35]Kuhn H The Hungarian Method for the Assignment Problem Naval Research Logistics Quarterly 1955 2 83 97 10.1002/nav.3800020109 · DOI · Paper page
  33. [36]Kendall M A New Measure of Rank Correlation Biometrika 1938 30 81 89 10.1093/biomet/30.1-2.81 · DOI
  34. [37]Airoldi E Blei D Erosheva E Handbook of Mixed Membership Models and Their Applications 2015 Boca Raton, Florida Chapman and Hall Press
  35. [38]Wang W Zhang Y On Fuzzy Clustering Validity Indices Fuzzy Sets Syst 2007 158 2095 2117 10.1016/j.fss.2007.03.004 · DOI
  36. [39]Bezdek J Cluster Validity with Fuzzy Sets Cybernetics 1974 3 58 73 10.1080/01969727308546047 · DOI
  37. [40]Dave R Validating Fuzzy Partitions Obtained Through C-Shells Partitions Pattern Recogn Lett 1996 17 613 623 10.1016/0167-8655(96)00026-8 · DOI
  38. [41]Xie X Beni G A Validity Measure for Fuzzy Clustering IEEE Transactions of Pattern Analysis and Machine Learning 1991 13 841 847 10.1109/34.85677 · DOI
  39. [42]Pal N Bezdek J On Cluster Validity of the Fuzzy C-Means Model IEEE Trans Fuzzy Syst 1995 3 370 379 10.1109/91.413225 · DOI
  40. [43]Garies S, Birtwhistle R, Drummond N, Queenan J, Williamson T. Data resource profile: national electronic medical record data from the Canadian Primary Care Sentinel Surveillance Network (CPCSSN). Int J Epidemiol. 2017;46(4):1091–2f. 28338877
  41. [44]Webster JJ, Kit C. Tokenization as the initial phase in NLP. In COLING 1992 volume 4: The 14th international conference on computational linguistics; 1992.
  42. [45]Díaz NPC, López MJM. An analysis of biomedical tokenization: problems and strategies. In Proceedings of the Sixth International Workshop on Health Text Mining and Information Analysis; 2015. pp. 40–9.
  43. [46]Hutter F Kotthoff L Vanschoren J Automated Machine Learning 2019 Springer

Cited by

Sorted by citations

Authors
Meaney, C., Stukel, T. A., Austin, P. C., Moineddin, R., Greiver, M., & Escobar, M.
Affiliations
University of Toronto (CA)
Corresponding author
Meaney, Christopher
ORCID
Venue
BMC Medical Informatics and Decision Making
Publisher
Springer Science and Business Media LLC
Published
2023-07-22
Volume
23
Issue
1
Pages
132-132
ISSN
1472-6947 (ISSN-L)
DOI
10.1186/s12911-023-02216-1
PMC
PMC10362613
OpenAlex
W4385144683
Cited by
23
References
0
Citations by year
2026: 13 · 2025: 6 · 2024: 4
FWCI
2.17
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
CIHR (FDN 143303)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
Keywords
topic modeling, topic model evaluation, model selection, non-negative matrix factorization, clinical text, electronic health records, primary care, reconstruction error, topic coherence, partition coefficient, partition entropy, model complexity, human validation, text summarization
MeSH terms
Algorithms, Canada, Humans, Retrospective Studies, Benchmarking, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review