PDF text classification to leverage information extraction from publication reports
Open accessAbstract
OBJECTIVES: Data extraction from original study reports is a time-consuming, error-prone process in systematic review development. Information extraction (IE) systems have the potential to assist humans in the extraction task, however majority of IE systems were not designed to work on Portable Document Format (PDF) document, an important and common extraction source for systematic review. In a PDF document, narrative content is often mixed with publication metadata or semi-structured text, which add challenges to the underlining natural language processing algorithm. Our goal is to categorize PDF texts for strategic use by IE systems. METHODS: We used an open-source tool to extract raw texts from a PDF document and developed a text classification algorithm that follows a multi-pass sieve framework to automatically classify PDF text snippets (for brevity, texts) into TITLE, ABSTRACT, BODYTEXT, SEMISTRUCTURE, and METADATA categories. To validate the algorithm, we developed a gold standard of PDF reports that were included in the development of previous systematic reviews by the Cochrane Collaboration. In a two-step procedure, we evaluated (1) classification performance, and compared it with machine learning classifier, and (2) the effects of the algorithm on an IE system that extracts clinical outcome mentions. RESULTS: The multi-pass sieve algorithm achieved an accuracy of 92.6%, which was 9.7% (p<0.001) higher than the best performing machine learning classifier that used a logistic regression algorithm. F-measure improvements were observed in the classification of TITLE (+15.6%), ABSTRACT (+54.2%), BODYTEXT (+3.7%), SEMISTRUCTURE (+34%), and MEDADATA (+14.2%). In addition, use of the algorithm to filter semi-structured texts and publication metadata improved performance of the outcome extraction system (F-measure +4.1%, p=0.002). It also reduced of number of sentences to be processed by 44.9% (p<0.001), which corresponds to a processing time reduction of 50% (p=0.005). CONCLUSIONS: The rule-based multi-pass sieve framework can be used effectively in categorizing texts extracted from PDF documents. Text classification is an important prerequisite step to leverage information extraction from PDF documents.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Two of the 2016 publications were from the same authors and aimed at the automation of Data Extraction (SLR8).
“Out of these publications, two were from the same authors who aimed at the automation of Data Extraction (SLR8) in the systematic review process [34,35].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · cited 3× · see citing works
“The SR process involves a series of scientifically rigorous steps(8), such as citation searching(9), abstract screening, full-text screening, data extraction, and article appraisal.”
BackgroundMethodThe WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · cited 1× · see citing works
“Several machine learning algorithms that have been implemented and integrated in the data mining software WEKA were used in this study (39).”
MethodEvidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769 · see citing works
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951 · see citing works
Combining Pattern Classifiers: Methods and Algorithms
Bagui, S. (2005) · Technometrics · doi:10.1198/tech.2005.s320 · Cited by 4,364 · see citing works
Yeh, M. (2014) · Journal of Nursing Research · doi:10.1097/jnr.0000000000000065 · Cited by 3,836 · see citing works
The class imbalance problem: A systematic study1
Japkowicz, N., & Stephen, S. (2002) · Intelligent Data Analysis · doi:10.3233/ida-2002-6504 · Cited by 3,295 · see citing works
Incorporating non-local information into information extraction systems by Gibbs sampling
Finkel, J. R., Grenager, T., & Manning, C. D. (2005) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1219840.1219885 · Cited by 3,028 · cited 1× · see citing works
“Second, we used the Stanford's Named Entity Recognition (NER) module(36) to detect PERSON, ORGANIZATION, and LOCATION entities in text.”
MethodKuncheva, L. I. (2004) · Wiley eBooks · doi:10.1002/0471660264 · Cited by 3,007 · see citing works
Eddy, S. R. (1996) · Current Opinion in Structural Biology · doi:10.1016/s0959-440x(96)80056-x · Cited by 1,445 · see citing works
How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-4-200708210-00179 · Cited by 838 · see citing works
Experimental perspectives on learning from imbalanced data
Hulse, J. V., Khoshgoftaar, T. M., & Napolitano, A. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273614 · Cited by 785 · see citing works
The STM Report: An overview of scientific and scholarly journal publishing
Ware, M., & Mabe, M. A. (2015) · Lincoln (University of Nebraska) · Cited by 396 · see citing works
Table extraction using conditional random fields
Pinto, D., McCallum, A., Wei, X., & Croft, W. B. (2003) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/860435.860479 · Cited by 358 · cited 1× · see citing works
“The ML baseline could be improved with sequential ML models such as Hidden Markov Model(52) and Conditional Random Field(53).”
MethodThe Global Evidence Mapping Initiative: Scoping research in broad topic areas
Bragge, P., Clavisi, O., Turner, T., Tavender, E., Collie, A., & Gruen, R. L. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-92 · Cited by 310 · cited 4× · see citing works
“About 23% of SRs have not been updated with new evidence within 2 years after the first publication (6), and many clinical questions are not addressed in existing SRs (7).”
BackgroundInfluentialAutomatic document metadata extraction using support vector machines
Giles, H. H. C. L., Manavoglu, E., Hongyuan Zha, B., Zhang, Z., & Fox, E. A. (2003) · Cited by 291 · see citing works
Medical Image Computing and Computer-Assisted Intervention – MICCAI 2011
Gabor Fichtinger, A., Peters, T., & Martel, A. L. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23626-6 · Cited by 262 · see citing works
ExaCT: automatic extraction of clinical trial characteristics from journal publications
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-56 · Cited by 188 · cited 1× · see citing works
“Natural language processing (NLP) research in the past decade has investigated techniques to extract study characteristics from biomedical publications (10-19).”
BackgroundBioRAT: extracting biological information from full-length papers
Corney, D. P. A., Buxton, B. F., Langdon, W. B., & Jones, D. T. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth386 · Cited by 175 · cited 1× · see citing works
“Natural language processing (NLP) research in the past decade has investigated techniques to extract study characteristics from biomedical publications (10-19).”
BackgroundSTM report : an overview of scientific and scholarly journal publishing
urfistinfo (2009) · OpenEdition (OpenEdition) · doi:10.58079/v4as · Cited by 139 · see citing works
Combining classifiers for robust PICO element detection.
Boudin, F., Nie, J.-Y., Bartlett, J. C., Grad, R., Pluye, P., & Dawes, M. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-29 · Cited by 131 · cited 1× · see citing works
“…information extraction (IE) studies select sources of extraction from MEDLINE ® abstracts, PubMed Central ® (PMC) archives, and journal websites (12, extraction practice requires that the source of extraction be the original full-text study reports (8), and the most common format for full-text…”
BackgroundJones, A. P., Remmington, T., Williamson, P. R., Ashby, D., & Smyth, R. L. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.11.024 · Cited by 114 · cited 1× · see citing works
“The extraction template produced by systematic reviewers were not designed as an NLP gold standard, and might contain human extraction errors (35).”
BackgroundLearning regular expressions for clinical text classification
Bui, D. D. A., & Zeng‐Treitler, Q. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002411 · Cited by 112 · cited 2× · see citing works
“In addition to bag of words features that have been used in designing machine learning models as well as used as baseline approaches (40-43), we also developed additional features that take advantage of external resources (MEDLINE ® metadata, PDFBox, and Stanford's NER) that were used in the…”
MethodSelf-citationPDF-TREX: An Approach for Recognizing and Extracting Tables from PDF Documents
Oro, E., & Ruffolo, M. (2009) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2009.12 · Cited by 95 · see citing works
Soguero-Ruíz, C., Hindberg, K., Rojo‐Álvarez, J. L., Skrøvseth, S. O., Godtliebsen, F., Mortensen, K. E., Revhaug, A., Lindsetmo, R., Augestad, K. M., & Jenssen, R. (2014) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2014.2361688 · Cited by 93 · cited 1× · see citing works
“In addition to bag of words features that have been used in designing machine learning models as well as used as baseline approaches (40-43), we also developed additional features that take advantage of external resources (MEDLINE® metadata, PDFBox, and Stanford's NER) that were used in the multi-pass sieve algorithm.”
MethodConstantin, A., Pettifer, S. R., & Воронков, А. (2013) · ACM Symposium on Document Engineering (DocEng) · doi:10.1145/2494266.2494271 · Cited by 89 · see citing works
Logical Structure Recovery in Scholarly Articles with Rich Document Features
Luong, M.-T., Nguyen, T., & Kan, M. (2010) · International Journal of Digital Library Systems · doi:10.4018/jdls.2010100101 · Cited by 86 · see citing works
Kim, M. S., Cho, K. T., Kwag, H. K., & Kim, J. H. (2004) · Lecture notes in computer science · doi:10.1007/b100557 · Cited by 81 · cited 1× · see citing works
“Commonly used approaches were machine learning (20-23) and rule-based or heuristics (24-29).”
MethodLayout and Content Extraction for PDF Documents
Chao, H., & Fan, J. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-28640-0_20 · Cited by 78 · see citing works
Automated information extraction of key trial design elements from clinical trial publications.
(2008) · PubMed Central · Cited by 54 · see citing works
PICO element detection in medical text without metadata: are first sentences enough?
Huang, K.-C., Chiang, I.-J., Xiao, F., Liao, C.-C., Liu, C. C.-H., & Wong, J.-M. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.07.009 · Cited by 52 · cited 1× · see citing works
“Natural language processing (NLP) research in the past decade has investigated techniques to extract study characteristics from biomedical publications (10-19).”
BackgroundEfficient Extraction of Protein-Protein Interactions from Full-Text Articles
Hakenberg, J., Leaman, R., Vo, N., Jonnalagadda, S., Sullivan, R., Miller, C., Tari, L., Baral, C. R., & Gonzalez‐Hernandez, G. (2010) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2010.51 · Cited by 41 · cited 1× · see citing works
“Natural language processing (NLP) research in the past decade has investigated techniques to extract study characteristics from biomedical publications (10-19).”
BackgroundSelf-citationXu, R., Hirano, Y., Tachibana, R., & Kido, S. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23626-6_23 · Cited by 36 · see citing works
A Hybrid Approach to Sentiment Sentence Classification in Suicide Notes
Sohn, S., Torii, M., Li, D., Wagholikar, K. B., Wu, S., & Liu, H. (2012) · Biomedical Informatics Insights · doi:10.4137/bii.s8961 · Cited by 35 · cited 1× · see citing works
“The approach is an alternative to the machine learning algorithms that are commonly used in text classification studies (43, 46, 47).”
MethodTeamBeam - Meta-Data Extraction from Scientific Literature
Kern, R., Jack, K., & Hristakeva, M. (2012) · D-Lib Magazine · doi:10.1045/july2012-kern · Cited by 30 · see citing works
Jonnalagadda, S., Li, D., Sohn, S., Wu, S. T.-I., Wagholikar, K. B., Torii, M., & Liu, H. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000766 · Cited by 28 · cited 1× · see citing works
“In previous studies, the multi-pass sieve framework has been successfully applied to solve co-reference resolution problem (37, 38).”
MethodSelf-citationBeel, J., Gipp, B., Shaker, A., & Friedrich, N. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15464-5_45 · Cited by 25 · see citing works
Klampfl, S., & Kern, R. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40501-3_15 · Cited by 23 · see citing works
García, M. A. M., Rodríguez, R. P., & Anido, L. (2015) · PeerJ · doi:10.7717/peerj.1279 · Cited by 23 · cited 1× · see citing works
“In addition to bag of words features that have been used in designing machine learning models as well as used as baseline approaches (40-43), we also developed additional features that take advantage of external resources (MEDLINE ® metadata, PDFBox, and Stanford's NER) that were used in the…”
MethodLogical Structure Recovery in Scholarly Articles with Rich Document Features
Luong, M.-T., Nguyen, T., & Kan, M. (2012) · IGI Global eBooks · doi:10.4018/978-1-4666-0900-6.ch014 · Cited by 22 · see citing works
Hsu, W. W. Y., Speier, W. F., & Taira, R. K. (2012) · PubMed Central · Cited by 21 · see citing works
Automatically finding relevant citations for clinical guideline development
Bui, D. D. A., Jonnalagadda, S., & Fiol, G. D. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.09.003 · Cited by 19 · cited 1× · see citing works
“The SR process involves a series of scientifically rigorous steps(8), such as citation searching(9), abstract screening, full-text screening, data extraction, and article appraisal.”
MethodSelf-citationA Comparison of Two Unsupervised Table Recognition Methods from Digital Scientific Articles
Klampfl, S., Jack, K., & Kern, R. (2014) · D-Lib Magazine · doi:10.1045/november14-klampfl · Cited by 17 · see citing works
Automatic Extracting of Patient-related Attributes: Disease, Age, Gender and Race
Zhu, H., Ni, Y., Peng, C., Qiu, Z., & Feng, C. (2012) · Studies in health technology and informatics · doi:10.3233/978-1-61499-101-4-589 · Cited by 17 · see citing works
DETECTION OF PROTEIN CATALYTIC SITES IN THE BIOMEDICAL LITERATURE
Verspoor, K. M., MacKinlay, A., Cohn, J. D., & Wall, M. E. (2012) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789814447973_0042 · Cited by 14 · cited 1× · see citing works
“Natural language processing (NLP) research in the past decade has investigated techniques to extract study characteristics from biomedical publications (10-19).”
BackgroundA Multi-Classifier Based Guideline Sentence Classification System
Song, M. H., Kim, S. H., Park, D. K., & Lee, Y. H. (2011) · Healthcare Informatics Research · doi:10.4258/hir.2011.17.4.224 · Cited by 13 · cited 1× · see citing works
“The approach is an alternative to the machine learning algorithms that are commonly used in text classification studies (43, 46, 47).”
MethodA comparison of layout based bibliographic metadata extraction techniques
Granitzer, M., Hristakeva, M., Knight, R., Jack, K., & Kern, R. (2012) · International Conference on Web Intelligence, Mining and Semantics · doi:10.1145/2254129.2254154 · Cited by 12 · see citing works
Research and Advanced Technology for Digital Libraries
Aalberg, T., Papatheodorou, C., Dobreva, M., Tsakonas, G., & Farrugia, C. J. (2013) · Lecture notes in computer science · doi:10.1007/978-3-319-67008-9 · Cited by 12 · see citing works
AUTOMATIC SUMMARIZATION OF CLINICAL ABSTRACTS FOR EVIDENCE-BASED MEDICINE
Summerscales, R. (2013) · Digital Commons - Andrews University (Andrews University) · Cited by 10 · see citing works
Table recognition evaluation and combination methods
Kboubi, F., Chabi, A. H., & Ahmed, M. B. (2005) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2005.224 · Cited by 10 · see citing works
Research and Advanced Technology for Digital Libraries
Berndt, R., Blümel, I., Clausen, M., Damm, D., Diet, J., Fellner, D. W., Fremerey, C., Klein, R., Krahl, F., Scherer, M., et al. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15464-5 · Cited by 10 · see citing works
Advances in Deep Parsing of Scholarly Paper Content
Schäfer, U., & Kiefer, B. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23160-5_9 · Cited by 9 · cited 1× · see citing works
“PDFBox is a free open-source tool commonly used in NLP solutions for PDF documents (31-33).”
MethodGarcía-Remesal, M., Maojo, V., & Crespo, J. (2010) · Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/iembs.2010.5627316 · Cited by 4 · cited 1× · see citing works
“PDFBox is a free open-source tool commonly used in NLP solutions for PDF documents (31-33).”
Method
Flagged first, then by citations
1 of the 55 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- [8]Higgins JP Green S Cochrane handbook for systematic reviews of interventions 2008 Wiley Online Library
- [24]Constantin A Pettifer S Voronkov A PDFX: fully-automated PDF-to-XML conversion of scientific literature. Proceedings of the 2013 ACM symposium on Document engineering 2013 ACM
- [50]Tong S Koller D Restricted bayes optimal classifiers 2000 AAAI/IAAI
Cited by
Sorted by citations
Automation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432 · see citing works
“Out of these publications, two were from the same authors who aimed at the automation of Data Extraction (SLR8) in the systematic review process [34,35].”
BackgroundA recent overview of the state-of-the-art elements of text classification
Mirończuk, M., & Protasiewicz, J. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.03.058 · Cited by 317 · see citing works
Baviskar, D., Ahirrao, S. N., Potdar, V. M., & Kotecha, K. V. (2021) · IEEE Access · doi:10.1109/access.2021.3072900 · Cited by 175 · see citing works
A tutorial on methodological studies: the what, when, how and why
Mbuagbaw, L. C. E., Lawson, D. O., Puljak, L., Allison, D. B., & Thabane, L. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01107-7 · Cited by 164 · see citing works
Using Electronic Health Records To Generate Phenotypes For Research
Pendergrass, S. A., & Crawford, D. C. (2019) · Current Protocols in Human Genetics · doi:10.1002/cphg.80 · Cited by 120 · see citing works
Bui, D. D. A., Del Fiol, G., Hurdle, J. F., & Jonnalagadda, S. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.10.014 · Cited by 70 · see citing works
“(2) Text Classification & Filtering: we used a text classification algorithm (27) to automatically categorize text snippets into five categories (TITLE, ABSTRACT, BODYTEXT, SEMISTRUCTURE, and METADATA).”
MethodSelf-citationChallenges and Advances in Information Extraction from Scientific Literature: a Review
Hong, Z., Ward, L. T., Chard, K., Blaiszik, B. J., & Foster, I. T. (2021) · JOM · doi:10.1007/s11837-021-04902-9 · Cited by 69 · see citing works
Wang, T., Liu, L., Liu, N., Zhang, H., Zhang, L., & Feng, S. (2020) · Applied Intelligence · doi:10.1007/s10489-020-01680-w · Cited by 64 · see citing works
dos Santos, Á. O., da Silva, E. S., Couto, L. M., Reis, G. V. L., & Belo, V. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104389 · Cited by 65 · see citing works
Budhi, G. S., Chiong, R., Pranata, I., & Hu, Z. (2021) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-020-09464-8 · Cited by 63 · see citing works
Text Classification: How Machine Learning Is Revolutionizing Text Categorization
Allam, H., Makubvure, L., Gyamfi, B. K., Kwadwo, G., & Akinwolere, K. (2025) · Information · doi:10.3390/info16020130 · Cited by 62 · see citing works
Scheibel, B., Mangler, J., & Rinderle‐Ma, S. (2021) · Computers in Industry · doi:10.1016/j.compind.2021.103442 · Cited by 60 · see citing works
Asif, M., Ali, I., Malik, M. S. A., Chaudary, M. H., Tayyaba, S., & Mahmood, M. T. (2019) · IEEE Access · doi:10.1109/access.2019.2903133 · Cited by 34 · see citing works
“’’ The methodology employed to investigate abovementioned research questions includes Information extraction using PDFBox tool [6] and preparation of manual generic structured annotated dataset as a training dataset for classification.”
BackgroundMethodResultInfluentialSustainable innovation in the metaverse: Blockchain's role in new business models
Sze, L. B., Salo, J., & Tan, T. M. (2024) · Digital Business · doi:10.1016/j.digbus.2024.100086 · Cited by 32 · see citing works
Jeon, J., Xu, X., Zhang, Y., Yang, L., & Cai, H. (2021) · Transportation Research Record Journal of the Transportation Research Board · doi:10.1177/03611981211001385 · Cited by 28 · see citing works
“Several researchers have used text parsing to extract clean sentences from various types of documents (e.g., financial prospectuses, patient records, and publication reports) ( 23–25 ).”
BackgroundNews Topic Classification Using Machine Learning Techniques
Sunagar, P., Kanavalli, A., Nayak, S. S., Mahan, S. R., Prasad, S., & Prasad, S. (2021) · Lecture notes in electrical engineering · doi:10.1007/978-981-33-4909-4_35 · Cited by 21 · see citing works
E-Mail Assistant – Automation of E-Mail Handling and Management using Robotic Process Automation
Khare, A., Singh, S., Mishra, R., Prakash, S., & Dixit, P. (2022) · International Conference on Decision Aid Sciences and Applications (DASA) · doi:10.1109/dasa54658.2022.9765017 · Cited by 14 · see citing works
“[16] have addressed the classification of PDF text to extract the information from publications or reports using Information Extraction (IE) systems.”
MethodThe UAB Informatics Institute and 2016 CEGS N-GRID de-identification shared task challenge
Bui, D. D. A., Wyatt, M., & Cimino, J. J. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.05.001 · Cited by 14 · see citing works
Self-citationPredicting rating polarity through automatic classification of review texts
Budhi, G. S., Chiong, R., Pranata, I., & Hu, Z. (2017) · IEEE Conference on Big Data and Analytics (ICBDA) · doi:10.1109/icbdaa.2017.8284101 · Cited by 13 · see citing works
“Variants of the model have been used to solve a range of classification problems [38-40].”
MethodSelf-enhancing GPS-Based Authentication Using Corresponding Address
Thảo, T. P., Irvan, M., Kobayashi, R., Yamaguchi, R. S., & Nakata, T. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-49669-2_19 · Cited by 11 · see citing works
Sousa, M. S. A., Peiris, S., Figueiró, M. F., Haby, M. M., Baraldi, A. C., Revéiz, L., & Souza, J. P. (2026) · Systematic Reviews · doi:10.1186/s13643-025-02842-y · Cited by 11 · see citing works
Conversational Text Extraction with Large Language Models Using Retrieval-Augmented Systems
Roy, S., Goswami, M., Nargund, N., Mohanty, S., & Pattnaik, P. K. (2024) · International Conference on Computational Intelligence and Networks (CINE) · doi:10.1109/cine63708.2024.10881808 · Cited by 10 · see citing works
Mull, H. J., Stolzmann, K. L., Kalver, E., Shin, M. H., Schweizer, M. L., Asundi, A., Mehta, P., Stanislawski, M. A., & Branch‐Elliman, W. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-1031-5 · Cited by 10 · see citing works
Classifying Pastebin Content Through the Generation of PasteCC Labeled Dataset
Riesco, A., Fidalgo, E., Al-Nabki, M. W., Jáñez-Martino, F., & Alegre, E. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-29859-3_39 · Cited by 8 · see citing works
Advanced text documents information retrieval system for search services
Chiranjeevi, H. S., & Manjula Shenoy, K. (2020) · Cogent Engineering · doi:10.1080/23311916.2020.1856467 · Cited by 8 · see citing works
Machine Learning Methods for Systematic Reviews:: A Rapid Scoping Review.
Roth, S. C., & Wermer‐Colan, A. (2023) · PubMed Central · doi:10.32481/djph.2023.11.008 · Cited by 7 · see citing works
“Duc An Bui, et al 41 utilized a PDF text classification tool to see how it helped with the data extraction stage of the systematic review, focusing on PDFBox (https:/PDF box/pdfbox.apache.org/) in combination with an annotation tool called GATE.”
MethodRiedel, M. C., Salo, T., Hays, J. S., Turner, M. D., Sutherland, M. T., Turner, J. A., & Laird, A. R. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00494 · Cited by 7 · see citing works
“, 2013) and we identified one paper (Bui et al., 2016) which attempted to classify text patterns according to which section of an article it appeared in (i.”
BackgroundMachine learning e revisão sistemática de literatura automatizada: uma revisão sistemática
Tsunoda, D. F., Moreira, P. S. D. C., & Guimarães, A. J. R. (2020) · Revista Tecnologia e Sociedade · doi:10.3895/rts.v16n45.12119 · Cited by 7 · see citing works
Enhanced Entity Extraction Using Big Data Mechanics
Dash, A., Pandey, M., & Rautaray, S. S. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-981-13-2673-8_8 · Cited by 6 · see citing works
MexPub: Deep Transfer Learning for Metadata Extraction from German Publications
Boukhers, Z., Beili, N., Hartmann, T., Goswami, P., & Zafar, M. A. (2021) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl52503.2021.00076 · Cited by 6 · see citing works
“in scientific documents using context-based classifiers such as Hidden Markov Models (HMMs) [26] and Conditional Random Fields (CRF) [8], [21] or models like Support Vector Machines (SVMs) [2], [9], [29].”
BackgroundUse of Natural Language Processing to Identify Inappropriate Content in Text
Merayo-Alba, S., Fidalgo, E., González‐Castro, V., Aláiz-Rodríguez, R., & Velasco-Mata, J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-29859-3_22 · Cited by 5 · see citing works
A Text Analytic Approach to Classifying Document Types
Walczak, S. M. (2017) · The Journal of Writing Analytics · doi:10.37514/jwa-j.2017.1.1.06 · Cited by 4 · see citing works
Timely, Efficient, and Living Systematic Reviews: Opportunities in International Development
Snilstveit, B., Stevenson, J., Shemilt, I., Clarke, M. J., Jiménez, E., & Thomas, J. (2018) · doi:10.51744/cip7 · Cited by 4 · see citing works
“, 2017; Noel-Storr et al 2014) Data extraction Text classification (Bui et al., 2016) Automated data extraction (Hsu et al.”
BackgroundMulti-label Text Categorization using Error-correcting Output Coding with Weighted Probability
Balamurugan, V., Vedanarayanan, V., Nisha, A. S. A., Narmadha, R., & Amirthalakshmi, T. M. (2022) · International Journal of Engineering · doi:10.5829/ije.2022.35.08b.08 · Cited by 3 · see citing works
Automatic Detection of the CaRS Framework in Scholarly Writing Using Natural Language Processing
Omotola, O., Nnamoko, N., Lam, C., Korkontzelos, I., Altham, C., & Barrowclough, J. (2025) · Electronics · doi:10.3390/electronics14142799 · Cited by 3 · see citing works
Extracción de información de documentos PDF para su uso en la indización automática de e-books
Leiva, I. G., Fujita, M. S. L., Redígolo, F. M., & Saran, J. F. (2022) · Transinformação · doi:10.1590/2318-0889202234e210069 · Cited by 3 · see citing works
The Effect of Knowledge Graph Schema on Classifying Future Research Suggestions
Alivanistos, D., Bijl, S. V. D., Cochez, M., & van Harmelen, F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65794-8_10 · Cited by 3 · see citing works
A <word, part of speech> embedding model for text classification
Liu, W., Liu, P., Yang, Y., Jing, Y., & Zhu, Z. (2019) · Expert Systems · doi:10.1111/exsy.12460 · Cited by 3 · see citing works
“In many domains, such as opinion mining, information retrieval, event detection, and natural language processing, understanding these data has a great consequence for real applications (Alt ı nel & Ganiz, 2016; Bui, Fiol, & Jonnalagadda, 2016; Cong, Yuan, & Liu, 2013; Vo & Zhang, 2015).”
BackgroundDigital document analytics using logistic regressive and deep transition-based dependency parsing
Rekha, D., Sangeetha, J., & Ramaswamy, V. (2021) · The Journal of Supercomputing · doi:10.1007/s11227-021-03973-4 · Cited by 2 · see citing works
Evaluating Document Analysis with kNN Based Approaches in Judicial Offices of Bangladesh
Islam, M. A., & Haque, M. J. (2018) · International Conference on Computing Methodologies and Communication (ICCMC) · doi:10.1109/iccmc.2018.8487847 · Cited by 2 · see citing works
“In [11] noun phrase chunking with filtering process has been implemented to analyze the text data and we also analyzed clause from the sentences with the tokenization process for the terms.”
MethodAn Empirical Method of Automatic Pattern Extraction for Clinical Text Classification
Hussain, M., Hussain, J., Ali, T., & Lee, S. (2020) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc44109.2020.9176503 · Cited by 2 · see citing works
“Text classification is an essential prerequisite of most of the text mining applications [1].”
BackgroundHtet, N. L., Peansupap, V., & Kaewmoracharoen, M. (2025) · International Journal of Construction Management · doi:10.1080/15623599.2025.2566404 · Cited by 1 · see citing works
“PDF documents often feature rich and varied logical structures, including semi-structured elements such as tables, figures, and multi-column layouts, as well as spatial coordinates that define the positioning of text and graphics (Bui et al. 2016).”
MAGNET Architecture Optimization on Multi-Label Text Classification
Abdurrazzaq, M. A., Saptawati, G. A. P., & Rusmawati, Y. (2021) · International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA) · doi:10.1109/icaicta53211.2021.9640263 · Cited by 1 · see citing works
Java Based Software Robot for Automatic E-mail Extraction
Singh, R., Upadhyay, S. K., & Sagar, N. (2023) · International Conference on Intelligent Systems, Advanced Computing and Communication (ISACC) · doi:10.1109/isacc56298.2023.10083905 · Cited by 1 · see citing works
“Information Extraction (IE) techniques have been used by Bui et al. [19] to classify PDF text in order to obtain data from reports and publications.”
MethodNamed Entity Extraction and Classification in Digital Publications
Wu, C., Lee, B. Y., Sun, J., Latt, Y. Y., Shepherd, K., & Watts, J. (2017) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2017-132
“The standard file format for publications is in Portable Document Format (PDF), which is not content-accessible [1].”
BackgroundIndex split decision tree and compositional deep neural network for text categorization
Ravikumar, N., & Selvan, T. (2017) · International Journal of Engineering & Technology · doi:10.14419/ijet.v7i1.1.9953
Automatic Summarization and Keyword Extraction from Multiple Wiki Articles and Books
Angayarkanni, S. P. (2020) · International Journal of Advanced Trends in Computer Science and Engineering · doi:10.30534/ijatcse/2020/160922020
“P+R Where β is a weighting factor that favors precision when β >1 and recall whenβ <1 [3].”
Background(2019) · International Journal of Recent Technology and Engineering (IJRTE) · doi:10.35940/ijrte.c5391.098319
“The rule-based multi-pass sieve technique was introduced for categorizing the texts from PDF documents [4].”
MethodHANDLING BIG TABULAR DATA OF ICT SUPPLY CHAINS: A MULTI-TASK, MACHINE-INTERPRETABLE APPROACH
Kohale, P. P., & Sharma, S. (2023) · ASIAN JOURNAL OF CONVERGENCE IN TECHNOLOGY · doi:10.33130/ajct.2023v09i02.001
Wickramaratne, H., & Ginige, T. N. (2023) · International Conference on Mathematical and Statistical Physics, Computational Science, Education and Communication (ICMSCE) · doi:10.1117/12.3011414
“Similar research explains how most of the existing Information extraction tools and systems are not designed to be used on PDFs even though Portable Document Format is a common source for systematic reviews 5 .”
MethodManoj, P., Nikhil, P., Gnanasekhar, V., Babu, T. S., Soudh, K. S., & Rajesh, J. T. (2025) · International Conference on Evolutionary Computing and Mobile Sustainable Networks (ICECMSN) · doi:10.1109/icecmsn68058.2025.11383057
Malashin, I. P., Masich, I. S., Тынченко, В. С., Бородулин, А. С., Gantimurov, A., Nelyub, V. A., Martysyuk, D., & Kurashkin, S. O. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3795719
“Tran et al. [28] demonstrate invoice-level extraction using graph convolutional networks, while Hansen et al. [11] and Bui et al. [4] develop neural and hybrid pipelines for PDF content classification, enabling downstream applications such as automated content recommendation or decision support.”
- Authors
- Bui, D. D. A., Del Fiol, G., & Jonnalagadda, S.
- Affiliations
- Northwestern University (US)
- Corresponding author
- Bui, Duy Duc An
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2016-06
- Volume
- 61
- Pages
- 141-148
- ISSN
- 1532-0464 (ISSN-L)
- PMC
- PMC4893911
- OpenAlex
- W2316779280
- Cited by
- 52
- References
- 55
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 3 · 2023: 5 · 2022: 3 · 2021: 10 · 2020: 8 · 2019: 6 · 2018: 5 · 2017: 5 · 2016: 1
- FWCI
- 4.67
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Biomedical Informatics (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Library of Medicine (R00LM011389, R01LM011416-01)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- information extraction, systematic reviews, semi-structured text, bibliographic metadata, PDF text extraction
- MeSH terms
- Machine Learning, Algorithms, Humans, Natural Language Processing, Publications, Review Literature as Topic, Information Storage and Retrieval, Narration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
