Toward an architecture for never-ending language learning
Open accessAbstract
We consider here the problem of building a never-ending language learner; that is, an intelligent computer agent that runs forever and that each day must (1) extract, or read, information from the web to populate a growing structured knowledge base, and (2) learn to perform this task better than on the previous day. In particular, we propose an approach and a set of design principles for such an agent, describe a partial implementation of such a system that has already learned to extract a knowledge base containing over 242,000 beliefs with an estimated precision of 74% after running for 67 days, and discuss lessons learned from this preliminary attempt to build a never-ending learning agent.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The NELL data set consists of facts represented by a triplet which involves two entities and their relation.
“It consists of facts represented by a triplet which involves two entities and their relation.”
From A Comprehensive Survey on Graph Neural Networks (2021) · cites this paper as [130]
The NELL data set is a knowledge graph obtained from the Never-Ending Language Learning project.
“The NELL data set is a knowledge graph obtained from the Never-Ending Language Learning project.”
From A Comprehensive Survey on Graph Neural Networks (2021) · cites this paper as [130]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Caruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636 · see citing works
Combining labeled and unlabeled data with co-training
Blum, A. L., & Mitchell, T. M. (1998) · Conference on Learning Theory · doi:10.1145/279943.279962 · Cited by 5,597 · see citing works
Self-citation(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791 · see citing works
Automatic acquisition of hyponyms from large text corpora
Hearst, M. A. (1992) · International Conference on Computational Linguistics · doi:10.3115/992133.992154 · Cited by 3,302 · see citing works
An Integrated Theory of the Mind.
Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C. J., & Qin, Y. (2004) · Psychological Review · doi:10.1037/0033-295x.111.4.1036 · Cited by 3,048 · see citing works
SOAR: An Architecture for General Intelligence
Laird, J. E., Newell, A., & Rosenbloom, P. S. (1987) · Artificial Intelligence · doi:10.1016/0004-3702(87)90050-6 · Cited by 2,535 · see citing works
Unsupervised word sense disambiguation rivaling supervised methods
Yarowsky, D. (1995) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/981658.981684 · Cited by 2,411 · see citing works
The Hearsay-II Speech-Understanding System: Integrating Knowledge to Resolve Uncertainty
Erman, L. D., Hayes-Roth, F., Lesser, V. R., & Reddy, D. R. (1980) · ACM Computing Surveys · doi:10.1145/356810.356816 · Cited by 1,357 · see citing works
Caruana, R. (1998) · Cited by 1,110 · see citing works
Unsupervised Models for Named Entity Classification
Collins, M., & Singer, Y. (1999) · Cited by 819 · see citing works
Learning dictionaries for information extraction by multi-level bootstrapping
Riloff, E., & Jones, R. (1999) · Cited by 675 · see citing works
Quinlan, J. R., & Cameron-Jones, R. M. (1993) · Lecture Notes in Computer Science · doi:10.1007/3-540-56602-3_124 · Cited by 528 · see citing works
Thrun, S., & Mitchell, T. M. (1995) · Robotics and Autonomous Systems · doi:10.1016/0921-8890(95)00004-y · Cited by 523 · see citing works
Self-citationCoupled semi-supervised learning for information extraction
Carlson, J. A., Betteridge, J., Wang, R. C., Junior, E. R. H., & Mitchell, T. M. (2010) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1718487.1718501 · Cited by 420 · see citing works
Self-citationEurisko: A program that learns new heuristics and domain concepts
Lenat, D. B. (1983) · Artificial Intelligence · doi:10.1016/s0004-3702(83)80005-8 · Cited by 269 · see citing works
Carbonell, J., Etzioni, O., Gil, Y., et al. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122353 · Cited by 232 · see citing works
Soar: An Architecture for General Intelligence
Laird, J. E., Newell, A., & Rosenbloom, P. S. (1987) · doi:10.21236/ada205407 · Cited by 219 · see citing works
Guiding Semi-Supervision with Constraint-Driven Learning
Chang, M., Ratinov, L., & Roth, D. (2007) · Cited by 192 · see citing works
A Probabilistic Model of Redundancy in Information Extraction
Downey, D., Etzioni, O., & Soderland, S. (2006) · doi:10.21236/ada454763 · Cited by 178 · see citing works
Active learning by labeling features
Druck, G., Settles, B., & McCallum, A. (2009) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1699510.1699522 · Cited by 164 · see citing works
Self-citationCounter-training in discovery of semantic patterns
Yangarber, R. (2003) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/1075096.1075140 · Cited by 133 · see citing works
Methods for domain-independent information extraction from the web: an experimental comparison
Etzioni, O., Cafarella, M., Downey, D., Popescu, A.-M., Shaked, T., Soderland, S. G., Weld, D. S., & Yates, A. (2004) · Cited by 112 · see citing works
Thrun, S., & Mitchell, T. M. (1995) · Springer eBooks · doi:10.1007/978-3-642-79629-6_7 · Cited by 102 · see citing works
Self-citationThe Hearsay-II Speech-Understanding System: Integrating Knowledge to Resolve Uncertainty
Erman, L. D., Hayes-Roth, F. A., Lesser, V., et al. (1981) · Elsevier eBooks · doi:10.1016/b978-0-934613-03-3.50029-5 · Cited by 87 · see citing works
A Mutually Beneficial Integration of Data Mining and Information Extraction
Nahm, U. Y., & Mooney, R. J. (2000) · Cited by 86 · see citing works
Names and similarities on the web
PAŞCA, M. A., Lin, D., Bigham, J. P., et al. (2006) · doi:10.3115/1220175.1220277 · Cited by 84 · see citing works
Entity extraction via ensemble semantics
Pennacchiotti, M., & Pantel, P. (2009) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1699510.1699542 · Cited by 80 · see citing works
Heterogeneous multitask learning with joint sparsity constraints
Yang, X., Kim, S. Y., & Xing, E. P. (2009) · Cited by 79 · see citing works
Strategies for lifelong knowledge extraction from the web
Banko, M., & Etzioni, O. (2007) · International Conference on Knowledge Capture · doi:10.1145/1298406.1298425 · Cited by 75 · see citing works
A design for the ICARUS architecture
Langley, P., McKusick, K. B., Allen, J. A., et al. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122365 · Cited by 62 · see citing works
Theo: A Framework for Self-Improving Systems
Mitchell, T. M., Allen, J., Chalasani, P., et al. (2014) · Psychology Press eBooks · doi:10.4324/9781315807843-20 · Cited by 54 · see citing works
Self-citationPRODIGY: An Integrated Architecture for Planning and Learning
Carbonell, J. G., Etzioni, O., Gil, Y., et al. (2018) · doi:10.1184/r1/6625442 · Cited by 45 · see citing works
Character-level analysis of semi-structured documents for set expansion
Wang, R. C., & Cohen, W. W. (2009) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1699648.1699697 · Cited by 41 · see citing works
Prodigy: An Integrated Architecture for Planning and Learning
Carbonell, J. G., Knoblock, C. A., & Minton, S. (2014) · Psychology Press eBooks · doi:10.4324/9781315807843-17 · Cited by 38 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Curran, J. R.; Murphy, T.; and Scholz, B. 2007. Minimising semantic drift with mutual exclusion bootstrapping. In Proc. of PACLING .
Cited by
Sorted by citations
A Comprehensive Survey on Graph Neural Networks
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. S. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2978386 · Cited by 11,000 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
A Survey on Knowledge Graphs: Representation, Acquisition, and Applications
Ji, S., Pan, S., Cambria, E., Marttinen, P., & Yu, P. S. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3070843 · Cited by 2,947 · see citing works
Knowledge Graph Embedding: A Survey of Approaches and Applications
Wang, Q., Mao, Z., Wang, B., & Guo, L. (2017) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2017.2754499 · Cited by 2,654 · see citing works
Critical analysis of Big Data challenges and analytical methods
Sivarajah, U., Kamal, M. M., Irani, Z., & Weerakkody, V. (2017) · Journal of Business Research · doi:10.1016/j.jbusres.2016.08.001 · Cited by 2,066 · see citing works
“Aggregating these data evidently goes beyond the abilities of current data integration systems (Carlson et al., 2010).”
BackgroundUnifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376 · see citing works
VQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570 · see citing works
SimplE Embedding for Link Prediction in Knowledge Graphs
Kazemi, S. M., & Poole, D. (2018) · arXiv · doi:10.48550/arxiv.1802.04868 · Cited by 476 · see citing works
“During the past two decades, several knowledge graphs (KGs) containing (perhaps probabilistic) facts about the world have been constructed (Carlson et al., 2010; Fader et al., 2011; Wu et al., 2012; Dong et al., 2014; Shin et al., 2015).”
BackgroundDual Graph Convolutional Networks for Graph-Based Semi-Supervised Classification
Zhuang, C., & Ma, Q. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186116 · Cited by 423 · see citing works
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224 · see citing works
Incorporating Vector Space Similarity in Random Walk Inference over Knowledge Bases
Gardner, M., Talukdar, P., Krishnamurthy, J., & Mitchell, T. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1044 · Cited by 189 · see citing works
Self-citationMeta-interpretive learning of higher-order dyadic datalog: predicate invention revisited
Muggleton, S. H., Lin, D., & Tamaddoni-Nezhad, A. (2015) · Machine Learning · doi:10.1007/s10994-014-5471-y · Cited by 184 · see citing works
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
Hancock, B., Bordes, A., Mazare, P.-E., & Weston, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1358 · Cited by 161 · see citing works
Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension
Yang, A., Wang, Q., Liu, J., Liu, K., Lyu, Y., Wu, H., She, Q., & Li, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1226 · Cited by 155 · see citing works
Kliegr, T., Bahník, Š., & Fürnkranz, J. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103458 · Cited by 139 · see citing works
Structure Learning via Parameter Learning
Wang, W. Y., Mazaitis, K., & Cohen, W. W. (2014) · Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management · doi:10.1145/2661829.2662022 · Cited by 47 · see citing works
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions
Roller, S., Boureau, Y.-L., Weston, J., Bordes, A., Dinan, E., Fan, A., Gunning, D., Ju, D., Li, M., Poff, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.12442 · Cited by 38 · see citing works
A Definition of Continual Reinforcement Learning
Abel, D., Barreto, A., Roy, B. V., Precup, D., Hasselt, H. V., & Singh, S. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2192 · Cited by 36 · see citing works
- Authors
- Carlson, A., Betteridge, J., Kisiel, B., Settles, B., Hruschka, E., & Mitchell, T.
- Affiliations
- Carnegie Mellon University (US)
- ORCID
- Mitchell, Tom
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2010-07-05
- Volume
- 24
- Issue
- 1
- Pages
- 1306-1313
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W1512387364
- Cited by
- 2,000
- References
- 34
- Citations by year
- 2026: 26 · 2025: 60 · 2024: 113 · 2023: 129 · 2022: 116 · 2021: 168 · 2020: 192 · 2019: 153 · 2018: 175 · 2017: 148 · 2016: 149 · 2015: 175 · 2014: 138 · 2013: 120 · 2012: 83
- FWCI
- 57.32
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Funders
- Conselho Nacional de Desenvolvimento Científico e Tecnológico
- Defense Advanced Research Projects Agency (FA8750)
- Topics
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- continual learning, web scraping, structured knowledge base, knowledge base population, design principles
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
