Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 6

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Virk: an active learning-based system for bootstrapping knowledge base development in the neurosciences

Ambert, Kyle H., Cohen, Aaron M., Burns, Gully A. P. C., Boudreau, Eilis, Sonmez, Kemal·2013·Frontiers in Neuroinformatics
DOI 10.3389/fninf.2013.00038PMC PMC3872296
Open this paper in CoChat — free account

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fninf.2013.00038

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Diagrammatic representation of the distribution of documents in our corpus, and workflow diagram for annotation in the active learning experiment. Documents were randomly allocated to either the initial training, active learning pool, validation, or test collections. A total of 962 documents were annotated during these experiments—670 during the active learning procedure, 92 during the random validation experiments, and an additional 200 for the hold-out test collection, used to evaluate the Virk system against the random system.
Figure 1 Diagrammatic representation of the distribution of documents in our corpus, and workflow diagram for annotation in the active learning experiment. Documents were randomly allocated to either the initial training, active learning pool, validation, or test collections. A total of 962 documents were annotated during these experiments—670 during the active learning procedure, 92 during the random validation experiments, and an additional 200 for the hold-out test collection, used to evaluate the Virk system against the random system.CC BY · PubMed Central
Figure 2 Performance statistics for the active learning system over iterations of document curation. The gray bars show the cumulating number of positively-annotated documents, while the black dotted line indicate the total number of documents annotated at a given iteration (increasing by 30 at each iteration after the first). The solid black line intersected with solid red lines indicates the estimated number of randomly-selected documents (±90% CI) that, at any iteration, would need to be annotated in order to obtain the same number of positive documents identified by Virk by that iteration. After three rounds of annotation, the average number of documents that would have to be read for a given number of positives is statistically significantly greater than that needing to be annotated with the Virk system.
Figure 2 Performance statistics for the active learning system over iterations of document curation. The gray bars show the cumulating number of positively-annotated documents, while the black dotted line indicate the total number of documents annotated at a given iteration (increasing by 30 at each iteration after the first). The solid black line intersected with solid red lines indicates the estimated number of randomly-selected documents (±90% CI) that, at any iteration, would need to be annotated in order to obtain the same number of positive documents identified by Virk by that iteration. After three rounds of annotation, the average number of documents that would have to be read for a given number of positives is statistically significantly greater than that needing to be annotated with the Virk system.CC BY · PubMed Central
Figure 3 Relative performance of a system trained using only the uncertain documents at each iteration of active learning (black), versus all documents annotated up to that point (red), in terms of AUC. At Iteration 1, both systems are trained using the same data (the 100 initially-annotated documents), and thus score the same. After that, the system trained using all the data consistently out-performs the one using only the uncertain data, though both begin to converge to similar values after 20 iterations.
Figure 3 Relative performance of a system trained using only the uncertain documents at each iteration of active learning (black), versus all documents annotated up to that point (red), in terms of AUC. At Iteration 1, both systems are trained using the same data (the 100 initially-annotated documents), and thus score the same. After that, the system trained using all the data consistently out-performs the one using only the uncertain data, though both begin to converge to similar values after 20 iterations.CC BY · PubMed Central
Figure 4 Performance evaluation comparing AUC of the Virk (red line) and Random Validation (black line) systems over iterations of active learning. The Random Validation system AUC was averaged over 10 random samplings, so that standard error could be calculated (bars). After six iteration, the Virk system outperform the 95% confidence interval for the random validation system.
Figure 4 Performance evaluation comparing AUC of the Virk (red line) and Random Validation (black line) systems over iterations of active learning. The Random Validation system AUC was averaged over 10 random samplings, so that standard error could be calculated (bars). After six iteration, the Virk system outperform the 95% confidence interval for the random validation system.CC BY · PubMed Central
Figure 5 Number of positive-class documents identified over 20 iterations by Virk (red line) and Random Validation (black line, ±95% confidence interval). After 20 iterations, the random validation system identified the number of positives found after only three iterations of the Virk system.
Figure 5 Number of positive-class documents identified over 20 iterations by Virk (red line) and Random Validation (black line, ±95% confidence interval). After 20 iterations, the random validation system identified the number of positives found after only three iterations of the Virk system.CC BY · PubMed Central
Figure 6 Goodness: work ratio over iterations of active learning. No data exists at the first iteration because no active learning has yet taken place. Between iterations 2 and 7, the goodness:work ratio increases, being approximately level until iteration 15, where it begins to decline.
Figure 6 Goodness: work ratio over iterations of active learning. No data exists at the first iteration because no active learning has yet taken place. Between iterations 2 and 7, the goodness:work ratio increases, being approximately level until iteration 15, where it begins to decline.CC BY · PubMed Central
Figure 7 The top 20 rank-ordered features, in terms of information gain, over iterations of active learning. The color of the text denotes which section of the document's associated MEDLINE record the term came from: either title (black), abstract (red), or MeSH (blue). Certain terms, such as ganglion are found across many iterations, though its position in the rank-ordered list changes, while others, such as “via” were less informative, appearing only in the first iteration.
Figure 7 The top 20 rank-ordered features, in terms of information gain, over iterations of active learning. The color of the text denotes which section of the document's associated MEDLINE record the term came from: either title (black), abstract (red), or MeSH (blue). Certain terms, such as ganglion are found across many iterations, though its position in the rank-ordered list changes, while others, such as “via” were less informative, appearing only in the first iteration.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 2 references

Research works printed but not linked: 29

  1. Ambert K. Cohen A. (2009). A system for classifying disease comorbidity status from medical discharge summaries using automated hotspot and negated concept detection. J. Am. Med. Inform. Assoc. 16, 590 10.1197/jamia.M3095 19390099 PMC2705265 · DOI
  2. Ambert K. Cohen A. (2011). k-information gain scaled nearest neighbors: a novel approach to classifying protein-protein interactions in free-text. IEEE Trans. Comput. Biol. Bioinform. 9, 305– 310 10.1109/TCBB.2011.32 21339533 · DOI
  3. Ambert K. Cohen A. (2012). Text-mining and neuroscience. Int. Rev. Neurobiol. 103, 109– 132 10.1016/B978-0-12-388408-4.00006-X 23195123 · DOI
  4. Ananiadou S. Rea B. Okazaki N. Procter R. Thomas J. (2009). Supporting systematic reviews using text mining. Soc. Sci. Comput. Rev. 27, 509– 523 10.1177/0894439309332293 · DOI
  5. Arens R. (2010). Learning SVM ranking function from user feedback using document metadata and active learning in the biomedical domain. Preference Learn. 11, 363 10.1007/978-3-642-14125-6_17 · DOI
  6. Cannon R. C. Gewaltig M.-O. Gleeson P. Bhalla U. S. Cornelis H. Hines M. L. (2007). Interoperability of neuroscience modeling software: current status and future directions. Neuroinformatics 5, 127– 138 10.1007/s12021-007-0004-5 17873374 PMC2658651 · DOI
  7. Cohen A. (2008). Five-way smoking status classification using text hot-spot identification and error-correcting output codes. J. Am. Med. Inform. Assoc. 15, 32 10.1197/jamia.M2434 17947623 PMC2274879 · DOI
  8. Cohen A. Adams C. Davis J. Yu C. Yu P. Meng W. (2010a). Evidence-based medicine, the essential role of systematic reviews, and the need for automated text mining tools, in Proceedings of the 1st ACM International Health Informatics Symposium (New York, NY: ACM), 376– 380
  9. Cohen A. Ambert K. McDonagh M. (2010b). A prospective evaluation of an automated classification system to support evidence-based medicine and systematic review, in AMIA Annual Symposium Proceedings. Vol. 2010 (Bethesda, MD: American Medical Informatics Association), 121 21346953 PMC3041348
  10. Cohen A. Ambert K. Yang J. Felder R. Sproat R. Roark B. (2010c). Ohsu/portland vamc team participation in the 2010 i2b2/va challenge tasks, in Proceedings of the 2010 i2b2/VA Workshop on Challenges in Natural Language Processing for Clinical Data (Boston, MA: i2b2).
  11. Cohen A. Ambert K. McDonagh M. (2009). Cross-topic learning for work prioritization in systematic review creation and update. J. Am. Med. Inform. Assoc. 16, 690– 704 10.1197/jamia.M3162 19567792 PMC2744720 · DOI · Paper page
  12. Ertekin S. Huang J. Giles C. (2007). Active learning for class imbalance problem, in Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (New York, NY: ACM), 823– 824
  13. Figueroa R. Zeng-Treitler Q. Ngo L. Goryachev S. Wiechmann E. (2012). Active learning for clinical text classification: is it better than random sampling? J. Am. Med. Inform. Assoc. 19, 809– 816 10.1136/amiajnl-2011-000648 22707743 PMC3422824 · DOI
  14. Forman G. (2006). Tackling concept drift by temporal inductive transfer, in Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (New York, NY: ACM), 252– 259
  15. Hirschman L. Burns G. Krallinger M. Arighi C. Cohen K. Valencia A. (2012). Text mining for the biocuration workflow. Database 2012: bas020 10.1093/database/bas020 22513129 PMC3328793 · DOI
  16. Howe D. Costanzo M. Fey P. Gojobori T. Hannick L. Hide W. (2008). Big data: the future of biocuration. Nature 455, 47– 50 10.1038/455047a 18769432 PMC2819144 · DOI
  17. Kouznetsov A. Matwin S. Inkpen D. Razavi A. Frunza O. Sehatkar M. (2009). Classifying biomedical abstracts using committees of classifiers and collective ranking techniques, Advances in Artificial Intelligence, eds Gao Y. Japkowicz N. (Berlin: Springer), 224– 228 10.1007/978-3-642-01818-3_29 · DOI
  18. Krallinger M. Leitner F. Valencia A. (2009). The BioCreative ii.5 challenge overview, in Proceedings of the BioCreative II.5 Workshop 2009 on Digital Annotations, Vol. 19, (Madrid).
  19. Liere R. Tadepalli P. (1997). Active learning with committees for text categorization. Proc. Natl. Conf. Artif. Intel. 4, 591– 597
  20. McCallum A. Nigam K. (1998). Employing em in pool-based active learning for text classification, in Proceedings of ICML-98, 15th International Conference on Machine Learning (San Francisco, CA: Morgan Kaufmann Publishers), 350– 358
  21. Mohamed T. Carbonell J. Ganapathiraju M. (2010). Active learning for human protein-protein interaction prediction. BMC Bioinform. 11(Suppl. 1), S57 10.1186/1471-2105-11-S1-S57 20122232 PMC3009530 · DOI
  22. Resnick P. Varian H. (1997). Recommender systems. Commun. ACM 40, 56– 58 10.1145/245108.245121 · DOI · Paper page
  23. Schohn G. Cohn D. (2000). Less is more: active learning with support vector machines, in Machine Learning-International Workshop then Conference (San Francisco, CA: Morgan Kaufmann Publishers Inc.), 839– 846
  24. Stephan K. Kamper L. Bozkurt A. Burns G. Young M. Kötter R. (2001). Advanced database methodology for the collation of connectivity data on the Macaque brain (CoCoMac). Philos. Trans. R. S. Lond. B Biol. Sci. 356, 1159 10.1098/rstb.2001.0908 11545697 PMC1088509 · DOI
  25. Subramaniyaswamy V. Vijayakumar V. Indragandhi V. (2013). A review of ontology-based tag recommendation approaches. Int. J. Intel. Sys. 28, 1054– 1071 10.1002/int.21616 · DOI
  26. Tong S. Koller D. (2002). Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. 2, 45– 66 10.1023/a:1009715923555 · DOI
  27. Uzuner Ö (2009). Recognizing obesity and comorbidities in sparse data. J. Am. Med. Inform. Assoc. 16, 561– 570 10.1197/jamia.M3115 19390096 PMC2705260 · DOI
  28. Wong W. Liu W. Bennamoun M. (2012). Ontology learning from text: a look back and into the future. ACM Comput. Surv. (CSUR) 44, 20 10.1145/2333112.2333115 · DOI
  29. Zhang Z. (2004). Weakly-supervised relation classification for information extraction, in Proceedings of the Thirteenth ACM International Conference on Information and Knowledge Management (New York, NY: ACM), 581– 588

Cited by

See all 6

Metadata

Authors
Ambert, K. H., Cohen, A. M., Burns, G. A. P. C., Boudreau, E., & Sonmez, K.
Venue
Frontiers in Neuroinformatics
Publisher
Frontiers Media SA
Published
2013
Volume
7
ISSN
1662-5196
DOI
10.3389/fninf.2013.00038
PMC
PMC3872296
Cited by
6
References
2

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Virk: an active learning-based system for bootstrapping knowledge base development in the neurosciences. https://app.cochat.ai/papers/10.3389/fninf.2013.00038
The paper
Ambert, K. H., Cohen, A. M., Burns, G. A. P. C., Boudreau, E., & Sonmez, K. (2013). Virk: an active learning-based system for bootstrapping knowledge base development in the neurosciences. Frontiers in Neuroinformatics. https://doi.org/10.3389/fninf.2013.00038