Demo Application for the AutoGOAL Framework
Open accessAbstract
Suilan Estevez-Velarde, Alejandro Piad-Morffis, Yoan Gutiérrez, Andres Montoyo, Rafael Muñoz-Guillena, Yudivián Almeida Cruz. Proceedings of the 28th International Conference on Computational Linguistics: System Demonstrations. 2020.
AI snapshot
Written from the paper’s full text.
TL;DR
AutoGOAL is a Python AutoML framework that uses a graph-based pipeline representation and probabilistic grammatical evolution to combine algorithms from multiple libraries, with an accompanying web demo. It reports competitive results against Auto-Weka, TPOT, Auto-Sklearn, and ML-Plan on classic datasets and two NLP challenges.
Key findings
- AutoGOAL provides 133+ adapters from 7 back-end libraries, including scikit-learn, nltk, gensim, spacy, keras, and pytorch, and supports text, images, and structured data.
- It offers a High-Level API for non-experts (black-box, scikit-learn-like) and a Low-Level API for experts (grammar-based solution space).
- Evaluation compares AutoGOAL with Auto-Weka, TPOT, Auto-Sklearn, and ML-Plan; AutoGOAL achieves comparative results on classic datasets with similar computational cost (1 hour per run).
- In NLP domains, AutoGOAL performs comparable to state-of-the-art hand-crafted solutions while requiring less expertise and effort (48 hours of execution); Table 1 reports AutoGOAL 21.1 on HAHA and 3.99 on MEDDOCAN (F1 for MEDDOCAN).
Methods
- Design: Python library with a graph-based representation for machine learning pipelines; core modules are Grammar (probabilistic context-free grammar), Optimisation (random search and probabilistic grammatical evolution), and Pipelines (Facade pattern connecting algorithms via type-annotated input/output).
- Evaluation: compared against Auto-Weka, TPOT, Auto-Sklearn, and ML-Plan; Table 1 reports results across datasets including Cars, Credit G., Abalone, Shuttle, Yeast, Dorothea, Gisette, HAHA, and MEDDOCAN, with other systems’ values obtained from ML-Plan (Mohr et al., 2018).
- Demo: a web application is available online, and full source code is provided under the MIT open-source license.
Limitations
- The paper does not state a dedicated limitations section; it describes AutoGOAL’s results as “comparative” rather than superior and notes the algorithm library is “undergoing continuous expansion.”
- Evident from the text: values for other AutoML systems were taken from ML-Plan rather than independently re-run, and no variance or statistical significance tests are reported.
- Evident from the provided text: the table formatting is garbled, making exact per-dataset comparisons difficult to verify.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
1 of 9 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured
- Premisecitation
AutoGOAL uses a search strategy based on probabilistic grammatical evolution to discover optimal machine learning pipelines.
“Furthermore, a search strategy based on probabilistic grammatical evolution is used to discover optimal machine learning pipelines (Estevez-Velarde et al., 2019) whose components can be from different back-end libraries.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Estevez-Velarde et al., 2019 source
Section 4 describes a modified probabilistic grammatical-evolution strategy for obtaining an optimal pipeline, and Section 4.2 explains how it searches for the best-performing pipeline. Our copy does not identify that strategy as AutoGOAL or establish that pipeline components can come from different back-end libraries. (LLM, conf=0.83)
Experimental — accuracy not yet measured
- Findingnone given
In NLP domains (text classification and entity recognition), AutoGOAL performs comparable to state-of-the-art solutions hand-crafted by human experts.
“In these domains AutoGOAL performs comparable to state-of-the-art solutions hand-crafted by human experts, while requiring considerably less expertise and effort (48 hours of execution).”
Quote in the paperSubstantial claim - Study design and conductcitation
AutoGOAL has been evaluated and compared to Auto-Weka, TPOT, Auto-Sklearn, and ML-Plan.
“AutoGOAL has been evaluated in different domains and compared to other AutoML tools, including Auto-Weka (Thornton et al., 2013), TPOT (Olson and Moore, 2016), Auto-Sklearn (Feurer et al., 2015), and ML-Plan (Mohr et al., 2018), see Table 1.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Thornton et al., 2013 source
- cites Olson and Moore, 2016
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Feurer et al., 2015
- Premiseown data
AutoGOAL provides 133+ adapters of existing algorithms from 7 different back-end libraries.
“AutoGOAL comes prepackaged with 133 plus adapters of existing algorithms, from 7 different back-end libraries, for multiple problems including text preprocessing, feature extraction, dataset augmentation, dimensionality reduction, as well as supervised and unsupervised learning techniques.”
- Also stated in: § 3 Implementation Details · plain
“A total of 133 algorithms from 7 different back-end libraries 4 are provided, several of which are semi-automatically created by code introspection (e.g., algorithms from scikit-learn and nltk which conform to a consistent API and the hyperparameters are documented in a consistent format), and the rest are manually added by the library developers.”
Quote in the paperNot a substantial claim - Also stated in: § 3 Implementation Details · plain
- Interpretationreasoning
AutoGOAL's main strength lies in its ability to combine different tools for solving complex problems beyond structured supervised learning, such as natural language processing, using virtually the same code.
“However, AutoGOAL's main strength lies in its ability to combine different tools for solving complex problems beyond structured supervised learning, such as natural language processing, using virtually the same code, by specifying the input and output datatypes.”
Quote in the paperSubstantial claim
- Interpretationnone given
AutoGOAL requires considerably less expertise and effort (48 hours of execution) in NLP domains.
“In these domains AutoGOAL performs comparable to state-of-the-art solutions hand-crafted by human experts, while requiring considerably less expertise and effort (48 hours of execution).”
Quote in the paperSubstantial claim - Premisereasoning
AutoGOAL uses a graph-based representation for machine learning pipelines.
“To unify disparate APIs into a single interface, AutoGOAL proposes a novel graph-based representation for machine learning pipelines.”
Quote in the paperSubstantial claim - Premisenone given
AutoGOAL provides a simple programming interface for creating automatically discoverable adapters for existing technologies.
“Extensibility: AutoGOAL proposes a simple programming interface that developers can implement to create an automatically discoverable adapter for an existing technology from any machine learning framework.”
Quote in the paperSubstantial claim
- May combine two claims. 6
- Classified as probably not a substantial claim. 7
May combine two claims.
Findingnone givenAutoGOAL achieves comparative results with other AutoML tools in classic datasets, with similar computational cost (1 hour per run).
“In terms of performance, AutoGOAL achieves comparative results with other AutoML tools in classic datasets, with similar computational cost (1 hour per run).”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningAutoGOAL greatly simplifies the application of machine learning for non-expert users while providing powerful low-level components for experts.
“AutoGOAL greatly simplifies the application of machine learning for non-expert users while providing powerful lowlevel components for experts to effectively optimise complex machine learning pipelines.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationnone givenAutoGOAL can be applied seamlessly to Natural Language Processing as well as structured classification problems, in contrast with alternative frameworks.
“In contrast with alternative frameworks, AutoGOAL can be applied seamlessly to Natural Language Processing as well as structured classification problems.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone givenAutoGOAL is designed with extensibility as a priority, enabling the addition of new algorithms from any machine learning library via a simple interface.
“The framework has been designed with extensibility as a priority, enabling the addition of new algorithms from any conceivable machine learning library by conforming to a simple interface.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone givenAutoGOAL is a framework in Python for automatically finding the best way to solve a given task.
“AutoGOAL is a framework in Python for automatically finding the best way to solve a given task.”
Quote in the paperNot a substantial claimMay combine two claims.
Study design and conductcitationAutoGOAL has been evaluated on classic datasets, HAHA, and MEDDOCAN.
“AutoGOAL is compared with other AutoML approaches in classic datasets (Dua and Graff, 2017) but can applied to more complex domains such as text classification in HAHA (Chiruzzo et al., 2019) and entity recognition in MEDDOCAN (Lara-Clares and Garcia-Serrano, 2019).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Dua and Graff, 2017
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Chiruzzo et al., 2019
- cites Lara-Clares and Garcia-Serrano, 2019
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisenone givenThe Grammar module automatically infers a context-free grammar from annotated classes.
“Given a collection of annotated classes, this module automatically infers a context-free grammar that describes the space of all possible instances of those classes.”
Quote in the paperNot a substantial claimMay combine two claims.
Premisenone givenAutoGOAL provides high-level classes that non-experts can use as black-box AutoML solutions, compatible with text, images, and structured data.
“Ease of use: AutoGOAL provides high-level classes that non-experts can use as black-box AutoML solutions, compatible with several data types including text, images and structured (tabular) data.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisereasoningThe Pipelines module connects algorithms in a graph where each path represents a possible pipeline.
“Classes implementing this pattern are automatically connected in a graph of algorithms where each path represents a possible pipeline for solving a specific problem (defined by the input and output datatypes).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe field of automatic machine learning (AutoML) has risen to prominence as a principled alternative for finding optimal or close to optimal solutions to complex machine learning problems.
“The field of automatic machine learning (AutoML) has risen to prominence as a principled alternative for finding optimal or close to optimal solutions to complex machine learning problems (Hutter et al., 2018).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Hutter et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationThe Optimisation module implements random search and probabilistic grammatical evolution.
“Two optimisation strategies are implemented: random search and probabilistic grammatical evolution (O'Neill and Ryan, 2001).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites O'Neill and Ryan, 2001 source
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone givenAutoGOAL can automatically infer input and output types from the dataset in many cases.
“In many cases AutoGOAL can automatically infer the input and output type from the dataset.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
Most existing AutoML tools focus on a specific family of algorithms or a specific problem setting.
“Most existing AutoML tools focus on a specific family of algorithms (such as neural networks) or a specific problem setting (such as supervised learning from tabular data).”
The paper relies on this without support.
Several challenges remain in complex domains such as natural language processing, where tools and technologies from different sources must be combined.
“Hence, despite the recent success of AutoML, several challenges still remain, specially in complex domains such as natural language processing, where tools and technologies from different sources must be combined.”
The paper relies on this without support.
AutoGOAL can be used in any scenario where several possible strategies are available to solve a given computational task.
“It has been designed mainly for automatic machine learning (AutoML) but it can be used in any scenario where several possible strategies are available to solve a given computational task.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Table caption calls all nine datasets classic
MinorThe table caption describes nine classic machine-learning datasets, but the evaluation section distinguishes the classic datasets from HAHA and MEDDOCAN, which it identifies as text classification and entity recognition tasks.
“for 9 classic machine learning datasets in terms of accuracy, except for MEDDOCAN, in which F 1 is used.”
“AutoGOAL is compared with other AutoML approaches in classic datasets (Dua and Graff, 2017) but can applied to more complex domains such as text classification in HAHA (Chiruzzo et al., 2019) and entity recognition in MEDDOCAN (Lara-Clares and Garcia-Serrano, 2019).”
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · cited 1× · see citing works
“High-Level API (non-experts): This API allows AutoGOAL to be used as a black-box classification or regression algorithm with an interface similar to the scikit-learn library (Pedregosa et al., 2011).”
MethodUCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 24,190 · see citing works
Hutter, F., Kotthoff, L., & Vanschoren, J. (2019) · The Springer series on challenges in machine learning · doi:10.1007/978-3-030-05318-5 · Cited by 1,552 · see citing works
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms
Thornton, C., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487629 · Cited by 1,359 · cited 1× · see citing works
“AutoGOAL has been evaluated in different domains and compared to other AutoML tools, including Auto-Weka (Thornton et al., 2013), TPOT (Olson and Moore, 2016), Auto-Sklearn (Feurer et al., 2015), and ML-Plan (Mohr et al., 2018), see Table 1.”
Method(2015) · neural information processing systems · Cited by 1,305 · see citing works
O'Neill, M., & Ryan, C. (2001) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.942529 · Cited by 951 · see citing works
TPOT: A Tree-Based Pipeline Optimization Tool for Automating Machine Learning
Olson, R. S., & Moore, J. H. (2019) · The Springer series on challenges in machine learning · doi:10.1007/978-3-030-05318-5_8 · Cited by 605 · cited 1× · see citing works
“AutoGOAL has been evaluated in different domains and compared to other AutoML tools, including Auto-Weka (Thornton et al., 2013), TPOT (Olson and Moore, 2016), Auto-Sklearn (Feurer et al., 2015), and ML-Plan (Mohr et al., 2018), see Table 1.”
MethodAuto-WEKA: Combined Selection and Hyperparameter Optimization of Classification AlgorithmsPreprint
Thornton, C., Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1208.3719 · Cited by 326 · see citing works
ML-Plan: Automated machine learning via hierarchical planning
Mohr, F., Wever, M., & Hüllermeier, E. (2018) · Machine Learning · doi:10.1007/s10994-018-5735-z · Cited by 175 · cited 2× · see citing works
“(Mohr et al., 2018).”
Method(2019) · IberLEF@SEPLN · Cited by 37 · see citing works
General-purpose hierarchical optimisation of machine learning pipelines with grammatical evolution
Estévez-Velarde, S., Gutiérrez, Y., Almeida-Cruz, Y., et al. (2020) · Information Sciences · doi:10.1016/j.ins.2020.07.035 · Cited by 20 · cited 1× · see citing works
“The latter performs a sampling/update cycle that selects the best performing instances according to some predefined metric (e.g., accuracy on a development set) and iteratively updates the internal probabilistic model of the sampler (Est´evez-Velarde et al., 2020).”
BackgroundSelf-citationAutoML Strategy Based on Grammatical Evolution: A Case Study about Knowledge Discovery from Text
Estevez-Velarde, S., Gutiérrez, Y., Montoyo, A., & Almeida-Cruz, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1428 · Cited by 15 · cited 1× · see citing works
“Furthermore, a search strategy based on probabilistic grammatical evolution is used to discover optimal machine learning pipelines (Estevez-Velarde et al., 2019) whose components can be from different back-end libraries.”
MethodSelf-citation(2019) · IberLEF@SEPLN · Cited by 3 · see citing works
Flagged first, then by citations
- Authors
- Estevez-Velarde, S., Piad-Morffis, A., Gutiérrez, Y., Montoyo, A., Muñoz-Guillena, R., & Almeida Cruz, Y.
- Affiliations
- University of Havana (CU)
- University of Alicante (ES)
- Venue
- Proceedings of the 28th International Conference on Computational Linguistics: System Demonstrations
- Published
- 2020
- Pages
- 18-22
- OpenAlex
- W3117357796
- Semantic Scholar
- ac71c2e38e1200ada23da2d4840ddec530b3d36f
- Cited by
- 0
- References
- 12
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- RUA, Repositorio Institucional de la Universidad de Alicante (Universidad de Alicante) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- dialogue systems, task-oriented dialogue, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
