Counterexample-Guided Data Augmentation
Open accessAbstract
We present a novel framework for augmenting data sets for machine learning based on counterexamples. Counterexamples are misclassified examples that have important properties for retraining and improving the model. Key components of our framework include a \textit{counterexample generator}, which produces data items that are misclassified by the model and error tables, a novel data structure that stores information pertaining to misclassifications. Error tables can be used to explain the model's vulnerabilities and are used to efficiently generate counterexamples for augmentation. We show the efficacy of the proposed framework by comparing it to classical augmentation techniques on a case study of object detection in autonomous driving based on deep neural networks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Ideas from verified inductive synthesis of programs can be useful in training AI components with provable guarantees.
“Ideas from verified inductive synthesis of programs can be useful in training AI components with provable guarantees (e.g.Dreossi et al., 2018;Abate et al., 2023).”
From Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems (2024) · cites this paper as Dreossi et al., 2018
The Conclusion describes the approach as counterexample-guided inductive synthesis applied to training an ML model, and reports improved image-classification accuracy. It does not establish provable guarantees for the trained model; the discussion of extending the method to system-level verification is presented as future work. The copy provides no page number for these passages. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Alur, R., Bodik, R., Juniwal, G., Martin, M. M. K., Raghothaman, M., Seshia, S. A., Singh, R., Solar-Lezama, A., Torlak, E., & Udupa, A. (2013) · 2013 Formal Methods in Computer-Aided Design · doi:10.1109/fmcad.2013.6679385 · Cited by 656 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 24
- Rajeev Alur, Rastislav Bodik, Garvit Ju- niwal, Milo M. K. Martin, Mukund Raghothaman, San- jit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. Syntax-guided syn- thesis. In Proceedings of the IEEE International Confer- ence on Formal Methods in Computer-Aided Design (FM- CAD), pages 1-17, October 2013.
- Dan Ciregan, Ueli Meier, and Jürgen Schmidhuber. Multi-column deep neural networks for im- age classification. In Computer vision and pattern recogni- tion (CVPR), 2012 IEEE conference on, pages 3642-3649. IEEE, 2012.
- Dan C Cires ¸an, Ueli Meier, Jonathan Masci, Luca M Gambardella, and Jürgen Schmidhuber. High-performance neural networks for visual object clas- sification. arXiv preprint arXiv:1102.0183, 2011. · arXiv
- Dreossi et al., 2017a] Tommaso Dreossi, Alexandre Donze, and Sanjit A. Seshia. Compositional falsification of cyber- physical systems with machine learning components. In Proceedings of the NASA Formal Methods Conference (NFM), pages 357-372, May 2017.
- Dreossi et al., 2017b] Tommaso Dreossi, Shromona Ghosh, Alberto L. Sangiovanni-Vincentelli, and Sanjit A. Seshia. Systematic testing of convolutional neural networks for autonomous driving. In ICML Workshop on Reliable Ma- chine Learning in the Wild (RMLW), 2017. Published on Arxiv: abs/1708.03309. · arXiv
- Tommaso Dreossi, Somesh Jha, and Sanjit A. Seshia. Semantic adversarial deep learning. In 30th International Conference on Computer Aided Verifi- cation (CAV), 2018.
- Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
- John H Halton. On the efficiency of cer- tain quasi-random sequences of points in evaluating multi- dimensional integrals. Numerische Mathematik, 2(1):84- 90, 1960.
- Susmit Jha and Sanjit A. Seshia. A Theory of Formal Synthesis via Inductive Learning. Acta Informatica, 54(7):693-726, 2017.
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.
- Xiaodan Liang, Zhiting Hu, Hao Zhang, Chuang Gan, and Eric P Xing. Recurrent topic-transition gan for visual paragraph generation. arXiv preprint arXiv:1703.07022, 2017. · arXiv
- Marco Marchesi. Megapixel size image creation using generative adversarial networks. arXiv preprint arXiv:1706.00082, 2017. · arXiv
- Niederreiter, 1988] Harald Niederreiter. Low-discrepancy and low-sequences. Journal of number theory, 30(1):51- 70, 1988.
- Stuart Russell, Tom Dietterich, Eric Horvitz, Bart Selman, Francesca Rossi, Demis Hassabis, Shane Legg, Mustafa Suleyman, Dileep George, and Scott Phoenix. Letter to the editor: Research priorities for ro- bust and beneficial artificial intelligence: An open letter. AI Magazine, 36(4), 2015.
- Sanjit A. Seshia, Dorsa Sadigh, and S. Shankar Sastry. Towards Verified Artificial Intelligence. ArXiv e-prints, July 2016.
- Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In Conference on Computer Vision and Pattern Recognition, CVPR, pages 761-769, 2016.
- Patrice Y Simard, David Steinkraus, John C Platt, et al. Best practices for convolutional neural networks applied to visual document analysis. In ICDAR, volume 3, pages 958-962, 2003.
- Ilya M Sobol. Uniformly distributed se- quences with an additional uniform property. USSR Computational Mathematics and Mathematical Physics, 16(5):236-242, 1976.
- Armando Solar-Lezama, Liviu Tancau, Rastislav Bodík, Sanjit A. Seshia, and Vijay A. Saraswat. Combinatorial sketching for finite programs. In Proceedings of the 12th International Conference on Ar- chitectural Support for Programming Languages and Op- erating Systems (ASPLOS), pages 404-415. ACM Press, October 2006.
- Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural net- works from simulation to the real world. In Conference on Intelligent Robots and Systems, IROS, pages 23-30. IEEE, 2017.
- David A van Dyk and Xiao-Li Meng. The art of data augmentation. Journal of Com- putational and Graphical Statistics, 10(1):1-50, 2001.
- Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987.
- Sebastien C Wong, Adam Gatt, Victor Stamatescu, and Mark D McDonnell. Understanding data augmentation for classification: when to warp? In Digital Image Computing: Techniques and Applications (DICTA), 2016 International Conference on, pages 1-6. IEEE, 2016.
- Yan Xu, Ran Jia, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, and Zhi Jin. Improved relation classification by deep recurrent neural networks with data augmentation. arXiv preprint arXiv:1601.03651, 2016. · arXiv
Cited by
Sorted by citations
Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
Ashmore, R., Calinescu, R., & Paterson, C. (2022) · ACM Computing Surveys · doi:10.1145/3453444 · Cited by 256 · see citing works
Requirements-driven test generation for autonomous vehicles with machine learning components
Tuncali, C. E., Fainekos, G., Prokhorov, D., Ito, H., & Kapinski, J. (2020) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2019.2955903 · Cited by 129 · see citing works
Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems
Dalrymple, D. "., Skalse, J., Bengio, Y., Russell, S., Tegmark, M., Seshia, S., Omohundro, S., Szegedy, C., Goldhaber, B., Ammann, N., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.06624 · Cited by 12 · see citing works
Self-citation
- Authors
- Dreossi, T., Ghosh, S., Yue, X., Keutzer, K., Sangiovanni-Vincentelli, A., & Seshia, S. A.
- Affiliations
- University of California, Berkeley (US)
- Venue
- Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
- Published
- 2018-07
- Pages
- 2071-2078
- OpenAlex
- W2803289520
- Cited by
- 60
- References
- 1
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 2 · 2023: 9 · 2022: 5 · 2021: 13 · 2020: 10 · 2019: 12 · 2018: 2
- FWCI
- 3.72
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1646208, 1739816, 1545126)
- Nvidia
- Defense Advanced Research Projects Agency (FA8750, FA8750-16-C-0043)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- counterexample generation, misclassification, model vulnerability, object detection, autonomous driving, deep neural network, data augmentation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
