CoChat
Sign Up
Cited by 1Multimodal Machine Learning ApplicationsSoftware Engineering ResearchTopic Modeling

AdaCoder: Adaptive Prompt Compression for Programmatic Visual Question Answering

Open access
Ukai, Mahiro, Kurita, Shuhei, Hashimoto, Atsushi, Ushiku, Yoshitaka, Inoue, Nakamasa·2024·Proceedings of the 32nd ACM International Conference on Multimedia
DOI 10.1145/3664647.3681010
Open this paper in CoChat — free accountOpen PDF

Abstract

Visual question answering aims to provide responses to questions given visual input. Recently, visual programmatic models (VPMs), which generate programs to answer questions through large language models (LLMs), have attracted attention. However, they often require long input prompts to provide the LLM with sufficient API usage details to generate relevant code. To address this limitation, we propose AdaCoder, an adaptive prompt compression framework for VPMs. AdaCoder operates in two phases: a compression phase and an inference phase. In the compression phase, given a preprompt that describes all API definitions with example code snippets, a set of compressed preprompts is generated, each depending on a specific question type. In the inference phase, AdaCoder predicts the question type and chooses the appropriate corresponding compressed preprompt to generate code to answer the question. In experiments, we apply AdaCoder to ViperGPT and demonstrate that it reduces token length by 71.1%, while maintaining or even improving the performance of visual question answering.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3664647.3681010

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Feedback-driven control and harness optimization methods use static analysis, runtime errors, tests, and human feedback to revise code through repeated execution.

    “feedback-driven control and harness optimization methods that use static analysis, runtime errors, tests, and human feedback to revise code through repeated execution [50, 51, 52, 53]”

    From Code as Agent Harness (2026) · cites this paper as [51]

    The error analysis (p. 7) manually counts coding errors, and the qualitative examples (p. 8) describe executing generated programs. Neither describes using static analysis, runtime errors, tests, or human feedback to revise code through repeated execution. The conclusion (p. 9) presents interactive code modification as future work. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. [1][1] Peter Anderson, Xiaodong He, Chris Buehler, et al. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 6077-6086. -
  2. [3][3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV) . 2425-2433. -
  3. [4][4] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2023. Token Merging: Your ViT But Faster. In Proc. International Conference on Learning Representations (ICLR) . -
  4. [5][5] Tom Brown, Benjamin Mann, Nick Ryder, et al. 2020. Language models are few-shot learners. In Proc. Annual Conference on Neural Information Processing Systems (NeurIPS) . 1877-1901. -
  5. [6][6] Jiali Chen, Zhenjun Guo, Jiayuan Xie, Yi Cai, and Qing Li. 2023. Deconfounded Visual Question Generation with Causal Inference. In Proc. ACM International Conference on Multimedia (ACMMM) . 5132-5142. -
  6. [7][7] Kang Chen, Tianli Zhao, and Xiangqian Wu. 2023. VTQA2023: ACM Multimedia 2023 Visual Text Question Answering Challenge. In Proc. ACM International Conference on Multimedia (ACMMM) . 9646-9650. -
  7. [10][10] Xiang Fu, Yuxin Hu, Baoxu Li, Yasheng Feng, Hao Wang, Xian Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. BLINK: Multimodal Large Language Models Can See but Not Perceive. In Proc. European Conference on Computer Vision (ECCV) . -
  8. [11][11] Jiaxin Ge, Sanjay Subramanian, Baifeng Shi, Roei Herzig, and Trevor Darrell. 2024. Recursive Visual Programming. In Proc. European Conference on Computer Vision (ECCV) . -
  9. [12][12] Tao Ge, Jing Hu, Li Dong, Shaoguang Mao, Yan Xia, Xun Wang, Si-Qing Chen, and Furu Wei. 2022. Extensible Prompts for Language Models on Zero-shot Language Style Customization. In Proc. Annual Conference on Neural Information Processing Systems (NeurIPS) . -
  10. [13][13] Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan T. Chakaravarthy, Yogish Sabharwal, and Ashish Verma. 2020. Power-bert: Accelerating BERT inference via progressive word-vector elimination. In Proc. International Conference on Machine Learning (ICML) . 3690-3699. -
  11. [14][14] Yash Goyal, Tejas Khot, et al. 2017. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 6904-6913. -
  12. [15][15] Tanmay Gupta and Aniruddha Kembhavi. 2022. Visual Programming: Compositional visual reasoning without training. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . -
  13. [16][16] Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. 2024. Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 9590-9601. -
  14. [17][17] Ziqi Huang, Hongyuan Zhu, Ying Sun, et al. 2021. A diagnostic study of visual question answering with analogical reasoning. In Proc. IEEE International Conference on Image Processing (ICIP) . IEEE, 2463-2467. -
  15. [18][18] Drew A. Hudson and Christopher D. Manning. 2019. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV) . -
  16. [20][20] Ziqi Jin and Wei Lu. 2023. Tab-CoT: Zero-shot Tabular Chain of Thought. In Proc. Findings of the Association for Computational Linguistics (ACL Findings) . -
  17. [21][21] Gyuwan Kim and Kyunghyun Cho. 2021. Length-adaptive transformer: Train once with length drop, use anytime with search. In Proc. Annual Meeting of the Association for Computational Linguistics (ACL) . 6501-6511. -
  18. [22][22] Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. 2022. Learned Token Pruning for Transformers. In Proc. ACM SIGKDD Conference on Knowledge Discovery and Data Mining (SIGKDD) . 784-794. -
  19. [23][23] Yunshi Lan, Xiang Li, Xin Liu, Yang Li, Wei Qin, and Weining Qian. 2023. Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts. In Proc. ACM International Conference on Multimedia (ACMMM) . -
  20. [24][24] Tung Le, Huy Tien Nguyen, and Minh Le Nguyen. 2021. Vision and text transformer for predicting answerability on visual question answering. In Proc. IEEE International Conference on Image Processing (ICIP) . IEEE, 934-938. -
  21. [25][25] Liunian Harold Li, Haoxuan You, Zhecan Wang, et al. 2021. Unsupervised Visionand-Language Pre-training Without Parallel Images and Captions. In Proc. Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) . -
  22. [26][26] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. 2022. Grounded language-image pre-training. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . -
  23. [27][27] Rengang Li, Cong Xu, Zhenhua Guo, Baoyu Fan, Runze Zhang, Wei Liu, Yaqian Zhao, Weifeng Gong, and Endong Wang. 2022. AI-VQA: Visual Question Answering based on Agent Interaction with Interpretability. In Proc. ACM International Conference on Multimedia (ACMMM) . 5274-5282. -
  24. [28][28] Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. 2015. Ask Your Neurons: A Neural-based Approach to Answering Questions about Images. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV) . 1-9. -
  25. [29][29] Jesse Mu, Xiang Lisa Li, and Noah Goodman. 2023. Learning to Compress Prompts with Gist Tokens. In Proc. Annual Conference on Neural Information Processing Systems (NeurIPS) . -
  26. [31][31] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023). - · arXiv
  27. [32][32] Argho Sarkar and Maryam Rahnemoonfar. 2022. Grad-CAM aware supervised attention for visual question answering for post-disaster damage assessment. In Proc. IEEE International Conference on Image Processing (ICIP) . IEEE, 3783-3787. -
  28. [33][33] Ruoyue Shen, Nakamasa Inoue, and Koichi Shinoda. 2024. Pyramid Coder: Hierarchical Code Generator for Compositional Visual Question Answering. In Proc. IEEE International Conference on Image Processing (ICIP) . -
  29. [34][34] Sanjay Subramanian, Medhini Narasimhan, et al. 2023. Modular Visual Question Answering via Code Generation. In Proc. Annual Meeting of the Association for Computational Linguistics (ACL) . -
  30. [35][35] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A Corpus for Reasoning About Natural Language Grounded in Photographs. In Proc. Annual Meeting of the Association for Computational Linguistics (ACL) . 6418-6428. -
  31. [36][36] Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. ViperGPT: Visual Inference via Python Execution for Reasoning. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV) . -
  32. [37][37] Gemini Team, Rohan Anil, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023). - · arXiv
  33. [38][38] Hugo Touvron, Louis Martin, Kevin Stone, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023). - · arXiv
  34. [39][39] Qingqing Wang, Liqiang Xiao, Yue Lu, Yaohui Jin, and Hao He. 2022. Towards Reasoning Ability in Scene Text Visual Question Answering. In Proc. ACM International Conference on Multimedia (ACMMM) . 2281-2289. -
  35. [41][41] Xiangyu Wu, Jianfeng Lu, Zhuanfeng Li, and Fengchao Xiong. 2022. Ques-toVisual Guided Visual Question Answering. In Proc. IEEE International Conference on Image Processing (ICIP) . IEEE, 4193-4197. -
  36. [42][42] Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. 2022. AVQA: A Dataset for Audio-Visual Question Answering on Videos. In Proc. ACM International Conference on Multimedia (ACMMM) . 34803491. -
  37. [43][43] Zichao Yang, Xiaodong He, Jianfeng Gao, et al. 2016. Stacked Attention Networks for Image Question Answering. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . -
  38. [44][44] Shunyu Yao, Dian Yu, Jeffrey Zhao, et al. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Proc. Annual Conference on Neural Information Processing Systems (NeurIPS) . -
  39. [45][45] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. 2016. Modeling context in referring expressions. In Proc. European Conference on Computer Vision (ECCV) . 69-85. -
  40. [46][46] Bowen Yuan, Sisi You, and Bing-Kun Bao. 2023. Self-PT: Adaptive Self-Prompt Tuning for Low-Resource Visual Question Answering. In Proc. ACM International Conference on Multimedia (ACMMM) . 5089-5098. -
  41. [47][47] Haotian Zhang and Wei Wu. 2022. Context Relation Fusion Model for Visual Question Answering. In Proc. IEEE International Conference on Image Processing (ICIP) . IEEE, 2112-2116. -
  42. [48][48] Kai Zhang, Li Mo, Wei Chen, Hao Sun, and Yejin Su. 2023. MagicBrush: A manually annotated dataset for instruction-guided image editing. In Proc. Annual Conference on Neural Information Processing Systems (NeurIPS) .

Cited by

Sorted by citations

Authors
Ukai, M., Kurita, S., Hashimoto, A., Ushiku, Y., & Inoue, N.
Affiliations
  • Institute of Science Tokyo (JP)
  • National Institute of Informatics (JP)
  • Omron Sinic X Corporation (Japan) (JP)
ORCID
Venue
Proceedings of the 32nd ACM International Conference on Multimedia
Publisher
ACM
Published
2024-10-28
Pages
9234-9243
DOI
10.1145/3664647.3681010
OpenAlex
W4403791611
Cited by
1
References
6
Citations by year
2025: 1
FWCI
0.21
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM International Conference on Multimedia (ACM MM) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
visual question answering, prompt compression, large language models, API usage, code generation, question classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review