CoChat
Sign Up
PreprintCited by 1Advanced Neural Network ApplicationsAutonomous Vehicle Technology and SafetyMultimodal Machine Learning Applications

FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning

Open access
Cao, Jiajun, Zhang, Qizhe, Jia, Peidong, Zhao, Xuhui, Lan, Bo, Zhang, Xiaoan, Li, Zhuo, Wei, Xiaobao +6 more·2025·arXiv
DOI 10.48550/arxiv.2507.23318arXiv 2507.23318
Open this paper in CoChat — free accountOpen PDF

Abstract

Vision-Language-Action (VLA) models have demonstrated significant potential in complex scene understanding and action reasoning, leading to their increasing adoption in end-to-end autonomous driving systems. However, the long visual tokens of VLA models greatly increase computational costs. Current visual token pruning methods in Vision-Language Models (VLM) rely on either visual token similarity or visual-text attention, but both have shown poor performance in autonomous driving scenarios. Given that human drivers concentrate on relevant foreground areas while driving, we assert that retaining visual tokens containing this foreground information is essential for effective decision-making. Inspired by this, we propose FastDriveVLA, a novel reconstruction-based vision token pruning framework designed specifically for autonomous driving. FastDriveVLA includes a plug-and-play visual token pruner called ReconPruner, which prioritizes foreground information through MAE-style pixel reconstruction. A novel adversarial foreground-background reconstruction strategy is designed to train ReconPruner for the visual encoder of VLA models. Once trained, ReconPruner can be seamlessly applied to different VLA models with the same visual encoder without retraining. To train ReconPruner, we also introduce a large-scale dataset called nuScenes-FG, consisting of 241K image-mask pairs with annotated foreground regions. Our approach achieves state-of-the-art results on the nuScenes open-loop planning benchmark across different pruning ratios.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2507.23318

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Autonomous driving requires displacement and velocity based on occupancy prediction, and robotic control demands rotation angle according to the grid map.

    “For example, autonomous driving [176, 109, 159, 18] would require displacement and velocity based on occupancy prediction, and robotic control [143, 166, 34] would demand rotation angle according to the grid map.”

    From Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality (2025) · cites this paper as [18]

    The Experimental Settings (p. 5) describe an autonomous-driving system evaluated on trajectory prediction, which supports the driving-planning aspect of the cited clause. The paper does not say that it requires displacement and velocity based on occupancy prediction; searches for those terms found no supporting passage. The separate robotic-control clause has its own citations and is not attributed to this source. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

  • By pruning or merging tokens, token reduction reduces computational cost and accelerates runtime, providing a practical solution for enhancing generative efficiency.

    “By pruning or merging tokens, token reduction [43, 46, 54, 57, 62, 69, 77, 92, 153, 182] reduces computational cost and accelerates runtime, providing a practical solution for enhancing generative efficiency [18, 34, 52, 69, 89, 93, 130, 148, 158, 159, 167, 177, 179, 183, 187, 188].”

    From Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality (2025) · cites this paper as [18]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 52

  1. Alvar, S. R.; Singh, G.; Akbari, M.; and Zhang, Y. 2025. Divprune: Diversity-based visual token pruning for large multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference , 9392-9401.
  2. Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 . · arXiv
  3. Bengio, Y.; L´ eonard, N.; and Courville, A. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 . · arXiv
  4. Bojarski, M.; Del Testa, D.; Dworakowski, D.; Firner, B.; Flepp, B.; Goyal, P.; Jackel, L. D.; Monfort, M.; Muller, U.; Nagy, J.; et al. 2016. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 . · arXiv
  5. Caesar, H.; Bankiti, V.; Lang, A. H.; Vora, S.; Liong, V. E.; Xu, Q.; Krishnan, A.; Pan, Y.; Baldan, G.; and Beijbom, O. 2020. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 11621-11631.
  6. Caesar, H.; Kabzan, J.; Tan, K. S.; Fong, W. K.; Wolff, E.; Lang, A.; Fletcher, L.; Beijbom, O.; and Omari, S. 2021. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106.11810 . · arXiv
  7. Cai, M.; Yang, J.; Gao, J.; and Lee, Y. J. 2024. Matryoshka multimodal models. arXiv preprint arXiv:2405.17430 . · arXiv
  8. Cao, J.; Zhang, Y.; Huang, T.; Lu, M.; Zhang, Q.; An, R.; Ma, N.; and Zhang, S. 2025. Move-kd: Knowledge distillation for vlms with mixture of visual encoders. In Proceedings of the Computer Vision and Pattern Recognition Conference , 19846-19856.
  9. Cha, J.; Kang, W.; Mun, J.; and Roh, B. 2024. Honeybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 13817-13827.
  10. Chen, L.; Zhao, H.; Liu, T.; Bai, S.; Lin, J.; Zhou, C.; and Chang, B. 2024. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision , 19-35. Springer.
  11. Chen, Y.; Bai, X.; Wang, Z.; Bai, C.; Dai, Y.; Lu, M.; and Zhang, S. 2025. StreamKV: Streaming Video QuestionAnswering with Segment-based KV Cache Retrieval and Compression. arXiv:2511.07278. · arXiv
  12. Chi, H.; Gao, H.-a.; Liu, Z.; Liu, J.; Liu, C.; Li, J.; Yang, K.; Yu, Y.; Wang, Z.; Li, W.; et al. 2025. Impromptu VLA: Open Weights and Open Data for Driving Vision-LanguageAction Models. arXiv preprint arXiv:2505.23757 . · arXiv
  13. Codevilla, F.; M¨ uller, M.; L´ opez, A.; Koltun, V.; and Dosovitskiy, A. 2018. Imitation learning for autonomous driving in urban environments. In Proceedings of the IEEE international conference on robotics and automation (ICRA) .
  14. Darcet, T.; Oquab, M.; Mairal, J.; and Bojanowski, P. 2023. Vision transformers need registers. arXiv preprint arXiv:2309.16588 . · arXiv
  15. Dhouib, M.; Buscaldi, D.; Vanier, S.; and Shabou, A. 2025. Pact: Pruning and clustering-based token reduction for faster visual language models. In Proceedings of the Computer Vision and Pattern Recognition Conference , 14582-14592.
  16. Ettinger, S.; Cheng, S.; Caine, B.; Liu, C.; Zhao, H.; Pradhan, S.; Chai, Y.; Sapp, B.; Qi, C. R.; Zhou, Y.; et al. 2021. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF international conference on computer vision , 9710-9719.
  17. Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2020. Generative adversarial networks. Communications of the ACM , 63(11): 139-144.
  18. Gu, J.; Sun, C.; and Zhao, H. 2021. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF international conference on computer vision , 15303-15312.
  19. He, K.; Chen, X.; Xie, S.; Li, Y.; Doll´ ar, P.; and Girshick, R. 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 16000-16009.
  20. Hu, W.; Dou, Z.-Y.; Li, L.; Kamath, A.; Peng, N.; and Chang, K.-W. 2024. Matryoshka query transformer for large vision-language models. Advances in Neural Information Processing Systems , 37: 50168-50188.
  21. Hu, Y.; Yang, J.; Chen, L.; Li, K.; Sima, C.; Zhu, X.; Chai, S.; Du, S.; Lin, T.; Wang, W.; et al. 2023. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 1785317862.
  22. Jiang, B.; Chen, S.; Xu, Q.; Liao, B.; Chen, J.; Zhou, H.; Zhang, Q.; Liu, W.; Huang, C.; and Wang, X. 2023. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 8340-8350.
  23. Lang, A. H.; Vora, S.; Caesar, H.; Zhou, L.; Yang, J.; and Beijbom, O. 2019. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 12697-12705.
  24. Li, W.; Yuan, Y.; Liu, J.; Tang, D.; Wang, S.; Qin, J.; Zhu, J.; and Zhang, L. 2025a. Tokenpacker: Efficient visual projector for multimodal llm. International Journal of Computer Vision , 1-19.
  25. Li, Y.; Wei, X.; Chi, X.; Li, Y .; Zhao, Z.; Wang, H.; Ma, N.; Lu, M.; and Zhang, S. 2025b. Manipdreamer3d: Synthesizing plausible robotic manipulation video with occupancyaware 3d trajectory. arXiv preprint arXiv:2509.05314 . · arXiv
  26. Li, Z.; Wang, W.; Li, H.; Xie, E.; Sima, C.; Lu, T.; Yu, Q.; and Dai, J. 2024. Bevformer: learning bird's-eye-view representation from lidar-camera via spatiotemporal transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence .
  27. Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual instruction tuning. Advances in neural information processing systems , 36: 34892-34916.
  28. Liu, Y.; Zhang, J.; Fang, L.; Jiang, Q.; and Zhou, B. 2021. Multimodal motion prediction with stacked transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 7577-7586.
  29. Ma, J.; Zhang, Q.; Lu, M.; Wang, Z.; Zhou, Q.; Song, J.; and Zhang, S. 2025. MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs. arXiv preprint arXiv:2508.21044 . · arXiv
  30. Prakash, A.; Chitta, K.; and Geiger, A. 2021. Multi-modal fusion transformer for end-to-end autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 7077-7087.
  31. Ren, T.; Liu, S.; Zeng, A.; Lin, J.; Li, K.; Cao, H.; Chen, J.; Huang, X.; Chen, Y.; Yan, F.; et al. 2024. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 . · arXiv
  32. Sha, H.; Zhuo, T. Y.; Richards, L. E.; Morin, S.; Karkus, P.; Adrien Taylor, A. T. B. W. W.; Bewley, A.; Shotton, J.; and Kanazawa, A. 2024. Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models. arXiv:2405.11623. · arXiv
  33. Shang, Y.; Cai, M.; Xu, B.; Lee, Y. J.; and Yan, Y. 2024. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 . · arXiv
  34. Sima, C.; Renz, K.; Chitta, K.; Chen, L.; Zhang, H.; Xie, C.; Beißwenger, J.; Luo, P.; Geiger, A.; and Li, H. 2024. Drivelm: Driving with graph visual question answering. In European conference on computer vision , 256-274. Springer.
  35. Tian, X.; Gu, J.; Li, B.; Liu, Y.; Wang, Y.; Zhao, Z.; Zhan, K.; Jia, P.; Lang, X.; and Zhao, H. 2024. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289 . · arXiv
  36. Wang, P.; Bai, S.; Tan, S.; Wang, S.; Fan, Z.; Bai, J.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; et al. 2024a. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191 . · arXiv
  37. Wang, S.; Yu, Z.; Jiang, X.; Lan, S.; Shi, M.; Chang, N.; Kautz, J.; Li, Y.; and Alvarez, J. M. 2025. Omnidrive: A holistic vision-language dataset for autonomous driving with counterfactual reasoning. In Proceedings of the Computer Vision and Pattern Recognition Conference , 2244222452.
  38. Wang, Y.; cheng, Z.; Yao, Y.; and zhang, W. 2024b. OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning. arXiv:2405.00835. · arXiv
  39. Wang, Z.; Bovik, A. C.; Sheikh, H. R.; and Simoncelli, E. P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4): 600-612.
  40. Wen, L.; chen, Y.; Li, Y.; Zhang, L.; Cui, C.; Zong, G.; Liu, J.; Li, R.; Yue, X.; and Qiao, Y. 2024. SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving. arXiv:2405.02213. · arXiv
  41. Wu, P.; Jia, X.; Chen, L.; Yan, J.; Li, H.; and Qiao, Y. 2022. Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong baseline. Advances in Neural Information Processing Systems , 35: 6119-6132.
  42. Xie, Z.; Zhang, Z.; Cao, Y.; Lin, Y .; Bao, J.; Yao, Z.; Dai, Q.; and Hu, H. 2022. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 96539663.
  43. Xu, Z.; Zhang, Y.; Xie, E.; Zhao, Z.; Guo, Y.; Zhang, L.; Qiao, Y.; Li, Z.; and Liu, S. 2024. DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model. arXiv:2310.05739. · arXiv
  44. Yang, C.; Sui, Y.; Xiao, J.; Huang, L.; Gong, Y.; Li, C.; Yan, J.; Bai, Y .; Sadayappan, P.; Hu, X.; et al. 2025a. Topv: Compatible token pruning with inference time optimization for fast and low-memory multimodal vision language model. In Proceedings of the Computer Vision and Pattern Recognition Conference , 19803-19813.
  45. Yang, S.; Chen, Y.; Tian, Z.; Wang, C.; Li, J.; Yu, B.; and Jia, J. 2025b. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference , 19792-19802.
  46. Zeng, W.; Wong, K.; Sun, J.; Tang, B.; Huer, F.; Girish, B. J.; Lu, F.; Chen, Y.; Xu, R.; Li, H.; Qiao, Y.; and Liu, S. 2024. OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model. arXiv:2405.12212. · arXiv
  47. Zhang, Q.; Cheng, A.; Lu, M.; Zhang, R.; Zhuo, Z.; Cao, J.; Guo, S.; She, Q.; and Zhang, S. 2024a. Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms. arXiv preprint arXiv:2412.01818 . · arXiv
  48. Zhang, Q.; Liu, M.; Li, L.; Lu, M.; Zhang, Y.; Pan, J.; She, Q.; and Zhang, S. 2025a. Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs. arXiv preprint arXiv:2506.10967 . · arXiv
  49. Zhang, S.; Fang, Q.; Yang, Z.; and Feng, Y. 2025b. Llavamini: Efficient image and video large multimodal models with one vision token. arXiv preprint arXiv:2501.03895 . · arXiv
  50. Zhang, Y.; Fan, C.-K.; Ma, J.; Zheng, W.; Huang, T.; Cheng, K.; Gudovskiy, D.; Okuno, T.; Nakata, Y.; Keutzer, K.; et al. 2024b. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417 . · arXiv
  51. Zhang, Z.; Liniger, A.; Dai, D.; Yu, F.; and Van Gool, L. 2021. End-to-end urban driving by imitating a reinforcement learning coach. In Proceedings of the IEEE/CVF international conference on computer vision , 15222-15232.
  52. Zhao, S.; Wang, Z.; Juefei-Xu, F.; Xia, X.; Liu, M.; Wang, X.; Liang, M.; Zhang, N.; Metaxas, D. N.; and Yu, L. 2025. Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction. In Proceedings of the Computer Vision and Pattern Recognition Conference , 29869-29879.

Cited by

Sorted by citations

Authors
Cao, J., Zhang, Q., Jia, P., Zhao, X., Lan, B., Zhang, X., Li, Z., Wei, X., Chen, S., Li, L., Liu, X., Lu, M., Wang, Y., & Zhang, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2507.23318
arXiv
2507.23318
OpenAlex
W4414920245
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-sa/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
vision-language-action models, visual token pruning, autonomous driving, pixel reconstruction, adversarial training, computational efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review