CoChat
Sign Up
PreprintCited by 0Topic ModelingHandwritten Text Recognition TechniquesNatural Language Processing Techniques

Memory-augmented agent training for business document understanding

Open access
Liu, Jiale, Zeng, Yifan, Højmark-Bertelsen, Malte, Gadeberg, Marie Normann, Wang, Huazheng, Wu, Qingyun·2024·arXiv
DOI 10.48550/arxiv.2412.15274arXiv 2412.15274
Open this paper in CoChat — free accountOpen PDF

Abstract

Traditional enterprises face significant challenges in processing business documents, where tasks like extracting transport references from invoices remain largely manual despite their crucial role in logistics operations. While Large Language Models offer potential automation, their direct application to specialized business domains often yields unsatisfactory results. We introduce Matrix (Memory-Augmented agent Training through Reasoning and Iterative eXploration), a novel paradigm that enables LLM agents to progressively build domain expertise through experience-driven memory refinement and iterative learning. To validate this approach, we collaborate with one of the world's largest logistics companies to create a dataset of Universal Business Language format invoice documents, focusing on the task of transport reference extraction. Experiments demonstrate that Matrix outperforms prompting a single LLM by 30.3%, vanilla LLM agent by 35.2%. We further analyze the metrics of the optimized systems and observe that the agent system requires less API calls, fewer costs and can analyze longer documents on average. Our methods establish a new approach to transform general-purpose LLMs into specialized business tools through systematic memory enhancement in document processing tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2412.15274

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Matrix, SAGE, and R2D2 compare reasoning traces against ground-truth answers to identify error sources and extract reflective insights.

    “Others adopt failure-driven reflection, exemplified by Matrix (Liu et al., 2024), SAGE (Liang et al., 2025), and R2D2 (Huang et al., 2025c), which compare reasoning traces against ground-truth answers to identify error sources and extract reflective insights.”

    From Memory in the Age of AI Agents (2025) · cites this paper as Liu et al., 2024

  • Matrix performs iterative optimization to combine execution trajectories and reflective insights with global memory, distilling task-agnostic principles that support reuse across scenarios.

    “For experiential memory, Matrix (Liu et al., 2024) performs iterative optimization to combine execution trajectories and reflective insights with global memory, distilling task-agnostic principles that support reuse across scenarios.”

    From Memory in the Age of AI Agents (2025) · cites this paper as Liu et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 26

Research works printed but not linked: 66

  1. In Burstein, J.; Doran, C.; and Solorio, T., eds., Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 307-317. Minneapolis, Minnesota: Association for Compu- tational Linguistics.
  2. Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
  3. Antonacopoulos, A.; Bridson, D.; Papadopoulos, C.; and Pletschacher, S. 2009. A realistic dataset for performance evaluation of document layout analysis. In 2009 10th In- ternational Conference on Document Analysis and Recogni- tion, 296-300. IEEE.
  4. Bajaj, P.; Campos, D.; Craswell, N.; Deng, L.; Gao, J.; Liu, X.; Majumder, R.; McNamara, A.; Mitra, B.; Nguyen, T.; et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268. · arXiv
  5. Brown, T. B. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. · arXiv
  6. Cai, T.; Wang, X.; Ma, T.; Chen, X.; and Zhou, D. 2023. Large language models as tool makers. arXiv preprint arXiv:2305.17126. · arXiv
  7. Cheng, J.; Liu, X.; Zheng, K.; Ke, P.; Wang, H.; Dong, Y.; Tang, J.; and Huang, M. 2023. Black-box prompt optimiza- tion: Aligning large language models without model train- ing. arXiv preprint arXiv:2311.04155. · arXiv
  8. Dong, Q.; Li, L.; Dai, D.; Zheng, C.; Wu, Z.; Chang, B.; Sun, X.; Xu, J.; and Sui, Z. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234. · arXiv
  9. Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. · arXiv
  10. Guo, X.; Huang, K.; Liu, J.; Fan, W.; Vélez, N.; Wu, Q.; Wang, H.; Griffiths, T. L.; and Wang, M. 2024. Embod- ied llm agents learn to cooperate in organized teams. arXiv preprint arXiv:2403.12482. · arXiv
  11. Hamdi, A.; Carel, E.; Joseph, A.; Coustaty, M.; and Doucet, A. 2021. Information extraction from invoices. In Interna- tional Conference on Document Analysis and Recognition, 699-714. Springer.
  12. Harley, A. W.; Ufkes, A.; and Derpanis, K. G. 2015. Evalua- tion of deep convolutional nets for document image classifi- cation and retrieval. In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), 991-995. IEEE.
  13. Hu, S.; Lu, C.; and Clune, J. 2024. Automated design of agentic systems. arXiv preprint arXiv:2408.08435. · arXiv
  14. Huang, J.; Chen, X.; Mishra, S.; Zheng, H. S.; Yu, A. W.; Song, X.; and Zhou, D. 2023. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798. · arXiv
  15. Ji, Z.; Yu, T.; Xu, Y.; Lee, N.; Ishii, E.; and Fung, P. 2023. Towards mitigating LLM hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, 1827-1843.
  16. Jiang, D.; Zhang, J.; Weller, O.; Weir, N.; Van Durme, B.; and Khashabi, D. 2024. Self-[in] correct: Llms strug- gle with refining self-generated responses. arXiv preprint arXiv:2404.04298. · arXiv
  17. Joshi, M.; Choi, E.; Weld, D.; and Zettlemoyer, L. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Barzilay, R.; and Kan, M.-Y., eds., Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1601-1611. Vancouver, Canada: Association for Computational Linguistics.
  18. Kamoi, R.; Zhang, Y.; Zhang, N.; Han, J.; and Zhang, R. 2024. When Can LLMs Actually Correct Their Own Mis- takes? A Critical Survey of Self-Correction of LLMs. arXiv preprint arXiv:2406.01297. · arXiv
  19. Krieger, F.; Drews, P.; Funk, B.; and Wobbe, T. 2021. In- formation extraction from invoices: a graph neural network approach for datasets with high layout variety. In Innova- tion Through Information Systems: Volume II: A Collection of Latest Research on Technology Issues, 5-20. Springer.
  20. Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K.; et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Associa- tion for Computational Linguistics, 7: 453-466.
  21. Liu, J.; Shen, D.; Zhang, Y.; Dolan, B.; Carin, L.; and Chen, W. 2021. What Makes Good In-Context Examples for GPT- 3? arXiv preprint arXiv:2101.06804. · arXiv
  22. Lu, Y.; Bartolo, M.; Moore, A.; Riedel, S.; and Stenetorp, P. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786. · arXiv
  23. Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; et al. 2024. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Sys- tems, 36.
  24. Min, S.; Lewis, M.; Zettlemoyer, L.; and Hajishirzi, H. 2021. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943. · arXiv
  25. Palm, R. B.; Winther, O.; and Laws, F. 2017. Cloudscan- a configuration-free invoice analysis system using recurrent neural networks. In 2017 14th IAPR International Confer- ence on Document Analysis and Recognition (ICDAR), vol- ume 1, 406-413. IEEE.
  26. Pan, L.; Saxon, M.; Xu, W.; Nathani, D.; Wang, X.; and Wang, W. Y. 2023. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.03188. · arXiv
  27. Prasad, A.; Hase, P.; Zhou, X.; and Bansal, M. 2022. Grips: Gradient-free, edit-based instruction search for prompting large language models. arXiv preprint arXiv:2203.07281. · arXiv
  28. Pryzant, R.; Iter, D.; Li, J.; Lee, Y. T.; Zhu, C.; and Zeng, M. 2023. Automatic prompt optimization with" gradient de- scent" and beam search. arXiv preprint arXiv:2305.03495. · arXiv
  29. Qian, C.; Han, C.; Fung, Y. R.; Qin, Y.; Liu, Z.; and Ji, H. 2023. Creator: Tool creation for disentangling abstract and concrete reasoning of large language models. arXiv preprint arXiv:2305.14318. · arXiv
  30. Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Su, J.; Duh, K.; and Carreras, X., eds., Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2383-2392. Austin, Texas: Association for Computational Linguistics.
  31. Riba, P.; Dutta, A.; Goldmann, L.; Fornés, A.; Ramos, O.; and Lladós, J. 2019. Table detection in invoice documents by graph neural networks. In 2019 International Conference on Document Analysis and Recognition (ICDAR), 122-127. IEEE.
  32. Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2024. Reflexion: Language agents with verbal re- inforcement learning. Advances in Neural Information Pro- cessing Systems, 36.
  33. Song, L.; Liu, J.; Zhang, J.; Zhang, S.; Luo, A.; Wang, S.; Wu, Q.; and Wang, C. 2024. Adaptive In-conversation Team Building for Language Model Agents. arXiv preprint arXiv:2405.19425. · arXiv
  34. Su, H.; Kasai, J.; Wu, C. H.; Shi, W.; Wang, T.; Xin, J.; Zhang, R.; Ostendorf, M.; Zettlemoyer, L.; Smith, N. A.; et al. 2022. Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975. · arXiv
  35. Sumers, T. R.; Yao, S.; Narasimhan, K.; and Griffiths, T. L. 2023. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427. · arXiv
  36. Talmor, A.; and Berant, J. 2018. The Web as a Knowledge- base for Answering Complex Questions. In North American Association for Computational Linguistics (NAACL).
  37. Tarawneh, A. S.; Hassanat, A. B.; Chetverikov, D.; Lendak, I.; and Verma, C. 2019. Invoice classification using deep fea- tures and machine learning techniques. In 2019 IEEE Jor- dan International Joint Conference on Electrical Engineer- ing and Information Technology (JEEIT), 855-859. IEEE.
  38. Wang, C.; Liu, X.; Yue, Y.; Tang, X.; Zhang, T.; Jiayang, C.; Yao, Y.; Gao, W.; Hu, X.; Qi, Z.; et al. 2023. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. arXiv preprint arXiv:2310.07521. · arXiv
  39. Wang, C.; Wu, Q.; Weimer, M.; and Zhu, E. 2021. Flaml: A fast and lightweight automl library. Proceedings of Machine Learning and Systems, 3: 434-447.
  40. Wang, Z. Z.; Mao, J.; Fried, D.; and Neubig, G. 2024. Agent workflow memory. arXiv preprint arXiv:2409.07429. · arXiv
  41. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of- thought prompting elicits reasoning in large language mod- els. Advances in neural information processing systems, 35: 24824-24837.
  42. Wu, Q.; Bansal, G.; Zhang, J.; Wu, Y.; Zhang, S.; Zhu, E.; Li, B.; Jiang, L.; Zhang, X.; and Wang, C. 2023. Autogen: Enabling next-gen llm applications via multi-agent conver- sation framework. arXiv preprint arXiv:2308.08155. · arXiv
  43. Wu, Y.; Jia, F.; Zhang, S.; Li, H.; Zhu, E.; Wang, Y.; Lee, Y. T.; Peng, R.; Wu, Q.; and Wang, C. 2024a. MathChat: Converse to Tackle Challenging Math Problems with LLM Agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.
  44. Wu, Y.; Yue, T.; Zhang, S.; Wang, C.; and Wu, Q. 2024b. StateFlow: Enhancing LLM Task-Solving through State- Driven Workflows. arXiv preprint arXiv:2403.11322. · arXiv
  45. Wu, Z.; Wang, Y.; Ye, J.; and Kong, L. 2022. Self-adaptive in-context learning: An information compression perspec- tive for in-context example selection and ordering. arXiv preprint arXiv:2212.10375. · arXiv
  46. Xia, X.; Liu, J.; Yu, J.; Shen, X.; Han, B.; and Liu, T. 2022. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh In- ternational Conference on Learning Representations.
  47. Xia, X.; Liu, J.; Zhang, S.; Wu, Q.; Wei, H.; and Liu, T. 2024. Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraints. In Forty-first International Conference on Machine Learning.
  48. Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. · arXiv
  49. Yang, C.; Wang, X.; Lu, Y.; Liu, H.; Le, Q. V.; Zhou, D.; and Chen, X. 2023. Large language models as optimizers. arXiv preprint arXiv:2309.03409. · arXiv
  50. Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W. W.; Salakhutdinov, R.; and Manning, C. D. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question An- swering. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
  51. Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; and Narasimhan, K. 2024. Tree of thoughts: Deliber- ate problem solving with large language models. Advances in Neural Information Processing Systems, 36.
  52. Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR).
  53. Yuan, L.; Chen, Y.; Wang, X.; Fung, Y. R.; Peng, H.; and Ji, H. 2023. Craft: Customizing llms by creating and retrieving from specialized toolsets. arXiv preprint arXiv:2309.17428. · arXiv
  54. Zhang, J.; Xiang, J.; Yu, Z.; Teng, F.; Chen, X.; Chen, J.; Zhuge, M.; Cheng, X.; Hong, S.; Wang, J.; et al. 2024a. AFlow: Automating Agentic Workflow Generation. arXiv preprint arXiv:2410.10762. · arXiv
  55. Zhang, S.; Wu, Y.; Zheng, Z.; Wu, Q.; and Wang, C. 2024b. Hypertime: Hyperparameter optimization for com- bating temporal distribution shifts. In Proceedings of the 32nd ACM International Conference on Multimedia, 4610- 4619.
  56. Zhang, S.; Xia, X.; Wang, Z.; Chen, L.-H.; Liu, J.; Wu, Q.; and Liu, T. 2023b. Ideal: Influence-driven selective annota- tions empower in-context learners in large language models. arXiv preprint arXiv:2310.10873. · arXiv
  57. Zhang, S.; Zhang, J.; Liu, J.; Song, L.; Wang, C.; Krishna, R.; and Wu, Q. 2024c. Offline Training of Language Model Agents with Functions as Learnable Weights. In Forty-first International Conference on Machine Learning.
  58. Zhao, A.; Huang, D.; Xu, Q.; Lin, M.; Liu, Y.-J.; and Huang, G. 2024. Expel: Llm agents are experiential learners. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 38, 19632-19642.
  59. Zhao, Z.; Wallace, E.; Feng, S.; Klein, D.; and Singh, S. 2021. Calibrate Before Use: Improving Few-shot Perfor- mance of Language Models. In Meila, M.; and Zhang, T., eds., Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 12697-12706.
  60. PMLR.
  61. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2023a. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 46595-46623.
  62. Zheng, X.; Yang, C.; Zhang, S.; Wang, Y.; Zhang, B.; Wu, Y.; Wu, Y.; Shao, L.; and Ji, R. 2023b. Ddpnas: Efficient neural architecture search via dynamic distribution pruning. International Journal of Computer Vision, 131(5): 1234- 1249.
  63. Zhong, X.; Tang, J.; and Yepes, A. J. 2019. Publaynet: largest dataset ever for document layout analysis. In 2019 International conference on document analysis and recog- nition (ICDAR), 1015-1022. IEEE.
  64. Zhou, D.; Schärli, N.; Hou, L.; Wei, J.; Scales, N.; Wang, X.; Schuurmans, D.; Cui, C.; Bousquet, O.; Le, Q.; et al. 2022a. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625. · arXiv
  65. Zhou, Y.; Muresanu, A. I.; Han, Z.; Paster, K.; Pitis, S.; Chan, H.; and Ba, J. 2022b. Large language mod- els are human-level prompt engineers. arXiv preprint arXiv:2211.01910. · arXiv
  66. Zhuge, M.; Wang, W.; Kirsch, L.; Faccio, F.; Khizbullin, D.; and Schmidhuber, J. 2024. Language agents as optimizable graphs. arXiv preprint arXiv:2402.16823. · arXiv

Authors
Liu, J., Zeng, Y., Højmark-Bertelsen, M., Gadeberg, M. N., Wang, H., & Wu, Q.
Affiliations
  • Pennsylvania State University (US)
  • Oregon State University (US)
Corresponding author
Liu, Jiale
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
39-53
ISSN
1865-0929 (ISSN-L)
DOI
10.48550/arxiv.2412.15274
arXiv
2412.15274
OpenAlex
W7117961343
Cited by
0
References
26
FWCI
0.00
Type
conference paper
Language
en
Open access
Licence
Locations
  • Communications in computer and information science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
memory-augmented agents, LLM agents, invoice processing, logistics, iterative learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review