Code4Struct: Code Generation for Few-Shot Event Structure Prediction
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Large Language Model (LLM) trained on a mixture of text and code has demonstrated impressive capability in translating natural language (NL) into structured code.We observe that semantic structures can be conveniently translated into code and propose CODE4STRUCT to leverage such text-tostructure translation capability to tackle structured prediction tasks.As a case study, we formulate Event Argument Extraction (EAE) as converting text into event-argument structures that can be represented as a class object using code.This alignment between structures and code enables us to take advantage of Programming Language (PL) features such as inheritance 1 and type annotation 2 to introduce external knowledge or add constraints.We show that, with sufficient in-context examples, formulating EAE as a code generation problem is advantageous over using variants of text-based prompts.Despite only using 20 training event instances for each event type, CODE4STRUCT is comparable to supervised models trained on 4,202 instances and outperforms current stateof-the-art (SOTA) trained on 20-shot data by 29.5% absolute F1.By leveraging the inheritance feature of PL, CODE4STRUCT can use 10-shot training data from a sibling event type to predict arguments for zero-resource event types and outperforms the zero-shot baseline by 12% absolute F1. 3 Event Argument Extraction Programming Language (Python) Event / Entity Type Transport, VEH Class definition class Transport, class VEH Hierarchical Event Ontology Movement:Transport Inheritance Inheritance is a way to create a hierarchy of classes in PL.A child class can base upon another class, retaining similar implementation.class Transport(Movement) Event Arguments vehicle Function arguments def function(vehicle=...) Argument ConstraintEach argument can has a list of multiple entities; Argument vehicle should be entities of type VEH. Type Annotation & Argument Default ValueType annotations are used by developers to indicate the data types of variables and input/outputs of functions.If a function is called without the argument, the argument gets its default value (a list in this case). def function( vehicle: List[VEH] = [], … ) Weakly-supervised InformationTransport Event describes someone transporting something in a vehicle from one place to another place. Docstring or Commentsclass Transport(Movement):""" self.agenttransported self.artifact in self.vehiclevehicle from self.origin place to self.destination place."""
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Self-citationEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · cited 1×
“Surprisingly, similar to the observation made by Min et al. (2022), using in-context examples from a random non-sibling type also benefits CODE4STRUCT performance, albeit not as helpful as sibling examples under 10-shot.”
MethodRethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2021) · arXiv · doi:10.48550/arxiv.2104.08786 · Cited by 615 · cited 3×
“We expect better selecting (Liu et al., 2021) and ordering (Lu et al., 2022) incontext examples can benefit CODE4STRUCT performance, which we leave for future work.”
BackgroundMethodProgPrompt: Generating Situated Robot Task Plans using Large Language Models
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., & Garg, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10161317 · Cited by 613 · cited 1×
“Code-LLM for Structured Task Sun et al. (2019); Singh et al. (2022) focus on procedural tasks that aim to control situated agents in an embodied environment by representing the procedure plan in code.”
BackgroundEntity, Relation, and Event Extraction with Contextualized Span Representations
Wadden, D., Wennberg, U., Luan, Y., & Hajishirzi, H. (2019) · arXiv · doi:10.48550/arxiv.1909.03546 · Cited by 612
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.556 · Cited by 610 · cited 3×
“We expect better selecting (Liu et al., 2021) and ordering (Lu et al., 2022) incontext examples can benefit CODE4STRUCT performance, which we leave for future work.”
BackgroundMethodCalibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · cited 1×
“Prior work (Zhao et al., 2021; Lu et al., 2022) has demonstrated a strong relationship between performance and in-context demonstrations; however, for ease of comparison to supervised baselines, we use the same set of examples from the training set for in-context learning.”
MethodPAL: Program-aided Language Models
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., & Neubig, G. (2022) · arXiv · doi:10.48550/arxiv.2211.10435 · Cited by 155 · cited 1×
“Gao et al. (2022) tackles math and symbolic reasoning tasks by decomposing the natural language problem into runnable steps using Code-LLM and delegating solution calculation to a PL interpreter.”
Method
Cited by
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144
Self-citationBhatia, S., Gandhi, T., Kumar, D., & Jalote, P. (2024) · Proceedings of the 1st International Workshop on Large Language Models for Code · doi:10.1145/3643795.3648396 · Cited by 55
EvEval: A Comprehensive Evaluation of Event Semantics for Large Language Models
Tao, Z., Jin, Z., Bai, X., Zhao, H., Feng, Y., Li, J., & Hu, W. (2023) · arXiv · doi:10.48550/arxiv.2305.15268 · Cited by 23
Retrieval-Augmented Code Generation for Universal Information Extraction
Guo, Y., Li, Z., Jin, X., Liu, Y., Zeng, Y., Liu, W., Li, X., Yang, P., Bai, L., Guo, J., et al. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-981-97-9434-8_3 · Cited by 23
(2023) · doi:10.18653/v1/2023.emnlp-main.824 · Cited by 4
Self-citation
Metadata
- Authors
- Wang, X., Li, S., & Ji, H.
- Venue
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2023
- Pages
- 3640-3663
- OpenAlex
- W4385571184
- Cited by
- 62
- References
- 13
- FWCI
- 11.72
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.