FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering
Open accessAbstract
Generative models have recently started to outperform extractive models in Open Domain Question Answering, largely by leveraging their decoder to attend over multiple encoded passages and combining their information.However, generative models tend to be larger than extractive models due to the need for a decoder, run slower during inference due to auto-regressive decoder beam search, and their generated output often suffers from hallucinations.We propose to extend transformer encoders with the ability to fuse information from multiple passages, using global representation to provide cross-sample attention over all tokens across samples.Furthermore, we propose an alternative answer span probability calculation to better aggregate answer scores in the global space of all samples.Using our proposed method, we outperform the current stateof-the-art method by 2.5 Exact Match score on the Natural Question dataset while using only 25% of parameters and 35% of the latency during inference, and 4.4 Exact Match on We-bQuestions dataset.When coupled with synthetic data augmentation, we outperform larger models on the TriviaQA dataset as well.The latency and parameter savings of our method make it particularly attractive for open-domain question answering, as these models are often compute-intensive.Global Tokens Global Encoder Layer Question + Passage 1 Passage Encoder Layer Question + Passage 2 Passage Encoder Layer Span Classifier 𝑃𝑃 𝑠𝑠 (𝑠𝑠 1 ) Span Classifier 𝑃𝑃 𝑠𝑠 (𝑠𝑠 2 ) 𝑃𝑃 𝐴𝐴 ("𝑁𝑁𝑁𝑁𝑁𝑁 𝑌𝑌𝑌𝑌𝑌𝑌𝑌𝑌") Σ Multi-Head Attention (Global Encoder Layer)x N Encoder Layers Q KV
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
On NaturalQuestions, with extra corpus, retrieval augmented models are better than other methods.
“For example, on NaturalQuestions [62], with extra corpus, retrieval augmented models [52,58] are better than other methods.”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [58]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575 · see citing works
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works
Semantic Parsing on Freebase from Question-Answer Pairs
Berant, J., Chou, A., Frostig, R., & Liang, P. (2013) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1160 · Cited by 2,318 · see citing works
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · see citing works
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
Chen, C.-F. R., Fan, Q., & Panda, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00041 · Cited by 2,242 · see citing works
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.10555 · Cited by 1,844 · see citing works
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · see citing works
Question Answering For Toxicological Information Extraction
Ferreira, B. C. L., Gonçalo Oliveira, H., Amaro, H., et al. (2022) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/oasics.slate.2022.3 · Cited by 1,535 · see citing works
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.74 · Cited by 1,109 · see citing works
Quantifying Attention Flow in Transformers
Abnar, S., & Zuidema, W. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.385 · Cited by 995 · see citing works
Latent retrieval for weakly supervised open domain question answering
Lee, K., Chang, M.-W., & Toutanova, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1612 · Cited by 771 · see citing works
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Yin, P., Neubig, G., Yih, W.-T., & Riedel, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.745 · Cited by 487 · see citing works
Simple and Effective Multi-Paragraph Reading Comprehension
Clark, C., & Gardner, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1078 · Cited by 402 · see citing works
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Xiong, L., Xiong, C., Li, Y., Tang, K., Liu, J., Bennett, P. N., Ahmed, J., & Overwikj, A. (2021) · International Conference on Learning Representations · Cited by 376 · see citing works
(2020) · International Conference on Machine Learning · Cited by 358 · see citing works
Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering
Wang, Z., Ng, P., Ma, X., Nallapati, R. M., & Xiang, B. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1599 · Cited by 246 · see citing works
LongT5: Efficient Text-To-Text Transformer for Long Sequences
Guo, M., Ainslie, J., Uthus, D., Ontanon, S., Ni, J., Sung, Y.-H., & Yang, Y. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.55 · Cited by 230 · see citing works
LongT5: Efficient Text-To-Text Transformer for Long Sequences
Guo, M., Ainslie, J., Uthus, D., Ontanon, S., Ni, J., Sung, Y.-H., & Yang, Y. (2021) · arXiv · doi:10.48550/arxiv.2112.07916 · Cited by 230 · see citing works
Knowledge-Grounded Dialogue Generation with Pre-trained Language Models
Zhao, X., Wu, W., Xu, C., Tao, C., Zhao, D., & Yan, R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.272 · Cited by 168 · see citing works
PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them
Lewis, P., Wu, Y., Liu, L., Minervini, P., Küttler, H., Piktus, A., Stenetorp, P., & Riedel, S. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00415 · Cited by 153 · see citing works
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
Miech, A., Alayrac, J.-B., Laptev, I., Sivic, J., & Zisserman, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00970 · Cited by 149 · see citing works
Evidence Aggregation for Answer Re-Ranking in Open-Domain Question AnsweringPreprint
Wang, S., Yu, M., Jiang, J., Zhang, W., Guo, X., Chang, S., Wang, Z., Klinger, T., Tesauro, G. J., & Campbell, M. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05116 · Cited by 143 · see citing works
Entity-Based Knowledge Conflicts in Question Answering
Longpre, S., Perisetla, K., Chen, A., Ramesh, N., DuBois, C., & Singh, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.565 · Cited by 101 · see citing works
End-to-End Training of Neural Retrievers for Open-Domain Question Answering
Sachan, D. S., Patwary, M. A., Shoeybi, M., Kant, N., Ping, W., Hamilton, W. L., & Catanzaro, B. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.519 · Cited by 74 · see citing works
Z-Code++: A Pre-trained Language Model Optimized for Abstractive SummarizationPreprint
He, P., Peng, B., Lu, L., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2208.09770 · Cited by 55 · see citing works
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Cheng, H., Shen, Y., Liu, X., He, P., Chen, W., & Gao, J. (2021) · doi:10.18653/v1/2021.acl-long.240 · Cited by 43 · see citing works
Open Domain Question Answering with A Unified Knowledge Interface
Ma, K., Cheng, H., Liu, X., Nyberg, E., & Gao, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.113 · Cited by 37 · see citing works
R2-D2: A Modular Baseline for Open-Domain Question Answering
Fajcik, M., Docekal, M., Ondrej, K., & Smrz, P. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.73 · Cited by 36 · see citing works
KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question AnsweringPreprint
Yu, D., Zhu, C., Fang, Y., Yu, W., Wang, S., Yi‐chong, X., Ren, X., Yang, Y., & Zeng, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04330 · Cited by 24 · see citing works
You Only Need One Model for Open-domain Question Answering
Lee, H., Kedia, A., Lee, J., Paranjape, A., Manning, C. D., & Woo, K.-G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.198 · Cited by 21 · see citing works
Self-citationCheng, H., Chang, M.-W., Lee, K., & Toutanova, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.501 · Cited by 18 · see citing works
Attention-guided Generative Models for Extractive Question AnsweringPreprint
Xu, P., Liang, D., Huang, Z., & Xiang, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06393 · Cited by 14 · see citing works
Ilinykh, N., & Dobnik, S. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.320 · Cited by 8 · see citing works
Simple and Efficient ways to Improve REALM
Balachandran, V., Vaswani, A., Tsvetkov, Y., & Parmar, N. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.16 · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- Sabrina J. Mielke, Arthur Szlam, Y-Lan Boureau, and Emily Dinan. 2020. Linguistic calibration through metacognition: aligning dialogue agent re- sponses with expected correctness. ArXiv preprint, abs/2012.14983. · arXiv
- Devendra Singh, Siva Reddy, Will Hamilton, Chris Dyer, and Dani Yogatama. 2021. End-to-end train- ing of multi-document reader and retriever for open- domain question answering. Advances in Neural Information Processing Systems, 34.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
- Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Trans- formers for longer sequences. In Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurIPS 2019, De- cember 8-14, 2019, Vancouver, BC, Canada, pages 9051-9062.
Cited by
Sorted by citations
Harnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644 · see citing works
- Authors
- Kedia, A., Zaidi, M. A., & Lee, H.
- Affiliations
- Samsung (South Korea) (KR)
- Venue
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Published
- 2022
- Pages
- 4246-4260
- OpenAlex
- W4385572958
- Cited by
- 14
- References
- 38
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 2 · 2023: 4
- FWCI
- 1.38
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- open-domain question answering, feature-level fusion, parameter efficiency, inference latency, hallucination mitigation, synthetic data augmentation, Natural Questions, TriviaQA
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
