The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Unbiased Scene Graph Generation From Biased Training
Open accessAbstract
Today's scene graph generation (SGG) task is still far from practical, mainly due to the severe training bias, e.g., collapsing diverse "human walk on / sit on / lay on beach" into "human on beach". Given such SGG, the down-stream tasks such as VQA can hardly infer better scene structures than merely a bag of objects. However, debiasing in SGG is not trivial because traditional debiasing methods cannot distinguish between the good and bad bias, e.g., good context prior (e.g., "person read book" rather than "eat") and bad long-tailed bias (e.g., "near" dominating "behind / in front of"). In this paper, we present a novel SGG framework based on causal inference but not the conventional likelihood. We first build a causal graph for SGG, and perform traditional biased training with the graph. Then, we propose to draw the counterfactual causality from the trained graph to infer the effect from the bad bias, which should be removed. In particular, we use Total Direct Effect (TDE) as the proposed final predicate score for unbiased SGG. Note that our framework is agnostic to any SGG model and thus can be widely applied in the community who seeks unbiased predictions. By using the proposed Scene Graph Diagnosis toolkit on the SGG benchmark Visual Genome and several prevailing models, we observed significant improvements over the previous state-of-the-art methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
MacKinnon, D. P., Fairchild, A. J., & Fritz, M. S. (2007) · Annual Review of Psychology · doi:10.1146/annurev.psych.58.110405.085542 · Cited by 5,688
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
Yan, S., Xiong, Y., & Lin, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12328 · Cited by 5,410 · cited 1×
“— yet on which are our core efforts made [69, 53, 6], then pretend that there is a graph — nothing but a sparse object layout with binary links, and finally shroud it into graph neural networks [63] for merely more contextual object representations [65, 17, 54].”
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks
Tai, K. S., Socher, R., & Manning, C. D. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1150 · Cited by 3,081
Causality: models, reasoning, and inference
(2000) · Choice Reviews Online · doi:10.5860/choice.38-1494a · Cited by 2,873
Torralba, A., & Efros, A. A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995347 · Cited by 2,440
Explanation in Causal Inference: Methods for Mediation and Interaction
VanderWeele, T. J. (2015) · Cited by 2,217
Causal Inference in Statistics
Pearl, J., Glymour, M., & Jewell, N. P. (2016) · Cited by 2,024
The Book of Why: The New Science of Cause and Effect
Goldberg, L. R. (2019) · Notices of the American Mathematical Society · doi:10.1090/noti1912 · Cited by 1,655
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., & Brendel, W. (2018) · arXiv · doi:10.48550/arxiv.1811.12231 · Cited by 1,490 · cited 2×
“With the power of counterfactual causality, we can remove the harmful bias from the good context bias, which cannot be easily identified by traditional debiasing methods such as data augmentation [9, 11] and unbiased learning [30].”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
Hudson, D. A., & Manning, C. D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00686 · Cited by 1,451
Scene Graph Generation by Iterative Message Passing
Xu, D., Zhu, Y., Choy, C. B., & Fei-Fei, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.330 · Cited by 1,271
Roese, N. J. (1997) · Psychological Bulletin · doi:10.1037/0033-2909.121.1.133 · Cited by 1,156 · cited 1×
“It means “counter to the facts” [46], and takes one step further that assigns the “clash of worlds” combination of values to variables.”
(2013) · Cited by 1,126
Image retrieval using scene graphs
Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D. A., Bernstein, M. S., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298990 · Cited by 1,101
Visual Relationship Detection with Language Priors
Lu, C., Krishna, R., Bernstein, M., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_51 · Cited by 1,069
Neural Motifs: Scene Graph Parsing with Global Context
Zellers, R., Yatskar, M., Thomson, S., & Choi, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00611 · Cited by 1,052
Pearl, J. (2022) · Probabilistic and Causal Inference · doi:10.1145/3501714.3501736 · Cited by 1,033
Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995
Exploring Visual Relationship for Image Captioning
Yao, T., Pan, Y., Li, Y., & Mei, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_42 · Cited by 870
Graph R-CNN for Scene Graph Generation
Yang, J., Lu, J., Lee, S., Batra, D., & Parikh, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_41 · Cited by 851
Image Generation from Scene Graphs
Johnson, J., Gupta, A., & Fei-Fei, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00133 · Cited by 836
Auto-Encoding Scene Graphs for Image Captioning
Yang, X., Tang, K., Zhang, H., & Cai, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01094 · Cited by 827
Self-citationEdge and Curve Detection for Visual Scene Analysis
Rosenfeld, A., & Thurston, M. (1971) · IEEE Transactions on Computers · doi:10.1109/t-c.1971.223290 · Cited by 781
Mediation analysis in epidemiology: methods, interpretation and bias
Richiardi, L., Bellocco, R., & Zugna, D. (2013) · International Journal of Epidemiology · doi:10.1093/ije/dyt127 · Cited by 781
Simon, H. A. (1990) · Utility and Probability · doi:10.1007/978-1-349-20568-4_5 · Cited by 705
A Political Mediation Model of Corporate Response to Social Movement Activism
King, B. G. (2008) · Administrative Science Quarterly · doi:10.2189/asqu.53.3.395 · Cited by 682
Visual Translation Embedding Network for Visual Relation Detection
Zhang, H., Kyaw, Z., Chang, S.-F., & Chua, T.-S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.331 · Cited by 586
Self-citationDon't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Agrawal, A., Batra, D., Parikh, D., & Kembhavi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00522 · Cited by 566
Learning to Compose Dynamic Tree Structures for Visual Contexts
Tang, K., Zhang, H., Wu, B., Luo, W., & Liu, W. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00678 · Cited by 543
Self-citationDetecting Visual Relationships with Deep Relational Networks
Dai, B., Zhang, Y., & Lin, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.352 · Cited by 502
Scene Graph Generation from Objects, Phrases and Region Captions
Li, Y., Ouyang, W., Zhou, B., Wang, K., & Wang, X. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.142 · Cited by 500
Graph-Structured Representations for Visual Question Answering
Teney, D., Liu, L., & Van Den Hengel, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.344 · Cited by 447
Knowledge-Embedded Routing Network for Scene Graph Generation
Chen, T., Yu, W., Chen, R., & Lin, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00632 · Cited by 432
Counterfactual VQA: A Cause-Effect Look at Language Bias
Niu, Y., Tang, K., Zhang, H., Lu, Z., Hua, X.-S., & Wen, J.-R. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01251 · Cited by 418
Self-citationWomen Also Snowboard: Overcoming Bias in Captioning Models
Hendricks, L. A., Burns, K., Saenko, K., Darrell, T., & Rohrbach, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_47 · Cited by 404
Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval
Schuster, S., Krishna, R., Chang, A., Fei-Fei, L., & Manning, C. D. (2015) · Proceedings of the Fourth Workshop on Vision and Language · doi:10.18653/v1/w15-2812 · Cited by 332
RESOUND: Towards Action Recognition Without Representation Bias
Li, Y., Li, Y., & Vasconcelos, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_32 · Cited by 332
Scene Graph Generation With External Knowledge and Image Reconstruction
Gu, J., Zhao, H., Lin, Z., Li, S., Cai, J., & Ling, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00207 · Cited by 308
Wang, T., Huang, J., Zhang, H., & Sun, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01077 · Cited by 291
Self-citationLearning Filter Pruning Criteria for Deep Convolutional Neural Networks Acceleration
He, Y., Ding, Y., Liu, P., Zhu, L., Zhang, H., & Yang, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00208 · Cited by 262
Self-citationGraphical Contrastive Losses for Scene Graph Parsing
Zhang, J., Shih, K. J., Elgammal, A., Tao, A., & Catanzaro, B. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01180 · Cited by 252
A Three-way Decomposition of a Total Effect into Direct, Indirect, and Interactive Effects
VanderWeele, T. J. (2013) · Epidemiology · doi:10.1097/ede.0b013e318281a64e · Cited by 242
Explainable and Explicit Visual Reasoning Over Scene Graphs
Shi, J., Zhang, H., & Li, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00857 · Cited by 223
Self-citationREPAIR: Removing Representation Bias by Dataset Resampling
Li, Y., & Vasconcelos, N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00980 · Cited by 223
Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels
Misra, I., Zitnick, C. L., Mitchell, M., & Girshick, R. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.320 · Cited by 199
RUBi: Reducing Unimodal Biases in Visual Question Answering
Cadène, R., Dancette, C., Ben-younes, H., Cord, M., & Parikh, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10169 · Cited by 194
Cognitive neuroscience of human counterfactual reasoning
Hoeck, N. V., Watson, P. D., & Barbey, A. K. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00420 · Cited by 174
Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition
Yin, G., Sheng, L., Liu, B., Yu, N., Wang, X., Shao, J., & Loy, C. C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_20 · Cited by 169
Deconfounded Image Captioning: A Causal Retrospect
Yang, X., Zhang, H., & Cai, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3121705 · Cited by 169
Self-citationThe Statistics of Causal Inference: A View from Political Methodology
Keele, L. (2015) · Political Analysis · doi:10.1093/pan/mpv007 · Cited by 168
Counterfactual Critic Multi-Agent Training for Scene Graph Generation
Chen, L., Zhang, H., Xiao, J., He, X., Pu, S., & Chang, S.-F. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00471 · Cited by 166
Self-citationAttentive Relational Networks for Mapping Images to Scene Graphs
Qi, M., Li, W., Yang, Z., Wang, Y., & Luo, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00408 · Cited by 165
Two Causal Principles for Improving Visual Dialog
Qi, J., Niu, Y., Huang, J., & Zhang, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01087 · Cited by 152
Self-citationDunn, G., Emsley, R., Liu, H., Landau, S., Green, J., White, I. R., & Pickles, A. R. (2015) · Health Technology Assessment · doi:10.3310/hta19930 · Cited by 131
Influence of resampling on accuracy of imbalanced classification
Burnaev, E. V., Erofeev, P., & Papanov, A. (2015) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2228523 · Cited by 113
Exploring Context and Visual Pattern of Relationship for Scene Graph Generation
Wang, W., Wang, R., Shan, S., & Chen, X. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00838 · Cited by 111
Krishna, R., Chami, I., Bernstein, M. S., & Fei-Fei, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00718 · Cited by 108
Explicit Bias Discovery in Visual Question Answering Models
Manjunatha, V., Saini, N., & Davis, L. S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00979 · Cited by 108
VrR-VG: Refocusing Visually-Relevant Relationships
Liang, Y., Bai, Y., Zhang, W., Qian, X., Zhu, L., & Mei, T. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01050 · Cited by 83
Causal Induction from Visual Observations for Goal Directed TasksPreprint
Nair, S., Zhu, Y., Savarese, S., & Fei-Fei, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.01751 · Cited by 65
Deconfounded Image Captioning: A Causal RetrospectPreprint
Yang, X., Zhang, H., & Cai, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.03923 · Cited by 45
Self-citationMüller, D., & Judd, C. M. (2014) · Wiley eBooks · doi:10.1002/9781118445112.stat06589 · Cited by 25
Cited by
Counterfactual VQA: A Cause-Effect Look at Language Bias
Niu, Y., Tang, K., Zhang, H., Lu, Z., Hua, X.-S., & Wen, J.-R. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01251 · Cited by 418
Self-citationCounterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification
Rao, Y., Chen, G., Lu, J., & Zhou, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00106 · Cited by 380
Causal Intervention for Weakly-Supervised Semantic Segmentation
Zhang, D., Zhang, H., Tang, J., Hua, X., & Sun, Q. (2020) · arXiv · doi:10.48550/arxiv.2009.12547 · Cited by 353
Self-citationCross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering
Liu, Y., Li, G., & Lin, L. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3284038 · Cited by 348
A Comprehensive Survey of Scene Graphs: Generation and Application
Chang, X., Ren, P., Xu, P., Li, Z., Chen, X., & Hauptmann, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3137605 · Cited by 324
Metadata
- Authors
- Tang, K., Niu, Y., Huang, J., Shi, J., & Zhang, H.
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 3713-3722
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3035017890
- Cited by
- 755
- References
- 73
- FWCI
- 35.36
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.