Cited by 17Digital Media Forensic DetectionMultimodal Machine Learning ApplicationsGenerative Adversarial Networks and Image Synthesis

Exploiting Modality-Specific Features for Multi-Modal Manipulation Detection and Grounding

Open access
Wang, Jiazhen, Liu, Bin, Miao, Changtao, Zhao, Zhiwei, Zhuang, Wanyi, Chu, Qi, Yu, Nenghai·2024·ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Abstract

AI-synthesized text and images have gained significant attention, particularly due to the widespread dissemination of multi-modal manipulations on the internet, which has resulted in numerous negative impacts on society. Existing methods for multi-modal manipulation detection and grounding primarily focus on fusing vision-language features to make predictions, while overlooking the importance of modality-specific features, leading to sub-optimal results. In this paper, we construct a simple and novel transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. To achieve this, we introduce visual/language pre-trained encoders and dual-branch cross-attention (DCA) to extract and fuse modality-unique features. Furthermore, we design decoupled fine-grained classifiers (DFC) to enhance modality-specific feature mining and mitigate modality competition. Moreover, we propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality using learnable queries, thereby improving the discovery of forged details. Extensive experiments on the DGM4dataset demonstrate the superior performance of our proposed model compared to state-of-the-art approaches.

Integrity signals

Registry

Crossref · 10.1109/icassp48485.2024.10448385

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The performance of any deepfake detection method is highly dependent on the quantitative and qualitative aspects of the datasets used for development.

    “The performance of any deepfake detection method is highly dependent on the quantitative and qualitative aspects of the datasets used for development [18, 21, 25, 40, 44, 47, 49, 51, 52, 61, 63, 69, 72, 75].”
  • Wang et al. [241] emphasized exploiting modality-specific features for enhanced detection and grounding of multi-modal manipulations.

    “Wang et al. [241] emphasized exploiting modalityspecific features for enhanced detection and grounding of multi-modal manipulations.”
  • Wang et al.'s framework incorporates modality-specific fine-grained classifiers to improve manipulation classification accuracy.

    “incorporates modalityspecific fine-grained classifiers to improve manipulation classification accuracy.”
  • Wang et al.'s dual-branch cross-attention framework facilitates bi-directional interaction between text and visual modalities.

    “Their dual-branch crossattention framework facilitates bi-directional interaction between text and visual modalities”

Cites

Flagged first, then by citations

1 of the 28 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Wang, J., Liu, B., Miao, C., Zhao, Z., Zhuang, W., Chu, Q., & Yu, N.
Affiliations
University of Science and Technology of China (CN)
Venue
ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Publisher
IEEE
Published
2024-04-14
Pages
4935-4939
ISSN
1520-6149 (ISSN-L)
OpenAlex
W4392903163
Cited by
12
References
28
Citations by year
2026: 4 · 2025: 8
FWCI
2.49
Type
conference paper
Language
en
Open access
Locations
  • IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Natural Science Foundation of China
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
modality-specific features, multimodal alignment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review