The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Study of Subjective and Objective Quality Assessment of Audio-Visual Signals
Abstract
The topics of visual and audio quality assessment (QA) have been widely researched for decades, yet nearly all of this prior work has focused only on single-mode visual or audio signals. However, visual signals rarely are presented without accompanying audio, including heavy-bandwidth video streaming applications. Moreover, the distortions that may separately (or conjointly) afflict the visual and audio signals collectively shape user-perceived quality of experience (QoE). This motivated us to conduct a subjective study of audio and video (A/V) quality, which we then used to compare and develop A/V quality measurement models and algorithms. The new LIVE-SJTU Audio and Video Quality Assessment (A/V-QA) Database includes 336 A/V sequences that were generated from 14 original source contents by applying 24 different A/V distortion combinations on them. We then conducted a subjective A/V quality perception study on the database towards attaining a better understanding of how humans perceive the overall combined quality of A/V signals. We also designed four different families of objective A/V quality prediction models, using a multimodal fusion strategy. The different types of A/V quality models differ in both the unimodal audio and video quality prediction models comprising the direct signal measurements and in the way that the two perceptual signal modes are combined. The objective models are built using both existing state-of-the-art audio and video quality prediction models and some new prediction models, as well as quality-predictive features delivered by a deep neural network. The methods of fusing audio and video quality predictions that are considered include simple product combinations as well as learned mappings. Using the new subjective A/V database as a tool, we validated and tested all of the objective A/V quality prediction models. We will make the database publicly available to facilitate further research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Self-citationLIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
Multiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253 · cited 10×
“Likewise, auditory masking principles are well-understood, and simultaneous masking is embodied by VQA models like SSIM and MS-SSIM [8], [9].”
BackgroundMethodInfluentialSelf-citationFSIM: A Feature Similarity Index for Image Quality Assessment
Zhang, L., Zhang, L., Mou, X., & Zhang, D. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2109730 · Cited by 5,203
Image information and visual quality
Sheikh, H. R., & Bovik, A. C. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.859378 · Cited by 4,104
Self-citationAn Algorithm for Intelligibility Prediction of Time–Frequency Weighted Noisy Speech
Taal, C. H., Hendriks, R. C., Heusdens, R., & Jensen, J. (2011) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2011.2114881 · Cited by 2,254
CNN architectures for large-scale audio classification
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952132 · Cited by 2,023
Evaluation of Objective Quality Measures for Speech Enhancement
Hu, Y., & Loizou, P. C. (2008) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2007.911054 · Cited by 1,830
Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index
Xue, W., Zhang, L., Mou, X., & Bovik, A. C. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2013.2293423 · Cited by 1,697
Self-citationA new standardized method for objectively measuring video quality
Pinson, M. H., & Wolf, S. (2004) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2004.834028 · Cited by 1,182 · cited 3×
“We used the following well-known video and audio quality predictors: • Video: VMAF [14], STRRED [13], SpEED [39], VQM [40], SSIM [8], MS-SSIM [9], VIFP [10], FSIM [41], GMSD [42]; • Audio: PEAQ [17], STOI [19], VISQOL [20], log-likelihood ratio (LLR) [21], signal-to-noise ratio (SNR), and segmental SNR (segSNR) [22].”
MethodContrast masking in human vision
Legge, G. E., & Foley, J. M. (1980) · Journal of the Optical Society of America · doi:10.1364/josa.70.001458 · Cited by 1,182
Study of Subjective and Objective Quality Assessment of Video
Seshadrinathan, K., Soundararajan, R., Bovik, A. C., & Cormack, L. K. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2042111 · Cited by 1,160
Self-citation(2009) · neural information processing systems · Cited by 962
Perceptual visual quality metrics: A survey
Lin, W., & Jay Kuo, C.-C. (2011) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2011.01.005 · Cited by 922
Natural signal statistics and sensory gain control
Schwartz, O., & Simoncelli, E. P. (2001) · Nature Neuroscience · doi:10.1038/90526 · Cited by 835
Motion Tuned Spatio-Temporal Quality Assessment of Natural Videos
Seshadrinathan, K., & Bovik, A. C. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2034992 · Cited by 708
Self-citationMethod, findings, and theory in studies of visual masking.
Kahneman, D. (1968) · Psychological Bulletin · doi:10.1037/h0026731 · Cited by 624
Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison
Chikkerur, S., Sundaram, V., Reisslein, M., & Karam, L. J. (2011) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2011.2104671 · Cited by 613
Blind Image Quality Estimation via Distortion Aggravation
Min, X., Zhai, G., Gu, K., Liu, Y., & Yang, X. (2018) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2018.2816783 · Cited by 508
Self-citationBlind Quality Assessment Based on Pseudo-Reference Image
Min, X., Gu, K., Zhai, G., Liu, J., Yang, X., & Chen, C. W. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2788206 · Cited by 450
Self-citationVideo Quality Assessment by Reduced Reference Spatio-Temporal Entropic Differencing
Soundararajan, R., & Bovik, A. C. (2013) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2012.2214933 · Cited by 368 · cited 4×
“Some of the most successful VQA models rely on natural scene statistics models to characterize quality, including VIF [10], STRRED [13], VMAF [14], SpEED [39], etc. Similar signal statistics have been observed to be fundamental to audio signals with similar implications for audio perception [52], [53].”
MethodInfluentialSelf-citationFILTERS FOR COMMON RESAMPLING TASKS
Turkowski, K. (1990) · Graphics Gems · doi:10.1016/b978-0-08-050753-8.50042-5 · Cited by 296
Automatic Prediction of Perceptual Image and Video Quality
Bovik, A. C. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2013.2257632 · Cited by 214
Self-citationStudy of Temporal Effects on Subjective Video Quality of Experience
Bampis, C. G., Li, Z., Moorthy, A. K., Katsavounidis, I., Aaron, A., & Bovik, A. C. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2729891 · Cited by 188
Self-citationA spatiotemporal most-apparent-distortion model for video quality assessment
Vu, P. V., Vu, C. T., & Chandler, D. M. (2011) · 2011 18th IEEE International Conference on Image Processing · doi:10.1109/icip.2011.6116171 · Cited by 180
A Basic Multimedia Quality Model
Hands, D. S. (2004) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2004.837233 · Cited by 163
SpEED-QA: Spatial Efficient Entropic Differencing for Image and Video Quality
Bampis, C. G., Gupta, P., Soundararajan, R., & Bovik, A. C. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2726542 · Cited by 150
Self-citationPerceptual-based quality assessment for audio–visual services: A survey
You, J., Reiter, U., Hannuksela, M. M., Gabbouj, M., & Perkis, A. (2010) · Signal Processing Image Communication · doi:10.1016/j.image.2010.02.002 · Cited by 129
A H.264/AVC video database for the evaluation of quality metrics
Francesca De Simone, A., Tagliasacchi, M., Naccari, M., Tubaro, S., & Ebrahimi, T. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5496296 · Cited by 126
Temporal Low-Order Statistics of Natural Sounds
Attias, H. T., & Schreiner, C. E. (1996) · Cited by 103
Audio-Visual Multimedia Quality Assessment: A Comprehensive Survey
Akhtar, Z., & Falk, T. H. (2017) · IEEE Access · doi:10.1109/access.2017.2750918 · Cited by 103
The Influence of Subjects and Environment on Audiovisual Subjective Tests: An International Study
Pinson, M., Janowski, L., Pépion, R., Huynh‐Thu, Q., Schmidmer, C., Corriveau, P., Younkin, A. C., Callet, P. L., Barkowsky, M., & Ingram, W. (2012) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2012.2215306 · Cited by 96
Recurrent and Dynamic Models for Predicting Streaming Video Quality of Experience
Bampis, C. G., Li, Z., Katsavounidis, I., & Bovik, A. C. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2815842 · Cited by 84
Self-citationPerceived Audiovisual Quality of Low-Bitrate Multimedia Content
WINKLER, S. S., & Faller, C. (2006) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2006.879871 · Cited by 79
Multiscale Approaches To Music Audio Feature Learning.
Dieleman, S., & Schrauwen, B. (2013) · Ghent University Academic Bibliography (Ghent University) · doi:10.5281/zenodo.1416675 · Cited by 77
Objective Assessment of Perceptual Audio Quality Using ViSQOLAudio
Sloan, C., Harte, N., Kelly, D., Kokaram, A. C., & Hines, A. (2017) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2017.2704421 · Cited by 53
Audio quality assessment using the mean structural similarity measure
Kandadai, S., Hardin, J. C., & Creusere, C. D. (2008) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2008.4517586 · Cited by 47
Combining audio and video metrics to assess audio-visual quality
Martinez, H. B., & Farias, M. C. Q. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-018-5656-7 · Cited by 22
Self-citationFull-reference audio-visual video quality metric
Martinez, H. B., & Farias, M. C. Q. (2014) · Journal of Electronic Imaging · doi:10.1117/1.jei.23.6.061108 · Cited by 20
Self-citation(2000) · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft)
Cited by
TOPIQ: A Top-Down Approach From Semantics to Distortions for Image Quality Assessment
Chen, C., Mo, J., Hou, J., Wu, H., Liao, L., Sun, W., Yan, Q., & Lin, W. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3378466 · Cited by 276
Perceptual video quality assessment: a survey
Min, X., Duan, H., Sun, W., Zhu, Y., & Zhai, G. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4133-3 · Cited by 218
Self-citationHuman-centered neural reasoning for subjective content processing: Hate speech, emotions, and humor
Kazienko, P., Bielaniewicz, J., Gruza, M., Kanclerz, K., Karanowski, K., Miłkowski, P., & Kocoń, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.01.010 · Cited by 38
Metadata
- Authors
- Min, X., Zhai, G., Zhou, J., Farias, M. C. Q., & Bovik, A. C.
- Venue
- IEEE Transactions on Image Processing
- Published
- 2020
- Volume
- 29
- Pages
- 6054-6068
- ISSN
- 1057-7149 (ISSN-L)
- 1941-0042
- OpenAlex
- W3019533076
- Cited by
- 328
- References
- 42
- FWCI
- 12.56
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.