The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Deep exemplar-based colorization
Open accessAbstract
We propose the first deep learning approach for exemplar-based local colorization. Given a reference color image, our convolutional neural network directly maps a grayscale image to an output colorized image. Rather than using hand-crafted rules as in traditional exemplar-based methods, our end-to-end colorization network learns how to select, propagate , and predict colors from the large-scale data. The approach performs robustly and generalizes well even when using reference images that are unrelated to the input grayscale image. More importantly, as opposed to other learning-based colorization methods, our network allows the user to achieve customizable results by simply feeding different references. In order to further reduce manual effort in selecting the references, the system automatically recommends references with our proposed image retrieval algorithm, which considers both semantic and luminance information. The colorization can be performed fully automatically by simply picking the top reference suggestion. Our approach is validated through a user study and favorable quantitative comparisons to the-state-of-the-art methods. Furthermore, our approach can be naturally extended to video colorization. Our code and models are freely available for public use.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738 · cited 1×
“We empirically chose the U-net architecture because of its effectiveness, as evidenced in many image generation tasks [Badrinarayanan et al. 2015; Yu and Koltun 2015; Zhang et al. 2017].”
MethodZhang, R., Isola, P., & Efros, A. A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,178
Reinhard, E., Adhikhmin, M., Gooch, B., & Shirley, P. (2001) · IEEE Computer Graphics and Applications · doi:10.1109/38.946629 · Cited by 2,937
SIFT Flow: Dense Correspondence across Scenes and Its Applications
Liu, C., Yuen, J., & Torralba, A. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.147 · Cited by 1,582
Colorization using optimization
Levin, A., Lischinski, D., & Weiss, Y. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015780 · Cited by 1,411
DAISY: An Efficient Dense Descriptor Applied to Wide-Baseline Stereo
Tola, E., Lepetit, V., & Fua, P. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.77 · Cited by 1,405
Neural Codes for Image Retrieval
Babenko, A., Slesarev, A., Chigorin, A., & Lempitsky, V. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_38 · Cited by 1,192
EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis
Sajjadi, M. S. M., Scholkopf, B., & Hirsch, M. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.481 · Cited by 1,058
Multi-scale Orderless Pooling of Deep Convolutional Activation Features
Gong, Y., Wang, L., Guo, R., & Lazebnik, S. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_26 · Cited by 1,030
DeepFlow: Large Displacement Optical Flow with Deep Matching
Weinzaepfel, P., Revaud, J., Harchaoui, Z., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.175 · Cited by 997
Learning Representations for Automatic Colorization
Larsson, G., Maire, M., & Shakhnarovich, G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_35 · Cited by 946
Photographic Image Synthesis with Cascaded Refinement Networks
Chen, Q., & Koltun, V. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.168 · Cited by 887
Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925974 · Cited by 814
Particular object retrieval with integral max-pooling of CNN activations
Tolias, G., Sicre, R., & Jégou, H. (2015) · arXiv · doi:10.48550/arxiv.1511.05879 · Cited by 744
Aggregating Local Deep Features for Image Retrieval
Yandex, A. B., & Lempitsky, V. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.150 · Cited by 674
Transferring color to greyscale images
Welsh, T. F., Ashikhmin, M., & Mueller, K. D. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566576 · Cited by 613
Real-time user-guided image colorization with learned deep priors
Zhang, R., Zhu, J.-Y., Isola, P., Geng, X., Lin, A. S., Yu, T., & Efros, A. A. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073703 · Cited by 569
Cheng, Z., Yang, Q., & Sheng, B. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.55 · Cited by 557
Fast image and video colorization using chrominance blending
Yatziv, L., & Sapiro, G. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.864231 · Cited by 529
StyleBank: An Explicit Representation for Neural Image Style Transfer
Chen, D., Yuan, L., Liao, J., Yu, N., & Hua, G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.296 · Cited by 502
Self-citationVisual attribute transfer through deep image analogy
Liao, J., Yao, Y., Yuan, L., Hua, G., & Kang, S. B. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073683 · Cited by 487
Self-citationLocal Color Transfer via Probabilistic Segmentation by Expectation-Maximization
Tai, Y.-W., Jia, J., & Tang, C.-K. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.215 · Cited by 332
Qu, Y., Wong, T.-T., & Heng, P.-A. (2006) · ACM Transactions on Graphics · doi:10.1145/1141911.1142017 · Cited by 297
Coherent Online Video Style Transfer
Chen, D., Liao, J., Yuan, L., Yu, N., & Hua, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.126 · Cited by 281
Self-citationImage colorization using similar images
Gupta, R. K., Chia, A. Y.-S., Rajan, D., Ng, E. S., & Zhiyong, H. (2012) · Proceedings of the 20th ACM international conference on Multimedia · doi:10.1145/2393347.2393402 · Cited by 256
An adaptive edge detection based colorization algorithm and its applications
Huang, Y., Tung, Y.-S., Chen, J.-C., Wang, S.-W., & Wu, J. (2005) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1101149.1101223 · Cited by 234
Automatic Image Colorization Via Multimodal Predictions
Charpiat, G., Hofmann, M., & Schölkopf, B. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88690-7_10 · Cited by 231
Learning Large-Scale Automatic Image Colorization
Deshpande, A., Rock, J., & Forsyth, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.72 · Cited by 214
Liu, X., Wan, L., Qu, Y., Wong, T., Lin, S. C.-F., Leung, C.-S., & Heng, P. (2008) · ACM Transactions on Graphics · doi:10.1145/1409060.1409105 · Cited by 203
Semantic colorization with internet images
Chia, A. Y.-S., Zhuo, S., Gupta, R. K., Tai, Y.-W., Cho, S.-Y., Tan, P., & Lin, S. (2011) · ACM Transactions on Graphics · doi:10.1145/2070781.2024190 · Cited by 177
Blind video temporal consistency
Bonneel, N., Tompkin, J., Sunkavalli, K., Sun, D., Paris, S., & Pfister, H. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818107 · Cited by 171
Semantic colorization with internet images
Chia, A. Y.-S., Zhuo, S., Gupta, R. K., Tai, Y.-W., Cho, S.-Y., Tan, P., & Lin, S. (2011) · Proceedings of the 2011 SIGGRAPH Asia Conference · doi:10.1145/2024156.2024190 · Cited by 167
Variational Exemplar-Based Image Colorization
Bugeau, A., Ta, V., & Papadakis, N. (2013) · IEEE Transactions on Image Processing · doi:10.1109/tip.2013.2288929 · Cited by 124
A Baseline for Visual Instance Retrieval with Deep Convolutional NetworksPreprint
Razavian, A. S., Sullivan, J., Maki, A., & Carlsson, S. (2014) · arXiv (Cornell University) · Cited by 118
Stereoscopic Neural Style Transfer
Chen, D., Yuan, L., Liao, J., Yu, N., & Hua, G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00696 · Cited by 110
Self-citationScribbler: Controlling Deep Image Synthesis with Sketch and ColorPreprint
Sangkloy, P., Lu, J., Chen, F., Yu, F., & Hays, J. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.00835 · Cited by 60
Cited by
Gated Context Aggregation Network for Image Dehazing and Deraining
Chen, D., He, M., Fan, Q., Liao, J., Zhang, L., Hou, D., Yuan, L., & Hua, G. (2019) · 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2019.00151 · Cited by 809
Self-citationImage-to-Image Translation: Methods and Applications
Pang, Y., Lin, J., Qin, T., & Chen, Z. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3109419 · Cited by 378
Learning by Aligning: Visible-Infrared Person Re-identification using Cross-Modal Correspondences
Park, H., Lee, S., Lee, J., & Ham, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01183 · Cited by 295
Cross-Domain Correspondence Learning for Exemplar-Based Image Translation
Zhang, P., Zhang, B., Chen, D., Yuan, L., & Wen, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00519 · Cited by 253
Self-citationLanguage-based colorization of scene sketches
Zou, C., Mo, H., Gao, C., Du, R., & Fu, H. (2019) · ACM Transactions on Graphics · doi:10.1145/3355089.3356561 · Cited by 86
Metadata
- Authors
- He, M., Chen, D., Liao, J., Sander, P. V., & Yuan, L.
- Venue
- ACM Transactions on Graphics
- Published
- 2018-07-30
- Volume
- 37
- Issue
- 4
- Pages
- 1-16
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2809852002
- Cited by
- 293
- References
- 44
- FWCI
- 16.08
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.