Understanding cities with machine eyes: A review of deep computer vision in urban analytics
Open accessThe DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Abstract
Abstract Modelling urban systems has interested planners and modellers for decades. Different models have been achieved relying on mathematics, cellular automation, complexity, and scaling. While most of these models tend to be a simplification of reality, today within the paradigm shifts of artificial intelligence across the different fields of science, the applications of computer vision show promising potential in understanding the realistic dynamics of cities. While cities are complex by nature, computer vision shows progress in tackling a variety of complex physical and non-physical visual tasks. In this article, we review the tasks and algorithms of computer vision and their applications in understanding cities. We attempt to subdivide computer vision algorithms into tasks, and cities into layers to show evidence of where computer vision is intensively applied and where further research is needed. We focus on highlighting the potential role of computer vision in understanding urban systems related to the built environment, natural environment, human interaction, transportation, and infrastructure. After showing the diversity of computer vision algorithms and applications, the challenges that remain in understanding the integration between these different layers of cities and their interactions with one another relying on deep learning and computer vision. We also show recommendations for practice and policy-making towards reaching AI-generated urban policies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
SSD: Single Shot MultiBox Detector
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_2 · Cited by 22,128
YOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.690 · Cited by 19,892
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631
YOLOv3: An Incremental Improvement
Redmon, J., & Farhadi, A. (2018) · arXiv · doi:10.48550/arxiv.1804.02767 · Cited by 17,141
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235 · cited 1×
“GANs are trained differently from the abovementioned deep models, not only in term of layers but rather, instead of the single end-to-end model, two parallel deep models are trained that compete with one another (Goodfellow, 2016; Goodfellow et al., 2014; Radford et al., 2015).”
BackgroundA Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738
Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.143 · Cited by 7,539
Rethinking Atrous Convolution for Semantic Image Segmentation
Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1706.05587 · Cited by 7,517
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · cited 2×
“…et al., 2013) - (Hester et al., 2017) DOUBLE DEEP QLEARNING - (van Hasselt, Guez, & Silver, 2015) DUEL DEEP Q-LEARNING - (Wang et al., 2015) A3C - (Mnih et al., 2016) 3.1 Classification Deep learning models, most specifically Convolutional Neural Networks (CNN), have shown substantial progress in…”
BackgroundInfluentialDeep learning and process understanding for data-driven Earth system science
Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., & Prabhat (2019) · Nature · doi:10.1038/s41586-019-0912-1 · Cited by 5,673
Generative Adversarial Networks
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2661 · Cited by 4,477
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2014) · arXiv · doi:10.48550/arxiv.1412.7062 · Cited by 4,213
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Brock, A., Donahue, J., & Simonyan, K. (2018) · arXiv · doi:10.48550/arxiv.1809.11096 · Cited by 3,717
Dueling Network Architectures for Deep Reinforcement Learning
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06581 · Cited by 3,300
Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks
Cha, Y., Choi, W., & Büyüköztürk, O. (2017) · Computer-Aided Civil and Infrastructure Engineering · doi:10.1111/mice.12263 · Cited by 3,295
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation
Lin, G., Milan, A., Shen, C., & Reid, I. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.549 · Cited by 3,287
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.629 · Cited by 3,021 · cited 4×
“Zhang et al. (2016) created stackGAN model to transform a text description of an image into a photo-realistic synthesis.”
BackgroundMethodInfluentialECO: Efficient Convolution Operators for Tracking
Danelljan, M., Bhat, G., Khan, F. S., & Felsberg, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.733 · Cited by 2,805
Deep Clustering for Unsupervised Learning of Visual Features
Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_9 · Cited by 2,674
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., & Dollár, P. (2014) · arXiv · doi:10.48550/arxiv.1405.0312 · Cited by 2,579
Deep learning for visual understanding: A review
Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.09.116 · Cited by 2,504
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494
Unsupervised deep embedding for clustering analysis
Xie, J., Girshick, R., & Farhadi, A. (2015) · arXiv · doi:10.48550/arxiv.1511.06335 · Cited by 2,476
DensePose: Dense Human Pose Estimation in the Wild
Guler, R. A., Neverova, N., & Kokkinos, I. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00762 · Cited by 1,851
Large Kernel Matters — Improve Semantic Segmentation by Global Convolutional Network
Peng, C., Zhang, X., Yu, G., Luo, G., & Sun, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.189 · Cited by 1,750 · cited 1×
“…continual success of CNN models for understanding and classifying the different components of the built environment at a pixel-level (Badrinarayanan et al., 2016; L.-C. Chen et al., 2016a, 2017; G. Lin et al., 2017; Long et al., 2015; Peng et al., 2017; F. Yu & Koltun, 2015; H. Zhao et al., 2017).”
BackgroundDenseASPP for Semantic Segmentation in Street Scenes
Yang, M., Yu, K., Zhang, C., Li, Z., & Yang, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00388 · Cited by 1,674
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017) · arXiv · doi:10.48550/arxiv.1708.02002 · Cited by 1,227
Learning to Track at 100 FPS with Deep Regression Networks
Held, D., Thrun, S., & Savarese, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_45 · Cited by 1,199
Action recognition with trajectory-pooled deep-convolutional descriptors
Wang, L., Qiao, Y., & Tang, X. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299059 · Cited by 1,194
Temporal Action Detection with Structured Segment Networks
Zhao, Y., Xiong, Y., Wang, L., Wu, Z., Tang, X., & Lin, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.317 · Cited by 892
Deep Q-learning From Demonstrations
Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., et al. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11757 · Cited by 889
Measuring human perceptions of a large-scale urban region using machine learning
Zhang, F., Zhou, B., Liu, L., Liu, Y., Fung, H. H., Lin, H., & Ratti, C. (2018) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2018.08.020 · Cited by 810 · cited 1×
“…tracker systems can be built based on correlation filtering and online learning techniques between consecutive frames (X. Zhang, Xia, Lu, Shen, & Zhang, 2018), the stateof-the-art research in object tracking uses an end-to-end CNN model to tackle both detection and tracking, which can add more…”
MOTS: Multi-Object Tracking and Segmentation
Voigtlaender, P., Krause, M., Osep, A., Luiten, J., Sekar, B. B. G., Geiger, A., & Leibe, B. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00813 · Cited by 658
Deep visual tracking: Review and experimental comparison
Li, P., Wang, D., Wang, L., & Lu, H. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.11.007 · Cited by 517
Faisal, A., Yigitcanlar, T., Kamruzzaman, M., & Currie, G. (2019) · Journal of Transport and Land Use · doi:10.5198/jtlu.2019.1405 · Cited by 427
Chu, Q., Ouyang, W., Li, H., Wang, X., Liu, B., & Yu, N. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.518 · Cited by 406
Object Detection from Video Tubelets with Convolutional Neural Networks
Kang, K., Ouyang, W., Li, H., & Wang, X. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.95 · Cited by 386
Paganini, M., de Oliveira, L., & Nachman, B. (2018) · Physical Review D · doi:10.1103/physrevd.97.014021 · Cited by 370
A patch-based convolutional neural network for remote sensing image classification
Sharma, A., Liu, X., Yang, X., & Shi, D. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.07.017 · Cited by 260
Application of Deep-Learning Methods to Bird Detection Using Unmanned Aerial Vehicle Imagery
Hong, S.-J., Han, Y., Kim, S.-Y., Lee, A.-Y., & Kim, G. (2019) · Sensors · doi:10.3390/s19071651 · Cited by 227
Deep Learning in Data-Driven Pavement Image Analysis and Automated Distress Detection: A Review
Gopalakrishnan, K. (2018) · Data · doi:10.3390/data3030028 · Cited by 210 · cited 1×
“So far, the applications of computer vision have been seen in a wide range of domains related to infrastructure and civil engineering (Gopalakrishnan, 2018; Griffiths & Boehm, 2018), most importantly in analysing defects (Feng, Liu, Kao, & Lee, 2017).”
BackgroundRegional deep learning model for visual tracking
Wu, G., Lu, W., Gao, G., Zhao, C., & Liu, J. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.10.064 · Cited by 78
Data-Driven Policy Making: The Policy Lab Approach
van Veenstra, A. F., & Kotterink, B. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-64322-9_9 · Cited by 60
Cited by
Street view imagery in urban analytics and GIS: A review
Biljecki, F., & Ito, K. (2021) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2021.104217 · Cited by 925
Huang, M. Q., Ninić, J., & Zhang, Q. B. (2021) · Tunnelling and Underground Space Technology · doi:10.1016/j.tust.2020.103677 · Cited by 370
A review of urban physical environment sensing using street view imagery in public health studies
Kang, Y., Zhang, F., Gao, S., Lin, H., & Liu, Y. (2020) · Annals of GIS · doi:10.1080/19475683.2020.1791954 · Cited by 292
Metadata
- Authors
- Ibrahim, M. R., Haworth, J., & Cheng, T.
- Venue
- Cities
- Publisher
- Elsevier BV
- Published
- 2020-01
- Volume
- 96
- Pages
- 102481
- ISSN
- 0264-2751 (ISSN-L)
- OpenAlex
- W2983285440
- Cited by
- 204
- References
- 56
- FWCI
- 10.35
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.