CoChat
Sign Up
Cited by 16Video Surveillance and Tracking MethodsEvacuation and Crowd DynamicsAdvanced Vision and Imaging

Crowd Density Forecasting by Modeling Patch-Based Dynamics

Open access
Minoura, Hiroaki, Yonetani, Ryo, Nishimura, Mai, Ushiku, Yoshitaka·2021·IEEE Robotics and Automation Letters
DOI 10.1109/lra.2020.3043169
Open this paper in CoChat — free account

Abstract

Forecasting human activities observed in videos is a long-standing challenge in computer vision and robotics and is also beneficial for various real-world applications such as mobile robot navigation and drone landing. In this work, we present a new forecasting task called crowd density forecasting. Given a video of a crowd captured by a surveillance camera, our goal is to predict how the density of the crowd will change in unseen future frames. To address this task, we developed the patch-based density forecasting networks (PDFNs), which directly forecasts crowd density maps of future frames instead of trajectories of each moving person in the crowd. The PDFNs represent crowd density maps based on spatially or spatiotemporally overlapping patches and learn a simple density dynamics of fewer people in each patch. Doing so allows us to efficiently deal with diverse and complex crowd density dynamics observed when input videos involve a variable number of crowds moving independently. Experimental results with several public datasets of surveillance videos demonstrate the effectiveness of our approaches compared with state-of-the-art forecasting methods.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/lra.2020.3043169

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The PDFN approach cannot always forecast sudden changes in walking directions, especially when they happened in later frames.

    “The approach cannot always forecast sudden changes in walking directions, especially when they happened in the later frames.”

    From Advanced Applications of Industrial Robotics: New Trends and Possibilities (2021) · cites this paper as [45]

    The Results (Section 4.6, p. 8) identify a forecasting failure: PDFN can miss a person entering or leaving the frame during future frames. The paper does not identify sudden changes in walking direction as the cause or say that such failures occur especially in later frames. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • PDFN-S and PDFN-ST outperformed baselines on all datasets, and PDFN-ST successfully forecasted dynamics of individuals, a small group, and a crowd.

    “Proposed patch-based models, PDFN-S and PDFN-ST, outperformed baselines on all the datasets.PDFN-ST successfully forecasted dynamics of individuals, a small group, and a crowd.”

    From Advanced Applications of Industrial Robotics: New Trends and Possibilities (2021) · cites this paper as [45]

    The Results (p. 8) say that PDFN outperformed baselines on all evaluated datasets and forecast the dynamics of individuals, a small group, and a crowd. But the Training Setup (p. 4) identifies the tested variants as PDFN-C and PDFN-D, not PDFN-S and PDFN-ST, so the results are not supported for the named models. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Patch-based density forecasting networks (PDFNs) directly forecast crowd density maps of future frames instead of trajectories of each moving person in the crowd.

    “The patch-based density forecasting networks (PDFNs) directly forecast crowd density maps of future frames instead of trajectories of each moving person in the crowd.”

    From Advanced Applications of Industrial Robotics: New Trends and Possibilities (2021) · cites this paper as [45]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 55

  1. [1][1] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. So- 2. cial lstm: Human trajectory prediction in crowded spaces. In Proc. CVPR , pages 961-971, 2016. 1, 2, 5, 6, 7 -
  2. [2][2] Florent Altch´ e and Arnaud de La Fortelle. An lstm network for highway trajectory prediction. In ITSC , pages 353-359, 2017. 1, 2, 8 -
  3. [3][3] Aniket Bera, Tanmay Randhavane, Rohan Prinja, and Dinesh Manocha. Sociosense: Robot navigation amongst pedestrians with social and psychological constraints. In Proc. IROS , pages 7018-7025, 2017. 1, 2 -
  4. [4][4] Gedas Bertasius, Aaron Chan, and Jianbo Shi. Egocentric basketball motion planning from a single first-person image. In Proc. CVPR , pages 5889-5898, 2018. 8 -
  5. [5][5] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In Proc. ICIP , pages 3464-3468, 2016. 6 -
  6. [6][6] Rohan Chandra, Uttaran Bhattacharya, Aniket Bera, and Dinesh Manocha. Traphic: Trajectory prediction in dense and heterogeneous traffic using weighted interactions. In Proc. CVPR , pages 8483-8492, 2019. 1, 2, 8 -
  7. [7][7] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, and James Hays. Argoverse: 3d tracking and forecasting with rich maps. In Proc. CVPR , pages 8748-8757, 2019. 2 -
  8. [8][8] Chiho Choi and Behzad Dariush. Looking to relations for future trajectory forecast. In Proc. ICCV , 2019. 2 -
  9. [9][9] Yanyan Fang, Biyun Zhan, Wandi Cai, Shenghua Gao, and Bo Hu. Locality-constrained spatial transformer network for video crowd counting. In Proc. ICME , pages 814-819. IEEE, 2019. 2, 3, 4 -
  10. [10][10] Junyu Gao, Wei Lin, Bin Zhao, Dong Wang, Chenyu Gao, and Jun Wen. C 3 framework: An open-source pytorch code for crowd counting. arXiv preprint arXiv:1907.02724 , 2019. 3, 4 - · arXiv
  11. [11][11] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proc. CVPR , pages 2255-2264, 2018. 1, 2 -
  12. [12][12] David Ha and J¨ urgen Schmidhuber. Recurrent world models facilitate policy evolution. In Proc. NeurIPS , pages 24502462, 2018. 4 -
  13. [13][13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. CVPR , pages 770-778, 2016. 4 -
  14. [14][14] Dirk Helbing and Peter Molnar. Social force model for pedestrian dynamics. Physical review E , 51(5):4282, 1995. 2 -
  15. [15][15] Adam Houenou, Philippe Bonnifait, V´ eronique Cherfaoui, and Wen Yao. Vehicle trajectory prediction based on motion model and maneuver recognition. In Proc. IROS , pages 4363-4369, 2013. 1, 2, 8 -
  16. [16][16] Boris Ivanovic and Marco Pavone. The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In Proc. ICCV , 2019. 1, 2, 5, 6, 7, 8 -
  17. [17][17] Ajay Jain, Sergio Casas, Renjie Liao, Yuwen Xiong, Song Feng, Sean Segal, and Raquel Urtasun. Discrete residual flow for probabilistic pedestrian behavior prediction. In Proc. CoRL , 2019. 1, 2, 8 -
  18. [18][18] Di Kang, Zheng Ma, and Antoni B Chan. Beyond counting: Comparisons of density maps for crowd analysis taskscounting, detection, and tracking. TCSVT , 29(5):1408-1422, 2018. 3 -
  19. [19][19] A. Kendon. Conducting interaction: Pattern of behavior in focused encounter . Cambridge University Press, 1990. 3 -
  20. [21][21] Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. Activity forecasting. In European Conference on Computer Vision , pages 201-214. Springer, 2012. 2 -
  21. [22][22] Henrik Kretzschmar, Markus Kuderer, and Wolfram Burgard. Learning to predict trajectories of cooperatively navigating agents. In Proc. ICRA , pages 4015-4020, 2014. 1, 2 -
  22. [23][23] Hoang M Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-agent imitation learning. In Proc. ICML , pages 1995-2003, 2017. 2 -
  23. [24][24] Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B. Choy, Philip H. S. Torr, and Manmohan Chandraker. Desire: Table 5: Base performances of crowd density maps on FDST : D KL , D IKL , D JS for the original crowd density maps computed directly from future video frames (lower scores mean better fit to ground truth maps). Table 6: Base performances of crowd density maps on UCY : D KL , D IKL , D JS for the original crowd density maps computed directly from future video frames (lower scores mean better fit to ground truth maps). Distant future prediction in dynamic scenes with interacting agents. In Proc. CVPR , 2017. 1, 2, 8 -
  24. [25][25] Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski. Crowds by example. Computer graphics forum , 26(3):655664, 2007. 2, 4 -
  25. [27][27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proc. ECCV , pages 740-755, 2014. 5 -
  26. [28][28] Weizhe Liu, Krzysztof Maciej Lis, Mathieu Salzmann, and Pascal Fua. Geometric and physical constraints for dronebased head plane crowd density estimation. In Proc. IROS , 2019. 3 -
  27. [29][29] Matthias Luber, Johannes A Stork, Gian Diego Tipaldi, and Kai O Arras. People tracking with human motion predictions from social forces. In Proc. ICRA , pages 464-469, 2010. 1, 2 -
  28. [30][30] Wenjie Luo, Bin Yang, and Raquel Urtasun. Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net. In Proc. CVPR , pages 3569-3577, 2018. 2 -
  29. [31][31] Wei-Chiu Ma, De-An Huang, Namhoon Lee, and Kris M Kitani. Forecasting interactive dynamics of pedestrians with fictitious play. In Proc. CVPR , pages 774-782, 2017. 2 -
  30. [32][32] Aashi Manglik, Xinshuo Weng, Eshed Ohn-Bar, and Kris M Kitani. Future near-collision prediction from monocular video: Feasibility, dataset, and challenges. In Proc. IROS , 2019. 1, 8 -
  31. [33][33] Riccardo Mazzon, Syed Fahad Tahir, and Andrea Cavallaro. Person re-identification in crowd. Pattern Recognition Letters , 33(14):1828-1837, 2012. 8 -
  32. [34][34] Ramin Mehran, Alexis Oyama, and Mubarak Shah. Abnormal crowd behavior detection using social force model. In Proc. CVPR , pages 935-942, 2009. 2 -
  33. [35][35] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proc. ICML , pages 807-814, 2010. 6 -
  34. [36][36] Yusuke Niitani, Toru Ogawa, Shunta Saito, and Masaki Saito. Chainercv: a library for deep learning in computer vision. In Proc. ACM'MM , 2017. 4 -
  35. [39][39] Nicholas Rhinehart, Kris M Kitani, and Paul Vernaza. R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In Proc. ECCV , pages 772-788, 2018. 2, 8 -
  36. [40][40] Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction conditioned on goals in visual multi-agent settings. In Proc. ICCV , 2019. 1, 2, 8 -
  37. [41][41] Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M Kitani, Dariu M Gavrila, and Kai O Arras. Human motion trajectory prediction: A survey. arXiv preprint arXiv:1905.06113 , 2019. 2 - · arXiv
  38. [42][42] Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical constraints. In Proc. CVPR , pages 1349-1358, 2019. 1, 2 -
  39. [43][43] Christoph Sch¨ oller, Vincent Aravantinos, Florian Lay, and Alois Knoll. The simpler the better: Constant velocity for pedestrian motion prediction. arXiv preprint arXiv:1903.07933 , 2019. 5, 6, 7 - · arXiv
  40. [44][44] Vishwanath A Sindagi and Vishal M Patel. A survey of recent advances in cnn-based single image crowd counting and density estimation. Pattern Recognition Letters , 107:3-16, 2018. 3 -
  41. [45][45] Hyun Soo Park, Jyh-Jing Hwang, Yedong Niu, and Jianbo Shi. Egocentric future localization. In Proc. CVPR , pages 4697-4705, 2016. 8 -
  42. [46][46] Luca Anthony Thiede and Pratik Prabhanjan Brahma. Analyzing the variety loss in the context of probabilistic trajec- tory prediction. In Proc. ICCV , pages 9954-9963, 2019. 2, 8 -
  43. [47][47] Pete Trautman, Jeremy Ma, Richard M Murray, and Andreas Krause. Robot navigation in dense human crowds: Statistical models and experimental studies of human-robot cooperation. IJRR , 34(3):335-356, 2015. 1, 2 -
  44. [48][48] Anirudh Vemula, Katharina Muelling, and Jean Oh. Social attention: Modeling attention in human crowds. In Proc. ICRA , pages 1-7, 2018. 1, 2 -
  45. [49][49] Qi Wang, Junyu Gao, Wei Lin, and Yuan Yuan. Learning from synthetic data for crowd counting in the wild. In Proc. CVPR , pages 8198-8207, 2019. 4, 8, 9, 10 -
  46. [50][50] SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional lstm network: A machine learning approach for precipitation nowcasting. In Proc. NeurIPS , pages 802-810, 2015. 3 -
  47. [51][51] Yanyu Xu, Zhixin Piao, and Shenghua Gao. Encoding crowd interaction with deep neural network for pedestrian trajectory prediction. In Proc. CVPR , pages 5275-5284, 2018. 1, 2 -
  48. [52][52] Takuma Yagi, Karttikeya Mangalam, Ryo Yonetani, and Yoichi Sato. Future person localization in first-person videos. In Proc. CVPR , pages 7593-7602, 2018. 8 -
  49. [53][53] Kuo-Hao Zeng, William B Shen, De-An Huang, Min Sun, and Juan Carlos Niebles. Visual forecasting by imitating dynamics in natural sequences. In Proc. ICCV , pages 29993008, 2017. 4 -
  50. [54][54] Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatiotemporal residual networks for citywide crowd flows prediction. In Proc. AAAI , 2017. 1, 2 -
  51. [55][55] Pu Zhang, Wanli Ouyang, Pengfei Zhang, Jianru Xue, and Nanning Zheng. Sr-lstm: State refinement for lstm towards pedestrian trajectory prediction. In Proc. CVPR , pages 12085-12094, 2019. 2 -
  52. [56][56] Shanghang Zhang, Guanhang Wu, Joao P Costeira, and Jos´ e MF Moura. Fcn-rlstm: Deep spatio-temporal neural networks for vehicle counting in city cameras. In Proc. ICCV , pages 3667-3676, 2017. 3 -
  53. [57][57] Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, and Ying Nian Wu. Multi-agent tensor fusion for contextual trajectory prediction. In Proc. CVPR , pages 12126-12134, 2019. 2 -
  54. [58][58] Bolei Zhou, Liu Liu, Aude Oliva, and Antonio Torralba. Recognizing city identity via attribute analysis of geo-tagged images. In Proc. ECCV , pages 519-534, 2014. 1, 2 -
  55. [60][60] Ali Zonoozi, Jung-jae Kim, Xiao-Li Li, and Gao Cong. Periodic-crn: A convolutional recurrent model for crowd density prediction with recurring periodic patterns. In Proc. IJCAI , pages 3732-3738, 2018. 1, 2

Cited by

Sorted by citations

Authors
Minoura, H., Yonetani, R., Nishimura, M., & Ushiku, Y.
Affiliations
  • Chubu University (JP)
  • Omron Sinic X Corporation (Japan) (JP)
ORCID
Venue
IEEE Robotics and Automation Letters
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2021-04
Volume
6
Issue
2
Pages
287-294
ISSN
  • 2377-3766 (ISSN-L)
  • 2377-3774
DOI
10.1109/lra.2020.3043169
OpenAlex
W3111021537
Cited by
16
References
3
Citations by year
2026: 1 · 2025: 3 · 2024: 2 · 2023: 1 · 2022: 4 · 2021: 4 · 2020: 1
FWCI
0.94
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Robotics and Automation Letters (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
  • Physical Sciences › Engineering › Ocean Engineering › Evacuation and Crowd Dynamics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
Keywords
density dynamics, surveillance video, mobile robot navigation, drone landing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review