CoChat
Sign Up
Cited by 95Pediatric Pain Management TechniquesEmotion and Mood RecognitionInfant Health and Development

Multimodal spatio-temporal deep learning approach for neonatal postoperative pain assessment

Open access
Salekin, Md Sirajus, Zamzmi, Ghada, Goldgof, Dmitry, Kasturi, Rangachar, Ho, Thao, Sun, Yu·2021·Computers in Biology and Medicine
DOI 10.1016/j.compbiomed.2020.104150PMC PMC7856028
Open this paper in CoChat — free account

Abstract

The current practice for assessing neonatal postoperative pain relies on bedside caregivers. This practice is subjective, inconsistent, slow, and discontinuous. To develop a reliable medical interpretation, several automated approaches have been proposed to enhance the current practice. These approaches are unimodal and focus mainly on assessing neonatal procedural (acute) pain. As pain is a multimodal emotion that is often expressed through multiple modalities, the multimodal assessment of pain is necessary especially in case of postoperative (acute prolonged) pain. Additionally, spatio-temporal analysis is more stable over time and has been proven to be highly effective at minimizing misclassification errors. In this paper, we present a novel multimodal spatio-temporal approach that integrates visual and vocal signals and uses them for assessing neonatal postoperative pain. We conduct comprehensive experiments to investigate the effectiveness of the proposed approach. We compare the performance of the multimodal and unimodal postoperative pain assessment, and measure the impact of temporal information integration. The experimental results, on a real-world dataset, show that the proposed multimodal spatio-temporal approach achieves the highest AUC (0.87) and accuracy (79%), which are on average 6.67% and 6.33% higher than unimodal approaches. The results also show that the integration of temporal information markedly improves the performance as compared to the non-temporal approach as it captures changes in the pain dynamic. These results demonstrate that the proposed approach can be used as a viable alternative to manual assessment, which would tread a path toward fully automated pain monitoring in clinical settings, point-of-care testing, and homes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.compbiomed.2020.104150

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Only 2 studies in this review used camera recordings to assess postoperative pain in neonates.

    “only 2 studies in this review used camera recordings to assess postoperative pain in neonates by capturing movements in premature infants [55,69].”

    From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [55]

    The Neonatal Pain Dataset section describes camera recordings of neonates, including nine monitored after surgery, and the Methodology section describes assessing postoperative pain from body movement in those videos. The paper reports gestational ages of 30–41 weeks for the overall dataset but does not establish that the postoperative group consisted of premature infants. The count of two studies is a statement about the citing review, not a finding this individual study can verify. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Late fusion combined outputs from modality-specific models without requiring explicit data-level alignment.

    “whereas late fusion combined outputs from modality-specific models without requiring explicit data-level alignment [39,54–56].”

    From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [55]

  • Regarding missing values within individual modalities, 36 studies focused on structured data, with only one study each considering waveforms and video modality.

    “Regarding missing values within individual modalities, 36 studies focused on structured data, with only one study each considering waveforms [39] and video modality [55].”

    From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [55]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. [1]Altman NS, 1992 An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician 46, 175– 185. doi: 10.1080/00031305.1992.10475879. · DOI
  2. [2]Brasher C, Gafsous B, Dugue S, Thiollier A, Kinderf J, Nivoche Y, Grace R, Dahmani S, 2014 Postoperative pain management in children and infants: an update. Pediatric Drugs 16, 129– 140. doi: 10.1007/s40272-013-0062-0. 24407716 · DOI
  3. [3]Breiman L, 2001 Random forests. Machine learning 45, 5– 32. doi: 10.1023/A:1010933404324. · DOI · Paper page
  4. [5]Celona L, Manoni L, 2017 Neonatal facial pain assessment combining hand-crafted and deep features, in: International Conference on Image Analysis and Processing, Springer pp. 197– 204. doi: 10.1007/978-3-319-70742-6_19. · DOI
  5. [6]Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L, 2009 Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee pp. 248– 255. doi: 10.1109/CVPR.2009.5206848. · DOI · Paper page
  6. [8]Dowell D, Arias E, Kochanek K, Anderson R, Guy GP, Losby JL, Baldwin G, 2017 Contribution of opioid-involved poisoning to the change in life expectancy in the united states, 2000–2015. Jama 318, 1065– 1067. doi: 10.1001/jama.2017.9308. 28975295 PMC5818798 · DOI
  7. [9]Fortier MA, Chung WW, Martinez A, Gago-Masague S, Sender L, 2016 Pain buddy: A novel use of m-health in the management of children’s cancer pain. Computers in biology and medicine 76, 202– 214. doi: 10.1016/j.compbiomed.2016.07.012. 27479493 PMC5639256 · DOI
  8. [10]Gan TJ, 2017 Poorly controlled postoperative pain: prevalence, consequences, and prevention. Journal of pain research 10, 2287. doi: 10.2147/JPR.S144066. 29026331 PMC5626380 · DOI
  9. [11]Grosse SD, Waitzman NJ, Yang N, Abe K, Barfield WD, 2017 Employer-sponsored plan expenditures for infants born preterm. Pediatrics 140, e20171078. doi: 10.1542/peds.2017-1078. 28933347 PMC6029853 · DOI
  10. [12]Grunau RE, Holsti L, Peters JW, 2006 Long-term consequences of pain in human neonates, in: Seminars in Fetal and Neonatal Medicine, Elsevier pp. 268– 275. doi: 10.1016/j.siny.2006.02.007. 16632415 · DOI
  11. [13]Haque MA, Bautista RB, Noroozi F, Kulkarni K, Laursen CB, Irani R, Bellantonio M, Escalera S, Anbarjafari G, Nasrollahi K,, 2018 Deep multimodal pain recognition: a database and comparison of spatio-temporal visual modalities, in: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), IEEE pp. 250– 257. doi: 10.1109/FG.2018.00044. · DOI
  12. [14]Hochreiter S, Schmidhuber J, 1997 Long short-term memory. Neural computation 9, 1735– 1780. doi: 10.1162/neco.1997.9.8.1735. 9377276 · DOI · Paper page
  13. [15]Hudson-Barr D, Capper-Michel B, Lambert S, Palermo TM, Morbeto K, Lombardo S, 2002 Validation of the pain assessment in neonates (pain) scale with the neonatal infant pain scale (nips). Neonatal Network 21, 15– 22. doi: 10.1891/0730-0832.21.6.15. 12240510 · DOI
  14. [16]Hummel P, Puchalski M, Creech S, Weiss M, 2008 Clinical reliability and validity of the n-pass: neonatal pain, agitation and sedation scale with prolonged pain. Journal of Perinatology 28, 55. doi: 10.1038/sj.jp.7211861. 18165830 · DOI
  15. [18]Koul A, Becchio C, Cavallo A, 2018 Cross-validation approaches for replicability in psychology. Frontiers in Psychology 9, 1117. doi: 10.3389/fpsyg.2018.01117. 30034353 PMC6043802 · DOI
  16. [20]Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL, 2014 Microsoft coco: Common objects in context, in: European conference on computer vision, Springer pp. 740– 755. doi: 10.1007/978-3-319-10602-1_48. · DOI · Paper page
  17. [21]Lin TY, RoyChowdhury A, Maji S, 2015 Bilinear cnn models for fine-grained visual recognition, in: Proceedings of the IEEE international conference on computer vision, pp. 1449– 1457. doi: 10.1109/ICCV.2015.170. · DOI
  18. [22]McNair C, Ballantyne M, Dionne K, Stephens D, Stevens B, 2004 Postoperative pain assessment in the neonatal intensive care unit. Archives of Disease in Childhood-Fetal and Neonatal Edition 89, F537– F541. doi: 10.1136/adc.2003.032961. 15499150 PMC1721796 · DOI
  19. [23]Melo G.M.d., Lélis A.L.P.d.A., Moura A.F.d., Cardoso MVLML, Silva V.M.d., 2014 Pain assessment scales in newborns: integrative review. Revista paulista de pediatria 32, 395– 402. doi: 10.1590/S0103-05822014000400017. 25511005 PMC4311795 · DOI
  20. [24]Oppenheim AV, 1999 Discrete-time signal processing. Pearson Education India.
  21. [25]Oppenheim AV, Schafer RW, 2004 From frequency to quefrency: A history of the cepstrum. IEEE signal processing Magazine 21, 95– 106. doi: 10.1109/MSP.2004.1328092. · DOI
  22. [26]Parkhi OM, Vedaldi A, Zisserman A, 2015 Deep face recognition, 41.1– 41.12doi: 10.5244/C.29.41. · DOI · Paper page
  23. [27]Penrose LS, 1946 The elementary statistics of majority voting. Journal of the Royal Statistical Society 109, 53– 57. doi: 10.2307/2981392. · DOI
  24. [28]Quinlan J, Lobo D, Levy N, 2020 Postoperative pain management: time to get back on track. Anaesthesia 75, e10– e13. doi: 10.1111/anae.14886. 31903581 · DOI
  25. [29]Redmon J, Farhadi A, 2018 Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767. · arXiv
  26. [30]Rish I,, 2001 An empirical study of the naive bayes classifier, in: IJCAI 2001 workshop on empirical methods in artificial intelligence, pp. 41– 46.
  27. [33]Salekin MS, Zamzmi G, Goldgof D, Kasturi R, Ho T, Sun Y, 2019a Multi-channel neural network for assessing neonatal pain from videos, in: 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), IEEE pp. 1551– 1556. doi: 10.1109/SMC.2019.8914537. · DOI
  28. [34]Salekin MS, Zamzmi G, Goldgof D, Kasturi R, Ho T, Sun Y, 2020 First investigation into the use of deep learning for continuous assessment of neonatal postoperative pain, in: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG), IEEE pp. 529– 533. doi: 10.1109/FG47880.2020.00082. · DOI
  29. [35]Salekin MS, Zamzmi G, Paul R, Goldgof D, Kasturi R, Ho T, Sun Y, 2019b Harnessing the power of deep learning methods in healthcare: Neonatal pain assessment from crying sound, in: 2019 IEEE Healthcare Innovations and Point of Care Technologies, (HIPOCT), IEEE pp. 127– 130. doi: 10.1109/HI-POCT45284.2019.8962827. · DOI
  30. [36]Seok HS, Choi BM, Noh GJ, Shin H, 2019 Postoperative pain assessment model based on pulse contour characteristics analysis. IEEE journal of biomedical and health informatics 23, 2317– 2324. doi: 10.1109/JBHI.2018.2890482. 30605112 · DOI
  31. [37]Sikka K, Ahmed AA, Diaz D, Goodwin MS, Craig KD, Bartlett MS, Huang JS, 2015 Automated assessment of children’s postoperative pain using computer vision. Pediatrics 136, e124– e131. doi: 10.1542/peds.2015-0029. 26034245 PMC4485009 · DOI
  32. [38]Simonyan K, Zisserman A, 2014 Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. · arXiv
  33. [40]Talbot K, Madden V, Jones S, Moseley G, 2019 The sensory and affective components of pain: are they differentially modifiable dimensions or inseparable aspects of a unitary experience? a systematic review. British journal of anaesthesia doi: 10.1016/j.bja.2019.03.033. PMC6676053 31053232 · DOI
  34. [41]Taylor BJ, Robbins JM, Gold JI, Logsdon TR, Bird T, Anand K, 2006 Assessing postoperative pain in neonates: a multicenter observational study. Pediatrics 118, e992– e1000. doi: 10.1542/peds.2005-3203. 17015519 · DOI
  35. [42]Temko A, Doyle O, Murray D, Lightbody G, Boylan G, Marnane W, 2015 Multimodal predictor of neurodevelopmental outcome in newborns with hypoxic-ischaemic encephalopathy. Computers in biology and medicine 63, 169– 177. doi: 10.1016/j.compbiomed.2015.05.017. 26093065 · DOI
  36. [44]Walker SM, 2014 Neonatal pain. Pediatric Anesthesia 24, 39– 48. doi: 10.1111/pan.12293. 24330444 PMC3995005 · DOI
  37. [45]Walter-Nicolet E, Annequin D, Biran V, Mitanchez D, Tourniaire B, 2010 Pain management in newborns. Pediatric Drugs 12, 353– 365. doi: 10.2165/11318900-000000000-00000. 21028915 · DOI
  38. [46]Weiser TG, Gawande A, 2015 Excess surgical mortality: strategies for improving quality of care, in: Essential Surgery: Disease Control Priorities, Third Edition (Volume 1). The International Bank for Reconstruction and Development/The World Bank. doi: 10.1596/978-1-4648-0346-8_ch16. 26740999 · DOI
  39. [47]Yang S, Luo P, Loy CC, Tang X, 2016 Wider face: A face detection benchmark, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5525– 5533. doi: 10.1109/CVPR.2016.596. · DOI
  40. [48]Zamzmi G, Kasturi R, Goldgof D, Zhi R, Ashmeade T, Sun Y, 2018 A review of automated pain assessment in infants: Features, classification tasks, and databases. IEEE reviews in biomedical engineering 11, 77– 96. doi: 10.1109/RBME.2017.2777907. 29989992 · DOI
  41. [49]Zamzmi G, Pai CY, Goldgof D, Kasturi R, Ashmeade T, Sun Y, 2019a A comprehensive and context-sensitive neonatal pain assessment using computer vision. IEEE Transactions on Affective Computing doi: 10.1109/TAFFC.2019.2926710. · DOI
  42. [50]Zamzmi G, Paul R, Salekin MS, Goldgof D, Kasturi R, Ho T, Sun Y, 2019b Convolutional neural networks for neonatal pain assessment. IEEE Transactions on Biometrics, Behavior, and Identity Science 1, 192– 200. doi: 10.1109/TBIOM.2019.2918619. · DOI

Cited by

Sorted by citations

Authors
Salekin, M. S., Zamzmi, G., Goldgof, D., Kasturi, R., Ho, T., & Sun, Y.
Affiliations
University of South Florida (US)
Corresponding author
Salekin, Md Sirajus
ORCID
Venue
Computers in Biology and Medicine
Publisher
Elsevier BV
Published
2021-02
Volume
129
Pages
104150
ISSN
0010-4825 (ISSN-L)
DOI
10.1016/j.compbiomed.2020.104150
PMC
PMC7856028
OpenAlex
W3110091040
Cited by
94
References
58
Citations by year
2026: 22 · 2025: 19 · 2024: 20 · 2023: 17 · 2022: 9 · 2021: 7
FWCI
7.93
Type
journal article
Language
en
Open access
Licence
Locations
  • Computers in Biology and Medicine (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Institutes of Health (R21NR018756)
Topics
  • Health Sciences › Medicine › Pediatrics, Perinatology and Child Health › Pediatric Pain Management Techniques
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Health Sciences › Health Professions › Pharmacy › Infant Health and Development
Keywords
pain assessment, multimodal pain assessment, spatio-temporal analysis, visual signals, vocalizations
MeSH terms
Deep Learning, Emotions, Humans, Infant, Newborn, Postoperative Pain, Pain, Postoperative

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review