Deep Learning Based Pain Recognition via Facial Expression Using Feature Fusion of Visual and Thermal Images
Abstract
Objective pain assessment in non-verbal patients remains a significant clinical challenge. While automated facial expression analysis offers a promising solution, existing methods often rely on simple cross-modality concatenation or shallow attention mechanisms, which typically fail to fully capture complex, interdependent cross-modal dynamics. To address this limitation, this study proposes a deep learning-based multimodal feature fusion framework that combines visual and thermal images to significantly enhance pain detection accuracy. The proposed framework integrates pretrained convolutional neural network backbones, specifically VGGFace, ResNet50, and DEYOLO, with two novel attention modules: Dual Semantic Enhancing Channel Weight Assignment (DECA), and Dual Spatial Enhancing Pixel Weight Assignment (DEPA) to adaptively optimize joint feature representations. For comparison, a hybrid baseline model that processes visual and thermal images separately was also developed, allowing a comparative analysis between fusion-based and non-fusion-based approaches. The model performance was systematically evaluated on the MIntPain dataset, which comprises 20 healthy subjects experiencing five distinct levels of pain intensity. To ensure data independence and prevent identity leakage across the training and testing phases, a strict subject-wise data split protocol was implemented with 15 subjects allocated for training and 5 subjects for testing. Experimental results demonstrate that the proposed multimodal fusion framework achieves superior performance, attaining a peak F1 score of 0.938 using the VGGFace backbone. Furthermore, external validation on the UNBC McMaster Shoulder Pain dataset yields a classification accuracy of 0.872, confirming the strong generalization capability and stability of the framework across unseen subjects. These findings highlight the effectiveness of visual-thermal synergy and the efficacy of the proposed DECA and DEPA modules, showcasing high potential for robust, non-invasive clinical pain-monitoring and assessment applications
Downloads
References
N. Ben Aoun, “A Review of Automatic Pain Assessment from Facial Information Using Machine Learning,” Jun. 01, 2024, Multidisciplinary Digital Publishing Institute (MDPI). doi: 10.3390/technologies12060092.
G. Bargshady, X. Zhou, R. C. Deo, J. Soar, F. Whittaker, and H. Wang, “Ensemble neural network approach detecting pain intensity from facial expressions,” Artif. Intell. Med., vol. 109, p. 101954, Sep. 2020, doi: 10.1016/j.artmed.2020.101954.
S. El Morabit, A. Rivenq, M.-E. Zighem, A. Hadid, A. Ouahabi, and A. Taleb-Ahmed, “Automatic Pain Estimation from Facial Expressions: A Comparative Analysis Using Off-the-Shelf CNN Architectures,” Electronics (Basel)., vol. 10, no. 16, p. 1926, Aug. 2021, doi: 10.3390/electronics10161926.
T. Alghamdi and G. Alaghband, “Facial Expressions Based Automatic Pain Assessment System,” Applied Sciences, vol. 12, no. 13, p. 6423, Jun. 2022, doi: 10.3390/app12136423.
T. Alghamdi and G. Alaghband, “SAFEPA: An Expandable Multi-Pose Facial Expressions Pain Assessment Method,” Applied Sciences, vol. 13, no. 12, p. 7206, Jun. 2023, doi: 10.3390/app13127206.
A. R. Wahab Sait and A. K. Dutta, “Ensemble Learning-Based Pain Intensity Identification Model Using Facial Expressions,” Journal of Disability Research, vol. 3, no. 3, Mar. 2024, doi: 10.57197/JDR-2024-0029.
M. S. Salekin, G. Zamzmi, D. Goldgof, R. Kasturi, T. Ho, and Y. Sun, “Multimodal spatio-temporal deep learning approach for neonatal postoperative pain assessment,” Comput. Biol. Med., vol. 129, p. 104150, Feb. 2021, doi: 10.1016/j.compbiomed.2020.104150.
N. Ben Aoun, “Deep Learning-Based Pain Intensity Estimation from Facial Expressions,” 2024, pp. 484–493. doi: 10.1007/978-3-031-64836-6_47.
S. A. S. Lakshminarayan, S. Hinduja, and S. Canavan, “Three-level Training of Multi-Head Architecture for Pain Detection,” in 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), IEEE, Nov. 2020, pp. 839–843. doi: 10.1109/FG47880.2020.00071.
A. Semwal and N. D. Londhe, “Computer aided pain detection and intensity estimation using compact CNN based fusion network,” Appl. Soft Comput., vol. 112, p. 107780, Nov. 2021, doi: 10.1016/j.asoc.2021.107780.
H. Pedersen, “Learning Appearance Features for Pain Detection Using the UNBC-McMaster Shoulder Pain Expression Archive Database,” 2015, pp. 128–136. doi: 10.1007/978-3-319-20904-3_12.
K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” Sep. 2014, doi: https://doi.org/10.48550/arXiv.1409.1556.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2016, pp. 770–778. doi: 10.1109/CVPR.2016.90.
M. A. Haque et al., “Deep Multimodal Pain Recognition: A Database and Comparison of Spatio-Temporal Visual Modalities,” in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), IEEE, May 2018, pp. 250–257. doi: 10.1109/FG.2018.00044.
Y. Lyu, I. Schiopu, and A. Munteanu, “Multi-modal neural networks with multi-scale RGB-T fusion for semantic segmentation,” Electron. Lett., vol. 56, no. 18, pp. 920–923, Sep. 2020, doi: 10.1049/el.2020.1635.
K. Munadi et al., “A Deep Learning Method for Early Detection of Diabetic Foot Using Decision Fusion and Thermal Images,” Applied Sciences, vol. 12, no. 15, p. 7524, Jul. 2022, doi: 10.3390/app12157524.
G. Bargshady, X. Zhou, R. C. Deo, J. Soar, F. Whittaker, and H. Wang, “The modeling of human facial pain intensity based on Temporal Convolutional Networks trained with video frames in HSV color space,” Appl. Soft Comput., vol. 97, p. 106805, Dec. 2020, doi: 10.1016/j.asoc.2020.106805.
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Jun. 2018, pp. 4510–4520. doi: 10.1109/CVPR.2018.00474.
M. Tan and Q. V. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” 36th International Conference on Machine Learning, ICML 2019, vol. 2019-June, Sep. 2020, [Online]. doi: 10.48550/arXiv.1905.11946
R. Islamadina, K. Saddami, M. Oktiana, T. F. Abidin, R. Muharar, and F. Arnia, “Performance of Deep Learning Benchmark Models on Thermal Imagery of Pain through Facial Expressions,” in 2022 IEEE International Conference on Communication, Networks and Satellite (COMNETSAT), IEEE, Nov. 2022, pp. 374–379. doi: 10.1109/COMNETSAT56033.2022.9994546.
Raihan Islamadina et al., “Learning Rate Analysis for Pain Recognition Through Viola-Jones and Deep Learning Methods,” Jurnal Nasional Teknik Elektro dan Teknologi Informasi, vol. 13, no. 2, pp. 77–83, May 2024, doi: 10.22146/jnteti.v13i2.9466.
S. Gkikas and M. Tsiknakis, “Automatic assessment of pain based on deep learning methods: A systematic review,” Apr. 01, 2023, Elsevier Ireland Ltd. doi: 10.1016/j.cmpb.2023.107365.
K. Gasmi et al., “Enhanced brain tumor diagnosis using combined deep learning models and weight selection technique,” Front. Neuroinform., vol. 18, Nov. 2024, doi: 10.3389/fninf.2024.1444650.
M. Hussain, M. O’Nils, J. Lundgren, and S. J. Mousavirad, “A Comprehensive Review on Deep Learning-Based Data Fusion,” IEEE Access, vol. 12, 2024, doi: 10.1109/ACCESS.2024.3508271.
S. Alphonse, S. Abinaya, and N. Kumar, “Pain assessment from facial expression images utilizing Statistical Frei-Chen Mask (SFCM)-based features and DenseNet,” Journal of Cloud Computing, vol. 13, no. 1, p. 142, Sep. 2024, doi: 10.1186/s13677-024-00706-9.
Y. Cheng and D. Kong, “CSINet: Channel–Spatial Fusion Networks for Asymmetric Facial Expression Recognition,” Symmetry (Basel)., vol. 16, no. 4, p. 471, Apr. 2024, doi: 10.3390/sym16040471.
P. Rodriguez et al., “Deep Pain: Exploiting Long Short-Term Memory Networks for Facial Expression Classification,” IEEE Trans. Cybern., vol. 52, no. 5, pp. 3314–3324, May 2022, doi: 10.1109/TCYB.2017.2662199.
Y. Chen et al., “DEYOLO: Dual-Feature-Enhancement YOLO for Cross-Modality Object Detection,” 2024, pp. 236–252. doi: 10.1007/978-3-031-78447-7_16.
M. Ferguson, R. Ak, Y.-T. T. Lee, and K. H. Law, “Automatic localization of casting defects with convolutional neural networks,” in 2017 IEEE International Conference on Big Data (Big Data), IEEE, Dec. 2017, pp. 1726–1735. doi: 10.1109/BigData.2017.8258115.
M. Liang, J. Hu, C. Bao, H. Feng, F. Deng, and T. L. Lam, “Explicit Attention-Enhanced Fusion for RGB-Thermal Perception Tasks,” IEEE Robot. Autom. Lett., vol. 8, no. 7, pp. 4060–4067, Jul. 2023, doi: 10.1109/LRA.2023.3272269.
R. Rossi, M. A. Lazarini, and K. Hirama, “Systematic Literature Review on the Accuracy of Face Recognition Algorithms,” EAI Endorsed Transactions on Internet of Things, vol. 8, no. 30, 2022, doi: 10.4108/eetiot.v8i30.2346.
K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks,” IEEE Signal Process. Lett., vol. 23, no. 10, pp. 1499–1503, Oct. 2016, doi: 10.1109/LSP.2016.2603342.
J. Brownlee, “Understand the Impact of Learning Rate on Neural Network Performance,” Machine Learning Mastery. Accessed: Jan. 24, 2021. [Online]. Available: https://machinelearningmastery.com/understand-the-dynamics-of-learning-rate-on-deep-learning-neural-networks/
G. Menchetti, Z. Chen, Di. J. Wilkie, R. Ansari, Y. Yardimci, and A. E. Cetin, “Pain detection from facial videos using two-stage deep learning,” in GlobalSIP 2019 - 7th IEEE Global Conference on Signal and Information Processing, Proceedings, 2019. doi: 10.1109/GlobalSIP45357.2019.8969274.
C. J. Clopper and E. S. Pearson, “The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial,” Biometrika, vol. 26, no. 4, 1934, doi: 10.2307/2331986.
A. Kimber, B. Efron, and R. J. Tibshirani, “An Introduction to the Bootstrap.,” The Statistician, vol. 43, no. 4, 1994, doi: 10.2307/2348146.
N. Zaeri and R. Qasim, “Thermal image identification against pose and expression variations using deep learning,” Journal of Engineering Research, vol. 12, no. 4, pp. 751–760, Dec. 2024, doi: 10.1016/j.jer.2023.10.043.
P. Lucey, J. F. Cohn, K. M. Prkachin, P. E. Solomon, and I. Matthews, “Painful data: The UNBC-McMaster shoulder pain expression archive database,” in 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops, FG 2011, 2011. doi: 10.1109/FG.2011.5771462.
Z. Li, W. L. Zheng, and B. L. Lu, “Multimodal Emotion Recognition with Missing Modality via a Unified Multi-task Pre-training Framework,” in MM 2025 - Proceedings of the 33rd ACM International Conference on Multimedia, Co-Located with MM 2025, 2025. doi: 10.1145/3746027.3755459.
X. Du, M. Safarzadeh, M. Zhu, S. Prasad, S. Das, and J. Chung, “An Explainable Transformer Model for Pain Intensity Assessment Using Multi-Modal Facial Sequential Images,” 2025. doi: 10.2139/ssrn.5201152.
O. El Othmani and S. Naouali, “Cross-spectral fusion of thermal and RGB imaging for objective pain estimation,” PLOS Digital Health, vol. 5, no. 5, p. e0001424, May 2026, doi: 10.1371/journal.pdig.0001424.
Copyright (c) 2026 Raihan Islamadina, Fitri Arnia, Taufik Fuadi Abidin, Rusdha Muharar, Aulia Syarif Aziz, Khairun Saddami
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlikel 4.0 International (CC BY-SA 4.0) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
.png)
.png)
.png)
.png)
.png)
