Analysis of Multimodal Biosignals during Surprise Conditions Correlates with Psychological Traits
Abstract
Surprise can simultaneously represent bad or good, pleasant or unpleasant, with the same experiences since understanding how humans' physiological qualities link with their emotional or mental health is required. We conducted quantitative research to concisely correlate mental stress and emotional issues by measuring brain activity, breathing, and heart rate in real time while executing specialized audio-visual stimulation to elicit a surprise event. We evaluated the frequency and temporal domain characteristics to determine if physiological measurements matched biochemical metrics and subjective stress assessments during the elicit surprise condition experiment. We discovered that the brain is still preferable to most in recognizing a human's psychological changes over a short period of time. The temporal (T3) (r = 0.544, p = 0.005) and frontal (Fz) (r = 0.519, p = 0.008) regions were shown to correlate with salivary amylase activity. In comparison to other channels, there was a negative association between stress perception and the occipital site (O1, r = -0.618, p = 0.001). We also found that heart rate variability activity correlates with arousal perception. By looking at specific multimodal biosignals, it is possible to understand human psychological traits by recording specific physiological signals for daily mental health monitoring.
Downloads
References
C. Yu, B. Ye, C. Lin, and Y. J. Wu, “Can Smart City Development Promote Residents’ Emotional Well-Being? Evidence From China,” IEEE Access, vol. 8, pp. 116024–116040, 2020, doi: https://doi.org/10.1109/ACCESS.2020.3004367.
D. Kahneman and A. Deaton, “High Income Improves Evaluation of Life but Not Emotional Well-Being,” Proceedings of the National Academy of Sciences, vol. 107, no. 38, pp. 16489–16493, Sep. 2010, doi: https://doi.org/10.1073/pnas.1011492107.
J. Du, J. Huang, Y. An, and W. Xu, “The Relationship between stress and negative emotion: The Mediating role of rumination,” Clinical Research and Trials, vol. 4, no. 1, 2018, doi: https://doi.org/10.15761/crt.1000208.
A. E. Gaffey and M. M. Wirth, “Psychophysiological Measures,” Encyclopedia of Quality of Life and Well-Being Research, pp. 5181–5184, 2014, doi: https://doi.org/10.1007/978-94-007-0753-5_2315.
K. Vora et al., “Emotion Recognition from Sensory and Bio-Signals: A Survey,” Proceedings of the 2nd International Conference on Data Engineering and Communication Technology, pp. 345–355, Oct. 2018, doi: https://doi.org/10.1007/978-981-13-1610-4_35.
G. Giannakakis, D. Grigoriadis, K. Giannakaki, O. Simantiraki, A. Roniotis, and M. Tsiknakis, “Review on psychological stress detection using biosignals,” IEEE Transactions on Affective Computing, pp. 1–1, 2019, doi: https://doi.org/10.1109/taffc.2019.2927337.
M. Egger, M. Ley, and S. Hanke, “Emotion Recognition from Physiological Signal Analysis: A Review,” Electronic Notes in Theoretical Computer Science, vol. 343, pp. 35–55, May 2019, doi: https://doi.org/10.1016/j.entcs.2019.04.009.
Md. A. Islam, Md. R. Z. Wami, and Md. R. Hayder, “Human Mental Condition Detection by EEG Signal and Time-Frequency Domain Analysis for Behavior,” 2021 International Conference on Science & Contemporary Technologies (ICSCT), Aug. 2021, doi: https://doi.org/10.1109/icsct53883.2021.9642653.
C. Zhu, P. Li, Z. Zhang, D. Liu, and W. Luo, “Characteristics of the regulation of the surprise emotion,” Scientific Reports, vol. 9, no. 1, May 2019, doi: https://doi.org/10.1038/s41598-019-42951-y.
A. Modirshanechi, J. Brea, and W. Gerstner, “A taxonomy of surprise definitions,” Journal of Mathematical Psychology, vol. 110, p. 102712, Sep. 2022, doi: https://doi.org/10.1016/j.jmp.2022.102712.
P. Ekman, R. Levenson, and W. Friesen, “Autonomic nervous system activity distinguishes among emotions,” Science, vol. 221, no. 4616, pp. 1208–1210, Sep. 1983, doi: https://doi.org/10.1126/science.6612338.
A. Barto, M. Mirolli, and G. Baldassarre, “Novelty or Surprise?,” Frontiers in Psychology, vol. 4, 2013, doi: https://doi.org/10.3389/fpsyg.2013.00907.
M. Faraji, K. Preuschoff, and W. Gerstner, “Balancing New against Old Information: The Role of Puzzlement Surprise in Learning,” Neural Computation, vol. 30, no. 1, pp. 34–83, Jan. 2018, doi: https://doi.org/10.1162/neco_a_01025.
V. Liakoni et al., “Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making,” NeuroImage, vol. 246, p. 118780, Feb. 2022, doi: https://doi.org/10.1016/j.neuroimage.2021.118780.
C. Findling, N. Chopin, and E. Koechlin, “Imprecise neural computations as a source of adaptive behaviour in volatile environments,” Nature Human Behaviour, vol. 5, no. 1, pp. 99–112, Jan. 2021, doi: https://doi.org/10.1038/s41562-020-00971-z.
J. W. Antony et al., “Behavioral, Physiological, and Neural Signatures of Surprise during Naturalistic Sports Viewing,” Neuron, vol. 109, no. 2, pp. 377-390.e7, Jan. 2021, doi: https://doi.org/10.1016/j.neuron.2020.10.029.
S. J. Gershman, M.-H. Monfils, K. A. Norman, and Y. Niv, “The computational nature of memory modification,” eLife, vol. 6, Mar. 2017, doi: https://doi.org/10.7554/elife.23763.
N. Rouhani and Y. Niv, “Signed and unsigned reward prediction errors dynamically enhance learning and memory,” eLife, vol. 10, Mar. 2021, doi: https://doi.org/10.7554/elife.61077.
H. A. Xu, A. Modirshanechi, M. P. Lehmann, W. Gerstner, and M. H. Herzog, “Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making,” PLOS Computational Biology, vol. 17, no. 6, p. e1009070, Jun. 2021, doi: https://doi.org/10.1371/journal.pcbi.1009070.
B. Gijsen, R. Montalto, J. Panneman, F. Falconieri, P. Wiper, and P. Zuraniewski, “Self-Healing for Cyber-Security,” 2021 Sixth International Conference on Fog and Mobile Edge Computing (FMEC), Dec. 2021, doi: https://doi.org/10.1109/fmec54266.2021.9732575.
M. Maheu, S. Dehaene, and F. Meyniel, “Brain signatures of a multiscale process of sequence learning in humans,” eLife, vol. 8, Feb. 2019, doi: https://doi.org/10.7554/elife.41541.
R. Hartmann, F. M. Schmidt, C. Sander, and U. Hegerl, “Heart Rate Variability as Indicator of Clinical State in Depression,” Frontiers in Psychiatry, vol. 9, Jan. 2019, doi: https://doi.org/10.3389/fpsyt.2018.00735.
N. K. Al-Qazzaz, M. K. Sabir, S. H. B. M. Ali, S. A. Ahmad, and K. Grammer, “Electroencephalogram Profiles for Emotion Identification over the Brain Regions Using Spectral, Entropy and Temporal Biomarkers,” Sensors, vol. 20, no. 1, p. 59, Dec. 2019, doi: https://doi.org/10.3390/s20010059.
K. M. Heilman, “Disorders of facial emotional expression and comprehension,” Handbook of Clinical Neurology, pp. 99–108, 2021, doi: https://doi.org/10.1016/b978-0-12-822290-4.00006-2.
T. Buchanan, D. Bibas, and R. Adolphs, “Salivary α-amylase levels as a biomarker of experienced fear,” Communicative & Integrative Biology, vol. 3, no. 6, pp. 525–527, Nov. 2010, doi: https://doi.org/10.4161/cib.3.6.12606.
N. Takai, M. Yamaguchi, T. Aragaki, K. Eto, K. Uchihashi, and Y. Nishikawa, “Effect of psychological stress on the salivary cortisol and amylase levels in healthy young adults,” Archives of Oral Biology, vol. 49, no. 12, pp. 963–968, Dec. 2004, doi: https://doi.org/10.1016/j.archoralbio.2004.06.007
N. Gullett, Zuzanna Zajkowska, A. Walsh, R. Harper, and V. Mondelli, “Heart rate variability (HRV) as a way to understand associations between the autonomic nervous system (ANS) and affective states: A critical review of the literature,” International Journal of Psychophysiology, vol. 192, pp. 35–42, Oct. 2023, doi: https://doi.org/10.1016/j.ijpsycho.2023.08.001.
M. M. Pulopulos, M.-A. Vanderhasselt, and R. De Raedt, “Association between changes in heart rate variability during the anticipation of a stressful situation and the stress-induced cortisol response,” Psychoneuroendocrinology, vol. 94, pp. 63–71, Aug. 2018, doi: https://doi.org/10.1016/j.psyneuen.2018.05.004.
R. Jerath and C. Beveridge, “Respiratory Rhythm, Autonomic Modulation, and the Spectrum of Emotions: The Future of Emotion Recognition and Modulation,” Frontiers in Psychology, vol. 11, Aug. 2020, doi: https://doi.org/10.3389/fpsyg.2020.01980.
Q. Zhang, X. Chen, Q. Zhan, T. Yang, and S. Xia, “Respiration-based emotion recognition with deep learning,” Computers in Industry, vol. 92–93, pp. 84–90, Nov. 2017, doi: https://doi.org/10.1016/j.compind.2017.04.005.
M. Jafari et al., “Emotion recognition in EEG signals using deep learning methods: A review,” Computers in Biology and Medicine, vol. 165, p. 107450, Oct. 2023, doi: https://doi.org/10.1016/j.compbiomed.2023.107450.
J. Zhang, D. Wu, F. Chen, Y. Shang, W. Zhang, and F. Wei, “Surprise Emotion Recognition of Pilots inside the Simulator Based on Multi-view Learning,” 2022 IEEE 4th International Conference on Civil Aviation Safety and Information Technology (ICCASIT), Oct. 2022, doi: https://doi.org/10.1109/iccasit55263.2022.9986697.
Copyright (c) 2024 Hendra Setiawan, Isnatin Miladiyah, Satyo Nuryadi, Alvin Sahroni
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlikel 4.0 International (CC BY-SA 4.0) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).