Analysis of Multimodal Biosignals during Surprise Conditions Correlates with Psychological Traits

Surprise can simultaneously represent bad or good, pleasant or unpleasant, with the same experiences since understanding how humans' physiological qualities link with their emotional or mental health is required. Our aim to conduct this quantitative research is to concisely correlate and objectively measure mental stress and emotional issues by measuring brain activity, breathing, and heart rate in real time while executing specialized audio-visual stimulation to elicit a surprise event. By proposing this study, we can evaluate and obtain a better understanding of how psychological changes correlate to physiological properties. We evaluated the frequency and temporal domain characteristics to determine if physiological measurements matched biochemical metrics and subjective stress assessments during the elicit surprise condition experiment. We discovered that the brain is still preferable to most in recognizing a human's psychological changes over a short period of time. The temporal (T3) (r = 0.544, p = 0.005) and frontal (Fz) (r = 0.519, p = 0.008) regions were shown to correlate with salivary amylase activity. In comparison to other channels, there was a negative association between stress perception and the occipital site (O1, r = -0.618, p = 0.001). We also found that heart rate variability activity correlates with arousal perception. By looking at specific multimodal biosignals, it is possible to understand human psychological traits by recording specific physiological signals for daily mental health monitoring.


I. INTRODUCTION
The rapid growth of smart cities has raised several serious concerns, one of which is about emotional well-being [1] There are numerous factors that contribute to a person's happiness or misery, and one of them is their emotional wellbeing.These factors include the intensity and frequency with which one experiences emotions such as fascination, joy, sadness, anxiety, affection, and anger, which determine whether one's life is good or bad [2].Emotional changes are related to changes in the stressor, and numerous studies have established a link between emotion and stress [3].
As the development of technology has already spread to many fields, one of the most important findings is how to estimate and recognize psychological changes, including emotion and stress, using physiological data.This approach is now being called psychophysiological, which correlates psychological processes and behavior and the impact of psychological or behavioral manipulations on physiology [4].Nowadays, stress and emotions primarily can be identified, estimated, or classified using several biosignals such as heart rate variability, brain activity, electrodermal, and face recognition [5,6].
Most previous studies have reported biosignals' effectiveness in classifying emotions and stress with various accuracies [6][7][8].However, the main issues are still how we can deeply understand specific psychological characteristics rather than binary or classification problems.While a deep investigation is established, more meaningful intakes can improve the detection method.
For instance, it is a surprise emotion.Surprise emotion is one of the basic emotions that can simultaneously represent bad or good, pleasant, and unpleasant with the same experiences [9].Another illustration: Picture waking up one morning and seeing that the street in front of your apartment has just been dusted with new snow.When you witness white streets after expecting a warm, sunny morning due to the weather prediction, you feel "surprised."As a result of surprise, numerous neurons in your brain and possibly other body parts change their activity [10].
The challenge to recognize specific emotions is understanding the characteristics or traits, including physiological aspects and the regulation regarding the surprise response.Since surprise response can elicit other emotions such as fear, happiness, joy, or stress, we believed that quantifying the biosignals during surprise conditions could enhance the study of human emotions especially the definition of surprise condition based on physiological background.
Several studies have already reported detecting surprise emotion on different conditions.One of the most fundamental findings was the study of Ekman et al., where the surprise emotion can be represented by the autonomic nervous changes during low heart rate activity [11].The challenge of detecting surprise emotion during the facial task was the same characteristic of happiness and sadness during low heart rate activity.
Over the last few decades, various definitions and formal measures of measures have been proposed and studied [12][13][14][15].According to this, the surprise measure has already yielded several results by utilizing brain measurement to explain the role of the surprise emotion itself [16][17][18][19].Also, previous research has identified surprise signatures in behavioral and physiological measurements [19][20][21].However, several questions remain, such as how the surprise emotion can be related to stress conditions and how it can be explained based on multimodal biosignals representation and biochemical measurement, such as saliva.Instead of employing a complex classification method to recognize the surprise emotion, our proposed study aims first to find out how the physiological properties respond to the surprise condition and then find out the meaning of each surprise condition according to the biosignal representation and how it corresponds to the psychological subjective evaluations.
Our contribution to this study is to build on previous work by using multimodal biosignals and analyzing how surprise conditions were connected to stress scales and subjective emotion evaluations using saliva biochemical characteristics over several visual and auditory surprise event stimuli trials.This study strives to ascertain whether surprise emotion can be described by the link between biosignals and subjective assessment (as a biomarker) and how we can deeply understand the type of surprise condition by processing and quantifying the physiological signal properties.In conclusion, this study provides significant insight into how the surprise emotion expresses distinct psychophysiological changes, whether they are positive or negative, resulting from visual and aural inputs, as well as how they relate to biosignals.

A. ETHICAL CONSIDERATION
Before the experiments, the subjects were informed of the research topic and procedure, and their informed consent was obtained.This study was conducted according to the principles of the Declaration of Helsinki, and all subjects' data were kept confidential and anonymous.This study followed the ethical committee's consideration, provided that it was already approved by the medical faculty of Universitas Islam Indonesia Yogyakarta with document number 19/Ka.Kom.Et/70/KE/VI/2022.

B. SUBJECTS
We realize that the study requires population diversification to get more general outcomes.The goal of our proposed study, on the other hand, is to show that the changes in physiological properties are caused by the stimulation itself and not by outside factors like background, age, or level of education, which could lead to biased results.
To ensure we had a uniform environment, five male subjects working at a university participated in this study.Their ages were between 22 and 30 years old (30.4 ± 9.6 years).The distribution of their heights and weights was 168.4 ± 7.8 cm. and 67.8 ± 7.8 kg, respectively, without any medication treatment during the experiment.The subjects were also healthy and did not experience any brain, respiratory, or cardiovascular diseases.We ensured the subjects got enough sleep before the experiment (pre-treatment) and during the experiment as well to avoid any different outcomes.

C. EXPERIMENTAL DESIGN
The experiment was held in a soundproofed room between 10:00 a.m. and 2:00 p.m., and each experiment took 20-30 minutes.The details of the experimental design can be seen in FIGURE 1.The subjects need to sit and relax in front of a 16inch monitor, as shown in FIGURE 2. This experiment was designed to elicit specific emotions and stress by stimulating random pictures to distract participants' attention and then showing a short video that suddenly surprised the subjects.Each subject saw five different videos or experienced five trials; then, twenty-five data points were collected to be analyzed, as illustrated in FIGURE 1.Each trial needs to be done in around 3 minutes, including the preparation for each participant.The detail of experiment steps as follows: 1) Participants came and sat down and relaxed in front of the monitor.2) The operator provided a cue when the experiment started.
3) In each trial, several random pictures are shown that represent a face or situation to distract the participants attention for 10 seconds.4) The main event was a surprise scene shown for 5-10 seconds.5) After finishing, the operator asked participants to fill out subjective assessments as psychological evaluations.6) Continue in this manner until each subject has completed five trials, each with a distinct scene depicting a surprising event.When the experiment began, we recorded their biosignals (respiration, cardiovascular system, and brain) during the experiment, which involved visual-audio stimulation.We also measured salivary amylase activity as a biochemical parameter for mental stress after each trial finished to accompany employing self-stress scoring, valence, and arousal to assess their subjective psychological condition.

D. DATA COLLECTION
We collected both physiological and psychological data among five participants.Firstly, the brain properties were collected by using an international 10-20 electrode placement system from 18 electrode sites that were placed on the participant's head's scalp.The heart and respiration data were collected through poly-channels from the brain activity recorder by using a limb lead 2 placement system and a nasal breathing sensor, respectively.For the psychological data, we evaluated it using a standardized questionnaire filled out by the participants after getting stimulated.

E. SUBJECTIVE PSYCHOLOGICAL ASSESSMENT
Since the experiment was designated to elicit mental changes (both emotion and stress), we assessed the psychological characteristics using several subjective evaluations by using questionnaires with Likert Scale scores.After each stimulation, the psychological assessment was held (a total of five trials for each subject) for 15 seconds.We demonstrated the Self-Assessment Manikin (SAM) to quantify the emotion (arousal and valence) from surprise emotion.The scores of valence and arousal ranged from -2 to 2, where the positive value means a pleasant emotion and the negative, an unpleasant one, with its intensity.To confirm the stressor level, we quantify the self-stress evaluation regarding their current stressor after seeing the visual stimulation from 1 to 5. A higher score means the stressor level has increased.

F. SIGNAL PROCESSING AND FEATURES EXTRACTION
Data from Russia's Deymed Mitsar EEG device was sampled at 250 Hz and provided brain activity (EEG), respiration, and heart rate activity all at the same time.The Arduino microcontroller sampled the other electrical signal from the PPG sensor at 160 Hz and solely used it as a comparison to the Mitsar device's heart rate activity.A Finite Impulse Response (FIR) bandpass filter with a cut-off frequency of 0.5 to 30 Hz was used to filter the entire data set (equation ( 1)).During the selected time segment,  " as filter coefficient filtering digitized data ( − ) from the biosignal recorder compensated k as filter length.We extracted the band power from EEG data regarding the fundamental band frequency, namely delta (<4 Hz), theta (4-7 Hz), alpha (8-12 Hz), and beta (>13 Hz).We employed the Fast Fourier Transform (FFT) based on equation ( 2) from the raw data during sixsecond visual stimulation on every trial of all subjects to obtain the power of frequency  (") .Then, the relative power metric was calculated by using equation (3).(2) From equation ( 3), the P represents the relative power of corresponding band power (delta, theta, alpha, and beta), where  (") is the spectral density of the k index of the frequency range.N is the total frequency index, and TP is the total power from desired band frequency (0.5 to 30 Hz).The heart rate activity was obtained from ECG lead II placement.We detected the Peak-to-Peak Interval from the heartbeat during the visual stimulation then quantified linear time-domain heart rate variabilities (HRV) parameters such as MeanRR, SDRR, rMSSD, and CVRR (ratio between MeanRR and SDRR).To extend the feature extraction, we also employed non-linear analysis by extracting the Poincare plot parameters, pSD1 and pSD2 [22].The heart rate activity features were presented in equations ( 4) - (8).
The last biosignal detected in our investigation was the rhythm of breathing.We used the Fast Fourier transform on equation ( 2) to determine the frequency components of the respiration sensor.We chose the frequency of the maximum peak as our fundamental frequency.

G. DATA ANALYSIS
We realized that this study was established only with five subjects, which means microscopic samples.Each subject experienced five trials.Therefore, the total number of observations consists of twenty-five data points.To interpret the results, we carefully observed the intersubject analysis.The population's center was calculated using the mean and standard error.Since the purpose of this study was to model the biosignals to represent human emotion and stress, we employed Pearson correlation analysis and supported it with descriptive analysis.We present the descriptive data using the mean and standard error (SE).
To do any statistical test, we applied non-parametric analysis (Kruskal-Wallis and posthoc Dunn) with Bonferroni adjustment to compare each combination of each factor, namely inter-subject and inter-trials analysis.We considered the data significantly different if the p-value was less than 0.05.Post-experiment, we collected several psychological assessments to obtain more comprehensive information than other studies that often compare it with limited psychological measurements.As provided in TABLE 1, we can see that after experiencing the surprise condition, the participants provide the valence score measurement that shows unpleasant emotions based on the valence measurement from the SAM questionnaire from the trials.In addition, the arousal measurement shows a neutral interest in the visual stimuli, which means that the unpleasant feeling is engaged fairly.We provided salivary amylase activity (sAA) to the subjects as a comparison to the psychological assessment.We found that the #1 and #2 trials had a higher sAA than the other trials.It shows that, post-experiment, the sAA level shows a significant  increase from the beginning of the experiment compared to the end of the trials.

B. BRAIN ACTIVITY
To begin, we examined the brain activity recorded by the EEG amplifier.Due to the visual cues employed in the experiment, the alpha and beta bands are frequently noticed in awake or active conditions.FIGURE 3 depicts participant brain activity during five trials while they were exposed to a surprise video or stimulus lasting five seconds.On the total electrode site, the beta band has a significantly larger relative power than the alpha band, indicating that the brain is in active condition.
As we observed the relative power of each frequency band, we found that the #2 trial had an inferior brain response compared to the other trials, especially in the frontal, central, and parietal lobes.The subject's alpha-band relative power can be observed from 10 to 18 relative power during the #2 trial.
Nonetheless, the beta band's relative power is doubled compared to the alpha ones, and the #2 trial is superior among other trials, both from the posterior and anterior of the brain.The results show that the relative power was higher than 30 during the #2 trial.According to this finding, we got similar results from the sAA measurement: the beginning of the trials produced a higher response compared to the end of the experiment as well.
To correlate the brain activity with psychological assessment, we demonstrated the Pearson correlation between each electrode's band power and psychological assessment, namely valence and arousal as emotion measurements and sAA with the stress scale to evaluate mental stress level (FIGURE 4 and FIGURE 5).Twenty-five data points were retrieved from each measure, both the physiological and psychological assessments.

FIGURE 4. The correlation of alpha-band and SAA as the stressful assessment based on biochemical measurement
According to TABLE 2, the EEG's band power is not well correlated with the emotion measurement (valence and arousal) in both alpha and beta band frequencies, where the correlation values are less than 0.05 (p-value > 0.05).Besides, the insight of those results is that the arousal assessment shows a quite potential and positive correlation between beta-band activity and arousal score assessment, especially in a frontal area such as F7, F3, and Fz with 0.38 (p-value = 0.06), 0.34 (p-value = 0.089), and 0.39 (p-value = 0.0512) correlation scores, respectively.Since the arousal scores have a positive correlation value with beta-band power, the higher beta-band activity represents the higher intensity (excitement) of a specific emotion (surprise emotion).
Then, we also investigated how brain activity correlates with the biochemical measurement (sAA) and stress scale selfassessment.FIGURE 4 shows that the alpha-band activity is primarily positively correlated with the stress assessment from their saliva samples.The most correlated electrode site is in the temporal area (T3), with a correlation value greater than 0.5, followed by Fz in the frontal area (p-value < 0.01).However, such a good correlation is not observable in betaband activity.
Lastly, after seeing the visual stimuli, we observed the correlation between brain activity and the stress scale selfassessment.We found that the alpha band showed a negative correlation and it did not represent the same trend when compared to the sAA assessment.As shown in FIGURE 5, the relative alpha power tends to correlate (p-value = 0.05) on most electrode sites except T4.The highest correlation was found in the occipital lobe (O1), where the correlation value was more than 0.61.

C. CARDIAC AND RESPIRATION ACTIVITY FIGURE 5. The correlation between alpha-band activity and the stress scale
The heart rate and respiration activity are two additional biosignals that should be investigated.The time domain parameters were extracted using both linear (MeanRR, SDRR, CVRR, and rMSSD) and non-linear (pSD1 and pSD2).For the respiration properties, we observed the peak frequency that corresponded to the breathing pattern.TABLE 3 shows that the correlation values were less than brain activity unless the correlation between the heart rate activity and arousal assessment could be examined as a possible metric, and the positive correlat ions were statistically significant on the SDRR, CVRR, pSD1, and pSD2 (correlation score > 0.6, pvalue < 0.01).The negative correlation can be found on MeanRR and rMSSD.However, the correlation properties are not as strong as those of the other parameters.According to  the results, we found that a surprise condition increased the heart rate and variability of the cardiovascular system.

D. ANALYSIS BETWEEN SUBJECTS AND TRIALS
To strengthen our findings, we established a non-parametric statistical analysis based on subject and trial factors.We applied the posthoc Dunn test to observe each factor's combination.Firstly, during inter-subject analysis, we found that several parameters were often significantly different (p < 0.05).FIGURE 6 shows that overall parameters mostly differentiate between subjects (S), especially S2 vs. S3, and S2 vs. S5 (MeanRR, SDRR, CVRR).Besides, the brain parameters on FIGURE 7 do not show any significant differences.This result proves that physiological properties between participants are highly different.Therefore, uniformizing the treatment is the most appropriate way to ensure any physiological differences between trials are based on the emotional changes and not biased by the physiological properties.
Conversely, the inter-trials analysis shows that only brain parameters had significant differences among all electrode sites, both in alpha and beta band frequencies.FIGURE 8 and FIGURE 9 demonstrates that the alpha band distinguishes all electrode sites between trial 2 and trial 5 (p < 0.01), then trial FIGURE 7. Brain properties between subjects on alpha band frequency 3 and trial 5 (p < 0.05).We discovered that all electrodes in the beta band, where the brain is actively working, are significantly different, with trial 2 vs. trial 5 most frequently occurring with a p-value less than 0.001 (p < 0.001), followed by trial 2 vs. trial 5 and trial 2 vs. trial 4 (p < 0.05).According to these results, we concluded that the stimulation between trials is able to elicit brain changes and prove that the brain has the most sensitive physiological properties to recognize emotion changes during surprise events.

IV. DISCUSSION
Surprise is one of the fundamental emotions because it is the only one that can convey both positive and negative emotions, or, as some researchers put it, pleasant and unpleasant emotions.Then, surprise has the potential to elicit additional psychological properties and emotions such as fear, amusement, happiness, and sadness, as well as possibly exacerbate human stressors.Our study discovered that the properties of salivary amylase, a biochemical process that represents a human stressor, can also change in response to surprise emotions.
We all fully comprehend that emotion and mental stress are processed by the central nervous system, or brain.Previous research on the detection of emotional and mental stress using physiological signals concluded that brain activity is the most sensitive biosignal and parameter for recognizing emotions, including surprise [23].During the surprise condition, our proposed study discovered a correlation between the frontal (Fz) and temporal (T3) lobes and sAA, a salivary enzyme that is typically used to assess mental state and represents sympathetic nervous system activity [24].Additionally, the result confirms the current consensus that the frontal lobe is the most critical brain region for emotion processing when compared to other regions [23].While it is well established that the temporal lobes are involved in the processing of affect and emotion, the relationship between the brain and sAA is unknown and requires further investigation.The most likely reason is due to the facial expression elicited by the visual stimuli of the surprise scene [24].
Other biological signals, such as heart rate and breathing activity, did not show any strong relationship with sAA except for arousal with heart rate activity.The arousal is mostly positively correlated with heart rate variability parameters (SDRR, CVRR, pSD1, and pSD2), and it depicts that the surprise emotion affects the variability of the autonomic nervous system.We confirmed in the previous study that arousal reflects the intensity of particular emotions and sympathetic activity [23,25].

FIGURE 9. Brain properties between trials on beta band frequency
During our study, the subject underwent five trials.We carefully observed and analyzed the physiological signals in relation to the psychological properties commonly used as a standard measurement.We found that arousal scores and sAA activity accurately show how physiological signals change during the surprise emotion.Surprise is a unique emotion because it can represent both good and bad emotions that are strongly linked to arousal and use the same brain regions for basic emotion processing.
The sAA associated with mental stress shares the same relationship as the brain activity associated with surprise emotion, and it extends another finding to the fear condition as well [25,26].One of the legacies of discovering how to distinguish basic emotions was through heart rate activity.Surprise emotions, like happiness and disgust, had low heart rate activity; however, this claim is considered the baseline finding [11].To deeply confirm, previous studies need to be obversed as a comparison.
Using physiological characteristics to detect surprise emotions, we contrasted our findings with those of earlier research.We discovered that the majority of research on HRV and respiratory features primarily addressed broad emotions like positive and negative affect [27].Furthermore, because of the cortisol response brought on by stress and the expectation of stressful situations, HRV does not adequately represent emotions [28].Furthermore, the breathing pattern detected certain emotions using rhythm, although only in general terms and not in terms of particular emotions [29,30].Therefore, we thought that the confirmation of our results about fewer significant variations between trials demonstrated the inability of HRV and breathing patterns to differentiate between certain emotional events or surprise conditions.
The most intriguing finding is that we have verified the brain's continued superiority in identifying and differentiating between distinct reactions to specific unexpected situations (trials comparison).Our results also demonstrate that the brain, or central nervous system, can distinguish between distinct emotion dimensions or objectives [31,32].Then, in contrast to heart rate and breathing rhythm, the earlier research "still" indicated that the brain is mostly utilized to aid in grasping emotion mechanisms.Because surprise feelings can simultaneously express both positive and negative emotions, research on surprise emotions is currently lacking, still not able to be replicated, and will be interesting in the future.
Because we discovered that specific emotions retain many of the same psychological properties as other psychological properties, we realized that the pipeline for detecting mental and/or psychological conditions must include more than one layer.We discovered that the first layer could be the heart rate activity, followed by the brain activity.This type of pipeline will be more efficient at reducing computation time when it is based on a single model that requires a large number of inputs to achieve high accuracy.
As usually expected, this study has limitations.Firstly, the number of subjects that were counted as microscopic subjects with a specific environment and demography.As we designed this study to observe the visibility needed to recognize a surprise condition with specific stimuli, we need to evaluate and validate it with various conditions, subject backgrounds, demographics, and real-world problems as future directions.Another important future issue is distinguishing different surprise conditions based on happy, sad, or angry emotions, which usually represent surprise conditions.Since the surprise condition is able to trigger other emotions, we need to demonstrate valuable physiological properties to discriminate against the surprise-based traits.Those future directions also need to consider various factors, such as genders, working environments, and types of jobs.
Thus, our future research should expand our sample size and validate our pipeline's ability to detect specific emotions in response to a variety of stimuli, including fear, happiness, sadness, disgust, and anger.Additionally, and perhaps most importantly, we must understand the physiological mechanism in order to characterize psychological properties with their associated meanings.Currently, we suggest focusing on brain studies as a superior feature compared to others.
Finally, our findings demonstrate a possible method to measure and detect specific psychological conditions by using multimodal biosignals with selected physiological properties.Based on our results, the brain sensor is the most important feature to evaluate the emotion and stressor compared to the others.Since a wearable brain sensor exists, the measurement can be attached during a meeting with the patient.The second layer is to utilize a heart rate sensor, which is much more affordable but lacks psychological dimension since the heart rate also represents the homeostatic condition.The appropriate objective measurement can be customized in accordance with psychological purposes.

VII. CONCLUSION
Regarding our findings, we concluded that our proposed study found that the surprise emotion still shares the same brain lobe with other emotions (the frontal lobe), and the other lobe (the temporal lobe) shows correlation with psychological properties called sAA as the biochemical parameter for mental stress assessment (p < 0.01).Another biosignal, heart rate activity, is correlated with the arousal score, which represents the intensity of emotion and means that during surprise conditions, it also affects autonomic nervous system activity.We confirmed that the brain is still a superior physiological biomarker to assess emotion changes and stressors rather than HRV features.However, future studies still need to be established to validate our pipeline planning to recognize specific emotions more effectively and provide meaningful results rather than an accuracy percentage.
We also would like to send our gratitude to our participants who are willing to join this study.

FIGURE 3 .
FIGURE 3. The EEG's band power on alpha and beta bands on a subject during experimentation from all trials

FIGURE 6 .
FIGURE 6.The correlation between alpha-band activity and the stress scale for each electrode

FIGURE 8 .
FIGURE 8. Brain properties between trials on alpha band frequency )