Method
Procedures
The philosophical foundation for the method of this study was based in the virtual
human interaction. Therefore, the fundamental assumption of the research design is that
the virtual static and motion stimuli are primary and necessary to understanding how an
individual senses others emotion through the digital platform as well as how an individual
fail to sense others emotions. Additionally, the participant encouraged
expressing comfort level of different relationships and acknowledged possible ambiguities
and contradictions in the dialogue.
Participant
Participants were 51 college students (20 men, 31 women) and 18 years of age (mean age = 25) or older from two universities (University of South Dakota and South Dakota State University) located in South Dakota. There were volunteer participants
and no credit and compensation were given for participating this study. 84.3% participants had lived in the Midwestern United States (e.g. Illinois, Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, North and South Dakota, Nebraska, Ohio, and Wisconsin). 15.7% participants were born in other places, but then moved to Midwestern and attended universities there. The ethnicity of participants is 84.3% of Caucasian, 7.84% of Asian, and 7.84% of His-panic.
Experimental Stimuli
A total of 103 survey questions are divided into four sections – (1) background information (7 questions), (2) emotional response (80 questions), (3) internet usage (9 questions), and (4) the comfort level in the video chat based on different relationships
(14 questions). In particular, the images and videos that used in the emotional response section of the survey, were created with four models from each generational group – Gen Z(Female), Millennial (Male), Gen X (Male), and Baby Boomer (Female). Gender
distribution of model was equal to two males and two females. There is a specific tone of colors that are a favor to the different gender. For example, warm tone is preferred by
women than men (Hillock 2003). Thus, colors were strategically removed from stimuli to ensure the direct influences from the visual, voice, and music only.
Procedure
The experiment was approved through the Institutional Review Board (IRB) of the University of South Dakota. In compliance with the IRB, all participant signed a consent form notifying them of their rights as a subject before the experiment began. A series of visual stimuli were used with voice and music to measure the effectiveness on sensing others emotional energy with eight questions. A sample question is “What emotion do you read?”. It is scored on a 2-point Likert scale. The voice and music have
positive and negative influence to support or distract the participant from sensing others emotion from given static image and video. The first-person point of view was used in
the positive voice - “I am so excited.” and the second-person point of view was used in the negative voice - “You hurt my feeling.” in order to create the personal connection
from the participants to the given visual stimuli. The positive and negative music was carefully chosen from the most well-known that have been used as sad and happy situation. “The lonely Man” from the incredible Hulk was used as sad music (negative)
Analysis of Facial Emotion Recognition Technology and Its Effectiveness 221 and “Merrie Melodies” from the Looney Tunes was used as a happy music (positive) in this survey. The comfort level in the video chat based on relationships was scored on a
six-point Likert scale ranging from very uncomfortable (1) to strongly comfortable (6). A sample question is “How do you feel when you do video chat?”.
The survey averaged 30 min in length and took place at two college locations - University of South Dakota and South Dakota State University. All participants were asked to complete their survey through the Qualtrics and no discussion was allowed
during the survey. There was an assistant in the lab to assist conducting survey and technical support.
Visual stimuli happy (left) and sad (right) emotional expressions
RESULTS
Data Screening. Prior to analysis, was explored to examine several crucial preliminary
topics, missing data, potential outliers, and normality. There was no missing data and a
total of 51 participants completed the experimental procedure. With all data point
included, it was normally distributed. The data was analyzed to compare the main effect
of voice and music (two independent variables) on sensing others emotional energy.
The Impact of Visual Stimuli Both Static Image and Video to Sense the Feeling
of Others. The study found that both static images and videos helps participants to
sense the feeling of others and taking them on as own significantly. The main effect of
both visual stimuli yielded an F(1, 50) = 0.009, p < .02, indicating a statistical significance.
In particular, video stimuli help participants to understand others feeling with
93% of chance, which is easier than static stimuli with 88% correctly reading others
feeling. In sum, participants correctly read the feeling of others regardless of the visual
stimuli type; however, the video stimuli help participants to sense the feeling of others
at much better rate (Table 1).
Table 1. Comparison between static image vs. video stimuli influences
The Impact of Music with Visual Stimuli to Sense the Feeling of Others. The study
found that both negative and positive music were influenced participants for their
decision making to understand others feeling. The main effect for music on sensing the
feeling of others with static image yielded an F ratio of F(1, 50) = 0.3, p < .001 and
with video yielded an F ratio of F(1, 50) = 0.3, p < .001, indicating a statistical significance.
The interaction of music and visual stimuli was significant and the result
indicated that the music influenced to sense the feeling of others for all participants. For
example, the music was distracted participants from reading correct feeling of others
when sad music with positive visual stimuli displayed together. However, the video
stimuli help to read the feeling of others with music influences whether it is a sad music
(negative) or happy music (positive) than the static image with music. In sum, the
music distorts the perception of reading feeling of others when the expression of visual
stimuli (e.g. happy) is opposite of the musical expression (e.g. sad) and increases the
perception of reading feeling of others when the expression of visual stimuli (e.g.
happy) is same as the musical expression (e.g. happy)
(Table 2). Comparison between static image with music vs. video with music influences
The Impact of Voice with Visual Stimuli to Sense the Feeling of Others. The study
found that both negative and positive voice were influenced participants for their
decision making to understand others feeling. The main effect for voice on sensing the
feeling of others with static image yielded an F ratio of F(1, 50) = 0.15, p < .001 and
with video yielded an F ratio of F(1, 50) = 0.2, p < .001, indicating a statistical significance.
The interaction of voice and visual stimuli was significant and the result
indicated that the voice influenced to sense the feeling of others for all participants. For
example, the voice was distracted participants from reading correct feeling of others
when sad music with positive visual stimuli displayed together. However, the video
stimuli help to read the feeling of others with voice influences whether it is a sad voice
(negative) or happy voice (positive) than the static image with voice. In sum, the voice
distorts the perception of reading feeling of others when the expression of visual stimuli
(e.g. happy) is opposite of the voice expression (e.g. sad) and increases the perception
of reading feeling of others when the expression of visual stimuli (e.g. happy) is same
as the voice expression (e.g. happy)
(Table 3). Comparison between static image with music vs. video with music influences
The Difference Between Voice and Music Influence. The study found that the voice
helps to sense the feeling of others significantly higher with p < .001 than the music.
Interaction of voice and visual stimuli was more significant than music and visual
stimuli. The result indicated that the voice influenced to sense the feeling of others for
all participants significantly better even pairing with opposite emotional expression
between two – e.g. negative voice (You hurt my feeling) and happy static image and
video stimuli (Table 4 and Fig 5).
(Table 4). Comparison between voice and music influences on sensing feeling of others
Fig. 5. Visual stimuli happy (left) and sad (right) emotional expressions
References.
Baumgardner, S.R., Crothers, M.K.: Positive Psychology. Pearson Education Inc., Upper
Saddle River (2009)
Cacioppo, J.T., Hawkley, L.C., Thisted, R.A.: Perceived social isolation makes me sad: 5-
year cross-lagged analyses of loneliness and depressive symptomatology in the Chicago
health, aging, and social relations study. Psychol. Aging 25(2), 453–463 (2010)
Collin, Catherine: The Psychology Book: Big Ideas Simply Explained. DK Publishing,
New York (2016)
Darwin, C., Prodger, P.: The Expression of the Emotions in Man and Animals. Oxford
University Press, New York (1998)
Ekman, P., Davidson, R.J., Friesen, W.V.: The Duchenne smile: emotional expression and
brain physiology: II. J. Pers. Soc. Psychol. 58(2), 342 (1990)
Gabrielsson, A.: Emotions in strong experiences with music. In: Juslin, P.N., Sloboda, J.A.
(eds.) Music and Emotion: Theory and Research, pp. 431–449. Oxford University Press,
New York (2001)
Hatfield, E., Cacioppo, J.T., Rapson, R.L.: Emotional contagion. Curr. Dir. Psychol. Sci. 2
(3), 96–99 (1993)
Henry, J.D., Von Hippel, W., Molenberghs, P., Lee, T., Sachdev, P.S.: Clinical assessment
of social cognitive function in neurological disorders. Nat. Rev. Neurol. 12(1), 28–39 (2016)
Hallock, J.: Colour assignment. (2003)
Fox, J.K.: Factors of Emotion Recognition in Faces: Three Perspectives (2004). http://www.
jyi.org
Larsen, J.T., McGraw, A.P., Cacioppo, J.T.: Can people feel happy and sad at the same
time? J. Pers. social Psychol. 81(4), 684–696 (2001)
Rainie, L., Zickuhr, K.: Video calling and video chat [Web blog post 2010]. http://www.
pewinternet.org/2010/10/13/video-calling-and-video-chat/
Barrett, L.F.: How To Understand Emotions? [Web video 2017]. https://www.youtube.com/
watch?v=GV7ui_KQXcE
Morland, R.L., Beach, S.R.: Exposure effects in the classroom: the development of affinity
among students. J. Exper. Soc. Psychol. 28, 255–276 (1992)
Rattue, P.: (Tuesday 14 February 2012). Sad Music Can Bring On Real Sadness [Web blog
post]. https://www.medicalnewstoday.com/articles/241653.php
Rozin, P., Berman, L., Royzman, E.: Biases in use of positive and negative words across
twenty natural languages. Cognit. Emot. 24(3), 536–538 (2010)
Soussignan, R.: A test of the facial feedback hypothesis. American Psychological
Association (2002)
Russell, J.A., Carroll, J.M.: On the bipolarity of positive and negative affect. Psychol. Bull.
125(1), 3–30 (1999)
