Text vs. Images: Understanding Emotional Expressions on Social Media During COVID-19 Pandemic

ABSTRACT
Due to the global spread of COVID-19, people all around the world have been forced to change the way they communicate and interact with others. Keeping social distance and wearing masks helps prevent the spread of coronavirus, and also makes online social platforms increase in demand in an unprecedented way (Flynn, 2008). Prolonged social isolation during COVID-19 is likely to have negative effects on mental health and communication on an individual. Researchers have found evidence for caused and elevated anxiety disorders such as somatization, post-traumatic stress disorder, panic disorders and depression amongst individuals during the COVID-19 pandemic (Meikle, 2016). Numerous studies have found that people only show their “good side” and positive emotions on social media. How does social media reveal our anxiety disorders during Covid? Do emotions expressed in pictures match with its text content on social media? In this research, 500 most recent selfies from individual accounts between December 1st and 10th in 2021 from age ranges 13 to 55 years old were downloaded for the study. The study used IBM Watson tone analyzer and SkyBiometry as tools for linguistic analysis and emotion detection. In addition, the research compared imagery and text content in social media as a function of emotional expression and methods.

ISBN 978-1-958651-25-4
DIO 10.54941/ahfe1002031
PUBLICATION DATE 2025
AUTHOR Qiuwen Li
CATEGORIES Data Visualization
KEYWORDS Emotional expressions, Communication, Social media, COVID-19, Photography posts, Text, Instagram, Social network, Attention theory, Mental health

Method

Procedures

The objective for this study was to examine how does social media reveal our anxiety disorders during the pandemic. This study conducted a content analysis of Instagram posts tagged with hashtags #covidlife, #covidusa, and #quarantine. 500 most recent selfies from individual accounts between December 1st and 10th in 2021 from age ranges 13 to 55 years old were downloaded for the study. To compare imagery and text content in social media as a function of emotional expression and methods, the study used open-source two recognition software. IBM Watson tone analyzer uses linguistic analysis to detect emotional tones in written text, and SkyBiometry as emotion detection found in selfie picture. All emotions were counted and categorized into three groups: positive, negative, and pre-affective. In addition, the study examined whether the emotions explicitly detected was clearly inconsistent when they were visual or textual for the same post. All variables were coded dichotomously (0 = absent, 1 = present) by one coder. For the reliability analysis, we randomly selected 10% of the material. The analysis generally indicated reliable measurement. Krippendorff’s α values were acceptable all (all values, α > .76).

Variables

Emotions Detection. The following emotions were coded when they were detected on the selfie picture: neutral, angry, disgusted, scared, happy, sad and surprised. If more than one emotion is present, the stronger one is shown. All emotions were counted and categorized into three groups: positive, negative, and pre-affective (can be positive and/or negative, depending on the goal conduciveness of the event).

Linguistic Tone Analysis. The following tones were coded when they were found in the text post: joy, fear, sadness, anger, analytical, confident and tentative. If more than one tone is present, the stronger one is shown. All tones were counted and categorized into three groups: positive, negative, and neutral.

Inconsistency. The research examined whether the emotions explicitly detected was clearly inconsistent when they were visual or textual for the same post. Inconsistency was coded as present (0 = absent, 1 = present). For instance, when the post presented a positive emotion found in selfie picture with a contradictory (negative or neutral tone) text post.

Number of Likes. The research explored the relationships between response behavior and emotions detected from the post. The study collected and analyzed how many likes each post received.

Demographics. A total of 500 most recent selfies from individual accounts between December 1st and 10th in 2021 were downloaded from both sex groups, 250 males and 250 females. The age of the individual was coded, whether the post included young individuals, older adults, or held no agerelated information.

RESULTS

The study found that positive emotions are still more prevalent than negative and pre-affective emotions on Instagram, for both selfie pictures and text posts during the COVID-19 pandemic. In Figure 6, positive emotions (44%) were detected on the selfie pictures as 10% more than negative emotions and 22% more than pre-affective emotions. Regardless of biological sex, positive emotions (54%) were detected on the text posts as 46% more than negative emotions and 16% more than pre-affective emotions. Compared with words, the results showed that people are more likely to reveal their negative emotions (e.g. sadness, angry) on image than text.

Single Project

Figure 1: Percentage of posts showing emotion categories. N D 500.

The research also examined gender-based differences in emotion expression on both pictures and words. Among posts, the highest percentage was positive emotions from females for both visual (28%) and textual (31%) contents. Within the negative and pre-affective emotions, there were same percentages for males and females found in the text posts. But in selfie pictures, the percentages of negative and pre-affective emotions from males were higher in comparison with females.

In terms of the relative numbers of likes received, females were received 2.8 times more likes than males. However, it is important to consider that there were variety numbers of likes for each post based on the popularity of an individual. Moreover, the study found that both male and female posted a significantly larger number of posts (53%) which visually and textual are inconsistent in emotion expression.

Single Project

Figure 2: This pie chart illustrates the percentages of each emotion categories detected on the selfie pictures between males and females.

Single Project

Figure 3: This pie chart illustrates the percentages of each emotion categories detected on the text posts between males and females.

Table 1. Instagram post of the total number of likes & inconsistency between picture and text.

Single Project

References.

Advice, S. G. S. (1911). Syracuse Post Standard. March, 28, 18.
Baumgardner, S.R., & Crothers, M.K. (2009). Positive Psychology. Pearson Education,Inc.
Czeisler, M. É., Lane, R. I., Petrosky, E., Wiley, J. F., Christensen, A., Njai, R., ... & Rajaratnam, S. M. (2020). Mental health, substance use, and suicidal ideation during the COVID-19 pandemic—United States, June 24–30, 2020. Morbidity and Mortality Weekly Report, 69(32), 1049.
Company’s News (2021). 15 Best Face Recognition APIs in 2021: RecFaces' Overview. Retrieved from https://recfaces.com/articles/face-recognition-apis
Cohn, M. A., & Fredrickson, B. L. (2009). Positive emotions. Oxford handbook of positive psychology, 2, 13–24. 9. https://datareportal.com/social-media-users
Daily Time Spent on Social Networking by Internet Users Worldwide from 2012 to 2020 (in minutes). Retrieved from https://www.statista.com/statistics/433871/da ily-social-media-usage-worldwide/
Dentzel, Z. (2013). How the internet has changed everyday life. Ch@ nge, 19. O’Keefe, G., Clarke-Pearson, K. (2011). Clinical report – The impact of social media on children, adolescents, and families. Pediatrics, 127, 800–804.
Darwin, C., & Prodger, P. (1998). The expression of the emotions in man and animals. Oxford University Press, USA.
Flynn, C. P. (2008). Social creatures: a human and animal studies reader. Lantern Books. Freitas, D. (2017). The happiness effect: How social media is driving a generation to appear perfect at any cost. Oxford University Press.
Fullerton, N. (2020). Instagram vs. Reality: The Pandemic’s Impact on Social Media and Mental Health.
Grieve, R., Indian, M., Witteveen, K., Tolan, G. A., & Marrington, J. (2013). Faceto-face or Facebook: Can social connectedness be derived online?. Computers in human behavior, 29(3), 604–609.
Global Social Media Stats. Retrieved from https://datareportal.com/social-mediausers Goffman, E. (1978). The presentation of self in everyday life (Vol. 21). London: Harmondsworth.
Gautam, S. (2020). The influence of COVID-19 on air quality in India: A boon or inutile. Bulletin of Environmental Contamination and Toxicology. https://doi.org/10.1007/s00128-020-02877-y.
Harrison, Jamie (2013). Battle of the Sexes: Are Women More Self-conscious in the Office than Men? Retrieved from https://www.blackenterprise.com/study-workpl ace-perception-office-women-care-more/
IBM Watson Tone Analyzer Review. Retrieved from https://www.slant.co/options/26844/~ibm-watson-tone-analyzer-review
Juno, Jennifer., Immunity., Wheatley, Adam (2021). Mounting evidence suggests COVID vaccines do reduce transmission. How does this work? Retrieved from https://www.gavi.org/vaccineswork/mounting-evidence-suggests-covid -vaccines-do-reduce-transmission-how-does-work
Li, Q., & Kim, Y. A. (2020, July). Happiness Participatory Media: Cultural Differences in Happiness on Instagram. In International Conference on Applied Human Factors and Ergonomics (pp. 394–402). Springer, Cham.
Li, Q., & Kim, Y. A. (2018, July). Analysis of Facial Emotion Recognition Technology and Its Effectiveness in Human Interaction. In International Conference on Applied Human Factors and Ergonomics (pp. 217-226). Springer, Cham.
Meikle, G. (2016). Social media: Communication, sharing and visibility. Routledge. Neumeier, M. (2005). The Brand Gap: Revised Edition. Peachpit Press.
Perry, Rose., Braren, Stephen Braren. (July 2021). Social Connectedness 101: Defining Social Connectedness. Retrieved from https://www.thesocialcreatures .org/thecreaturetimes/social-connectedness-101-defining-social-connectedness#: ~{}:text=Humans%20are%20social%20creatures%20who,even%20how%20l ong%20we%20live.
Rodriguez, Tori (2013). Negative Emotions Are Key to Well-Being. Retrieved from https://www.scientificamerican.com/article/negative-emotions-key-well-being/
Social Video: The Marketing Trend that’s Worth Investing. Retrieved from in https: //www.bluecorona.com/blog/social-video-marketing-trend/
Soucheray S (June 29, 2021). US COVID-19 cases rise, likely due to Delta variant. CIDRAP. Retrieved July 11, 2021.
WHO Coronavirus (COVID-19) Dashboard. Retrieved from https://covid19. who.int/
Yean, Chew (2015). Emotion Detection and Recognition from Text Using Deep Learning. Retrieved from https://devblogs.microsoft.com/cse/2015/11/29/emotion-de tection-and-recognition-from-text-using-deep-learning/