No AccessJournal of Speech, Language, and Hearing ResearchResearch Article1 Feb 2016

Prosody and Semantics Are Separate but Not Separable Channels in the Perception of Emotional Speech: Test for Rating of Emotions in Speech


    Our aim is to explore the complex interplay of prosody (tone of speech) and semantics (verbal content) in the perception of discrete emotions in speech.


    We implement a novel tool, the Test for Rating of Emotions in Speech. Eighty native English speakers were presented with spoken sentences made of different combinations of 5 discrete emotions (anger, fear, happiness, sadness, and neutral) presented in prosody and semantics. Listeners were asked to rate the sentence as a whole, integrating both speech channels, or to focus on one channel only (prosody or semantics).


    We observed supremacy of congruency, failure of selective attention, and prosodic dominance. Supremacy of congruency means that a sentence that presents the same emotion in both speech channels was rated highest; failure of selective attention means that listeners were unable to selectively attend to one channel when instructed; and prosodic dominance means that prosodic information plays a larger role than semantics in processing emotional speech.


    Emotional prosody and semantics are separate but not separable channels, and it is difficult to perceive one without the influence of the other. Our findings indicate that the Test for Rating of Emotions in Speech can reveal specific aspects in the processing of emotional speech and may in the future prove useful for understanding emotion-processing deficits in individuals with pathologies.


    • Algom, D., Chajut, E., & Lev, S. (2004). A rational look at the emotional Stroop phenomenon: A generic slowdown, not a Stroop effect.Journal of Experimental Psychology: General, 133, 323–338. doi:10.1037/0096-3445.133.3.323
    • Ardila, A. (2007). Normal aging increases cognitive heterogeneity: Analysis of dispersion in WAIS-III scores across age.Archives of Clinical Neuropsychology, 22, 1003–1011.
    • Astésano, C., Besson, M., & Alter, K. (2004). Brain potentials during semantic and prosodic processing in French.Cognitive Brain Research, 18, 172–184.
    • Bagby, R. M., Parker, J. D. A., & Taylor, G. J. (1994). The twenty-item Toronto Alexithymia Scale—I. Item selection and cross-validation of the factor structure.Journal of Psychosomatic Research, 38, 23–32.
    • Balota, D. A., Yap, M. J., Cortese, M. J., Hutchison, K. A., Kessler, B., Loftis, B., … Treiman, R. (2007). The English Lexicon Project.Behavioral Research Methods, 39, 445–459.
    • Beaucousin, V., Lacheret, A., Turbelin, M.-R., Morel, M., Mazoyer, B., & Tzourio-Mazoyer, N. (2007). fMRI study of emotional speech comprehension.Cerebral Cortex, 17, 339–352. doi:10.1093/cercor/bhj151
    • Ben-David, B. M., & Algom, D. (2009). Species of redundancy in visual target detection.Journal of Experimental Psychology: Human Perception and Performance, 35, 958–976. doi:10.1037/a0014511
    • Ben-David, B. M., Multani, N., Durham, N. A.-M., Green, R., & van Lieshout, P. H. H. M. (2014, July). A new tool for assessing impairments of identification of emotions in spoken language following traumatic brain injury—the Test of Rating of Emotions in Speech (T-RES). Paper presented at the 2014 Mid-Year Meeting of the International Neuropsychological Society, Jerusalem, Israel.
    • Ben-David, B. M., Multani, N., Shakuf, V., & van Lieshout, P. H. H. M. (2013, June). Inter-generational communication breakdown: Age-related differences in processing of emotions in speech. Paper presented at the 20th World Congress of the International Association of Gerontology and Geriatrics, Seoul, South Korea.
    • Ben-David, B. M., Nguyen, L. L. T., & van Lieshout, P. H. H. M. (2011). Stroop effects in persons with traumatic brain injury: Selective attention, speed of processing, or color-naming? A meta-analysis.Journal of International Neuropsychological Society, 17, 354–363. doi:10.1017/S135561771000175X
    • Ben-David, B. M., & Schneider, B. A. (2009). A sensory origin for color-word Stroop effects in aging: A meta-analysis.Aging, Neuropsychology, and Cognition, 16, 505–534. doi:10.1080/13825580902855862
    • Ben-David, B. M., & Schneider, B. A. (2010). A sensory origin for color-word Stroop effects in aging: Simulating age-related changes in color-vision mimics age-related changes in Stroop.Aging, Neuropsychology, and Cognition, 17, 730–746. doi:10.1080/13825585.2010.510553
    • Ben-David, B. M., Tewari, A., Shakuf, V., & van Lieshout, P. H. H. M. (2014). Stroop effects in Alzheimer's disease: Selective attention, speed of processing, or color-naming? A meta-analysis.Journal of Alzheimer's Disease, 38, 923–938.
    • Ben-David, B. M., Thayapararajah, A., & van Lieshout, P. H. H. M. (2013). A resource of validated digital audio recordings to assess identification of emotion in spoken language after a brain injury.Brain Injury, 27, 248–250. doi:10.3109/02699052.2012.740648
    • Ben-David, B. M., Tse, V. Y. Y., & Schneider, B. A. (2012). Does it take older adults longer than younger adults to perceptually segregate a speech target from a background masker?.Hearing Research, 290, 55–63. doi: 10.1016/j.heares.2012.04.022
    • Ben-David, B. M., van Lieshout, P. H. H. M., & Leszcz, T. (2011). A resource of validated affective and neutral sentences to assess identification of emotion in spoken language after a brain injury.Brain Injury, 25, 206–220. doi:10.3109/02699052.2010.536197
    • Bertels, J., Kolinsky, R., Pietrons, E., & Morais, J. (2011). Long-lasting attentional influence of negative and taboo words in an auditory variant of the emotional Stroop task.Emotion, 11, 29–37. doi:10.1037/a0022017
    • Bowers, D., Bauer, R. M., & Heilman, K. M. (1993). The nonverbal affect lexicon: Theoretical perspectives from neuropsychological studies of affect perception.Neuropsychology, 7, 433–444.
    • Bowers, D., Blonder, L. X., & Heilman, K. M. (1999). Florida Affect Battery: A manual. Gainesville, FL: Center for Neuropsychological Studies.
    • Breitenstein, C., Daum, I., & Ackermann, H. (1998). Emotional processing following cortical and subcortical brain damage: Contribution of the fronto-striatal circuitry.Behavioural Neurology, 11(1), 29–42.
    • Brown, R. (1973). A first language: The early stages. Cambridge, MA: Harvard University Press.
    • Carton, J. S., Kessler, E. A., & Pape, C. L. (1999). Nonverbal decoding skills and relationship well-being in adults.Journal of Nonverbal Behavior, 23, 91–100.
    • Cicero, B. A., Borod, J. C., Santschi, C., Erhan, H. M., Obler, L. K., Agosti, R. M., … Grunwald, I. S. (1999). Emotional versus nonemotional lexical perception in patients with right and left brain damage.Neuropsychiatry, Neuropsychology, & Behavioral Neurology, 12, 255–264.
    • Craik, F. I. M. (2002). Levels of processing: Past, present … and future?.Memory, 10, 305–318. doi:10.1080/09658210244000135
    • Craik, F. I. M., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research.Journal of Verbal Learning and Verbal Behavior, 11, 671–684.
    • Dupuis, K., & Pichora-Fuller, M. K. (2010). Use of affective prosody by young and older adults.Psychology and Aging, 25, 16–29. doi:10.1037/a0018777
    • Egloff, B., & Schmukle, S. C. (2004). Gender differences in implicit and explicit anxiety measures.Personality and Individual Differences, 36, 1807–1815.
    • Ethofer, T., Anders, S., Erb, M., Herbert, C., Wiethoff, S., Kissler, J., … Wildgruber, D. (2006). Cerebral pathways in processing of affective prosody: A dynamic causal modeling study.NeuroImage, 30, 580–587. doi:10.1016/j.neuroimage.2005.09.059
    • Garner, W. R., & Felfoldy, G. L. (1970). Integrality of stimulus dimensions in various types of information processing.Cognitive Psychology, 1, 225–241.
    • Grant, D. A. (1948). The latin square principle in the design and analysis of psychological experiments.Psychological Bulletin, 45, 427–442.
    • Grimshaw, G. M., Séguin, J. A., & Godfrey, H. K. (2009). Once more with feeling: The effects of emotional prosody on hemispheric specialisation for linguistic processing.Journal of Neurolinguistics, 22, 313–326.
    • Henry, J. D., Phillips, L. H., Crawford, J. R., Theodorou, G., & Summers, F. (2006). Cognitive and psychosocial correlates of alexithymia following traumatic brain injury.Neuropsychologia, 44, 62–72.
    • Honkalampi, K., Hintikka, J., Tanskanen, A., Lehtonen, J., & Viinamäki, H. (2000). Depression is strongly associated with alexithymia in the general population.Journal of Psychosomatic Research, 48, 99–104.
    • Icht, M., & Ben-David, B. M. (2014). Oral-diadochokinesis rates across languages: English and Hebrew norms.Journal of Communication Disorders, 48, 27–37.
    • Icht, M., & Ben-David, B. M. (2015). Oral-diadochokinetic rates for Hebrew-speaking school-age children: Real words vs. non-words repetition.Clinical Linguistics & Phonetics, 29, 102–114.
    • Ishii, K., Reyes, J. A., & Kitayama, S. (2003). Spontaneous attention to word content versus emotional tone: Differences among three cultures.Psychological Science, 14, 39–46.
    • Jacob, H., Brück, C., Plewnia, C., & Wildgruber, D. (2014). Cerebral processing of prosodic emotional signals: Evaluation of a network model using rTMS.PloS ONE, 9(8), e105509.
    • Jones, D. (2010, June25). A WEIRD view of human nature skews psychologists' studies.Science, 328(5986), 1627.
    • Joukamaa, M., Saarijärvi, S., Muuriaisniemi, M.-L., & Salokangas, R. K. R. (1996). Alexithymia in a normal elderly population.Comprehensive Psychiatry, 37, 144–147.
    • Juslin, P. N., & Laukka, P. (2003). Communication of emotions in vocal expression and music performance: Different channels, same code?.Psychological Bulletin, 129, 770–814.
    • Kitayama, S. (1996). Remembrance of emotional speech: Improvement and impairment of incidental verbal memory by emotional voice.Journal of Experimental Social Psychology, 32, 289–308.
    • Kitayama, S., & Ishii, K. (2002). Word and voice: Spontaneous attention to emotional utterances in two languages.Cognition and Emotion, 16, 29–59.
    • Kotz, S. A., Dengler, R., & Wittfoth, M. (2015). Valence-specific conflict moderation in the dorso-medial PFC and the caudate head in emotional speech.Social Cognitive and Affective Neuroscience, 10(2), 165–171.
    • Kotz, S. A., & Paulmann, S. (2007). When emotional prosody and semantics dance cheek to cheek: ERP evidence.Brain Research, 1151, 107–118. doi:10.1016/j.brainres.2007.03.015
    • Laukka, P. (2003). Categorical perception of emotion in vocal expression.Annals of the New York Academy of Sciences, 1000, 283–287.
    • Lund, K., & Burgess, C. (1996). Producing high-dimensional semantic spaces from lexical co-occurrence.Behavior Research Methods, Instruments, & Computers, 28(2), 203–208.
    • Mehrabian, A., & Wiener, M. (1967). Decoding of inconsistent communications.Journal of Personality and Social Psychology, 6, 109–114.
    • Melara, R. D., & Algom, D. (2003). Driven by information: A tectonic theory of Stroop effects.Psychological Review, 110, 422–471.
    • Mitchell, R. L. C. (2006). How does the brain mediate interpretation of incongruent auditory emotions? The neural response to prosody in the presence of conflicting lexico-semantic cues.European Journal of Neuroscience, 24, 3611–3618. doi:10.1111/j.1460-9568.2006.05231.x
    • Mitchell, R. L. C., Elliott, R., Barry, M., Cruttenden, A., & Woodruff, P. W. R. (2003). The neural response to emotional prosody, as revealed by functional magnetic resonance imaging.Neuropsychologia, 41, 1410–1421.
    • Mitchell, R. L. C., & Ross, E. D. (2008). fMRI evidence for the effect of verbal complexity on lateralisation of the neural response associated with decoding prosodic emotion.Neuro-psychologia, 46, 2880–2887. doi:10.1016/j.neuropsychologia.2008.05.024
    • Morton, J. B., & Trehub, S. E. (2001). Children's understanding of emotion in speech.Child Development, 72, 834–843.
    • Nowicki, S., Jr., & Duke, M. P. (1994). Individual differences in the nonverbal communication of affect: The Diagnostic Analysis of Nonverbal Accuracy scale.Journal of Nonverbal Behavior, 18, 9–35.
    • Nygaard, L. C., & Lunders, E. R. (2002). Resolution of lexical ambiguity by emotional tone of voice.Memory & Cognition, 30, 583–593.
    • Nygaard, L. C., & Queen, J. S. (2008). Communicating emotion: Linking affective prosody and word meaning.Journal of Experimental Psychology: Human Perception and Performance, 34, 1017–1030. doi:10.1037/0096-1523.34.4.1017
    • Paulmann, S., Jessen, S., & Kotz, S. A. (2009). Investigating the multimodal nature of human communication: Insights from ERPs.Journal of Psychophysiology, 23, 63–76.
    • Paulmann, S., & Kotz, S. A. (2008). An ERP investigation on the temporal dynamics of emotional prosody and emotional semantics in pseudo- and lexical-sentence context.Brain and Language, 105, 59–69. doi:10.1016/j.bandl.2007.11.005
    • Paulmann, S., & Pell, M. D. (2011). Is there an advantage for recognizing multi-modal emotional stimuli?.Motivation and Emotion, 35, 192–201.
    • Paulmann, S., Pell, M. D., & Kotz, S. A. (2008). Functional contributions of the basal ganglia to emotional prosody: Evidence from ERPs.Brain Research, 1217, 171–178. doi:10.1016/j.brainres.2008.04.032
    • Pell, M. D., Jaywant, A., Monetta, L., & Kotz, S. A. (2011). Emotional speech processing: Disentangling the effects of prosody and semantic cues.Cognition & Emotion, 25, 834–853. doi:10.1080/02699931.2010.516915
    • Pell, M. D., Monetta, L., Paulmann, S., & Kotz, S. A. (2009). Recognizing emotions in a foreign language.Journal of Nonverbal Behavior, 33, 107–120.
    • Pell, M. D., Paulmann, S., Dara, C., Alasseri, A., & Kotz, S. A. (2009). Factors in the recognition of vocally expressed emotions: A comparison of four languages.Journal of Phonetics, 37, 417–435.
    • Raven, J. C. (1965). Guide to using the Colored Progressive Matrices Sets A, Ab, and B. Dumfries, Scotland: Griem.
    • Roche, J. M., Petters, B., & Dale, R. (2014). “Your tone says it all”: The processing and interpretation of affective language.Speech Communication, 66, 47–64.
    • Scherer, K. R. (2003). Vocal communication of emotion: A review of research paradigms.Speech Communication, 40, 227–256.
    • Scherer, K. R., Banse, R., & Wallbott, H. G. (2001). Emotion inferences from vocal expression correlate across languages and cultures.Journal of Cross-Cultural Psychology, 32, 76–92.
    • Schirmer, A., & Kotz, S. A. (2006). Beyond the right hemisphere: Brain mechanisms mediating vocal emotional processing.Trends in Cognitive Sciences, 10, 24–30.
    • Schröder, M. (2001). Emotional speech synthesis: a review.In P. Dalsgaard, B. Lindberg, H. Benner, & Z.-H. Tan (Eds.), Eurospeech 2001 Scandinavia: 7th European Conference on Speech Communication and Technology (pp. 561–564). Baixas, France: International Speech Communication Association.
    • Schwartz, R., & Pell, M. D. (2012). Emotional speech processing at the intersection of prosody and semantics.PLoS ONE, 7(10), e47279.
    • Shakuf, V., & Ben-David, B. M. (2014, February). Who is afraid of the bear? Hebrew test for recognition of emotions in spoken language for the diagnosis and evaluation of abilities and disabilities. Paper presented at the 50th Annual Meeting of the Israeli Speech, Hearing and Language Association, Tel Aviv, Israel.
    • Tanenhaus, M. K., Magnuson, J. S., Dahan, D., & Chambers, C. (2000). Eye movements and lexical access in spoken-language comprehension: Evaluating a linking hypothesis between fixations and linguistic processing.Journal of Psycholinguistic Research, 29, 557–580.
    • Vingerhoets, G., Berckmoes, C., & Stroobant, N. (2003). Cerebral hemodynamics during discrimination of prosodic and semantic emotion in speech studied by transcranial Doppler ultrasonography.Neuropsychology, 17, 93–99.
    • Wambacq, I. J. A., & Jerger, J. F. (2004). Processing of affective prosody and lexical-semantics in spoken utterances as differentiated by event-related potentials.Cognitive Brain Research, 20, 427–437. doi:10.1016/j.cogbrainres.2004.03.015
    • Wechsler, D. (2008). Wechsler Adult Intelligence Scale–Fourth Edition. San Antonio, TX: Pearson.
    • Wildgruber, D., Ackermann, H., Kreifelts, B., & Ethofer, T. (2006). Cerebral processing of linguistic and emotional prosody: fMRI studies.Progress in Brain Research, 156, 249–268. doi:10.1016/S0079-6123(06)56013-3
    • Wittfoth, M., Schröder, C., Schardt, D. M., Dengler, R., Heinze, H.-J., & Kotz, S. A. (2010). On emotional conflict: Interference resolution of happy and angry prosody reveals valence-specific effects.Cerebral Cortex, 20, 383–392. doi:10.1093/cercor/bhp106
    • Wurm, L. H., Vakoch, D. A., Strasser, M. R., Calin-Jageman, R., & Ross, S. E. (2001). Speech perception and vocal expression of emotion.Cognition and Emotion, 15, 831–852.
    • Zupan, B., Neumann, D., Babbage, D. R., & Willer, B. (2009). The importance of vocal affect to bimodal processing of emotion: Implications for individuals with traumatic brain injury.Journal of Communication Disorders, 42, 1–17.

    Additional Resources