Acoustics of Emotional Prosody Produced by Prelingually Deaf Children With Cochlear Implants

Monita Chatterjee, Aditya M Kulkarni, Rizwan M Siddiqui, Julie A Christensen, Mohsen Hozan, Jenni L Sis, Sara A Damm, Monita Chatterjee, Aditya M Kulkarni, Rizwan M Siddiqui, Julie A Christensen, Mohsen Hozan, Jenni L Sis, Sara A Damm

Abstract

Purpose: Cochlear implants (CIs) provide reasonable levels of speech recognition quietly, but voice pitch perception is severely impaired in CI users. The central question addressed here relates to how access to acoustic input pre-implantation influences vocal emotion production by individuals with CIs. The objective of this study was to compare acoustic characteristics of vocal emotions produced by prelingually deaf school-aged children with cochlear implants (CCIs) who were implanted at the age of 2 and had no usable hearing before implantation with those produced by children with normal hearing (CNH), adults with normal hearing (ANH), and postlingually deaf adults with cochlear implants (ACI) who developed with good access to acoustic information prior to losing their hearing and receiving a CI. Method: A set of 20 sentences without lexically based emotional information was recorded by 13 CCI, 9 CNH, 9 ANH, and 10 ACI, each with a happy emotion and a sad emotion, without training or guidance. The sentences were analyzed for primary acoustic characteristics of the productions. Results: Significant effects of Emotion were observed in all acoustic features analyzed (mean voice pitch, standard deviation of voice pitch, intensity, duration, and spectral centroid). ACI and ANH did not differ in any of the analyses. Of the four groups, CCI produced the smallest acoustic contrasts between the emotions in voice pitch and emotions in its standard deviation. Effects of developmental age (highly correlated with the duration of device experience) and age at implantation (moderately correlated with duration of device experience) were observed, and interactions with the children's sex were also observed. Conclusion: Although prelingually deaf CCI and postlingually deaf ACI are listening to similar degraded speech and show similar deficits in vocal emotion perception, these groups are distinct in their productions of contrastive vocal emotions. The results underscore the importance of access to acoustic hearing in early childhood for the production of speech prosody and also suggest the need for a greater role of speech therapy in this area.

Keywords: acoustics; children; cochlear implants; emotion; production; speech; vocal.

Copyright © 2019 Chatterjee, Kulkarni, Siddiqui, Christensen, Hozan, Sis and Damm.

Figures

Figure 1
Figure 1
Group differences in acoustic features of emotional productions. (Top to bottom – left panels) These figures show boxplots of mean F0 (Hz), F0 s.d. (Hz), Intensity (dB), Duration (s), and Spectral Centroid (Hz) values estimated for each sentence (abscissa) recorded by the participants in each emotion (happy: red; sad: blue). Data from the four groups of participants are represented in the four panels (left to write: ACI, ANH, CCI, and CNH). (Top to bottom – right panels) These figures show boxplots of the mean values of these acoustic features computed across the 20 sentences recorded in each emotion by individual participants. The abscissa shows the four groups (ACI, ANH, CCI, and CNH). Happy and sad emotions are again shown in red and blue colors.
Figure 2
Figure 2
Boxplots of acoustic contrasts between happy and sad emotions for mean F0 (upper) and F0 s.d. (lower) for each sentence (abscissa) and for the four groups (see legend).
Figure 3
Figure 3
(A–E) Values of acoustic features (A: mean F0; B: F0 s.d.; C: Intensity; D: Duration; E: Spectral Centroid) of the happy (red) and sad (blue) emotions recorded by CNH and CCI, plotted against their age (abscissa). For each acoustic feature, left- and right-hand panels show results in CCI and CNH, respectively, and upper and lower plots show results in female and male participants, respectively. The differently shaped symbols and lines in each color represent individual sentences recorded in each emotion.
Figure 4
Figure 4
(Top to bottom) Mean F0, F0 s.d., and Intensity of productions by CCI, plotted against their age at implantation. Left- and right-hand panels show results in female and male participants, respectively. Red and blue symbols represent happy and sad emotions, respectively, and the differently shaped symbols and the lines represent individual sentences recorded in each emotion.

References

    1. Banse R., Scherer K. R. (1996). Acoustic profiles in vocal emotion expression. J. Pers. Soc. Psychol. 70, 614–636. 10.1037/0022-3514.70.3.614, PMID:
    1. Bates D., Maechler M., Bolker B., Walker S. (2015). Fitting linear mixed-effects models using lme4. J. Stat. Softw. 67, 1–48. 10.18637/jss.v067.i01
    1. Boersma P. (2001). Praat, a system for doing phonetics by computer. Glot Int. 5, 341–345.
    1. Boersma P., Weenink D. (2019). Praat: doing phonetics by computer [Computer program]. Version 6.0.52 Available at: (Accessed May 2, 2019).
    1. Chatterjee M., Peng S. C. (2008). Processing F0 with cochlear implants: modulation frequency discrimination and speech intonation recognition. Hear. Res. 235, 143–156. 10.1016/j.heares.2007.11.004, PMID:
    1. Chatterjee M., Zion D. J., Deroche M. L., Burianek B. A., Limb C. J., Goren A. P., et al. . (2015). Voice emotion recognition by cochlear-implanted children and their normally-hearing peers. Hear. Res. 322, 151–162. 10.1016/j.heares.2014.10.003, PMID:
    1. Damm S. A., Sis J. L., Kulkarni A. M., Chatterjee M. (2019). How vocal emotions produced by children with cochlear implants are perceived by their hearing peers. J. Speech Lang. Hear. Res. (in press).
    1. Deroche M. L., Kulkarni A. M., Christensen J. A., Limb C. J., Chatterjee M. (2016). Deficits in the sensitivity to pitch sweeps by school-aged children wearing cochlear implants. Front. Neurosci. 10:73. 10.3389/fnins.2016.00073, PMID:
    1. Fox J., Weisberg S. (2011). An {R} companion to applied regression. 2nd Edn. Thousand Oaks CA: Sage; Available at: (Accessed July 1, 2019).
    1. Gilbers S., Fuller C., Gilbers D., Broersma M., Goudbeek M., Free R., et al. . (2015). Normal-hearing listeners’ and cochlear implant users’ perception of pitch cues in emotional speech. i-Perception 6:0301006615599139., PMID:
    1. Green T., Faulkner A., Rosen S., Macherey O. (2005). Enhancement of temporal periodicity cues in cochlear implants: effects on prosodic perception and vowel identification. J. Acoust. Soc. Am. 118, 375–385. 10.1121/1.1925827
    1. Hopyan-Misakyan T. M., Gordon K. A., Dennis M., Papsin B. C. (2009). Recognition of affective speech prosody and facial affect in deaf children with unilateral right cochlear implants. Child Neuropsychol. 15, 136–146. 10.1080/09297040802403682, PMID:
    1. Jiam N. T., Caldwell M., Deroche M. L., Chatterjee M., Limb C. J. (2017). Voice emotion perception and production in cochlear implant users. Hear. Res. 352, 30–39., PMID:
    1. Kuznetsova A., Brockhoff P. B., Christensen R. H. B. (2017). lmerTest package: tests in linear mixed effects models. J. Stat. Softw. 82, 1–26. 10.18637/jss.v082.i13
    1. Laan G. P. M. (1997). The contribution of intonation, segmental durations, and spectral features. Speech Comm. 22, 43–65. 10.1016/S0167-6393(97)00012-5
    1. Luo X., Fu Q.-J., Galvin J. J., 3rd. (2007). Vocal emotion recognition by normal-hearing listeners and cochlear implant users. Trends Amplif. 11, 301–315. 10.1177/1084713807305301
    1. Luo X., Kern A., Pulling K. R. (2018). Vocal emotion recognition performance predicts the quality of life in adult cochlear implant users. J. Acoust. Soc. Am. 144, EL429–EL435. 10.1121/1.5079575, PMID:
    1. Nakata T., Trehub S. E., Kanda Y. (2012). Effect of cochlear implants on children’s perception and production of speech prosody. J. Acoust. Soc. Am. 131, 1307–1314. 10.1121/1.3672697
    1. Nash J. C., Varadhan R. (2011). Unifying optimization algorithms to aid software system users: optimx for R. J. Stat. Softw. 43, 1–14. 10.18637/jss.v043.i09
    1. Paquette S., Ahmed G. D., Goffi-Gomez M. V., Hoshino A. C. H., Peretz I., Lehmann A. (2018). Musical and vocal emotion perception for cochlear implant users. Hear. Res. 370, 272–282. 10.1016/j.heares.2018.08.009, PMID:
    1. Peng S. C., Lu N., Chatterjee M. (2009). Effects of cooperating and conflicting cues on speech intonation recognition by cochlear implant users and normal hearing listeners. Audiol. Neurotol. 14, 327–337. 10.1159/000212112, PMID:
    1. Peng S.-C., Lu H.-P., Lu N., Lin Y.-S., Deroche M. L. D., Chatterjee M. (2017). Processing of acoustic cues in lexical-tone identification by pediatric cochlear-implant recipients. J. Speech Lang. Hear. Res. 60, 1223–1235. 10.1044/2016_JSLHR-S-16-0048, PMID:
    1. Peng S.-C., Tomblin J. B., Cheung H., Lin Y.-S., Wang L.-S. (2004). Perception and production of mandarin tones in prelingually deaf children with cochlear implants. Ear Hear. 25, 251–264. 10.1097/01.AUD.0000130797.73809.40, PMID:
    1. Peng S.-C., Tomblin J. B., Turner C. W. (2008). Production and perception of speech intonation in pediatric cochlear implant recipients and individuals with normal hearing. Ear Hear. 29, 336–351. 10.1097/AUD.0b013e318168d94d, PMID:
    1. Přibilová A., Přibil J. (2009). “Spectrum modification for emotional speech synthesis” in Multimodal signals: Cognitive and algorithmic issues. Berlin, Heidelberg: Springer, 232–241.
    1. R Core Team (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; Available at: (Accessed January 5, 2019).
    1. Scherer K. R. (2003). Vocal communication of emotion: a review of research paradigms. Speech Comm. 40, 227–256. 10.1016/S0167-6393(02)00084-5
    1. Schorr E. A., Roth F. P., Fox N. A. (2009). Quality of life for children with cochlear implants: perceived benefits and problems and the perception of single words and emotional sounds. J. Speech Lang. Hear. Res. 52, 141–152. 10.1044/1092-4388(2008/07-0213), PMID:
    1. See R. L., Driscoll V. D., Gfeller K., Kliethermes S., Oleson J. (2013). Speech intonation and melodic contour recognition in children with cochlear implants and with normal hearing. Otol. Neurotol. 34, 490–498. 10.1097/MAO.0b013e318287c985, PMID:
    1. Sueur J. (2018). Sound analysis and synthesis with R. Culemborg, Netherlands: Springer, Use R! Series.
    1. Sueur J., Aubin T., Simonis C. (2008). Seewave: a free modular tool for sound analysis and synthesis. Bioacoustics 18, 213–226. 10.1080/09524622.2008.9753600
    1. Tinnemore A. R., Zion D. J., Kulkarni A. M., Chatterjee M. (2018). Children’s recognition of emotional prosody in spectrally-degraded speech is predicted by their age and cognitive status. Ear Hear. 39, 874–880. 10.1097/AUD.0000000000000546, PMID:
    1. Van De Velde D. J., Schiller N. O., Levelt C. C., Van Hueven V. J., Beers M., Briaire J. J., et al. . (2019). Prosody perception and production by children with cochlear implants. J. Child Lang. 46, 111–141. 10.1017/S0305000918000387, PMID:
    1. Wickham H. (2016). ggplot2: Elegant graphics for data analysis. New York: Springer-Verlag.
    1. Winn M. B., Chatterjee M., Idsardi W. J. (2012). The use of acoustic cues for phonetic identification: effects of spectral degradation and electric hearing. J. Acoust. Soc. Am. 131, 1465–1479. 10.1121/1.3672705

Source: PubMed

スポンサーと協力者

医学的状態

薬物療法

3
購読する