No AccessJournal of Speech, Language, and Hearing ResearchResearch Article1 Aug 2014

Absolute and Relative Reliability of Percentage of Syllables Stuttered and Severity Rating Scales

    Purpose

    Percentage of syllables stuttered (%SS) and severity rating (SR) scales are measures in common use to quantify stuttering severity and its changes during basic and clinical research conditions. However, their reliability has not been assessed with indices measuring both relative and absolute reliability. This study was designed to provide such information. Relative reliability deals with the rank order of participants in a sample, whereas absolute reliability measures the closeness of scores to one other and to a hypothetical true score.

    Method

    Eighty-seven adult participants who stutter received a 10-min unscheduled telephone call. Three experienced judges measured %SS and also used a 9-point SR scale to measure stuttering severity from recordings of the telephone calls.

    Results

    Relative intrajudge and interjudge reliability were satisfactory for both scales. However, absolute intrajudge and interjudge reliability were not satisfactory. Results showed that paired-judge SR and %SS procedures improved absolute reliability compared with single-judge measures. Additionally, the paired-judge procedure improved relative reliability from high to very high levels.

    Conclusion

    Measurement of group changes of stuttering severity can be done in research contexts using either %SS or SR. However, for detecting changes within individuals using such measures, a paired-judge procedure is a more reliable method.

    References

    • Allen, C., Nikolopoulos, T. P., Dyar, D., & O'Donoghue, G. M. (2001). Reliability of a rating scale for measuring speech intelligibility after pediatric cochlear implantation.Otology & Neurotology, 22, 631–633.
    • Armson, J., & Kiefte, M. (2008). The effect of SpeechEasy on stuttering frequency, speech rate, and speech naturalness.Journal of Fluency Disorders, 33, 120–134.
    • Aron, M. L. (1967). The relationships between measurements of stuttering behavior.Journal of the South African Logopedic Society, 14, 15–34.
    • Atkinson, G. (2003). What is this thing called measurement error?.T. Reilly, & M. Marfell-Jones (Eds.), Kinathropometry VIII: Proceeding of the 8th International Conference of Kinathropometry (ISAK) (pp. 3–13). New York, NY: Routledge.
    • Atkinson, G., & Nevill, A. M. (1998). Statistical methods for assessing measurement error (reliability) in variables relevant to sport medicine.Sports Medicine, 26, 217–238.
    • Batterham, A. M., & George, K. P. (2003). Reliability in evidence-based clinical practice: A primer for allied health professionals.Physical Therapy in Sport, 4, 122–128.
    • Beal, D. S., Quraan, M. A., Cheyne, D. O., Taylor, M. J., Gracco, V. L., & De Nil, L. F. (2011). Speech-induced suppression of evoked auditory fields in children who stutter.NeuroImage, 54, 2994–3003.
    • Beilby, J. M., Byrnes, M. L., & Yaruss, J. S. (2012). Acceptance and commitment therapy for adults who stutter: Psychosocial adjustment and speech fluency.Journal of Fluency Disorders, 37, 289–299.
    • Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement.Lancet, 1, 307–310.
    • Bland, J. M., & Altman, D. G. (1995). Comparing methods of measurement: Why plotting difference against standard method is misleading.Lancet, 346, 1085–1087.
    • Bland, J. M., & Altman, D. G. (1999). Measuring agreement in method comparison studies.Statistical Methods in Medical Research, 8, 135–160.
    • Boberg, E., & Kully, D. (1994). Long-term results of an intensive treatment program for adults and adolescents who stutter.Journal of Speech and Hearing Research, 37, 1050–1059.
    • Bothe, A. K. (2008). Identification of children's stuttered and nonstuttered speech by highly experienced judges: Binary judgments and comparisons with disfluency-types definitions.Journal of Speech, Language, and Hearing Research, 51, 867–878.
    • Brundage, S. B., Bothe, A. K., Lengeling, A. N., & Evans, J. J. (2006). Comparing judgments of stuttering made by students, clinicians, and highly experienced judges.Journal of Fluency Disorders, 31, 271–283.
    • Bruton, A., Conway, J. H., & Holgate, S. T. (2000). Reliability: What is it and how is it measured?.Physiotherapy, 86, 94–99.
    • Carey, B., O'Brian, S., Onslow, M., Block, S., Packman, A., & Jones, M. (2010). Randomized controlled non-inferiority trial of a telehealth treatment for chronic stuttering: The Camperdown Program.International Journal of Language and Communication Disorders, 45, 108–120.
    • Chang, S. E., Horwitz, B., Ostuni, J., Reynolds, R., & Ludlow, C. L. (2011). Evidence of left inferior frontal-premotor structural and functional connectivity deficits in adults who stutter.Cerebral Cortex, 21, 2507–2518.
    • Choo, A. L., Chang, S. E., Zengin-Bolatkale, H., Ambrose, N. G., & Loucks, T. M. (2012). Corpus callosum morphology in children who stutter.Journal of Communication Disorders, 45, 279–289.
    • Cordes, A. K. (1994). The reliability of observational data: I. Theories and methods for speech-language pathology.Journal of Speech and Hearing Research, 37, 264–279.
    • Cordes, A. K., & Ingham, R. J. (1995). Judgments of stuttered and nonstuttered intervals by recognized authorities in stuttering research.Journal of Speech and Hearing Research, 38, 33–41.
    • Cordes, A. K., & Ingham, R. J. (1999). Effects of time-interval judgment training on real-time measurement of stuttering.Journal of Speech, Language, and Hearing Research, 42, 862–879.
    • Costello, J. M. (1975). Time-out procedures for the modification of stuttering: Three case studies.Journal of Speech and Hearing Disorders, 40, 216–231.
    • Cream, A., O'Brian, S., Jones, M., Block, S., Harrison, E., Lincoln, M., … Onslow, M. (2010). Randomized controlled trial of video self-modeling following speech restructuring treatment for stuttering.Journal of Speech, Language, and Hearing Research, 53, 887–897.
    • Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando, FL: Holt, Rinehart & Winston.
    • Cullinan, W. L., Prather, E. M., & Williams, D. E. (1963). Comparison of procedures for scaling severity of stuttering.Journal of Speech and Hearing Research, 6, 187–194.
    • Curran, M. F., & Hood, S. B. (1977). Listener ratings of severity for specific disfluency types in children.Journal of Fluency Disorders, 2, 87–98.
    • de Vet, H. C., Terwee, C. B., Knol, D. L., & Bouter, L. M. (2006). When to use agreement versus reliability measures.Journal of Clinical Epidemiology, 59, 1033–1039.
    • Domholdt, E. (2005). Rehabilitation research: Principles and applications (3rd ed.). Philadelphia, PA: Elsevier Saunders.
    • Donoghue, D., & Stokes, E. K. (2009). How much change is true change? The minimum detectable change of the Berg Balance Scale in elderly people.Journal of Rehabilitation Medicine, 41, 343–346.
    • Eve, C. L., Onslow, M., Andrews, C., & Adams, R. (1995). Clinical measurement of early stuttering severity: The reliability of a 10-point scale.Australian Journal of Human Communication Disorders, 23, 26–39.
    • Femrell, L., Åvall, M., & Lindström, E. (2012). Two-year follow-up of the Lidcombe Program in ten Swedish-speaking children.Folia Phoniatrica et Logopaedica, 64, 248–253.
    • Foundas, A. L., Bollich, A. M., Feldman, J., Corey, D. M., Hurley, M., Lemen, L. C., & Heilman, K. M. (2004). Aberrant auditory processing and atypical planum temporale in developmental stuttering.Neurology, 63, 1640–1646.
    • Foundas, A. L., Corey, D. M., Angeles, V., Bollich, A. M., Crabtree-Hartman, E., & Heilman, K. M. (2003). Atypical cerebral laterality in adults with persistent developmental stuttering.Neurology, 61, 1378–1385.
    • Gallop, R. F., & Runyan, C. M. (2012). Long-term effectiveness of the SpeechEasy fluency enhancement device.Journal of Fluency Disorders, 37, 334–343.
    • Guyatt, G., Walter, S., & Norman, G. (1987). Measuring change over time: Assessing usefulness of evaluative instruments.Journal of Chronic Diseases, 40, 171–178.
    • Harasym, J., & Langevin, M. (2012). Stuttering treatment for a school-age child with Down syndrome: A descriptive case report.Journal of Fluency Disorders, 37, 253–262.
    • Harris, V., Onslow, M., Packman, A., Harrison, E., & Menzies, R. (2002). An experimental investigation of the impact of the Lidcombe Program on early stuttering.Journal of Fluency Disorders, 27, 203–214.
    • Harrison, E., Onslow, M., Andrews, C., Packman, A., & Webber, M. (1998). Control of stuttering with prolonged speech: Development of a one-day instatement program.In A. Cordes, & R. J. Ingham (Eds.), Treatment efficacy in stuttering: A search for empirical bases (pp. 191–212). San Diego, CA: Singular.
    • Harrison, E., Onslow, M., & Menzies, R. (2004). Dismantling the Lidcombe Program of early stuttering intervention: Verbal contingencies for stuttering and clinical measurement.International Journal of Language & Communication Disorders, 39, 257–267.
    • Hewat, S., Onslow, M., Packman, A., & O'Brian, S. (2006). A Phase II clinical trial of self-imposed time-out treatment for stuttering in adults and adolescents.Disability and Rehabilitation, 28, 33–42.
    • Hopkins, W. G. (2000). Measures of reliability in sports medicine and science.Sports Medicine, 30, 1–15.
    • Ingham, R. J., & Cordes, A. (1992). Interclinic differences in stuttering-event counts.Journal of Fluency Disorders, 17, 171–176.
    • Ingham, R. J., Kilgo, M., Ingham, J. C., Moglia, R., Belknap, H., & Sanchez, T. (2001). Evaluation of a stuttering treatment based on reduction of short phonation intervals.Journal of Speech, Language, and Hearing Research, 44, 1229–1244.
    • Johnson, W. (1955). The time, the place, and the problem.In W. Johnson, & R. R. Leutenegger (Eds.), Stuttering in children and adults (pp. 3–24). Minneapolis: University of Minnesota Press.
    • Jones, M., Dobson, A., & O'Brian, S. (2011). A graphical method for assessing agreement with the mean between multiple observers using continuous measures.International Journal of Epidemiology, 40, 1308–1313.
    • Jones, M., Onslow, M., Packman, A., O'Brian, S., Hearne, A., Williams, S., … Schwarz, I. (2008). Extended follow-up of a randomized controlled trial of the Lidcombe Program of Early Stuttering Intervention.International Journal of Language & Communication Disorders, 43, 649–661.
    • Jones, M., Onslow, M., Packman, A., Williams, S., Ormond, T., Schwarz, I., & Gebski, V. (2005). Randomised controlled trial of the Lidcombe programme of early stuttering intervention.British Medical Journal, 331, 659–661.
    • Karimi, H., O'Brian, S., Onslow, M., Jones, M., Menzies, R., & Packman, A. (2013). Unscheduled telephone calls to measure percent syllables stuttered during clinical trials.Journal of Speech, Language, and Hearing Research, 56, 1455–1461.
    • Kreiman, J., & Gerratt, B. R. (2011). Comparing two methods for reducing variability in voice quality measurements.Journal of Speech, Language, and Hearing Research, 54, 803–812.
    • Kreiman, J., Gerratt, B. R., & Ito, M. (2007). When and why listeners disagree in voice quality assessment tasks.The Journal of the Acoustical Society of America, 122, 2354–2364.
    • Kreiman, J., Gerratt, B. R., Kempster, G. B., Erman, A., & Berke, G. S. (1993). Perceptual evaluation of voice quality: Review, tutorial, and a framework for future research.Journal of Speech and Hearing Research, 36, 21–40.
    • Kully, D., & Boberg, E. (1988). An investigation of interclinic agreement in the identification of fluent and stuttered syllables.Journal of Fluency Disorders, 13, 309–318.
    • Langevin, M., Huinck, W. J., Kully, D., Peters, H. F. M., Lomheim, H., & Tellers, M. (2006). A cross-cultural, long-term outcome evaluation of the ISTAR Comprehensive Stuttering Program across Dutch and Canadian adults who stutter.Journal of Fluency Disorders, 31, 229–256.
    • Langevin, M., Kully, D., Teshima, S., Hagler, P., & Narasimha Prasad, N. G. (2010). Five-year longitudinal treatment outcomes of the ISTAR Comprehensive Stuttering Program.Journal of Fluency Disorders, 35, 123–140.
    • Lewis, D., & Sherman, D. (1951). Measuring the severity of stuttering.Journal of Speech Disorders, 16, 320–326.
    • Lincoln, M., & Onslow, M. (1997). Long-term outcome of early intervention for stuttering.American Journal of Speech-Language Pathology, 6, 51–58.
    • MacDonald, J. D., & Martin, R. R. (1973). Stuttering and disfluency as two reliable and unambiguous response classes.Journal of Speech and Hearing Research, 16, 691–699.
    • Martin, R. R., Haroldson, S. K., & Triden, K. A. (1984). Stuttering and speech naturalness.Journal of Speech and Hearing Disorders, 49, 53–58.
    • Martin, R., Haroldson, S., & Woessner, G. (1988). Perceptual scaling of stuttering severity.Journal of Fluency Disorders, 13, 27–47.
    • Mulcahy, K., Hennessey, N., Beilby, J., & Byrnes, M. (2008). Social anxiety and the severity and topography of stuttering in adolescents.Journal of Fluency Disorders, 33, 306–319.
    • Naylor, R. V. (1953). A comparative study of methods of estimating the severity of stuttering.Journal of Speech and Hearing Disorders, 18, 30–37.
    • Neef, N. E., Jung, K., Rothkegel, H., Pollok, B., von Gudenberg, A. W., Paulus, W., & Sommer, M. (2011). Right-shift for non-speech motor processing in adults who stutter.Cortex, 47, 945–954.
    • Nelson, M. (1997). The validation of a dietary assessment.In B. M. Margetts, & M. Nelson (Eds.), Design concepts in nutritional epidemiology (pp. 242–272). Oxford, England: Oxford Medical Publications.
    • Nevill, A. N., & Atkinson, G. (1997). Assessing agreement between measurements recorded on a ratio scale in sports medicine and sports science.British Journal of Sports Medicine, 31, 314–318.
    • O'Brian, S., Jones, M., Pilowsky, R., Onslow, M., Packman, A., & Menzies, R. (2010). A new method to sample stuttering in preschool children.International Journal of Speech-Language Pathology, 12, 173–177.
    • O'Brian, S., Packman, A., & Onslow, M. (2004). Self-rating of stuttering severity as a clinical tool.American Journal of Speech-Language Pathology, 13, 219–226.
    • O'Brian, S., Packman, A., & Onslow, M. (2008). Telehealth delivery of the Camperdown Program for adults who stutter: A Phase I trial.Journal of Speech, Language, and Hearing Research, 51, 184–195.
    • O'Brian, S., Packman, A., Onslow, M., Cream, A., O'Brian, N., & Bastock, K. (2003). Is listener comfort a viable construct in stuttering research?.Journal of Speech, Language, and Hearing Research, 46, 503–509.
    • O'Brian, S., Packman, A., Onslow, M., & O'Brian, N. (2003). Generalizability theory II: Application to perceptual scaling of speech naturalness in adults who stutter.Journal of Speech, Language, and Hearing Research, 46, 718–723.
    • O'Brian, S., Packman, A., Onslow, M., & O'Brian, N. (2004). Measurement of stuttering in adults: Comparison of stuttering-rate and severity-scaling methods.Journal of Speech, Language, and Hearing Research, 47, 1081–1087.
    • O'Donoghue, P.2012). Statistics for sport and exercise studies. London, England: Routledge.
    • Olds, T. (2002). Five errors about error.Journal of Science and Medicine in Sport, 5, 336–340.
    • Onslow, M., Adams, R., & Ingham, R. (1992). Reliability of speech naturalness ratings of stuttered speech during treatment.Journal of Speech and Hearing Research, 35, 994–1001.
    • Onslow, M., Andrews, C., & Costa, L. (1990). Parental severity scaling of early stuttered speech: Four case studies.Australian Journal of Human Communication Disorders, 18, 47–61.
    • Onslow, M., Andrews, C., & Lincoln, M. (1994). A control/experimental trial of an operant treatment for early stuttering.Journal of Speech and Hearing Research, 37, 1244–1259.
    • Onslow, M., Costa, L., Andrews, C., Harrison, E., & Packman, A. (1996). Speech outcomes of a prolonged-speech treatment for stuttering.Journal of Speech and Hearing Research, 39, 734–749.
    • Onslow, M., Packman, A., Stocker, S., van Doorn, J., & Siegel, G. M. (1997). Control of children's stuttering with response-contingent time-out: Behavioral, perceptual and acoustic data.Journal of Speech, Language, and Hearing Research, 40, 121–133.
    • Packman, A., Onslow, M., & van Doorn, J. (1994). Prolonged speech and modification of stuttering: Perceptual, acoustic, and electroglottographic data.Journal of Speech and Hearing Research, 37, 724–737.
    • Rousey, C. L. (1958). Stuttering severity during prolonged spontaneous speech.Journal of Speech and Hearing Research, 1, 40–47.
    • Ryan, B. P., & Van Kirk Ryan, B. (1983). Programmed stuttering therapy for children: Comparison of four establishment programs.Journal of Fluency Disorders, 8, 291–322.
    • Ryan, B.P., & Van Kirk Ryan, B. (1995). Programmed stuttering treatment for children: Comparison of two establishment programs through transfer, maintenance, and follow-up.Journal of Speech and Hearing Research, 38, 61–75.
    • Saliva, J., Ysseldyke, J. E., & Bolt, S. (2010). Assessment in special and inclusive education (11th ed.). New York, NY: Houghton-Mifflin.
    • Schiavetti, N., Martin, R. R., Haroldson, S. K., & Metz, D. E. (1994). Psychophysical analysis of audiovisual judgments of speech naturalness of nonstutterers and stutterers.Journal of Speech and Hearing Research, 37, 46–52.
    • Sherman, D., & McDermott, R. (1958). Individual ratings of severity of moments of stuttering.Journal of Speech and Hearing Research, 1, 61–67.
    • Shrivastav, R., Sapienza, C. M., & Nandur, V. (2005). Application of psychometric theory to the measurement of voice quality using rating scales.Journal of Speech, Language, and Hearing Research, 48, 323–335.
    • Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability.Psychological Bulletin, 86, 420–428.
    • Sim, J., & Wright, C. C. (2005). The kappa statistic in reliability studies: Use, interpretation, and sample size requirements.Physical Therapy, 85, 257–268.
    • Stuart, A., Kalinowski, J., Saltuklaroglu, T., & Guntupalli, V. K. (2006). Investigations of the impact of altered auditory feedback in-the-ear devices on the speech of people who stutter: One-year follow-up.Disability and Rehabilitation, 28, 757–765.
    • Tasko, S. M., McClean, M. D., & Runyan, C. M. (2007). Speech motor correlates of treatment-related changes in stuttering severity and speech naturalness.Journal of Communication Disorders, 40, 42–65.
    • Teshima, S., Langevin, M., Hagler, P., & Kully, D. (2010). Post-treatment speech naturalness of comprehensive stuttering program clients and differences in ratings among listener groups.Journal of Fluency Disorders, 35, 44–58.
    • Tinsley, H. E., & Weiss, D. J. (1975). Interrater reliability and agreement of subjective judgments.Journal of Counseling Psychology, 22, 358–374.
    • Trajkowski, N., Andrews, C., Onslow, M., O'Brian, S., Packman, A., & Menzies, R. (2011). A Phase II trial of the Westmead Program: Syllable-timed speech treatment for preschool children who stutter.International Journal of Speech-Language Pathology, 13, 500–509.
    • Unger, J. P., Glück, C. W., & Cholewa, J. (2012). The immediate effects of AAF devices on the characteristics of stuttering: A clinical analysis.Journal of Fluency Disorders, 37, 122–134.
    • Van Borsel, J., & Eeckhout, H. (2008). The speech naturalness of people who stutter speaking under delayed auditory feedback as perceived by different groups of listeners.Journal of Fluency Disorder, 33, 241–251.
    • Weir, J. P. (2005). Quantifying test–retest reliability using the intraclass correlation coefficient and the SEM.Journal of Strength and Conditioning Research, 19, 231–240.
    • Williams, D. E., & Kent, L. R. (1958). Listener evaluations of speech interruptions.Journal of Speech and Hearing Research, 1, 124–131.
    • Young, M. A. (1969a). Observer agreement: Cumulative effects of rating many samples.Journal of Speech and Hearing Research, 12, 135–143.
    • Young, M. A. (1969b). Observer agreement: Cumulative effects of repeated ratings of the same samples and of knowledge of group results.Journal of Speech and Hearing Research, 12, 144–155.
    • Young, M. A. (1970). Anchoring and sequence effects for the category scaling of stuttering severity.Journal of Speech and Hearing Research, 13, 360–368.

    Additional Resources