Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information

Abstract

Objective: To develop a reliable and valid instrument to assess the understandability and actionability of print and audiovisual materials.

Methods: We compiled items from existing instruments/guides that the expert panel assessed for face/content validity. We completed four rounds of reliability testing, and produced evidence of construct validity with consumers and readability assessments.

Results: The experts deemed the PEMAT items face/content valid. Four rounds of reliability testing and refinement were conducted using raters untrained on the PEMAT. Agreement improved across rounds. The final PEMAT showed moderate agreement per Kappa (Average K=0.57) and strong agreement per Gwet's AC1 (Average=0.74). Internal consistency was strong (α=0.71; Average Item-Total Correlation=0.62). For construct validation with consumers (n=47), we found significant differences between actionable and poorly-actionable materials in comprehension scores (76% vs. 63%, p<0.05) and ratings (8.9 vs. 7.7, p<0.05). For understandability, there was a significant difference for only one of two topics on consumer numeric scores. For actionability, there were significant positive correlations between PEMAT scores and consumer-testing results, but no relationship for understandability. There were, however, strong, negative correlations between grade-level and both consumer-testing results and PEMAT scores.

Conclusions: The PEMAT demonstrated strong internal consistency, reliability, and evidence of construct validity.

Practice implications: The PEMAT can help professionals judge the quality of materials (available at: http://www.ahrq.gov/pemat).

Keywords: Assessment; Audiovisual materials; Clear communication; Educational materials; Health literacy; Instrument development; Measurement; Patient education; Plain language; Readability.

Conflict of interest statement

All authors state that there are no other actual or potential conflicts of interest including any financial, personal or other relationships with other people or organizations within 3 years of beginning the submitted work that could inappropriately influence, or be perceived to influence, their work.

Copyright © 2014 Elsevier Ireland Ltd. All rights reserved.