In the first installment of our exploration into equity and access in translated texts, we delved into linguistic and cultural considerations for ensuring content is comprehensible, accessible, and inclusive for all. Now, in part two, we shift our focus to the research methodologies employed to evaluate the comprehensibility (including the evaluation of plain language, as described in part one) of any content, translated or not. Understanding how to effectively assess the accessibility of texts is paramount to ensuring that all readers have equitable access to information.

Let’s first review some common methods used to evaluate the comprehensibility and accessibility of texts, namely message testing and usability testing. Then we will explore the types of quantitative and qualitative data that can help us evaluate comprehensibility of texts/content.

Message Testing

Message testing involves gathering feedback from users regarding their experience with a message—i.e., a core piece of information, usually ranging from a few words to a paragraph or so, that a brand, organization, company, etc. wants to convey to its audience. Message testing focuses on aspects like clarity, ease of understanding, and overall ability to navigate and use the text as intended. By incorporating user feedback, organizations can iteratively improve the accessibility of their content. Some of the specific approaches to message testing include:

  • Comprehension Testing: Comprehension testing assesses how well individuals understand the content of a text. This can involve reading comprehension exercises or asking participants to summarize the main points of the text. Through comprehension testing, we can learn whether the language and structure of the text effectively convey the intended message.
  • Paraphrase Testing: Paraphrase testing involves asking participants to restate the content of a text in their own words. This method not only evaluates comprehension but also provides valuable information about the clarity and simplicity of the language used.
  • 5-Second Tests: In this rapid evaluation method, participants are shown a piece of material for only five seconds before being asked recall questions. This approach simulates real-world scenarios where individuals may only have a brief encounter with the text and provides insight into its immediate impact and memorability.
  • Think-Aloud Protocols: Think-aloud protocols require participants to verbalize their thoughts and reactions as they engage with texts. This method provides insight into participants’ cognitive processes, allowing researchers to identify points of confusion or difficulty in real-time.
  • A/B Testing or Split Testing:  A/B testing involves presenting two or more variations to users and comparing metrics such as click-through rates or completion rates, to determine which version is more effective in achieving its intended goals.

Usability Testing  

Usability testing is a cornerstone of user experience (UX) research, focusing on how effectively users can interact with a text or interface to accomplish specific tasks. While message testing is usually applied on a short piece of text or content, usability testing is typically applied to an interface that has multiple pathways through it, such as an app or website. Within testing an interface’s usability, we can focus on the accessibility of the text or content and how it affects the user’s experience with the interface. This method goes beyond simply assessing comprehension; it delves into the entire user journey, from initial engagement to task completion.

During usability testing, researchers observe participants as they engage with the text or interface, paying close attention to their interactions, navigation patterns, and decision-making processes. By closely observing user behavior, researchers can identify areas of friction, confusion, or inefficiency that may hinder comprehension and accessibility.

Moreover, usability testing incorporates gathering direct feedback from users. Participants are encouraged to articulate their thoughts, feelings, and frustrations as they navigate the content. This qualitative feedback provides invaluable insights into users’ perceptions, preferences, and pain points, shedding light on areas for improvement.

Usability testing can take various forms, including:

  • First Click Test for Websites: Specifically applicable to digital content, the first click test evaluates where users click first when presented with a webpage. This method helps identify whether the layout and design of the website effectively guide users to the information they seek.
  • Task-Based Testing: Participants are given specific tasks to complete using the text or interface. Researchers observe how users approach each task, where they encounter difficulties, and how they navigate through challenges. This method helps identify usability issues that may impede users’ ability to accomplish their goals effectively.
  • Prototype Testing: In the early stages of development, researchers may share prototypes or wireframes with prospective users to gather feedback and iterate on design concepts. This iterative approach allows for early identification and resolution of usability issues before the final product is launched.
  • Accessibility Testing: Usability testing can also encompass evaluating the accessibility of content for users with disabilities. Researchers assess factors such as screen reader compatibility, keyboard navigation, and color contrast to ensure that the content is inclusive and accessible to all users. There are many online resources providing guidelines for accessibility, such as the Web Content Accessibility Guidelines (WCAG) international standard (https://www.w3.org/WAI/standards-guidelines/wcag/ ).

Overall, usability testing plays a critical role in evaluating the effectiveness of texts and interfaces in facilitating user comprehension and accessibility. By combining observational insights with direct user feedback, researchers can identify and address usability issues, ultimately enhancing the user experience and promoting equitable access to information for all users.

Many of these message and usability testing approaches involve collecting new qualitative or quantitative data, or a combination of the two. In the following section, we break out the specific qualitative and quantitative data collection options most often used for evaluating the accessibility and comprehensibility of texts.

Quantitative Data Collection Methods

Quantitative research methods involve the collection and analysis of numerical data to quantify patterns and relationships. When evaluating texts, quantitative methods can provide valuable insights into the effectiveness of the language and its impact on comprehension. Some key quantitative data collection methods used to evaluate the accessibility of texts include:

  • Surveys and Questionnaires: Surveys and questionnaires can be used to gather feedback from a large number of participants regarding their perception of the clarity and comprehensibility of texts. Questions may assess factors such as readability, relevance, and ease of understanding.
  • Quantitative Content Analysis: Content analysis involves systematically coding and categorizing textual data to identify patterns and trends. Quantitative content analysis can be used to analyze the linguistic features of texts, such as word frequency, sentence length, and readability scores.
  • Eye-Tracking Studies: Eye-tracking studies measure where participants look and for how long, providing insights into their visual attention and reading behavior. By tracking eye movements while participants engage with texts, researchers can identify areas of the text that may cause confusion or difficulty.
  • Usage Analytics: For digital content, usage analytics can provide quantitative data on user interactions, such as page views, time spent on page, and click-through rates. By analyzing usage data, researchers can assess how users engage with content and identify areas for improvement.

Qualitative Data Collection Methods

Qualitative research methods involve the collection and analysis of non-numerical data to understand the underlying motivations, attitudes, and experiences of individuals. When evaluating texts for accessibility and comprehensibility, qualitative methods can provide rich insights into the nuances of language and cultural interpretation. Some key qualitative data collection methods used to evaluate accessibility include:

  • In-Depth Interviews: In-depth interviews allow researchers to explore participants’ perspectives in detail, uncovering their experiences with and interpretations of texts. Through open-ended questions and probing techniques, researchers can gain a deeper understanding of the factors that influence comprehension and accessibility.
  • Focus Groups: Focus groups bring together a small group of participants to discuss their perceptions of texts in a collaborative setting. By facilitating group discussion and interaction, researchers can uncover shared experiences and perspectives, as well as differences in interpretation and understanding.
  • Observational Studies: Observational studies involve observing participants’ behavior in naturalistic settings, such as reading texts in their everyday environment. For example, an airport might want research into how customers interact with new signage in a recently remodeled terminal. By observing how individuals interact with content in real-world contexts, researchers can gain insights into the practical challenges and barriers to comprehension.

In conclusion, evaluating the comprehensibility of texts requires a multifaceted approach that integrates both quantitative and qualitative research methods. By combining quantitative data collection methods such as surveys and content analysis with qualitative methods such as interviews and observational studies, researchers can gain a comprehensive understanding of the linguistic and cultural factors that influence accessibility. Through rigorous research and evaluation, we can strive towards creating content that is equitable, accessible, and inclusive for all. If you’d like to talk about content equity and accessibility in more depth, just send us a message.