Testing for UX Writers: Know When Your Words Are Working

Posted 2 months ago by Annie Adams

Welcome to the latest instalment of my series on writing as user experience design. I’m discussing key principles of user-centered design with the aim of applying these ideas to writers’ contributions UX and UI.

Let’s talk about testing. As UX writers, we’re well aware that we work in a space obsessed with evaluating usability. And we love it! We’re in this for the users, after all. We try to understand their every value, want, and need so we can craft perfect pieces of UX copy and microcopy.

Yet our niche of the UX world is still new. While UX designers and researchers have a marvellous array of methods at their disposal, I’m often left wondering how UX writers can test our words. How do we narrow in on our part of the design process? What testing methods are available to us right now?

I don’t just mean readability scores. Flesch-Kincaid Index and similar metrics of readability can be a great baseline for user-friendly text, but they aren’t perfect. Nor can the algorithms powering these scores tell if your words are striking the right tone. As anyone writing in a particular brand voice can attest, every word needs to reflect the persona encoded by the internal style guide. Flesch-Kincaid can’t help you with that.

(I promise I’m not trying to be too harsh against readability scores. I use Hemingway all the time to keep my writing concise!)

Turns out, we aren’t starting from scratch when it comes to testing content and microcopy. Here are a few methods from around the web to help you start building testing into your process!

A note on participants: When testing comprehension, it’s important to recruit participants representative of your actual users. Proxy users aren’t enough. Running cloze tests on stakeholders from the marketing department or surveying engineers walking down the hallway won’t work here. These insiders bring company-specific assumptions and knowledge and can’t provide an accurate measure of comprehensibility.

Cloze Testing

Source: stock.tookapic.com via pixabay.com

Evaluate users’ comprehension with a cloze test, sometimes called a cloze deletion test. Based on Gestalt theory of mental self-organization (“cloze” is related to “closure,” or the mechanism by which individuals understand an incomplete object as complete), this method was first proposed by W. L. Taylor in 1974.

Cloze tests are great for figuring out if your intended audience actually understands what you’ve written.

How to run a cloze test:

  1. Select a piece of text and replace every Nth word with blanks. A higher N value makes the test easier. In typical tests, N = 6. Of course, UX writers working with microcopy will need to have a much smaller N!
  2. Ask each participant to read the altered text and fill in the blanks with their best guesses about the missing words.
  3. Calculate the score: the percentage of correctly guessed words. Synonyms and misspellings are allowed (in fact, if a synonym recurs, maybe that’s the word you’ve been looking for). If your participants score 60% or higher on average, your text is reasonably comprehensible for this user profile.

Highlighter testing

Source: pixabay.com via pexels.com

How to run highlighter testing

  1. Select a piece of text and determine what you’re testing for. Code a set of highlighters according to these factors. In the linked example, participants highlighted words that made them feel more confident about the service in green and words that made them feel less confident about the service in red.
  2. Provide participants with a printout of the text, the coded highlighters, and clear instructions about marking the page. If you’re testing for more than one value pair, avoid confusion by providing the user with a clean printout and new instructions for each pair.
  3. Calculate the results. The colors provide an easily assessed visual indicator of what text works and what might need rewriting. If you’re a metrics person, you can also calculate statistics based on counts of color-coded lines.

Comprehension survey

Photo by pulkit jain on Unsplash

Think back to your high school English class. Do you remember having weekly quizzes to make sure you were reading and understanding the assigned books? Your teacher was on to something.

Comprehension surveys are basically quizzes that test whether a user understands a piece of content. You can test long pieces, short strings, and everything in between.

How to run a comprehension survey

  1. Determine the text you’d like to test. You can present it all at once, line-by-line, or a mix of both.
  2. Develop questions. A List Apart has great tips for writing high quality questions, including stating questions in positive form, providing only one correct or clearly best answer in a multiple choice list, and providing incorrect but plausible distractor answers. You might want to add an option for “I don’t know,” as random guessing isn’t helpful here.
  3. Provide your participant with the content and questionnaire. This isn’t the SAT, so keep things short — twenty minutes is more than enough time to gauge your user’s comprehension.
  4. Calculate the scores. If you use a survey platform like SurveyMonkey or through a UX testing tool such as Optimal Workshop, you’ll get a handy score report automatically!

I hope these testing methods are useful for you. If you’re a UX writer or content writer and have other ideas about measuring the effectiveness of your words, let the world know in the comments!

This article was originally published on Annie’s Medium page.

Get started with Marvel Enterprise

Get started with Marvel Enterprise

From CEOs to marketing, get your entire organisation collaborating in one place.

Get started with Enterprise

UX Writer for Gap Inc (Gap, Banana Republic, Athleta, Old Navy, Intermix). Visit me at https://www.annietadams.com.

Related Posts

There are lots of great tools for drawing wireframes today. However, I still prefer my good ol’ paper wireframing kit. In this post I’ll tell you why and explain how paper wireframing will make you a better designer. In 2012 I was planning on taking my wireframing skills to the next level. I had gotten the excellent app Paper by… Read More →

Your portfolio is a story about you and your work — your expression of growth through all the obstacles, surprises, successes and failures. This is your epic. Illuminate the arc of your career and why you matter. For the audience, your portfolio showcases your problem-solving superpowers, hard and soft skills, and your ability to reflect upon your successes and failures…. Read More →

Animation is a big part of user experience. When it comes to transitions in mobile apps, there are a lot of things you can communicate very subtly with animation. Send the message, open Settings, check the box, navigate to another page — these are all moments of change. Animating a transition is an excellent way to reinforce the user’s action…. Read More →