Patrick Madden

>Developing questions is the aspect of research that non-researchers normally feel the most comfortable with. However, it is something that is deceptively difficult to do well and is the place where things are most likely to go wrong in the survey process. People make entire careers out of the study of questionnaire development and there are many books and scholarly articles devoted to the topic. I am clearly not one of these scholars, but I’ve at least seen enough to know all the things I don’t know about developing questions for surveys.

So, what don’t I know?

The goal of a survey question is to gather reliable and consistent information on something you want to measure, such as the number of times people have been to the doctor in the past year. Sounds easy enough. But how people define the word ‘doctor’ is very different, especially as you cross cultures. Are we talking about primary care physicians only? Does a trip to the emergency room count? How about a visit to a public clinic? And what do we mean by year? Do we mean the previous 12 months from this very moment, or the last calendar year of 2009? Subtle differences, but to the extent that survey questions can be interpreted differently by respondents, it means that our question is vague and will produce inconsistent data.

Types of Questions

There are two basic types of questions that make up the average survey: closed-ended and open-end questions.

Closed-ended questions are those that require yes/no or a multiple choice answer, such as ‘do you love puppies?’ Open-ended questions cannot be answered in a simple yes/no response and require a longer and more varied response. Something like: ‘Please tell me the reasons why you hate puppies?’

Generally speaking, closed-ended questions are simple to ask, analyze and interpret. However, they do require you to phrase questions into something that can be answered using multiple choice. That is not always possible or practical. Open-ended questions can capture more detailed information and are useful when you don’t know what types of answers the respondents are going to provide, but they do require more coding and analysis time after the survey is completed.

Biased or Leading Questions

Ok, that question about puppies above is actually a bad survey question, because it is called a leading question. By using the word ‘love’ and phrasing the question in that way, I am pushing respondents to answer a certain way (in this example, to say they love of puppies). If they answer ‘no,’ then they look like a bad person.

Leading questions and biased surveys are something that I commonly run into, especially with the market research crowd (I rarely do political polling since it makes me feel dirty). These surveys seem to be designed to confirm a point of view. Political push polls are the most common example of these types of surveys, but there are other subtle ways that questions can lead respondents – such as loaded questions, biased phrasing of question wording or unbalanced response scales.

The unbalanced scale:

Please tell me your opinion of puppies, using the following scale of 1-5:

1. I love them more than life itself
2. I love them a lot
3. I love them
4. I like them

The results of my survey show that 100% of people like puppies.

Additionally, the order that you put your questions in can affect how respondents answer them, that is, one question can bias a later question. If you ask a respondent to rate a series of factors on importance, and then the respondent is asked an open-ended question on what is important, their comments will largely be limited to the factors already mentioned. Ask the open-ended question first, and a greater diversity of factors are likely to be mentioned.

Check out StatPac for a good list of more subtle ways to bias survey questions.

Ask Actionable Questions

That question about puppies above is bad for another reason too: It’s not going to give me any information that is actionable. So what if everyone loves puppies? What can I do with that data?

A mistake that people often make is asking questions simply to ask them because they think they should or because they have “space” in their survey. Make sure that every question in a survey serves a purpose and that you will use the information that comes out of it. The best way to do this is to clearly define the goals of your survey. What are your hypotheses? What are the things you want your survey to answer? Make survey every survey question builds towards those goals.

The Lowest Common Denominator

Maybe that sounds a little mean, but it is important to assure that your survey questions can be understood all respondents, not just those with advanced degrees. At minimum, questions need to be put in plain English (think 6th-8th grade reading level), and organizational jargon needs to be stripped away. If you conduct research on short term disability policies, your internal definition of STD is probably a little different than the average person. It’s usually a good idea to let someone outside your industry or area of expertise read your survey to make sure it can be understood by the average person.

For very sophisticated research, cognitive testing can be done to be sure that each question is understood by the respondent as intended. Most studies don’t have a budget to allow this, so you’ll need to rely on past research about what respondents have found confusing in the past. And if there is no past research, well, my best advice is to keep it simple.

Response Scales

There is the issue of what are the appropriate question response scales to use (another topic people have written entire books about). Scales can be either even or odd. Even scales allow no middle choice. Research suggests that an uneven scale is better because it allows a respondent to indicate that they are in the middle and could go either way.

Scales can also be anchored or unanchored. An anchored scale has words attached to it, such as “very satisfied” or “somewhat satisfied”, and they are preferred because they add clarity on the meaning of the divisions on scales. Alternatively, an unanchored scale might be worded, “on a scale of 1-10 where 10 is very satisfied…” More divisions allow finer distinctions in responses but they also make wording impractical.

Should you label every point in your scale? Generally, yes if possible, as labeling every point increases understanding of your scale and leads to more consistent responses.

How many distinct points should you have in your scale? Five will usually do for most questions, but if you believe that your data is going to be skewed either negatively or positively, then a seven or ten point scale will produce a wider range of answers.

The Final Word

Now that you’ve crafted all your survey questions and given them appropriate response scales, you get to worry about the order to put them in. The order of questions in a survey can be important. If you ask a question early in a survey, the responses will be different than the same question asked later, after a respondent had time to think about a topic. Later responses are more likely to provide more depth than earlier ones

Thinking about the research objectives and the specific informational needs, an experienced researcher will craft questions in different ways to completely surround a topic. For example, you might ask the respondent to name something top of mind, and then ask about familiarity with similar items, and then ask for rankings of these items on particular characteristics. You might also include an open-ended question on why one something is different than another.

***

So survey development is easy, right? For more information on designing questions for surveys, I’d reccommend the following books:

http://rcm.amazon.com/e/cm?t=patrmadd0a-20&o=1&p=8&l=bpl&asins=0787970883&fc1=000000&IS2=1&lt1=_blank&m=amazon&lc1=0000FF&bc1=000000&bg1=FFFFFF&f=ifrhttp://rcm.amazon.com/e/cm?t=patrmadd0a-20&o=1&p=8&l=bpl&asins=0749450282&fc1=000000&IS2=1&lt1=_blank&m=amazon&lc1=0000FF&bc1=000000&bg1=FFFFFF&f=ifrhttp://rcm.amazon.com/e/cm?t=patrmadd0a-20&o=1&p=8&l=bpl&asins=078797546X&fc1=000000&IS2=1&lt1=_blank&m=amazon&lc1=0000FF&bc1=000000&bg1=FFFFFF&f=ifr