Patrick Madden

>One of the first questions that usual comes up when you are planning to conduct a survey is, how many surveys do I need to do to get statistically valid results? If you’ll allow me to channel Bill Clinton for a second, the answer really depends on what your definition of ‘statistically valid’ is. While there is no universal rule for what constitutes statistical validity or how many surveys you need to do to get ‘valid results,’ typically survey researchers will recommend a baseline of 400 completed interviews. Why 400? Well, it’s a nice round number that will provide an overall margin of error of +/-5% at 95% confidence*. People like round numbers.

In reality, the number of people that you should survey depends on a number of factors related to your target population and the way you’d like to analyze your survey results. Here are the questions you need to answer before deciding how big of a sample you need for your survey.

1. What is the size of the target population: Who is the group of people that you are surveying and how large is this group (e.g., customers, your employees, men age 18-24 still living at home with their parents)? The fewer people in your group (i.e., your target population), the fewer surveys you’ll usually need to do. Why? If you survey 5% or more of the population, then you will need to adjust your sampling error by something called a finite population correction factor.

The math behind this adjustment isn’t important, but the table below shows how many surveys you’d have to do to achieve a margin of error of +/-5% at 95% confidence at different population sizes:

Sample size needed to achieve a margin of error the same as 400 surveys at different population sizes

2. Will you be segmenting your results: 400 completed surveys will give you a margin or error of +/-5% OVERALL, but what if you want to break out your analysis by demographic factors, such as region, age, gender or income? If the results of your survey rely heavily on specific group segments, then it is a good idea to make sure you do enough surveys within each segment of the population to get generalizable results by those groups.**

Wait, how many surveys do you have to do to generalizable results by segment? Well, it’s a bit like the chicken and the egg situation; it depends on what you will be using the data for. A rule of thumb researchers usually use is to conduct at least 100 surveys per segment. Again, 100 surveys is a nice round number which produces a nice round margin of error of +/-10% at 95% confidence. That, and people tend to freak out when you tell them they need to complete 400 surveys per segment.

3. What is the pain to you or your organization if the survey results are incorrect: Let’s be honest, not all decisions carry the same importance (and yes there is such a thing as a stupid question). So as the financial, strategic, or social importance of your decision rises, so does the need to ensure that your survey results are accurate.

The go-to gold standard margin of error of +/-5% at 95% confidence means that if 50% of respondents indicated something in a survey, then 19 out of 20 times (your 95% confidence level), between 45% and 55% of the overall population would provide the same response.

Unfortunately, there are still a lot of things that can go wrong even if the survey data is collected correctly. If the difference between 45% of your customers saying something and 55% saying something is a make or break deal for your organization, then you need to do more surveys and shoot for a smaller margin of error. If the chance of getting an erroneous survey measurement 1 time out of 20 is still too risky for the type of decision you will be making based on the data, then maybe you should up your confidence level to 99%.

The reverse is true as well. If you are going to be measuring the same questions multiple times in different surveys, then you can do fewer surveys and still protect yourself from the one bad survey result. Sometimes you are simply looking to see if customers have any interest in a product, service or concept. In that case, doing 100 surveys would like be enough as it would provide a margin or error of +/-10% at 95% confidence.

The following table displays how many surveys need to be done to achieve a specific margin of error at different confidence levels:

Sample Size needed to achieve a specific margin of error at different confidence levels

If the table is not specific enough for your needs, you can always calculate the number of surveys you need to do yourself using this general formula in Excel:

“=(CI*CI)*(p*(1-p))/(MOE*MOE)”

where

CI = required confidence level (where the standard score of 1.96 is used for a 95% confidence interval)
p = estimated proportion of response (use .5 for the most conservative result)
m = margin of error (5%, or 0.05, for example)

Ok, maybe it’s not super easy. If math isn’t your thing, then you can use an online sample size calculator to identify what sample size you need based on the population size, margin of error and confidence level. Just make sure to think about the answer those questions listed above first.

—

* Technically, 384 surveys produces an overall margin of error of almost exactly +/-5.0%. However, as I mentioned, researchers like nice round numbers and will therefore usually round that number up to 400.

** The need to segment or split out your survey results inevitability brings us down the road to quotas and stratified sampling, but that is a conversation for another day.