You don’t need to be a statistician to get usable and insightful data from your participants
A perfect survey can be time-consuming, often taking hours or even days to devise. Odds are, you took a survey (or many surveys) and wondered, “Just how do they get it?” Well, worry not, for here are a few tips to help you craft that survey like a pro!
Take some time and find the question you want answered
A survey is an effective tool that provides some insight into what the respondents think and feel, and how they act, and helps better understand how the wider population or demographic thinks, feels, and behaves. The key to using this tool though, is first narrowing down on the specific question you want answered.
Here’s an example to illustrate this. Suppose you’re a part of a large multinational corporation XYZ Corp, which sells breakfast cereal in many countries. They want to set up shop in India, and they task you with the important job of doing all the preliminary market research. The easiest and most effective way to go about this would be through a survey and, long story short, you would want the question of “what does the Indian consumer eat in the morning?” answered, among a plethora of other questions. This one question would serve as your springboard, helping you draft and structure your questionnaire, such that the data from your survey answers (or comes close to) the very first question.
Make sure the answer to your question can be found in a relevant, specific, and quantifiable way. And remember, it is important to not rush through this stage, as it would end up having an impact on your survey every step of the way, so take it slow and take your time!
Narrow down on your target demographic
It isn’t always helpful (or useful) to cast a wide net and fish out data from across many boards (age groups, urban/rural demographics, occupations etc.). More data does not mean more useful data.
Circling back to the example before, you wouldn’t want to ask men and women over and above their 60s about their thoughts on a breakfast cereal for kids. It would make a lot more sense instead to focus on what kids and their parents think of, as the latter buys the breakfast cereal, and the former eats it.
It is important to keep in mind that this is contingent on what your product is, and who it caters to. A more general, broad-base use product would require surveying a wider demographic, while a specific use-case product would only require surveying its target audience.
Approach the questionnaire as if you were to answer them
Studies by survey experts from Stanford have shown that answering survey questionnaires is a 5-step process. Respondents read the question, figure out what it is trying to ask for, search their memory for something relevant to this case, integrate the various instances in their memory to form a single judgement, and finally translate the judgment to a choice in the questionnaire. While drafting questions, it would be helpful to keep this process in mind, not only to make the respondent’s task easier, but to also obtain more accurate data and subsequent results.
Research has also shown that there are broadly two types of respondents.
optimizers and satisficers
Optimizers are those people who deeply engage with the survey, closely looking at every question and option, and only then choosing what they fall under. Satisficers on the other hand do not engage as obsessively as their counterparts, and will choose the first option they see that they fall under. They often respond with neutral choices, or do not closely read the questions, and are seen to populate the dataset with unusable data.
Keeping this in mind, surveys should be prepared such that it is able to accurately gauge and engage the satisficers. Surveys have to be made as accessible as possible, using simple words and consistent syntax, and choices should be mutually exclusive (no overlap between the choices). The use of double negatives and double-barreled questions (questions that ask two things at once) are to be avoided.
Every question should add value to your survey
Each question in a survey questionnaire should build towards your goal, and help you answer the fundamental question you set out to answer with the survey. When data from the responses to two questions give you similar insights, it would have been better to do away with one of them or phrase the other one better.
If it is of no use knowing the precise location of your respondent, or their exact age in years, don’t ask them said questions. Stick to framing questions that produce usable data and help generate meaningful results.
Stop stopping at yes/no binaries
Often, most questions can’t be answered with a simple ‘yes’ or ‘no’. Now, while it would be unwise (and inadvisable) to allow participants to author lengthy essays, it is equally unwise to constrain them to mere yes or no, especially in questions where personal preference plays a pivotal role.
Now, XYZ Corp (our example from before) wants you to ascertain what breakfast cereal adult millennials prefer to eat. A roundabout way to go about it would be to make a lengthy list of breakfast cereals in the market, and frame them as “yes or no” questions for the respondent to answer. This would make the survey needlessly long, and the questions would progressively diminish in value added, in terms of usable insight gained (as mentioned in the previous point). A smarter way to approach this would be by narrowing down on 5 of the top-selling cereals and making the respondent choose one.
Solutions to problems are seldom black or white; use Multiple-choice questions when choices are involved, giving you far more useful insights.
No one likes vague and open-ended questions
Open-ended questions often force the respondent to really think about their answers, which may leave them perplexed and frustrated for having to form a coherent response. They would also inevitably lead to data that is very varied and may hamper on its usefulness.
Similarly, giving the respondent a lengthy list of extents to which they agree or disagree with something would leave them cognitively burdened (a smarter way of saying really stressed out). Taking our example of XYZ Corp, imagine one of the questions you would have to ask was “how much do you like sugary cereal?” and you give them the following options:
- Strongly disagree
- Moderately disagree
- Slightly disagree
- Neutral
- Slightly agree
- Moderately agree
- Strongly agree
This exhausting array of options would leave the respondent scratching their heads trying to figure out the difference between a “Moderately disagree” and a “Slightly disagree.” Instead, the following would be easier for the respondent to choose:
- Extremely grateful
- Very grateful
- Moderately grateful
- Slightly grateful
- Not at all grateful
Remember, we want people to engage with the questionnaire, not scratch their heads trying to figure out what to say or choose or thrown aback by unclear language. It is important to be succinct, and refrain from asking abstract questions.
Avoid using unbalanced scales
Giving the respondent three ‘bad’ options and one neutral or ‘good’ option. These would tilt them towards choosing one of the ‘bad’ options, given that they outweigh the number of neutral and ‘good’ ones.
While it may make sense to often add a “neutral” midpoint, some studies have shown that people who rush through surveys without much thought (adding nothing useful in terms of insights) choose the “neutral” response, which would skew the data and impact the analysis.
Sandwich the hard questions between the easier ones
People hate to have to think hard from the get-go. Asking the ‘meat’ of your survey in the very beginning is sure to drive them away. Asking it to them at the very end too is sure to give inaccurate responses, as the respondents are sure to be fatigued by the many questions before it. The best way to go about then, would be to push the harder questions in between the easier to answer and the more straightforward ones, as they would be deeply engaged with the survey, as compared to when they are just about to start filling out the questionnaire, or about to finish it.
Another way to approach the harder, more thought-provoking questions is to ensure that they are easy to understand and leave no room for interpretation. A good question should also avoid the use of controversial language, as it is sure to drive engagement down and would consequently lead to inaccurate results.
Ultimately questions should be grouped and appear in a logical order, without much glaring contrast between two corresponding questions, ensuring that they do not elicit an implicitly contrasting response.
Test your survey out with friends and family
This one’s the most important tip of them all. The best way to figure out if your survey works is to get a few people to answer it. This is something that should be done during the survey concept stage, as a sort of pilot run to use to figure out any chinks in it. Even if you think that your survey is concise and engaging, it is best to test it out with a few folks (perhaps 5) to catch for those pesky errors and problems that do not appear without a second pair of eyes.
