Survey Coding in Market Research | Is There a Bias?

As previously defined in one of the Bunker’s prior post, survey coding is the process of taking the open-end responses and categorizing them into groups.  Once coded, they can be analyzed in the same way single and multiple response questions can be.  So, the process is fairly simple.  It usually involves looking at exported data on an Excel sheet, adding columns for coding, and reading through the 0pen-ends and assigning each open-end with a category code.  From there you can subtotal the codes and divide them by your pool to assign representative percentages.

What is survey coding in market research?  Here is an example:

The process seems very simple, but there are many factors that contribute to coding that make it unscientific.  Research & Marketing Strategies (RMS) – a market research company in Syracuse, NY – utilizes coding in almost every single quantitative report sent to clients.  It’s extremely valuable for clients (and much easier) for them to be able to see a percentage breakdown of how respondents answered a question rather than having them read through 800 open-ends.  However, caution should be taken when reviewing any coded question.  Here are a few reasons survey coding is not a scientific and fool-proof process:

1) Survey coding is subjective – if you were to take three separate analysts and have them code the same identical 100 open-ends, you will undoubtedly end up with three very different graphs.  Not only will you end up with different percentages for each category but most likely different categories as well. You could argue that even the same analysts would code the data differently depending on the day.  It helps to create some kind of consistency with codes, either referring to a past project with that client, or using the same analyst to code questions from the survey.  It also helps if you know if there are any specific topics that the client would like pulled from the open-ends and you can be sure to create a category for that.  For example, if a client wants to find out how many people referenced the “Christmas special” on the question about content of advertising, you can ensure that a category in your graph will depict that.

2) Processes differ – some analysts will limit coding to using just one all-encompassing code, while others could decide to use up to four or five if the open-end covers a wide array of topics.  On top of the decision to use codes per open-end, some analysts will aggregate the statistics by percentage of responses while others will use percentage of respondents.  We endorse the percentage of respondents.

Now, I am not arguing that you should not use survey coding in a report.  In fact, for previously stated reasons, it offers analysts and clients both major benefits that I believe outweigh the subjectivity of the process.  Other substitutes for coding of open-ends that seem to be a nice supplement are using Wordles in our PowerPoint reports.  These Wordles encompass all words used in verbatim responses and are able to highlight the words used most often by respondents in larger font.  A second option is limiting the scope of the open-ended question to force the respondent to eliminate interpretation from the analyst.  Instead of asking “when you think of client XYZ, what comes to mind?,” you may want to ask “when you think of client XYZ, give me one word or phrase that comes to mind.”  By limiting the scope of the response, it eliminates additional details and cuts right to the core of the question (and it will save you hours of coding – You’re welcome!)

Do you have questions about coding or how surveys can benefit your business?  Contact our Business Development Director Sandy Baker at SandyB@RMSresults.com or by calling us at 315-635-9802.

Read Full Post »

6 Survey Coding Tips | Survey Research in Central NY

Including open-end responses in a survey is always a great way to get information; types of information that weren’t touched upon in the script or answers that wouldn’t have previously been listed for a question.  It can be a great source for qualitative data within a quantitative survey.  Survey coding is the process of taking the open-end responses and categorizing them into groups.  Once coded, they can be analyzed in the same way multiple response questions can be.  The survey coding process can be very tedious in order to ensure reliable results.  After reading through 500 comments of your dissatisfied customers, you may feel like you need counseling, but the results will be invaluable to your company.  So ask your market research vendor to send you all of the open-end responses in an excel file at the conclusion of the fieldwork.

The survey coding process for open-end questions must be undertaken carefully, as responses can be open to judgment and interpretation of the individual.  The results can vary from person to person depending on what code you use for the open-ended comment.  The process can be very subjective, even though one may think market research analysis is meant to be objective.

Here are a few tips for survey coding:

1. Read through all open-ended responses ahead of time.  This helps the analyst get a feel for the themes that are recurring in the data set.  It will also help the analyst understand how the population is responding to the specific survey question.  It can be surprising how many people answer an open-ended question in a similar fashion to one another without any guidance. 
2. Start by creating a lot of categories before narrowing the field.  Once all of the categories are laid out and initially coded, begin to further combine the data to limit the analysis to about 8-12 codes.
3. Make sure everyone’s comment counts.  A reporting standard for RMS is to create 8-12 all-encompassing survey coding categories, even if there are a few outliers that are lumped into the category of “Other,” they must be mentioned and/or footnoted.
4. Create accurate and unambiguous codes, which cover the responses they apply to.  If someone were to look at the code title after reviewing a question, it should be clear what type of comments fell under it.  It may help to create a pop-out box in the report with further detail on the code with some explanation. For example, stay away from creating a category like “Service Related” – does that mean Customer Service? Billing Service? Service in relation to usage?
5. Feel free to use more than one code.  Many times respondents offer multiple comments on a question spanning from topic A to topic Z.  If a response was limited to only one code, the respondent’s other areas of concern would be understated.
6. Consider coding % of respondents rather than % of responses.  This will give an equal value to all responses.  This prevents those who express concern in multiple areas from overpowering those who had a single code response.

The responses to open-ended questions are the most raw and unaffected parts of survey analysis.  The questions are completely unaided, and respondents can say or write anything that comes to mind.  They are not limited to the selecting choices or guided in their response.  With open-ended questions you get a true sense for how the respondents feel.  At the same time, the survey coding can be compared and analyzed for more quantitative/ statistical analysis.  The process of reviewing all open-ends and survey coding has become an important standard in reports delivered by Research & Marketing Strategies (RMS) and the results are always very rewarding for our clients.

Read Full Post »