How to Summarize Survey Comments and Feedback in Six Sigma

Surveys are great tools for gathering information about a process or problem. Often times, the problem we encounter has no data to support it (perception only), or it’s unclear how the bad the problem is. Surveys can help us reach out to those experiencing the problem to get their input and answer some of these questions, especially customers of our work.

In order to apply Six Sigma methods, questions should consider using a Likert scale, most commonly a 1-5 rating, such as:

1. Strongly disagree
2. Disagree
3. Neither agree nor disagree
4. Agree
5. Strongly agree

1 to 10 scales are also used, but it may provide too much granularity, making it hard for the responder to decide between a 3 and a 4, or an 8 or a 9.

The Likert scale works really well for getting started with many improvements, since the lack of data is a common problem I encounter when working with clients. After gathering input from an initial survey, a follow-up survey can be performed after improvements are made, to determine if the scores increased significantly (if higher is better).

Another element of a good survey is to allow comments or text fields to capture any other information not asked specifically within the survey. This allows the user to clarify their response, explain their answers in more detail, or provide additional ideas, feedback or solutions. It’s frustrating for me to fill out a survey and feel like my major complaint or issue wasn’t even asked about.

I had a problem during a canceled flight last year, and when I filled out the survey they sent out, it didn’t ask me any questions related to the ticketing process, and there was no place to enter my comments. I tend to think this is done intentionally by companies, but that’s a discussion for another day…

After you receive the feedback from the comments in your survey, most people simply read through them to get some valuable insights and ideas, but it usually ends there.

Instead, take the comments and break them up into categories, and generate a Pareto chart.

Let’s take a look at an example…

When I worked in the corporate world, we conducted a survey of our Lean and Six Sigma practitioners, to see what they wanted from the corporate Lean office. Here is an example of some of the text comments we received. Below each comment, I’ve categorized the comment into one or more topics.

• “Back to the Basics – more 5 why training, root cause analysis, red bins, etc. Quality teams have been established, some are flourishing, some are floundering. Need to get back to 5 why analysis and also some of the change management that goes along with quality issues.”
  • Category #1: Focus on basic quality tools
  • Category #2: Change Management
• “Focus on Advanced Quality Planning to pro-actively avoid customer complaints. Support the quality organization with teaching and implementing fundamental APQP concepts.”
  • Category: Advanced quality implementation
• “Develop strategic plans on how we will turn the current customer dissatisfaction around and implement those across the enterprise (i.e. implement stop production, shipping if needed, error proofing, etc.). This has to be cascaded down to actions which can be understood by all. No vague statement like ‘ you need to make better quality’. What can be done concretely. An Annual Operating Plan (AOP) plan is not a strategic plan which will turn us around. It is a cost savings plan which every company has to have to stay financially healthy. Our customers don’t care about our AOP plan (shareholders do), they want to know i.e. how we will improve our quality.”
  • Category: Strategic Plan for Quality
• “Revisit what we do as a lean organization and ask ourselves what truly adds value to our customer. Let’s not things just because we have always done them. (i.e. the whole certification process which is now a check in the box and looses all credibility of what lean really should be).”
  • Category: Review Lean certification intent
• “Get back to understanding and doing the lean basics right. instead of checking the box. Ask what is the purpose. Training and support to help employees minimize multi-tasking within their own area.”
  • Category #1: Focus on Lean Basics
  • Category #2: Reduce multi-tasking across company

Here is the full data set from the list of comments, broken out by category/topic, and sorted from most frequent to least frequent into a Pareto chart.

Based on these results, we determined that we should also work on these areas for the upcoming year:

• multi-tasking
• lean basics
• leadership coaching for the upcoming year.

We might have been able to gather some of that input from just reading through the comments, but with so much feedback, that can be overwhelming. The categorization and graphical view helps us stay focused on the most important topics, and allows us to give feedback to the responders that their idea was heard, and why it did or did not make the top of the list.

One of the key elements of Six Sigma is to let the data guide us to prioritize, as we all have limited time and resources. That is a big reason why the simple Pareto chart is so critical.

How do you ensure the proper categorization of these comments?

This is a valid concern, and there are formal Six Sigma techniques that could be used like Attribute Agreement Analysis (or Attribute Gage R&R) to prove whether your categories are correct or not. But to make it simple, I suggest having 2-3 people categorize them on their own, then get together and compare notes, so you have one final answer. I wouldn’t get too complicated unless you’re going to use this survey on a frequent basis.

I would also recommend you make improvements to your survey in the future, based on comments you received. You might decide to remove some questions, or create some new questions.

Using Word Clouds

One newer and simpler method of organizing text is to look at keywords, and display by size based on frequency into a word cloud.

In this cloud I generated at WordClouds.com by simply pasting the comments into the website, we can see that the following words stand out:

• plan
• quality
• customer
• lean
• strategic
• organization

This can be helpful to identifying commonly used words and potential topics, but I feel this approach can be harder to develop tangible action plans.

Have you categorized comments into a Pareto chart before? What about a word cloud? If so, how did it work out for you?