Surveys & Feedback prompts

Analyse these open-text responses.

RESPONSES:
"""
[PASTE]
"""

THE QUESTION THEY WERE ANSWERING: [EXACT WORDING]
WHO ANSWERED: [RESPONDENT DESCRIPTION AND HOW THEY WERE RECRUITED]
RESPONSE COUNT: [TOTAL, AND OUT OF HOW MANY ASKED]
WHAT I WANT TO LEARN: [PURPOSE]

Produce:

1. THE RESPONSE RATE CAVEAT - who answered the open question versus who was asked. People with strong feelings answer free-text questions; the silent majority is missing. State what this sample can support.

2. THE THEMES - with counts and the percentage of responses. For each: the theme, how many mentioned it, two or three representative verbatim quotes, and the range of intensity within it.

3. THE CODING FRAME - how you grouped responses, stated explicitly so I can check the judgement. Flag any response that could reasonably have gone in two themes.

4. FREQUENCY VERSUS INTENSITY - separate these. A theme mentioned by five people with great force is different from one mentioned in passing by forty. Both matter and they are different findings.

5. THE QUESTION WORDING EFFECT - assess whether the exact question wording steered the answers. If it asked 'what could be improved', you will get complaints regardless of overall satisfaction, and those complaints are not a satisfaction measure.

6. THE UNPROMPTED - things mentioned that the question did not ask about. These carry more weight than answers to the question, because nobody was steered toward them.

7. WHAT NOBODY SAID that you would expect. Note it, carefully, as an observation.

8. THE OUTLIERS - individual responses that are unusual and worth reading in full. Sometimes one person articulates something the themes flatten. Quote them.

9. THE LANGUAGE - the words respondents use for the thing you are asking about. Useful for future surveys, for marketing and for search.

10. THE FINDINGS ranked by confidence, with what each rests on.

Do not report a theme as a finding on the basis of two responses. State counts everywhere.

Review and fix these survey questions.

MY QUESTIONS:
"""
[PASTE]
"""

WHAT I WANT TO LEARN: [RESEARCH QUESTION]
WHO ANSWERS: [RESPONDENTS]
WHAT DECISION THIS INFORMS: [PURPOSE]

For each question, report: Original | Problem | Fixed version | Why it matters.

Check for:

1. LEADING - assumes an answer, or signals the one you want. 'How helpful was X?' presupposes it was helpful. 'Do you agree that...' invites agreement.
2. DOUBLE-BARRELLED - two questions in one, which cannot be answered. 'Was the service fast and friendly?'
3. LOADED TERMS - emotive or evaluative words that shift responses.
4. ASSUMED KNOWLEDGE OR BEHAVIOUR - questions that assume the respondent has done something or knows a term. Needs a filter question first.
5. UNBALANCED SCALES - more positive than negative options, no neutral midpoint where one is warranted, or verbal labels that are not evenly spaced.
6. MISSING OPTIONS - no 'none of these', 'not applicable' or 'don't know' where those are real answers. Forcing a choice manufactures data.
7. OVERLAPPING CATEGORIES - ranges like '1-5, 5-10' where 5 fits both.
8. RECALL BURDEN - asking about behaviour too far back or too precisely to remember accurately.
9. SOCIAL DESIRABILITY - questions where the honest answer is embarrassing. Suggest indirect phrasing.
10. HYPOTHETICALS - 'would you use', 'would you pay'. People answer these generously and then do not. Replace with past behaviour.
11. JARGON - your internal terminology.

Then:
- THE ORDER EFFECTS - where an earlier question primes a later one. Reorder.
- THE MISSING QUESTION - what my research question needs that I have not asked.
- THE QUESTIONS TO CUT - anything not tied to a decision, or answerable from data I already have.
- THE LENGTH WARNING - how many questions before completion rate drops meaningfully.

Help me interpret these survey results.

RESULTS:
"""
[PASTE - QUESTIONS AND RESPONSE DISTRIBUTIONS]
"""

SURVEY QUESTIONS AS WORDED: [EXACT WORDING]
WHO WAS SURVEYED AND HOW: [SAMPLE AND RECRUITMENT]
RESPONSE RATE: [RESPONSES OUT OF INVITATIONS]
WHAT I WANT TO CONCLUDE: [YOUR INTENDED READING]

Produce:

1. THE RESPONSE RATE PROBLEM first. Who responded and who did not, and how they likely differ. A 12% response rate does not give you the view of your customers; it gives you the view of the 12% who respond to surveys, who are systematically different. Everything below is subject to this.

2. WHAT THE NUMBERS SAY - literally, question by question, without interpretation.

3. THE WORDING EFFECT - for each key result, how much the exact question wording shaped it. Report the finding as an answer to the question actually asked, not to the question I meant.

4. THE BASE PROBLEM - for every percentage, the number it is based on. A percentage of a filtered subgroup can be based on very few responses. Flag any figure resting on a small base; these are where over-reading happens.

5. THE MIDDLE - how respondents used the neutral or middle option. Heavy use of the midpoint often means the question was unclear or irrelevant to them, not that they are neutral.

6. WHAT CANNOT BE CONCLUDED - specifically. Surveys measure stated attitudes at a moment in time among people who responded. They do not measure behaviour, they do not establish causes, and they do not predict what people will do.

7. MY INTENDED READING, ASSESSED - I said what I want to conclude. Is it supported? Where I am over-reading, say so directly.

8. THE DIFFERENCES BETWEEN GROUPS - if I have any, whether they are large enough to mean anything given the sample sizes in each group. Small subgroup differences are usually noise.

9. WHAT TO INVESTIGATE FURTHER - what the survey suggests but cannot confirm, and how you would find out.

10. THE HONEST HEADLINE - the strongest claim this survey actually supports.

Help me design satisfaction measurement.

WHAT I WANT TO KNOW: [YOUR GOAL]
WHAT I SELL OR PROVIDE: [CONTEXT]
CUSTOMER VOLUME: [HOW MANY, AND INTERACTION FREQUENCY]
WHEN I COULD ASK: [TOUCHPOINTS AVAILABLE]
WHAT I WOULD DO WITH THE RESULT: [THE ACTION]
EXISTING MEASUREMENT: [IF ANY]

Produce:

1. WHAT YOU ACTUALLY WANT TO KNOW - 'are customers satisfied' is rarely the useful question. More useful: are they getting the outcome they came for, will they come back, what is stopping them, and where does the experience break. Restate the goal.

2. THE MEASUREMENT OPTIONS - with an honest assessment of each:
   - A single satisfaction rating: simple, comparable over time, tells you almost nothing about why
   - A recommendation-likelihood score: widely used, widely over-interpreted. It is a number, not a diagnosis, and the scoring convention discards most of the information.
   - Effort or ease: often more predictive of repeat behaviour than satisfaction
   - Outcome achievement: did you get what you came for? Frequently the most useful single question and the least asked.
   - Open text: the only part that tells you what to change
   Recommend a combination for my situation and say why.

3. THE ESSENTIAL OPEN QUESTION - whichever scale you use, one free-text question does most of the work. Write it for my context. 'What is the main reason for your score?' is the standard and it is hard to beat.

4. WHEN TO ASK - the touchpoint matters more than the wording. Ask too early and they have no experience to report; too late and they have forgotten. For my situation, when.

5. THE RESPONSE BIAS - who will answer. Satisfaction surveys are answered by the delighted and the furious. Say how this skews the result and what to do about it.

6. WHAT THE NUMBER WILL NOT TELL YOU - and the discipline of not reporting a score as though it were a diagnosis.

7. THE ACTION LOOP - a score nobody acts on is theatre. Given my stated action, what result would trigger what response, and who owns it.

8. THE GAMING WARNING - if anyone's performance is judged on this, they will influence it: asking only happy customers, asking at a favourable moment, or coaching for a high score. Name the specific risks here.

9. THE MINIMUM VIABLE VERSION - the smallest thing that would be useful, if the full design is more than I will sustain.

Turn this feedback into a priority list.

FEEDBACK:
"""
[PASTE - SURVEY RESPONSES, REVIEWS, SUPPORT TICKETS, OR ALL OF THEM, LABELLED BY SOURCE]
"""

WHAT I PROVIDE: [CONTEXT]
MY RESOURCES: [WHAT YOU CAN REALISTICALLY CHANGE]
WHO GAVE THIS FEEDBACK: [AND HOW THEY WERE SELECTED]

Produce:

1. THE SOURCE BIAS - each feedback source over-represents someone. Reviews come from the extremes, support tickets from people with problems, surveys from responders. Say what this combined set over- and under-represents.

2. THE THEMES - with counts by source. A theme appearing across several sources is stronger evidence than one concentrated in a single channel.

3. THE CLASSIFICATION - each theme as one of:
   - REAL PROBLEM: something is broken or missing
   - EXPECTATION PROBLEM: it works as designed and was described badly. Usually the cheapest to fix.
   - WRONG CUSTOMER: people who should never have bought or signed up. A targeting problem.
   - PREFERENCE: someone wants it different, not broken. Changing may make it worse for others.

4. THE VOCAL MINORITY CHECK - for each theme, how many people, out of how many total. One articulate person can make a preference sound like a crisis. State counts everywhere.

5. THE SILENT MAJORITY - for each theme, who is affected but did not write in. Usually a multiple of those who did. Note where the true impact is likely much larger than the feedback volume suggests, and where it is not.

6. THE PRIORITY MATRIX - each theme scored on: how many are affected, how badly, whether it causes them to leave, and effort to fix. Rank by (impact / effort).

7. THE QUICK WINS - expectation problems and documentation gaps, which cost little and remove real friction.

8. WHAT NOT TO CHANGE - requests that would damage the offering for your main audience, come from people outside your target, or would take resources from something that matters more. Name them explicitly, because the loudest requests are frequently in this category.

9. WHAT TO INVESTIGATE - themes where the feedback signals a problem but not its cause.

10. WHAT TO TELL THE PEOPLE who gave feedback, including on the things you will not do.

Compare these survey results across periods.

PERIOD 1 RESULTS:
"""
[PASTE]
"""

PERIOD 2 RESULTS:
"""
[PASTE]
"""

WHAT CHANGED BETWEEN THEM: [ANY BUSINESS CHANGES]
SAMPLE SIZES: [EACH PERIOD]
WERE THE QUESTIONS IDENTICAL: [yes / no - if no, what changed]
WERE THE RESPONDENTS COMPARABLE: [same recruitment method? same population?]

Produce:

1. THE COMPARABILITY CHECK first, and it may end the analysis. If the questions changed, the samples were recruited differently, or the population shifted, the periods are not comparable and any difference is uninterpretable. Say so plainly rather than proceeding.

2. THE DIFFERENCES - question by question: period 1, period 2, the change, and whether it is larger than you would expect from sampling variation alone given these sample sizes. Show rough reasoning. Most period-over-period survey movements in small samples are noise.

3. THE NOISE THRESHOLD - state roughly how large a difference would need to be, at these sample sizes, before it means anything. Then apply it.

4. THE COMPOSITION CHECK - did the mix of respondents change? A satisfaction score can fall because a different group answered, not because anyone's satisfaction changed. Compare respondent characteristics if available; if not, flag this as an unresolved alternative explanation.

5. WHAT MOVED TOGETHER - if several questions moved in the same direction, that may indicate a genuine shift, or a change in who responded, or a mood effect. Distinguish.

6. AGAINST THE BUSINESS CHANGES - where a movement coincides with something I told you changed. Say 'coincides with', not 'caused by', and state what would establish causation.

7. WHAT DID NOT CHANGE - often the more important finding, especially if you expected it to.

8. THE HONEST SUMMARY - what can be said with confidence, what is suggestive, and what is indistinguishable from noise. Be willing to conclude that nothing detectable changed.

9. WHAT TO DO DIFFERENTLY next time to make the comparison cleaner: identical questions, identical recruitment, larger samples, or tracking the same respondents.

Do not report a movement as a change if it falls within normal variation.

Help me improve my survey response rate.

CURRENT RESPONSE RATE: [PERCENTAGE, AND OUT OF HOW MANY]
SURVEY: [PASTE IT, OR DESCRIBE LENGTH AND QUESTIONS]
HOW IT IS SENT: [CHANNEL AND TIMING]
WHO IS ASKED: [AUDIENCE AND THEIR RELATIONSHIP TO YOU]
INVITATION WORDING: [PASTE]
INCENTIVE: [IF ANY]

Produce:

1. THE LENGTH PROBLEM - count the questions and estimate completion time honestly. Length is the single largest determinant of response rate. Which questions can be cut? For each, ask whether any answer would change a decision, and cut it if not.

2. THE INVITATION - rewrite it. It should say: what it is about, how long it takes (honestly - understating it destroys trust and increases abandonment), what will be done with the answers, and whether it is anonymous. Under 80 words.

3. THE TIMING - when it is sent relative to the experience being asked about, and the day and time of sending. Match to when the audience is receptive.

4. THE FIRST QUESTION - it determines whether people continue. It should be easy, relevant and clearly connected to the stated purpose. A demographic question first loses people immediately.

5. THE FRICTION - mobile usability, whether a login is required, the number of clicks to start, question types that are tedious on a phone (matrix grids especially), and whether progress is shown.

6. THE INCENTIVE QUESTION - whether to offer one, and the honest trade-off: incentives raise response rates and attract people motivated by the incentive rather than the topic, which can distort the sample. Say whether it is worth it here. A prize draw and a guaranteed small reward affect the sample differently.

7. THE REMINDER - one, at a sensible interval, worded differently from the original. More than two reminders damages the relationship for diminishing returns.

8. THE TRUST FACTORS - anonymity, what happens to the data, who sees it, and whether anything will actually change as a result. The last one matters most for repeat surveys: people stop answering when nothing ever changes.

9. THE FEEDBACK LOOP - telling respondents what you learned and what you changed. This is the highest-return investment in future response rates and almost nobody does it.

10. THE HONEST CEILING - a realistic response rate for this audience and channel, so I am not chasing an impossible number. And the reminder that a higher response rate matters mainly because it reduces bias, not because the count looks better.