ChatGPT is a fast, cheap way to make sense of a pile of Etsy reviews — and it is genuinely useful for it, if you know the one thing it does badly. Paste in a few dozen reviews and it will group themes, surface complaints and draft replies in seconds. But it will also, with total confidence, hand you a customer quote that no customer ever wrote, a percentage it pulled from nowhere, and a keyword attached to the wrong product. Those are not rare glitches; they are how a language model behaves when it is asked to summarise and has no rule forcing it to stay literal. This guide gives you eight prompts that get real value out of ChatGPT for review analysis, then the three specific failures to watch for and exactly how to check its output before you trust a word of it. ChatGPT is the tool being discussed here; our own engine, Kiln, is mentioned only where it removes these failure modes by design.
The short version
- ChatGPT is good at grouping review themes and drafting replies — use it, with your eyes open.
- Its biggest failure is inventing quotes: it will produce a "customer quote" no customer wrote.
- It also invents counts and percentages, and attaches a keyword to the wrong listing.
- Verify every quote with a literal find in your real reviews; if it is not a word-for-word match, drop it.
- The checks are exactly what a purpose-built tool enforces in code, so nothing ungrounded reaches you.
01Why sellers reach for ChatGPT to read reviews
Reading reviews properly is real work. A shop with a few hundred reviews holds a genuinely useful dataset — the themes buyers care about, the complaints that recur, the exact words they search with — but pulling that out by hand is an evening you will not get back. ChatGPT looks like the perfect shortcut: paste the reviews, ask a question, get an instant, well-organised answer. And for a lot of tasks it is genuinely good — it will group themes, spot patterns and draft a reply faster than you could. The catch is that a language model is built to produce fluent, plausible text, not to stay faithful to a source. When you ask it to summarise reviews, it will sometimes fill gaps with things that sound right rather than things that are true — and it presents both with the same confidence. Used well, with the checks in this guide, it is a strong first pass. Used blindly, it will quietly put words in your customers' mouths. The trick is knowing which parts to trust and which to verify.
→ run the same analysis with every quote verified
02First, get your reviews out of Etsy
ChatGPT cannot see your shop — it only knows what you paste in — so step one is getting your reviews into text. Etsy lets you download your reviews: from Shop Manager, open Settings and then Options and use the Download Data tool, or export them from your Reviews page (help.etsy.com article 360000572708). That gives you a file you can open and copy from. If you only want a quick pass, you can also copy the visible reviews straight off your public Reviews page. Either way, paste the raw review text — including the star ratings, because you will need them for honest sentiment — into ChatGPT along with one of the prompts below. Keep the batches reasonable; very long pastes get truncated or summarised, which is one of the ways detail quietly goes missing. Work through your reviews in chunks and the model has less room to gloss over what it cannot fit.
You can download your reviews from Etsy: open Shop Manager, go to Settings then Options, and use Download Data, or export from your Reviews page (help.etsy.com article 360000572708). Paste the text into ChatGPT — it cannot read your shop on its own.
03Eight prompts that actually help
Give ChatGPT a specific job and it does far better than a vague "analyse these." These eight cover most of what a seller needs. One — themes with counts: "Here are my Etsy reviews. Group them into the recurring themes buyers mention, and tell me how many distinct reviews raise each theme." Two — recurring complaints: "From these reviews, list only the complaints that appear in more than one review, and include the star rating of each." Three — missing title keywords: "Here is my listing title: [paste it]. Here are its reviews. List the product words buyers use that are NOT already in my title." Four — honest sentiment: "Sort these reviews into positive (4-5 stars), mixed (3 stars) and negative (1-2 stars) using the star rating, not the wording." Five — per listing: "Group these reviews by the item they mention, and for each item list what buyers praise and complain about." Six — a competitor: "Here are a competitor's public reviews. What do their buyers complain about most, and what words do they use that I could target?" Seven — product ideas: "From these reviews, list any variations or additions buyers said they wished existed." Eight — a reply draft: "Draft a short, professional public reply to this negative review that acknowledges it, takes responsibility, and points to a resolution, in under four sentences." Notice that the strongest prompts force the model toward the source — count distinct reviews, use the star not the tone, only complaints that recur — which is also your first line of defence against the failures below.
04The three things ChatGPT will get wrong
For all its usefulness, ChatGPT fails at review analysis in three specific, repeatable ways, and every one of them can embarrass you if it reaches a listing or a customer. First, it invents quotes. Ask it for "a representative customer quote" and it will often produce a polished sentence in quotation marks that reads exactly like a real review — because no buyer wrote it. It has blended several reviews into a plausible-sounding fabrication. Second, it invents numbers. Ask how many buyers mention shipping and it may answer "about 40%" with total confidence, having counted nothing; the figure is a guess dressed as data. Third, it mis-attributes words to the wrong listing. Paste a mixed batch and it may read "these bridesmaid proposal boxes were perfect" in one review and cheerfully attach "bridesmaid proposal" as a keyword to your coffee mug, because it lost track of which review belonged to which item. Each failure comes from the same root: the model optimises for a fluent, satisfying answer, and without a hard rule tying every claim back to the source text, "fluent" and "true" quietly diverge.
Before you put any quote from ChatGPT into a listing, an ad, or a reply, search your actual reviews for that exact sentence. If it is not there word-for-word, it is invented — do not use it. The same goes for any number it gives you.
ChatGPT will hand you a customer quote in quotation marks that no customer ever wrote — and it will sound completely real.
Skip the fact-checking
Review Intelligence runs the same analysis but every quote is a literal match from a real review, and keywords only appear if a buyer actually wrote them.
05How to check ChatGPT before you trust it
The fix is a habit, not a better prompt: treat every concrete claim as unverified until you have checked it against the source. For quotes, this is simple and non-negotiable — take any sentence ChatGPT gives you in quotation marks and search your actual reviews for it word-for-word. If it is not there exactly, it is invented; discard it, no matter how good it sounds. Never put an unverified quote into a listing, an ad, a social post or a reply. For numbers, do not accept any percentage or count unless you can see the model actually counted, or you count it yourself; when in doubt, ask it to "list the exact reviews that mention this" and check the list is real. For per-listing keywords, confirm the reviewer who used the word actually bought that item before you add the word to that listing's title. It sounds laborious because it is — which is the honest catch with using a general chatbot for this. It will do the grouping in seconds and then quietly hand you three things you have to fact-check by hand.
06Or let the checks run for you
The reason a general chatbot needs all that fact-checking is that nothing forces it to stay literal. A purpose-built tool can make that impossible by design, and that is the difference with Review Intelligence. It runs the same analysis you would prompt ChatGPT for — themes, complaints, sentiment, the keywords your titles miss — on any public Etsy shop, but every quote it shows is a literal, word-for-word match from a real review, a keyword only appears if a buyer actually wrote it, and sentiment comes straight from the star rating rather than the model's read of the tone. There are no invented percentages, because the counting is done in code, not by the language model. Our engine, Kiln, does the reading; the guarantees do the fact-checking you would otherwise do by hand. So you can keep ChatGPT for the quick, low-stakes first pass and the reply drafts — and when you need numbers you can act on and quotes you can safely publish, let the checks run for you instead of running them yourself.