ChatGPT Works in the Playground But Breaks in Your Product

2026-06-04 · 12 min read

You pasted customer emails into ChatGPT and the summary was perfect. You wired the same prompt to the API and users got nonsense, refusals, or $800 bills in a week. The playground is not production.

What breaks when you go live

• Real user input is messier than your test examples
• No output validation; bad JSON crashes downstream code
• No rate limits; one user loops the feature and spikes cost
• No logging; you cannot reproduce the bad answer
• Model updates behind the scenes change behavior silently

Minimum production checklist

• Prompt stored in version control, not edited only in OpenAI dashboard
• Structured output schema validated before CRM or UI update
• Per-user or per-tenant daily token cap
• Fallback message when API timeout or refusal
• Sample of inputs/outputs logged (redact PII) for debugging
• Golden test set run before every prompt deploy

Human-in-the-loop for high-stakes actions

Draft emails, suggest CRM fields, classify tickets: auto with review queue. Refunds, legal text, medical adjacency: human approval required. Match autonomy to downside of being wrong.

Cost surprises and how to prevent them

Use cheaper models for classification, premium models for generation. Cache repeated queries. Cap max tokens per request. Show finance a dashboard of cost per feature before launch, not after the spike.