Intercom published on the OpenAI blog how it built Fin, the agent that today closes support tickets across chat, email, and voice at industrial volumes. Three lessons, all operational.
First: A/B testing on models in production, not just in the lab. When a new model comes out, they run it in parallel with the live one on a slice of real traffic, and measure resolution rate and CSAT before promoting it. Second: structured offline evaluations before deploy. No "feels better", only reference datasets and numeric gates. Third: model-agnostic architecture. Routing decides which model answers which request, and swapping it doesn't require rewriting the product.
The interesting point for anyone building agents in the enterprise isn't Fin itself. It's that Intercom has codified an evaluation cycle that looks like an industrial process: offline gates, live A/B, model swap as configuration, not as refactor. Most enterprise AI projects today have none of these three things. You deploy the model that was trendy at kickoff and you hope.
Why this matters for anyone building enterprise AI: without a numeric gate and an architecture that lets you swap the model, every release is a bet.