What the Azure OpenAI Demo Doesn't Show You
The demo works. Three months into production, the same system is producing wrong answers on a predictable subset of inputs, costs three times what was projected, and has no mechanism to detect when it is failing. Here is what the demo leaves out.