The fact that the development environment "can return a sentence" only proves that the shortest link is available, and does not mean that the application is suitable for production. Before going online, you need to verify failure paths, key boundaries, streaming integrity, cost records, and model switching, rather than just testing one successful request.
Check before you start
Prepare separately for production environment:
- Independent API Key, not shared with personal testing.
- Fixed real model ID and prepare controllable surrogate model.
- A set of desensitized test inputs covering short requests, long requests, streaming and tool calls.
- You can view logs of status code, request ID, time taken and number of retries.
Configuration steps
1. Connection and authentication
- Base URL ends up with no duplicates
/v1/v1。 - API Key only exists in server-side environment variables or key systems.
- There is no automatic infinite retry when testing a 401.
- Keys can be revoked and rotated independently.
- The front end, logs, error pages and monitoring events will not expose the complete key.
2. Models and protocols
- model use
/v1/modelsOr the real ID returned by the console. - Verify individually that the Chat Completions, Responses, or APIs the target client actually depends on.
- After the normal text is successful, the streaming output, tool calling, image input and other capabilities can be independently verified.
- Applications can give explicit errors when a model is unavailable, rather than silently switching and returning results of different quality.
3. Timeout and retry
It is recommended to configure the connection timeout, read timeout and total task timeout separately. There must be a maximum number of retries, and random jitter must be added to avoid a large number of requests hitting the service again at the same time.
| scene | Default processing direction |
|---|---|
| 400、401、model not found | Fix request without automatically retrying |
| 429 | Check quota and concurrency, read Retry-After (if any), limited backoff |
| Part 5xx | Log request ID, limited retry or switch to alternate path |
| Network outage | First determine whether the request may have been executed, and then decide whether to retry |
Tool calls with side effects must first design idempotent keys or deduplication mechanisms. Otherwise, a "retry" may result in repeated message sending, repeated writing, or repeated deductions.
4. Logging and Observability
Each request for suggestion records:
timestamp, request_id, model, endpoint, status,
latency_ms, first_token_ms, input_tokens, output_tokens,
retry_count, final_resultDo not save the complete API Key, user privacy, unmasked file contents or complete prompt words in the logs. When it is really necessary to retain it, access permissions, retention period and deletion mechanism should be set.
5. Cost and Capacity
- Record the number of requests, input and output usage, and failed retries.
- Set budgets or request caps per user, per task, and per day.
- Use real business samples to estimate, not just one sentence test.
- Use the console's live information as the source of truth for prices, model availability, and billing rules.
6. Release and rollback
- Preserve the previous version of configuration and model routing.
- Start with a small staged rollout, then increase traffic gradually.
- Ability to configure streaming, tool calls, or high-cost models to be turned off.
- Rollback does not rely on redistribution.
- Customer service troubleshooting information does not contain sensitive user data.
FAQ
Does having a backup model count as disaster recovery?
Doesn’t count. Alternate models may have different context, tool calls, structured output, and cost characteristics. It is necessary to use the same set of business tests to verify and clarify under what conditions the switch will occur, how to notify users, and when to resume.
Should the timeout be set as long as possible?
No. If it is too short, long tasks will be accidentally killed, and if it is too long, connections and concurrency will be occupied. Different policies should be set according to the request type, such as general question and answer, long text generation and Agent tool tasks are configured separately.
What is the most important thing before going online?
There is no single answer. The minimum is: keys are not leaked, failures are not infinitely retried, requests can be located, costs are observable, and configurations can be rolled back. If one of these five items is missing, it will be difficult to stop losses quickly after a problem occurs.
Next step
Write the list in this article into the release process and re-execute it every time you change the model, SDK, proxy layer or timeout policy. Don’t take a passing test as a permanent conclusion.