Let me conclude first: Capability degradation cannot be confirmed with a single question. Keep the model, parameters, and test set fixed; repeat samples at different times; then compare protocol fields, token usage, dynamic capability probes, streaming, tool calls, and real business outcomes.
Open the LLM API gateway check, you can first generate a sub-report, and then supplement the business question set according to the method in this article.
Define capability degradation first
The "becoming stupid" that users feel may come from multiple aspects: model routing changes, gateways appending system prompts, context truncation, incompatible tool calls, request parameters being filtered, peak period throttling, or random output fluctuations of the model itself. Black box detection can only find that the results are inconsistent with the statement, and cannot determine the internal cause based on the phenomenon alone.
Therefore, it is recommended to split the problem into three levels:
| Hierarchy | Questions that need to be answered | main evidence |
|---|---|---|
| Protocol layer | Does the API return a standard structure as declared? | status code,model、usage, streaming events |
| Ability level | Whether key tasks are completed stably | Dynamic questions, tool calls, structured output, business question sets |
| time layer | Whether continuous drift occurs in the same configuration | Multi-period reporting, latency, error rate, token and result changes |
Repeatable detection steps
1. Fixed request conditions
Record Base URL, model ID,temperature, maximum output length, system prompts and test time. When comparing two API gateways, except for the API Key and address, the remaining conditions should be consistent. Otherwise the difference in results cannot be attributed.
2. Check the structured signal first
Prioritize checking authentication, HTTP status, response model,request ID,usage and Token arithmetic. Structured field exceptions do not necessarily mean that the model will be replaced, but they can quickly expose gateway compatibility, billing transparency or routing declaration issues.
Special attention is required:/models Not opening or returning 401, 403, or 429 may just be permissions and rate limiting strategies, and cannot be used as evidence of fraud alone.
3. Re-verify key capabilities
Cover at least the following three categories:
- Dynamic questions randomly generated by the server prevent suppliers from pre-setting answers to fixed questions.
- SSE streaming output, checking event format, delta content, and end tag.
- Tool call, check if the tool name, JSON parameters and random nonce are returned unchanged.
If your business relies on code, long text, retrieval, or structured JSON, you should also include your own desensitized samples. General dynamic questions can only check part of the ability and cannot replace production tasks.
4. Repeat the test at different times
It is recommended to run once at low peak and once at peak, and retest the next day. A failure may come from network, rate limiting or upstream fluctuations; multiple key capabilities continue to decline and can be repeated under the same conditions, which is worth upgrading to a high-risk signal.
What phenomena deserve special investigation?
- Request model and response
modelLong-term inconsistency, and the service provider cannot interpret the alias mapping. usagePersistently missing or apparently inconsistent Token arithmetic.- Usable tool calls, streaming output, or structured results continue to fail.
- The constraint compliance rate of the same business problem set dropped significantly, while the request parameters did not change.
- Systematic differences in model claims, capability results, or output structures between peak and off-peak periods.
These signals indicate that "investigation is needed", but they do not mean that fraud or intelligence has been proven. Gateway upgrades, model version changes, and client differences may also produce similar phenomena.
Don’t just look at the total score
The comprehensive score in the detection tool measures the compatibility of the protocol with key capabilities, not the model intelligence score or the manufacturer certification score. You should first check whether the failed item affects your business, and then check whether the results of multiple rounds are consistent.
For example, tool call failure has limited impact on ordinary text summarization, but it may be a blocking problem for Agent tasks; knowledge cutoff readme and output style are weak signals and are only suitable for long-term comparison and should not be judged as true or false independently.
Recommended comparison records to save
time, base_url, requested_model, response_model,
status, latency_ms, input_tokens, output_tokens,
dynamic_probe, sse, tool_call, business_suite, notesDo not save the complete API Key, user privacy, or non-desensitized prompt words when recording. After an exception occurs, it is easier to locate the problem by retesting it using the same question set and providing the service provider with the request time and desensitization request ID than just saying "the model has become stupid."
Next step
Run it once firstStandard Model Testing, read againHow to interpret test reports. If the main exception is that the request model is inconsistent with the response model, continue to checkModel impersonation and route substitution checks。