LLM API comparison test template: results can be reproduced and costs can be verified

Let me conclude first: To fairly compare two LLM API entrances, conditions other than Base URL and API Key must be fixed, sampling repeated over multiple periods, and compatibility, quality, stability, Token, and billing recorded separately. If you only compare a speed test screenshot or a fixed question, the conclusion cannot be reproduced.

Define the decision problem before comparing

Different businesses require different weights. First write down clearly what this test will decide:

• Common Conversation: Focus on success rate, first byte latency, context, and price.
• Code Agent: Focus on real protocol requirements such as tool invocation, structured parameters, long-term task stability, and Responses API.
• Content production: Focus on constraint compliance, factual errors, long-form consistency, and manual rework time.
• Batch processing: Focus on throughput, 429s, retry costs, billing verifyability, and failure recovery.

Without clear business goals, overall scores can easily mask real blockers.

control variables

In addition to the service address and key, the following conditions must be consistent:

If the model IDs or protocols returned by the two portals are not equivalent, they should be marked as different products and should not be forcibly merged into "Same Model Hengping".

Suggested four-level question set

Layer One: Protocol Smoke

Verify /models, minimal chat, error response,model、usage, SSE and tool calls. When the agreement does not meet business requirements, there is no need to continue to expand the sample.

Level 2: Dynamic ability questions

Each round uses a local script to generate random numbers, random field names or random nonces, check calculations, constraint compliance and tool parameter return. Dynamic questions can reduce fixed answer pollution, but cannot prove the identity of the underlying model.

The third level: public tasks

Prepare automatic judgment tasks such as code repair, information extraction, long text constraints, JSON Schema and multi-round dialogues. Questions, expected results, and evaluators must be versioned.

Level 4: Real business samples

Extract and desensitize historical failures. Record task completion rate, manual modification time and business interruption reasons. Real samples are most valuable for decision-making, but they cannot leak user data or internal keys.

Original record template

Keep one row of raw data per request:

run_id,provider,started_at,model_requested,model_returned,http_status,first_byte_ms,total_ms,input_tokens,output_tokens,total_tokens,retry_count,probe_id,passed,error_type,request_id

Create another test configuration file:

suite_version: "2026-07-14.1"
temperature: 0
max_tokens: 512
timeout_seconds: 60
retries: 0
stream: false
sample_windows:
  - low_peak
  - high_peak
  - next_day

Do not record the complete API Key, non-desensitized prompt words, user files, or personal information in the response. Request IDs should also be confirmed not to contain sensitive data before being made public.

Indicators to be calculated

Availability

• Raw success rate: The proportion of first requests that return valid results.
• Final success rate: including the success ratio after limited retries.
• Error distribution: 401, 429, 5xx, timeout, and protocol parsing errors are counted separately.

Don't count success after infinite retries as entry stability. First-time failure rate determines true latency and additional costs.

delay

• First byte delay: How long it takes for users to see the first piece of content in streaming scenarios.
• Total time taken: How long it takes for the task to be completed completely.
• P50/P95: Observe typical experience and tail slow requests respectively.

Capability quality

• Constraint compliance rate.
• JSON or tool parameter parsing success rate.
• Code test pass rate.
• Real business one-time completion rate.
• Average manual rework minutes.

Token and cost

Summarize input, output, failed requests and retry tokens respectively. Then calculate based on the price snapshot during the test:

Complete task cost = Successful request fee + Failed request fee + Retry fee + Manual rework cost

Prices and model availability change. Record the lookup time and source in each report, and never present a one-time snapshot as permanent pricing.

Sample size and interpretation boundaries

Protocol checks can be run in small numbers, but stability and latency will require at least dozens of requests to see an initial distribution. The smaller the sample, the less likely it is to use conclusions such as "inevitably faster" and "absolutely stable".

Model output is random. Even if temperature=0,Different service implementations, versions and parallelism,strategies may also produce differences. Competency conclusions should be combined with multiple questions, multiple rounds and real business baselines.

The following phenomena can only be marked as "requires verification":

• Return model Inconsistent with the request.
• usage Missing or Token arithmetic exception.
• A certain round of dynamic questions failed.
• Output style or knowledge cutoff readme changes.

Only when multiple pieces of evidence continue to appear and can be reproduced is it appropriate to upgrade the risk level. Black box results cannot replace model manufacturer certification.

Minimal structure when publishing reports

An auditable public report contains at least:

1. Test date, region, network and client version.
2. Model ID, protocol, parameters, and question set version.
3. Sample size, raw data fields, and whether failures are counted.
4. Success rate, P50/P95, capability indicators, Token and cost.
5. Known limitations, exception explanations, and retest plans.
6. A description that the key and business data have been desensitized.

Can be used firstAPI gateway detection toolGenerate protocols and behavioral sampling, and then followTest report interpretation guideView itemized evidence. Continue to use during the selection stage12-item API gateway test list, to avoid a single total score replacing business judgment.