News

Artificial Analysis tightens endpoint test as 24 of 103 trail references

Artificial Analysis says Endpoint Accuracy Index v1.1 uses more long-context repeats and tighter confidence intervals, with 24 of 103 measured serverless endpoints scoring statistically below self-hosted references.

D
Nov 19, 2026 · 2 min read

Artificial Analysis tightened its Endpoint Accuracy Index methodology and said 24 of 103 measured serverless endpoints scored statistically below corresponding self-hosted model references. The Endpoint Accuracy Index v1.1 release classified the remaining 79 endpoints as statistically in line with those references.

The revision raises the number of long-context repeats per question from 10 to 25, corrects two answer keys and changes the model used to grade answer equality. Across 48 endpoints and reference deployments scored under both versions, Artificial Analysis said the changes narrowed the average 95% confidence interval on its AA-LCR-25 long-context evaluation from plus or minus 4.7 points to plus or minus 2.6 points. The organization cautioned that raw AA-LCR-25 scores from v1.1 are not directly comparable with v1.0 scores.

The index compares a hosted endpoint with a self-hosted reference running the same model. Under the published methodology, an endpoint is in line when its reference-inclusive 95% confidence interval includes 100. An endpoint is below the reference when the entire interval falls below 100.

The composite gives equal weight to three evaluations: a 500-task tool-calling subset of the Berkeley Function Calling Leaderboard, a 250-question subset of Humanity’s Last Exam for hard reasoning, and the 25-question AA-LCR-25 long-context recall test.

Across 45 endpoints scored under both index versions, Artificial Analysis said the average composite confidence interval narrowed from plus or minus 5.5 points to plus or minus 4.3 points. The composite score moved by 1.6 points on average. Ten endpoints changed classification: six moved from in line to below the reference, while four moved from below the reference to in line.

The v1.1 results span six open-weights models. The release added GLM-5.3 Flash and Muse Spark Open, replaced DeepSeek V4 Pro with DeepSeek V4.1 Flash 1017, and removed GLM-5.2. Its aggregate table placed all 22 measured DeepSeek V4.1 Flash 1017 endpoints in line with the reference. Below-reference counts were 10 of 26 for GLM-5.3, five of 22 for GLM-5.3 Flash, five of 16 for gpt-oss-120b, two of 12 for Kimi K3 and two of five for Muse Spark Open.

Artificial Analysis attributed known shortfalls to output-token caps, reduced reasoning, tool-call parsing or formatting, and request limits or stream behavior. It also reported one endpoint below the reference on both the hard-reasoning and long-context tests without observed truncation or lower reasoning-token counts, leaving that result without one of those observed explanations.

The organization said its references used official model weights at recommended precision on vLLM or SGLang, running on Nvidia B200, B300 or GB200 systems. It used each model’s highest reasoning mode and each endpoint’s highest supported output and context limits. The results were measured from Oct. 8 through Nov. 4, 2026, so they are point-in-time comparisons rather than live monitoring.

More news