I'm still using DS-flash - it beats frontier model on speed and cost keeping good enough code quality.
It depends on the harness and constraints you give to the model but for most of the tasks its a great fit, I have not switched from DS since couple of weeks already (previously I would switch to Opus for some heavier work but I stopped recently).
My harness consists of multi-level AGENTS.md and SOUL.md containing "grug brain developer" and "asd-ste100" type of speech for clearer communication.
This might be the first time I’ve favourited a post without upvoting it.
This is really sus. I’d love a more detailed explanation and “take” from AA on what problem they felt needed correcting and why things have shuffled the way they have. And how they can claim their index has relevance if positions can be switched around so easily.
It is embarrassing how much they suck up to american labs while those models only perform the first few days after release. With open weights you can be sure the quality stays the same unlike the closed source crap. How about a retest after some time?
The natural language choice in multilingual index seems pretty strange and biased to me, they tried to encompass a broad swath of Asian languages, but then there's like four European romance languages. They didn't include any Slavic one i.e Russian, or Nordic one like Swedish, or Finno-Ugric. And these romance ones are probably overrepresented in the training data anyways
Glancing over the methodology[1] made me curious about three things:
1. Is there a benchmark measuring how much better an LLM performs when you tell it "you are an X"?
2. Does telling an AI agent "you are an AI agent" actually help? Is it the LLM equivalent of reminding a senior developer they're a senior developer, making them feel guilty about being lazy and finally getting them to do their job?
3. What happens when the persona conflicts with the task? If you tell it "you are a PM" and then ask it to write code, does it struggle? Produce worse code? Spawn a sub-agent and tell it they're a developer?
Yeah, this smells. Ain't no way Terra went up and Deepseek so far low. I have had the later beat the former on multiple criteria by simply being less distracted.
There was a thread here about families of models responding in the same patterns to specific queries like "say a random word", and it takes less than two minutes to verify it is true. It'll take a bit of effort to not convince me this change wasn't reverse-benchmaxxed to screw everyone other than OpenAI. I'll be less distrustful after everyone is done IPO-ing.
chinese labs are probably benchmaxxing but i'd like to see when the index changes relative to the position of the best chinese models (last suspiciously-timed update was when glm-5.3-flash was looking really strong)
Honestly it does somewhat match what I’ve felt playing a lot with all OpenAI / Claude / deepseek models
One thing I’m still surprised / unsure on is haiku 5.5 max besting latest deepseek - but also I think the last haiku was so incredibly bad I still hold some bias whilst using the new one.
We need an AI benchmark made in neutral Switzerland.
I'm still using DS-flash - it beats frontier model on speed and cost keeping good enough code quality. It depends on the harness and constraints you give to the model but for most of the tasks its a great fit, I have not switched from DS since couple of weeks already (previously I would switch to Opus for some heavier work but I stopped recently).
My harness consists of multi-level AGENTS.md and SOUL.md containing "grug brain developer" and "asd-ste100" type of speech for clearer communication.
This might be the first time I’ve favourited a post without upvoting it.
This is really sus. I’d love a more detailed explanation and “take” from AA on what problem they felt needed correcting and why things have shuffled the way they have. And how they can claim their index has relevance if positions can be switched around so easily.
From my rough recollection, all 3 of the past updates all had the effects of decreasing the score of open weight models.
One was rushed and same-day updated as GLM 5.3 (iirc) got added; of course its main change to rankings was lower GLM scores.
Fun fact: By changing the weights of the input you can change the output as you want.
It is embarrassing how much they suck up to american labs while those models only perform the first few days after release. With open weights you can be sure the quality stays the same unlike the closed source crap. How about a retest after some time?
The natural language choice in multilingual index seems pretty strange and biased to me, they tried to encompass a broad swath of Asian languages, but then there's like four European romance languages. They didn't include any Slavic one i.e Russian, or Nordic one like Swedish, or Finno-Ugric. And these romance ones are probably overrepresented in the training data anyways
Such a big re-ordering. The same week Anthropic files its S-1. Right...
Glancing over the methodology[1] made me curious about three things:
1. Is there a benchmark measuring how much better an LLM performs when you tell it "you are an X"?
2. Does telling an AI agent "you are an AI agent" actually help? Is it the LLM equivalent of reminding a senior developer they're a senior developer, making them feel guilty about being lazy and finally getting them to do their job?
3. What happens when the persona conflicts with the task? If you tell it "you are a PM" and then ask it to write code, does it struggle? Produce worse code? Spawn a sub-agent and tell it they're a developer?
[1] https://artificialanalysis.ai/methodology/intelligence-bench...
> GLM-5.3 (max) score highest at 32, and Claude Haiku 5.5 (max) outscores all of them
That seems improbable
"The FrontierCode verifier grades each attempt with the task's tests and rubric. We use GPT-6 Luna (medium) as the LLM judge"
Why are they using a closed source LLM as a judge? Closed source providers can see your traffic and adjust as they see fit.
Why did they call it “humanities last exam”?
Will an LLM topping this bench be the end of humanity? The doomer rhetoric has no place in what should be an objective measure.
Open-weight models
Kimi K3.1 Pro: −9
DeepSeek V4.1 Flash 1017: −13
GLM-5.3: −13
MiMo-V2.6-Pro: −16
Proprietary models
GPT-6 Astra: +4
GPT-6.1 Sol: +3
GPT-6.1 Terra: +3
Claude Opus 5.5: 0
Claude Sonnet 5.5: −2
Claude Fable 5.1: −3
Gemini 4 Argon (high): −3
Muse Wonder Full: −6
Claude Haiku 5.5: −7
Muse Wonder: −9
Gemini 3.8 Flash (high): −10
Grok 4.7 (high): −11
Is there affiliation with OpenAI or coincidence?
Yeah, this smells. Ain't no way Terra went up and Deepseek so far low. I have had the later beat the former on multiple criteria by simply being less distracted.
There was a thread here about families of models responding in the same patterns to specific queries like "say a random word", and it takes less than two minutes to verify it is true. It'll take a bit of effort to not convince me this change wasn't reverse-benchmaxxed to screw everyone other than OpenAI. I'll be less distrustful after everyone is done IPO-ing.
My experience is not consistent with yours.
It's wild seeing opus 5.5 up there so high, while i'm still using Fable 5.1 because I couldn't stand how opus writes.
Opus 5 writes like its off it's meds, opus 5.5 is a lot less weird that way.
chinese labs are probably benchmaxxing but i'd like to see when the index changes relative to the position of the best chinese models (last suspiciously-timed update was when glm-5.3-flash was looking really strong)
Interesting to see the open weight models drop so drastically from the changes here.
They perform pretty badly on the new benchmarks introduced.
Honestly it does somewhat match what I’ve felt playing a lot with all OpenAI / Claude / deepseek models
One thing I’m still surprised / unsure on is haiku 5.5 max besting latest deepseek - but also I think the last haiku was so incredibly bad I still hold some bias whilst using the new one.
Comparing the scoring rubric for AAv5.0 vs AAv4.3:
AAv5.0: 25% Knowledge work: AA-Briefcase v1.1 - 15% AutomationBench-AA - 5% GDP.pdf - 5% (-5%) 25% Coding: FrontierCode 2.0 - 15% (new) Terminal-Bench4.0 - 10% 25% Science: Terminal-Bench Science .1- 15% (new) CritPt v2 (hard subset) - 10% (new version) 25% General: AA-Omniscience v1.1 - 15% HLE-Diamond - 10%
AAv4.3: 30% Agents: AA-Briefcase v1.1 - 15% GDPval-AA v2 - 10% AutomationBench-AA - 5% 20% Coding: Terminal-Bench4.0 - 10% SciCode - 10% 30% General: AA-Omniscience (Accuracy) - 10% AA-Omniscience (Non-hallucination) - 5% GDP.pdf - 10% AA-LCR v1.1 - 5% 20% Scientific Reasoning: HLE - 10% CritPt 10%
tldr: gained Terminal-Bench Science and FrontierCode, dropped SciCode, GDPval-AA and AA-LCR. Lowered weight of GDP.pdf.