I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations.
Catching a lab cheating specifically on my one dumb benchmark would be really funny.
Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than anything I was considering.
His conclusion:
> Nothing jumped out at me. I couldn’t find a case where the pelican-bicycle images looked noticeably better than the rest of that model’s grid.
What if they’re not pelicanmaxxing, but svgmaxxxing in general?
Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge.
Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.
How useful actually is this? It generates SVGs of pelicans on bicycles, sure, and some of them are (almost) spatially correct. But, none of them look good.
AI image generation suffers from this more generally. You can generate pictures of pelicans, sure. Newer models clearly generate images with more pelican-ness than before. But all of it is still uglier than sin. Drawing things accurately is one thing, making results that someone might actually want to use (without embarrassing themselves) is something else.
Even if the models don't break through any particular "uglier than sin" barrier, with a bit more work, presumably the SVGs could become importable into an editor that would let a human apply taste and discretion. Seems to me like a heck of a head-start.
As for conventional diffusion-model stuff, I happen to think there are some pieces of AI art that still look really good even knowing they're AI.
No, the point is that a general-ish ability to draw good SVGs is a useful ability in itself. People need SVGs for all sorts of purposes, and if AI can generate one for them, that's mostly useful (discussions about art and employment etc notwithstanding).
That said, I think this would correlate relatively little with general programming ability. They're not unrelated, of course, but being able to generate code that paints an accurate + esthetically pleasing image is quite different from generating code that achieves a non-spatial goal.
I think a more fundamental test is SVG art creation in general. Perhaps a pipeline to take any image, caption it, ask the LLM for an SVG, rasterize to an image, and finally either use a deterministic visual similarity check or ask another LLM to be the judge and score how close the SVG is to the original image.
> All 21 pelican-bicycle images, across all seven labs, face right. No other animal/vehicle combination does that.
> However, facing right is common: 60% of all 1,008 images do it. How common depends on the animal and the vehicle, and bicycles are one of the two vehicles where it’s strongest
Of course the pelican on the bicycle is facing right. The drivetrain on a bicycle is on the right side. If you want any representation of a bicycle that shows the drivetrain you're going to show the right side of it if you want to do so without the frame occluding it. It's an excellent bet that their training data reflects this.
As near as I can tell, all of the bicycles are shown facing right, regardless of the direction the animal is facing (GPT 5.6-Terra, Sample 1/3). Also, in every case where the rider has legs (i.e. not the whale) both of the rider's legs are on the right side of the bicycle. This suggests a pretty serious lack of actual understanding of how a bicycle works.
I'm glad someone ran the numbers on this. Every single Simon Willison post of an SVG is followed with someone dismissing it saying "I'm sure they train on it by now." This is despite a good blog post with sound logic on how easy that is to catch. [1] Glad to see someone took the time for a quantitative analysis of dumb little animals riding dumb little bikes.
There's that version of the argument version, but there's also the softer version: that there used to be no training material of illustrated pelicans on bicycles, but now you have actual artistically talented individuals drawing it and that could improve the performance even though the AI labs are sucking it up no differently than everything else.
This post proves that hasn't happened yet, either. Although maybe the bad results posted online are being trained on and that explains the UNDER performance.
I don't think this small amount generalization to other animals and vehicles is strong evidence they haven't trained on this, either directly or more generally.
Why let your dreams be dreams? This is a perfect example of following a hypothesis. I love when people dive into an esoteric subject and just go full swing. Reminds me a CGP Grey and the name Tiffany. Sometimes you just need to know.
Pelicans aside, we need to remember that benchmarks are the only good quantitative way we have of comparing models. If someone has complaints about “benchmaxxing”, please ask them to contribute a better benchmark! It is valuable work and very appreciated.
If you train towards the test, you aren't necessarily improving overall fitness, but you are destroying the value of that test over time because you're decreasing its correlation with overall fitness.
> If someone has complaints about “benchmaxxing”, please ask them to contribute a better benchmark!
I don't think I'd go that far!
When someone says a model has been benchmaxxed, what they really mean is that it performs better in benchmarks compared to their real world experience. That's a real thing, I've certainly experienced it with some models.
...my take is that some things in life just resist quantitative measurements. Who is the best job candidate? What is the best programming language? Add AI models to the pile.
This is funny, I actually did a similar experiment just yesterday.
Looking for evidence of the same, but with another twist: checking if the models would choose to create a pelican on a bicycle, if no specific bird or method of transportation was specified.
Simple as they are, there are some really aesthetically pleasing penguins on skateboards in there, including from less capable models. (In fact, I would say the Opus series got progressively worse at it over time.)
I find your analysis much more convincing than TFA, since it doesn't require a subjective evaluation and is more robust to animal/transport complexity.
I feel like getting LLMs to spit out an SVG is akin to getting a human artist to draw something by just reciting a list of coordinates. It's insanely hard and unnatural.
Image generation models nowadays can easily generate a photorealistic pelican riding a bicycle, where the bicycle has perfect structure. But it is, of course, only a raster image.
It seems that we're missing a kind of step to decompose an image into a list of instructions (say, SVG paths, or even brush strokes with a real brush) to reproduce it properly. Doing so would probably need a true understanding of the structure of the scene, which is something that AI still struggles with to this day.
I don’t quite agree. Good human artist can visualize in their mind how to draw a picture, i think. Which i think is no different than LLM doing SVG drawing in their “head”. Anthropic’s recent post call this head-space “workspace”.
It just might feel foreign to human who does not have a SVG trained head-space.
I know. I'm just thinking about how to make AI create SVGs better... in theory, a sufficiently smart AI could "generate an image in its head", think about it, and then output the SVG paths to produce said image. Intuitively that would be somewhat closer to how human artists convert artistic visions into a sequence of arm movements while holding a brush (obviously, humans don't hold a fully formed, photorealistic image in the head while drawing, but rather vague concepts, but still).
Image models that support text output like Image2, or general text models that can read images like Claude can vectorize raster images. But they aren't very good at it, doing it manually in Inkscape still produces better quality even when done by non-artists.
It's a method to grade LLM output - as such it's something that will receive focus in correcting for. As soon as people who have a say in where funding is going noticed it as a metric the labs started caring about their performance in it. In the best case the labs are focusing on improving SVG capabilities in general and optimizing Pelican production as part of that initiative - but now that it's a known measure it is no longer reliable.
I've had the feeling that labs aren't pelicanmaxxing specifically but that they do have some sort of RL environment for SVGs that they are letting the AIs overcook in. Specifically I'm thinking of the gemini 3.1 pro annoucnement that seemed to have a huge leap in animated SVG performance but not much else impressive about it.
So they aren't pelicanmaxxing but they are benchmaxxing in a way. The benefit of the pelican was originally that uplift on the pelican signaled an overall uplift on intelligence. I don't believe that is the case anymore and it is just another jagged edge of model intelligence.
So this is not my experience at all for asking about simple SVG icons for web-pages. Here is one of the examples I have tried for in the past, make a simple cartoon SVG knife for a map icon for a crime map.
Can see the images for ChatGPT/Claude (Sonnet 5), and Gemini are all quite bad.
Jagged edge of LLMs. How do you explain being able to generate very complicated shapes in the Pelican example but cannot make a much simpler icon without just alluding to it is in the training data?
If you look at the raster image ChatGPT generated, that is fine. It is just this example (and other simple SVG icons I have asked for) result in pretty bad SVGs. It just makes me highly suspicious that the LLMs are learning shape primitives and extrapolating to new shapes, vs just having a big dictionary of prior examples and stitching them together.
That’s an impressive image, but what a mistake it was to click the second link (on mobile without an ad blocker). I wouldn’t send it to anyone I respect ...
OK, so we've done animal_vehicle, how about new SVG ideas each time? I just tried "make an SVG of a man sitting in a chair at a computer behind a desk" which gives more interesting results than the animalVehicle test.
I find it humorous that the animal + plane combo appears to be such an outlier. I assume this is due to the models assuming the user mean plain and misspelled it in the prompt.
I think it's actually due to "pelican on a plane" isn't the same as "pelican on an airplane" (Sonnet5 @ Flamingo x Plane), some consistent and warranted semantic/linguistic confusion!
Either you've memorized the outline (or detailed component shapes) of, say, a horse, or you haven't. Memorizing the outline of a pelican isn't going to help you with the horse.
You could train a model to do something a bit different like a pencil sketch, or vector graphic sketch, of something given a photo of it, and expect that to be a generalized skill, but if you are asking the model to do it "from memory" then memorizing a pelican is no substitute for not having memorized a horse.
He didn't. That's why the article exists. You have to do the science to see if it does.
He was asking the question - do we see gains across other tasks? The underlying question was: Is the additional attention given to this specific task creating a false impression of progress?
Just click through the models. At a glance (and highly subjective) I don't see anything jumping out as oom worse than anything else. I only noticed a model placing the animal inside a plane (with seat and small window) but other than that, they all seem similar inside each model to me.
If an AI researcher was going to pelicanmaxx, they would almost certainly apply the augmentations mentioned in the article during training, e.g. randomly selecting animals and conveyances. You’d want a model that generalizes well, just sfting in that specific prompt would be pretty bush league for a frontier lab.
I don’t have any reason to believe they are gaming the benchmark, just saying. I do find the idea of a data labeller having to generate thousands of svgs of different animals on different modes of transportation quite funny though.
At this point, I think there are so many pelican images in the pretraining data that drawing a pelican no longer makes sense as a model evaluation task.
I am not sure if this is how it works but let's say there was a reddit thread talking about the pelican benchmark and in it someone posts mockup examples of what an ideal result would look like
aren't some LLM going to digest that thread at some point and indirectly learn from it?
basically my point is originally this was a good benchmark because it was an absurd never-seen-before thing, but now that it is in content, some models are going to get a benefit in education?
you'd need the "AI" equivalent of an old-school "google whack", something with no previous results
They ingest so much data that a couple of reddit threads do not move the needle.
It is the reinforcement learning that produces more tangible results with less data, but it is something that the AI labs specifically selects and it is not picked up unknowingly
The models definitely seem to pay attention to the tests.
Since the tests can be generally gamed with directing descriptions at it non-deterministically, there's a greater chance the questions solution can be found.
Of course, hopefully the models are instead adding patterns and types of questions as well and it makes the models more capable, but it may be limited in how it transfers to other types of questions in breadth or depth.
> Every single one is a pelican, on a bicycle, first try. When every student gets an A, the exam has stopped grading.
Numerous pelicans and their bikes are clearly horribly malformed. In fact none of the bike frames are correct. Fable and Opus come close, but the top of the diamond is disconnected in Fable's case and the head tube is misaligned with the front fork in Opus's case.
And of course, as the parent post shows, labs don't actually seem to be training on the pelican bike case.
Agreed. And more; the Macbooks are pretty much the same - some are god approximations, some are terrible, all of them are recognisably a MacBook. And if you start using it they can train on it.
The problem isn't the test, its that is a public test.
Simon has previously said he has a list of secret prompts (at least one of which he "burned" as a demonstration a while ago). That's what makes it a good test - his commentary on the public test is something of a proxy for non-public tests. This makes it a good benchmark.
Asking questions that are absurd is like interviewing developers and asking absurd questions on the grounds that it tests creative and critical thinking.
Remember these Microsoft interview questions designed to identify the best developers?
"If you could eliminate one U.S. state, which one would it be?"
"How would you move Mount Fuji?"
Absurd interview questions have an air of legitimacy due to the quasi sophisticated justifications put forward for why they are good tests.
Absurd interview questions are not good tests of people or LLMs.
> Asking questions that are absurd is like interviewing developers and asking absurd questions on the grounds that it tests creative and critical thinking.
Yes; what's wrong with that?
Do you suppose that it doesn't test those qualities?
It was never intended as a meaningful benchmark. The surprising thing was that for the first ~12 months performance on the stupid pelican benchmark did seem to correspond to the performance of the models on other tasks.
That pattern no longer holds - Fable 5 and GPT-5.6 have both been out-pelicaned by lesser models now.
As I understand it; the point is to ask for an SVG which would demonstrate a conceptual understanding of what is being asked for and that is an important test IMO.
What sufficiently hard, but useful, problem would you ask the model for?
"Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."
Or the more pop layman version
"When a measure becomes a metric/KPI, it ceases to be a good measure."
Story time, I live in Argentina, and we don't have Big Macs, the main Mc Donald's brand, here, because during the CFK presidency, one of her tactics was to Goodhart economic metrics. Even the informal obscure ones like the [Big Mac Index](https://en.wikipedia.org/wiki/Big_Mac_Index), I don't know the precise details, but the Big Mac ended up being a very cheap item, like 2 or 3 times cheaper than actual menu items, but it was never on the advertised menu, and it also ended up being very small compared to the other burgers, so it wasn't even like a hack, a shrinkflation type of deal.
But hey, anyone who read the Big Mac Index table would never find Argentina at the bottom of that list along with a couple of other countries with bad brands, so the ploy worked. And now we live with the aftershock, the brand never really turned around, other brands with ridiculous names took over it like the McTasty, which makes me sound like that skit from Tarantino's Pulp Fiction.
This is fantastic
I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations.
Catching a lab cheating specifically on my one dumb benchmark would be really funny.
Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than anything I was considering.
His conclusion:
> Nothing jumped out at me. I couldn’t find a case where the pelican-bicycle images looked noticeably better than the rest of that model’s grid.
What if they’re not pelicanmaxxing, but svgmaxxxing in general?
Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge.
Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.
But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.
https://www.youtube.com/watch?v=jgYYOUC10aM
reminds me of this Key and Peele skit
How useful actually is this? It generates SVGs of pelicans on bicycles, sure, and some of them are (almost) spatially correct. But, none of them look good.
AI image generation suffers from this more generally. You can generate pictures of pelicans, sure. Newer models clearly generate images with more pelican-ness than before. But all of it is still uglier than sin. Drawing things accurately is one thing, making results that someone might actually want to use (without embarrassing themselves) is something else.
Even if the models don't break through any particular "uglier than sin" barrier, with a bit more work, presumably the SVGs could become importable into an editor that would let a human apply taste and discretion. Seems to me like a heck of a head-start.
As for conventional diffusion-model stuff, I happen to think there are some pieces of AI art that still look really good even knowing they're AI.
Vibecoding a SVG based metroidvania as we speak! This is gonna be lit!
AI can generate a fairly satisfactory SVG for a favicon now (programmer art quality at least).
well that holds IF svgmaxxing is 100% "code-writing-maxxing"
...which.. hmm I dunno if they are same or not
No, the point is that a general-ish ability to draw good SVGs is a useful ability in itself. People need SVGs for all sorts of purposes, and if AI can generate one for them, that's mostly useful (discussions about art and employment etc notwithstanding).
That said, I think this would correlate relatively little with general programming ability. They're not unrelated, of course, but being able to generate code that paints an accurate + esthetically pleasing image is quite different from generating code that achieves a non-spatial goal.
I don't see why that's true. LLMs don't have to only be good at code-writing.
Funnily enough, not that niche, because I have tried many times to do it as part of a wider project.
Addressed in the article, in case you're curious.
Well, it’s mentioned as a limitation of the analysis, very much not ruled out (or in.)
That simonw is causing labs to do extra fine-tuning runs for this seems highly probable :)
I agree, other formats, both textual and binary should be tested.
Simon I hope from this day hence, your bio always includes:
"Simon Willison, among other things, is an advocate for the inclusion of pelican geometry in LLM training datasets."
I think a more fundamental test is SVG art creation in general. Perhaps a pipeline to take any image, caption it, ask the LLM for an SVG, rasterize to an image, and finally either use a deterministic visual similarity check or ask another LLM to be the judge and score how close the SVG is to the original image.
Fidelity to the original is definitely not how humans would measure "art" in this context.
True, maybe we can call the generated SVG something else than art.
> Catching a lab cheating specifically on my one dumb benchmark would be really funny.
Similar thing happened when TPC came up with SQL benchmarks.
If you're not good at TPC, your engineering team is no good.
If you're good at TPC, then (as a customer) we will actually include you in a bake-off benchmark for our specific problem.
Winning on it is the price of admittance into the game, especially in a crowded market.
But how narrowly you benchmarket matters, you can't just hard-code that specific scenario & not fix anything adjacent while you're at it.
For example when it comes to GPUs, the "Quack3" (sic) benchmark on ATI cards comes to mind.
Perhaps also vary the bird? Wikipedia tells me pelicans are in the order _Pelecaniformes_ so shoebills or herons might do.
> I've been casually spot-checking other animals in other vehicles
Snakes on a plane, weasels on a diesel, spiders on a glider, baboons on a balloon, goats on a boat.
> All 21 pelican-bicycle images, across all seven labs, face right. No other animal/vehicle combination does that.
> However, facing right is common: 60% of all 1,008 images do it. How common depends on the animal and the vehicle, and bicycles are one of the two vehicles where it’s strongest
Of course the pelican on the bicycle is facing right. The drivetrain on a bicycle is on the right side. If you want any representation of a bicycle that shows the drivetrain you're going to show the right side of it if you want to do so without the frame occluding it. It's an excellent bet that their training data reflects this.
Citation: https://www.rei.com/c/bikes
Edited to add:
As near as I can tell, all of the bicycles are shown facing right, regardless of the direction the animal is facing (GPT 5.6-Terra, Sample 1/3). Also, in every case where the rider has legs (i.e. not the whale) both of the rider's legs are on the right side of the bicycle. This suggests a pretty serious lack of actual understanding of how a bicycle works.
I'm glad someone ran the numbers on this. Every single Simon Willison post of an SVG is followed with someone dismissing it saying "I'm sure they train on it by now." This is despite a good blog post with sound logic on how easy that is to catch. [1] Glad to see someone took the time for a quantitative analysis of dumb little animals riding dumb little bikes.
1. https://simonwillison.net/2025/Nov/13/training-for-pelicans-...
There's that version of the argument version, but there's also the softer version: that there used to be no training material of illustrated pelicans on bicycles, but now you have actual artistically talented individuals drawing it and that could improve the performance even though the AI labs are sucking it up no differently than everything else.
This post proves that hasn't happened yet, either. Although maybe the bad results posted online are being trained on and that explains the UNDER performance.
I don't think this small amount generalization to other animals and vehicles is strong evidence they haven't trained on this, either directly or more generally.
Why let your dreams be dreams? This is a perfect example of following a hypothesis. I love when people dive into an esoteric subject and just go full swing. Reminds me a CGP Grey and the name Tiffany. Sometimes you just need to know.
> The more plausible story is SVGmaxxing
Exactly--and you have to ask yourself at this point what "maxxing" really means, since "get better at drawing SVGs" is a useful skill.
Really awful how the AI labs are skillmaxxing /s
Pelicans aside, we need to remember that benchmarks are the only good quantitative way we have of comparing models. If someone has complaints about “benchmaxxing”, please ask them to contribute a better benchmark! It is valuable work and very appreciated.
It's a problem because of Goodhart's law.
If you train towards the test, you aren't necessarily improving overall fitness, but you are destroying the value of that test over time because you're decreasing its correlation with overall fitness.
> If someone has complaints about “benchmaxxing”, please ask them to contribute a better benchmark!
I don't think I'd go that far!
When someone says a model has been benchmaxxed, what they really mean is that it performs better in benchmarks compared to their real world experience. That's a real thing, I've certainly experienced it with some models.
...my take is that some things in life just resist quantitative measurements. Who is the best job candidate? What is the best programming language? Add AI models to the pile.
This is funny, I actually did a similar experiment just yesterday.
Looking for evidence of the same, but with another twist: checking if the models would choose to create a pelican on a bicycle, if no specific bird or method of transportation was specified.
My version of it: https://www.modelbias.ai/pelican-on-a-bicycle-test
Simple as they are, there are some really aesthetically pleasing penguins on skateboards in there, including from less capable models. (In fact, I would say the Opus series got progressively worse at it over time.)
I find your analysis much more convincing than TFA, since it doesn't require a subjective evaluation and is more robust to animal/transport complexity.
Thanks! Because I think that models are becoming better at creating SVGs in general. If you look at Claude Fable 5 and Kimi K3 for example.
In my tests it did create bicycles the most, but this is just a general bias I believe, as tested here: https://www.modelbias.ai/prompt/transport
I feel like getting LLMs to spit out an SVG is akin to getting a human artist to draw something by just reciting a list of coordinates. It's insanely hard and unnatural.
Image generation models nowadays can easily generate a photorealistic pelican riding a bicycle, where the bicycle has perfect structure. But it is, of course, only a raster image.
It seems that we're missing a kind of step to decompose an image into a list of instructions (say, SVG paths, or even brush strokes with a real brush) to reproduce it properly. Doing so would probably need a true understanding of the structure of the scene, which is something that AI still struggles with to this day.
I don’t quite agree. Good human artist can visualize in their mind how to draw a picture, i think. Which i think is no different than LLM doing SVG drawing in their “head”. Anthropic’s recent post call this head-space “workspace”.
It just might feel foreign to human who does not have a SVG trained head-space.
The pelican on a bicycle test is specifically about generating an SVG, fyi, not a raster.
I know. I'm just thinking about how to make AI create SVGs better... in theory, a sufficiently smart AI could "generate an image in its head", think about it, and then output the SVG paths to produce said image. Intuitively that would be somewhat closer to how human artists convert artistic visions into a sequence of arm movements while holding a brush (obviously, humans don't hold a fully formed, photorealistic image in the head while drawing, but rather vague concepts, but still).
Image models that support text output like Image2, or general text models that can read images like Claude can vectorize raster images. But they aren't very good at it, doing it manually in Inkscape still produces better quality even when done by non-artists.
It's a method to grade LLM output - as such it's something that will receive focus in correcting for. As soon as people who have a say in where funding is going noticed it as a metric the labs started caring about their performance in it. In the best case the labs are focusing on improving SVG capabilities in general and optimizing Pelican production as part of that initiative - but now that it's a known measure it is no longer reliable.
I've had the feeling that labs aren't pelicanmaxxing specifically but that they do have some sort of RL environment for SVGs that they are letting the AIs overcook in. Specifically I'm thinking of the gemini 3.1 pro annoucnement that seemed to have a huge leap in animated SVG performance but not much else impressive about it.
So they aren't pelicanmaxxing but they are benchmaxxing in a way. The benefit of the pelican was originally that uplift on the pelican signaled an overall uplift on intelligence. I don't believe that is the case anymore and it is just another jagged edge of model intelligence.
Underlying data is available on GitHub: https://github.com/dylanjcastillo/blog/tree/main/_extras/pel...
It's really refreshing to see someone publish a null result.
So this is not my experience at all for asking about simple SVG icons for web-pages. Here is one of the examples I have tried for in the past, make a simple cartoon SVG knife for a map icon for a crime map.
https://x.com/CrimeDecoder/status/2080008114615537766
Can see the images for ChatGPT/Claude (Sonnet 5), and Gemini are all quite bad.
Jagged edge of LLMs. How do you explain being able to generate very complicated shapes in the Pelican example but cannot make a much simpler icon without just alluding to it is in the training data?
Does asking for a dagger help?
If you look at the raster image ChatGPT generated, that is fine. It is just this example (and other simple SVG icons I have asked for) result in pretty bad SVGs. It just makes me highly suspicious that the LLMs are learning shape primitives and extrapolating to new shapes, vs just having a big dictionary of prior examples and stitching them together.
join me in building the ideal training set for pelicans riding bicycles: https://github.com/scosman/pelicans_riding_bicycles
Oh great! You've now made it a lot easier for LLMs to train on this dataset!
Your next iteration will need different animals and different transportation options. You'll run out after a few iterations.
"benchmaxxing by generalizing" is not really benchmaxxing
Another point for consideration: Specialized SVG models create way better looking pelicans riding a bicycle. (E.g. Refract V4: https://jumpshare.com/s/8liB7Aiuoo3yucbWGXjZ mirror: https://postimg.cc/McV70p84 )
That’s an impressive image, but what a mistake it was to click the second link (on mobile without an ad blocker). I wouldn’t send it to anyone I respect ...
The name is "Recraft V4", and from looking it up: yeah, it sure seems like whatever black magic they use for SVG generation kicks ass.
Hilarious. Could you imagine being a programmer at an AI company and this is your assigned task?
OK, so we've done animal_vehicle, how about new SVG ideas each time? I just tried "make an SVG of a man sitting in a chair at a computer behind a desk" which gives more interesting results than the animalVehicle test.
There probably good set of images of that description already so it does exercise the inference capability of the model
I find it humorous that the animal + plane combo appears to be such an outlier. I assume this is due to the models assuming the user mean plain and misspelled it in the prompt.
I think it's actually due to "pelican on a plane" isn't the same as "pelican on an airplane" (Sonnet5 @ Flamingo x Plane), some consistent and warranted semantic/linguistic confusion!
I noticed this happens especially with herons. My guess is it's because the model links "heron" to Heron's formula and the Cartesian plane
GLM has 2 combos of "on a plane" literally sitting inside a plane, with a window and a bit of wing showing. That's funny.
... Is that not how it should be interpreted?
You seem to assume training on pelican would not result in improved performance on other similar tasks. Why?
Either you've memorized the outline (or detailed component shapes) of, say, a horse, or you haven't. Memorizing the outline of a pelican isn't going to help you with the horse.
You could train a model to do something a bit different like a pencil sketch, or vector graphic sketch, of something given a photo of it, and expect that to be a generalized skill, but if you are asking the model to do it "from memory" then memorizing a pelican is no substitute for not having memorized a horse.
He didn't. That's why the article exists. You have to do the science to see if it does.
He was asking the question - do we see gains across other tasks? The underlying question was: Is the additional attention given to this specific task creating a false impression of progress?
AI rating AI? Am I missing something.
That seems to be how we signal "objectivity" nowadays.
Having an objective score is quite difficult. Maybe it would be better to do a pairwise comparison and calculate ELO?
Just click through the models. At a glance (and highly subjective) I don't see anything jumping out as oom worse than anything else. I only noticed a model placing the animal inside a plane (with seat and small window) but other than that, they all seem similar inside each model to me.
If you want to, go ahead, but it seems to me the author already exceeded the energy expenditure that this question warranted.
If an AI researcher was going to pelicanmaxx, they would almost certainly apply the augmentations mentioned in the article during training, e.g. randomly selecting animals and conveyances. You’d want a model that generalizes well, just sfting in that specific prompt would be pretty bush league for a frontier lab.
I don’t have any reason to believe they are gaming the benchmark, just saying. I do find the idea of a data labeller having to generate thousands of svgs of different animals on different modes of transportation quite funny though.
At this point, I think there are so many pelican images in the pretraining data that drawing a pelican no longer makes sense as a model evaluation task.
How does attempt 2 by Llama 4 Maverick look like a bald eagle??
I am not sure if this is how it works but let's say there was a reddit thread talking about the pelican benchmark and in it someone posts mockup examples of what an ideal result would look like
aren't some LLM going to digest that thread at some point and indirectly learn from it?
basically my point is originally this was a good benchmark because it was an absurd never-seen-before thing, but now that it is in content, some models are going to get a benefit in education?
you'd need the "AI" equivalent of an old-school "google whack", something with no previous results
* https://en.wikipedia.org/wiki/Googlewhack
They ingest so much data that a couple of reddit threads do not move the needle.
It is the reinforcement learning that produces more tangible results with less data, but it is something that the AI labs specifically selects and it is not picked up unknowingly
It's incredible that each model has it's own style that remains relatively consistent throughout all of the different generated examples.
The models definitely seem to pay attention to the tests.
Since the tests can be generally gamed with directing descriptions at it non-deterministically, there's a greater chance the questions solution can be found.
Of course, hopefully the models are instead adding patterns and types of questions as well and it makes the models more capable, but it may be limited in how it transfers to other types of questions in breadth or depth.
https://playcode.io/blog/macbook-svg-benchmark
I think we should stop using pelican benchmark.
I disagree with this in the blog post:
> Every single one is a pelican, on a bicycle, first try. When every student gets an A, the exam has stopped grading.
Numerous pelicans and their bikes are clearly horribly malformed. In fact none of the bike frames are correct. Fable and Opus come close, but the top of the diamond is disconnected in Fable's case and the head tube is misaligned with the front fork in Opus's case.
And of course, as the parent post shows, labs don't actually seem to be training on the pelican bike case.
Agreed. And more; the Macbooks are pretty much the same - some are god approximations, some are terrible, all of them are recognisably a MacBook. And if you start using it they can train on it.
The problem isn't the test, its that is a public test.
Simon has previously said he has a list of secret prompts (at least one of which he "burned" as a demonstration a while ago). That's what makes it a good test - his commentary on the public test is something of a proxy for non-public tests. This makes it a good benchmark.
The pelican prompt is ridiculous.
Test the LLLM against things you want it to do.
Asking questions that are absurd is like interviewing developers and asking absurd questions on the grounds that it tests creative and critical thinking.
Remember these Microsoft interview questions designed to identify the best developers?
"If you could eliminate one U.S. state, which one would it be?"
"How would you move Mount Fuji?"
Absurd interview questions have an air of legitimacy due to the quasi sophisticated justifications put forward for why they are good tests.
Absurd interview questions are not good tests of people or LLMs.
Relevant questions are good tests.
> Asking questions that are absurd is like interviewing developers and asking absurd questions on the grounds that it tests creative and critical thinking.
Yes; what's wrong with that?
Do you suppose that it doesn't test those qualities?
The whole point of "AI" is arbitrary task completion. Why isn't a SVG drawing relevant for that?
> The pelican prompt is ridiculous
Yes, deliberately so.
It was never intended as a meaningful benchmark. The surprising thing was that for the first ~12 months performance on the stupid pelican benchmark did seem to correspond to the performance of the models on other tasks.
That pattern no longer holds - Fable 5 and GPT-5.6 have both been out-pelicaned by lesser models now.
As I understand it; the point is to ask for an SVG which would demonstrate a conceptual understanding of what is being asked for and that is an important test IMO.
What sufficiently hard, but useful, problem would you ask the model for?
> Test the LLLM against things you want it to do
I agree, it is ridiculous to ask an LLM to replace an artist.
https://en.wikipedia.org/wiki/Goodhart%27s_law
"Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."
Or the more pop layman version
"When a measure becomes a metric/KPI, it ceases to be a good measure."
Story time, I live in Argentina, and we don't have Big Macs, the main Mc Donald's brand, here, because during the CFK presidency, one of her tactics was to Goodhart economic metrics. Even the informal obscure ones like the [Big Mac Index](https://en.wikipedia.org/wiki/Big_Mac_Index), I don't know the precise details, but the Big Mac ended up being a very cheap item, like 2 or 3 times cheaper than actual menu items, but it was never on the advertised menu, and it also ended up being very small compared to the other burgers, so it wasn't even like a hack, a shrinkflation type of deal.
But hey, anyone who read the Big Mac Index table would never find Argentina at the bottom of that list along with a couple of other countries with bad brands, so the ploy worked. And now we live with the aftershock, the brand never really turned around, other brands with ridiculous names took over it like the McTasty, which makes me sound like that skit from Tarantino's Pulp Fiction.
How did the president manage to influence McDonalds' local business decisions? And how did that lead to McDonald's pulling out of the country?
I wish I could downvote this for Pelicanmaxxing lmao.
We're going to keep maxxmaxxing forever.
Would you prefer the term Pelicangate?