A little irrelevant, but I need to vent about something MUD adjacent.
I really wish MUDs were still a thing. The text-based roleplaying communities were largely swallowed by Discord, where as before they existed scattered over proboards, jcink, and a few others.
Discord, being the imperfect text platform, limits the text and leaves most interaction limited by awkward formatting. To boot, even when a better ecosystem is invented, players refuse to give it a shot even if directly invited, because they got comfortable with Discord's terribleness.
It's so frustrating, and I don't know how to voice it, or even what can be done.
MUDs never disappeared. People stopped finding them engaging enough to use. Blaming Discord reverses the causality. MUDs lost engagement long before disco was created. People congregate on disco because it offers a more compelling social interface, even if it lacks the depth of a persistent game world. The opportunity I'd draw from this isn’t to convince people to return to older and less engaging interfaces. It’s to build persistent roleplaying systems around the interaction modes that have proven to be more engaging already. Potentially combine voice, speech recognition, text, tts, and synthesized characters with a proper world simulation underneath. If roleplaying communities already gather in disco, then deliver the experience where they already are. The interface was always irrelevant to what made MUDs fun anyway. Imagination and connecting with other people was always the thing.
I specifically outlined non-MUD role playing environments because I know MUDs lost popularity long before Discord. I am outlining that any form of tech, besides Discord, is sidelines -- MUDs and a hopeful resurgence as well.
I agree with you. You've also got more / easier competition. IIRC, MUDs peaked in the mid-late 90s. MMOs were really the death knell, then the explosion of other types of games, mobile games, then to your point things like Discord.
One adjacent project we're exploring for the future is if you took all of the current understanding of game design and mechanics with modern AI functionality, knowledge bases, etc, and built a MUD, what would it look like and would it be enough to bring back players from other games?
There is an ongoing project doing this (that launches this weekend I believe) to revive a MUD called Shadows of Isildur. I think a lot of effort has been put into the new changes to the codebase to bring QoL features that a modern audience would expect while keeping the bones of the game the same. I'm not certain how it will fare as, to be honest, MUDs take an inordinate amount of time to really groove into and immerse yourself in - but I am excited to see efforts like this invested on as we older MUDers find ourselves finally reaching the point of having free time again.
Would be kind of a cool experiment for a mud-style interface, where each npc/avatar was backed by a lower-cost LLM... similar rules for the npcs in the game, but a background feed, and history of interactions with other players as background.
Maybe limiting npc's to only a certain number of moves that aren't a response to other users per day...
The people who played MUDs for the grind will, genuinely, have a lot more fun playing PoE or some other modern adventure game. MUDs were the only option for that sort of adventure experience for a while but there are other options now that simply have more tools at their disposal than text based communication offers.
If a MUD is going to survive today it's going to be driven by those RP focused players where the freedom to describe actions in text allows a flexibility that's unrealistic in graphical games.
I honestly worry that the relevance of text-based systems is largely tied with literacy and how it interacts with the player's imagination -- something largely in decline, at least in the US.
Simutronics is has been running and developing Gemstone and DragonRealms for 30+ years and counting. The communities are about 400-1000 players each. They've embraced microtransactions (in the form of quarterly events), but both games are entirely playable without participating in that aspect.
With modern technology, training in both games is very automatable. But developing and optimizing your training routines is (in my opinion) incredibly fun. Actual roleplaying is less common it was in the 90s, but there are still nightly events where dozens of players will get together to socialize and explore the worlds in-character.
I highly recommend it, and for those who don't have the time to grind, you can join the shattered server where 24/7 scripting is allowed and encouraged. F2P subscriptions don't work on that server, unfortunately, but it has been a great hobby for someone like me who used to play back in the day but could never play it without scripting due to time constraints.
MUDs are still kicking around and if you'd like to play in one you'll find a few solid communities with good engagement that would love to include you. They're in an odd spot though since MUDs existed in two sorts of categories 1. I really want to adventure and text is the only way to do that (these sorts died off to MMOs) and 2. I want to roleplay - those survive but the mechanical portions of those games are becoming more and more irrelevant as new games are constantly supplying better systems and mechanisms than MUDs were able to.
This leaves MUDs in a weird place since things like MUSHes generally have most of the roleplaying tools without as much weird code for attacks and PVP interactions being involved so I've seen those thrive more recently.
The MUD I play (Aardwolf) averages about 200 players online at any time, but it's effectively dead as a MUD. There's very little player interaction, which is what made MUDs special originally. Part of this is that nearly all multiplayer features have been abandoned, and the rest of the features are centered around loot box equivalent features. Despite this, there are a couple hundred people who seem to enjoy the gamification. The MUD is highly optimized for scripting, so it scratches the side project programming itch for me.
I have a pile of ideas and not enough time to get to them all, but one of them is the opposite direction. Embrace Discord, and build a MUD whose social systems are funneled through it.
Technically MUDs still are a thing, there's a good number still running. Though sounds like you don't really just want a MUD where you want to go out mobbing but a more social thing (but not IRC?).
I suspect MUDs may have a bit of a resurgence soon, I've seen more and more interest in them as of late simply because they seem like an excellent pairing with LLMs which easily consume text. Though I think for your needs someone would need to work on the barrier to entry to getting on a MUD as well as the modern UI/UX expected today.
Having stepped away from MUDs ourselves for years, it's been interesting to see what the community has been cooking up in terms of UI/UX, Evennia, MUDlet, TUIs, etc.
Fully agree here. I love MUDs. The shear level of creativity that you can have in an MUD is practically infinite, and MUDs are just so fun to play in general. Especially when you start adding in triggers and sound packs and all that
I think this could be solved with bridge bots to make your MUD multiplatform. You could have a dedicated web app, an IRC bridge, a Discord bridge, etc.
Discord-brained users can still use your MUD and contribute to network effects, and you can provide an off-ramp ("I just want to play VarelionScape, maybe I'll just open the dedicated website on my phone instead of going on Discord and digging through the server list").
About 15 years ago, for a while I ran a MUD over AX.25, because MUDs aren't nerdy enough on their own, and amateur radio - even packet radio - isn't nerdy enough on its own.
All three of us that used it found it pretty entertaining.
Very cool! Why reimplement the "moo code", though? Is it "just" for backward compatibility, or did you determine that none of the existing languages can be easily modified to live in a MOO environment?
I started playing on an LP MUD in the late '90s. Over the past 20 years, I tried a few times to implement a similar environment, only with everyone being a "wizard". LPC was also a prototype-based OO language with multiple inheritance, and it supported live development, with code for objects stored in files and dynamically reloadable. I would always hit some kind of blocker, no matter the language I chose for the implementation. It looks like MOOs did (and do) what I wanted, but I just wasn't aware of their existence. :( Maybe if I used MOOs as a model, I'd get better results... but I'm not sure if I want to go all in on DB. I tried looking up how the code is loaded into the DB in the book (https://timbran.org/book), but couldn't find it. Is there an explanation somewhere of how MOOs are bootstrapped from sources, how you can maintain sources in files on disk, and how new definitions are pulled into the DB at runtime?
There's all sorts of reasons, not sure what level of detail you want me to get into. I've got about 30 years of thoughts on this.
For one, I think MOO is kind of shite in lots of ways. But I and others spent lots of time 20+ years ago building alternatives and nobody used them. There's intrinsic value to just picking up and then improving on something that has established history. It gathers an audience of users, has name recognition, and most importantly keeps you responsible from an engineering dev POV because you are forced to focus on delivering a thing that works and you have a way to prove it. As Linux has shown, incremental improvement on a somewhat flawed but relatively popular and known foundation has value. mooR cakes over a lot of the ugly parts of MOO and honestly, I'm pretty proud of it.
Secondly, existing "mainstream" languages are not appropriate, mostly. None of them or their runtimes are really based on a properly sandboxed persistent model. I poked at WASM, and I could write a screed about that, but wrong system. MOO (and successors to it) were built with the idea of lots of people jamming on one shared world, and it shows.
Finally, I have all sorts of ideas on what I think is "better" than MOO, and did eventually open the floodgate this spring and start building out the "dream system" here: https://github.com/timbran-project/mica -- but as you can see from relative star count of one vs the other.. my first point has relvance.
There should be an explanation in the book on how loading works. If not, I'll need to clarify. Documentation writing sucks. But the TLDR is that one of the arguments you pass to the system at start is basically a source directory of ("objdef") human readable / editable source files, and that gets compiled into a persistent binary database. Once that's done, restarts load directly from that DB/image, not source. (But the system can periodically export or "dump" objdef files as well).
Note this is a bit different from classic LambdaMOO, which had no such process. It had a primitive text database that was not human editable really, and you started from what people passed around of those.
Right. I kind of forgot that MUDs are supposed to be social. Seeing as the popularity of MUDs plummeted, I also changed my goals over time: I want to build a persistent, text-based (but with rich formatting support, including eventually GUI widgets) live programmable environment with network transparency and multiple concurrent sessions allowed. Since it's personal/local space first, and only allows others to "visit" (and even that is mostly for my own remote sessions), I don't need many users initially. If I was trying to build something that emphasizes the multi-user aspect, the choice of continuing one of the historical lineages would be much harder to avoid.
> Secondly, existing "mainstream" languages are not appropriate, mostly. None of them or their runtimes are really based on a properly sandboxed persistent model.
You're right. And even if you broaden the search so much that "mainstream" is barely visible from the cut-off point, you're left with basically just Smalltalk. My current attempt is based on GToolkit distribution of Pharo Smalltalk - it does check some boxes, but it's class-based (not a big problem given the dynamism) and VERY single-user by design (quite a big problem, and thinking about eventual security/permissions/capabilities handling gives me a headache). Still, from all the prototypes I made over the years, this attempt went the farthest; I think I'll stick with it for a while longer.
> Finally, I have all sorts of ideas on what I think is "better" than MOO
Thank you, I will definitely check it out :)
> But the TLDR is that one of the arguments you pass to the system at start is basically a source directory of ("objdef") human readable / editable source files, and that gets compiled into a persistent binary database.
Thanks. That's exactly how a Smalltalk image is built, too, at least with the tooling I'm using (GT). You start an image, point it to a set of Git repos with Smalltalk code, which is then loaded into the image, classes are created, and methods get compiled. You then dump the new image. When you start it the next time, it already contains all the code, and you can change it inside the image at runtime, as much as you want. At some point, you export the changed code back to Git repos, and regenerate the image. Since the image contains all the live objects as well, you can close it, and all the state is restored when you reopen it.
Thanks for the Discord invite, I just joined. I'll go over the book and try to run mooR and mica first, and will reach out for help if needed.
For mica specifically: I'm not 100% sure, but I seem to remember you describing the idea behind it much earlier (here on HN) than this spring. I think it was in comments under a Python-Prolog bridge or something like that. Good to see you had the time and motivation to implement it :) I'll definitely give it a read!
If you ask an LLM to solve a multiplication problem using reasoning without code tools, depending on the model, it will get 15 digit (15D) * 15 digit multiplication correct (123456789012345 x 998765432109876). It will take between 4000 and 8000 tokens if Sonnet. Eventually with enough digits, it will start only solving the problem 80% ... then 70% of the time until it has so many digits it will never solve. There will be a certain number of digits where it will not converge on a solution nor will it stop working thinking it can solve it. That is very, very expensive. It takes a lot of runs to determine the probability it will solve it.
What you can do now, this is likely the most important thing, is change the prompt and evaluate how many tokens and at what speed it takes to accomplish the task. Sure the measurements of each LLM are important! Nonetheless, if you can say to a company that you have a technique to tune prompts and evaluate them so instead of spending $1 X 100,000 times a day, they can instead spend $0.90 X 100,000 times a day, you will make a ton of money.
Withered technology - that's a nice turn of phrase. I think I'll start using that to refer to some of the 2000s era MS tech we're saddled with that gets a coat of paint and some KBs to cover CVEs but ultimately is rather withered...
I was a bit afraid that MUD was an acronym for something else, I clicked in about 10% hope that this will be for multi user dungeons and behold, thank you. Interesting read.
That brought back memories when we were trying to find free terminal in few departments at our uni to get access to one that has internet connectivity.
+1 to using MUD as a learning tool. Learn about agent loops, prompting iteration, etc.
I vibe coded a few room proof of concept MUD. I added a tools called `oracle` (aka me) that it could ask me questions to help along its way; and the ability to interject in the loop with a hint. Just as I might want an llm to stop to ask me for help instead of plodding along...
Interesting to limit the number of turns and see where it gets stuck or how quickly it can finish.
Not Hitchhiker's specifically, we'd considered another existing one but as soon as we started digging into IP law related to MUDs, we decided if we're going to do it, we'll just start our own from scratch.
that's funny. the very first thing i had an llm do was to ask it to pretend to be the HHGTTG infocom game. Can't remember how it did.. not amazingly but was interesting still.
Thanks! Yeah, it started with a friend's idea to take a MUD we played in the 90s / early 2000s that had been open sourced and adapt it to mobile games. He's a middle school teacher and wanted to introduce the next generations to MUDs.
We then decided to pivot and thought, what if this would be a good eval environment for LLMs from the simplistic standpoint of, text is their native environment.
That's fair. We leaned on LLMs to help draft parts of the paper but disclosed it in the acknowledgements section. We didn't appreciate the amount of work this "hobby" experiment would require, and so looked for ways to help get this out the door.
A little irrelevant, but I need to vent about something MUD adjacent.
I really wish MUDs were still a thing. The text-based roleplaying communities were largely swallowed by Discord, where as before they existed scattered over proboards, jcink, and a few others.
Discord, being the imperfect text platform, limits the text and leaves most interaction limited by awkward formatting. To boot, even when a better ecosystem is invented, players refuse to give it a shot even if directly invited, because they got comfortable with Discord's terribleness.
It's so frustrating, and I don't know how to voice it, or even what can be done.
MUDs never disappeared. People stopped finding them engaging enough to use. Blaming Discord reverses the causality. MUDs lost engagement long before disco was created. People congregate on disco because it offers a more compelling social interface, even if it lacks the depth of a persistent game world. The opportunity I'd draw from this isn’t to convince people to return to older and less engaging interfaces. It’s to build persistent roleplaying systems around the interaction modes that have proven to be more engaging already. Potentially combine voice, speech recognition, text, tts, and synthesized characters with a proper world simulation underneath. If roleplaying communities already gather in disco, then deliver the experience where they already are. The interface was always irrelevant to what made MUDs fun anyway. Imagination and connecting with other people was always the thing.
I specifically outlined non-MUD role playing environments because I know MUDs lost popularity long before Discord. I am outlining that any form of tech, besides Discord, is sidelines -- MUDs and a hopeful resurgence as well.
I agree with you. You've also got more / easier competition. IIRC, MUDs peaked in the mid-late 90s. MMOs were really the death knell, then the explosion of other types of games, mobile games, then to your point things like Discord.
One adjacent project we're exploring for the future is if you took all of the current understanding of game design and mechanics with modern AI functionality, knowledge bases, etc, and built a MUD, what would it look like and would it be enough to bring back players from other games?
There is an ongoing project doing this (that launches this weekend I believe) to revive a MUD called Shadows of Isildur. I think a lot of effort has been put into the new changes to the codebase to bring QoL features that a modern audience would expect while keeping the bones of the game the same. I'm not certain how it will fare as, to be honest, MUDs take an inordinate amount of time to really groove into and immerse yourself in - but I am excited to see efforts like this invested on as we older MUDers find ourselves finally reaching the point of having free time again.
Would be kind of a cool experiment for a mud-style interface, where each npc/avatar was backed by a lower-cost LLM... similar rules for the npcs in the game, but a background feed, and history of interactions with other players as background.
Maybe limiting npc's to only a certain number of moves that aren't a response to other users per day...
It could be a lot of fun.
A MUD exists as a platform for creative writing - introducing LLMs into one would deprive it of its primary function.
Most people played MUDs for the grind. Levels and loot. They care about the story as much as you care about it in Path of Exile.
The people who played MUDs for the grind will, genuinely, have a lot more fun playing PoE or some other modern adventure game. MUDs were the only option for that sort of adventure experience for a while but there are other options now that simply have more tools at their disposal than text based communication offers.
If a MUD is going to survive today it's going to be driven by those RP focused players where the freedom to describe actions in text allows a flexibility that's unrealistic in graphical games.
I honestly worry that the relevance of text-based systems is largely tied with literacy and how it interacts with the player's imagination -- something largely in decline, at least in the US.
Sadly, you're not wrong... It might not get off the ground, but if we're able to, we'll give it a shot.
I really, really hope you lot manage to!
Simutronics is has been running and developing Gemstone and DragonRealms for 30+ years and counting. The communities are about 400-1000 players each. They've embraced microtransactions (in the form of quarterly events), but both games are entirely playable without participating in that aspect.
With modern technology, training in both games is very automatable. But developing and optimizing your training routines is (in my opinion) incredibly fun. Actual roleplaying is less common it was in the 90s, but there are still nightly events where dozens of players will get together to socialize and explore the worlds in-character.
I highly recommend it, and for those who don't have the time to grind, you can join the shattered server where 24/7 scripting is allowed and encouraged. F2P subscriptions don't work on that server, unfortunately, but it has been a great hobby for someone like me who used to play back in the day but could never play it without scripting due to time constraints.
MUDs are still kicking around and if you'd like to play in one you'll find a few solid communities with good engagement that would love to include you. They're in an odd spot though since MUDs existed in two sorts of categories 1. I really want to adventure and text is the only way to do that (these sorts died off to MMOs) and 2. I want to roleplay - those survive but the mechanical portions of those games are becoming more and more irrelevant as new games are constantly supplying better systems and mechanisms than MUDs were able to.
This leaves MUDs in a weird place since things like MUSHes generally have most of the roleplaying tools without as much weird code for attacks and PVP interactions being involved so I've seen those thrive more recently.
The MUD I play (Aardwolf) averages about 200 players online at any time, but it's effectively dead as a MUD. There's very little player interaction, which is what made MUDs special originally. Part of this is that nearly all multiplayer features have been abandoned, and the rest of the features are centered around loot box equivalent features. Despite this, there are a couple hundred people who seem to enjoy the gamification. The MUD is highly optimized for scripting, so it scratches the side project programming itch for me.
I have a pile of ideas and not enough time to get to them all, but one of them is the opposite direction. Embrace Discord, and build a MUD whose social systems are funneled through it.
Technically MUDs still are a thing, there's a good number still running. Though sounds like you don't really just want a MUD where you want to go out mobbing but a more social thing (but not IRC?).
I suspect MUDs may have a bit of a resurgence soon, I've seen more and more interest in them as of late simply because they seem like an excellent pairing with LLMs which easily consume text. Though I think for your needs someone would need to work on the barrier to entry to getting on a MUD as well as the modern UI/UX expected today.
Having stepped away from MUDs ourselves for years, it's been interesting to see what the community has been cooking up in terms of UI/UX, Evennia, MUDlet, TUIs, etc.
Fully agree here. I love MUDs. The shear level of creativity that you can have in an MUD is practically infinite, and MUDs are just so fun to play in general. Especially when you start adding in triggers and sound packs and all that
> even what can be done.
I think this could be solved with bridge bots to make your MUD multiplatform. You could have a dedicated web app, an IRC bridge, a Discord bridge, etc.
Discord-brained users can still use your MUD and contribute to network effects, and you can provide an off-ramp ("I just want to play VarelionScape, maybe I'll just open the dedicated website on my phone instead of going on Discord and digging through the server list").
That's a good idea, thanks! We'd only considered the web app + using a client.
discworld mud (any many others) ase still totally worth playing
> I really wish MUDs were still a thing.
About 15 years ago, for a while I ran a MUD over AX.25, because MUDs aren't nerdy enough on their own, and amateur radio - even packet radio - isn't nerdy enough on its own.
All three of us that used it found it pretty entertaining.
You would probably like my project https://github.com/timbran-project/moor
Very cool! Why reimplement the "moo code", though? Is it "just" for backward compatibility, or did you determine that none of the existing languages can be easily modified to live in a MOO environment?
I started playing on an LP MUD in the late '90s. Over the past 20 years, I tried a few times to implement a similar environment, only with everyone being a "wizard". LPC was also a prototype-based OO language with multiple inheritance, and it supported live development, with code for objects stored in files and dynamically reloadable. I would always hit some kind of blocker, no matter the language I chose for the implementation. It looks like MOOs did (and do) what I wanted, but I just wasn't aware of their existence. :( Maybe if I used MOOs as a model, I'd get better results... but I'm not sure if I want to go all in on DB. I tried looking up how the code is loaded into the DB in the book (https://timbran.org/book), but couldn't find it. Is there an explanation somewhere of how MOOs are bootstrapped from sources, how you can maintain sources in files on disk, and how new definitions are pulled into the DB at runtime?
There's all sorts of reasons, not sure what level of detail you want me to get into. I've got about 30 years of thoughts on this.
For one, I think MOO is kind of shite in lots of ways. But I and others spent lots of time 20+ years ago building alternatives and nobody used them. There's intrinsic value to just picking up and then improving on something that has established history. It gathers an audience of users, has name recognition, and most importantly keeps you responsible from an engineering dev POV because you are forced to focus on delivering a thing that works and you have a way to prove it. As Linux has shown, incremental improvement on a somewhat flawed but relatively popular and known foundation has value. mooR cakes over a lot of the ugly parts of MOO and honestly, I'm pretty proud of it.
Secondly, existing "mainstream" languages are not appropriate, mostly. None of them or their runtimes are really based on a properly sandboxed persistent model. I poked at WASM, and I could write a screed about that, but wrong system. MOO (and successors to it) were built with the idea of lots of people jamming on one shared world, and it shows.
Finally, I have all sorts of ideas on what I think is "better" than MOO, and did eventually open the floodgate this spring and start building out the "dream system" here: https://github.com/timbran-project/mica -- but as you can see from relative star count of one vs the other.. my first point has relvance.
There should be an explanation in the book on how loading works. If not, I'll need to clarify. Documentation writing sucks. But the TLDR is that one of the arguments you pass to the system at start is basically a source directory of ("objdef") human readable / editable source files, and that gets compiled into a persistent binary database. Once that's done, restarts load directly from that DB/image, not source. (But the system can periodically export or "dump" objdef files as well).
Note this is a bit different from classic LambdaMOO, which had no such process. It had a primitive text database that was not human editable really, and you started from what people passed around of those.
Hope that helps. Come by the discord if you have questions: https://discord.com/invite/Ec94y5983z
Also I just finally put up a new blog post on dev status https://timbran.org/catching-up-with-moor.html
EDIT: I've done some book edits and added a chapter here which I hope helps: https://timbran.org/book/2.0-dev/html/the-system/bootstrappi...
> building alternatives and nobody used them
Right. I kind of forgot that MUDs are supposed to be social. Seeing as the popularity of MUDs plummeted, I also changed my goals over time: I want to build a persistent, text-based (but with rich formatting support, including eventually GUI widgets) live programmable environment with network transparency and multiple concurrent sessions allowed. Since it's personal/local space first, and only allows others to "visit" (and even that is mostly for my own remote sessions), I don't need many users initially. If I was trying to build something that emphasizes the multi-user aspect, the choice of continuing one of the historical lineages would be much harder to avoid.
> Secondly, existing "mainstream" languages are not appropriate, mostly. None of them or their runtimes are really based on a properly sandboxed persistent model.
You're right. And even if you broaden the search so much that "mainstream" is barely visible from the cut-off point, you're left with basically just Smalltalk. My current attempt is based on GToolkit distribution of Pharo Smalltalk - it does check some boxes, but it's class-based (not a big problem given the dynamism) and VERY single-user by design (quite a big problem, and thinking about eventual security/permissions/capabilities handling gives me a headache). Still, from all the prototypes I made over the years, this attempt went the farthest; I think I'll stick with it for a while longer.
> Finally, I have all sorts of ideas on what I think is "better" than MOO
Thank you, I will definitely check it out :)
> But the TLDR is that one of the arguments you pass to the system at start is basically a source directory of ("objdef") human readable / editable source files, and that gets compiled into a persistent binary database.
Thanks. That's exactly how a Smalltalk image is built, too, at least with the tooling I'm using (GT). You start an image, point it to a set of Git repos with Smalltalk code, which is then loaded into the image, classes are created, and methods get compiled. You then dump the new image. When you start it the next time, it already contains all the code, and you can change it inside the image at runtime, as much as you want. At some point, you export the changed code back to Git repos, and regenerate the image. Since the image contains all the live objects as well, you can close it, and all the state is restored when you reopen it.
Thanks for the Discord invite, I just joined. I'll go over the book and try to run mooR and mica first, and will reach out for help if needed.
For mica specifically: I'm not 100% sure, but I seem to remember you describing the idea behind it much earlier (here on HN) than this spring. I think it was in comments under a Python-Prolog bridge or something like that. Good to see you had the time and motivation to implement it :) I'll definitely give it a read!
That's a cool project, thanks for sharing!
ssh to late.sh
https://news.ycombinator.com/item?id=49001127
> the measurements of each LLM
I think you have the correct idea.
If you ask an LLM to solve a multiplication problem using reasoning without code tools, depending on the model, it will get 15 digit (15D) * 15 digit multiplication correct (123456789012345 x 998765432109876). It will take between 4000 and 8000 tokens if Sonnet. Eventually with enough digits, it will start only solving the problem 80% ... then 70% of the time until it has so many digits it will never solve. There will be a certain number of digits where it will not converge on a solution nor will it stop working thinking it can solve it. That is very, very expensive. It takes a lot of runs to determine the probability it will solve it.
What you can do now, this is likely the most important thing, is change the prompt and evaluate how many tokens and at what speed it takes to accomplish the task. Sure the measurements of each LLM are important! Nonetheless, if you can say to a company that you have a technique to tune prompts and evaluate them so instead of spending $1 X 100,000 times a day, they can instead spend $0.90 X 100,000 times a day, you will make a ton of money.
Withered technology - that's a nice turn of phrase. I think I'll start using that to refer to some of the 2000s era MS tech we're saddled with that gets a coat of paint and some KBs to cover CVEs but ultimately is rather withered...
Haha, thanks. I wish I'd come up with that. It's from Gunpei Yokoi of Nintendo fame.
I was a bit afraid that MUD was an acronym for something else, I clicked in about 10% hope that this will be for multi user dungeons and behold, thank you. Interesting read.
That brought back memories when we were trying to find free terminal in few departments at our uni to get access to one that has internet connectivity.
+1 to using MUD as a learning tool. Learn about agent loops, prompting iteration, etc.
I vibe coded a few room proof of concept MUD. I added a tools called `oracle` (aka me) that it could ask me questions to help along its way; and the ability to interject in the loop with a hint. Just as I might want an llm to stop to ask me for help instead of plodding along...
Interesting to limit the number of turns and see where it gets stuck or how quickly it can finish.
Speaking of MUD, has anyone tried attaching an AI to the Hitchhikers MUD[1]?
[1]: https://www.bbc.co.uk/programmes/articles/1g84m0sXpnNCv84GpN...
This is a text-based game, but it is not a MUD. The MU in MUD stands for multi-user.
Not Hitchhiker's specifically, we'd considered another existing one but as soon as we started digging into IP law related to MUDs, we decided if we're going to do it, we'll just start our own from scratch.
that's funny. the very first thing i had an llm do was to ask it to pretend to be the HHGTTG infocom game. Can't remember how it did.. not amazingly but was interesting still.
That’s a fun design that brings back some old memories.
Thanks! Yeah, it started with a friend's idea to take a MUD we played in the 90s / early 2000s that had been open sourced and adapt it to mobile games. He's a middle school teacher and wanted to introduce the next generations to MUDs.
We then decided to pivot and thought, what if this would be a good eval environment for LLMs from the simplistic standpoint of, text is their native environment.
"We did not choose a MUD because it is charming. We chose it because its constraints make behavior measurable." I so fucking hate these LLMisms
That's fair. We leaned on LLMs to help draft parts of the paper but disclosed it in the acknowledgements section. We didn't appreciate the amount of work this "hobby" experiment would require, and so looked for ways to help get this out the door.