Author here. I’m happy to see one of my projects on Hacker News. This has been a fun one. One evening you just try to disassemble it and wonder where the code is. The following months were a truly satisfying experience, reverse-engineering this diamond.
There is still a functioning Forth interpreter implemented in the game. If they hadn’t removed all the word names, it would have been possible to debug at any time and analyze the program state and call functions live with real Forth code. Some crazy feature at that time.
Some early source code snippets of the game have even survived and can still be found on archive.org [0]. By today’s standards, they’re almost unreadable.
A fantastic read, and really interesting to see the use of Forth. I remember Forth having a bit of popularity in the 80s. This was such an amazing game, especially in that you felt like the world was huge with the encouragement to just explore.
The other game this reminds me of is a game for the TI99/4a called Tunnels of Doom. It was a cartridge game that also had a floppy or cassette data load. It had a dynamic dungeon creation so every time you played the game you got a new unique experience. That would be an equally challenging one to reverse engineer due to the oddity of the GROM/GPL architecture in the TI99/4a.
Hard to convey how effective Starflight’s game design was within the limits of the day.
The embedding of the story within what was almost entirely free-form exploration & adventure across a huge galaxy was masterful.
You could feel how close the creators were to the edge of what was possible with the save game system: Basically, the disk was a memory image. As you played the game would rewrite itself, so if you got stuck there was no “reset”. The documentation was emphatic: Only play with a copy of the original disk!!
I loved Starlight but I'm not sure it was procedural world generation. I mean there was a map of stars printed with the game so they weren't changing. There was a small bit of variation in terms of what one found on planets and so on...the key was it felt like an open world because it was big enough and there was nothing stopping you from doing what you liked and when (except resources).
It was procedural at least in the sense that you couldn't store the data for all the planets in memory (or even store it on disk) on the 1980s systems it ran on. So you needed a way to generate the data on the fly.
yeah but you're dismissing the fact that this was just a pregen table of data back then. They made a map based on that, sure, but from that table came... everything else and you can't store all that data on floppy.
Similar techniques apply today. Pregen like 100,000 stars. Give them names and locations in the galaxy, treat them as your "locations of interest" with a seed. The rest can just be another cloud of particles with no interest and if the player visits, you can RNG whatever based on the seed. No two systems can share a seed. They can, however, share a branch.
Procedural generation can use a fixed seed, it's not too uncommon. For instance, Elder Scrolls 2: Daggerfall's map was procedurally generated, but is the same for every player.
They bragged about it being procedural in interviews.
What I was never clear on was the degree of cherry picking they did. There were 800 worlds on something like nine disks, each unique and peppered with minerals and artifacts.
Early in the 2000's one of the original authors of Sfarflight posted a massive dump of information online: design documents and source code. That is the "Technical Articles Saved from Oblivion" link in this repository. Sadly no one seems to have managed to save a full copy! This is one I kick myself about: I browsed this back when it was up but didnt think to download. These days I make a point to request Internet Archives to save things like this if I ever come across it.
About halfway through my reverse-engineering process, I came across those documents and realized that roughly 50% were missing. I searched pretty much the entire web to see if anyone had stored these files elsewhere, but so far I haven’t had any luck.
I'm not sure what you're referring to since the files T.C. Lee posted on that geocities site are the only design documents he (or anyone else) ever released, and those are preserved in the SFFiles.zip found at the oocities mirror linked from the github there. That zip contained only a partial dump of some source code, while that sonic.net page contains a more complete copy (but lacks any kind of design documents whatsoever). Was there something else you were referring to?
I seem to recall browsing what appeared to be the complete source back in the day. I put in a bit of an effort to get it compiling, but it was only released as poorly scanned printouts of the source code and OCR wasn't so good then so the project was bigger than I hoped.
That would be the source code posted to that sonic.net site. It was a mix of raw dumps of FORTH blocks and printouts converted to PDFs. It wasn't the full source code, and there weren't any design documents in there.
If anybody is interested in just playing this game, the Sega Genesis version of this game is arguably the definitive one. It was released later on and took everything in the PC version and amped it up a step. It's amazing and easy to find emulators + the ROM online. I replayed it fairly recently and it definitely held up very well.
It's open world, clue based, and with some amazing plot. Like the article mentions, the ending twist alone makes it worth suffering through your notepad of a million little hints thrown at you throughout the game. It's also the sort of game you can beat in [literally] 5 minutes if you know everything ahead of time. So I would strongly recommend avoiding walkthroughs/spoilers. The whole game is about piecing together the puzzle of what's happening and it's amazingly immersive.
> for this game you can throw the usual tools away...
> The reason is that Starflight was written in Forth
I recently reverse-engineered another game from the 80's, and had a similar issue because it was written with Turbo Pascal.
The different tools didn't like quirks such as data stored inline between the instructions or every library function having a different calling convention.
Turns out the simplest way to handle this was to write my own decompiler, since Turbo Pascal 3 didn't do any optimisations.
> There is no set path, allowing players to switch freely between mining, ship-to-ship combat, and alien diplomacy. The broader plot of the game emerges slowly
This reminds me of Star Control 2, aka The Ur-Quan Masters
> The game influenced the design of numerous other games for decades after its release.
And in SC2 wiki page I see this:
> Once Reiche and Ford conceived Star Control 2, they would draw large inspiration from Starflight.
Never played Starflight before but seems right up my alley as SC2 is one of my favorite games
Edit:
Just finished reading the rest of the readme. Very cool! I honestly knew nothing about Forth (just that it was a programming language) and now I want to play around with it.
Wow, so cool! I did not know Starflight was written in Forth.
Starflight was my first experience with game patching. When I got my copy it would not run on my PC. I believe it was because I had an EGA graphics adapter. I wrote a letter explaining my problem and mailed it to Electronic Arts. They mailed a letter back. It said there should be a program on my computer named debug.com, and gave instructions about how to load the Starflight executable in it, replace a couple of bytes, and save it out. I followed the patch instructions and it got the game working! After which I spent many happy hours with my friends exploring the Starflight universe.
Starflight is the great grandfather of No Man’s Sky, and the first game I availed myself of using Internet databases to find obscure facts about the game. There was no website. It was some group of people curating a text document that was reposted to Usenet periodically with additions and corrections.
I don’t know when I started the game, but I finished it the summer after my freshman year, which is where I first got Internet access.
Awesome! I got this game for Christmas in 3rd grade and was completely hooked! It really captured my imagination and immersed me into this seemingly limitless world. It took me a couple of years to complete, and I kept of journal of exploration experiences.
Probably worth mentioning that writing a big project in Forth is more like creating an OOP framework.(if you are disciplined)
The end result of that is one doesn't write the program in "Forth" per se but in the domain specific language you create for the job. This is how Forth gets more productive than pure assembly language if the team documents things well and follows the prescribed system.
This was the era before optimizing compilers.[1] The overwhelming majority of commercial games were shipping hand-coded assembly still. Forth had the advantage of low overhead, no-worse-than-a-compiler speed, and better-than-assembly productivity. It was a small time window, but a good fit in the moment.
[1] Non-trivial optimizations were just starting to show up on big systems, but Microsoft C in 1985 was still a direct translator.
...Forth? Wow. I wonder how much code change was necessary between the various systems. It's hard to imagine a Megadrive Forth compiler, but then again, the game was on several other M68k systems so maybe it wasn't as hard...
It is really, really easy to write a Forth interpreter (You can write a simple one in an afternoon). It's often the first software written for an architecture. The structure of Forth means that the hardware-dependent parts are contained in a small number of words (sort of like functions in other languages but not exactly). Forth can be implemented on tiny microcontrollers; a Megadrive would be luxury.
It's.. not a compiler (besides I had Forth on my C64). Maybe one can call it a translator to ad-hoc bytecode. I also had USCD Pascal on that C64 which translated to bytecode. This was more JVM-like. So nothing hard about it.
Forth is a compiler but what it "compiles" is not standard. The implementor decides what they need.
Forth can compile pointers to native code that are the VM's instructions, called direct threading.
Forth can compile pointers to pointers to native code VM instructions, called indirect threading.
Forth can compile byte code like OpenFirmware/OpenBoot.
And modern systems compile optimized native code (VFX Forth, SwiftForth) but still remain fully interactive at the console.
Point of information: By and large FORTHs did not use bytecode. Execution tokens (XTs) were usually stored as a function of the default word size, which typically was 16 bits. There were some FORTHs that went out of their way to use token threading so they could store programs in byte codes, but those were pretty rare. Rarer still were programs that mixed byte code with word-sized code (although one such scheme is described in an issue of Forth Dimensions).
Author here. I’m happy to see one of my projects on Hacker News. This has been a fun one. One evening you just try to disassemble it and wonder where the code is. The following months were a truly satisfying experience, reverse-engineering this diamond.
There is still a functioning Forth interpreter implemented in the game. If they hadn’t removed all the word names, it would have been possible to debug at any time and analyze the program state and call functions live with real Forth code. Some crazy feature at that time.
Some early source code snippets of the game have even survived and can still be found on archive.org [0]. By today’s standards, they’re almost unreadable.
[0] https://web.archive.org/web/20030906124225/http://www.sonic....
"By today's standards"? no that's just FORTH code
There is a VM project now using the reversing work as a basis: https://git.information-superhighway.net/SpindleyQ/starfligh...
Microblog: https://gamemaking.social/@SpindleyQ/115058149348319018
A fantastic read, and really interesting to see the use of Forth. I remember Forth having a bit of popularity in the 80s. This was such an amazing game, especially in that you felt like the world was huge with the encouragement to just explore.
The other game this reminds me of is a game for the TI99/4a called Tunnels of Doom. It was a cartridge game that also had a floppy or cassette data load. It had a dynamic dungeon creation so every time you played the game you got a new unique experience. That would be an equally challenging one to reverse engineer due to the oddity of the GROM/GPL architecture in the TI99/4a.
Hard to convey how effective Starflight’s game design was within the limits of the day.
The embedding of the story within what was almost entirely free-form exploration & adventure across a huge galaxy was masterful.
You could feel how close the creators were to the edge of what was possible with the save game system: Basically, the disk was a memory image. As you played the game would rewrite itself, so if you got stuck there was no “reset”. The documentation was emphatic: Only play with a copy of the original disk!!
Everyone goes on about Elite, but Elite was just a sandbox… (an amazing sandbox)
This was crazy! World gen is hard. Proc world gen is NP hard. Story driven proc world gen with persistence in 1986 was Kaku level brilliance.
I loved Starlight but I'm not sure it was procedural world generation. I mean there was a map of stars printed with the game so they weren't changing. There was a small bit of variation in terms of what one found on planets and so on...the key was it felt like an open world because it was big enough and there was nothing stopping you from doing what you liked and when (except resources).
It was procedural at least in the sense that you couldn't store the data for all the planets in memory (or even store it on disk) on the 1980s systems it ran on. So you needed a way to generate the data on the fly.
yeah but you're dismissing the fact that this was just a pregen table of data back then. They made a map based on that, sure, but from that table came... everything else and you can't store all that data on floppy.
Similar techniques apply today. Pregen like 100,000 stars. Give them names and locations in the galaxy, treat them as your "locations of interest" with a seed. The rest can just be another cloud of particles with no interest and if the player visits, you can RNG whatever based on the seed. No two systems can share a seed. They can, however, share a branch.
Procedural generation can use a fixed seed, it's not too uncommon. For instance, Elder Scrolls 2: Daggerfall's map was procedurally generated, but is the same for every player.
They bragged about it being procedural in interviews.
What I was never clear on was the degree of cherry picking they did. There were 800 worlds on something like nine disks, each unique and peppered with minerals and artifacts.
O cool! This approach was used in WASTELAND too, on four floppies (the progenitor of the FALLOUT series).
Early in the 2000's one of the original authors of Sfarflight posted a massive dump of information online: design documents and source code. That is the "Technical Articles Saved from Oblivion" link in this repository. Sadly no one seems to have managed to save a full copy! This is one I kick myself about: I browsed this back when it was up but didnt think to download. These days I make a point to request Internet Archives to save things like this if I ever come across it.
About halfway through my reverse-engineering process, I came across those documents and realized that roughly 50% were missing. I searched pretty much the entire web to see if anyone had stored these files elsewhere, but so far I haven’t had any luck.
I'm not sure what you're referring to since the files T.C. Lee posted on that geocities site are the only design documents he (or anyone else) ever released, and those are preserved in the SFFiles.zip found at the oocities mirror linked from the github there. That zip contained only a partial dump of some source code, while that sonic.net page contains a more complete copy (but lacks any kind of design documents whatsoever). Was there something else you were referring to?
I seem to recall browsing what appeared to be the complete source back in the day. I put in a bit of an effort to get it compiling, but it was only released as poorly scanned printouts of the source code and OCR wasn't so good then so the project was bigger than I hoped.
That would be the source code posted to that sonic.net site. It was a mix of raw dumps of FORTH blocks and printouts converted to PDFs. It wasn't the full source code, and there weren't any design documents in there.
You can see on archive.org where it captured directory info for files which were not stored:
https://web.archive.org/web/20031218202130/http://www.sonic....
When you navigate those directory archives many of the files are missing. E.g.
https://web.archive.org/web/20030529230704/http://www.sonic....
Some design documents ARE there, e.g. "Specification of 3d display for starquest" : https://web.archive.org/web/20030719111039if_/http://www.son...
Well....... shit. I'd been in there previously and never noticed that. Disregard.
If anybody is interested in just playing this game, the Sega Genesis version of this game is arguably the definitive one. It was released later on and took everything in the PC version and amped it up a step. It's amazing and easy to find emulators + the ROM online. I replayed it fairly recently and it definitely held up very well.
It's open world, clue based, and with some amazing plot. Like the article mentions, the ending twist alone makes it worth suffering through your notepad of a million little hints thrown at you throughout the game. It's also the sort of game you can beat in [literally] 5 minutes if you know everything ahead of time. So I would strongly recommend avoiding walkthroughs/spoilers. The whole game is about piecing together the puzzle of what's happening and it's amazingly immersive.
Generally a game that was way ahead of its time.
> for this game you can throw the usual tools away...
> The reason is that Starflight was written in Forth
I recently reverse-engineered another game from the 80's, and had a similar issue because it was written with Turbo Pascal.
The different tools didn't like quirks such as data stored inline between the instructions or every library function having a different calling convention.
Turns out the simplest way to handle this was to write my own decompiler, since Turbo Pascal 3 didn't do any optimisations.
> There is no set path, allowing players to switch freely between mining, ship-to-ship combat, and alien diplomacy. The broader plot of the game emerges slowly
This reminds me of Star Control 2, aka The Ur-Quan Masters
> The game influenced the design of numerous other games for decades after its release.
And in SC2 wiki page I see this:
> Once Reiche and Ford conceived Star Control 2, they would draw large inspiration from Starflight.
https://en.wikipedia.org/wiki/Star_Control_II
Never played Starflight before but seems right up my alley as SC2 is one of my favorite games
Edit: Just finished reading the rest of the readme. Very cool! I honestly knew nothing about Forth (just that it was a programming language) and now I want to play around with it.
Wow, so cool! I did not know Starflight was written in Forth.
Starflight was my first experience with game patching. When I got my copy it would not run on my PC. I believe it was because I had an EGA graphics adapter. I wrote a letter explaining my problem and mailed it to Electronic Arts. They mailed a letter back. It said there should be a program on my computer named debug.com, and gave instructions about how to load the Starflight executable in it, replace a couple of bytes, and save it out. I followed the patch instructions and it got the game working! After which I spent many happy hours with my friends exploring the Starflight universe.
You can read more about this gem of a game upon the digital antiquarian:
https://www.filfre.net/2014/10/starflight/
Starflight is the great grandfather of No Man’s Sky, and the first game I availed myself of using Internet databases to find obscure facts about the game. There was no website. It was some group of people curating a text document that was reposted to Usenet periodically with additions and corrections.
I don’t know when I started the game, but I finished it the summer after my freshman year, which is where I first got Internet access.
One of my all-time favourite games. A family member had Starflight on Genesis. I still pull it up in an emulator from time to time.
Random edit: It's also how I learned the word obsequious.
[delayed]
lol! Me too! I remember the alien interaction dialog and this word.
I think we will see more and more games and applications be fully reversed engineered in the coming month due to increase coding agent capabilities.
Awesome! I got this game for Christmas in 3rd grade and was completely hooked! It really captured my imagination and immersed me into this seemingly limitless world. It took me a couple of years to complete, and I kept of journal of exploration experiences.
Thank you for posting this!
> Back in the 80ths, an unknown company called Binary Systems published the game Starflight.
80s - 80ths is a fraction (1/80).
Also - the publisher was Electronic Arts, already well-established at that point. Binary Systems were the developers.
Loved this game as a young kid. Great to see it. Somewhere I also found a web implementation of that black and white codewheel that came with it.
I find it curious that the game was written in Forth. Certainly a very unusual choice for a commercial game.
Probably worth mentioning that writing a big project in Forth is more like creating an OOP framework.(if you are disciplined)
The end result of that is one doesn't write the program in "Forth" per se but in the domain specific language you create for the job. This is how Forth gets more productive than pure assembly language if the team documents things well and follows the prescribed system.
This was the era before optimizing compilers.[1] The overwhelming majority of commercial games were shipping hand-coded assembly still. Forth had the advantage of low overhead, no-worse-than-a-compiler speed, and better-than-assembly productivity. It was a small time window, but a good fit in the moment.
[1] Non-trivial optimizations were just starting to show up on big systems, but Microsoft C in 1985 was still a direct translator.
This is the way. Best thing since the reincarnation of Wall Street Raider.
This is my all time favorite DOS game. It felt open world before people used that phrase. Amazing game.
I believe this is only the second game I ever finished. But also the first I used the internet to finish.
...Forth? Wow. I wonder how much code change was necessary between the various systems. It's hard to imagine a Megadrive Forth compiler, but then again, the game was on several other M68k systems so maybe it wasn't as hard...
It is really, really easy to write a Forth interpreter (You can write a simple one in an afternoon). It's often the first software written for an architecture. The structure of Forth means that the hardware-dependent parts are contained in a small number of words (sort of like functions in other languages but not exactly). Forth can be implemented on tiny microcontrollers; a Megadrive would be luxury.
It's.. not a compiler (besides I had Forth on my C64). Maybe one can call it a translator to ad-hoc bytecode. I also had USCD Pascal on that C64 which translated to bytecode. This was more JVM-like. So nothing hard about it.
Forth is a compiler but what it "compiles" is not standard. The implementor decides what they need. Forth can compile pointers to native code that are the VM's instructions, called direct threading. Forth can compile pointers to pointers to native code VM instructions, called indirect threading. Forth can compile byte code like OpenFirmware/OpenBoot.
And modern systems compile optimized native code (VFX Forth, SwiftForth) but still remain fully interactive at the console.
Point of information: By and large FORTHs did not use bytecode. Execution tokens (XTs) were usually stored as a function of the default word size, which typically was 16 bits. There were some FORTHs that went out of their way to use token threading so they could store programs in byte codes, but those were pretty rare. Rarer still were programs that mixed byte code with word-sized code (although one such scheme is described in an issue of Forth Dimensions).
This game looked very familiar and after googling around it looks like it is recognized as the progenitor of Starcontrol game.