I like this, but unfortunately it doesn't solve one annoying problem: lexical scope doesn't work and it will fail in an unexpected way.
If you reference something lexically, your code fails at runtime. Want to use an import? You have to use import() inside the closure you pass to spawn(). Typescript doesn't know this. Your language server doesn't know this. Access a variable that shadows a built in global? Now you're accessing the built in global.
The only way this could even be addressed is by having a full on parser. Even then you can't guarantee things will work.
I think the only "fix" is for JS to introduce a new syntax to have a function that can't access lexical scope, returning a value that either extends a subclass of Function or has a cheeky symbol set on it. At least then, it'll fail at compile time.
A linter rule provided by the library could be helpful here. I know it's just a workaround but probably easier than going for a solution that does compile time checks.
This should be the expected behavior when multithreading. It is the expected behavior when executing a child process, such as node’s child_process.fork.
Fork, and normal worker threads always enter a script, there's clearly no shared lexical scope. This spawn method executes a function, but that fn can't interact with the scope outside
While I agree with GP that this should be the expected behavior, your comment raises what I think is a large problem/wild-goose-chase in ‘modern’ language designs implementing concurrency.
The push from language designers (this applies across the high/low level spectrum and at all ranges of success for languages) to make concurrent code ‘look just like’ linearly read, synchronous, single-threaded code is pervasive and seems to avoid large pushback by users of the language. The complaints that should be made against this syntax design become complaints that code doesn’t do what developers think it should.
My position is that concurrent (and parallel) code IS NOT sequential code and languages should embrace those differences. The move to or design of async/await is often explicitly argued for from this position. But the semantic differences in concurrent code IMO should not be obscured or obfuscated by seeking to conform that code to sequential code’s syntax.
I’d love a way to be able to specify that sort of thing. I wrote a little server-side JSX rendering layer, and event handlers were serialized to strings, and so they had similar restrictions.
This looks great. If it works as well as the readme suggests, this’ll let me reach for Bun in some of the scenarios where I currently reach for Go. Typescript has become my favorite language, but the lack of efficient multithreading is sometimes a deal breaker.
(I suspect, to paraphrase Greenspun's rule, any sufficiently complicated app using Web Workers contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of this library...)
Documentation here is exceptionally well written for a JS project, although move() doing different things depending on the type of data you pass to it feels like a foot-gun, and also how is it blocking access to arrays you pass to it?
From an overall system point of view, this is the current pinnacle of footgun design.
The OS does thread management and scheduling, facilitates IPC, locking, etc. All of this is one big largely-solved problem (at least for the kind of things most people are doing in JavaScript today). But because of history, we now have a very popular language and runtimes that are trying to replicate all these features, reinventing wheels, and adding layers on inefficiency to the overall execution.
I don’t disagree with you about the additional inefficiency that is very likely to accumulate as JS adds more and more ‘features’ (via the language, frameworks, or libraries). But as a genuine question, isn’t this reimplementation (or any comparable library for multithreading) required by JavaScript’s position on sandboxing. I would be suspicious of intent if browsers were allowed to spawn any number of threads to execute non-trusted scripts at the level typically seen from more native application code.
I like this, but unfortunately it doesn't solve one annoying problem: lexical scope doesn't work and it will fail in an unexpected way.
If you reference something lexically, your code fails at runtime. Want to use an import? You have to use import() inside the closure you pass to spawn(). Typescript doesn't know this. Your language server doesn't know this. Access a variable that shadows a built in global? Now you're accessing the built in global.
The only way this could even be addressed is by having a full on parser. Even then you can't guarantee things will work.
I think the only "fix" is for JS to introduce a new syntax to have a function that can't access lexical scope, returning a value that either extends a subclass of Function or has a cheeky symbol set on it. At least then, it'll fail at compile time.
A linter rule provided by the library could be helpful here. I know it's just a workaround but probably easier than going for a solution that does compile time checks.
This should be the expected behavior when multithreading. It is the expected behavior when executing a child process, such as node’s child_process.fork.
Fork, and normal worker threads always enter a script, there's clearly no shared lexical scope. This spawn method executes a function, but that fn can't interact with the scope outside
While I agree with GP that this should be the expected behavior, your comment raises what I think is a large problem/wild-goose-chase in ‘modern’ language designs implementing concurrency.
The push from language designers (this applies across the high/low level spectrum and at all ranges of success for languages) to make concurrent code ‘look just like’ linearly read, synchronous, single-threaded code is pervasive and seems to avoid large pushback by users of the language. The complaints that should be made against this syntax design become complaints that code doesn’t do what developers think it should.
My position is that concurrent (and parallel) code IS NOT sequential code and languages should embrace those differences. The move to or design of async/await is often explicitly argued for from this position. But the semantic differences in concurrent code IMO should not be obscured or obfuscated by seeking to conform that code to sequential code’s syntax.
I’d love a way to be able to specify that sort of thing. I wrote a little server-side JSX rendering layer, and event handlers were serialized to strings, and so they had similar restrictions.
This looks great. If it works as well as the readme suggests, this’ll let me reach for Bun in some of the scenarios where I currently reach for Go. Typescript has become my favorite language, but the lack of efficient multithreading is sometimes a deal breaker.
Exactly my thoughts. The only incompatibility with Bun is the unavailability of the `using` keyword:
> If you are using Bun (which doesn't natively support using and uses a transpiler which is incompatible with this library)...
I skimmed the issues but I couldn't find any issues on Bun regarding this except for: https://github.com/oven-sh/bun/discussions/4325
I'm confused why drop() is a function that you have to import inside the closure instead of a method.
This is incredible! The SharedJsonBuffer got me all excited!
Writing module bundlers in Javascript had diminishing returns from multi threading because of the overhead of serializing and deserializing ASTs.
I wonder how far something like this would push the ceiling. Would love to see some benchmarks of this thing hauling ASTs around.
This seems very much worth a look!
(I suspect, to paraphrase Greenspun's rule, any sufficiently complicated app using Web Workers contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of this library...)
The implementation of the shared json buffer is nuts
Documentation here is exceptionally well written for a JS project, although move() doing different things depending on the type of data you pass to it feels like a foot-gun, and also how is it blocking access to arrays you pass to it?
It's part of the web platform: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
From an overall system point of view, this is the current pinnacle of footgun design.
The OS does thread management and scheduling, facilitates IPC, locking, etc. All of this is one big largely-solved problem (at least for the kind of things most people are doing in JavaScript today). But because of history, we now have a very popular language and runtimes that are trying to replicate all these features, reinventing wheels, and adding layers on inefficiency to the overall execution.
Sigh.
I don’t disagree with you about the additional inefficiency that is very likely to accumulate as JS adds more and more ‘features’ (via the language, frameworks, or libraries). But as a genuine question, isn’t this reimplementation (or any comparable library for multithreading) required by JavaScript’s position on sandboxing. I would be suspicious of intent if browsers were allowed to spawn any number of threads to execute non-trusted scripts at the level typically seen from more native application code.
This is cool! Hope we can get multi-threaded wasm some time soon.