Science should be more like this, in current times, yes.
But until much of academia is burned to the ground, or until science can be properly separated from modern academia, this will never be so. The current academic incentives are all wrong: low-quality research is rewarded and results in publications, whereas high-quality research (that takes time, and usually reveals that most exciting publications depend on p-hacking or other highly data-dependent analyses and selective presentations) is not published or actively blocked during peer review.
So instead you get BS arguments about how data can't be released for various privacy concerns (when in reality the vast majority of most datasets are trivial to scrub of identifying factors, and even in more complex datasets where you need to consider k-anonymity, it is still trivial to release data that allows replication of core analyses), and academic science is increasingly irrelevant unless it is tied to tech and industry, where producing junk actually has real negative economic and personal consequences.
I don't know what world this article / post lives in, but it isn't the messy world of actual reality.
It's changing, though, and articles like this are important. TFA is arguing for change in the future, not presenting this as a fait accompli.
In the 27 years I've been in academia, I've seen a lot of progress in data openness (NCBI GEO was a game-changer) and FOSS analysis software (it's now widely expected that a high impact pub will make all data and code available for review, and then publicly available upon manuscript publication; most major journals will not allow submission without this). It is becoming common for big journals to specifically ask reviewers to review the analysis code. It is starting to become more and more common for papers to release all the code used to generate all the figures (including supplementary figures)
There is still a long way to go, I agree. But it's always better to light candles than curse darkness, etc.
> academic science is increasingly irrelevant unless it is tied to tech and industry
While I have some sympathy with a lot of your bitterness, this statement is insulting silliness that a quick look at the list of Nobel Prizes in physiology and medicine would prove wrong. Almost all major breakthroughs in the applied sphere stem from decades of basic research that happened just because it interested someone.
> it's now widely expected that a high impact pub will make all data and code available for review, and then publicly available upon manuscript publication
I am also in academia and don't think this is true at all for data, not even remotely, unless by "publicly available" you mean "available to most academics or academic labs after an often tedious and slow approval process requiring an academic email and various signed agreements". Maybe what you are saying is true in some very specific domains (e.g. machine learning research), but in general what you are saying here is IMO wildly out of touch with present realities in the vast majority of fields, but especially those involving human subjects.
> While I have some sympathy with a lot of your bitterness, this statement is insulting silliness that a quick look at the list of Nobel Prizes in physiology and medicine would prove wrong. Almost all major breakthroughs in the applied sphere stem from decades of basic research that happened just because it interested someone.
Nobel Prizes are so rare they don't speak at all to the generalizations I am making here. Also, much medical academic research is successful because it is in fact ultimately industry-funded. It is of course though highly dependent on the academic subfield, for sure, and I was painting with a broad brush.
If I had to narrow things, STEM academic research isn't so bad, so long as we exclude social science from STEM. Much social science research needs to be defunded ASAP. And I'm not claiming industry research doesn't also have warped incentives. But, on balance, I'd wager outside of pure math/physics and certain more algorithmic/pure domains in comp sci, the smartest people today are going to choose (and be found in) industry, not academia.
Hear hear! There are so many obvious improvements to how almost everything is done. For instance, in medicine review articles as a class of articles largely represent a giant waste of time. RCTs flatten all their gathered data during publishing, summarizing complex trial data, which is gathered but never published, into a few numbers. Then review articles take a bunch of flattened data, discard the articles that don't fit the exact question they are reviewing, and then publish a doubly flattened conclusion. If any of the included articles turn out to have flaws, if treatments change in retrospect, if you are looking for the answer to a slightly different question or you are looking at a different subgroup, then the review is useless and has to be repeated.
All of these tens of thousands of man-hours could be replaced by a few GitHub repos, if only RCTs would just publish their damn data. Then you could just run and rerun the statistics on whatever subgroup you're looking for, instead of combing through decades of review articles answering slightly different questions, looking for the answer between the lines. With LLMs making mining of large scale datasets almost trivial (with the process most likely becoming trustworthy within a few years), the current status quo is looking more and more antiquated.
If you want to be even more radical, hospitals could just publish their data continuously. Of course, it is easy to point to the risks of doing so, but what's often ignored is the benefits. It is hard to overstate just how many medical mysteries a hospital encounters on a daily basis, how much unknown we are navigating in practice. The current norm is that 99.99% of these cases are never published, and are only ever thought about by a small group of people who happened to be at work. Particularly, when someone dies of something no one figured out, it is never published anywhere, because even if you tried it is not interesting reading material for a journal to publish. And no one ever tries because they're scared of being called out for a mistake. A hospital is essentially a continuously running and extremely interesting experiment, where 99.99999% of all results are thrown in the garbage, and the only published data is subject to extreme selection bias.
All of this could be different, and the risks involved are actually quite small in practice. It is easy to automatically anonymize data quite well, but extremely difficult to absolutely guarantee that it is anonymous. And since current ethical norms are extremely averse to any degree of risk, and usually entirely ignore potential benefits, we all suffer for it. It is not entirely unlikely that someone reading this post will one day die because of something that could have been prevented, had things been different.
Yup, strongly agree with all of this, especially the RCT stuff.
This has all been profoundly obvious for at least well over a decade or even two now. A consequence has been that too many serious people are driven away from academia and research, to the detriment of science generally.
I've no idea what to do about all this, because people have voiced obvious and easy solutions for decades, but they are all routinely ignored.
Make analysis code available. Make anonymized or perturbed data available for download without people having to jump through hoops to get it. If you have highly sensitive data, release only the variables or other statistics needed to reproduce core analyses. Don't only do garbage null-hypothesis significance testing or statistical analyses on the full data, also do ML approaches were you have to actually show your analyses replicate on held-out subsets, and report this. Make reviews open (anonymizing as needed) so we can see when biased or incompetent reviewers are blocking good publications. Allow public review (or at least broader academic open review, in some form), since it is no longer defensible to delegate review and decisions to one or two random people that just happen to be emailed and have the time / are on some editorial / review board. Publish null findings / results, if only in minimal forms so we don't waste time and money trying to reproduce garbage. Make articles available and don't charge insane article processing fees or open access fees of thousands of USD (especially since hosting fees are not that crazy, and also because journals don't do any of the formatting work half the time anyway, and make academics or RAs or students do all the typesetting and formatting most of the time, even though now this can all be automated with template files, mostly).
Most of these things are easy to do for the majority of papers, especially in the past 20 years with the internet and modern tech and software. Plenty of frameworks exist already that have done most and/or at least some of these things, but, collectively, academia is decades behind overall.
I agree with the general sentiment, but there's one major caveat. We should implement reproducible programs on top of a virtual machine spec like JVM or WebAssembly rather than replicate the entire environment. It's more practical to do and doesn't push software towards further centralization. Let people use whatever OS and VM implementation they want, or even write their own.
> Every result is instantly reproducible. When you read a paper claiming that a new drug reduces symptoms by 30%, you click a link and watch the exact analysis run in your browser. The data processing, statistical tests, and visualizations execute in seconds using the same environment the authors used—preserved perfectly through reproducible containers.
Particularly the "data availability" and "Availability and peer review of computer code and algorithm".
However, in my limited experience, of trying to reproduce certain scRNA-seq processing pipelines, in practice it's never available as just a Github link. I can understand that some/many researcher's code is not in good shape, so I think it'll be quite a stretch to have this available.
I do think it's laudable though, to try and make it available. It would certainly have been very useful for me in the past.
> modern science is synonymous with open source software.
Another problem with reproducibility is the openness of the underlying data. Many academics are terrified of giving away the golden goose and the software is often useless without the data.
However many scientists do work openly, e.g. The Journal of Open Source Software:
JOSS is great, I’ve both published with and reviewed for them, and enjoyed it more than traditional journals. The process felt more constructive than destructive, in a sense.
I believe they’re always looking for new volunteers to review papers, so please do volunteer if you are able.
> TL;DR I claim that modern science is synonymous with open source software
That's a strong statement that's not supported by the arguments and IMO misguided.
I don't have a problem with "open", but rather with "software".
Both science and software deal with models, however the focus is quite different. I suspect you conflate theory with models.
The goal of science is to produce and test theories — that's an inductive/abductive process. A model, regardless of whether it's reified into mathematical formulas or software, is a means of making a theory operational enough that its consequences can be derived and confronted with observations.
Software often starts downstream of this: it's a reification of theories, models, algorithms, or findings that are the result of research. Of course software can also be used as part of the research process itself. The distinction is roughly the familiar one between research and development.
I think it is worse than that. Science is a process for resolving disagreements, ambiguity, and uncertainty, and also for discovering abstractions (patterns) among phenomena. It is social and can not be reduced to a binary / digital file, as it is dynamic and ongoing, and, fundamentally, exploratory.
Software is a static program and basically none of these things.
Software development is kind of like science, in some ways, in that you discover abstractions and patterns, and this requires resolving disagreements and ambiguity between you and your users, but in the end, the user demands are usually fairly concrete and specific (though no one may know how to express those demands precisely, initially), and the process is not really exploratory in the way science is.
> Science is a process for resolving disagreements, ambiguity, and uncertainty,
More dramatically stated as: Science progresses one funeral at a time.
> An important scientific innovation rarely makes its way by gradually winning over and converting its opponents. What does happen is that its opponents gradually die out, and that the growing generation is familiarized with the ideas from the beginning.
Thank you for engaging. This is a much more insightful take.
If I'm reading yiur argument right, you're saying that deployed models are downstream versions (reified) of aa theory. Theory, being the actual object of science.
I think this misrepresents science. Science is the ability to build testable knowledge. From that,how would you separate the test from the science? In fact, in an ideal world, why wouldn't you want your theory to be put in a format that's executable? I'm not saying that those things are always the same, my (provocative) title is based on a dream where we can imagine theory and model coexist because software is now a thing.
> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
Nice read. I believe that traditional software is a great way to showcase the proofs when it comes to physics and mathematics. You can easily code up a theorem in a language of your choice and justify that 'Okay, the output matches the expected value'.
I am particularly fascinated by labs like DeepMind [https://deepmind.google/science/]. The recent advances in their frontier models that are able to predict diseases before they're diagnosed is incredible. This is what AI should be built for and actually do!
Thanks! I appreciate that. Your point about DeepMind and frontier models is spot on. When they "embody"/build on the science done before them we get absolutely mindblowing synergies. But I wonder what happens when the LLMs become way smarter that us: why even loop us in? I guess that's related to the recent field medalist letter https://mathandai.org/
TL;DR I claim that modern science is synonymous with open source software. This post explains why, why it matters, and what you can (and should) do next.
Like if CERN published the discovery of the Higgs boson with 99.9997% certainty, but refusing to tell you how they calculated that number, or what equipment they used and how they calibrated it, in order to prevent other labs from copying their methods.
Or like a machine learning lab claiming SOTA on a benchmark, beating a well-known method that they re-implemented, possibly with bugs, on their private dataset, for millions of compute. But you don't get the source to check, and they don't release any intermediate results or ablation experiments. Aka, from the outside you can't distinguish it from corporate marketing.
Equivalent? In the analogy of the math or physics results it would be a mental model in someone's brain that you can't access or verify. You just hope it's true
You argue that open software is science, which is not at all the same as claiming that science is software. ("is" in this context is not equivalence -- "a poodle is a dog" != "a dog is a poodle".)
I agree the post is muddy about whether the relationship is bijective (equivalent, poodle=dog) or injective (onto, poodle is a dog).
I make it slightly more precise in the statement "I posit that open source software is a necessary condition if we are to science in a computerized world". That's where the "is" comes from in the title. Throughout history, this definitely has not been the case. I'm arguing that's changing.
It's not just "muddy", it's thoroughly inconsistent and impenetrable (which probably has a lot to do with why there is so little engagement here). You say "TL;DR I claim that modern science is synonymous with open source software" which is radically different from your "slightly more precise" statement.
I won't put any more time into this ... good luck in figuring out what it is you really want to claim and presenting a coherent and cogent argument for it.
Science should be more like this, in current times, yes.
But until much of academia is burned to the ground, or until science can be properly separated from modern academia, this will never be so. The current academic incentives are all wrong: low-quality research is rewarded and results in publications, whereas high-quality research (that takes time, and usually reveals that most exciting publications depend on p-hacking or other highly data-dependent analyses and selective presentations) is not published or actively blocked during peer review.
So instead you get BS arguments about how data can't be released for various privacy concerns (when in reality the vast majority of most datasets are trivial to scrub of identifying factors, and even in more complex datasets where you need to consider k-anonymity, it is still trivial to release data that allows replication of core analyses), and academic science is increasingly irrelevant unless it is tied to tech and industry, where producing junk actually has real negative economic and personal consequences.
I don't know what world this article / post lives in, but it isn't the messy world of actual reality.
It's changing, though, and articles like this are important. TFA is arguing for change in the future, not presenting this as a fait accompli.
In the 27 years I've been in academia, I've seen a lot of progress in data openness (NCBI GEO was a game-changer) and FOSS analysis software (it's now widely expected that a high impact pub will make all data and code available for review, and then publicly available upon manuscript publication; most major journals will not allow submission without this). It is becoming common for big journals to specifically ask reviewers to review the analysis code. It is starting to become more and more common for papers to release all the code used to generate all the figures (including supplementary figures)
There is still a long way to go, I agree. But it's always better to light candles than curse darkness, etc.
> academic science is increasingly irrelevant unless it is tied to tech and industry
While I have some sympathy with a lot of your bitterness, this statement is insulting silliness that a quick look at the list of Nobel Prizes in physiology and medicine would prove wrong. Almost all major breakthroughs in the applied sphere stem from decades of basic research that happened just because it interested someone.
> it's now widely expected that a high impact pub will make all data and code available for review, and then publicly available upon manuscript publication
I am also in academia and don't think this is true at all for data, not even remotely, unless by "publicly available" you mean "available to most academics or academic labs after an often tedious and slow approval process requiring an academic email and various signed agreements". Maybe what you are saying is true in some very specific domains (e.g. machine learning research), but in general what you are saying here is IMO wildly out of touch with present realities in the vast majority of fields, but especially those involving human subjects.
> While I have some sympathy with a lot of your bitterness, this statement is insulting silliness that a quick look at the list of Nobel Prizes in physiology and medicine would prove wrong. Almost all major breakthroughs in the applied sphere stem from decades of basic research that happened just because it interested someone.
Nobel Prizes are so rare they don't speak at all to the generalizations I am making here. Also, much medical academic research is successful because it is in fact ultimately industry-funded. It is of course though highly dependent on the academic subfield, for sure, and I was painting with a broad brush.
If I had to narrow things, STEM academic research isn't so bad, so long as we exclude social science from STEM. Much social science research needs to be defunded ASAP. And I'm not claiming industry research doesn't also have warped incentives. But, on balance, I'd wager outside of pure math/physics and certain more algorithmic/pure domains in comp sci, the smartest people today are going to choose (and be found in) industry, not academia.
Hear hear! There are so many obvious improvements to how almost everything is done. For instance, in medicine review articles as a class of articles largely represent a giant waste of time. RCTs flatten all their gathered data during publishing, summarizing complex trial data, which is gathered but never published, into a few numbers. Then review articles take a bunch of flattened data, discard the articles that don't fit the exact question they are reviewing, and then publish a doubly flattened conclusion. If any of the included articles turn out to have flaws, if treatments change in retrospect, if you are looking for the answer to a slightly different question or you are looking at a different subgroup, then the review is useless and has to be repeated.
All of these tens of thousands of man-hours could be replaced by a few GitHub repos, if only RCTs would just publish their damn data. Then you could just run and rerun the statistics on whatever subgroup you're looking for, instead of combing through decades of review articles answering slightly different questions, looking for the answer between the lines. With LLMs making mining of large scale datasets almost trivial (with the process most likely becoming trustworthy within a few years), the current status quo is looking more and more antiquated.
If you want to be even more radical, hospitals could just publish their data continuously. Of course, it is easy to point to the risks of doing so, but what's often ignored is the benefits. It is hard to overstate just how many medical mysteries a hospital encounters on a daily basis, how much unknown we are navigating in practice. The current norm is that 99.99% of these cases are never published, and are only ever thought about by a small group of people who happened to be at work. Particularly, when someone dies of something no one figured out, it is never published anywhere, because even if you tried it is not interesting reading material for a journal to publish. And no one ever tries because they're scared of being called out for a mistake. A hospital is essentially a continuously running and extremely interesting experiment, where 99.99999% of all results are thrown in the garbage, and the only published data is subject to extreme selection bias.
All of this could be different, and the risks involved are actually quite small in practice. It is easy to automatically anonymize data quite well, but extremely difficult to absolutely guarantee that it is anonymous. And since current ethical norms are extremely averse to any degree of risk, and usually entirely ignore potential benefits, we all suffer for it. It is not entirely unlikely that someone reading this post will one day die because of something that could have been prevented, had things been different.
Yup, strongly agree with all of this, especially the RCT stuff.
This has all been profoundly obvious for at least well over a decade or even two now. A consequence has been that too many serious people are driven away from academia and research, to the detriment of science generally.
I've no idea what to do about all this, because people have voiced obvious and easy solutions for decades, but they are all routinely ignored.
Which are the obvious and easy solutions?
Make analysis code available. Make anonymized or perturbed data available for download without people having to jump through hoops to get it. If you have highly sensitive data, release only the variables or other statistics needed to reproduce core analyses. Don't only do garbage null-hypothesis significance testing or statistical analyses on the full data, also do ML approaches were you have to actually show your analyses replicate on held-out subsets, and report this. Make reviews open (anonymizing as needed) so we can see when biased or incompetent reviewers are blocking good publications. Allow public review (or at least broader academic open review, in some form), since it is no longer defensible to delegate review and decisions to one or two random people that just happen to be emailed and have the time / are on some editorial / review board. Publish null findings / results, if only in minimal forms so we don't waste time and money trying to reproduce garbage. Make articles available and don't charge insane article processing fees or open access fees of thousands of USD (especially since hosting fees are not that crazy, and also because journals don't do any of the formatting work half the time anyway, and make academics or RAs or students do all the typesetting and formatting most of the time, even though now this can all be automated with template files, mostly).
Most of these things are easy to do for the majority of papers, especially in the past 20 years with the internet and modern tech and software. Plenty of frameworks exist already that have done most and/or at least some of these things, but, collectively, academia is decades behind overall.
I agree with the general sentiment, but there's one major caveat. We should implement reproducible programs on top of a virtual machine spec like JVM or WebAssembly rather than replicate the entire environment. It's more practical to do and doesn't push software towards further centralization. Let people use whatever OS and VM implementation they want, or even write their own.
> Every result is instantly reproducible. When you read a paper claiming that a new drug reduces symptoms by 30%, you click a link and watch the exact analysis run in your browser. The data processing, statistical tests, and visualizations execute in seconds using the same environment the authors used—preserved perfectly through reproducible containers.
At least some journals have this as a stipulation e.g. https://www.nature.com/nature-portfolio/editorial-policies/r...
Particularly the "data availability" and "Availability and peer review of computer code and algorithm".
However, in my limited experience, of trying to reproduce certain scRNA-seq processing pipelines, in practice it's never available as just a Github link. I can understand that some/many researcher's code is not in good shape, so I think it'll be quite a stretch to have this available.
I do think it's laudable though, to try and make it available. It would certainly have been very useful for me in the past.
Replace "Open" with free as in "freedom" gnu.org
> modern science is synonymous with open source software.
Another problem with reproducibility is the openness of the underlying data. Many academics are terrified of giving away the golden goose and the software is often useless without the data.
However many scientists do work openly, e.g. The Journal of Open Source Software:
https://joss.theoj.org/
JOSS is great, I’ve both published with and reviewed for them, and enjoyed it more than traditional journals. The process felt more constructive than destructive, in a sense.
I believe they’re always looking for new volunteers to review papers, so please do volunteer if you are able.
Science is open, but science is not software and software definitely not science.
Did you read the post...?
Yes. It conflates a bunch of things
> TL;DR I claim that modern science is synonymous with open source software
That's a strong statement that's not supported by the arguments and IMO misguided.
I don't have a problem with "open", but rather with "software".
Both science and software deal with models, however the focus is quite different. I suspect you conflate theory with models.
The goal of science is to produce and test theories — that's an inductive/abductive process. A model, regardless of whether it's reified into mathematical formulas or software, is a means of making a theory operational enough that its consequences can be derived and confronted with observations.
Software often starts downstream of this: it's a reification of theories, models, algorithms, or findings that are the result of research. Of course software can also be used as part of the research process itself. The distinction is roughly the familiar one between research and development.
I think it is worse than that. Science is a process for resolving disagreements, ambiguity, and uncertainty, and also for discovering abstractions (patterns) among phenomena. It is social and can not be reduced to a binary / digital file, as it is dynamic and ongoing, and, fundamentally, exploratory.
Software is a static program and basically none of these things.
Software development is kind of like science, in some ways, in that you discover abstractions and patterns, and this requires resolving disagreements and ambiguity between you and your users, but in the end, the user demands are usually fairly concrete and specific (though no one may know how to express those demands precisely, initially), and the process is not really exploratory in the way science is.
It just really isn't a very good comparison IMO.
> Science is a process for resolving disagreements, ambiguity, and uncertainty,
More dramatically stated as: Science progresses one funeral at a time.
> An important scientific innovation rarely makes its way by gradually winning over and converting its opponents. What does happen is that its opponents gradually die out, and that the growing generation is familiarized with the ideas from the beginning.
https://en.wikipedia.org/wiki/Planck%27s_principle
Thank you for engaging. This is a much more insightful take.
If I'm reading yiur argument right, you're saying that deployed models are downstream versions (reified) of aa theory. Theory, being the actual object of science.
I think this misrepresents science. Science is the ability to build testable knowledge. From that,how would you separate the test from the science? In fact, in an ideal world, why wouldn't you want your theory to be put in a format that's executable? I'm not saying that those things are always the same, my (provocative) title is based on a dream where we can imagine theory and model coexist because software is now a thing.
https://news.ycombinator.com/newsguidelines.html
> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
Arguments are fine. Vacuous statements need grounding. The follow-up is way more detailed
Nice read. I believe that traditional software is a great way to showcase the proofs when it comes to physics and mathematics. You can easily code up a theorem in a language of your choice and justify that 'Okay, the output matches the expected value'.
I am particularly fascinated by labs like DeepMind [https://deepmind.google/science/]. The recent advances in their frontier models that are able to predict diseases before they're diagnosed is incredible. This is what AI should be built for and actually do!
Thanks! I appreciate that. Your point about DeepMind and frontier models is spot on. When they "embody"/build on the science done before them we get absolutely mindblowing synergies. But I wonder what happens when the LLMs become way smarter that us: why even loop us in? I guess that's related to the recent field medalist letter https://mathandai.org/
We really need to ban tech people from using the word "open".
Erm. Why?
TL;DR I claim that modern science is synonymous with open source software. This post explains why, why it matters, and what you can (and should) do next.
What's the equivalent of closed source?
Like if CERN published the discovery of the Higgs boson with 99.9997% certainty, but refusing to tell you how they calculated that number, or what equipment they used and how they calibrated it, in order to prevent other labs from copying their methods.
Or like a machine learning lab claiming SOTA on a benchmark, beating a well-known method that they re-implemented, possibly with bugs, on their private dataset, for millions of compute. But you don't get the source to check, and they don't release any intermediate results or ablation experiments. Aka, from the outside you can't distinguish it from corporate marketing.
It’s research happening privately without publishing, usually going into products
Equivalent? In the analogy of the math or physics results it would be a mental model in someone's brain that you can't access or verify. You just hope it's true
Open source is much more than source available though. Its about licensing.
But that doesn't map to software at all right?
You argue that open software is science, which is not at all the same as claiming that science is software. ("is" in this context is not equivalence -- "a poodle is a dog" != "a dog is a poodle".)
I agree the post is muddy about whether the relationship is bijective (equivalent, poodle=dog) or injective (onto, poodle is a dog). I make it slightly more precise in the statement "I posit that open source software is a necessary condition if we are to science in a computerized world". That's where the "is" comes from in the title. Throughout history, this definitely has not been the case. I'm arguing that's changing.
It's not just "muddy", it's thoroughly inconsistent and impenetrable (which probably has a lot to do with why there is so little engagement here). You say "TL;DR I claim that modern science is synonymous with open source software" which is radically different from your "slightly more precise" statement.
I won't put any more time into this ... good luck in figuring out what it is you really want to claim and presenting a coherent and cogent argument for it.
Thanks for the well wishes