Book publishers used to print India/SEA-only editions of books only sold in those countries, significantly cheaper than in the US or Europe.
Then a Thai guy realized that this would be a good business opportunity: buy cheaper books and then import them to the US. Wiley brought him to court, went to the SCOTUS, Wiley lost the case. So they ended up printing cheaper edition books.
Mind you "price discrimination" like this still exist in the digital world where locality is easier to be enforced. For example Steam has extensive regional pricing across countries so the same game can be significantly cheaper in Russia, India, Brazil etc. compared to US or EU
When Kirtsaeng moved to the US in 1997 to pursue an undergraduate degree in mathematics at
Cornell University,[4] he discovered that textbooks (not just those published by Wiley, but
of other publishers too) were considerably more expensive to buy in the United States than
in his home country. Kirtsaeng asked his relatives from Thailand to buy such books at home
and ship them to him to sell at a profit. He sold the imported books on eBay, making $1.2
million in revenue, although the parties disputed the net profit amount.
A text book case (ha!) about one of the mechanisms that enable the free markets and trade to bring the prices of goods down.
Wow, I've only ever heard of regional pricing being described as an overwhelmingly positive concept. When phrased as "price discrimination", it invokes a completely different set of (negative) emotions in me.
It's weirdly uncomfortable knowing that phrasing has such a big impact on one's emotions. It really shows how vulnerable we are to manipulation.
Wow, how did we go from "it's impossible to enforce region locking online because you can transmit information instantly across the world" to "locality more easily enforced in the digital world"?
The wonderful world of “no ownership” means that if the store owner suspects anything is afoot they can pester you for evidence of residence and ban your account. Instead of saving 20% you risk losing 80% (don’t know the real price difference).
I graduated a decade ago but there was a pretty decent post-semester market for people selling their textbooks to other students (mostly via posting in various Facebook groups). Given the cost of college nowadays, if that ends up saving students a bit of money, it's probably worth it over saving the textbooks themselves.
Most years my parents ended up just driving my stuff to my aunt's house where she kept it in her garage over the summer after I packed it all up since I grew up several states away but my aunt loved in the area, and by the end of senior year, the textbooks I never managed to sell had added up to almost an entire extra bin (which was heavier than any of the others because they mostly just had clothes). For people who didn't have parents who loved driving everywhere in a large enough car or relatives in the area, this seems like it would be even more annoying (potentially ending up with just donating the leftover textbooks or maybe leaving them on the street like furniture and stuff often is at the end of the year on campuses). If people want to keep their textbooks, more power to them, but I'm not convinced that having a cheaper way that also simplifies the logistics of moving isn't better.
I'm a professor at a community college in Silicon Valley, and my students use online textbooks. I try to use Creative Commons or other libre textbooks, but sometimes I use paid textbooks when they are heads-and-shoulders better than their libre alternatives. Some e-textbooks can be accessed on a subscription basis. I admit I prefer non-subscription materials, but a colleague advised me that often the book that students learn from is different from a good reference book that students can use once they've already learned the material. For example, my colleagues and I have had great success with an online, interactive textbook for discrete math. While the subscription is unfortunately only valid for the duration of the course, once students have learned discrete math, they could buy a used copy of Rosen's discrete math textbook as a reference.
The nice thing about e-textbooks is not needing to carry around a bunch of heavy books. I remember the tomes I had in my college days, such as Stewart's Calculus.
> The nice thing about e-textbooks is not needing to carry around a bunch of heavy books. I remember the tomes I had in my college days, such as Stewart's Calculus.
I'm not sure how relatable this is for people outside of my age group (late millennial), but growing up we used to get chided about stuffing too much in our backpacks and shown presentations about how we were going to give ourselves scoliosis, but then had like two minutes between classes that might be on the other side of the building (meaning no time to stop at our lockers other than at lunch) and we'd get chided even worse if we didn't have our textbooks available for the days we happened to need them in class (which of course we were never told in advance, and some classes had multiple textbooks that wouldn't all get used every time).
I wish that we had iPads or Chromebooks or whatever back when I was in school not even because I would have wanted to have been able to surf the web or play games or whatever, but just to have a solution to having to pick between having a sore back or getting worse scolding (with a side course of hypocrisy either way).
FreeBSD didn’t have memory overcommit and instead used strict swap reservation - each allocated anonymous memory page was supposed to have a corresponding swap page. This required 2x RAM swap space, otherwise you would get “out of swap” when forking a large process. FreeBSD implemented memory overcommit around 2000.
Oh, so that's where that old nugget of wisdom came from! I've heard the rule about making your swap at least 2x your RAM for ages and thought it was just some old rule of thumb from the 80s. I didn't know there used to be a legitimate reason for it.
I remember how NetBSD promoted itself as running on many more
toasters than Linux once.
Then some NetBSD dev wrote on their mailing list that this is
no longer true. Linux runs on more toasters now. (And also top 500 supercomputers, but toasters are the real metal to the petal test.)
It's an interesting piece of history too. I kind of evaluate it a
bit differently, e. g. my summary is "momentum beats academic
perfection". Which is not completely what it is about, but it is
my own imperfect TL;DR summary.
> but toasters are the real metal to the petal test.
Or is that a new tongue in cheek idiom?
AFAIK, it’s peDal, not peTal, and the other way around: pedal to the metal. The literal meaning is to push your accelerator pedal down to the floor of your car, which is made of metal.
This basically fits my stereotype of BSD being a little bit more hardcore while Linux is a little more accessible… when the question was “can you install an OS on a toaster,” BSD had an advantage. Now that normal engineers have to make IOT toasters (for some reason) Linux should have the advantage, right?
Seems like at a glance that website if it sees a referral from ycombinator, it redirects to that image.... In a private window it loads the 'intended' page.....
Just blackholing the request would be kinder than taking it out on the person following the link, given that their gripe seems to be more with the one who posted it (and that as a deterrent, it doesn't seem to be particularly effective given that it's been years since the first time I've seen this conversation play out in comments)
I don’t understand the part about using heuristics and deciding what counts as used memory…
Used memory for the system is always total minus available.
Heuristics? I would hope that the system knows precisely what is using every single byte of physical and virtual memory. Is this a reporting problem? Why do we have to settle for heuristics and not the exact number?
> I would hope that the system knows precisely what is using every single byte of physical and virtual memory.
Of course the system knows what is using every page. The difficulty is really in how to account for pages that are backed by disk.
If you count all of those as free, that's not accurate. If you count all of those as used, that's not accurate either. Additionally, FreeBSD (at least) doesn't have separate queues for disk backed pages, so there's not really a good way to know how much of your active (or inactive) memory is disk backed.
As an additional caveat that measuring active/inactive has costs. In the past, FreeBSD wouldn't really do the work for that until it needed to... I know some stuff changed, but I don't remember where it ended up; it wasn't great when it bulk marked a ton of pages as inactive and then the active ones would fault back in.
This is only really a problem if you accept overcommit as a force of nature that can't be changed or tweaked (you can still do address space reservation without needing overcommit)
If you don't, it becomes rather easy (and strict commit accounting is done for example on Linux even if it isn't used in some cases)
Memory mapped files can be entirely recreated from the disk so no need to charge for them. Anonymous pages (whether private or shareable) have to be charged. Shareable memory is the harder one to charge. (The case where a mapping is used by only one can get charged as private commit.) These two previous cases are charged even if in a swap file or whatnot
Thrashing is a well known known issue that can occur with swap, but it can also happen from page cache or memory mapped files. Indeed not having swap enabled can make things worse, as private pages that haven't been used in a hours cannot be swapped out to keep the important files cached or memory mapped.
Realistically for measuring physical memory sufficiency, you care about memory/data of any type (even files) that will be used in use upcoming time period, and ensuring that a sufficient percentage of it can be held in physical memory to avoid thrashing.
This is hard! Technically impossible to know for the general case (halting problem), and all methods of trying to approximate it involves trade-offs.
The thing is it's easy to define free, unused memory. But a lot of the used memory is your system caching stuff that would be free if you needed more than what's actually free. So you can see you have 1g of free memory out of your 4g, but then you allocate 3g and it will do without a sweat and you'd be confused. So you have to go and dig for what those caches are and report that they're effectively free too.
> The “cached” memory includes tmpfs and ramfs for seemingly no reason.
If you're curious why that is by the way, it's because that's actually how these are implemented (tmpfs/ramfs is just a mount to a filesystem where the files never get marked clean[1])
AFAIK the only way for you to figure out how much of your disks is actually cached involves enumerating all tmpfs and ramfs mounts, summing their sizes, and subtracting the sum from the cache size reported by the kernel.
Well, what's the alternative? Invent a new type of memory reservations specifically to account for tmpfs/ramfs mounts? That'd violate your own stated desired goal of
> The OS should just expose a counter for available memory instead of having applications understand every type of memory reservation.
I think a memory pressure indicator is useful. For example, if you’re writing a garbage collector, you can choose to hold off from returning pages back to the kernel when the system is under low pressure, and do this more often under high pressure.
That's very, very marginally useful. If your application recently ran GC when the system was under low pressure and freed a bunch of pages (and still holds to them), and then some other application suddenly acquires a lot of memory, your application won't free those pages until next GC happens. Which most likely won't until you've used up all those freed pages, and at that point a lot of swapping will already have had happen.
Explicitly set per-application upper bounds à la GOMEMLIMIT and Java's -Xmx, together with cgroup's enforcement, seem to be much more useful in practice.
Or there was a proposal for some sort of SIGLOWEM signal that would be sent to all processes in the system (with default disposition of "do nothing") that'd allow applications to release some of its non-vital memory holdings: also a much more timelier notification.
That metric would give you a number of bytes which can be used for pages not backed by files, but it won't give you actual memory usage statistics:
It won't count executable pages and memory-mapped file use as "used" memory, so your system might display gigabytes "free" when it's starving, executables getting paused when code pages are paged-in from disk.
It's just less useful than what's displayed now. "Everyone is doing it wrong" is usually a signal that you're missing something.
As for the “unauthorized” sign:
Book publishers used to print India/SEA-only editions of books only sold in those countries, significantly cheaper than in the US or Europe. Then a Thai guy realized that this would be a good business opportunity: buy cheaper books and then import them to the US. Wiley brought him to court, went to the SCOTUS, Wiley lost the case. So they ended up printing cheaper edition books.
https://en.wikipedia.org/wiki/Kirtsaeng_v._John_Wiley_%26_So....
Mind you "price discrimination" like this still exist in the digital world where locality is easier to be enforced. For example Steam has extensive regional pricing across countries so the same game can be significantly cheaper in Russia, India, Brazil etc. compared to US or EU
An example: https://steamdb.info/app/413150/
https://partner.steamgames.com/pricing/explorer
https://partner.steamgames.com/doc/store/pricing/currencies
Wow, I've only ever heard of regional pricing being described as an overwhelmingly positive concept. When phrased as "price discrimination", it invokes a completely different set of (negative) emotions in me.
It's weirdly uncomfortable knowing that phrasing has such a big impact on one's emotions. It really shows how vulnerable we are to manipulation.
> It's weirdly uncomfortable knowing that phrasing has such a big impact on one's emotions. It really shows how vulnerable we are to manipulation.
My dad would discuss it in high school english courses: terrorist vs freedom fighter etc
That's why the current crop of authoritarian "think of the children" propaganda is so unreasonably effective
Wow, how did we go from "it's impossible to enforce region locking online because you can transmit information instantly across the world" to "locality more easily enforced in the digital world"?
Most people haven't started using vpns yet.
The wonderful world of “no ownership” means that if the store owner suspects anything is afoot they can pester you for evidence of residence and ban your account. Instead of saving 20% you risk losing 80% (don’t know the real price difference).
You can't fool me with a VPN when your credit card is a US one.
Silicon valley happened
If you like this kind of post, you might like this “htop explained” post.
https://peteris.rocks/blog/htop/
htop explained*
I was looking forward for web protocol, but alas...
It was a typo. Thanks, fixed.
But there is an http headers article you might like.
https://peteris.rocks/blog/exotic-http-headers/
I”d instead suggest RFC 9110, HTTP Semantics: <https://www.rfc-editor.org/info/rfc9110/>
Thank you for such a quality post.
I wonder if these btop fixes got into the standard ports collection? Or even upstream?
I like the command for viewing the ARC cache size, never knew that. It's only 2GB on my system (of 64GB RAM).
Ports is on 1.4.7.
* https://freshports.org/sysutils/btop/
The fix is still pending being even made to the origin version.
* https://github.com/aristocratos/btop/pull/1728
Thanks! I will keep an eye on that.
>I like the command for viewing the ARC cache size
zfs-stats is also nice for more zfs internals like hit-rate L2 etc..
https://www.freshports.org/sysutils/zfs-stats
Thanks! I didn't know that one.
the end struck me - a picture of an os book. I wonder if students these days retain their books after college, or do they get returned as a rental?
I graduated a decade ago but there was a pretty decent post-semester market for people selling their textbooks to other students (mostly via posting in various Facebook groups). Given the cost of college nowadays, if that ends up saving students a bit of money, it's probably worth it over saving the textbooks themselves.
Most years my parents ended up just driving my stuff to my aunt's house where she kept it in her garage over the summer after I packed it all up since I grew up several states away but my aunt loved in the area, and by the end of senior year, the textbooks I never managed to sell had added up to almost an entire extra bin (which was heavier than any of the others because they mostly just had clothes). For people who didn't have parents who loved driving everywhere in a large enough car or relatives in the area, this seems like it would be even more annoying (potentially ending up with just donating the leftover textbooks or maybe leaving them on the street like furniture and stuff often is at the end of the year on campuses). If people want to keep their textbooks, more power to them, but I'm not convinced that having a cheaper way that also simplifies the logistics of moving isn't better.
I'm a professor at a community college in Silicon Valley, and my students use online textbooks. I try to use Creative Commons or other libre textbooks, but sometimes I use paid textbooks when they are heads-and-shoulders better than their libre alternatives. Some e-textbooks can be accessed on a subscription basis. I admit I prefer non-subscription materials, but a colleague advised me that often the book that students learn from is different from a good reference book that students can use once they've already learned the material. For example, my colleagues and I have had great success with an online, interactive textbook for discrete math. While the subscription is unfortunately only valid for the duration of the course, once students have learned discrete math, they could buy a used copy of Rosen's discrete math textbook as a reference.
The nice thing about e-textbooks is not needing to carry around a bunch of heavy books. I remember the tomes I had in my college days, such as Stewart's Calculus.
> The nice thing about e-textbooks is not needing to carry around a bunch of heavy books. I remember the tomes I had in my college days, such as Stewart's Calculus.
I'm not sure how relatable this is for people outside of my age group (late millennial), but growing up we used to get chided about stuffing too much in our backpacks and shown presentations about how we were going to give ourselves scoliosis, but then had like two minutes between classes that might be on the other side of the building (meaning no time to stop at our lockers other than at lunch) and we'd get chided even worse if we didn't have our textbooks available for the days we happened to need them in class (which of course we were never told in advance, and some classes had multiple textbooks that wouldn't all get used every time).
I wish that we had iPads or Chromebooks or whatever back when I was in school not even because I would have wanted to have been able to surf the web or play games or whatever, but just to have a solution to having to pick between having a sore back or getting worse scolding (with a side course of hypocrisy either way).
just hint students towards anna's archive and then sky's the limit
I bought as few textbooks as I could, but the few that I did buy are sitting in my parents' basement bookshelves somewhere.
In college ATM and while I usually keep old textbooks, I have sold a couple on ebay to free up shelf space.
Great job on getting the fixes merged!
Interesting post, it made me wonder. At one time FreeBSD swap usage/logic was far better than what Linux did. Is that still the case ?
FreeBSD didn’t have memory overcommit and instead used strict swap reservation - each allocated anonymous memory page was supposed to have a corresponding swap page. This required 2x RAM swap space, otherwise you would get “out of swap” when forking a large process. FreeBSD implemented memory overcommit around 2000.
Oh, so that's where that old nugget of wisdom came from! I've heard the rule about making your swap at least 2x your RAM for ages and thought it was just some old rule of thumb from the 80s. I didn't know there used to be a legitimate reason for it.
See this retrocomputing.SE question https://retrocomputing.stackexchange.com/questions/32492/ori...
Yes, It's just not every tool is aware of ZFS ARC. Which is what this post is about. Author just describes in an odd way.
It's a bit odd that you have to query a filesystem-specific value to get simple memory usage stats.
Despite having File System, ZFS is a lot more than just file system. Sooner you understand and make peace with it, easier it is to use it.
Which is exactly (part of) why people call ZFS a "rampant layering violation"!
I remember how NetBSD promoted itself as running on many more toasters than Linux once.
Then some NetBSD dev wrote on their mailing list that this is no longer true. Linux runs on more toasters now. (And also top 500 supercomputers, but toasters are the real metal to the petal test.)
These fights always remind me of:
https://www.jwz.org/doc/worse-is-better.html
It's an interesting piece of history too. I kind of evaluate it a bit differently, e. g. my summary is "momentum beats academic perfection". Which is not completely what it is about, but it is my own imperfect TL;DR summary.
> but toasters are the real metal to the petal test.
Or is that a new tongue in cheek idiom?
AFAIK, it’s peDal, not peTal, and the other way around: pedal to the metal. The literal meaning is to push your accelerator pedal down to the floor of your car, which is made of metal.
"metal to the petal" gives me more of a "gilding the lily" flavor, especially in cnotrast to the original construction.
Still not sure if it was on purpose, but I read it as a reference to a "but can it run DOOM?" benchmark.
This basically fits my stereotype of BSD being a little bit more hardcore while Linux is a little more accessible… when the question was “can you install an OS on a toaster,” BSD had an advantage. Now that normal engineers have to make IOT toasters (for some reason) Linux should have the advantage, right?
Normal engineers don't do that either.
why did that url point me to a scrotum in an egg cup
Seems like at a glance that website if it sees a referral from ycombinator, it redirects to that image.... In a private window it loads the 'intended' page.....
HN has disappointed JWZ
Just blackholing the request would be kinder than taking it out on the person following the link, given that their gripe seems to be more with the one who posted it (and that as a deterrent, it doesn't seem to be particularly effective given that it's been years since the first time I've seen this conversation play out in comments)
ZFS cache. The end.
User installs an unfamiliar server OS with an enterprise filesystem and is stunned when it works differently. I fail to see a teachable moment here.
I've noticed that the various tools report different amounts of free memory. I appreciate finally having an explanation for why.
Sure, but also some tools needed fixing.
This is why I use Linux. :>
Poor FreeBSD folks though. After so many years trying to present themselves as better alternative, the road just got steeper ...
The OS Crusades are over man
For real though, what is the point of running a bsd on a personal computer? Seems like intentionally handicapping yourself.
I have FreeBSD on all my personal machines and i feel the opposite of handicapped, however i use Linux (CachyOS) for gaming (as a VM with pass-thru).
Consistent ZFS tooling around the OS and ZFS support out of the box, which always works.
So more so because of the storage drivers? I don’t have the specific concern, maybe that’s why I don’t understand the draw.
No ZFS no problem :)
Netcraft confirms it.
~all Browsers on FreeBSD (Chromium/Firefox/Librewolf) have Linux as User-Agent.
I don’t understand the part about using heuristics and deciding what counts as used memory…
Used memory for the system is always total minus available.
Heuristics? I would hope that the system knows precisely what is using every single byte of physical and virtual memory. Is this a reporting problem? Why do we have to settle for heuristics and not the exact number?
> I would hope that the system knows precisely what is using every single byte of physical and virtual memory.
Of course the system knows what is using every page. The difficulty is really in how to account for pages that are backed by disk.
If you count all of those as free, that's not accurate. If you count all of those as used, that's not accurate either. Additionally, FreeBSD (at least) doesn't have separate queues for disk backed pages, so there's not really a good way to know how much of your active (or inactive) memory is disk backed.
As an additional caveat that measuring active/inactive has costs. In the past, FreeBSD wouldn't really do the work for that until it needed to... I know some stuff changed, but I don't remember where it ended up; it wasn't great when it bulk marked a ton of pages as inactive and then the active ones would fault back in.
This is only really a problem if you accept overcommit as a force of nature that can't be changed or tweaked (you can still do address space reservation without needing overcommit)
If you don't, it becomes rather easy (and strict commit accounting is done for example on Linux even if it isn't used in some cases)
Memory mapped files can be entirely recreated from the disk so no need to charge for them. Anonymous pages (whether private or shareable) have to be charged. Shareable memory is the harder one to charge. (The case where a mapping is used by only one can get charged as private commit.) These two previous cases are charged even if in a swap file or whatnot
> If you count all of those as free, that’s not accurate.
Why not? It depends on what you’re measuring. Physical memory? They count as free. Virtual memory? They count as used.
The ambiguities only arise when we stop making that distinction very clear.
> It depends on what you’re measuring.
What I want to know is do I have enough physical memory for what I'm running.
Sure, I can drop disk backed pages and recreate them as needed, but when it happens too often, there goes performance.
Yep.
Thrashing is a well known known issue that can occur with swap, but it can also happen from page cache or memory mapped files. Indeed not having swap enabled can make things worse, as private pages that haven't been used in a hours cannot be swapped out to keep the important files cached or memory mapped.
Realistically for measuring physical memory sufficiency, you care about memory/data of any type (even files) that will be used in use upcoming time period, and ensuring that a sufficient percentage of it can be held in physical memory to avoid thrashing.
This is hard! Technically impossible to know for the general case (halting problem), and all methods of trying to approximate it involves trade-offs.
The thing is it's easy to define free, unused memory. But a lot of the used memory is your system caching stuff that would be free if you needed more than what's actually free. So you can see you have 1g of free memory out of your 4g, but then you allocate 3g and it will do without a sweat and you'd be confused. So you have to go and dig for what those caches are and report that they're effectively free too.
Instantly reclaimable disk caches should count as available, and they do.
This isn’t hard. The OS should just expose a counter for available memory instead of having applications understand every type of memory reservation.
edit:
Linux does this, but it has its own share of issues with memory counters. The “cached” memory includes tmpfs and ramfs for seemingly no reason.
> The “cached” memory includes tmpfs and ramfs for seemingly no reason.
If you're curious why that is by the way, it's because that's actually how these are implemented (tmpfs/ramfs is just a mount to a filesystem where the files never get marked clean[1])
[1]: https://www.kernel.org/doc/Documentation/filesystems/ramfs-r...
That’s clever. Makes for terrible UX, though.
AFAIK the only way for you to figure out how much of your disks is actually cached involves enumerating all tmpfs and ramfs mounts, summing their sizes, and subtracting the sum from the cache size reported by the kernel.
Well, what's the alternative? Invent a new type of memory reservations specifically to account for tmpfs/ramfs mounts? That'd violate your own stated desired goal of
> The OS should just expose a counter for available memory instead of having applications understand every type of memory reservation.
I don’t see how it contradicts my goal. I want memory counters that abstract away kernel behavior that the application has no business accounting for.
The applications, frankly, have no business accounting any memory at all outside of what they use themselves via sbrk/mmap.
I think a memory pressure indicator is useful. For example, if you’re writing a garbage collector, you can choose to hold off from returning pages back to the kernel when the system is under low pressure, and do this more often under high pressure.
That's very, very marginally useful. If your application recently ran GC when the system was under low pressure and freed a bunch of pages (and still holds to them), and then some other application suddenly acquires a lot of memory, your application won't free those pages until next GC happens. Which most likely won't until you've used up all those freed pages, and at that point a lot of swapping will already have had happen.
Explicitly set per-application upper bounds à la GOMEMLIMIT and Java's -Xmx, together with cgroup's enforcement, seem to be much more useful in practice.
Or there was a proposal for some sort of SIGLOWEM signal that would be sent to all processes in the system (with default disposition of "do nothing") that'd allow applications to release some of its non-vital memory holdings: also a much more timelier notification.
Ostensibly you could subtract "Shmem"[1] in /proc/meminfo from the cached value... maybe?
Do agree it's not the best UX and utilities should probably do a better job at showing that
[1]: https://man7.org/linux/man-pages/man5/proc_meminfo.5.html
That metric would give you a number of bytes which can be used for pages not backed by files, but it won't give you actual memory usage statistics:
It won't count executable pages and memory-mapped file use as "used" memory, so your system might display gigabytes "free" when it's starving, executables getting paused when code pages are paged-in from disk.
It's just less useful than what's displayed now. "Everyone is doing it wrong" is usually a signal that you're missing something.
Linux MemAvailable from /proc/meminfo is just an estimation calculated as an arbitrary percentage (50%) of free and potentially reclaimable memory.
You can't determine how much of the memory can actually be reclaimed under memory pressure until you try to reclaim it.
> You can't determine how much of the memory can actually be reclaimed under memory pressure until you try to reclaim it.
Yes, and I’m arguing that we should be able to, by having the kernel keep track.
You will be surprised by how inaccurate memory measurements are.
Not after reading this article. :-)