Amazing article. It just goes to show how many problems they are solving just to work around bugs and limitations within Apple itself. I’d wager a guess that if Apple made a new Xserve with the same performance as a Mac mini but twice the price people would still buy them, because figuring out data center operations is expensive.
Apple still lives in the world where software developed in IDE on a personal computer and shipped as a boxed software on a disquette with a licence key. Xcode cloud is a joke. All their automation is a joke. It’s not surprising even their own software development stuck because of their ecosystem because they use same tools, same cryptic software signing with their plists, p8, chains, certificates, lack of ui testing, horrible cpp, objc legacy. You just drowning in this environment. How I wish we have proper web app apis at last
Android tooling is cross-platform and easily executed in your choice of CI provider with cheap Linux ARM minutes. Simulator-based UI tests are headless (there are headless tests for iOS but not macOS). It's not even remotely a close comparison.
No, the apple ecosystem will always hold a monopoly over stupid upper class people to fleece. Apple will say "jump" and Developers will ask how high (if they know what's good for their wallet)
They might still be living in that world, but they also live in a world where they can do exactly $0 of R&D of actual rack server SKUs and still have people buying their high-end MBP SKUs at halfway-decent used car prices.
"Luckily by now we found almost perfect cables that bend exactly as we want them to bend. And we bought the entire supply of them. There are zero more of them available to order on Amazon." Everyone is hoarding at any cost...
Amazon also has much tighter integration and offers the Macs bare-metal. This has the downside that you can only gets Macs as dedicated hosts, so it's another layer to manage, and that it takes around an hour to wipe the Mac whenever you shut it down, even if you're not going to release the underlying dedicated host. Also for the last year, even grabbing a Mac dedicated hosts often runs into "None available" for hours at a time. Amazon tells you "try this other zone" but even that is no guarantee. I wasn't even able to get our TAMs to tell me whether trying other regions was likely to help. Besides that, it takes 5-10 minutes to launch a Mac, and once you acquire one, there's a 24 hour minimum billing period due to Apple licensing restrictions. Which is all to say, as a single customer, you can't really build an elastically scaled Mac CI/CD farm on top of AWS. You really need to size for your peak load. Which is expensive compared to just buying your own Macs, but $former_employer didn't want to be in the datacenter business.
FWIW, I believe MacStadium was the first to offer Macs in datacenters as a service:
> A well-known limitation is that Apple only allows two macOS VMs per physical Mac. It can be a Mac Pro with 24 cores and 192 GB memory, yet it is limited to running only two guest VMs. This limitation leaks into our product offering: we can only slice Mac resources in two halves. So we run instances with 6 or 12 cores on 14-core Mac Minis and 8/16-core instances on 18-core MacBooks (we reserve a couple cores for management overhead).
I don't know about selling, but Apple permits up to 2 virtualized Macs to run on a Mac. If they are circumventing this limit that would be an issue, but it doesn't sound like they are.
Ah okay - and the 24 hour minimum rental time doesn't apply to VMs? Im just curious how they are getting around the limits that Mac Stadium and AWS have.
They aren't getting around any limitations. How you decide to bill for meeting the limitations is a business decision, and you can do it via either higher average rates and allow more granular rentals to be financially covered that way or by requiring a 24 hour minimum.
I.e. just because Apple requires you to do things in 24 hour increments does not mean your billing has to expose it in exactly the same way.
xtool [0] came up on here a while ago [1]. I think it still requires you get a big xcode app bundle off a macos machine and drop that into xtool, since there's no other way to get the iOS library header files from Apple. The other parts of xcode just aren't used as far as I know. Other than that caveat, you're running builds in xtool's own environment on whatever platform you choose. You can even do code signing.
That's one of the reasons to keep it as far as possible from anywhere you need trust (like prod): software, like soylent green, is made of people. And you can't trust people like him.
I really wonder if Apple's logic to require a sale of a unit for 2 VMs is fiscally correct. Seems like it would be better to open it up to virtualization such that more devs can make iOS and macOS apps. Is the community of devs being forced to buy mac hw just do dev a iOS/macOs app so large that its worth putting us through all this HASSLE? Maybe they are worried about clone machine/farms that will eat into their sales. Idk. Because they are so VM unfriendly, I refuse to deal with Apple's bs.
The cost of cloud Mac hardware is quite frankly ludicrous across the board. Namespace's minimum for an M6 is, what, $0.06/minute for 6 vCPU? And that's on the cheap end - Semaphore wants $0.09/minute for 4 vCPU and GitHub Actions wants $0.062/minute for M1 hardware (forget AWS pricing even which is even more atrocious).
If I run just a UI test suite for an hour, that's $3.60/run, which is now higher than what I'm paying for AI inference that made the change that triggered the run. That's crazy! If I ran even just one run per night that would run me more than $100/month. A Mac Mini is $900 - I break even in nine months by buying the hardware outright. If I run multiple UI test suites per day, I break even much faster. These ARM chips are so power-light that the cost of electricity is practically irrelevant, and after all, these aren't rack-mounted monstrosities with 90 dB fans - they're meant to cool themselves sitting on your desk. Maybe it's different if you need like 60 of them?
Is anyone even running the numbers before they sign up? Or are companies really running that many Minis and are paying the premium to outsource the DCOps?
I looked at our GitHub Actions monthly expenses, then at the price of a M4 Mac mini (just when they were released, and still cheap), and decided to buy one as it would pay for itself in less than a year. There are definitely still cases when buying hardware still makes sense, over renting in the cloud.
I have a test suite that needs to run on a Mac in a project that merges < 20 PRs a year. No point in provisioning our own Mac and maintaining it for that.
The registry distribution subject "for another blog post" is the most interesting part of this to me. I'm very interested in the efficiency/locality metrics that went into designing that. I look forward to that future post.
I am lost, doing it in a Hackintosh is feasible but against EULA terms? I remember having a Hackintosh around the office two years ago (yes, it was x64 based). BTW I just found "ARM is not a universal architecture, there will be no more Hackintosh" [1].
Amazing article. It just goes to show how many problems they are solving just to work around bugs and limitations within Apple itself. I’d wager a guess that if Apple made a new Xserve with the same performance as a Mac mini but twice the price people would still buy them, because figuring out data center operations is expensive.
[dead]
Apple still lives in the world where software developed in IDE on a personal computer and shipped as a boxed software on a disquette with a licence key. Xcode cloud is a joke. All their automation is a joke. It’s not surprising even their own software development stuck because of their ecosystem because they use same tools, same cryptic software signing with their plists, p8, chains, certificates, lack of ui testing, horrible cpp, objc legacy. You just drowning in this environment. How I wish we have proper web app apis at last
and what kind of world do we live in now
A world where people use Claude to operate doomed SaaS products
Android tooling is cross-platform and easily executed in your choice of CI provider with cheap Linux ARM minutes. Simulator-based UI tests are headless (there are headless tests for iOS but not macOS). It's not even remotely a close comparison.
That must be why everyone’s trying to release Android apps now.
No, the apple ecosystem will always hold a monopoly over stupid upper class people to fleece. Apple will say "jump" and Developers will ask how high (if they know what's good for their wallet)
They might still be living in that world, but they also live in a world where they can do exactly $0 of R&D of actual rack server SKUs and still have people buying their high-end MBP SKUs at halfway-decent used car prices.
So I get why they don't care.
>How I wish we have proper web app apis at last
You mean APIs that inconsiderate developers use to release a simple calculator in the form of a 300+MB binary that eats a quarter of my RAM?
> You can't build iOS apps without a Mac.
I'm hoping AI will change that soon.
This article is dated November 3rd yet I've read it months ago, it was already in my history.
Title should be edited to "Putting hundreds of MacBooks in a data center (because Apple is silly)" imo
"Luckily by now we found almost perfect cables that bend exactly as we want them to bend. And we bought the entire supply of them. There are zero more of them available to order on Amazon." Everyone is hoarding at any cost...
Hmm hundreds of laptop batteries in a datacenter doesn't seem too smart.
> Another source of pain: FileVault disk encryption. Or rather decryption.
Amazon solves this by disabling FileVault and warns against enabling it.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-...
Amazon also has much tighter integration and offers the Macs bare-metal. This has the downside that you can only gets Macs as dedicated hosts, so it's another layer to manage, and that it takes around an hour to wipe the Mac whenever you shut it down, even if you're not going to release the underlying dedicated host. Also for the last year, even grabbing a Mac dedicated hosts often runs into "None available" for hours at a time. Amazon tells you "try this other zone" but even that is no guarantee. I wasn't even able to get our TAMs to tell me whether trying other regions was likely to help. Besides that, it takes 5-10 minutes to launch a Mac, and once you acquire one, there's a 24 hour minimum billing period due to Apple licensing restrictions. Which is all to say, as a single customer, you can't really build an elastically scaled Mac CI/CD farm on top of AWS. You really need to size for your peak load. Which is expensive compared to just buying your own Macs, but $former_employer didn't want to be in the datacenter business.
FWIW, I believe MacStadium was the first to offer Macs in datacenters as a service:
https://9to5mac.com/2020/12/27/macstadium-mac-mini/
Honestly, my first thought was "does the datacentre know ?"
12 MacBooks per shelf, if we say roughly 5U per shelf (21.12/4.445 rounded up), so roughly 9 shelves per 47U rack, so 100-ish MacBooks per rack.
That's a scary density of LiPo battery.
The author complains about the lack of redundant power in Mac Minis, but there are other ways to get redundant power into one AC input.
Isnt it against the Apple EULA to sell VMs of Macos? Love the creative problem solving with the HW.
> A well-known limitation is that Apple only allows two macOS VMs per physical Mac. It can be a Mac Pro with 24 cores and 192 GB memory, yet it is limited to running only two guest VMs. This limitation leaks into our product offering: we can only slice Mac resources in two halves. So we run instances with 6 or 12 cores on 14-core Mac Minis and 8/16-core instances on 18-core MacBooks (we reserve a couple cores for management overhead).
I don't know about selling, but Apple permits up to 2 virtualized Macs to run on a Mac. If they are circumventing this limit that would be an issue, but it doesn't sound like they are.
Ah okay - and the 24 hour minimum rental time doesn't apply to VMs? Im just curious how they are getting around the limits that Mac Stadium and AWS have.
They aren't getting around any limitations. How you decide to bill for meeting the limitations is a business decision, and you can do it via either higher average rates and allow more granular rentals to be financially covered that way or by requiring a 24 hour minimum.
I.e. just because Apple requires you to do things in 24 hour increments does not mean your billing has to expose it in exactly the same way.
Man, kudos to the person or AI who designed their website. Easy to read, easy to find any information, with eye catching branding.
Author here. Happy to answer questions.
> You can't build iOS apps without a Mac.
I'm not sure this is true anymore.
xtool [0] came up on here a while ago [1]. I think it still requires you get a big xcode app bundle off a macos machine and drop that into xtool, since there's no other way to get the iOS library header files from Apple. The other parts of xcode just aren't used as far as I know. Other than that caveat, you're running builds in xtool's own environment on whatever platform you choose. You can even do code signing.
[0] https://xtool.sh
[1] https://news.ycombinator.com/item?id=44011515
Technically you can, legally not sure
It is.
You can build iOS apps on Omarchy now
Omarchy is even less production ready than racking up laptops.
Plus there's the issue that DHH is a racist twat.
That's one of the reasons to keep it as far as possible from anywhere you need trust (like prod): software, like soylent green, is made of people. And you can't trust people like him.
I really wonder if Apple's logic to require a sale of a unit for 2 VMs is fiscally correct. Seems like it would be better to open it up to virtualization such that more devs can make iOS and macOS apps. Is the community of devs being forced to buy mac hw just do dev a iOS/macOs app so large that its worth putting us through all this HASSLE? Maybe they are worried about clone machine/farms that will eat into their sales. Idk. Because they are so VM unfriendly, I refuse to deal with Apple's bs.
> Is the community of devs being forced to buy mac hw just do dev a iOS/macOs app so large
Yes.
Why else would you buy such an expensive machine with such a crappy operating system?
It's Unix and it can run any pro creative app you like.
Not crappy. Skill issue.
The cost of cloud Mac hardware is quite frankly ludicrous across the board. Namespace's minimum for an M6 is, what, $0.06/minute for 6 vCPU? And that's on the cheap end - Semaphore wants $0.09/minute for 4 vCPU and GitHub Actions wants $0.062/minute for M1 hardware (forget AWS pricing even which is even more atrocious).
If I run just a UI test suite for an hour, that's $3.60/run, which is now higher than what I'm paying for AI inference that made the change that triggered the run. That's crazy! If I ran even just one run per night that would run me more than $100/month. A Mac Mini is $900 - I break even in nine months by buying the hardware outright. If I run multiple UI test suites per day, I break even much faster. These ARM chips are so power-light that the cost of electricity is practically irrelevant, and after all, these aren't rack-mounted monstrosities with 90 dB fans - they're meant to cool themselves sitting on your desk. Maybe it's different if you need like 60 of them?
Is anyone even running the numbers before they sign up? Or are companies really running that many Minis and are paying the premium to outsource the DCOps?
> A Mac Mini is $900 - I break even in nine months by buying the hardware outright.
When you take resale value into account, purchase makes even more sense than renting.
Used M1 Mac Minis from 2020 are currently selling for about $500 when they originally retailed for $699.
I looked at our GitHub Actions monthly expenses, then at the price of a M4 Mac mini (just when they were released, and still cheap), and decided to buy one as it would pay for itself in less than a year. There are definitely still cases when buying hardware still makes sense, over renting in the cloud.
> A Mac Mini is $900 - I break even in nine months by buying the hardware outright.
The clue to their decision may be right there, potentially going for an increase in sales.
But nowadays it seems they suffer from a lack of vision and drive; perhaps (hopefully) they’ll come around.
I have a test suite that needs to run on a Mac in a project that merges < 20 PRs a year. No point in provisioning our own Mac and maintaining it for that.
The registry distribution subject "for another blog post" is the most interesting part of this to me. I'm very interested in the efficiency/locality metrics that went into designing that. I look forward to that future post.
I am lost, doing it in a Hackintosh is feasible but against EULA terms? I remember having a Hackintosh around the office two years ago (yes, it was x64 based). BTW I just found "ARM is not a universal architecture, there will be no more Hackintosh" [1].
[1] https://www.reddit.com/r/hackintosh/comments/1uboij0/arm_is_...
[dead]
Wild to see them using MacBook Pros and not Mac Mini/Studio. Curious how that possible makes sense for economics and cooling.
They answer that in literally the first paragraph in the article.
TL:DR; They couldn't buy enough minis or studios.
it's "literally" the 5th paragraph, but yeah, it's there.
If only they covered that in the article you didn't read.
Did you literally open the link, see a picture of Macbooks, and come directly back here before reading a single word?
tbh, I skipped to the part about Virtualization.framework and skipped the part that started "Chances are you’ve seen me on X: "
So they buy up the machines, likely create supply issues (add waiting time etc.), then rent them out at premium?
People are outraged landlords doing that with housing.
Yet someone such rent seeking behaviour is acceptable?
Get fucked.