👏 more 👏 developers 👏 need 👏 to 👏 hear 👏 this
I can count on one hand the number of my clients over the past couple of years who haven't either over-architected for scale or were unnecessarily concerned about it (prior to coming to me for strategic advice, of course 😉).
You don't need to understand Distributional Little's Law to figure this out, it's obvious with primary school level math.
(screenshot excerpt from https://tailscale.com/blog/new-internet)
commentary/addition on blogpost
@ryantownsend I think their answer to "how did we get here" is missing a crucial bit of nuance: the influence of tech company blogs.
I've noticed a pretty clear pattern over the years where some tech company has to solve a problem at scale, blogs about how they've done it, and then there's a horde of developers who excitedly reads that blogpost and declares it the new best practice, elevating it to the level of "you are not a serious developer if you don't do this", often egged on by the tech company in question.
When prodded as to why, given that they are not a big tech company themselves, the answer has always been the same: a *belief*, not merely a hope, that they will be soon. Many of them outright reasoning along the lines of "if I do the same thing as this big tech company, I *will* grow as big as them and get rich, because clearly it worked for them".
This is, IMO, a much stronger effect than that article implies; this kind of technical writing actually drives entire hype cycles and steers entire developer communities in specific directions (as opposed to individual developers just being a bit optimistic), and I've seen it happen first-hand many times.
commentary/addition on blogpost
@joepie91 @ryantownsend on the other hand I've talked with an admin who very much prefers to deploy to kubernetes rather than bare metal, because k8s makes deploying and managing things and keeping services running much easier and more uniform. Yes, even if it's a single python script running on a single low powered machine.
commentary/addition on blogpost
@viq @ryantownsend I've seen this motivation quite a few times, and I strongly suspect it's similar to the whole thing with microservices and how it supposedly allows for decoupling code, when that benefit actually comes from code modularization and that's only a tangential side effect of microservices. It's not that microservices don't *also* have this benefit, it's that you could have had that same benefit without all the complexity of microservices.
Same thing with deploying to kubernetes; the actual benefit here is a consistent deployment environment (which is a valid thing to want!). You don't *need* an ultra-scalable system (with all of the associated complexity) for that, you just need a consistent environment, and there are many other options for that that aren't kubernetes-shaped, often with a significantly lower complexity cost.
commentary/addition on blogpost
@viq @ryantownsend Two options that immediately come to mind, with different philosophies behind them, would be NixOS and Dokku. Or, if you're willing to compromise a bit on environment consistency, the many deployment tools like Chef and Puppet (their main problem is that they can only *add* to the environment).
Dokku is essentially a single-machine self-hostable Heroku-like environment, that AIUI provides a consistent environment much like Heroku does. I'm personally not a fan of the Heroku model so I haven't used it, but from what I've heard from others, it works pretty much exactly as promised.
NixOS is a declarative Linux distribution that works kind of like a deployment system like Chef/Puppet, but implemented as a local package/system manager first and foremost, and with much stronger guarantees about environment consistency/determinism (also stronger than containers).
It can pretty trivially be extended to multiple systems with third-party tools if wanted, but it's system-local in its core; using it across systems basically just entails pre-building the other systems (which is just a build product, from its perspective) and pushing your build to them.
NixOS' documentation still needs a lot of work so its apparent complexity is higher than what's strictly needed to work with it, and Dokku is probably more immediately accessible, but I bring it up because it is a good example of what *can* be achieved without 'hyperscale' systems today.
commentary/addition on blogpost
@joepie91 @ryantownsend like OpenShift, or what would some examples be?
commentary/addition on blogpost
@viq @ryantownsend I'm more thinking of platforms like Heroku (ironically) and the smattering of other "we will host your app for you [with various degrees of add-on services]" providers. They kind of by definition provide a standardized environment, at least within the confines of their own service.
But I find them uninteresting due to the vendor lock-in, so I don't track very closely which ones are currently alive :)
commentary/addition on blogpost
@joepie91 @ryantownsend that's fair, I just wanted to better understand what exactly category we're talking about, maybe some examples.
On a side note, I've seen fly.io mentioned in that category, and that they supposedly do some interesting stuff.
commentary/addition on blogpost
@viq @ryantownsend I would count them as one such platform, yeah. Though again with the same vendor lock-in caveat :)
(And they do piecemeal billing, like AWS, which is a very good way to get a surprise bill at the end of the month IME)
commentary/addition on blogpost
@joepie91 @viq Heroku/Render don’t really have vendor lock-in as they use pretty standard open source tech like Postgres/Redis and their VMs just run whatever code you deploy, nothing fancy there either.
In a commercial environment, it doesn’t really make sense to run this yourself unless you have specialised infrastructure requirements – PaaS have entire teams dedicated to uptime/security etc, accessing that for as little as $50/mo is unachievable in-house.
commentary/addition on blogpost
@ryantownsend @viq Over the years, I've had a peek behind the curtains of a lot of infrastructure/service/security providers, in various capacities and to various extents, and I honestly can't say that I share that reasoning.
I'm certainly not saying that there are no competent people working at such companies, but there's often a very big cliff between "the security/reliability posture that is advertised or implied" and the "the posture that the provider actually has", frequently because of overworked infra/security teams or micromanagers meddling and not letting them do what's needed.
There are probably specific providers which have competent teams (though even that only gets you so far, at a certain scale it becomes unmanageable). But I think that outsourcing it to a competent-*appearing* provider and assuming that takes care of your security/reliability, is a very dangerous thing to be doing. You're mostly just paying for a security blanket at that point - which of course will be cheaper than the real thing.
Basically: if your company is dependent on IT infrastructure, you *must* have someone in your company who understands that infrastructure, its weaknesses, and who has both the ability and access to recover from its failures. Whether you outsource things to a third-party provider or not. And by that point, running a standard Linux server is not a very tall order either.
commentary/addition on blogpost
@joepie91 @ryantownsend thanks. Dokku I will read about. NixOS both me and the person in question are familiar with. I don't remember their opinion of it, but still they expressed strong preference for deploying things on k8s.
commentary/addition on blogpost
@viq @ryantownsend My suspicion would be that that's because tooling and documentation for k8s are better than for NixOS, as that is usually the reason; which can be a valid reason for an immediate choice, of course, but is also somewhat of a chicken-and-egg problem :)
Short of massive capital injections, things are generally only going to improve if people use them, and so it often pays off in the long term to look more closely at the 'minimum complexity' introduced by different options, and select one with a low complexity even if it means a bit more work to figure out how it works. (In general, not just regarding NixOS)
commentary/addition on blogpost
@viq (Of course there are also many PaaS options, as @ryantownsend alluded to, but as those generally come with some form of vendor lock-in, I did not include them here)