Forest @forestjohnson@pixie.town

@nolan @shadowfacts

Honestly I shouldn't have even mentioned the batching concept, 1 request per event is fine.

The main thing is getting that 12 concurrent requests number up, way way up, until its not the limiting factor any more. I am reading about this right now 👀 https://blog.saeloun.com/2022/03/01/ruby-fibers-101.html

Maybe mastodon can use that with its sidekiq, instead of threads, and that fixes it ?

@f0x Well it could be tho.. there's no reason why the keys in IndexedDB can't have the timestamp as the 1st part of the key.. Then different timelines are just different filters on the same sorted set 🤔

I predict that you'll end up de-normalizing the data at least a little bit or at very least having multiple "indexes" within the indexedDB in order to look things up in various different ways.. But I could be wrong lol, Ive never tried to make a fedi client

@nolan

> The problem is that the code is using 90s concurrency technology

Think oldschool apache 1.0 or even older, inetd, versus a modern application like nginx.

The former uses 1 operating system thread per request, while the latter uses epoll / asychronous io / event loops to handle thousands of concurrent requests all on the same OS thread.

This might be a bit of an inflammatory statement but really, Literally ALL the high-scale "fast" web tech uses the multiple-concurrent-things-on-same-thread design. So the fediverse software just has to adopt that if its going to "scale".

Again, It's not a matter of buying a faster computer or using more energy to run the service, it's just a way of having your computer do 1000s of things at once (or even millions depending on the system) without the computer getting bogged down switching between tasks like it does with OS threads.

----------------

Hugo Gameiro of masto.host goes on:

> Then, for each reply you receive to that post, 3K jobs are created, so your followers can see that reply without leaving their server or looking at your profile. Then you reply to the reply you got, another 3K jobs are created and so on.

Another thing I'm noticing here: Really there should be 1 queue for each server that you federate with, or the queue should be somehow "partitioned" on the remote server URL. Instead of spawning a new task for each individual message to a server, maybe it should just build up the events destined for a specific server & then send them all at once in a batch. During normal operation when its not backed up, the batches will only have 1 message, but when it gets backed up, batching could help dramatically. If there are 100 events queued up for the average server, then batching would make the consumption of the queue 100 times faster. IDK if ActivityPub supports this kind of batching tho.

At any rate I've never used Ruby, I could be wrong about a lot of the stuff im saying about OS threads etc with how that relates to Ruby code, but at the end of the day, that 12 number is sticking out like a sore thumb. I'm very very confident that the "actual solution" will be to remove the ceiling on the # of concurrent operations. We have the technology; it's just software. No need to buy a new server.

@nolan

So, I have never used Sidekiq before but Queues of various kinds, distributed systems and their performance issues used to be my bread and butter at my old job.

From the aral post, I think the most important part is to listen to what the server admin is saying about how the problem occurs:

> (you have 23k followers, let’s assume 3k different servers), as soon as you create the post 3k Sidekiq jobs are created. At your current plan you have 12 Sidekiq threads, so to process 3k jobs it will take a while because it can only deal with 12 at a time.

Right away I am already seeing architectural red flags here. If 12 different other servers that I am trying to push updates to are all currently overwhelmed and timing out, it sounds like the entire process will grind to a halt. But there's reason for that; it's not the server who has a lot of push work to do can't do it all in time!!

The problem is that the logic that the server uses to push updates is highly flawed. First of all, an arbitrary limit of 12 in-flight requests max is pretty damn low. I would adjust that up to 1000 or higher. A raspberry pi can handle 1000s of concurrent http requests no problem. HOWEVER I suspect that "worker count" of 12 or whatever it is, is ridiculously low for some reason -- It sounds like each worker is its own OS thread, so spawning 1000s of them could do terrible things to the poor CPU, cause it to spend way too much time context-switching between OS threads.

This sounds like oldschool 1-thread-per-request concurrency. Still works fine in 2022 as long as the # of things happening at once is close to the # of processor cores you have. But when you start having (or wanting to have, for performance reasons) 1000s of things going on at once, you start looking at implementing asynchronous IO properly.

couroutines, goroutines, greenlets, async tasks, lightweight threads, fibers, actors, event loop, epoll, async io, they go by many different names depending on the language du jour. But they are all ways of having 1 single OS thread doing multiple things at once without triggering OS-level context switches.

Key takeaways so far:

--------------------------------------

The problem is that the code is using 90s concurrency technology

Buying a faster computer wont really fix the problem, although it might ameliorate it a bit

Properly fixing the problem involves refactoring the code so that it can do 1000s of things at once.

Note that doing 1000 things at once is totally normal for modern applications and it does not require a fast computer.

@Literally @joepie91 my point is, it only takes 1 person who is capable of giving up their free time in order to get yelled at.
People do the darndest things.

Honestly, I wish more people would yell at **me** lol

@ironiridis @reese 2/2

I've never really liked code using abbrevs or single letter names outside of tight loops even before I took that class.

And back when I used to work as a developer I would often run into comments that had become confusing or misleading as the code changed but folks forgot to update the comments

@ironiridis @reese 1/2 yeah I totally forgot about the part where he's dangerously obsessed with minimizing function size... Mostly I just remembered the naming stuff. Even if this guy is the worst person ever it doesn't disqualify him from having 1 or 2 good ideas. I had no idea about any of this, i just remembered the long names / explain with names instead of with comments concepts. I think they hold up 🤷