Forest @forestjohnson@pixie.town

@reese https://funkwhale.audio

@ashfurrow@masto.ashfurrow.com @nolan

I wish you the best and I hope that in the future you will have time for it, or whatever else you wanna do. I saw a lot of posts from mastodon.technology so thanks for hosting it :)

@nolan @shadowfacts Ok, reading more:

> While fibers provide a delightful interface to work with concurrency the real drawback of Ruby and other GIL (global interpreter lock) based languages is that the execution of threads is limited to only one native thread (per process) at a time.

It sounds like it's similar to Node.js, where one can make multiple-process copies of the application which collaborate.. Not unlike the signup table at a large event where you go to a different queue depending on the first letter of your last name, a-f, line 1 g-o line 2, and p-z line 3, something like that

@ashfurrow@masto.ashfurrow.com @nolan I still think that if this operation of pushing/federating posts to followers' servers was implemented in "lightweight threads / async io" all the way through, no limit on # of concurrent transactions, it would improve the throughput dramatically all on the same machine without a hardware upgrade. Especially if it can use lock-free techniques like partitioning. But also, I've never written a line of ruby nor cloned mastodon's code, so I have no idea how much of an undertaking that would really be.

@ashfurrow@masto.ashfurrow.com @nolan Ah nice, so maybe its the database driver that needs to work based on fibers instead of a fixed-size thread pool?

Or maybe a different way of doing the required transaction might free up the system from this tyrannical "12" limit.

@nolan @shadowfacts

But the "more than 1 message per mail truck" sounds like its not in the protocol. So I think fibers is the only way for mastodon.

@nolan @shadowfacts

Sorry about my super long winded, poorly written posts before, the sort of silly analogy I can make goes like this:

In the nation of Mastodon Server #42069, there is a postal service. All outgoing mail goes to the postal outbox. Postal workers drive to the outbox, pick up One (1) message, then read the URL address on it and drive to that server to deliver it. Then they drive back to the outbox & repeat. There are only 12 postal workers. Problem is that when folks follower counts start rising they start getting followers from thousands of different servers, and every time they do anything, it puts thousands of messages in that outbox. Poor 12 workers can't deliver them all 1 at a time even if they were superhuman HTTP client machines.

My opinion: remove the limit on # of postal workers completely, or, if you absolutely can't do that, then try to make more than 1 message per mail truck.

@nolan @shadowfacts

Honestly I shouldn't have even mentioned the batching concept, 1 request per event is fine.

The main thing is getting that 12 concurrent requests number up, way way up, until its not the limiting factor any more. I am reading about this right now 👀 https://blog.saeloun.com/2022/03/01/ruby-fibers-101.html

Maybe mastodon can use that with its sidekiq, instead of threads, and that fixes it ?

@f0x Well it could be tho.. there's no reason why the keys in IndexedDB can't have the timestamp as the 1st part of the key.. Then different timelines are just different filters on the same sorted set 🤔

I predict that you'll end up de-normalizing the data at least a little bit or at very least having multiple "indexes" within the indexedDB in order to look things up in various different ways.. But I could be wrong lol, Ive never tried to make a fedi client

@nolan

> The problem is that the code is using 90s concurrency technology

Think oldschool apache 1.0 or even older, inetd, versus a modern application like nginx.

The former uses 1 operating system thread per request, while the latter uses epoll / asychronous io / event loops to handle thousands of concurrent requests all on the same OS thread.

This might be a bit of an inflammatory statement but really, Literally ALL the high-scale "fast" web tech uses the multiple-concurrent-things-on-same-thread design. So the fediverse software just has to adopt that if its going to "scale".

Again, It's not a matter of buying a faster computer or using more energy to run the service, it's just a way of having your computer do 1000s of things at once (or even millions depending on the system) without the computer getting bogged down switching between tasks like it does with OS threads.

----------------

Hugo Gameiro of masto.host goes on:

> Then, for each reply you receive to that post, 3K jobs are created, so your followers can see that reply without leaving their server or looking at your profile. Then you reply to the reply you got, another 3K jobs are created and so on.

Another thing I'm noticing here: Really there should be 1 queue for each server that you federate with, or the queue should be somehow "partitioned" on the remote server URL. Instead of spawning a new task for each individual message to a server, maybe it should just build up the events destined for a specific server & then send them all at once in a batch. During normal operation when its not backed up, the batches will only have 1 message, but when it gets backed up, batching could help dramatically. If there are 100 events queued up for the average server, then batching would make the consumption of the queue 100 times faster. IDK if ActivityPub supports this kind of batching tho.

At any rate I've never used Ruby, I could be wrong about a lot of the stuff im saying about OS threads etc with how that relates to Ruby code, but at the end of the day, that 12 number is sticking out like a sore thumb. I'm very very confident that the "actual solution" will be to remove the ceiling on the # of concurrent operations. We have the technology; it's just software. No need to buy a new server.