Forest @forestjohnson@pixie.town

@nolan @shadowfacts

Sorry about my super long winded, poorly written posts before, the sort of silly analogy I can make goes like this:

In the nation of Mastodon Server #42069, there is a postal service. All outgoing mail goes to the postal outbox. Postal workers drive to the outbox, pick up One (1) message, then read the URL address on it and drive to that server to deliver it. Then they drive back to the outbox & repeat. There are only 12 postal workers. Problem is that when folks follower counts start rising they start getting followers from thousands of different servers, and every time they do anything, it puts thousands of messages in that outbox. Poor 12 workers can't deliver them all 1 at a time even if they were superhuman HTTP client machines.

My opinion: remove the limit on # of postal workers completely, or, if you absolutely can't do that, then try to make more than 1 message per mail truck.

@nolan AIUI, it's basically an architectural issue with Sidekiq. Jobs run synchronously, and one worker thread runs one job, so an in-flight network request blocks an entire worker thread. So you have to put a lot more work into tuning Sidekiq to fully utilize the CPU/network resources.

I strongly suspect anything with green threads (Go, Elixir, etc.) would manage it better, because you can just kick off a green thread for every job and when one suspends while waiting for a network request, the OS thread can just switch to running a different job—so there's always forward progress.

I don't know if those languages have facilities for tuning the runtime based on network bandwidth, but they would at least get you a lot closer to CPU saturation.