Follow

cyberia.club matrix server outage RCA

RCA is that a cyberia.club user was in a 4chan-associated room on matrix.org which was experiencing an extreme volume of hate message spam from a common "chud" server.

Since we had an incomplete block on the offending server / since matrix-synapse's block feature is buggy, this resulted in what appeared to be an ` O(n*m)` number of error messages in our log file, where `n` is the # of spam messages and `m` is the average number of times that federation was attempted for each message. This log spam filled our disk and caused matrix to crash.

insight was gained by creating a histogram of bytes-per-minute in the log file, and by examining the rooms related to those high-volume error messages.

The issue was resolved by "purging" the room via the matrix administration API, (kick all local users, delete, & block) and then manually deleting all `federation_inbound_events_staging` rows associated with that room from the database.

AHHHH I was wrong turns out it was a typo bug in the matrix version we were using. Updating to matrix-synapse v1.50.2 fixes it.

Sign in to participate in the conversation
Pixietown

Small server part of the pixie.town infrastructure. Registration is closed.