Forest @forestjohnson@pixie.town

Here is the pull request to fix the issue in matrix-synapse

https://github.com/matrix-org/synapse/pull/11819

cyberia.club matrix server outage RCA

RCA is that a cyberia.club user was in a 4chan-associated room on matrix.org which was experiencing an extreme volume of hate message spam from a common "chud" server.

Since we had an incomplete block on the offending server / since matrix-synapse's block feature is buggy, this resulted in what appeared to be an ` O(n*m)` number of error messages in our log file, where `n` is the # of spam messages and `m` is the average number of times that federation was attempted for each message. This log spam filled our disk and caused matrix to crash.

insight was gained by creating a histogram of bytes-per-minute in the log file, and by examining the rooms related to those high-volume error messages.

The issue was resolved by "purging" the room via the matrix administration API, (kick all local users, delete, & block) and then manually deleting all `federation_inbound_events_staging` rows associated with that room from the database.

this is what I worked on today

https://git.sequentialread.com/forest/config-lite

streaming again today https://stream.sequentialread.com/

Music rn:
✖️✖️🐧🐧 Wasted Penguinz

Netizens are always telling me that self-hosting is too difficult for the average person, citing security risks and complexity.

Well, security risks and complexity didn't seem to stop everyone from running a web browser!

I believe the difference boils down to investment. It's harder to make a profit from helping folks self-host. "Teach a Man to Fish" and all. So the industry never worked on making self-hosting easy and "normal" in the same way that web browsing was made easy and "normal".

How did they do it for web browsing? A whole heck of a lot of work, mostly around user interface design and usability. The tech giants of today, Apple, Microsoft, and Amazon all got their start by imagining new, easier to navigate interfaces that everyone could use, not just the tech elite.

I'm not out to "make it big" in the same way that they did, but I **AM** inspired by the change-making disruptive power that they wielded, especially the transformative power of software usability.

Most of the hosting and server software of today is designed by and for technical people, almost exclusively in professional contexts. Their designs either start from the business goals or from the computer hardware.

For contrast, successful designs for user-focused products (client software) unsurprisingly always start with the user & how to interface with the user, everything else is derived from that.

But the boots-on-the-ground digital laborer who is tasked with deploying and maintaining these systems is almost never considered in the design process. They are simply expected to be "professional" and to be able to handle it. Any ease-of-use or convenience they enjoy was probably "snuck in" on the side, while the manager wasn't looking.

I see this tendency towards profit-driven and inhumane/elitist design in the server software arena as the ultimate "root cause" to explain why hosting / application security is hard, as well as why so many security vulnerabilities and server misconfigurations exist in the first place.

It's not easy because no one was ever paid to make it easy. Because if you gave it all away, what would money, power, and software platforms be worth?

Well, spoiler alert, I think it would be worth a hell of a lot, actually. The potential benefit to everyone & to the economy and society would be enormous.

So I start all of my designs from the self-hoster, the individual person maintaining the system. From there, I work towards the "hardware" and the real world, things like the "dorm room LAN" network environment, the Total Cost of Ownership of a server, etc.

Ultimately, I think I'm trying to bring imaginative & disruptive "big-tech" usability to the world of "small-tech" open-source self-hosting platforms and tools.

I spent all my time today finally getting around to fixing that reflection-based environment variable config override thingy

@dumpsterqueer : I was talking about this back when you were thinking about refactoring the configuration library for GTS

I know the ship has already sailed and I'm not recommending it for GTS; I just wanna show it off I guess 😇

It's a bit arcane but it appears to actually work; and the error/warning logs are the best ive ever seen on a library like this: https://git.sequentialread.com/forest/file-and-or-env-config

OMFG brain fart, I meant **WireGuard** not shark 🦈🦈🦈

streaming today, working on version 2 (from-scratch rewrite) of my threshold reverse tunnel app

https://stream.sequentialread.com/

music today: 🔥💀 AFI ❄️🥀

now with QUIC, hopefully able to compete with Wireshark on latency

im cooking up a hair brained scheme to host twitch.tv out of moms basement

Finally now we're cookin!! down to about 250ms to join a torrent swarm and torrent a file 😎 Here are some stream clips!!

- furiously looking up information on node.js webtorrent/bittorrent tracker (part 1, typescript/socket listening issues, video cuts out)

14min https://picopublish.sequentialread.com/files/stream_clip_2022-01-02_13-46/

- part 2, partial success after I realized first clip got messed up

6min https://picopublish.sequentialread.com/files/stream_clip_2022-01-02_15-09/

part 3, two torrents completed in 500ms

6min https://picopublish.sequentialread.com/files/stream_clip_2022-01-02_17-20/

writing typescript type defs for webtorrent/bittorrent-tracker https://stream.sequentialread.com/

music: 🍄 Gnome 🧙🎸

streaming again today, more WebTorrent adventures https://stream.sequentialread.com/

😋

back at it again https://stream.sequentialread.com/

hopefully I can torrent some bytes today. music: gold panda

continuing my bad habit of starting a million projects: https://git.sequentialread.com/forest/tuber

🍠 Tuber: Serve Your Media Without Limits From a "Potato" Computer Hosted in Mom's Basement: Take our Beloved "Series of Tubes" to Full Power

Nowhere close to operational yet but as far as I can tell there's no reason why it won't work? I'm excited about this one.

I'm streaming now, finally taking some time to start working on that p2p cdn idea i had

https://stream.sequentialread.com

The idea is its an HTTP reverse proxy you can put on any web server and you can configure it to offer media files and downloads over WebTorrent instead of over HTTP
inspired by https://github.com/Novage/p2p-media-loader but for HTTP in general instead of just the <video> and <audio> elements in a page

Music rn: 🎹🌌 MGMT 🌀🎸

We got a little carried away with the little cakes decoration process

I'm proud of what cyberia.club has done with capsul.org, our home-grown "cloud" VPS provider over the past couple of years. We keep it running day in and day out, and others have joined us in our repos / made a fork, embarking on their own journey.

https://sequentialread.com/capsul-rumors-my-demise-greatly-exaggerated/

@j3s

Every time I travel south from my icy domain (Minnesota) I'm reminded of how many homeless people are out there struggling. We don't see nearly as many homeless folks daily up here because the winter will either force them out or kill them.

I met a lot of homeless folks in Atlanta. The Covid economy has been cruel to many.

One man I spoke to told me about how his mother never really loved him, and always told him that you can't change the world, and it would be silly to try. But he found God at a homeless mission, and realized how much he had been lied to and abused by the system.

After I mentioned I was traveling/just a visitor, he asked what brought me here, so I tried to explain that I was part of a group who're trying to make it possible for anyone to "own" their own presence online as a part of a community that they have a say in, not like how it is with facebook or google. To my surprise, he instantly picked up on what I was saying and seemed to be cheered up by hearing about it. I explained that the server was as if it was "a car running without oil" and he told me to make sure I oil that machine well.

I really hope at some point our society can progress to the point where folks don't get left behind and kicked to the curb. This guy was shivering cold standing under an awning to stay out of the rain.

I just returned from Atlanta, Georgia where @j3s and I did maintenance on Baikal, cyberia's server and the workhorse behind https://capsul.org

Baikal has been running for years with an inefficient QEMU VirtIO configuration which caused its SSD disks to wear out much faster than they should, and our decision to use NVME drives for some of the disks in the ZFS pool almost prevented us from being able to expand disk capacity to meet demand: We kept mailing PCI-e to NVME adapters to the datacenter and they kept saying "It doesn't fit".

Life happened and it got kinda bad, with a legitimate risk of everything going kablooie, so we had to work hard to get everything fixed in short order and ensure that our little project has a future. I suppose it's not-so-little anymore, depending on your perspective.

Attached are a couple fun pictures from the trip. You can also see a video I took of us replacing one of the SSDs that was wearing out here: https://picopublish.sequentialread.com/files/zfs_disk_replacement/

I was sweating bullets while j3s carefully pulled the disk out of the server while it was running 😅