Mastodon fully operational again Matrix operational - irc bridges stuck on weird ipv6 issue - telegram bridge stuck on python bullshit Gitea up WriteAs up Plume mirror up
terra & helios fully upgraded and freshly rebooted after like 200 days
maybe next time shit breaks I'll practice writing formal post-mortems, but not today
irc bridges have an IP_FREEBIND setup so they can use all ipv6 addresses in a block for outgoing connections (one per user), which conflicts with certain iptables MASQUARADE rules for wireguard routing, so temporarliy routing that outgoing traffic through luna again.
and the upgrade broke python shit (as always..) and virtualenvs had to be reset too.. (garbage)
everything is up and running again 🎉 🎊 🎉 🎊
maybe next time shit breaks I'll practice writing formal post-mortems, but not today
irc bridges have an IP_FREEBIND setup so they can use all ipv6 addresses in a block for outgoing connections (one per user), which conflicts with certain iptables MASQUARADE rules for wireguard routing, so temporarliy routing that outgoing traffic through luna again.
and the upgrade broke python shit (as always..) and virtualenvs had to be reset too.. (garbage)