Cloud "oops turns out our distributed system is centralized in a single datacenter" Flare

"Well, we *thought* we had High Availability, but we never actually tested that"

- Cloudflare, supposed distributed systems experts, processing a double-digit percentage of the world's web traffic

I heard there's this well-tested distributed network routing protocol, that is extremely resilient to provider failures. I think it's called BGP - perhaps we should all be using that instead.

🤔 Cloudflare seems to actually have pissed off the HN commenters this time

Some other juicy bits about the #Cloudflare outage:
- No 24/7 (experienced) technician availability at the datacenter that hosted their control plane(!)
- No end-to-end service dependency tracking or diagrams
- Therefore, supposedly HA services depending on non-HA infrastructure
- Even if the "redundant" setup *did* work (it didn't), all three locations would be physically within *the same earthquake zone*

This is absolute clowncar level network administration, frankly, for something the size and importance of Cloudflare.