@theresnotime@fox.nexus Adding this rather important context here as well: that article is quite old, and was written in a very specific context: companies maliciously and/or negligently using robots.txt (for business reasons usually), and then shutting down with almost zero notice, leading to lots of people losing their data and there being no way to get it back.