why is cloudflare blocking an internet archive address as an AI scraper
why is the internet archive requesting pages with a fake user agent
@ben It actually might not (technically) be fake; I think there was an experiment at some point to archive some things with a headless browser that wouldn't be navigable in a more traditional scraper, similar to how search engines sometimes do this
@joepie91 @ben interesting
I was looking up something else I recalled that cloudflare had and found the exact user agent in tech info page for cloudflare always online thing
the thing that uses the internet archive when the website is down
I—
Small server part of the pixie.town infrastructure. Registration is closed.
@joepie91 @ben interesting
I was looking up something else I recalled that cloudflare had and found the exact user agent in tech info page for cloudflare always online thing
the thing that uses the internet archive when the website is down
I—