why is cloudflare blocking an internet archive address as an AI scraper

why is the internet archive requesting pages with a fake user agent

Follow

@ben It actually might not (technically) be fake; I think there was an experiment at some point to archive some things with a headless browser that wouldn't be navigable in a more traditional scraper, similar to how search engines sometimes do this

· · Web · 1 · 0 · 4

@joepie91 @ben interesting

I was looking up something else I recalled that cloudflare had and found the exact user agent in tech info page for cloudflare always online thing

the thing that uses the internet archive when the website is down

I—

Sign in to participate in the conversation
Pixietown

Small server part of the pixie.town infrastructure. Registration is closed.