Follow
would anyone be angry if i had my own search engine with a simple crawler
like yacy or smth
ill post the user agent if anyone wants to not be indexed by me. that is if it doesnt already comply with robots.txt
🌬️
@lyncia I'll probably be spending my own back up shortly 😊 Those were useful. Even just the searchX interface so I don't have to deal with corpo garbage
itd probabpy use less bandwidth than the ravenous ai scrapers tbh