@angilly i appreciate you taking these concerns in stride. In the future though, I'd recommend having the public discussion *before* starting to collect any data
I have a separate instance that doesn't really federate, and it's been quite 'interesting' to monitor the logs for all these scrapers.
Also Friendica is very noisy for some reason? requesting a bunch of endpoints on discovery even tho you don't really federate with them
@angilly on my instances these requests get denied, yes. but just because data is public does not mean people consent to it being scraped, processed, and published elsewhere. If arbitrary full-text search was desireable on the fediverse, it would've been there.
It's not a technical challenge that needs solving, it's a social decision to do without it.
https://docs.joinmastodon.org/user/network/#search
> It deliberately does not allow searching for arbitrary strings in the entire database, in order to reduce the risk of abuse by people searching for controversial terms to find people to dogpile.
https://blog.joinmastodon.org/2018/07/cage-the-mastodon/
> Mastodon deliberately does not support arbitrary search
Assorted list of Fediverse scrapers you might want to block on the network level (some more malicious than others)
https://git.pixie.town/f0x/nixos/src/branch/main/nodes/aura/configuration.nix#L100
another scraper by an (ex-?)twitter employee, running without recognizable user-agent (uses the typhoeus library with defaults)
creating some full-text search federation wide search... https://macaw.social/@angilly/109597402157254670
fediverse.network scrapes a whole range of endpoints, but the site it refers to is just a parked domain
another scraper in my logs, "fedimapper" by https://hachyderm.io/@tedivm/109569119442998113
not sure why it needs to scrape my domain_blocks with that stated purpose
@researchfairy it's a type of (tasty) pastry from the balkans https://en.wikipedia.org/wiki/B%C3%B6rek
@researchfairy the Börek
@TakeV like two ships federating in the night
https://arloparks.bandcamp.com/album/collapsed-in-sunbeams
still so long till the next #BandcampFriday though (February 3rd)
@dragon hey heavy metal deathtrap trying to avoid others is my band name, how did you know
@kim slacks off (it's a video meeting anyways) slack on
@dragon high on car-centric infrastructure and control of a metal deathmachine
maybe i should try writing happy poetry lol, but then it's easy to get sappy? can't win huh (and can't write, anyways)
@kim time for some.... slacking off
⚠️ READ BEFORE FOLLOWING ⚠️
if i don't know you from elsewhere (under same nick), shoot me an introductory DM first (following back is fine)
I do anarchist tech stuff and run free services at https://pixie.town
I program, solder rgb led thingies, and fly fpv quadcopters
en: they/them
nl: die/dies (langzaldieleven.nl)
“i don't trust like that”
not a furry, actually
Extreme coffee-out-of-a-wineglass Energy
something something trans list stop scraping bios
and now a word from our sponsors (screenreader warning it's zalgo)
T̀ͧ̓̑͐̓̍̂̏҉̴̷͚̦̤͙̜̖͙̝͟ợ̵͈̗̮̲̥͕̼̩̭̞̙͉̆ͮͧ̉̒́̑̍̋ͭ̌ͭ̒̉́̕͟ ̐̅̈́ͯ҉̸̴҉̹̟͕̖̠̟̤͕į̸̙̮͓̤̠̘̫̦̥̣̻͚̣̎ͭͯ̋̉͝n̔̄̏̈́̃̇͛̂̋̇̐́͘͝҉͙͔̠͇̖̤̹̭̱̪v̴̴̛̘̠̰̹͚̱͉̳̘̥̞̳̪͈ͥͭ̅ͥͦ̀͛̔̃̃̎͋̋̎͐͌ͪ̚͟͢ͅö́́̎ͬ̔͑̆̃̅̒̿ͪͯ̓͏̞̱̜͍̬̗̹̫̝̪͓͕̳̬̰͘͝kͥ̒ͣͦ̌͛̃͒̀̿ͣͪͤͬ̍ͮ̚̚̕͝҉̹̰̟̰̻̻͍̠̗̳̬̬̬̞̟̹̩͇́͜ẹ̴̡̨̱̹͍̯̱̗̗͍̬̐ͣ̑͑̐̓̈̑ͥ̅́̇̃͒̀̃̂́ ̨̛͖̬͇̣͔̼̥̬̝̥̣̭̝̪͎͈̌̅͆̉̀͘͜ͅẗ́̄͊̌̍̆́̿́̊ͣͮ̅ͥͩ̔̏͏̧̳͎̥͈ͅh̴̴͇̻ͧ̍̐̈͐̎͛́̀̽̃̒̔͢͢ȩ̸̶̶̟̗̮̺̭̥͕̭͎̺̙͎̖͔ͪ̑͛̓̅ͪ̄́ͧ͡ͅ ̡̧͇̤͚̻̬͉͔̥̫̟̙ͮͩ͌̿́̆͋͜h̵̨̭̰͎̭̱͊͒́͒͆̎ͮ̈́̆ͪͧ̚͞î̛̦̞͓͖̭͈̮͔̩͙̱̖̞̳̥̦̩ͭ̂̏͒ͨ̃̿̽̓͑ͫ̕͝͡vͧ͋ͪ̌̂̑́͌̂̒͑ͮ̋̂ͫ̈́҉̹͜͢ȩ̡̖̯̞̺̭̗͔͇̻̤̼͈̙̞͉͙̈ͤ͊ͨ̀̆͆͒̓̄̿ͭ̃̚͜͝͡-̶̪̪̠̝̜̯̜̹̭̯͎͍̲̱͉ͪ̏͒̊ͫ̀̈͘͡m̸̪̘͙̰͚̗̳͕̟̖̿̌͐̔̐̈̽̃ͯ̅͢ͅͅi̸̷̧̛͍̝̦̫̮̤̐͑͗̏ͬn̡̨͆ͩͤͫ̔̈́̈́͊͐̂͛̀̚͞҉̜͍̝̰̱͚̜̹̞̝̞͈d̢̫͕͚͕̥̰̝͆͗́ͨ͑̈́̓͜ ̡̩̜͎̳͎͂̓ͫͭ͐̀͡ȑ̷ͭ̑ͪͭ͋͢͏͕̳̟͜ͅͅe̴͌̅ͣ̾͒̔́̊̔ͭ̅̄̇͏͎͉͈̤̙p̀ͥ̈ͨͩ͛ͥͣ͗̄̈́̚҉̢͔͉͍̹̮͉̺r̵̸̡̩͎̱̟̺̟̞͈̯̯̪̹͂́ͣ̐͑̒̒̀ͧͩ̿ͮ̕͞ě̵̡̱͈̜̯̳͍̝̦̜̫͈̜̗̘̪̪̓͆͑͋ͮͯͪ̅̂͐̔̆̃ͫ͑̾͒͢ͅş̶͓͉͚̜̪̜͓̘̻̃̔ͨ́̀ͅẻ̵͇͈̮̝̠͖͍̫͉͓̪̠͔̬͕͛̊͐̎̓̽ͫ̌ͧ̅̿́͘n̛͚̺͈͍̰͉͙̤̘̺͖͉̤͖̈͑͑̍̅ͪ̎͂́ͦ̒ͣ̋̆̄̄̍̃̊͟t̵̛͙͚̥͇̫̻̞͖͕̰͈̩̰̱͉ͣ̃ͫ̋̍̈ͥ͗̎ͭ͋͜i̵̡̤͇̣̰̦̟̭̮̩̲͔̭̟̖̹̙ͥ̆̋ͫ̓͌̒̾̍̄̾̎̂͂̏̇ͩ̚͢n̶̮̹̤̻͈̙͔͎̦̟ͫ̀͌͛̋̌̽̀̓̂̕g̷̣͖̠̩͈̲̥͍̦̘̺̏̍͛͋̎͛͒ͪ̇ͮ͠͝ ͦ͂́̿͐̅̌̊̌̉̍̀҉҉͈͖̮̩͎̮̬͖c͖̬̠̫̠̫̗̉̾͋͒̏̄̈́ͬ̊̓͘͝h̴̷̨͉͖̱̗̪̣͕̮͓͕̺͖͈͙̥̬͓̟ͣ̏̀͐̀́̍ͪ̋͒͐ͪ͐́̕a͍͈͉͎̥̠͍͛ͭ͛̃ͫ͒͋́͟ö͙̻͔̙͖̰́̋̑́͜s̶̸̫̖̫͇̣̻̺̹͔ͧ͐̂̈́ͮ͋̌͠.̰̯̞͎̗̺̠͔̫͍̖ͮͦ̒̏̈̾ͭͧ̉͘͢͠