Show newer

My search engine crawler so far seems to be perfectly fine with about 20MB of heap size (ie. "true" memory use), though it doesn't do the actual content indexing yet. Promising results so far though!

Show thread

laziness is a myth created to shame poor, working class and disabled people into working their life away.

you have value even if you cant work. you have value even if all you do is survive. fuck this shit and burn it down.

never seen a billionaire called lazy? yeah there’s a reason.

At first glance, I seem to be getting a roughly 4% robots.txt rejection rate on a pile of personal websites, which is honestly lower than I had expected

Show thread

@arianvp Yeeeep. I've had much the same experience with Google's JS libraries.

idle thought maybe hot take 

The whole thing with "official fan $thing" and the paradox therein, seems like it's really just a version of the "temporarily embarrassed millionaire" phenomenon, where someone is trying to appear grassroots and independent, but really their actual hope in doing so is to become the Big Guy, not for legitimate grassroots reasons

@arianvp That's one of the industries it happens for, yeah, the thing with the infosec industry is that they're extremely easy to buy goodwill from, you just hire a couple security folks who are well-liked in the community, have them write a couple really interesting and cool blogposts with knowledge that's hard to find in a concise format elsewhere, and bam, you've bought the market.

See also: Cloudflare, Tailscale, ...

(There's some other industries where this happens, DevOps is another one, but it's why this particular predatory model happens so often in specific industries)

@diedofheartbreak@plush.city Thankfully I'm running at an extremely slow crawling interval in dev mode precisely so that this doesn't happen if I fuck something up 😅

Disentangling colonial thought patterns, whiteness is a death cult (reading links on extinction, genocide, colonialism) 

When you wish extinction on *everyone*, you're wishing violence on the people we forced to be complicit in our sins; through slavery, through colonialism, through domination and subjugation, through abusively wielding global power.

The human race is not the problem. White people and white supremacy are the problem.

Show thread

I find it slightly ironic that I found out that my suspicion of robots.txt matching in my crawler being broken was correct, by noticing it trying to crawl a Twitter page, a site which famously disallows any and all robots.

Time to fix that I suppose, and figure out why on earth it's not working...

bitwarden, enshittification, etc., personal frustration 

@zrb@astrodon.social I haven't; the problem with password managers is that it's really easy to implement them wrong in subtle ways, and so I couldn't really recommend anything unless I know it's been reviewed by trustworthy security experts...

bitwarden, enshittification, etc., personal frustration (2) 

So like, consider this your early warning for Tailscale and all the other currently-hip "open-source-friendly" things that are rapidly buying goodwill with neat technical blogposts. We'll see where they are in a year or five.

(And this one isn't exactly an 'early' warning anymore, but Microsoft "embracing open-source" is absolutely one of these as well.)

Show thread

bitwarden, enshittification, etc., personal frustration 

And once again, a "FOSS" thing that people *assured* me was legitimate FOSS when I refused to use it because of its commercial VC-y setup... is starting to lock down their project after increased VC funding.

How many times does this need to happen before people stop trying to argue about this shit with me and just accept that there is a very good reason I will not touch anything that looks VC-y, no matter how "we're FOSS, honest, promise" it claims to be?

When will people learn that corporate FOSS with an unclear business model *will* end up backstabbing them, and it's just a matter of 'when', not a matter of 'if'?

"But this one is different!" No it's fucking not, and I fucking *told* you that this would happen and why, but you didn't want to listen did you?

@lianna@mastodon.gamedev.place Different kind of project :) Marginalia intends to index the web broadly, but I have different goals - hence also why I scoped my question to just robots.txt

: What would the most sensible way to implement robots.txt for a search engine project? The intended behaviour is to be reasonably conservative; ideally it should tolerate some weirdness in the robots.txt and default to "do not crawl" in cases of doubt as to the author's intention, but still have good coverage of sites that clearly did not intend to block crawlers (and default to "crawl" if there is no robots.txt or equivalent at all).

(Asking because there's not one robots.txt standard, and perhaps people here have preferences on what variant is the best choice to support here)

Ik probeer mijn leerlingen steeds weer bij te brengen dat ze altijd kritisch moeten blijven nadenken en volledig zelfstandig moeten durven zijn, maar vervelend genoeg is er dan altijd wel ééntje die daar niet zomaar in mee wil gaan. 🙄

@freakazoid Actually, a better example: the common implementation of "meritocracy" in FOSS is pretty much exactly what you describe, and it basically always results in a toxic environment. Likewise, meritocracy has a kernel of truth to it, but it can only work as part of a bigger change.

The ADHD experience of coming back in the morning to one's browser with a couple hundred open tabs: "did I already read this article, or did I just skim it and leave it for later?" 😓

@freakazoid I feel like that's overly optimistic, honestly. It's not entirely *wrong* - removing money does remove an avenue of power accumulation. But... it is not the only one, and unless you also get rid of the underlying power-seeking culture, people will find another way to accumulate and misuse power, most likely reinventing money, or religion, or something like it.

Getting rid of money can definitely *help*, in that it removes a pillar of power, and so weakens the overall system of hierarchy, and makes it easier to knock it over. But you do still need to fight the concept of 'deserved' hierarchy itself to actually get there!

(You can see this happening at a small scale with the many local initiatives that try to exist 'without money', but then do not recognize or address power relations otherwise, and ultimately end up succumbing to some power struggle from out of left field)

Show older
Pixietown

Small server part of the pixie.town infrastructure. Registration is closed.