I find it slightly ironic that I found out that my suspicion of robots.txt matching in my crawler being broken was correct, by noticing it trying to crawl a Twitter page, a site which famously disallows any and all robots.

Time to fix that I suppose, and figure out why on earth it's not working...

Aha! Accidentally case-sensitive regex strikes again

Great, robots.txt parsing is now working correctly!

At first glance, I seem to be getting a roughly 4% robots.txt rejection rate on a pile of personal websites, which is honestly lower than I had expected

My search engine crawler so far seems to be perfectly fine with about 20MB of heap size (ie. "true" memory use), though it doesn't do the actual content indexing yet. Promising results so far though!

lmfao, I've implemented a feature that ignores tech company sites effectively by looking for any occurrence of the text 'Pricing' as the sole content of a HTML element

It's absolutely hilarious that this works

Not happy about the CPU time yet; it's currently clocking somewhere between 15 and 50 milliseconds of CPU time per request, which really isn't acceptable, so that needs much more optimization I think. Profiler time, I guess?

I have extended my tech corp detection to corps in general by also detecting a "Careers" button