Details are here, for anyone who wants to see some concrete numbers: https://github.com/oxigraph/oxigraph/discussions/924
I should probably start tagging these project update posts with #seekseek or something.
Anyway, current status: rewrite of the scraping backend is almost done. I'm a lot happier with this version than with the previous one, and this one should be a lot more suitable for the original goal of making it easier for people to build their own search engines.
Some big items remaining: switching to embedded Oxigraph instead of a stand-alone server (requires writing some Neon/Rust bindings), rewiring the code so that it can actually load multiple configuration modules with their own namespaces (as it's meant to do), implementing auto-expiry of dependents, worker threads, custom TTLs, and converting existing scraper modules to the new API.
The API didn't change *much*, but enough to need some changes. That should actually end up simplifying the modules!
Speaking of which: if you're interested in building your own search engine for something, and want to test out this software, let me know! All it should require to know is basic (JS) programming knowledge, and jQuery syntax. The backend handles the rest of the complexity. The software will run on a laptop easily.
(For hopefully obvious reasons, I will not assist with unethical projects like scraping personal information)
seekseek.org devlog #2
Today I'm fixing the last few issues remaining before I can start writing real-world scrapers for the new backend!
I think all the major outstanding issues have now been fixed, though I'm sure that as soon as I start trying to use it, I will find a few stragglers.
Meanwhile, listening to a set by Mandidextrous