Forest @forestjohnson@pixie.town

@djh I'm starting to remember this now.. The z curve has slightly more discontinuities iirc but its also easier to find the sub sections. My method is kinda "brute force" based on sampling a reduced resolution version of the curve in order to decide where to split up the sections before running the actual DB queries.

But in my opinion, the most important thing to call out about all of this is that you don't need different database software / you don't need to modify the database at all. The app that talks to the database just needs to Use this "one weird trick" to generate keys and query ranges.

@djh

i don't understand what the bigmin thing is, Are you talking about an in-memory search or a database (on disk)?

I made my own version of software that does this. It uses a different space filling curve but the idea is the same. How I thought abt the query running really far outside the requested area: essentially you have to decide whether you want to prioritize fewer individual IO operations or less wasted IO bandwidth.

Since data on disks is laid out as one sequence, typically it's faster for a disk to read a little bit more data if it only has to do one single sequential scan, as opposed to reading a bunch of different little bits and pieces.

My solution simply downsampled the curve until It was no sweat for the computer to make a list of every single point on the curve within the queried area. From there, I settled on an algorithm that looked at the points on the curve and decided how many segments it would split them into. If the distance between two points was larger than the queried area, then I wouldnt join those two points into a contiguous segment, I would split the segments at that point.

The cool thing: this 1 rule perfectly expresses the trade-off between how many IO operations and how much wasted bandwidth. And it's tweakable at query time, not at indexing time. So you can set a coefficient for that threshold depending on the performance characteristics of your disk when handling your given data set.

https://sequentialread.com/building-a-spatial-index-supporting-range-query-using-space-filling-hilbert-curve/

@jamiem @notplants @mayfirst

I'm blocking the ui thread until I know whether the email was accepted or rejected. So I can't just wait 30 seconds for a bounce notif and then assume success. No one would use this site if you have to wait 30s every time you log in

@jamiem @notplants @mayfirst

I had considered that option, but I decided against it because I want to be able to tell the difference between time out and a success.

@drahardja get in the Eva shingy