Hm. Has anyone done any work on modular databases, by any chance?
That is, databases which don't follow the paradigm of "a single schema for all the persistent state in your application", and which instead let different modules/packages do their own data management, but in a shared data store where data from different packages can link to each other easily.
@rtn I'm familiar with it (and have used some of it), but it's not really what I'm looking for in the sense that it doesn't actually solve any of the hard problems!
Sure, the modules are modular, and you *could* manually assemble it into some sort of modular data management setup. But it leaves the "how" of that undefined (especially things like relational integrity), and that is precisely the thing I am trying to work out here 🙂
@joepie91 Aye. I figured that was the case but I thought I'd mention it just in case. And I agree, it's modular but leaves a lot of work and knowledge to the developer to design the rest.
@joepie91 Not really work on such, but Rama might be interesting to check out if you’re looking for references for the kind of a thing you describe. Ref. https://redplanetlabs.com
The framework as such might not be as interesting as the concepts they’re building it around.
@jhulkko Thanks - it's going to take me a while to read through this, I'm currently reading the "we rewrote Mastodon" post.
So far I'm not really encountering anything new as it seems to all be about the data encapsulation rather than the decoupling of concerns and interoperability (which is the harder part I'm mainly trying to figure out), but perhaps I just haven't gotten to the right section yet :)
The design so far does seem to roughly make sense, so hopefully they also talk about the rest of the owl!
@joepie91 i feel xtdb and datomic enable modularity very well:
Everything is decomposed into qartuples of entity-attribute-value-time. Entity roughly corresponds to row, attribute roughly to column, value to cell and time to a fourth colour, you didn't know was there.
Attributes are namespaced and can be attached to entities without prior coordination. This enables composition of single data points from different modules.
1/2
@joepie91 There is a type of attribute called a ref, which can link to a different entity in a generic way.
This decomposition of tables into tuples and adding time as a dimension brings several benefits, and easier modularity is one of them. It's like taking the most useful graph db schema and making it canon.
2/2
@joepie91 Have you tried out anything from the level ecosystem? Key value store in the bottom. It's built to be modular. https://github.com/level/awesome