I'm genuinely stuck. The Matrix S2S spec seems effectively unimplementable in its current state, with core functionality being undefined, which leaves me with only a few options:
1. Try to implement it anyway, and be chasing undefined behaviour and broken rooms for the next 5 years
2. Try to get all the holes in the spec resolved, which will probably take a decade at this pace, if it ever succeeds at all (and will burn me out, guaranteed)
3. Fork the protocol into something that's actually strictly specified, but this will fragment the ecosystem because of missing compatibility
4. Just give up on implementing a Matrix server. Take the (extremely rare for me) decision to just abandon this project entirely.
All of these options suck. I see no remaining solutions until the Matrix core/spec developers actually start prioritizing fixing the spec over other things.
Now what the hell do I do?
@joepie91 @ShadowJonathan I totally see your pain, though in this case forking seems very counter productive unless there is considerably large community ready to push the new effort forward.
Matrix itself will hit hard walls of their own doing unless those problems will get resolved. E.g. JSON serialization is a mess and the protocol will never work as intended between different implementations unless that stuff gets fixed.
So, in this case the best way forward would probably be fixing the spec and related documentation, or at least put as much effort to that as viable.
If you are to consider alternatives, then Simplex protocol might be worth checking out as well. Personally I haven’t read that spec, but few people seem to like it.
@jhulkko @ShadowJonathan@tech.lgbt I literally cannot fix the spec problems, because there is no way for me to know what goes in the missing bits - that information is locked up in the heads of the core developers.
And I've already put a ton of work towards filing issues and figuring things out etc., and it's just not sustainable - there's insufficient proactive cooperation from core devs to actually get anywhere.
@joepie91 @ShadowJonathan That might actually be a problem related to the complexity of the overall system, to a degree that some parts of the spec just feel too daunting to touch for about anyone involved.
Neil tried to push that kind of core spec changes forward from within the organisation, but could not get anything forward despite his efforts.
I would assume solving the problems in a way that provides reasonable migration path, and having a preliminary implementation to show viability would be needed to push such changes forward.
Matrix protocol kind of has ”technical debt” in some of its core parts and specs that is hard to justify repairing due to effort and breakage required. Until it will be too painful not to. And currently that pain is only suffered by community projects and the Dendrite devs. Synapse has no problem to remain compatible with itself, and that pain is not really visible there.
@jhulkko @ShadowJonathan@tech.lgbt If this is the case, and "we just don't know how most of the protocol works, and we can't justify fixing it" is accepted in core as the status quo...
Then I have to ask, by what definition is this an open protocol?
@joepie91 @ShadowJonathan Everything in it is open, but not perfect. And each contributing entity had their own set of priorities, which affect decisions on what they actively push forward as their own effort.
I don’t think the project would block advancements in the core spec area, if the effort required from the spec team is running MSCs through the process with viable prototype implementations etc in place. Problem here is probably the amount of effort required that the core project should find resources for, if these changes are expected to be figured out by them based on problems and bug tickets.
The foundation still relies too much on single entity paying for the effort, and that can never be without side effects. I don’t believe there is malice involved in this.
@jhulkko @ShadowJonathan@tech.lgbt Is it actually open *in practice*, though, if implementing the spec does not result in a compatible implementation? (Which it definitely will not, this is well-established)
Like, Microsoft got (rightly) raked over the coals for publishing an "open" document format for Office and then immediately deviating from that in their own implementation, making all *real-world* Office documents incompatible with spec implementations. This was concluded to not be a true open standard.
Aside from the intentions, what's the material difference with this situation as it stands?
@jhulkko @ShadowJonathan@tech.lgbt (And if I'm being honest, I think this is almost entirely a problem of prioritization - the handful of people who *could* do this job are all busy doing things that many others could be doing, instead of the things that only they can do...)
@jhulkko @ShadowJonathan@tech.lgbt As for "blocking core spec work" - I have made suggestions for procedural improvements in the past, but these were shot down.
Ultimately this all hinges on the core team; tasks need to be done that other people simply *cannot* do, because of institutional knowledge (about the design rationale, missing bits, etc.). This knowledge needs to be made available.
Until that happens, I don't see how any amount of community contributions could possibly fix this issue, because nobody actually knows how it's *supposed* to work.