I'm genuinely stuck. The Matrix S2S spec seems effectively unimplementable in its current state, with core functionality being undefined, which leaves me with only a few options:
1. Try to implement it anyway, and be chasing undefined behaviour and broken rooms for the next 5 years
2. Try to get all the holes in the spec resolved, which will probably take a decade at this pace, if it ever succeeds at all (and will burn me out, guaranteed)
3. Fork the protocol into something that's actually strictly specified, but this will fragment the ecosystem because of missing compatibility
4. Just give up on implementing a Matrix server. Take the (extremely rare for me) decision to just abandon this project entirely.
All of these options suck. I see no remaining solutions until the Matrix core/spec developers actually start prioritizing fixing the spec over other things.
Now what the hell do I do?
Unqualified suggestion
@joepie91
Seems like a carefully considered and thoughtful implementation of 3. might also help with 2. You could try to keep it as compatible as possible and lobby for the core/spec developers to simply merge your specifications.
I am probably not qualified to answer this though as I have no experience with this kind of thing.
clover 🍀⛤
@joepie91@social.pixie.town fork the XMPP protocol to add missing features like spaces
@weedtofu There's a reason I'm trying to work with Matrix; XMPP has fundamental problems like centralized MUCs
@joepie91@social.pixie.town do the thing that one meme said
"there are 13 protocols, but i'll make a protocol to unify them all! there are now 14 protocols"
@joepie91 @ShadowJonathan I totally see your pain, though in this case forking seems very counter productive unless there is considerably large community ready to push the new effort forward.
Matrix itself will hit hard walls of their own doing unless those problems will get resolved. E.g. JSON serialization is a mess and the protocol will never work as intended between different implementations unless that stuff gets fixed.
So, in this case the best way forward would probably be fixing the spec and related documentation, or at least put as much effort to that as viable.
If you are to consider alternatives, then Simplex protocol might be worth checking out as well. Personally I haven’t read that spec, but few people seem to like it.
@jhulkko @ShadowJonathan@tech.lgbt I literally cannot fix the spec problems, because there is no way for me to know what goes in the missing bits - that information is locked up in the heads of the core developers.
And I've already put a ton of work towards filing issues and figuring things out etc., and it's just not sustainable - there's insufficient proactive cooperation from core devs to actually get anywhere.
@joepie91 @ShadowJonathan That might actually be a problem related to the complexity of the overall system, to a degree that some parts of the spec just feel too daunting to touch for about anyone involved.
Neil tried to push that kind of core spec changes forward from within the organisation, but could not get anything forward despite his efforts.
I would assume solving the problems in a way that provides reasonable migration path, and having a preliminary implementation to show viability would be needed to push such changes forward.
Matrix protocol kind of has ”technical debt” in some of its core parts and specs that is hard to justify repairing due to effort and breakage required. Until it will be too painful not to. And currently that pain is only suffered by community projects and the Dendrite devs. Synapse has no problem to remain compatible with itself, and that pain is not really visible there.
@jhulkko @ShadowJonathan@tech.lgbt If this is the case, and "we just don't know how most of the protocol works, and we can't justify fixing it" is accepted in core as the status quo...
Then I have to ask, by what definition is this an open protocol?
@joepie91 @ShadowJonathan Everything in it is open, but not perfect. And each contributing entity had their own set of priorities, which affect decisions on what they actively push forward as their own effort.
I don’t think the project would block advancements in the core spec area, if the effort required from the spec team is running MSCs through the process with viable prototype implementations etc in place. Problem here is probably the amount of effort required that the core project should find resources for, if these changes are expected to be figured out by them based on problems and bug tickets.
The foundation still relies too much on single entity paying for the effort, and that can never be without side effects. I don’t believe there is malice involved in this.
@jhulkko @ShadowJonathan@tech.lgbt Is it actually open *in practice*, though, if implementing the spec does not result in a compatible implementation? (Which it definitely will not, this is well-established)
Like, Microsoft got (rightly) raked over the coals for publishing an "open" document format for Office and then immediately deviating from that in their own implementation, making all *real-world* Office documents incompatible with spec implementations. This was concluded to not be a true open standard.
Aside from the intentions, what's the material difference with this situation as it stands?
@jhulkko @ShadowJonathan@tech.lgbt As for "blocking core spec work" - I have made suggestions for procedural improvements in the past, but these were shot down.
Ultimately this all hinges on the core team; tasks need to be done that other people simply *cannot* do, because of institutional knowledge (about the design rationale, missing bits, etc.). This knowledge needs to be made available.
Until that happens, I don't see how any amount of community contributions could possibly fix this issue, because nobody actually knows how it's *supposed* to work.
@jhulkko @ShadowJonathan@tech.lgbt (And if I'm being honest, I think this is almost entirely a problem of prioritization - the handful of people who *could* do this job are all busy doing things that many others could be doing, instead of the things that only they can do...)
@joepie91 Mix of 1. and 3, if it's possible to "just" define the undefined bits. The internet is made of rough consensus and running code, and if your code is the first to run reliably, others will follow your implementation. (Note that I don't know the state of the art in the Matrix world, so I can't tell if this is really possible)
@Hurgotron The problem I foresee with that approach is that Synapse is by far the dominant implementation, and so in the case of incompatibilities between my homeserver and Synapse, my homeserver is almost certainly the one that's gonna be blamed for being 'wrong'
@joepie91 why use an existing s2s interface if all of them suck?
@SoniEx2 Because not implementing the S2S spec means your server cannot federate with other Matrix servers, and therefore isn't compatible with the protocol.
@joepie91 yes, but at least the other options don't tie down the c2s protocol to the s2s implementation.
so they would be easier to make a custom network for. it could even be properly federated (like matrix), and have virtual hosting.
... eh nvm we're not gonna be able to sell anyone on this idea are we
@joepie91 this makes me feel less crazy, thanks... i came to the same conclusion a few years ago and gave up on it.
@joepie91 why not just use XMPP?
@ch0ccyra1n I was hoping not to have to add a "don't talk to me about XMPP" comment to my post, but...
Yeah, no interest in relitigating the topic of XMPP because those discussions always end the same way and it's never constructive
@joepie91 oh that's fair. Sorry for asking
@joepie91 Matrix always struck me as a “one vendor”-protocol. I don’t know if this is deliberate or by accident…
@nblr I don't think it's deliberate. I do think there are real governance issues.
@joepie91 the SS API should be implementable fine. If you’ve found UB, please file it against github.com/matrix-org/matrix-spec/issues. If you’ve found security concerns, please route them to security@matrix.org. There should be no need to fork. (Each MSC is effectively a small fork anyway).
@joepie91 …or if your concerns are about governance, please talk to matthew or amandine directly (or one of the other directors of the foundation). (currently we don’t know which concerns you’re talking about.)
Well. I've just learned a couple of things about the #Matrix spec and its process that are giving me serious concerns about the long term.
I'm refraining from sharing details here right now, because they would be extremely easy to misinterpret, and it's not an immediate problem to worry about.
However, I do think that it's time to start seriously thinking about forking the protocol, and I've created a room for talking about this: #fork:pixie.town
If you're interested in working on this (as a developer, as a technical writer, as a spec designer, or something else), then please join the room!
However, it's important to recognize that such a spec fork will impact many existing communities and developers, and so breaking compatibility is not to be taken lightly. If possible, it should be avoided.
So if you feel like burning everything down and starting over, this is probably not the project for you. Also, I do expect everybody *not* to harass Matrix core folks over this.
If those things are not an issue for you, then please join!