@amy I've been working for a while on a better streams specification for JS (interoperable with Node streams and papers over their bullshit because wow they are terrible); would you want to try out those and see if it works better?

Caveat: API is simple but documentation is incomplete and there may still be some rough edges in the reference implementation (which I'm working on polishing out of the design before releasing it). And I can answer any questions about how to work with them of course, and fix issues.

@joepie91@social.pixie.town oh sure! And also, lmao I'm really glad to know it's not just me that finds node streams to be weird, from the implementor side in particular. Using them isn't any worse than any callback based API.

@amy Alright, I'll put together a short thing later today on how they work at a high level 🙂 The 'getting started' so to say. And then I can answer any questions going from that. If that works for you?

@joepie91@social.pixie.town yeah sure 😊

@amy I... may have written a bit more than I had initially anticipated 😅 Here's the introduction so far: https://gist.github.com/joepie91/f4f9a56d37a5935833c242defd4f732c

Feel free to ask if anything is missing/unclear!

@amy (Also feel free to ask if/when you would like a copy of the spec in its current state - I haven't included it because it's not final yet and may have changed by the time it is needed, so probably best to work from an up-to-date copy by that time)

@joepie91@social.pixie.town oh this reminds me a ton of observables (and especially reactivex, or the C++ ranges concept) but is unfortunately therefore probably not what I need at this moment. Because I need to wrap duplex streams to manipulate traffic from a socket going in both directions.

Also, and this isn't a problem with your library which seems very nicely thought out(!) but I greatly prefer interfaces with strong types around them because I have a poor working memory (but usually great long term memory) and types help me keep from shooting myself in the foot by forgetting part way through doing something what kinds of things I'm juggling prior to runtime.

@amy So the duplex streams, and specifically sockets, are a bit of a weird case - it *does* support these, but you need to explicitly specify in from-node-stream whether you want to wrap the readable or writable side (and so just have wrap the same duplex stream twice, once on either end, and it'll work fine).

I did look into duplex streams for Promistreams originally but concluded that they break almost every streams mechanic (or make it unreliable), and ultimately don't actually offer anything extra beyond what "an object containing a sink and a source stream as its properties" would, so that's the intended way to do duplex-like streams in Promistreams.

Also, I've actually been experimenting a lot lately with more ergonomic representation of network sockets in Promistreams, the idea being that a 'server' is just represented as a stream of clients, and a 'client' has a .receive and a .send property which are Promistreams representing the respective sides of the underlying stream.

Many of my recent changes to the low-level abstractions and the spec have actually been exactly to make this sort of thing work reliably! I'm not entirely happy with where it is yet (needs more testing), but a rough example of what that looks like is here: https://gist.github.com/joepie91/b583ce10f9dd63e3ad746c9a2a1f37e0?ts=4

(But you can just use from-node-stream today on a regular duplex socket and it'll just work fine, the example above is just experimentation beyond that!)

On the types point; I don't use TS, but I do generally use very strict argument validation on library surfaces (stricter than what TS can check in places), and with understandable error messages, so it'll tell you if anything invalid is passed in. Quite a few of the stream packages probably don't have this set up yet, but will in the future.

Here's an example of a stream package with relatively complex validation rules: https://git.cryto.net/promistream/simple-source/src/branch/master/index.js#L17-L23 (though most packages are single-argument).

This won't help type propagation in TS of course, at least until tsc improves its inference. I'd be open to someone contributing (maintenance for) TS definitions, but that's not really something I'd want to commit to myself, as I don't use it. I can understand if that's a reason not to use it!

Does that answer your points, or did I miss or misunderstand anything?

(As an aside, I wasn't sure if your problem description implied needing to insert streams inside of an existing unmodifiable Node.js streams pipeline - but if that is the case, that *will* be possible, I just have not gotten around to writing the to-node-stream module yet. Once I do, it will also support creating Duplex Node streams from two Promistreams.)

@joepie91@social.pixie.town I'm curious where you see TS type inference to be insufficient. The only things I haven't yet been able to come up with inference for in my own code are things that I think would require full turing-completeness* to express (something they've explicitly designed against supporting because nobody wants a repeat of C++ templates), though I'll grant you it can be challenging to express certain concepts with it.

That server example is quite nice and that API would certainly work for my needs, though I don't have a stream of sockets that I'm handling. Just one at a time and I need to wrap it and "tap" into it to do some manipulation of the data as it goes back and forth. The operations I'm doing are fully symmetric so it doesn't matter what direction data is flowing in. I suppose you could say my needs are isomorphic to simply logging all the data back and forth. I just don't want to write this logic in many places heh.

I struggle a little to see what exactly your argument validation is doing there? But it seems like it's more than type inference, and actually validation and handling, which is definitely a runtime operation even with strong types, unless you're talking dependent types. (From the names alone it seems like you're expressing optional arguments and functions that take 1-N arguments with types that vary based on the number of arguments, which typescript can trivially represent in its types.)


*Though you can get really quite far without it: a couple examples of things I've been able to write are a fully type safe parser generator, and an abstraction over VSCode's command/RPC system that allows me to create command interfaces and then those are used to deduce the complete valid set of arguments for a single general purpose execCommand function across all the interfaces defined in the code base. (It's pretty neat cause it creates type-level tuples of command names, their arguments, and return types and then unpacks the tuples to do argument deduction based on the name of the command.)