@amy So the duplex streams, and specifically sockets, are a bit of a weird case - it *does* support these, but you need to explicitly specify in from-node-stream whether you want to wrap the readable or writable side (and so just have wrap the same duplex stream twice, once on either end, and it'll work fine).

I did look into duplex streams for Promistreams originally but concluded that they break almost every streams mechanic (or make it unreliable), and ultimately don't actually offer anything extra beyond what "an object containing a sink and a source stream as its properties" would, so that's the intended way to do duplex-like streams in Promistreams.

Also, I've actually been experimenting a lot lately with more ergonomic representation of network sockets in Promistreams, the idea being that a 'server' is just represented as a stream of clients, and a 'client' has a .receive and a .send property which are Promistreams representing the respective sides of the underlying stream.

Many of my recent changes to the low-level abstractions and the spec have actually been exactly to make this sort of thing work reliably! I'm not entirely happy with where it is yet (needs more testing), but a rough example of what that looks like is here: https://gist.github.com/joepie91/b583ce10f9dd63e3ad746c9a2a1f37e0?ts=4

(But you can just use from-node-stream today on a regular duplex socket and it'll just work fine, the example above is just experimentation beyond that!)

On the types point; I don't use TS, but I do generally use very strict argument validation on library surfaces (stricter than what TS can check in places), and with understandable error messages, so it'll tell you if anything invalid is passed in. Quite a few of the stream packages probably don't have this set up yet, but will in the future.

Here's an example of a stream package with relatively complex validation rules: https://git.cryto.net/promistream/simple-source/src/branch/master/index.js#L17-L23 (though most packages are single-argument).

This won't help type propagation in TS of course, at least until tsc improves its inference. I'd be open to someone contributing (maintenance for) TS definitions, but that's not really something I'd want to commit to myself, as I don't use it. I can understand if that's a reason not to use it!

Does that answer your points, or did I miss or misunderstand anything?

(As an aside, I wasn't sure if your problem description implied needing to insert streams inside of an existing unmodifiable Node.js streams pipeline - but if that is the case, that *will* be possible, I just have not gotten around to writing the to-node-stream module yet. Once I do, it will also support creating Duplex Node streams from two Promistreams.)

@joepie91@social.pixie.town I'm curious where you see TS type inference to be insufficient. The only things I haven't yet been able to come up with inference for in my own code are things that I think would require full turing-completeness* to express (something they've explicitly designed against supporting because nobody wants a repeat of C++ templates), though I'll grant you it can be challenging to express certain concepts with it.

That server example is quite nice and that API would certainly work for my needs, though I don't have a stream of sockets that I'm handling. Just one at a time and I need to wrap it and "tap" into it to do some manipulation of the data as it goes back and forth. The operations I'm doing are fully symmetric so it doesn't matter what direction data is flowing in. I suppose you could say my needs are isomorphic to simply logging all the data back and forth. I just don't want to write this logic in many places heh.

I struggle a little to see what exactly your argument validation is doing there? But it seems like it's more than type inference, and actually validation and handling, which is definitely a runtime operation even with strong types, unless you're talking dependent types. (From the names alone it seems like you're expressing optional arguments and functions that take 1-N arguments with types that vary based on the number of arguments, which typescript can trivially represent in its types.)


*Though you can get really quite far without it: a couple examples of things I've been able to write are a fully type safe parser generator, and an abstraction over VSCode's command/RPC system that allows me to create command interfaces and then those are used to deduce the complete valid set of arguments for a single general purpose execCommand function across all the interfaces defined in the code base. (It's pretty neat cause it creates type-level tuples of command names, their arguments, and return types and then unpacks the tuples to do argument deduction based on the name of the command.)

@amy (Response to your usecase at the bottom of this post, beyond the line separator, feel free to skip all the stuff above it if you're not in the mood for discussion about types/validation!)

To clarify on the inference point: I meant when not using TS yourself :) tsc has some very limited inference abilities on "just" JS, when it doesn't have explicit type annotations to go off anywhere, but it gets very easily confused, and doesn't infer anywhere near as well in that case as eg. Tern does.

I've found that it usually only takes 3 steps of indirection or so before it loses track of what types are produced by what, and it doesn't understand conditional cases at all (whereas eg. Tern would consider the union of possible types in its inference), and you end up with it just not knowing the types of anything. I've heard complaints that some JS libraries without explicit type definitions can cause this kind of 'type loss' even in TS codebases.

None of this happens if you're writing code in TS yourself AFAIK; as long as it regularly gets explicit type specifications, and everything inbetween also has type definitions, it seems to do fine.

On the validation: yep, it's runtime validation, the motivations are explained a bit more here, albeit in slightly more fiery language due to a number of unpleasant past discussions with other folks: https://www.npmjs.com/package/@validatem/core (though that's not required reading for using Promistreams). I just personally haven't found much value in static typing beyond that, and strong typing (ie. no type coercion) unfortunately isn't a thing in JS either way.

The specific example I linked makes use of `@validatem/wrap-value-as-option` to accept either a single value or an options object, and if it gets a single value, it turns it into an options object with that value as the `onRequest` property. That's because often you don't need to specify any custom teardown code for a source stream, so then you can just pass the "value-producing" callback directly to it and you're done.

--

Onto your actual usecase, if you just need to wrap a single socket at a time, then yeah, the experimental stuff is not needed; just a fromNodeStream.fromReadable(socket) and fromNodeStream.fromWritable(socket) should provide what you need, I think? And then the resulting streams can be used in a normal Promistreams pipeline, and it *should* now correctly handle the socket being closed on either end (by happy-aborting the other side). Please let me know if it doesn't.

The resulting sink stream should take Buffers, and the resulting source stream should produce Buffers, so then you could use a stream/pipeline-generating function like the third example in the introductory post to generate the same 'transform' (logger?) and use that on both ends. And then do whatever else needs to happen in the pipeline, basically. And perhaps pass them around bundled in an object if needed.

I'm not sure what the rest of your usecase looks like - if you're okay with sharing more detail (basically an end-to-end description of where the data comes from and where it ultimately goes to in what form), then I can probably provide a more concrete example of how that might be done in a Promistreams pipeline.

Every pipeline needs to be actively 'driven', so unless you're feeding all the received data immediately back into the sending side (after logging), some kind of other destination will need to be specified as a Promistream, using a Node stream wrapper or otherwise :)

@amy Belated addition: the example of Validatem use that I linked could indeed be expressed in most static type systems, but Validatem is what I use everywhere by default, including many cases where it couldn't be; the list of validators can give an idea of that: https://validatem.cryto.net/modules/

There just aren't very many of these cases in Promistreams specifically, so it's hard to show that in context 😅

@joepie91@social.pixie.town sure. Validation is a different job than types. Types just reduce the amount you need by proof.

Anyway, I looked closely at promistreams and while cool I don't think they fit my needs closely enough to to pick up an untyped dependency.

@amy No worries! The offer of help remains open in case you run into something else later where they'd be a good fit :)