Raw access to stdio fd's opened in child_process.spawn()

[isaacs]

API: https://gist.github.com/2294039

Re: #2729

[isaacs]

Note: createPipe part of the API is out.

[isaacs]

Possible informative use case stuff for what people want from the libuv bits: Keno/julia#5

[Keno]

I am the Julia developer that is currently working on the libuv integration. While I can't speak to the node-side interface, I did want to mention what our use case for pipes/spawn is. However, first of all a huge THANK YOU for making libuv available. Now, Julia supports the following syntax:

run(`ls`|`sort`)

which basically does exactly what you would expect it to do (pipe the output from ls to sort). I have been able to get some of the rudimentary cases to work by allowing libuv to reuse the already created fd, but for the more complex stuff we need access to both file handlers. Now, I see a few options here (again just the libuv parts):

1. Add an extra field to uv_pipe_t for the other file handler (see the issue @isaacs linked for @vtjnash 's ideas on how this is best done.
2. Introduce an extra field for the other fd, but make it a separate struct type to not pay the overhead when we do not need access to the other fd (somthing like UV_DUAL_NAMED_PIPE).
3. Make it explicit that uv_pipe_t is the wrapper for only one end of the file handle and create a new method that takes two initialized pipes and assigns the fd's to them. In uv_spawn, we could then check if there's already an fd associated with the pipe and if there is use it. If not, we can fall back on the current behavior and discard the fd that gets set on the child's stdio.

Now, I'm tending towards option 2, because that would be most explicit. Option 1 would work also if we are willing to accept the additional overhead of one field if we don't need both fd's. Option 3 would probably require the least code changes, but also create a bit of overhead in the case where both fd's are needed (since it involves creating an extra new uv_pipe_t). If I had to fix this temporarily I would choose 3, but as it seems like the process spawning API is being reworked anyway, it might be worth thinking about other solutions.

That's basically how I see it from the Julia+libuv point of view. If you need more information about how we are using libuv, feel free to ask.

Also, I'll just cc @piscisaureus and @bnoordhuis as it is mostly about libuv.

[bnoordhuis]

@loladiro: Thanks for the detailed description. Something I've been tinkering with is an interface that looks like uv_pipe(stream1, stream2) which copies[1] the output from stream1 to stream2. Would that work for you? Are there times when you munge the data before passing it on?

[1] It's actually a zero copy operation on Linux thanks to the magic of the splice() syscall.

[Keno]

I think that would work for the use cases we have (There's no case where we need to do anything with the data before passing it yet). I'll have to play around with the actual implementation once that is available. Also, how would this be implemented on Windows, as the primary reasons for us switching to libuv was platform independence?

[bnoordhuis]

Also, how would this be implemented on Windows, as the primary reasons for us switching to libuv was platform independence?

The interface and behavior would be the same but (maybe) not as optimized.

[indutny]

Can this be closed now? We still don't have API for piping two streams to each other.

[Keno]

For the julia side of things we need joyent/libuv#451 or something similar to get access to the streams that used to be implicitly created in uv_spawn. I can't speak for the node side though.

[indutny]

Ok, closing this issue as it's no more related to node.