Re: Process.BeginErrorReadLine and BeginOutputReadLine; buffer not flushed?

On Oct 14, 2:19 pm, "Peter Duniho" <NpOeStPe...@xxxxxxxxxxxxxxxx>
wrote:
On Tue, 14 Oct 2008 10:56:13 -0700, robert.waters  

<robert.wat...@xxxxxxxxx> wrote:
I have a thread that uses the Process object to execute an external
script, and redirects the stdout and stderr output from that script
asyncronously (using BeginErrorReadLine and BeginOutputReadLine and a
DataReceivedEventHandler for each output handle).
However, the output is not being sent to the event handlers when it is
flushed in the external script.

The (perl) script is simple, it just iterates and prints a string to
stdout when the iterator is even, and a string to stderr when it is
odd; output is flushed after each print.  When I run this script in
the console, I get the expected output (alternating stdout/stderr
strings).  However, when I run this script using the Process object,
the strings appear in bunches of one or the other.  [...]

I suspect that this is a side-effect of the complete end-to-end  
implementation details.  Unfortunately, none of the parts you're using  
provide any sort of guarantee that the stdout and stderr will in any way  
be synchronized, which means that there's a fundamental problem with the  
specific approach.

That said, you _might_ be able to improve the situation by handling the  
reads yourself rather than letting the Process "read line" behavior do  
it.  In particular, call BeginRead() directly on the stream returned by  
Process.StandardOutput.BaseStream (and likewise for StandardError).  That  
will at least allow you to bypass the buffering that goes on in the text  
reader (but you'll have to handle the character output encoding yourself)..

I'm not all that hopeful the above would work though.  I mean, it's worth  
a try.  But it's not just the StreamReader that does buffering.  The  
OS/console itself is also buffering.  There's a certain amount of output  
that the process can send out on the stdout and stderr streams, a few  
hundred bytes if I recall correctly, before the process itself will  
block.  That means that if the process gets ahead of your reading process  
at all, the reading process can still wind up in a thread that's happy to  
keep reading the stream it's working on, while the OS buffers output sent  
to the other string.

All of this assumes that there's not actually a set order to how output  
goes to each stream.  I'm assuming that your "odd/even" test case is just  
a simplification to narrow down the problem and that in reality you've got  
output going to each string arbitrarily.  Of course, if it's not, then one  
obvious solution is to simply set up a bottleneck queue that alternates  
between which stream it allows output from, on a line-per-line basis.

But somehow I suspect your actual real-world problem isn't that simple.  
The bottom line here is that you seem to be using stdout and stderr in a  
way that is contrary to their intended use, wanting for a guarantee that  
was never designed into the console output architecture in the first  
place.  Ultimately, that sort of approach is likely to lead to failure, or  
at the very least to some fairly awkward or ugly hacking to get things to  
work the way you want.

Pete

Thanks Pete. "Ugly Hacking" is looking like the only resolution to
this problem.
I found some related information:
"The Process class use a internal class AsyncStreamReader of which
constructor the buffer size is fixed as 1024."; so, without re-
implementing the Process class, I am out of luck.

The bottom line here is that you seem to be using stdout and stderr in a
way that is contrary to their intended use
Pet, you've read me perfectly. I have a tendency to use stderr as a
logging mechanism (for debugging), which keeps stdout clean-and-neat.
I suppose I'll have to start logging to file like everyone else ;)

Thank you so much for your time and your help; I wish my question
could have been as succinct as your answer.
-Robert
.

Relevant Pages

  • Re: Process.BeginErrorReadLine and BeginOutputReadLine; buffer not flushed?
    ... script, and redirects the stdout and stderr output from that script ... That means that if the process gets ahead of your reading process at all, the reading process can still wind up in a thread that's happy to keep reading the stream it's working on, while the OS buffers output sent to the other string. ...
    (microsoft.public.dotnet.languages.csharp)
  • Open3 with nonblocking reads
    ... I'm currently trying to execute a script within a script, ... single line should contain if it's STDOUT or STDERR, ... STDOuts were passed to the logfile. ...
    (comp.lang.perl.misc)
  • Re: order of execution
    ... stdout and stderr are different streams. ... the standard error stream is not fully buffered; ... C defines three buffering modes for streams: ...
    (comp.lang.c)
  • Re: coloring stdout and stderr
    ... everything printed to stdout shows up in green and everything printed to ... stderr becomes red. ... I tried pipes, but without much success, as ... output to a terminal only line buffering is used (i.e. ...
    (comp.unix.programmer)
  • Re: Filter stdout and stderr Together
    ... I would like to write a script that filters lines from both stdout and ... script with the following syntax: ... stderr as expected, but with particular lines filtered out. ...
    (comp.unix.shell)