Author: Ottavio Caruso Date: To: Hampshire LUG Discussion List Subject: Re: [Hampshire] unwanted buffering of ssh output when piping into awk
Victor Churchill wrote: > I want to deconstruct the log file lines and reformat them for legibility.
> So I pipe the stream coming from ssh into awk on my desktop machine.
> When I try
> ssh qq2 "tail -f /var/log/httpd/custom_log | awk '{printf (\"%9d %15s
> %7s %s %60s\n\", \$NF, \$1, \$3, \$4, \$7)}' "
> I see nothing at first, and then a burst of lines, then nothing, etc.
>
> Replacing the 'tail' command with a while-loop saying 'date' every
> second, I get a batch of around 140 lines at a time which is
> remarkably close to 4K bytes.
>
> I can run the tail output into an awk process on the server
> ssh qq2 "tail blah | awk 'blah' " (with appropriate quote-escapes) and
> that works but I'd rather do the post-processing on my machine.
>
> After checking ssh and awk manpages the only clue I found was to add
> fflush() to the wak code, but with no observable difference.
>
> ssh server "command" gives every impression of giving immediate
> line-by-line output without being buffered by ssh or bash. And awk
> reads from stdin with a newline record separator. So who is doing the
> 4KB buffering (if that is what is happening)?
>
