logging problems - "Pipe Shut"

[Gaute Amundsen]

> Ah, OK, you won't be able to process log files larger than about 2 GB on
> a 32-bit machine.  I should probably figure out a way to get around
> that.

Ok, now I have set up /var/log/varnish/varnish.log to rotate every hour, and a 
postrotate action on that to pipe through varnishncsa and append 
to /var/log/varnish/varnish.access_log which in turn gets rotated and 
processed by awstats dayly.

That should take care of the 2G limit in varnishncsa at least. 
I was hoping it would help with the logging it self but apparently not..

I have about 20 logfiles by now, all under 100M, and have made various 
observations:

Everything went well during the night. The accesstimes are about a minute past 
the hour, and the last date string in the log corresponds with that.

Then at 09:13 suddenly it's all "Pipe Shut read(read)" and "Pipe Shut 
write(read)" again. The situation when I noticed this, about 12:30, this was 
in 'log.1' whith modtime 09:13 and 'log' was still 0 bytes.
I guess that just means warnishlog had stopped responding to SIGHUP.

Wil try to do a restart insted in postrotate to see if runningtime has 
anything to do with it.

That's the logging problem, now for the processing problem:

I made a small script to run varnish(log|ncsa) on every file and get the 
returnstatus. log works fine, and a tail of that diplays sensible output and 
timestamps, but ncsa segfaults on about half of them. 
No pattern discernible..

I tried to make a small python script to feed a log chunk by chunk into ncsa 
on stdin, but I was not able to make that work reliably enough to get any 
hint what precisely is causing the problem. Don't even know if ncsa can work 
like that..
Tried to look for the last output lines from nscalogs in the varnishlog as 
well, but nothing jumps out so far...

Starting to run out of things to try here...
Ideas?

Gaute