GRSEC and Varnish

[Mark Moseley]

On Thu, Feb 4, 2010 at 7:59 AM, Bernardf FRIT <bernard at frit.net> wrote:
> Mark Moseley a écrit :
>>
>> grsec will often report that signals were sent, not that grsec
>> necessarily sent that signal itself. I don't think I've ever actually
>> seen it report itself sending a signal to a process. So varnishd could
>> be segfaulting for some reason and grsec is just reporting this. If
>> you're getting a core file, try loading it into gdb and using 'bt' on
>> it, to see where it's dying.
>
> I cannot get any core file.
>>
>> One other thing to try: As soon as it happens, try using 'dmesg' (or
>> "dmesg -s 131072" in case you have lots of things logging to the
>> kernel logs) and grep for PAX. It's not likely, but PAX could be
>> killing it due to some violation. And for whatever reason, the PAX
>> message doesn't show up in the logs, just in dmesg.
>
> Nothing more in dmesg than in kern.log. ;-[
>
> varnishd[27069]: segfault at 1000 ip 000000000043abf0 sp 0000000045696ae0
> error 4 in varnishd[400000+50000]
> grsec: From 80.13.9.224: signal 11 sent to
> /usr/sbin/varnishd[varnishd:27069] uid/euid:65534/65534
> gid/egid:65534/65534, parent /usr/sbin/varnishd[varnishd:11327] uid/euid:0/0
> gid/egid:0/0
> varnishd[27992]: segfault at 1000 ip 000000000043abf0 sp 0000000048140ae0
> error 4 in varnishd[400000+50000]
> grsec: From 90.4.162.78: signal 11 sent to
> /usr/sbin/varnishd[varnishd:27992] uid/euid:65534/65534
> gid/egid:65534/65534, parent /usr/sbin/varnishd[varnishd:11327] uid/euid:0/0
> gid/egid:0/0
> varnishd[28119]: segfault at ed000 ip 000000000043abf0 sp 0000000047ceaae0
> error 4 in varnishd[400000+50000]
> grsec: From 90.44.219.18: signal 11 sent to
> /usr/sbin/varnishd[varnishd:28119] uid/euid:65534/65534
> gid/egid:65534/65534, parent /usr/sbin/varnishd[varnishd:11327] uid/euid:0/0
> gid/egid:0/0

Try putting "ulimit -c unlimited" in your varnishd rc file. I haven't
needed to get a varnishd core file before, so maybe the devs might be
able to advise if there's other steps necessary as well. There should
also be some logs saying that it died (or at least that it restarted);
dunno what your distro you're using, but in debian, those typically
end up in /var/log/syslog.

You could tail varnishncsa to see if there's a common request where it
seems to segfault at and if there is, you could attach to varnishd
with "gdb /path/to/varnishd <pid of varnishd>" and try to trigger it.
Then get the backtrace with 'bt'. But be aware that it'll bog it down
dreadfully, so i wouldn't advise it in production.