"not cacheable" even on static files
Baptiste
bedis9 at gmail.com
Fri Jan 13 09:07:13 CET 2012
Hi Harry,
Is HAProxy in front of your Varnish servers or between varnish and Apache?
You could use HAProxy to mitigate the attack, if you know the pattern,
I can help on this point.
cheers
On Thu, Jan 12, 2012 at 10:37 PM, Harry Sufehmi <sufehmi at gmail.com> wrote:
> Hi folks, for the past few days I've been puzzled by this problem -
> Varnish have refused to cache even static files. While it was fine
> before, and will cache even dynamic HTML files (with proper cache
> headers, of course)
>
> The problem begun when my client reported that his website was
> constantly going up & down. When I checked, it was under some sort of
> DDoS attack. This is not news, because his website was pretty much
> under 24x7 syn flood attack.
> However, this attack is now able to bypass HAproxy & Varnish - and hit
> Apache directly, right where it hurts most.
>
> To cut the long story short - basically now Varnish refuses to cache
> almost everything.
> The X-Cacheable header that I enabled contains the dreaded "NO:Not
> Cacheable" status. And I've not been able to find out why.
>
> This is where I stuck, if I can find out the cause, the I'd be able to
> rectify it.
>
> Anyway, the DDoS attack may not be related at all to the current
> problem (varnish not caching), I included the story just for the sake
> of completeness.
>
> Let's proceed to some facts that I've gathered :
>
> Same GIF file - but different sizes (note the "Received" column) :
> http://minus.com/mbawzSZUxJ#3
>
> The URLs with port 8181 are direct requests to Varnish, bypassing HAproxy.
>
> HTTP Headers produced by Varnish : http://minus.com/mbawzSZUxJ#4
>
> HTTP Headers produced by HAproxy : http://minus.com/mbawzSZUxJ#2
>
> List of cookies sent by browser & received from Varnish :
> http://minus.com/mbawzSZUxJ#1
> (stripped clean by varnish, basically)
>
> Please find the result of varnishstat -1 & content of
> /etc/varnish/default.vcl attached to the end of this email.
>
> Varnish is running with the following parameters :
>
> $ ps aux|grep varnish
> myuser 3327 0.0 0.0 3324 796 pts/6 S+ 04:36 0:00 grep varnish
> root 8441 0.0 1.0 86024 83532 pts/2 SL+ Jan12 0:01
> /usr/sbin/varnishd -P /var/run/varnishd.pid -a 0.0.0.0:8181 -f
> /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 180 -w 2,1000,30 -s
> malloc,2G -d
>
>
> Hopefully someone will be able to point / give me a hint to the right direction.
>
>
> Thanks,
> Harry
>
> ===================
>
> $ telnet localhost 6082
> Trying 127.0.0.1...
> Connected to localhost.
> Escape character is '^]'.
> 200 199
> -----------------------------
> Varnish Cache CLI 1.0
> -----------------------------
> Linux,2.6.32-25-generic-pae,i686,-smalloc,-hcritbit
>
> Type 'help' for command list.
> Type 'quit' to close CLI session.
>
> stats
> 200 1978
> 16564302 Client connections accepted
> 16514647 Client requests received
> 2378064 Cache hits
> 988836 Cache misses
> 592750 Backend conn. success
> 462880 Backend conn. failures
> 13096234 Backend conn. reuses
> 63305 Backend conn. was closed
> 13160380 Backend conn. recycles
> 82 Fetch head
> 9937334 Fetch with Length
> 1856 Fetch chunked
> 245 Fetch wanted close
> 66 Fetch failed
> 26436 N struct sess_mem
> 26091 N struct sess
> 18773 N struct object
> 18862 N struct objectcore
> 6827 N struct objecthead
> 3 N struct vbe_conn
> 107 N worker threads
> 26496 N worker threads created
> 144188 N overflowed work requests
> 1 N backends
> 932755 N expired objects
> 1851315 N LRU moved objects
> 12437230 Objects sent with write
> 16564302 Total Sessions
> 16514647 Total Requests
> 870 Total pipe
> 13146877 Total pass
> 13630039 Total fetch
> 6464791313 Total header bytes
> 122615308544 Total body bytes
> 16564289 Session Closed
> 11 Session Linger
> 17 Session herd
> 1364697125 SHM records
> 93867958 SHM writes
> 2574 SHM flushes due to overflow
> 446883 SHM MTX contention
> 540 SHM cycles through buffer
> 10815252 SMA allocator requests
> 37554 SMA outstanding allocations
> 45497812 SMA outstanding bytes
> 114575871831 SMA bytes allocated
> 114530374019 SMA bytes free
> 505674 SMS allocator requests
> 211877406 SMS bytes allocated
> 211877406 SMS bytes freed
> 13688813 Backend requests made
> 1 N vcl total
> 1 N vcl available
> 1 N total active purges
> 1 N new purges added
> 3365986 HCB Lookups without lock
> 122890 HCB Lookups with lock
> 122890 HCB Inserts
> 99485 Client uptime
> 16426 Backend conn. retry
> 3690588 Fetch no body (304)
>
> =================== default.vcl ===============
>
> $ cat /etc/varnish/default.vcl
>
> # Default backend definition. Set this to point to your content
> # server.
> backend default {
> .host = "127.0.0.1";
> .port = "81";
> }
>
>
> sub vcl_deliver {
> if (obj.hits > 0) {
> set resp.http.X-Cache = "HIT";
> } else {
> set resp.http.X-Cache = "MISS";
> }
> }
>
> # Below is a commented-out copy of the default VCL logic. If you
> # redefine any of these subroutines, the built-in logic will be
> # appended to your code.
> #
> sub vcl_recv {
>
> if (req.backend.healthy) {
> set req.grace = 180s;
> } else {
> set req.grace = 1h;
> }
> }
>
>
>
> sub vcl_fetch {
>
> set beresp.grace = 1h;
>
> # Varnish determined the object was not cacheable
> if (!beresp.cacheable) {
> set beresp.http.X-Cacheable = "NO:Not Cacheable";
>
> # You don't wish to cache content for logged in users
> } elsif (req.http.Cookie ~ "(UserID|_session)") {
> set beresp.http.X-Cacheable = "NO:Got Session";
> return(pass);
>
> # You are respecting the Cache-Control=private header from the backend
> } elsif (beresp.http.Cache-Control ~ "private") {
> set beresp.http.X-Cacheable = "NO:Cache-Control=private";
> return(pass);
>
> # You are extending the lifetime of the object artificially
> } elsif (beresp.ttl < 1s) {
> set beresp.ttl = 5s;
> set beresp.grace = 5s;
> set beresp.http.X-Cacheable = "YES:FORCED";
>
> # Varnish determined the object was cacheable
> } else {
> set beresp.http.X-Cacheable = "YES";
> }
>
> if (req.url ~
> "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$")
> {
> unset beresp.http.set-cookie;
> set beresp.ttl = 24h;
> }
>
> return(deliver);
>
> # if (!beresp.cacheable) {
> # return (pass);
> # }
> # if (beresp.http.Set-Cookie) {
> # return (pass);
> # }
> # return (deliver);
> }
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
More information about the varnish-misc
mailing list