"not cacheable" even on static files

[Harry Sufehmi]

Hi folks, for the past few days I've been puzzled by this problem -
Varnish have refused to cache even static files. While it was fine
before, and will cache even dynamic HTML files (with proper cache
headers, of course)

The problem begun when my client reported that his website was
constantly going up & down. When I checked, it was under some sort of
DDoS attack. This is not news, because his website was pretty much
under 24x7 syn flood attack.
However, this attack is now able to bypass HAproxy & Varnish - and hit
Apache directly, right where it hurts most.

To cut the long story short - basically now Varnish refuses to cache
almost everything.
The X-Cacheable header that I enabled contains the dreaded "NO:Not
Cacheable" status. And I've not been able to find out why.

This is where I stuck, if I can find out the cause, the I'd be able to
rectify it.

Anyway, the DDoS attack may not be related at all to the current
problem (varnish not caching), I included the story just for the sake
of completeness.

Let's proceed to some facts that I've gathered :

Same GIF file - but different sizes (note the "Received" column) :
http://minus.com/mbawzSZUxJ#3

The URLs with port 8181 are direct requests to Varnish, bypassing HAproxy.

HTTP Headers produced by Varnish : http://minus.com/mbawzSZUxJ#4

HTTP Headers produced by HAproxy : http://minus.com/mbawzSZUxJ#2

List of cookies sent by browser & received from Varnish :
http://minus.com/mbawzSZUxJ#1
(stripped clean by varnish, basically)

Please find the result of varnishstat -1 & content of
/etc/varnish/default.vcl attached to the end of this email.

Varnish is running with the following parameters :

$ ps aux|grep varnish
myuser   3327  0.0  0.0   3324   796 pts/6    S+   04:36   0:00 grep varnish
root      8441  0.0  1.0  86024 83532 pts/2    SL+  Jan12   0:01
/usr/sbin/varnishd -P /var/run/varnishd.pid -a 0.0.0.0:8181 -f
/etc/varnish/default.vcl -T 127.0.0.1:6082 -t 180 -w 2,1000,30 -s
malloc,2G -d


Hopefully someone will be able to point / give me a hint to the right direction.


Thanks,
Harry

===================

$ telnet localhost 6082
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
200 199
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,2.6.32-25-generic-pae,i686,-smalloc,-hcritbit

Type 'help' for command list.
Type 'quit' to close CLI session.

stats
200 1978
    16564302  Client connections accepted
    16514647  Client requests received
     2378064  Cache hits
      988836  Cache misses
      592750  Backend conn. success
      462880  Backend conn. failures
    13096234  Backend conn. reuses
       63305  Backend conn. was closed
    13160380  Backend conn. recycles
          82  Fetch head
     9937334  Fetch with Length
        1856  Fetch chunked
         245  Fetch wanted close
          66  Fetch failed
       26436  N struct sess_mem
       26091  N struct sess
       18773  N struct object
       18862  N struct objectcore
        6827  N struct objecthead
           3  N struct vbe_conn
         107  N worker threads
       26496  N worker threads created
      144188  N overflowed work requests
           1  N backends
      932755  N expired objects
     1851315  N LRU moved objects
    12437230  Objects sent with write
    16564302  Total Sessions
    16514647  Total Requests
         870  Total pipe
    13146877  Total pass
    13630039  Total fetch
  6464791313  Total header bytes
122615308544  Total body bytes
    16564289  Session Closed
          11  Session Linger
          17  Session herd
  1364697125  SHM records
    93867958  SHM writes
        2574  SHM flushes due to overflow
      446883  SHM MTX contention
         540  SHM cycles through buffer
    10815252  SMA allocator requests
       37554  SMA outstanding allocations
    45497812  SMA outstanding bytes
114575871831  SMA bytes allocated
114530374019  SMA bytes free
      505674  SMS allocator requests
   211877406  SMS bytes allocated
   211877406  SMS bytes freed
    13688813  Backend requests made
           1  N vcl total
           1  N vcl available
           1  N total active purges
           1  N new purges added
     3365986  HCB Lookups without lock
      122890  HCB Lookups with lock
      122890  HCB Inserts
       99485  Client uptime
       16426  Backend conn. retry
     3690588  Fetch no body (304)

=================== default.vcl ===============

$ cat /etc/varnish/default.vcl

# Default backend definition.  Set this to point to your content
# server.
 backend default {
     .host = "127.0.0.1";
     .port = "81";
 }


sub vcl_deliver {
        if (obj.hits > 0) {
                set resp.http.X-Cache = "HIT";
        } else {
                set resp.http.X-Cache = "MISS";
        }
}

# Below is a commented-out copy of the default VCL logic.  If you
# redefine any of these subroutines, the built-in logic will be
# appended to your code.
#
 sub vcl_recv {

  if (req.backend.healthy) {
    set req.grace = 180s;
  } else {
    set req.grace = 1h;
  }
 }



 sub vcl_fetch {

   set beresp.grace = 1h;

    # Varnish determined the object was not cacheable
    if (!beresp.cacheable) {
        set beresp.http.X-Cacheable = "NO:Not Cacheable";

    # You don't wish to cache content for logged in users
    } elsif (req.http.Cookie ~ "(UserID|_session)") {
        set beresp.http.X-Cacheable = "NO:Got Session";
        return(pass);

    # You are respecting the Cache-Control=private header from the backend
    } elsif (beresp.http.Cache-Control ~ "private") {
        set beresp.http.X-Cacheable = "NO:Cache-Control=private";
        return(pass);

    # You are extending the lifetime of the object artificially
    } elsif (beresp.ttl < 1s) {
        set beresp.ttl   = 5s;
        set beresp.grace = 5s;
        set beresp.http.X-Cacheable = "YES:FORCED";

    # Varnish determined the object was cacheable
    } else {
        set beresp.http.X-Cacheable = "YES";
    }

    if (req.url ~
"\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$")
{
            unset beresp.http.set-cookie;
            set beresp.ttl   = 24h;
    }

    return(deliver);

#     if (!beresp.cacheable) {
#         return (pass);
#     }
#     if (beresp.http.Set-Cookie) {
#         return (pass);
#     }
#     return (deliver);
 }