VCL config file for specific URLs

[pub crawler]

Thanks for your input Rob.

JPEG's, GIFs, etc. are all fine to cache, as they are very static in
nature in our environment.
Currently we have Varnish setup to cache:
ico|html|htm|bmp|png|gif|jpg|jpeg|swf|css|js|rss|xml

Our issue is our app servers get overwhelmed, become a large
bottleneck and eventually fail under high load.  We can add more
servers in horizontal scaling mode, but creates more wasted power,
more machines to maintain, etc.  Our need it to address raised site
load mostly due to search spiders that are out of control but
necessary.

So we thought getting Varnish to cache all these dynamic pages would
alleviate load on our application servers.

Essentially, it is fine for Varnish to cache all of our ColdFusion
pages (.cfm), except a handful of pages like these:
http://www.website.com/Template/members/
http://www.website.com/Template/dsp_addreview.cfm/flat/ID=157
etc.

Any idea of how to accomplish the cache exception of these handful of pages?


> I'm not sure of the best way to supply a large list of URLs to pipe, but I'd
> suggest that you think about turning the logic around.  I don't know the
> nature of your site, but presumably it's safe to cache all JPEGs and
> similar.  How much load would be alleviated by caching everything whose
> content type is not text/html?
>
> Then, for text/html, is it possible for you to edit your backend site and
> add a header such as "X-Is-Cacheable: yes" in your index.cfm and Review.cfm?
>  Then, in vcl_fetch, you'd do something like:
>
> if content-type = text/html
>   if X-Is-Cacheable = yes
>       cache this
>   else
>       don't cache
>   end if
> else
>   cache this
> end if
>
>
> You might find this approach simpler than writing a big long list of pages
> not to cache.
>
>
> Rob
>