compression-at-edge, in front of varnish reverse proxy, gzip-compresses some content, but not other ?

[Ben DJ]

nginx compression-at-edge (in front of a reverse proxy) compresses
some content, not other ?

I've an nginx + varnish + apache2 stack.  Nginx serves as redirector,
ssl handshake, and gzip compression.  Varnish serves as a
reverse-proxy, and apache 'just' hosts apps & serves content.

I'm trying to get gzip compression behaving properly.  I'm getting
intermittent results -- some content seems to be gzipped, some not.

I'm looking for some help figuring out WHERE the problem lies, and
what to do to fix it.

In nginx conf, I've,

	...
	http {
		...
		gzip              on;
		gzip_http_version 1.0;
		gzip_comp_level   9;
		gzip_proxied      any;
		gzip_buffers      16 8k;
		gzip_min_length   0;
		gzip_types text/plain text/css text/xml text/javascript
application/x-javascript;
		gzip_disable "MSIE [1-6].(?!.*SV1)";
		gzip_vary         on;
	
		upstream varnish {
			server 127.0.0.1:8090 weight=10 max_fails=3 fail_timeout=15s;
		}
	
		server {
			listen                    x.y.z.w:443;
			...
			location / {
				proxy_pass             http://varnish;
				proxy_redirect         off;
				proxy_set_header       Host $host;
				proxy_set_header       X-Real-IP $remote_addr;
				proxy_set_header       X-Forwarded-For $proxy_add_x_forwarded_for;
				proxy_set_header       X-Client-Verify SUCCESS;
				proxy_set_header       X-SSL-Subject $ssl_client_s_dn;
				proxy_set_header       X-SSL-Issuer  $ssl_client_i_dn;
			}
		}
		...


In varnish config, per chat @ #irc, I've replaced,

	if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
		# No point in compressing these
		remove req.http.Accept-Encoding;
	} elsif (req.http.Accept-Encoding ~ "gzip") {
		# if the browser supports it, we'll use gzip
		set req.http.Accept-Encoding = "gzip";
	} elsif (req.http.Accept-Encoding ~ "deflate") {
		# next, try deflate if it is supported
		set req.http.Accept-Encoding = "deflate";
	} else {
		# unknown algorithm. Probably junk, remove it
		remove req.http.Accept-Encoding;
	}

with a,

	...
	if (req.http.Accept-Encoding) {
		remove req.http.Accept-Encoding;
	}
	...

clause, since the compression is to be done only at the nginx 'edge'.

Apache's compression (gzip, default, or otherwise), is completely disabled.

Atm, a check of my test site with YSlow complains that .js's are NOT
being compressed.  Checking with LiveHTTPHeaders Firefox plugin,
shows, e.g., the .js in question NOT being gzipped, but a .gif *is*,
e.g.

	...
	----------------------------------------------------------

	https://my.site.com/main/apostrophePlugin/js/jquery.keycodes-0.2.js

	

	GET /main/apostrophePlugin/js/jquery.keycodes-0.2.js HTTP/1.1
	Host: my.site.com
	User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.4)
Gecko/20100417
	Accept: */*
	Accept-Language: en-us,en;q=0.5
	Accept-Encoding: gzip,deflate
	Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
	Keep-Alive: 115
	Connection: keep-alive
	Referer: https://my.site.com/main/
	Cookie: SESS6fa8cdc2d7064704bbda0c83e2c2588c=94889db68945e19ed6f666b7e00cdd36;
symfony=3KOH8Qk0hV%2C%2CvXLi0PK5YmdenP1
	If-Modified-Since: Tue, 27 Apr 2010 18:18:18 GMT
	If-None-Match: "362e4-1008-4853810064180"
	Authorization: Digest username="admin", realm="AUTH my.site.com",
nonce="5GopBDmFBAA=99f7be8796e018dde459a07178393d235366ecd9",
uri="/main/apostrophePlugin/js/jquery.keycodes-0.2.js", algorithm=MD5,
response="b04cb995cd1f86a67197aab3b5a5dbc9", qop=auth, nc=000001c9,
cnonce="9dbeaefee4d57b12"
	Cache-Control: max-age=0

	

	HTTP/1.1 304 Not Modified
	Server: nginx/0.8.35
	Date: Wed, 28 Apr 2010 00:39:48 GMT
	Connection: keep-alive
	Etag: "362e4-1008-4853beaefee80"
	Expires: Sat, 01 May 2010 00:39:48 GMT
	Cache-Control: max-age=259200
	Content-Length: 0
	X-Varnish: 940462008
	Age: 0
	Via: 1.1 varnish

	----------------------------------------------------------

	https://my.site.com/apostrophePlugin/images/a-special-blank.gif

	GET /apostrophePlugin/images/a-special-blank.gif HTTP/1.1
	Host: my.site.com
	User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.4)
Gecko/20100417
	Accept: image/png,image/*;q=0.8,*/*;q=0.5
	Accept-Language: en-us,en;q=0.5
	Accept-Encoding: gzip,deflate
	Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
	Keep-Alive: 115
	Connection: keep-alive
	Referer: https://my.site.com/main/
	Cookie: SESS6fa8cdc2d7064704bbda0c83e2c2588c=94889db68945e19ed6f666b7e00cdd36;
symfony=3KOH8Qk0hV%2C%2CvXLi0PK5YmdenP1

	HTTP/1.1 404 Not Found
	Server: nginx/0.8.35
	Date: Wed, 28 Apr 2010 00:39:48 GMT
	Content-Type: text/html; charset=iso-8859-1
	Transfer-Encoding: chunked
	Connection: keep-alive
	Vary: Accept-Encoding, accept-language,accept-charset
	Content-Language: en
	X-Varnish: 940462009 940461993
	Age: 87
	Via: 1.1 varnish
	Content-Encoding: gzip
	----------------------------------------------------------
	...


where, iiuc, the presence/absence of "Content-Encoding: gzip" defines
whether or not the item in question was succesfully gzipped.

Any ideas why/where the .js is not getting gzipped?

Thanks,

Ben