Varnish logging

Dirk Gomez lists at dirkgomez.de
Thu Apr 19 21:37:19 CEST 2007 

Hi list,

The setting is: a lot of virtual hosts running on top of Zope. We  
deliver statistics generated with server logfiles.

We are perfectly content with post-processing a big logfile with  
entries for several domains - we had a script like that when we were  
still using Squid.

Comparing the logfiles generated by Squid and Varnish we find that  
those generated by Varnish lack any Host prefix in the relevant 'GET'  
field so we are unable to process them as we did with Squid's.

As the manpage does not say anything about a configuration option to  
alter this behaviour, we'd be thankful for hints in this case.  
Filtering before logging was proposed in an earlier post- in our case  
this does not seem to be a good solution as there are lots of  
changing domains involved and so the configuration would be quite  
tedious- it would be entirely sufficient to have the host logged by  
default.

Appendix: Some lines of the logfiles, for comparison

Squid:

89.59.80.86 - - [06/Apr/2007:13:28:05 +0200] "GET http:// 
op.elseware.de:8080/content/logobao.gif HTTP/1.1" 200 17554 "http:// 
op.elseware.de/content/e3224/e10/e589/e594/e702/index_ger.html"  
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR  
2.0.50727)" TCP_REFRESH_MISS:DIRECT
89.59.80.86 - - [06/Apr/2007:13:28:05 +0200] "GET http:// 
op.elseware.de:8080/common/masses.jpg HTTP/1.1" 200 2681 "http:// 
op.elseware.de/content/e3224/e10/e589/e594/e702/index_ger.html"  
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR  
2.0.50727)" TCP_MEM_HIT:NONE
89.59.80.86 - - [06/Apr/2007:13:28:05 +0200] "GET http:// 
op.elseware.de:8080/favicon.ico HTTP/1.1" 200 1407 "-" "Mozilla/4.0  
(compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727)"  
TCP_MEM_HIT:NONE
87.160.144.136 - - [06/Apr/2007:13:28:08 +0200] "GET http:// 
www.csg.elseware.com:8080/content/e1756/index_html? HTTP/1.1" 200  
14826 "http://www.csg.elseware.com/content/e1756" "Mozilla/5.0  
(Windows; U; Windows NT 5.1; de; rv:1.8.0.11) Gecko/20070312 Firefox/ 
1.5.0.11" TCP_MISS:DIRECT


Varnish:

87.160.200.220 - - [10/Apr/2007:00:36:05 +0200] "GET /sites/nele- 
elseware.de/myzms/content/imgb01_ger.jpg HTTP/1.1" 304 - "http:// 
www.nele-elseware.de/content/e531/index_ger.html" "Mozilla/4.0  
(compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:36:26 +0200] "GET /sites/nele- 
elseware.de/myzms/content/e531/e706/milonga_del_serafin_ger.mp3 HTTP/ 
1.1" 304 - "http://www.nele-elseware.de/content/e531/index_ger.html"  
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:36:31 +0200] "GET /sites/nele- 
elseware.de/myzms/content/e531/e706/milonga_del_serafin_ger.mp3 HTTP/ 
1.1" 304 - "-" "NSPlayer/9.0.0.3250 WMFSDK/9.0"
87.160.200.220 - - [10/Apr/2007:00:41:23 +0200] "GET /sites/nele- 
elseware.de/myzms/content/e531/e686/el_choclo_ger.mp3 HTTP/1.1" 304 -  
"http://www.nele-elseware.de/content/e531/index_ger.html" "Mozilla/ 
4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:45:40 +0200] "GET /sites/nele- 
elseware.de/myzms/content/e531/e543/a_los_paisanos_ger.pdf HTTP/1.1"  
200 479110 "http://www.nele-elseware.de/content/e531/index_ger.html"  
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:29 +0200] "GET /common/logo.gif  
HTTP/1.1" 304 - "http://www.nele-elseware.de/content/e531/ 
index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;  
SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:29 +0200] "GET /sites/nele- 
elseware.de/myzms/content/imga01_ger.jpg HTTP/1.1" 304 - "http:// 
www.nele-elseware.de/content/e531/index_ger.html" "Mozilla/4.0  
(compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /common/deu.jpg  
HTTP/1.1" 304 - "http://www.nele-elseware.de/content/e531/ 
index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;  
SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /common/eng.jpg  
HTTP/1.1" 304 - "http://www.nele-elseware.de/content/e531/ 
index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;  
SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /common/esp.jpg  
HTTP/1.1" 304 - "http://www.nele-elseware.de/content/e531/ 
index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;  
SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /sites/nele- 
elseware.de/myzms/content/imgb01_ger.jpg HTTP/1.1" 304 - "http:// 
www.nele-elseware.de/content/e531/index_ger.html" "Mozilla/4.0  
(compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /misc_/zms/ 
mime_type.application_pdf.gif HTTP/1.1" 304 - "http://www.nele- 
elseware.de/content/e531/index_ger.html" "Mozilla/4.0 (compatible;  
MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /misc_/zms/ 
mime_type.audio_basic.gif HTTP/1.1" 304 - "http://www.nele- 
elseware.de/content/e531/index_ger.html" "Mozilla/4.0 (compatible;  
MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:46:41 +0200] "GET /sites/nele- 
elseware.de/myzms/content/e531/e684/ahora_no_me_conoces-nele- 
elseware_ger.mp3 HTTP/1.1" 304 - "http://www.nele-elseware.de/content/ 
e531/index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT  
5.1; SV1; HbTools 4.8.4)"
87.160.200.220 - - [10/Apr/2007:00:49:59 +0200] "GET /sites/nele- 
elseware.de/myzms/content/e531/e706/milonga_del_serafin_ger.mp3 HTTP/ 
1.1 HTTP/1.1" 304 - "-" "NSPlayer/9.0.0.3250 WMFSDK/9.0"
66.249.65.110 - - [10/Apr/2007:00:53:27 +0200] "GET /e43/e59/ 
sitemap_ger.html HTTP/1.1 HTTP/1.1" 304 - "-" "Mozilla/5.0  
(compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
88.198.251.202 - - [10/Apr/2007:01:19:14 +0200] "" 304 - "-" "-"
85.25.129.88 - - [10/Apr/2007:01:24:26 +0200] "GET / HTTP/1.0 HTTP/ 
1.1" 304 - "-" "Mozilla"
88.198.251.202 - - [10/Apr/2007:03:50:50 +0200] "" 304 - "-" "-"
207.46.98.56 - - [10/Apr/2007:03:55:56 +0200] "GET /robots.txt HTTP/ 
1.0 HTTP/1.1" 304 - "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"
66.249.65.110 - - [10/Apr/2007:04:08:24 +0200] "GET /e31/e78/ 
sitemap_ger.html HTTP/1.1 HTTP/1.1" 304 - "-" "Mozilla/5.0  
(compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
65.55.208.24 - - [10/Apr/2007:04:11:10 +0200] "GET /robots.txt HTTP/ 
1.0 HTTP/1.1" 304 - "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"
65.55.208.24 - - [10/Apr/2007:04:11:13 +0200] "GET /e53/ 
sitemap_ger.html HTTP/1.0 HTTP/1.1" 304 - "-" "msnbot/1.0 (+http:// 
search.msn.com/msnbot.htm)"
65.54.188.19 - - [10/Apr/2007:05:00:54 +0200] "GET /robots.txt HTTP/ 
1.0 HTTP/1.1" 304 - "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"
65.54.188.19 - - [10/Apr/2007:05:00:55 +0200] "GET /content/ 
index_ger.html HTTP/1.0 HTTP/1.1" 304 - "-" "msnbot/1.0 (+http:// 
search.msn.com/msnbot.htm)"
207.46.98.56 - - [10/Apr/2007:06:26:38 +0200] "GET /robots.txt HTTP/ 
1.0 HTTP/1.1" 304 - "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"
207.46.98.56 - - [10/Apr/2007:06:26:39 +0200] "GET /e61/ 
index_print_ger.html HTTP/1.0 HTTP/1.1" 304 - "-" "msnbot/1.0  
(+http://search.msn.com/msnbot.htm)"

-- Dirk

More information about the varnish-misc mailing list