New challenge for Varnish!

[Rob S]

zabrane Mikael wrote:
> Hi Rob,
>
>     I don't think that's going to be possible with a pure VCL
>     approach.  Can you give us a better idea of exactly what you're
>     trying to achieve, as there might be a simpler / more
>     Varnish-friendly way of achieving what you're after.
>
>
> To make it simple, I'd like to store for a document A, all its raw 
> headers (both request and response) for later processing.
> A cron job will collect them (i.e these stored files) and send them to 
> a complex analysis system.
>
>     You can certainly use varnishlog to see the full headers for a
>     given UUID
>
>
> I'll give it try!
>
>     but the body content is more difficult.
>
>
> Is there another way to achieve that?
> Maybe by using some C to implement a callback for handling raw 
> "request" and "response".
You could use some C, but I don't know enough to comment about race 
conditions between concurrent requests if the same UUID came in at about 
the same time, or how to work gracefully if Varnish queued some 
concurrent requests to the same resource. 

If the UUID isn't that important, I'd consider doing the following:

Set up additional backends, on 127.0.0.2, called 'loggedsquid' and 
'loggedcustomerserver'.  Then, approximately have:


  // set default backend
  set req.backend = squid;

  if (req.http.X-RAWDATA != "") {
        set req.backend = customserver;
  }

 if (req.http.X-STORE!= "") {
     set req.backend = loggedsquid;

     if (req.http.X-RAWDATA != "") {
        set req.backend = loggedcustomserver;
     }
  }

THEN, on the varnish server, run (approximately) " tcpdump -s 0 -w 
loggedtraffic.pcap -d lo ip and host 127.0.0.2 "

Finally, on the cron, you could parse the pcap file, as that'd have the 
full headers and body, all in one coherent place.  Indeed, if the UUID 
is essential, there's no reason why you couldn't parse it out of the 
pcap file, work out the tcp port number for that request, then reparse 
the pcap to write out the packet contents to a specific file for that UUID.

I suppose it all depends on how regularly this logging would happen.


Rob