[Varnish] #1830: VSL API: "duplicate link" errors in request grouping when vsl_buffer is increased

[Varnish]

#1830: VSL API: "duplicate link" errors in request grouping when vsl_buffer is
increased
-------------------------------------------+----------------------
 Reporter:  geoff                          |       Type:  defect
   Status:  new                            |   Priority:  normal
Milestone:  Varnish 4.0 release            |  Component:  varnishd
  Version:  4.0.3                          |   Severity:  normal
 Keywords:  VSL,duplicate link,vsl_buffer  |
-------------------------------------------+----------------------
 We have an app that uses the VSL API to extract data from the log, always
 in request grouping. Last week we increased vsl_buffer (to reduce SHM lock
 contention), and are now encountering very many "duplicate link" errors in
 the synthetic VSL record. It turns out that the errors no longer occur
 when we set vsl_buffer back to the lower value, and appear again when it
 is increased (we can "turn it on and off" by changing vsl_buffer with
 param.set).

 Previously we had vsl_buffer==default==4k, and increased it to 32k. By
 tweaking with param.set, I find that the errors are essentially gone with
 vsl_buffer up to about 5400 to 5500, although this was at low load on the
 site. The VSL errors seem to gradually occur more frequently as we
 increase vsl_buffer.

 I could get the same effect by running:

 {{{
 # with the same -L and -T parameters used by the app
 $ varnishlog -g request -q VSL
 }}}

 The query starts getting hits when vsl_buffer is above the same threshold,
 and more hits as vsl_buffer is increased.

 The site uses extensive nesting of ESI includes, up to at least
 esi_level==6, so request grouping assembles a large tree. As far as we can
 tell, we're not losing any of the data that the app is meant to collect; I
 wouldn't say this is certain, but so far VSL appears to me to be
 presenting all of the data that it should be. Put another way: so far I
 can't see anything else that is wrong due to the "duplicate links". So the
 problem is not a show-stopper (although we have the problem that we log
 all VSL records as errors, so our logs are currently being flooded).

-- 
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1830>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator