From hetardik.p at gmail.com Sat Mar 2 08:43:48 2019 From: hetardik.p at gmail.com (Hardik) Date: Sat, 2 Mar 2019 14:13:48 +0530 Subject: Reg: printing ReqURL two times in varnishlog Message-ID: Hi Team, we can see ReqURL is getting printed twice in varnish shm and so in varnishlog. It is causing problems while processing this logs further. some time it is very big URL and have parse twice. Is it possible to make it print only once. or any other work around ? ** << Request >> 313464548 -- Begin req 313464547 rxreq -- Timestamp Start: 1548836427.883183 0.000000 0.000000 -- Timestamp Req: 1548836427.883183 0.000000 0.000000 -- ReqStart 127.0.0.1 16962 a3 -- ReqMethod GET *-- ReqURL /abc/xyz/image.jpeg* -- ReqProtocol HTTP/1.1 -- ReqHeader X-Forwarded-For: x.x.x.x -- ReqHeader Host: aaa -- ReqHeader Connection: close -- ReqHeader accept: */* -- ReqHeader x-playback-session-id: xyz -- ReqHeader user-agent: AppleCoreMedia/1.0.0.16C101 (iPhone; U; CPU OS 12_1_2 like Mac OS X; en_us) -- ReqHeader accept-language: en-us -- ReqHeader accept-encoding: identity -- ReqHeader cookie: dtCookie==xyz -- ReqUnset X-Forwarded-For: x.x.x.x -- ReqHeader X-Forwarded-For: x.x.x.x, 127.0.0.1 -- VCL_call RECV -- VCL_Log x-xid:313464548 -- VCL_Log req-source:Client *-- ReqURL /abc/xyz/image.jpeg* -- ReqHeader X-VCL-Host: aaa -- VCL_Log checking aaa -- ReqHeader aaa Waiting for reply. Thank you in Advance. - Hardik -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Sat Mar 2 11:00:47 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Sat, 2 Mar 2019 01:00:47 -1000 Subject: Reg: printing ReqURL two times in varnishlog In-Reply-To: References: Message-ID: ReqURL gets printed every time you set req.url, so unless you don't modify it, you will have multiple lines. To be fair, it looks like this is an issue with your log parser. Just ignore those lines if you've already seen one and you're done. On Fri, Mar 1, 2019, 22:47 Hardik wrote: > Hi Team, > > we can see ReqURL is getting printed twice in varnish shm and so in > varnishlog. It is causing problems while processing this logs further. some > time it is very big URL and have parse twice. Is it possible to make it > print only once. or any other work around ? > > ** << Request >> 313464548 > -- Begin req 313464547 rxreq > -- Timestamp Start: 1548836427.883183 0.000000 0.000000 > -- Timestamp Req: 1548836427.883183 0.000000 0.000000 > -- ReqStart 127.0.0.1 16962 a3 > -- ReqMethod GET > *-- ReqURL /abc/xyz/image.jpeg* > -- ReqProtocol HTTP/1.1 > -- ReqHeader X-Forwarded-For: x.x.x.x > -- ReqHeader Host: aaa > -- ReqHeader Connection: close > -- ReqHeader accept: */* > -- ReqHeader x-playback-session-id: xyz > -- ReqHeader user-agent: AppleCoreMedia/1.0.0.16C101 (iPhone; U; CPU > OS 12_1_2 like Mac OS X; en_us) > -- ReqHeader accept-language: en-us > -- ReqHeader accept-encoding: identity > -- ReqHeader cookie: dtCookie==xyz > -- ReqUnset X-Forwarded-For: x.x.x.x > -- ReqHeader X-Forwarded-For: x.x.x.x, 127.0.0.1 > -- VCL_call RECV > -- VCL_Log x-xid:313464548 > -- VCL_Log req-source:Client > *-- ReqURL /abc/xyz/image.jpeg* > -- ReqHeader X-VCL-Host: aaa > -- VCL_Log checking aaa > -- ReqHeader aaa > > Waiting for reply. Thank you in Advance. > > - Hardik > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinakee at waltzz.com Tue Mar 5 07:15:12 2019 From: pinakee at waltzz.com (Pinakee BIswas) Date: Tue, 5 Mar 2019 12:45:12 +0530 Subject: ESI slowing down page loading Message-ID: <4dcf855d-bc21-b8de-15e2-7d3d5aa68838@waltzz.com> Hi, We are using varnish 4.1.8 and Django 1.11 for our application. We are using ESI to break down caching on a page. Part of pages which are dynamically loaded are fetched from the backend server using ESI but if there are multiple ESI calls on a page, the server response time and hence, the page loading increases. This is because of the increase in the number of calls to the backend server before Varnish serves the page to the client. Could you please let me know if there is a way to optimize the above? Thanks, Pinakee From hetardik.p at gmail.com Thu Mar 7 07:30:09 2019 From: hetardik.p at gmail.com (Hardik) Date: Thu, 7 Mar 2019 13:00:09 +0530 Subject: Reg: printing ReqURL two times in varnishlog In-Reply-To: References: Message-ID: Thank you Guillaume. we already planed to ignore that tag once its done in parser as you mentioned. Actually main reason behind asking this question was.. to use varnish shm maximum. If we print req.url multiple times then it will occupy more space in varnish shm. Correct me if i am wrong. One more question I have regarding timestamp... Timestamps related to Bereq are added in req Timestamps ( timers ) ? Timestamp Fetch: 1548836428.021777 *0.138593* 0.138593 Timestamp Resp: 1548836428.070475 0.187292 0.048588 Timestamp BerespBody: 1548836428.070220 *0.186739* 0.048602 *As per above logs, we can see 0.138593* < *0.186739 but it should >0.186739 in fetch timestamp right?* *or req and bereq counters are totally different and no relation between those ? Your comments will be really helpful.* Thank you Hardik On Sat, 2 Mar 2019 at 16:31, Guillaume Quintard < guillaume at varnish-software.com> wrote: > ReqURL gets printed every time you set req.url, so unless you don't modify > it, you will have multiple lines. > > To be fair, it looks like this is an issue with your log parser. Just > ignore those lines if you've already seen one and you're done. > > On Fri, Mar 1, 2019, 22:47 Hardik wrote: > >> Hi Team, >> >> we can see ReqURL is getting printed twice in varnish shm and so in >> varnishlog. It is causing problems while processing this logs further. some >> time it is very big URL and have parse twice. Is it possible to make it >> print only once. or any other work around ? >> >> ** << Request >> 313464548 >> -- Begin req 313464547 rxreq >> -- Timestamp Start: 1548836427.883183 0.000000 0.000000 >> -- Timestamp Req: 1548836427.883183 0.000000 0.000000 >> -- ReqStart 127.0.0.1 16962 a3 >> -- ReqMethod GET >> *-- ReqURL /abc/xyz/image.jpeg* >> -- ReqProtocol HTTP/1.1 >> -- ReqHeader X-Forwarded-For: x.x.x.x >> -- ReqHeader Host: aaa >> -- ReqHeader Connection: close >> -- ReqHeader accept: */* >> -- ReqHeader x-playback-session-id: xyz >> -- ReqHeader user-agent: AppleCoreMedia/1.0.0.16C101 (iPhone; U; >> CPU OS 12_1_2 like Mac OS X; en_us) >> -- ReqHeader accept-language: en-us >> -- ReqHeader accept-encoding: identity >> -- ReqHeader cookie: dtCookie==xyz >> -- ReqUnset X-Forwarded-For: x.x.x.x >> -- ReqHeader X-Forwarded-For: x.x.x.x, 127.0.0.1 >> -- VCL_call RECV >> -- VCL_Log x-xid:313464548 >> -- VCL_Log req-source:Client >> *-- ReqURL /abc/xyz/image.jpeg* >> -- ReqHeader X-VCL-Host: aaa >> -- VCL_Log checking aaa >> -- ReqHeader aaa >> >> Waiting for reply. Thank you in Advance. >> >> - Hardik >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Mar 7 18:50:27 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 7 Mar 2019 10:50:27 -0800 Subject: Reg: printing ReqURL two times in varnishlog In-Reply-To: References: Message-ID: The VSL is a circular buffer, so consuming more space only means you'll go through it more quickly, usually not a big deal. The Fetch timestamp means you have something to send, if streaming is on (and it is by default), you can start sending well before the body is completely processed. -- Guillaume Quintard On Wed, Mar 6, 2019 at 11:32 PM Hardik wrote: > Thank you Guillaume. we already planed to ignore that tag once its done in > parser as you mentioned. > Actually main reason behind asking this question was.. to use varnish shm > maximum. If we print req.url multiple times then it will occupy more space > in varnish shm. Correct me if i am wrong. > > One more question I have regarding timestamp... > Timestamps related to Bereq are added in req Timestamps ( timers ) ? > > Timestamp Fetch: 1548836428.021777 *0.138593* 0.138593 > Timestamp Resp: 1548836428.070475 0.187292 0.048588 > > Timestamp BerespBody: 1548836428.070220 *0.186739* 0.048602 > > *As per above logs, we can see 0.138593* < *0.186739 but it should > >0.186739 in fetch timestamp right?* > *or req and bereq counters are totally different and no relation between > those ? Your comments will be really helpful.* > > Thank you > Hardik > > On Sat, 2 Mar 2019 at 16:31, Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> ReqURL gets printed every time you set req.url, so unless you don't >> modify it, you will have multiple lines. >> >> To be fair, it looks like this is an issue with your log parser. Just >> ignore those lines if you've already seen one and you're done. >> >> On Fri, Mar 1, 2019, 22:47 Hardik wrote: >> >>> Hi Team, >>> >>> we can see ReqURL is getting printed twice in varnish shm and so in >>> varnishlog. It is causing problems while processing this logs further. some >>> time it is very big URL and have parse twice. Is it possible to make it >>> print only once. or any other work around ? >>> >>> ** << Request >> 313464548 >>> -- Begin req 313464547 rxreq >>> -- Timestamp Start: 1548836427.883183 0.000000 0.000000 >>> -- Timestamp Req: 1548836427.883183 0.000000 0.000000 >>> -- ReqStart 127.0.0.1 16962 a3 >>> -- ReqMethod GET >>> *-- ReqURL /abc/xyz/image.jpeg* >>> -- ReqProtocol HTTP/1.1 >>> -- ReqHeader X-Forwarded-For: x.x.x.x >>> -- ReqHeader Host: aaa >>> -- ReqHeader Connection: close >>> -- ReqHeader accept: */* >>> -- ReqHeader x-playback-session-id: xyz >>> -- ReqHeader user-agent: AppleCoreMedia/1.0.0.16C101 (iPhone; U; >>> CPU OS 12_1_2 like Mac OS X; en_us) >>> -- ReqHeader accept-language: en-us >>> -- ReqHeader accept-encoding: identity >>> -- ReqHeader cookie: dtCookie==xyz >>> -- ReqUnset X-Forwarded-For: x.x.x.x >>> -- ReqHeader X-Forwarded-For: x.x.x.x, 127.0.0.1 >>> -- VCL_call RECV >>> -- VCL_Log x-xid:313464548 >>> -- VCL_Log req-source:Client >>> *-- ReqURL /abc/xyz/image.jpeg* >>> -- ReqHeader X-VCL-Host: aaa >>> -- VCL_Log checking aaa >>> -- ReqHeader aaa >>> >>> Waiting for reply. Thank you in Advance. >>> >>> - Hardik >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Mar 11 08:09:37 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 11 Mar 2019 09:09:37 +0100 Subject: ESI slowing down page loading In-Reply-To: <4dcf855d-bc21-b8de-15e2-7d3d5aa68838@waltzz.com> References: <4dcf855d-bc21-b8de-15e2-7d3d5aa68838@waltzz.com> Message-ID: Hello Pinakee, On Tue, Mar 5, 2019 at 8:16 AM Pinakee BIswas wrote: > > Hi, > > We are using varnish 4.1.8 and Django 1.11 for our application. > > We are using ESI to break down caching on a page. Part of pages which > are dynamically loaded are fetched from the backend server using ESI but > if there are multiple ESI calls on a page, the server response time and > hence, the page loading increases. This is because of the increase in > the number of calls to the backend server before Varnish serves the page > to the client. > > Could you please let me know if there is a way to optimize the above? Unfortunately there's little you can do to lower the latency on heavy ESI misses or passes with Varnish Cache. Varnish Plus on the other hand has parallel ESI that should bring down latency to the longest fetch instead of all of them: https://info.varnish-software.com/blog/varnish-lab-parallel-esi https://info.varnish-software.com/blog/parallel-versus-serial-esi-processing-benchmarks Dridi From hetardik.p at gmail.com Tue Mar 12 10:07:07 2019 From: hetardik.p at gmail.com (Hardik) Date: Tue, 12 Mar 2019 15:37:07 +0530 Subject: reg: not getting ReqAcct tag in varnishlog Message-ID: Hi Team, I am not getting *ReqAcct* tag in vanishlog. From this tag I am reading *total bytes* served. As this tag is not present we are loosing billing. Out of 100 Requests 90 requests I am not getting ReqAcct tag. Mostly this problem have seen for mobile requests but not all time. *Error cases when ReqAcct is not present in varnishlog* 1. - VSL timeout - End synt 2. - VSL store overflow - End synth 3. Pastebin 1. - VCL_return deliver 2. - Timestamp Process: 1548350435.479732 0.000604 0.000604 3. - RespHeader Connection: keep-alive 4. - Debug "Hit idle send timeout, wrote = 293944/20534511; retrying%00" 5. - Debug "Hit idle send timeout, wrote = 91224/20240567; retrying%00" 6. - Debug "Hit idle send timeout, wrote = 149144/20149343; retrying%00" 7. - Debug "Hit idle send timeout, wrote = 165072/20000199; retrying%00" 8. - Debug "Hit idle send timeout, wrote = 85432/19835127; retrying%00" 9. - Debug "Hit idle send timeout, wrote = 233128/19749695; retrying%00" 10. - VSL timeout 11. - End synth Is it any change require regarding this? can we change any tunable parameter ? any VCL flow change can help ? Thank you Hardik -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Mar 12 10:50:10 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 12 Mar 2019 11:50:10 +0100 Subject: reg: not getting ReqAcct tag in varnishlog In-Reply-To: References: Message-ID: Hello Hardik, > Is it any change require regarding this? can we change any tunable parameter ? any VCL flow change can help ? Can you please tell us more about which version of Varnish you are using? $ varnishd -V Please also specify which OS you are running Varnish on, and how you installed Varnish. Thanks, Dridi From hetardik.p at gmail.com Tue Mar 12 10:51:32 2019 From: hetardik.p at gmail.com (Hardik) Date: Tue, 12 Mar 2019 16:21:32 +0530 Subject: reg: not getting ReqAcct tag in varnishlog In-Reply-To: References: Message-ID: Hi Dridi, Varnish version - 6.0.1 OS - centos 7 I checked out vanish 6.0.1 source code, built rpm and installed. Please let me know if more information is required. Thank you Hardik On Tue, 12 Mar 2019 at 16:20, Dridi Boukelmoune wrote: > Hello Hardik, > > > Is it any change require regarding this? can we change any tunable > parameter ? any VCL flow change can help ? > > Can you please tell us more about which version of Varnish you are using? > > $ varnishd -V > > Please also specify which OS you are running Varnish on, and how you > installed Varnish. > > Thanks, > Dridi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Mar 12 11:15:35 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 12 Mar 2019 12:15:35 +0100 Subject: reg: not getting ReqAcct tag in varnishlog In-Reply-To: References: Message-ID: On Tue, Mar 12, 2019 at 11:54 AM Hardik wrote: > > Hi Dridi, > > Varnish version - 6.0.1 > OS - centos 7 > > I checked out vanish 6.0.1 source code, built rpm and installed. > > Please let me know if more information is required. Do you have a reliable way to reproduce this? If so, can you describe it and does it still happen with 6.0.3? Thanks, Dridi From pinakee at waltzz.com Tue Mar 12 11:41:44 2019 From: pinakee at waltzz.com (Pinakee BIswas) Date: Tue, 12 Mar 2019 17:11:44 +0530 Subject: ESI slowing down page loading In-Reply-To: References: <4dcf855d-bc21-b8de-15e2-7d3d5aa68838@waltzz.com> Message-ID: <917f82fa-1f78-4079-762c-c8858fb9b9e1@waltzz.com> Hi Dridi, Thanks for your response and the details. I did get the info on PESI from someone from the varnish plus team. Would certainly look into it but being a young company, would have also factor in the cost considering our web traffic. Thanks, Pinakee On 11/03/19 1:39 PM, Dridi Boukelmoune wrote: > Hello Pinakee, > > On Tue, Mar 5, 2019 at 8:16 AM Pinakee BIswas wrote: >> Hi, >> >> We are using varnish 4.1.8 and Django 1.11 for our application. >> >> We are using ESI to break down caching on a page. Part of pages which >> are dynamically loaded are fetched from the backend server using ESI but >> if there are multiple ESI calls on a page, the server response time and >> hence, the page loading increases. This is because of the increase in >> the number of calls to the backend server before Varnish serves the page >> to the client. >> >> Could you please let me know if there is a way to optimize the above? > Unfortunately there's little you can do to lower the latency on heavy > ESI misses or passes with Varnish Cache. > > Varnish Plus on the other hand has parallel ESI that should bring down > latency to the longest fetch instead of all of them: > > https://info.varnish-software.com/blog/varnish-lab-parallel-esi > https://info.varnish-software.com/blog/parallel-versus-serial-esi-processing-benchmarks > > Dridi From slink at schokola.de Tue Mar 12 12:02:32 2019 From: slink at schokola.de (Nils Goroll) Date: Tue, 12 Mar 2019 13:02:32 +0100 Subject: reg: not getting ReqAcct tag in varnishlog In-Reply-To: References: Message-ID: Hi, On 12/03/2019 11:07, Hardik wrote: > *Error cases when ReqAcct is not present in varnishlog* > > 1. > -?? VSL??????????? timeout > -?? End??????????? synt > > 2. > -?? VSL??????????? store overflow > -?? End??????????? synth The VSL clients (varnishlog and friends) store transactions in the client program's memory when varnishd comes close to overwriting the respective region of the VSL shared memory ring. The -L and -T options allow you to specify these limits, see for example the varnishlog man page. I _suspect_ you are seeing this with the -g request or -g session transaction grouping (please specify how you invoke the respective command). If yes, an alternative would be to reduce the time the VCL client needs to store open transactions by switching to the vxid or raw grouping (-g argument). hth, Nils From dridi at varni.sh Tue Mar 12 14:18:09 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 12 Mar 2019 15:18:09 +0100 Subject: ESI slowing down page loading In-Reply-To: <917f82fa-1f78-4079-762c-c8858fb9b9e1@waltzz.com> References: <4dcf855d-bc21-b8de-15e2-7d3d5aa68838@waltzz.com> <917f82fa-1f78-4079-762c-c8858fb9b9e1@waltzz.com> Message-ID: Hi Pinakee, On Tue, Mar 12, 2019 at 12:41 PM Pinakee BIswas wrote: > > Hi Dridi, > > Thanks for your response and the details. > > I did get the info on PESI from someone from the varnish plus team. > > Would certainly look into it but being a young company, would have also > factor in the cost considering our web traffic. That's perfectly understandable. You can also try Varnish Plus cloud images to give a try to pESI without a significant upfront investment, and see at the very least whether that brings an improvement to your case. I think it's available for the 4.1 series too. By the way, I encourage you to update to the latest 4.1 release (4.1.11 if my memory serves well) and plan an upgrade to 6.0 LTS because 4.1 is going EOL soon for Varnish Cache (in case you missed the announcement). Cheers, Dridi From hetardik.p at gmail.com Wed Mar 13 14:52:30 2019 From: hetardik.p at gmail.com (Hardik) Date: Wed, 13 Mar 2019 20:22:30 +0530 Subject: reg: not getting ReqAcct tag in varnishlog In-Reply-To: References: Message-ID: Hi Dridi, We should able to recreate with load and mobile requests. I have not tried with 6.0.3. @Nils, We are seeing issue with both varnishlog and varnishlog with -g option. But here problem is, shared memory it self does not have ReqAcct tag I think ( please correct me if I am wrong). Because all the clients which are reading shm all are getting same thing..means no ReqAcct. But yes I am agree that impact with "varnishlog -g session" is more. So If shared memory it self has no ReqAcct tag then all clients will also not get right ? How to fix this problem ? Please help with some details which I can understand because we are loosing bills for which we are serving traffic...! *Normal varnish command we use to grep running logs* varnishlog -g request -q "ReqURL ~ '/abc/xyz'" *command uses to read shared memory directly for billing* varnishlog -g session --> we are already planing to use "varnishlog -g xvid" for billing api. Because I understood is, -g session option is taking more time to arrange in particular order and delivery final output. Please help with some more detail.. It will really helpful. Thank you Hardik On Tue, 12 Mar 2019 at 17:32, Nils Goroll wrote: > Hi, > > On 12/03/2019 11:07, Hardik wrote: > > *Error cases when ReqAcct is not present in varnishlog* > > > > 1. > > - VSL timeout > > - End synt > > > > 2. > > - VSL store overflow > > - End synth > > The VSL clients (varnishlog and friends) store transactions in the client > program's memory when varnishd comes close to overwriting the respective > region > of the VSL shared memory ring. > > The -L and -T options allow you to specify these limits, see for example > the > varnishlog man page. > > I _suspect_ you are seeing this with the -g request or -g session > transaction > grouping (please specify how you invoke the respective command). If yes, an > alternative would be to reduce the time the VCL client needs to store open > transactions by switching to the vxid or raw grouping (-g argument). > > hth, Nils > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Mar 13 17:19:17 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 13 Mar 2019 18:19:17 +0100 Subject: reg: not getting ReqAcct tag in varnishlog In-Reply-To: References: Message-ID: On Wed, Mar 13, 2019 at 3:56 PM Hardik wrote: > > Hi Dridi, > > We should able to recreate with load and mobile requests. I have not tried with 6.0.3. I guess there's no need for that. Your varnishlog setup is stretched too thin to cope with your load. I was completely oblivious to the problems slink spotted right away... > @Nils, > We are seeing issue with both varnishlog and varnishlog with -g option. But here problem is, shared memory it self does not have ReqAcct tag I think ( please correct me if I am wrong). Because all the clients which are reading shm all are getting same thing..means no ReqAcct. But yes I am agree that impact with "varnishlog -g session" is more. > > So If shared memory it self has no ReqAcct tag then all clients will also not get right ? How to fix this problem ? Please help with some details which I can understand because we are loosing bills for which we are serving traffic...! You could try avoiding grouping, use varnishncsa with a custom format, overall store and process less data in memory. > Normal varnish command we use to grep running logs > varnishlog -g request -q "ReqURL ~ '/abc/xyz'" > > command uses to read shared memory directly for billing > varnishlog -g session > --> we are already planing to use "varnishlog -g xvid" for billing api. Because I understood is, -g session option is taking more time to arrange in particular order and delivery final output. Please help with some more detail.. It will really helpful. There are few practical uses of -g session for live traffic. This works better on offline logs for example when examining traffic post-mortem. If only ReqAcct is important, and the rest of the information you need is available on the request side, you should definitely stop using the -g option, and use the -c option to further limit internal processing. If you need information from the backend transactions too, try to figure how you could make this information available on the client side. Dridi From hetardik.p at gmail.com Sun Mar 17 12:09:13 2019 From: hetardik.p at gmail.com (Hardik) Date: Sun, 17 Mar 2019 17:39:13 +0530 Subject: reg: not getting ReqAcct tag in varnishlog In-Reply-To: References: Message-ID: Thanks a lot Dridi & Team for details.. You could try avoiding grouping, use varnishncsa with a custom format, overall store and process less data in memory. *-I was analyzing varnishncsa logs as well and could see problem with it also. Means missing ReqAcct sometime.* *Does that means shared memory its self does not have ReqAcct tag ? * *-Is grouping happen to clients memory ? means client reads from shared memory and then does grouping in client's memory ? Please give some details so based on that I can arrange the things. Also I am curious to understand. Any document link or point out any coding part, I will go through that as well.* If only ReqAcct is important, and the rest of the information you need is available on the request side, you should definitely stop using the -g option, and use the -c option to further limit internal processing. *-Yes, we are reading now request side only ( till now was reading session also but found out other alternative so stopped that after your comment ). * *-I have few doubts here.. is there a difference between NOT assigning -g option to g_arg(means keep default to xvid ) and assigning -c option ? * *-Currently I have stopped using -g option in service which is reading shm so kept to default option. To by pass berequest tags I have started using "VSL_t_bereq" structure member. If -c option is still better than this then I will test out that also. Please suggest..* Thank you Hardik On Wed, 13 Mar 2019 at 22:49, Dridi Boukelmoune wrote: > On Wed, Mar 13, 2019 at 3:56 PM Hardik wrote: > > > > Hi Dridi, > > > > We should able to recreate with load and mobile requests. I have not > tried with 6.0.3. > > I guess there's no need for that. Your varnishlog setup is stretched > too thin to cope with your load. > > I was completely oblivious to the problems slink spotted right away... > > > @Nils, > > We are seeing issue with both varnishlog and varnishlog with -g option. > But here problem is, shared memory it self does not have ReqAcct tag I > think ( please correct me if I am wrong). Because all the clients which are > reading shm all are getting same thing..means no ReqAcct. But yes I am > agree that impact with "varnishlog -g session" is more. > > > > So If shared memory it self has no ReqAcct tag then all clients will > also not get right ? How to fix this problem ? Please help with some > details which I can understand because we are loosing bills for which we > are serving traffic...! > > You could try avoiding grouping, use varnishncsa with a custom format, > overall store and process less data in memory. > > > Normal varnish command we use to grep running logs > > varnishlog -g request -q "ReqURL ~ '/abc/xyz'" > > > > command uses to read shared memory directly for billing > > varnishlog -g session > > --> we are already planing to use "varnishlog -g xvid" for billing api. > Because I understood is, -g session option is taking more time to arrange > in particular order and delivery final output. Please help with some more > detail.. It will really helpful. > > There are few practical uses of -g session for live traffic. This > works better on offline logs for example when examining traffic > post-mortem. > > If only ReqAcct is important, and the rest of the information you need > is available on the request side, you should definitely stop using the > -g option, and use the -c option to further limit internal processing. > > If you need information from the backend transactions too, try to > figure how you could make this information available on the client > side. > > Dridi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hetardik.p at gmail.com Mon Mar 18 11:09:51 2019 From: hetardik.p at gmail.com (Hardik) Date: Mon, 18 Mar 2019 16:39:51 +0530 Subject: Reg: exact field for response time Message-ID: Hi Team, query is not related to any issue. Need understanding/clarification regarding exact response time fields between varnish 3 and varnish 6.1. *varnish 3* ReqEnd c 877345549 1233949945.075706005 1233949945.075754881 0.017112017 0.000022888 *0.000025988* https://github.com/jib/libvmod-timers/blob/master/src/vmod_timers.c The sixth and last number (0.000025988) is the time from we start delivering the object until the request completes. *varnish 6.1* Timestamp Resp: 1501601912.806787 0.048125 *0.000037* https://feryn.eu/blog/varnishlog-measure-varnish-cache-performance/#The_Timestamp_tag -> Are both fields give same values, means same meaning or both are different? In varnish 3 I was reading last field. And as per explanation its time from delivery to end of request ( and not total time from start of request till end, plz correct if wrong). Based on this explanation I am reading resp time only and no total time in resp tag. Do I have to read 0.048125 to be compatible with varnish 3 or I am doing correct ? Thank you Hardik -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Mon Mar 18 11:22:09 2019 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 18 Mar 2019 12:22:09 +0100 Subject: Reg: exact field for response time In-Reply-To: References: Message-ID: <008d5d67-011f-a56a-ad38-e7d9a90aafa4@uplex.de> On 3/18/19 12:09, Hardik wrote: > > *varnish 6.1* > Timestamp Resp: 1501601912.806787 0.048125 *0.000037* > > https://feryn.eu/blog/varnishlog-measure-varnish-cache-performance/#The_Timestamp_tag > > > -> Are both fields give same values, means same meaning or both are > different? In varnish 3 I was reading last field. And as per explanation > its time from delivery to end of request ( and not total time from start of > request till end, plz correct if wrong). Based on this explanation I am > reading resp time only and no total time in resp tag. Do I have to read > 0.048125 to be compatible with varnish 3 or I am doing correct ? man vsl(7) is your friend here: The format is: %s: %f %f %f | | | | | | | +- Time since last timestamp | | +---- Time since start of work unit | +------- Absolute time of event +----------- Event label So the second number is the time since request processing began, and the third number is the time since the previous timestamp. Timestamp:Resp usually comes after Timestamp:Process: Process Processing finished, ready to deliver the client response. Resp Delivery of response to the client finished. So in your example, it's 48.125ms since the start of request processing, and 37?s since the previous timestamp, usually meaning 37?s for response delivery. The second field of Timestamp:Resp (48ms here) is a good measure of the total time to process the request. HTH, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From dridi at varni.sh Mon Mar 18 15:13:01 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 18 Mar 2019 16:13:01 +0100 Subject: reg: not getting ReqAcct tag in varnishlog In-Reply-To: References: Message-ID: On Sun, Mar 17, 2019 at 1:12 PM Hardik wrote: > > Thanks a lot Dridi & Team for details.. Before I can answer your questions, can you explain exactly what you are trying to do? We could probably give you better advice if we knew what you need to collect. Dridi From hetardik.p at gmail.com Mon Mar 18 15:50:32 2019 From: hetardik.p at gmail.com (Hardik) Date: Mon, 18 Mar 2019 21:20:32 +0530 Subject: Reg: exact field for response time In-Reply-To: <008d5d67-011f-a56a-ad38-e7d9a90aafa4@uplex.de> References: <008d5d67-011f-a56a-ad38-e7d9a90aafa4@uplex.de> Message-ID: Hi Geoff, Thanks for reply. I understood meaning of the fields in varnish 6. One answer I got that we should consider 0.048125 as total response time. But my actual question is different. I am not getting difference between varnish 3 and varnish 6. Exact question if I ask.. *varnish 3* ReqEnd c 877345549 1233949945.075706005 1233949945.075754881 0.017112017 0.000022888 *0.000025988* *varnish 6.1* Timestamp Resp: 1501601912.806787* 0.048125* *0.000037* *0.000025988 (varnish 3) is equivalent to 0.048125 or 0.000037 (varnish 6) ?* we have upgraded from 3 to 6 so not able to decide exact field which I was reading in case of varnish 3. *Thank you* *Hardik* On Mon, 18 Mar 2019 at 16:52, Geoff Simmons wrote: > On 3/18/19 12:09, Hardik wrote: > > > > *varnish 6.1* > > Timestamp Resp: 1501601912.806787 0.048125 *0.000037* > > > > > https://feryn.eu/blog/varnishlog-measure-varnish-cache-performance/#The_Timestamp_tag > > > > > > -> Are both fields give same values, means same meaning or both are > > different? In varnish 3 I was reading last field. And as per explanation > > its time from delivery to end of request ( and not total time from start > of > > request till end, plz correct if wrong). Based on this explanation I am > > reading resp time only and no total time in resp tag. Do I have to read > > 0.048125 to be compatible with varnish 3 or I am doing correct ? > > man vsl(7) is your friend here: > > The format is: > > %s: %f %f %f > | | | | > | | | +- Time since last timestamp > | | +---- Time since start of work unit > | +------- Absolute time of event > +----------- Event label > > So the second number is the time since request processing began, and the > third number is the time since the previous timestamp. > > Timestamp:Resp usually comes after Timestamp:Process: > > Process > Processing finished, ready to deliver the client response. > > Resp Delivery of response to the client finished. > > So in your example, it's 48.125ms since the start of request processing, > and 37?s since the previous timestamp, usually meaning 37?s for response > delivery. > > The second field of Timestamp:Resp (48ms here) is a good measure of the > total time to process the request. > > > HTH, > Geoff > -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hetardik.p at gmail.com Mon Mar 18 16:51:05 2019 From: hetardik.p at gmail.com (Hardik) Date: Mon, 18 Mar 2019 22:21:05 +0530 Subject: reg: not getting ReqAcct tag in varnishlog In-Reply-To: References: Message-ID: Hi Dridi, I am reading few tags for Billing purpose. I have added VMOD for this.In vmod currently I was passing "-g session" option to varnish callback function. But *I found out that not getting ReqAcct tag. Also I am missing lots of logs itself.* After with discussion with you guys it seems problem is due to "-g session" option. So I removed that and tested again and looks better then before. Reduced frequency of missing ReqAcct and Log loss decreased a lot. Now, my doubt, Is it better not to use any option and keep default setting, or to use -c option ? So I can minimize log loss as many as I can. Here is small from vmod which is reading shared memory (*it was lot easier in varnish 3 because I was able to call dispatch function directly*), vut = VUT_Init(argv[0], 1, argv, &s); vut->dispatch_f = VarnishLog::handler; vut->dispatch_priv = this; vut->g_arg = 3; <---- I commented out this now ( -g session ) vut->sighup = 1; vut->sighup_f = VarnishLog::sighup; VUT_Setup(vut); VUT_Main(vut); VUT_Fini(&vut); I can not change above whole setup but can modify few things in that. Now, If you can answer previous questions will be really helpful. Particularly how -g session option creating problem ? Thank you Hardik On Mon, 18 Mar 2019 at 20:43, Dridi Boukelmoune wrote: > On Sun, Mar 17, 2019 at 1:12 PM Hardik wrote: > > > > Thanks a lot Dridi & Team for details.. > > Before I can answer your questions, can you explain exactly what you > are trying to do? > > We could probably give you better advice if we knew what you need to > collect. > > Dridi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Mar 18 19:12:34 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 18 Mar 2019 20:12:34 +0100 Subject: reg: not getting ReqAcct tag in varnishlog In-Reply-To: References: Message-ID: On Mon, Mar 18, 2019 at 5:53 PM Hardik wrote: > > Hi Dridi, > > I am reading few tags for Billing purpose. I have added VMOD for this.In vmod currently I was passing "-g session" option to varnish callback function. But I found out that not getting ReqAcct tag. Also I am missing lots of logs itself. > > After with discussion with you guys it seems problem is due to "-g session" option. So I removed that and tested again and looks better then before. Reduced frequency of missing ReqAcct and Log loss decreased a lot. > > Now, my doubt, Is it better not to use any option and keep default setting, or to use -c option ? So I can minimize log loss as many as I can. > > Here is small from vmod which is reading shared memory (it was lot easier in varnish 3 because I was able to call dispatch function directly), > vut = VUT_Init(argv[0], 1, argv, &s); > vut->dispatch_f = VarnishLog::handler; > vut->dispatch_priv = this; > vut->g_arg = 3; <---- I commented out this now ( -g session ) > vut->sighup = 1; > vut->sighup_f = VarnishLog::sighup; > VUT_Setup(vut); > VUT_Main(vut); > VUT_Fini(&vut); So you wrote your own log utility in C++? Wow! > I can not change above whole setup but can modify few things in that. > > Now, If you can answer previous questions will be really helpful. Particularly how -g session option creating problem ? Sorry but this is not what I was looking for. [1] Can you give me a list of log records you need to collect? And possibly how you are trying to group them if they come from different transactions? There's a lot we can do without building a new utility. Dridi [1] http://xyproblem.info/ From hetardik.p at gmail.com Tue Mar 19 00:39:09 2019 From: hetardik.p at gmail.com (Hardik) Date: Tue, 19 Mar 2019 06:09:09 +0530 Subject: reg: not getting ReqAcct tag in varnishlog In-Reply-To: References: Message-ID: Hi Dridi, Can you give me a list of log records you need to collect? SLT_Timestamp : SLT_ReqStart : SLT_ReqMethod : SLT_ReqURL: SLT_ReqProtocol : SLT_RespStatus : SLT_ReqHeader : SLT_RespHeader : SLT_ReqAcct : SLT_BereqAcct : SLT_VCL_Log : And possibly how you are trying to group them if they come from different transactions? *I am reading based xid ( by FD ). Means reading full records per fd.* Please let me know if any other information I can provide.. If this is not related to my problem still I am curious to know how grouping is happening. You can point out some code or links with some details, I will go through. Thank you Hardik On Tue, 19 Mar 2019 at 00:43, Dridi Boukelmoune wrote: > On Mon, Mar 18, 2019 at 5:53 PM Hardik wrote: > > > > Hi Dridi, > > > > I am reading few tags for Billing purpose. I have added VMOD for this.In > vmod currently I was passing "-g session" option to varnish callback > function. But I found out that not getting ReqAcct tag. Also I am missing > lots of logs itself. > > > > After with discussion with you guys it seems problem is due to "-g > session" option. So I removed that and tested again and looks better then > before. Reduced frequency of missing ReqAcct and Log loss decreased a lot. > > > > Now, my doubt, Is it better not to use any option and keep default > setting, or to use -c option ? So I can minimize log loss as many as I can. > > > > Here is small from vmod which is reading shared memory (it was lot > easier in varnish 3 because I was able to call dispatch function directly), > > vut = VUT_Init(argv[0], 1, argv, &s); > > vut->dispatch_f = VarnishLog::handler; > > vut->dispatch_priv = this; > > vut->g_arg = 3; <---- I commented out this now ( -g session ) > > vut->sighup = 1; > > vut->sighup_f = VarnishLog::sighup; > > VUT_Setup(vut); > > VUT_Main(vut); > > VUT_Fini(&vut); > > So you wrote your own log utility in C++? Wow! > > > I can not change above whole setup but can modify few things in that. > > > > Now, If you can answer previous questions will be really helpful. > Particularly how -g session option creating problem ? > > Sorry but this is not what I was looking for. [1] > > Can you give me a list of log records you need to collect? And > possibly how you are trying to group them if they come from different > transactions? There's a lot we can do without building a new utility. > > Dridi > > [1] http://xyproblem.info/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hetardik.p at gmail.com Tue Mar 19 06:18:40 2019 From: hetardik.p at gmail.com (Hardik) Date: Tue, 19 Mar 2019 11:48:40 +0530 Subject: Reg: exact field for response time In-Reply-To: References: <008d5d67-011f-a56a-ad38-e7d9a90aafa4@uplex.de> Message-ID: This line is creating confusion for me.. https://github.com/jib/libvmod-timers/blob/master/src/vmod_timers.c The sixth and last number (0.000025988) is the time from we start delivering the object until the request completes. Thanks Hardik On Mon, 18 Mar 2019 at 21:20, Hardik wrote: > Hi Geoff, > > Thanks for reply. I understood meaning of the fields in varnish 6. > > One answer I got that we should consider 0.048125 as total response time. > But my actual question is different. > I am not getting difference between varnish 3 and varnish 6. > > Exact question if I ask.. > > *varnish 3* > ReqEnd c 877345549 1233949945.075706005 1233949945.075754881 0.017112017 > 0.000022888 *0.000025988* > > *varnish 6.1* > Timestamp Resp: 1501601912.806787* 0.048125* *0.000037* > > *0.000025988 (varnish 3) is equivalent to 0.048125 or 0.000037 (varnish 6) > ?* > > we have upgraded from 3 to 6 so not able to decide exact field which I was > reading in case of varnish 3. > > *Thank you* > *Hardik* > > > > On Mon, 18 Mar 2019 at 16:52, Geoff Simmons wrote: > >> On 3/18/19 12:09, Hardik wrote: >> > >> > *varnish 6.1* >> > Timestamp Resp: 1501601912.806787 0.048125 *0.000037* >> > >> > >> https://feryn.eu/blog/varnishlog-measure-varnish-cache-performance/#The_Timestamp_tag >> > >> > >> > -> Are both fields give same values, means same meaning or both are >> > different? In varnish 3 I was reading last field. And as per explanation >> > its time from delivery to end of request ( and not total time from >> start of >> > request till end, plz correct if wrong). Based on this explanation I am >> > reading resp time only and no total time in resp tag. Do I have to read >> > 0.048125 to be compatible with varnish 3 or I am doing correct ? >> >> man vsl(7) is your friend here: >> >> The format is: >> >> %s: %f %f %f >> | | | | >> | | | +- Time since last timestamp >> | | +---- Time since start of work unit >> | +------- Absolute time of event >> +----------- Event label >> >> So the second number is the time since request processing began, and the >> third number is the time since the previous timestamp. >> >> Timestamp:Resp usually comes after Timestamp:Process: >> >> Process >> Processing finished, ready to deliver the client response. >> >> Resp Delivery of response to the client finished. >> >> So in your example, it's 48.125ms since the start of request processing, >> and 37?s since the previous timestamp, usually meaning 37?s for response >> delivery. >> >> The second field of Timestamp:Resp (48ms here) is a good measure of the >> total time to process the request. >> >> >> HTH, >> Geoff >> -- >> ** * * UPLEX - Nils Goroll Systemoptimierung >> >> Scheffelstra?e 32 >> 22301 Hamburg >> >> Tel +49 40 2880 5731 >> Mob +49 176 636 90917 >> Fax +49 40 42949753 >> >> http://uplex.de >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From hetardik.p at gmail.com Tue Mar 19 10:00:55 2019 From: hetardik.p at gmail.com (Hardik) Date: Tue, 19 Mar 2019 15:30:55 +0530 Subject: Reg: exact field for response time In-Reply-To: References: <008d5d67-011f-a56a-ad38-e7d9a90aafa4@uplex.de> Message-ID: Hi Geoff & Team, Please help with understanding. Thank you Hardik On Tue, 19 Mar 2019 at 11:48, Hardik wrote: > This line is creating confusion for me.. > > https://github.com/jib/libvmod-timers/blob/master/src/vmod_timers.c > The sixth and last number (0.000025988) is the time from we start > delivering the object until the request completes. > > Thanks > Hardik > > On Mon, 18 Mar 2019 at 21:20, Hardik wrote: > >> Hi Geoff, >> >> Thanks for reply. I understood meaning of the fields in varnish 6. >> >> One answer I got that we should consider 0.048125 as total response time. >> But my actual question is different. >> I am not getting difference between varnish 3 and varnish 6. >> >> Exact question if I ask.. >> >> *varnish 3* >> ReqEnd c 877345549 1233949945.075706005 1233949945.075754881 0.017112017 >> 0.000022888 *0.000025988* >> >> *varnish 6.1* >> Timestamp Resp: 1501601912.806787* 0.048125* *0.000037* >> >> *0.000025988 (varnish 3) is equivalent to 0.048125 or 0.000037 (varnish >> 6) ?* >> >> we have upgraded from 3 to 6 so not able to decide exact field which I >> was reading in case of varnish 3. >> >> *Thank you* >> *Hardik* >> >> >> >> On Mon, 18 Mar 2019 at 16:52, Geoff Simmons wrote: >> >>> On 3/18/19 12:09, Hardik wrote: >>> > >>> > *varnish 6.1* >>> > Timestamp Resp: 1501601912.806787 0.048125 *0.000037* >>> > >>> > >>> https://feryn.eu/blog/varnishlog-measure-varnish-cache-performance/#The_Timestamp_tag >>> > >>> > >>> > -> Are both fields give same values, means same meaning or both are >>> > different? In varnish 3 I was reading last field. And as per >>> explanation >>> > its time from delivery to end of request ( and not total time from >>> start of >>> > request till end, plz correct if wrong). Based on this explanation I am >>> > reading resp time only and no total time in resp tag. Do I have to read >>> > 0.048125 to be compatible with varnish 3 or I am doing correct ? >>> >>> man vsl(7) is your friend here: >>> >>> The format is: >>> >>> %s: %f %f %f >>> | | | | >>> | | | +- Time since last timestamp >>> | | +---- Time since start of work unit >>> | +------- Absolute time of event >>> +----------- Event label >>> >>> So the second number is the time since request processing began, and the >>> third number is the time since the previous timestamp. >>> >>> Timestamp:Resp usually comes after Timestamp:Process: >>> >>> Process >>> Processing finished, ready to deliver the client response. >>> >>> Resp Delivery of response to the client finished. >>> >>> So in your example, it's 48.125ms since the start of request processing, >>> and 37?s since the previous timestamp, usually meaning 37?s for response >>> delivery. >>> >>> The second field of Timestamp:Resp (48ms here) is a good measure of the >>> total time to process the request. >>> >>> >>> HTH, >>> Geoff >>> -- >>> ** * * UPLEX - Nils Goroll Systemoptimierung >>> >>> Scheffelstra?e 32 >>> 22301 Hamburg >>> >>> Tel +49 40 2880 5731 >>> Mob +49 176 636 90917 >>> Fax +49 40 42949753 >>> >>> http://uplex.de >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Tue Mar 19 12:36:06 2019 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 19 Mar 2019 13:36:06 +0100 Subject: Reg: exact field for response time In-Reply-To: References: <008d5d67-011f-a56a-ad38-e7d9a90aafa4@uplex.de> Message-ID: On 3/19/19 11:00, Hardik wrote: > > Please help with understanding. Well I tried with my previous answer, but you didn't seem to be satisfied with that. I'm not sure what else I can say. >> This line is creating confusion for me.. >> >> https://github.com/jib/libvmod-timers/blob/master/src/vmod_timers.c >> The sixth and last number (0.000025988) is the time from we start >> delivering the object until the request completes. That appears to be exactly the same as what we had in the old wiki, which as I recall was the only documentation we ever had for ReqEnd in Varnish 3: https://varnish-cache.org/trac/wiki/Varnishlog The statement about the sixth and last number has a bit of a grammatical error, but what it's saying is that it's the duration of delivery -- the time needed to send the client response. >>> I am not getting difference between varnish 3 and varnish 6. That's where you're losing me, and I suspect the problem is that you're looking for something Varnish 6 that corresponds exactly to what you were reading in Varnish 3. Generally speaking, they don't correspond. At the risk of repeating the answer that you didn't like, timestamps in Varnish 6 give you three values: * an absolute time (Unix epoch) * time elapsed since the beginning of request processing * time elapsed since the last timestamp However displeasing it may be, that's the way it is. In *practical* terms, you probably *are* getting a corresponding value in the third field of Timestamp:Resp, because of the fact that it follows Timestamp:Process. Again repeating the reference to https://varnish-cache.org/docs/trunk/reference/vsl.html#timestamps: Process Processing finished, ready to deliver the client response. Resp Delivery of response to the client finished. Since Timestamp:Resp follows Timestamp:Process, and the third field is the difference since the last timestamp, then the third field of Timestamp:Resp is the difference between "ready to deliver the client response" and "Delivery of response to the client finished". In other words, the duration of delivery. So if you're insisting on asking "is that field in Varnish 6 the same as the field in Varnish 3", then fine, they're essentially the same thing. The reason I'm reluctant to just say yes is that it's possible (although very uncommon) for some *other* timestamp to appear before Timestamp:Resp in the log. If that does happen (a rare case, if at all), then they're not the same. Hardik, I believe you'll be much better off if you come to terms with the fact that the timestamps in Varnish 3 and Varnish 6 just aren't the same. You're using Varnish 6 now, so go with the way it works. Trying my best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From dridi at varni.sh Tue Mar 19 16:51:33 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 19 Mar 2019 17:51:33 +0100 Subject: reg: not getting ReqAcct tag in varnishlog In-Reply-To: References: Message-ID: Hi, Thank you for the details, more questions to come. On Tue, Mar 19, 2019 at 1:42 AM Hardik wrote: > > Hi Dridi, > > Can you give me a list of log records you need to collect? > > SLT_Timestamp : Do you need all timestamps or a specific metric? > SLT_ReqStart : > SLT_ReqMethod : > SLT_ReqURL: > SLT_ReqProtocol : > SLT_RespStatus : > SLT_ReqHeader : > SLT_RespHeader : > SLT_ReqAcct : > SLT_BereqAcct : Do you need the BereqAcct records for all transactions? Including cache hits? This one is tricky in terms of billing. > SLT_VCL_Log : > > And > possibly how you are trying to group them if they come from different > transactions? You can do the grouping with the -g option, but that didn't go well for you so that's what I'm trying to figure out. > I am reading based xid ( by FD ). Means reading full records per fd. What does FD mean here? File descriptor? From ReqStart? > Please let me know if any other information I can provide.. > > > If this is not related to my problem still I am curious to know how grouping is happening. You can point out some code or links with some details, I will go through. Well, utilities like varnishlog or varnishncsa accumulate transactions via libvarnishapi in memory (which may take a long time) and then libvarnishapi presents them in order. So utilities don't assume this logic and simply get the data presented to them in a callback function. That's where timeouts, overruns or transaction limits may result in data loss since slow log consumers don't slow down Varnish, and Varnish isn't slowed down by logs more than writing them to memory requires. Dridi From hetardik.p at gmail.com Wed Mar 20 06:14:03 2019 From: hetardik.p at gmail.com (Hardik) Date: Wed, 20 Mar 2019 11:44:03 +0530 Subject: Reg: exact field for response time In-Reply-To: References: <008d5d67-011f-a56a-ad38-e7d9a90aafa4@uplex.de> Message-ID: Thanks Geoff. It cleared my doubt. Means Both fields in 3 and 6 are same ( Agree with you that its not always). "At the risk of repeating the answer that you didn't like, timestamps in Varnish 6 give you three values" *This line made my day...* One question now .. what time should give to customer as a response time? Is it 2nd field(Total round trip time) or 3rd field(only response time after processed fetched content) ? Timestamp Resp: 1501601912.806787* 0.048125* *0.000037* Reason of asking this question is, Only first time request will go to origin. After that next all the request will be served from cache. Also generally we do not have control over other network (customer's network) when request goes to origin. As per this understanding, I should give 3rd field to customer as per response time. Please correct me if I am wrong. Thanks again Geoff. -Hardik On Tue, 19 Mar 2019 at 18:06, Geoff Simmons wrote: > On 3/19/19 11:00, Hardik wrote: > > > > Please help with understanding. > > Well I tried with my previous answer, but you didn't seem to be > satisfied with that. I'm not sure what else I can say. > > >> This line is creating confusion for me.. > >> > >> https://github.com/jib/libvmod-timers/blob/master/src/vmod_timers.c > >> The sixth and last number (0.000025988) is the time from we start > >> delivering the object until the request completes. > > That appears to be exactly the same as what we had in the old wiki, > which as I recall was the only documentation we ever had for ReqEnd in > Varnish 3: > > https://varnish-cache.org/trac/wiki/Varnishlog > > The statement about the sixth and last number has a bit of a grammatical > error, but what it's saying is that it's the duration of delivery -- the > time needed to send the client response. > > >>> I am not getting difference between varnish 3 and varnish 6. > > That's where you're losing me, and I suspect the problem is that you're > looking for something Varnish 6 that corresponds exactly to what you > were reading in Varnish 3. > > Generally speaking, they don't correspond. At the risk of repeating the > answer that you didn't like, timestamps in Varnish 6 give you three values: > > * an absolute time (Unix epoch) > * time elapsed since the beginning of request processing > * time elapsed since the last timestamp > > However displeasing it may be, that's the way it is. > > In *practical* terms, you probably *are* getting a corresponding value > in the third field of Timestamp:Resp, because of the fact that it > follows Timestamp:Process. > > Again repeating the reference to > https://varnish-cache.org/docs/trunk/reference/vsl.html#timestamps: > > Process > Processing finished, ready to deliver the client response. > Resp > Delivery of response to the client finished. > > Since Timestamp:Resp follows Timestamp:Process, and the third field is > the difference since the last timestamp, then the third field of > Timestamp:Resp is the difference between "ready to deliver the client > response" and "Delivery of response to the client finished". In other > words, the duration of delivery. > > So if you're insisting on asking "is that field in Varnish 6 the same as > the field in Varnish 3", then fine, they're essentially the same thing. > > The reason I'm reluctant to just say yes is that it's possible (although > very uncommon) for some *other* timestamp to appear before > Timestamp:Resp in the log. If that does happen (a rare case, if at all), > then they're not the same. > > Hardik, I believe you'll be much better off if you come to terms with > the fact that the timestamps in Varnish 3 and Varnish 6 just aren't the > same. You're using Varnish 6 now, so go with the way it works. > > > Trying my best, > Geoff > -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hetardik.p at gmail.com Wed Mar 20 09:37:22 2019 From: hetardik.p at gmail.com (Hardik) Date: Wed, 20 Mar 2019 15:07:22 +0530 Subject: reg: not getting ReqAcct tag in varnishlog In-Reply-To: References: Message-ID: Hi Dridi, Do you need all timestamps or a specific metric? Regarding timestamp, want to read two tags, Timestamp Start: *1516269224.184112* 0.000000 0.000000 Timestamp Resp: 1516269224.184920 *0.000808 0.000087* Do you need the BereqAcct records for all transactions? Including cache hits? Sorry it is my mistake. I am not reading any of the beck-end records. So can ignore BereqAcct. I need fields from Req records only. What does FD mean here? File descriptor? From ReqStart? Yes, Its file descriptor. And yes reading from ReqStart till ReqAcct. Using switch case to read needed records. ex: switch (tag) { case SLT_ReqURL: Using below condition as I do not want to read Sess and Bereq records.( Please correct me if doing something wrong) if (!VSL_Match(vsl, t->c) || (t->type == VSL_t_bereq) || (t->type == VSL_t_sess)) continue; Well, utilities like varnishlog or varnishncsa accumulate transactions via libvarnishapi in memory (which may take a long time) and then libvarnishapi presents them in order. So utilities don't assume this logic and simply get the data presented to them in a callback function. Thanks for this.. It means varnishlog (any utility) asks for logs which will come via libvarnishapi. Just trying to confirm that what i understood from above is correct. Here, memory means varnishd process memory or utility's memory ? As per my understadning its varnishd's memory because libvarnishapi will be holding logs to arrange. In case of varnish 3, we were able to call direct dispatch function so it was faster, if my understanding is correct. Are log loss and missing of ReqAcct tag both problems are related ? I am thinking missing of ReqAcct is due to timeout problem also. For example mobile request are taking more time to serve.. In this case we might hit timeout of reading logs from VSL right ? I do not know in detail. But if buy modifying any timeout value can help please point out, I will try that sure. Thank you hardik On Tue, 19 Mar 2019 at 22:22, Dridi Boukelmoune wrote: > Hi, > > Thank you for the details, more questions to come. > > On Tue, Mar 19, 2019 at 1:42 AM Hardik wrote: > > > > Hi Dridi, > > > > Can you give me a list of log records you need to collect? > > > > SLT_Timestamp : > > Do you need all timestamps or a specific metric? > > > SLT_ReqStart : > > SLT_ReqMethod : > > SLT_ReqURL: > > SLT_ReqProtocol : > > SLT_RespStatus : > > SLT_ReqHeader : > > SLT_RespHeader : > > SLT_ReqAcct : > > SLT_BereqAcct : > > Do you need the BereqAcct records for all transactions? Including cache > hits? > > This one is tricky in terms of billing. > > > SLT_VCL_Log : > > > > And > > possibly how you are trying to group them if they come from different > > transactions? > > You can do the grouping with the -g option, but that didn't go well > for you so that's what I'm trying to figure out. > > > I am reading based xid ( by FD ). Means reading full records per fd. > > What does FD mean here? File descriptor? From ReqStart? > > > Please let me know if any other information I can provide.. > > > > > > If this is not related to my problem still I am curious to know how > grouping is happening. You can point out some code or links with some > details, I will go through. > > Well, utilities like varnishlog or varnishncsa accumulate transactions > via libvarnishapi in memory (which may take a long time) and then > libvarnishapi presents them in order. So utilities don't assume this > logic and simply get the data presented to them in a callback > function. > > That's where timeouts, overruns or transaction limits may result in > data loss since slow log consumers don't slow down Varnish, and > Varnish isn't slowed down by logs more than writing them to memory > requires. > > > > Dridi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniele.piaggesi at bonsaimeme.com Wed Mar 20 10:30:06 2019 From: daniele.piaggesi at bonsaimeme.com (Daniele Piaggesi) Date: Wed, 20 Mar 2019 11:30:06 +0100 Subject: Hit idle send timeout when I try to stream mp3 Message-ID: <2F2FC97B-E61A-4D01-ADBD-85E34F44A69D@bonsaimeme.com> Dear all I have a problem with my Varnish installation when I try to stream mp3 files through Varnish Cache 4. This is the scenario. I have a Drupal website in which editors can upload mp3s. These mp3s are listed in a section of website and the end user can listen using an HTML 5 player. The stack is: Varnish 4 <-> Nginx 1.11<-> Drupal 7 on a GNU/Linux Debian Jessie. Varnish and Nginx are installed with Debian packages. My Varnish configuration is here: https://pastebin.com/8Kw1b2mL When I try to listen an mp3 directly through Nginx all works well: player loads the file and I can listen the mp3. If I try to do the same through Varnish, the player remains in loading and mp3 doesn?t start. I tried to download the mp3 file using curl and my request goes in timeout. I had a look at varnishlog and this is the output: * << Request >> 1409738 - Begin req 1246052 rxreq - Timestamp Start: 1552587068.541997 0.000000 0.000000 - Timestamp Req: 1552587068.541997 0.000000 0.000000 - ReqStart 93.147.150.135 15330 - ReqMethod GET - ReqURL /sites/default/files/audio/radio_interviews/20180927-rds-gr_rds_1700-170602593m_1.mp3 - ReqProtocol HTTP/1.1 - ReqHeader Host: www.xxx.it - ReqHeader Connection: keep-alive - ReqHeader Pragma: no-cache - ReqHeader Cache-Control: no-cache - ReqHeader Accept-Encoding: identity;q=1, *;q=0 - ReqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36 - ReqHeader chrome-proxy: frfr - ReqHeader Accept: */* - ReqHeader Referer: http://www.xxx.it/path/to/url - ReqHeader Accept-Language: en-US,en;q=0.9,es;q=0.8,fr;q=0.7,it;q=0.6 - ReqHeader Cookie: SESScookieagreed=2; has_js=1; _ga=GA1.2.950111086.1552579260; _gid=GA1.2.112403313.1552579260; __atuvc=2%7C11; _gat_UA-57096474-1=1 - ReqHeader Range: bytes=0- - ReqHeader X-Forwarded-For: 93.147.150.135 - VCL_call RECV - VCL_acl NO_MATCH allowed_monitors - ReqUnset X-Forwarded-For: 93.147.150.135 - ReqHeader X-Forwarded-For: 93.147.150.135, 93.147.150.135 - ReqURL /sites/default/files/audio/radio_interviews/20180927-rds-gr_rds_1700-170602593m_1.mp3 - ReqHeader x-range: bytes=0- - ReqUnset Cookie: SESScookieagreed=2; has_js=1; _ga=GA1.2.950111086.1552579260; _gid=GA1.2.112403313.1552579260; __atuvc=2%7C11; _gat_UA-57096474-1=1 - VCL_return hash - ReqUnset Accept-Encoding: identity;q=1, *;q=0 - VCL_call HASH - ReqUnset Range: bytes=0- - VCL_return lookup - Hit 229378 - VCL_call HIT - VCL_return fetch - VCL_Error change return(fetch) to return(miss) in vcl_hit{} - VCL_call MISS - VCL_return fetch - Link bereq 1409739 fetch - Timestamp Fetch: 1552587068.542520 0.000523 0.000523 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx/1.11.5 - RespHeader Date: Thu, 14 Mar 2019 18:11:08 GMT - RespHeader Last-Modified: Fri, 28 Sep 2018 08:26:56 GMT - RespHeader ETag: "5bade5d0-a0b94" - RespHeader Content-Type: audio/mpeg - RespHeader Content-Length: 658324 - RespHeader X-Cacheable: YES - RespHeader X-Varnish: 1409738 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - RespHeader X-Cache: MISS - RespHeader X-Cookie: - RespHeader grace: - RespHeader X-Varnish-Server: www.xxx.it - VCL_return deliver - Timestamp Process: 1552587068.542536 0.000539 0.000016 - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Debug "Hit idle send timeout, wrote = 247608/658701; retrying" - Debug "Write error, retval = -1, len = 411093, errno = Resource temporarily unavailable" - Timestamp Resp: 1552587188.539239 119.997242 119.996703 - ReqAcct 733 0 733 377 658324 658701 - End I?m not a Varnish ?guru?, but it seems that the error is: - Debug "Hit idle send timeout, wrote = 247608/658701; retrying" - Debug "Write error, retval = -1, len = 411093, errno = Resource temporarily unavailable" I search a lot on Google but I didn?t find anything about that, except for a timeout problem that doesn?t seem to me because timeout settings are set to 60s (first_byte), same as Nginx. Any help is really appreciated. If you need some other infos, let me know. Thanks in advance Daniele Daniele Piaggesi Mobile: +39 393 880 78 50 Skype: g0blin79 E-mail: daniele.piaggesi at bmeme.com ------------------------------------------ Bonsaimeme S.r.l. Via del Porto Fluviale, 9 00154 Roma - Italy Phone: +39 06 98 26 04 39 Fax: +39 06 94 81 02 03 ------------------------------------------- bmeme.com *** Prima di stampare, pensa all'ambiente! *** *** Before printing think about environment and costs *** Le informazioni, i dati e le notizie contenute nella presente comunicazione e i relativi allegati sono di natura privata e come tali possono essere riservate e sono, comunque, destinate esclusivamente ai destinatari indicati in epigrafe. La diffusione, distribuzione e/o la copiatura del documento trasmesso da parte di qualsiasi soggetto diverso dal destinatario ? proibita, sia ai sensi dell?art. 616 c.p., sia ai sensi del D.Lgs. n. 196/2003. Se avete ricevuto questo messaggio per errore, vi preghiamo di distruggerlo e di darcene immediata comunicazione anche inviando un messaggio all?indirizzo email: info at bonsaimeme.com . Il testo della email potrebbe contenere opinioni personali e non necessariamente riconducibili a quelle di Bonsaimeme S.r.l. --- --- --- This e-mail (including attachments) is intended only for the recipient(s) named above. It may contain confidential or privileged information and should not be read, copied or otherwise used by any other person. If you are not the named recipient, please contact: info at bonsaimeme.com and delete the e-mail from your system. Rif. D.L. 196/2003. -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Wed Mar 20 11:45:53 2019 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 20 Mar 2019 12:45:53 +0100 Subject: Reg: exact field for response time In-Reply-To: References: <008d5d67-011f-a56a-ad38-e7d9a90aafa4@uplex.de> Message-ID: <263b86ff-19c3-fa8e-55df-f63ea242ed1f@uplex.de> On 3/20/19 07:14, Hardik wrote: > > what time should give to customer as a response time? Is it 2nd field(Total > round trip time) or 3rd field(only response time after processed fetched > content) ? > Timestamp Resp: 1501601912.806787* 0.048125* *0.000037* > > Reason of asking this question is, Only first time request will go to > origin. After that next all the request will be served from cache. Also > generally we do not have control over other network (customer's network) > when request goes to origin. As per this understanding, I should give 3rd > field to customer as per response time. Please correct me if I am wrong. The third field of Resp is not a useful measurement. It's the time taken for Varnish userland code to complete network send operations -- when the syscalls say they're done, there's essentially nothing left for Varnish to do with the response, so Varnish writes the final timestamp. But return from the syscalls for network send may mean nothing more than: the data has been placed on queues in the TCP stack. It doesn't tell you anything about the network send, or even if the data has been sent on the network yet at all. As a practical matter, if you tell your customer that "response time" was 37 microseconds, they probably won't believe you. (I wouldn't.) From what you've said, it sounds like you're looking for something like: the time taken to process the request, but not including the time for a fetch from the origin server. Is that about right? For that, you'd need to do more than read one field from one Timestamp entry -- you'll need to read at least two, maybe more, and then do some arithmetic. The best measure for the total time of request processing is the 2nd field in Timestamp:Resp, 48ms in your example above. The best measure for the fetch time is in the backend log, the 3rd field of Timestamp:Beresp, maybe added to the 3rd field of Timestamp:BerespBody. So you'd have to find the backend log corresponding to client log. -g request can help you with that. Then (if this is what you're after), subtract the fetch time from the total request processing time. There are some other ways to go about it, but it depends on what exactly you want to measure as "response time". And since I may have misunderstood what you're trying to measure, I'll stop there. HTH, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From dridi at varni.sh Wed Mar 20 15:32:11 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 20 Mar 2019 16:32:11 +0100 Subject: reg: not getting ReqAcct tag in varnishlog In-Reply-To: References: Message-ID: On Wed, Mar 20, 2019 at 10:40 AM Hardik wrote: > > Hi Dridi, > > Do you need all timestamps or a specific metric? > Regarding timestamp, want to read two tags, > Timestamp Start: 1516269224.184112 0.000000 0.000000 > Timestamp Resp: 1516269224.184920 0.000808 0.000087 > > Do you need the BereqAcct records for all transactions? Including cache hits? > Sorry it is my mistake. I am not reading any of the beck-end records. So can ignore BereqAcct. > I need fields from Req records only. Ok, in this case you can probably get away with just varnishncsa to collect all you need. No grouping (the default -g vxid), client mode (-c) only, with a custom -F format to grab only what you need. This should help reduce the churn to the point where you lose data. If you struggle with this, I can help you later with that, but start by reading the following manuals: - varnishncsa - vsl - vsl-query For example, the format for the timestamps you wish to collect would look like this: > %{VSL:Timestamp:Start[1]}x %{VSL:Timestamp:Resp[2]}x %{VSL:Timestamp:Resp[3]}x Rinse and repeat for all the things you need to capture for the logs, put them in the order you prefer and off you go. No need to write your own utility. > What does FD mean here? File descriptor? From ReqStart? > Yes, Its file descriptor. And yes reading from ReqStart till ReqAcct. Using switch case to read needed records. If you already work with VXIDs, the FD becomes redundant. Dridi From cristian.c at istream.today Wed Mar 27 13:29:52 2019 From: cristian.c at istream.today (cristian.c at istream.today) Date: Wed, 27 Mar 2019 15:29:52 +0200 Subject: Varnish removing tags incorrectly fromURL Message-ID: <009d01d4e4a1$2919eee0$7b4dcca0$@istream.today> Hello, I would highly appreciate if I get some help on the following issu: The query string from the end (== erstellen4) is being incorrectly appended to the token because the Varnish is not removing the tags correctly. https://xxxxxx.my/aa/?ResetPasswordToken=4P/weCg49hetX25dVAJxGW0i2GcwuN3bB3z xbMiYLo+3Kfpk199F9ZjwvSP3g8mrPq/opmCosoDmkTHYx3CYK+ABEFrF92y+R0V9icpnLep+f+z fPJjVOZ+M6wa1egt+GNktWIdBIruXXREYAboEQyBtHmgGJQe25KoCUvfUe1ySZlcFre5Dk913ktB D/wvwrtt/O6T2e9aUn2aiKkKdtA==&utm_source=acc_activation&utm_medium=email&utm _campaign=FW_new_customer_activation_2-2019032713&utm_content=Zugangsdaten+e rstellen4 This is what I get by looking at the logs: After reset password token: 4P/weCg49hetX25dVAJxGW0i2GcwuN3bB3zxbMiYLo3Kfpk199F9ZjwvSP3g8mrPq/opmCosoDmk THYx3CYK ABEFrF92yR0V9icpnLepfzfPJjVOZM6wa1egtGNktWIdBIruXXREYAboEQyBtHmgGJQe25KoCUvf Ue1ySZlcFre5Dk913ktBD/wvwrtt/O6T2e9aUn2aiKkKdtA== erstellen4 This is my varnish config : # # Marker to tell the VCL compiler that this VCL has been adapted to the # new 4.0 format. vcl 4.0; import directors; import std; acl monitoring { "localhost"; "192.xxx.xxx.xxx"/32; /* Collector */ "83.xxx.xxx.xxx"/32; /* LB */ } acl purge { "xxx.xxx.xxx.xxx"/32; /* */ "xxx.xxx.xxx.xxx"/32; /* */ } include "/etc/varnish/backend.vcl"; sub vcl_init { include "/etc/varnish/director.vcl"; } sub vcl_recv { # Happens before we check if we have this in cache already. # # Typically you clean up the request here, removing cookies you don't need, #A #rewriting the request, etc. #set req.backend_hint = vweb.backend(req.http.X-Forwarded-For); #set req.backend_hint = fbdirector.backend(); # # Set hash directory with hashing option X-Forwarded for becuase we use nginx between the client and vanrish #set req.backend_hint = hashdirector.backend(req.http.X-Forwarded-For); # # Monitoring for FortiADC if faild, the proxy is taken out, if all fails then hit maintance page. if (req.method == "GET" && req.url == "/varnish-status") { if (client.ip ~ monitoring) { #if (std.healthy(hashdirector.backend(req.http.X-Forwarded-For))) { return(synth(200, "OK")); #} else { # return(synth(503, "No backends available")); #} } else { return(synth(403, "Access denied.")); } } include "/etc/varnish/vhost.vcl"; # Remove the proxy header (see https://httpoxy.org/#mitigate-varnish) unset req.http.proxy; # Allow purging if (req.method == "PURGE") { if (!client.ip ~ purge) { # purge is the ACL defined at the begining # Not from an allowed IP? Then die with an error. return (synth(405, "IP: " + client.ip + " is not allowed to send PURGE requests.")); } # If you got this stage (and didn't error out above), purge the cached result return (purge); } # Only allow BAN requests from IP addresses in the 'purge' ACL. if (req.method == "BAN") { # Same ACL check as above: if (!client.ip ~ purge) { return (synth(405, "IP: " + client.ip + " is not allowed to send BAN requests.")); } # manual sudo varnishadm "ban req.http.host ~ www.mydomain.com" ban("req.http.host ~ " + req.http.host); # Throw a synthetic page so the request won't go to the backend. return (synth(200, "BAN for " + req.http.host + " done")); } # Only cache GET or HEAD requests. This makes sure the POST requests are always passed. if (req.method != "GET" && req.method != "HEAD") { return (pass); } # Some generic URL manipulation, useful for all templates that follow # First remove the Google Analytics added parameters, useless for our backend if (req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteu rl)=") { set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=( [A-z0-9_\-\.%25]+)", ""); set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)= ([A-z0-9_\-\.%25]+)", "?"); set req.url = regsub(req.url, "\?&", "?"); set req.url = regsub(req.url, "\?$", ""); } # Strip hash, server doesn't need it. if (req.url ~ "\#") { set req.url = regsub(req.url, "\#.*$", ""); } # Strip a trailing ? if it exists if (req.url ~ "\?$") { set req.url = regsub(req.url, "\?$", ""); } # Some generic cookie manipulation, useful for all templates that follow # Remove the "has_js" cookie set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", ""); # Remove any Google Analytics based cookies set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "_gat=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "utmctr=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "utmcmd.=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "utmccn.=[^;]+(; )?", ""); # Remove DoubleClick offensive cookies set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", ""); # Remove the Quant Capital cookies (added by some plugin, all __qca) set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", ""); # Remove the AddThis cookies set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", ""); # Remove a ";" prefix in the cookie if present set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", ""); # Are there cookies left with only spaces or that are empty? if (req.http.cookie ~ "^\s*$") { unset req.http.cookie; } # Large static files are delivered directly to the end-user without # waiting for Varnish to fully read the file first. # Varnish 4 fully supports Streaming, so set do_stream in vcl_backend_response() if (req.url ~ "^[^?]*\.(7z|avi|bz2|flac|flv|gz|mka|mkv|mov|mp3|mp4|mpeg|mpg|ogg|ogm|opus|r ar|tar|tgz|tbz|txz|wav|webm|xz|zip)(\?.*)?$") { unset req.http.Cookie; return (hash); } #Remove all cookies for static files if (req.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|j s|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf|ogg|ogm|opus|pdf|png|ppt|pptx|ra r|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx |xml|xz|zip)(\?.*)?$") { unset req.http.Cookie; return (hash); } # Send Surrogate-Capability headers to announce ESI support to backend set req.http.Surrogate-Capability = "key=ESI/1.0"; if (req.http.Authorization) { # Not cacheable by default return (pass); } if (req.url == "/checksite.aspx") { # Dont cache monitoring url return (pass); } return (hash); } sub vcl_backend_response { # Happens after we have read the response headers from the backend. # # Here you clean the response headers, removing silly Set-Cookie headers # and other mistakes your backend does. # # set beresp.http.X-Backend = beresp.backend.name; # Remove some headers: ASP version unset beresp.http.X-Powered-By; # Remove cookie with empty basketid useless... VL should fix on backend if (beresp.http.set-cookie == "BasketID=; path=/") { unset beresp.http.set-cookie; } # Pause ESI request and remove Surrogate-Control header if (beresp.http.Surrogate-Control ~ "ESI/1.0") { unset beresp.http.Surrogate-Control; set beresp.do_esi = true; } # Enable cache for all static files # The same argument as the static caches from above: monitor your cache size, if you get data nuked out of it, consider giving up the static file cache. # Before you blindly enable this, have a read here: https://ma.ttias.be/stop-caching-static-files/ if (bereq.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|j s|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf|ogg|ogm|opus|pdf|png|ppt|pptx|ra r|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx |xml|xz|zip)(\?.*)?$") { unset beresp.http.set-cookie; } # Large static files are delivered directly to the end-user without # waiting for Varnish to fully read the file first. # Varnish 4 fully supports Streaming, so use streaming here to avoid locking. if (bereq.url ~ "^[^?]*\.(7z|avi|bz2|flac|flv|gz|mka|mkv|mov|mp3|mp4|mpeg|mpg|ogg|ogm|opus|r ar|tar|tgz|tbz|txz|wav|webm|xz|zip)(\?.*)?$") { unset beresp.http.set-cookie; set beresp.do_stream = true; # Check memory usage it'll grow in fetch_chunksize blocks (128k by default) if the backend doesn't send a Content-Length header, so only enable it for big objects } # Don't cache 50x responses if (beresp.status == 500 || beresp.status == 502 || beresp.status == 503 || beresp.status == 504) { return (abandon); } if (bereq.http.Cookie ~ "(UserID|_session)") { #set beresp.http.X-Cacheable = "NO:Got Session"; set beresp.uncacheable = true; return (deliver); } elsif (beresp.ttl <= 0s) { # Varnish determined the object was not cacheable #set beresp.http.X-Cacheable = "NO:Not Cacheable"; } elsif (beresp.http.set-cookie) { # You don't wish to cache content for logged in users #set beresp.http.X-Cacheable = "NO:Set-Cookie"; set beresp.uncacheable = true; return (deliver); } elsif (beresp.http.Cache-Control ~ "private") { # You are respecting the Cache-Control=private header from the backend #set beresp.http.X-Cacheable = "NO:Cache-Control=private"; set beresp.uncacheable = true; return (deliver); } else { # Varnish determined the object was cacheable #set beresp.http.X-Cacheable = "YES"; } return(deliver); } sub vcl_deliver { # Happens when we have all the pieces we need, and are about to send the # response to the client. # # You can do accounting or modifying the final object here. if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and the number of hits, disable when not needed set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } # Unset some headers unset resp.http.Via; unset resp.http.X-Varnish; # Please note that obj.hits behaviour changed in 4.0, now it counts per objecthead, not per object # and obj.hits may not be reset in some cases where bans are in use. See bug 1492 for details. # So take hits with a grain of salt set resp.http.X-Cache-Hits = obj.hits; } Regards, Cris -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Mar 27 15:39:48 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 27 Mar 2019 16:39:48 +0100 Subject: Varnish removing tags incorrectly fromURL In-Reply-To: <009d01d4e4a1$2919eee0$7b4dcca0$@istream.today> References: <009d01d4e4a1$2919eee0$7b4dcca0$@istream.today> Message-ID: On Wed, Mar 27, 2019 at 2:33 PM wrote: > > Hello, > > > > I would highly appreciate if I get some help on the following issu: If you need to filter out or extract parameters from a query-string I recommend this: https://github.com/Dridi/libvmod-querystring/#vmod-querystring If you are running Varnish below 6.0 I encourage you to upgrade but meanwhile you also have this: https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring It should be much easier than using regular expressions, at the expense of having to manage a vmod. Dridi From cristian.c at istream.today Thu Mar 28 13:00:26 2019 From: cristian.c at istream.today (cristian.c at istream.today) Date: Thu, 28 Mar 2019 15:00:26 +0200 Subject: Varnish removing tags incorrectly fromURL In-Reply-To: References: <009d01d4e4a1$2919eee0$7b4dcca0$@istream.today> Message-ID: <00e001d4e566$36f70410$a4e50c30$@istream.today> Hello, Thank you for answering, I'm running on version 5.2.1, the problem is that I can't install this version https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring because there is no "configure" file inside (./configure not working) if I download the last version of vmod I get the following error configure error: varnish version 6.0.0. or higher is required ... Thank you! Regards, Cristian -----Original Message----- From: Dridi Boukelmoune Sent: Wednesday, March 27, 2019 5:40 PM To: cristian.c at istream.today Cc: varnish-misc Subject: Re: Varnish removing tags incorrectly fromURL On Wed, Mar 27, 2019 at 2:33 PM wrote: > > Hello, > > > > I would highly appreciate if I get some help on the following issu: If you need to filter out or extract parameters from a query-string I recommend this: https://github.com/Dridi/libvmod-querystring/#vmod-querystring If you are running Varnish below 6.0 I encourage you to upgrade but meanwhile you also have this: https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring It should be much easier than using regular expressions, at the expense of having to manage a vmod. Dridi From cristian.c at istream.today Thu Mar 28 14:41:04 2019 From: cristian.c at istream.today (cristian.c at istream.today) Date: Thu, 28 Mar 2019 16:41:04 +0200 Subject: Varnish removing tags incorrectly fromURL In-Reply-To: <00e001d4e566$36f70410$a4e50c30$@istream.today> References: <009d01d4e4a1$2919eee0$7b4dcca0$@istream.today> <00e001d4e566$36f70410$a4e50c30$@istream.today> Message-ID: <00e301d4e574$45ceba10$d16c2e30$@istream.today> Update: Ignore this... I have found. thx -----Original Message----- From: cristian.c at istream.today Sent: Thursday, March 28, 2019 3:00 PM To: 'Dridi Boukelmoune' Cc: 'varnish-misc' Subject: RE: Varnish removing tags incorrectly fromURL Hello, Thank you for answering, I'm running on version 5.2.1, the problem is that I can't install this version https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring because there is no "configure" file inside (./configure not working) if I download the last version of vmod I get the following error configure error: varnish version 6.0.0. or higher is required ... Thank you! Regards, Cristian -----Original Message----- From: Dridi Boukelmoune Sent: Wednesday, March 27, 2019 5:40 PM To: cristian.c at istream.today Cc: varnish-misc Subject: Re: Varnish removing tags incorrectly fromURL On Wed, Mar 27, 2019 at 2:33 PM wrote: > > Hello, > > > > I would highly appreciate if I get some help on the following issu: If you need to filter out or extract parameters from a query-string I recommend this: https://github.com/Dridi/libvmod-querystring/#vmod-querystring If you are running Varnish below 6.0 I encourage you to upgrade but meanwhile you also have this: https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring It should be much easier than using regular expressions, at the expense of having to manage a vmod. Dridi From guillaume at varnish-software.com Thu Mar 28 15:42:35 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 28 Mar 2019 08:42:35 -0700 Subject: Varnish removing tags incorrectly fromURL In-Reply-To: <00e301d4e574$45ceba10$d16c2e30$@istream.today> References: <009d01d4e4a1$2919eee0$7b4dcca0$@istream.today> <00e001d4e566$36f70410$a4e50c30$@istream.today> <00e301d4e574$45ceba10$d16c2e30$@istream.today> Message-ID: Hello Cristian, Would you care sharing your findings in order to help people having the same issue in the future? Cheers, On Thu, Mar 28, 2019, 07:42 wrote: > Update: Ignore this... I have found. thx > > -----Original Message----- > From: cristian.c at istream.today > Sent: Thursday, March 28, 2019 3:00 PM > To: 'Dridi Boukelmoune' > Cc: 'varnish-misc' > Subject: RE: Varnish removing tags incorrectly fromURL > > Hello, > > Thank you for answering, I'm running on version 5.2.1, the problem is > that I can't install this version > https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring > because there is no "configure" file inside (./configure not working) if > I download the last version of vmod I get the following error configure > error: varnish version 6.0.0. or higher is required ... > > Thank you! > > > Regards, > Cristian > > -----Original Message----- > From: Dridi Boukelmoune > Sent: Wednesday, March 27, 2019 5:40 PM > To: cristian.c at istream.today > Cc: varnish-misc > Subject: Re: Varnish removing tags incorrectly fromURL > > On Wed, Mar 27, 2019 at 2:33 PM wrote: > > > > Hello, > > > > > > > > I would highly appreciate if I get some help on the following issu: > > If you need to filter out or extract parameters from a query-string I > recommend this: > > https://github.com/Dridi/libvmod-querystring/#vmod-querystring > > If you are running Varnish below 6.0 I encourage you to upgrade but > meanwhile you also have this: > > https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring > > It should be much easier than using regular expressions, at the expense of > having to manage a vmod. > > > Dridi > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cristian.c at istream.today Fri Mar 29 07:16:05 2019 From: cristian.c at istream.today (cristian.c at istream.today) Date: Fri, 29 Mar 2019 09:16:05 +0200 Subject: Varnish removing tags incorrectly fromURL In-Reply-To: References: <009d01d4e4a1$2919eee0$7b4dcca0$@istream.today> <00e001d4e566$36f70410$a4e50c30$@istream.today> <00e301d4e574$45ceba10$d16c2e30$@istream.today> Message-ID: <011b01d4e5ff$46f5ebd0$d4e1c370$@istream.today> Hi, It was just my mistake, you must download the tar.gz version from releases tab : https://github.com/Dridi/libvmod-querystring/releases Regards, Cristian From: Guillaume Quintard Sent: Thursday, March 28, 2019 5:43 PM To: cristian.c at istream.today Cc: varnish-misc Subject: Re: Varnish removing tags incorrectly fromURL Hello Cristian, Would you care sharing your findings in order to help people having the same issue in the future? Cheers, On Thu, Mar 28, 2019, 07:42 > wrote: Update: Ignore this... I have found. thx -----Original Message----- From: cristian.c at istream.today > Sent: Thursday, March 28, 2019 3:00 PM To: 'Dridi Boukelmoune' > Cc: 'varnish-misc' > Subject: RE: Varnish removing tags incorrectly fromURL Hello, Thank you for answering, I'm running on version 5.2.1, the problem is that I can't install this version https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring because there is no "configure" file inside (./configure not working) if I download the last version of vmod I get the following error configure error: varnish version 6.0.0. or higher is required ... Thank you! Regards, Cristian -----Original Message----- From: Dridi Boukelmoune > Sent: Wednesday, March 27, 2019 5:40 PM To: cristian.c at istream.today Cc: varnish-misc > Subject: Re: Varnish removing tags incorrectly fromURL On Wed, Mar 27, 2019 at 2:33 PM > wrote: > > Hello, > > > > I would highly appreciate if I get some help on the following issu: If you need to filter out or extract parameters from a query-string I recommend this: https://github.com/Dridi/libvmod-querystring/#vmod-querystring If you are running Varnish below 6.0 I encourage you to upgrade but meanwhile you also have this: https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring It should be much easier than using regular expressions, at the expense of having to manage a vmod. Dridi _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Fri Mar 29 09:02:52 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 29 Mar 2019 10:02:52 +0100 Subject: Varnish removing tags incorrectly fromURL In-Reply-To: <011b01d4e5ff$46f5ebd0$d4e1c370$@istream.today> References: <009d01d4e4a1$2919eee0$7b4dcca0$@istream.today> <00e001d4e566$36f70410$a4e50c30$@istream.today> <00e301d4e574$45ceba10$d16c2e30$@istream.today> <011b01d4e5ff$46f5ebd0$d4e1c370$@istream.today> Message-ID: On Fri, Mar 29, 2019 at 8:18 AM wrote: > > Hi, > > It was just my mistake, you must download the tar.gz version from releases tab : https://github.com/Dridi/libvmod-querystring/releases I would have been surprised if a 1.x release didn't work for you :) Thanks for confirming and don't forget to move to 6.0 LTS to get bug fixes regularly. Dridi