reg: not getting ReqAcct tag in varnishlog

[Hardik]

Hi Dridi,

Do you need all timestamps or a specific metric?
Regarding timestamp, want to read two tags,
Timestamp      Start: *1516269224.184112* 0.000000 0.000000
Timestamp      Resp: 1516269224.184920 *0.000808 0.000087*

Do you need the BereqAcct records for all transactions? Including cache
hits?
Sorry it is my mistake. I am not reading any of the beck-end records. So
can ignore BereqAcct.
I need fields from Req records only.

What does FD mean here? File descriptor? From ReqStart?
Yes, Its file descriptor. And yes reading from ReqStart till ReqAcct. Using
switch case to read needed records.
ex:
switch (tag)
{
     case SLT_ReqURL:


Using below condition as I do not want to read Sess and Bereq records.(
Please correct me if doing something wrong)
             if (!VSL_Match(vsl, t->c) || (t->type == VSL_t_bereq)
                                      || (t->type == VSL_t_sess))
                        continue;

Well, utilities like varnishlog or varnishncsa accumulate transactions
via libvarnishapi in memory (which may take a long time) and then
libvarnishapi presents them in order. So utilities don't assume this
logic and simply get the data presented to them in a callback
function.
Thanks for this..
It means varnishlog (any utility) asks for logs which will come via
libvarnishapi. Just trying to confirm that what i understood from above is
correct. Here, memory means varnishd process memory or utility's memory ?
As per my understadning its varnishd's memory because libvarnishapi will be
holding logs to arrange.
 In case of varnish 3, we were able to call direct dispatch function so it
was faster, if my understanding is correct.

Are log loss and missing of ReqAcct tag both problems are related ?
I am thinking missing of ReqAcct is due to timeout problem also. For
example mobile request are taking more time to serve.. In this case we
might hit timeout of reading logs from VSL right ? I do not know in detail.
But if buy modifying any timeout value can help please point out, I will
try that sure.

Thank you
hardik


On Tue, 19 Mar 2019 at 22:22, Dridi Boukelmoune <dridi at varni.sh> wrote:

> Hi,
>
> Thank you for the details, more questions to come.
>
> On Tue, Mar 19, 2019 at 1:42 AM Hardik <hetardik.p at gmail.com> wrote:
> >
> > Hi Dridi,
> >
> > Can you give me a list of log records you need to collect?
> >
> > SLT_Timestamp :
>
> Do you need all timestamps or a specific metric?
>
> > SLT_ReqStart :
> > SLT_ReqMethod :
> > SLT_ReqURL:
> > SLT_ReqProtocol :
> > SLT_RespStatus :
> > SLT_ReqHeader :
> > SLT_RespHeader :
> > SLT_ReqAcct :
> > SLT_BereqAcct :
>
> Do you need the BereqAcct records for all transactions? Including cache
> hits?
>
> This one is tricky in terms of billing.
>
> > SLT_VCL_Log :
> >
> > And
> > possibly how you are trying to group them if they come from different
> > transactions?
>
> You can do the grouping with the -g option, but that didn't go well
> for you so that's what I'm trying to figure out.
>
> > I am reading based xid ( by FD ). Means reading full records per fd.
>
> What does FD mean here? File descriptor? From ReqStart?
>
> > Please let me know if any other information I can provide..
> >
> >
> > If this is not related to my problem still I am curious to know how
> grouping is happening. You can point out some code or links with some
> details, I will go through.
>
> Well, utilities like varnishlog or varnishncsa accumulate transactions
> via libvarnishapi in memory (which may take a long time) and then
> libvarnishapi presents them in order. So utilities don't assume this
> logic and simply get the data presented to them in a callback
> function.
>
> That's where timeouts, overruns or transaction limits may result in
> data loss since slow log consumers don't slow down Varnish, and
> Varnish isn't slowed down by logs more than writing them to memory
> requires.
>
>
>
> Dridi
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20190320/7358e06b/attachment-0001.html>