Re: increase sess_workspace to fix assert error?

Tue Apr 16 15:54:11 CEST 2013

Hi Rubén,

Apparently we made a mistake a while ago by not confirming the varnish 3.0 defaults compared to 2.1. So we were happily still running with a sess_workspace of 16k, instead of 64k.
This worked fine with our previous config, but since we added our sharding vmod, 16k is not enough anymore.

So, the fixed config with 64k as sess_workspace has been running on one of our varnish servers for almost a week now.
And it's been working fine. Even when a Justin Bieber link went viral yesterday, which put our systems to the real test.
Not to mention this happened at the same time it's already peak-hour.

And Varnish (of course;) happily served thousands of requests per second.

So yes, I can now confirm: increasing sess_workspace fixed the assertion errors in my case.

regards,

Enno van Amerongen

  _____  

From: Rubén Romero [mailto:ruben at varnish-software.com]
To: Enno van Amerongen [mailto:enno at tty.nl]
Sent: Sun, 14 Apr 2013 12:52:48 +0200
Subject: Re: increase sess_workspace to fix assert error?

Hi Enno,  

Increasing it will likely help. Not sure to how much, but do give it as much as you can and let us know how it went :-)

Best,
  -
  Rubén Romero
  Varnish Software

Den 2. apr. 2013 21.18 skrev "Enno van Amerongen" <enno at tty.nl> følgende:

Dear List,

After deployment of a vmod we've written (that enables us to select a director by id in url), we've been experiencing assert errors.  

Below are the stack traces, but to not waste your time, here's what I think it boils down to:

{ overflow#012    id = "sess",#012    {s,f,r,e} = {0x7fed76034c78,+16384,(nil),+16384},#012  }  

4k as session workspace has worked fine for us for years, but our vmod sharding adds hundreds of directors (as opposed to 4), while the number of backends stayed the same (about 25).  

So, my question is, do I just need to increase sess_workspace to fix this? If not, what else ;)

Regards,

Enno van Amerongen  

Both stacktraces  

Apr  2 18:43:32 obfuscated /var/www/varnish[12750]: Child (20498) Panic message: Assert error in vh_get_var_alloc(), vmod_var.c line 77:#012  Condition((v) != 0) not true.#012errno = 32 (Broken pipe)#012thread = (cache-worker)#012ident = Linux,2.6.32-5-amd64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012  0x439199: pan_backtrace+19#012  0x43946e: pan_ic+1ad#012  0x7fe137efe347: _end+7fe13786898f#012  0x7fe137efe862: _end+7fe137868eaa#012  0x7fecc5b93e73: _end+7fecc54fe4bb#012  0x7fecc5b939ea: _end+7fecc54fe032#012  0x7fecc5ba24f1: _end+7fecc550cb39#012  0x4417bd: VCL_recv_method+6d#012  0x41d8f5: cnt_recv+262#012  0x41e68d: CNT_Session+5cb#012sp = 0x7fed81293008 {#012  fd = 2849, id = 2849, xid = 918440385,#012  client = 10.1.1.1 51854,#012  step = STP_RECV,#012  handling = deliver,#012  err_code = 200, err_reason = (null),#012  restarts = 0, esi_level = 0#012  flags = #012  bodystatus = 4#012  ws = 0x7fed81293080 { overflow#012    id = "sess",#012    {s,f,r,e} = {0x7fed81293c78,+16384,(nil),+16384},#012  },#012  http[req] = {#012    ws = 0x7fed81293080[sess]#012      "GET",#012      "/f/146057654_2.jpg",#012      "HTTP/1.1",#012      "User-Agent: Opera/9.80 (Windows NT 6.1; WOW64) Presto/2.12.388 Version/12.14",#012      "Host: img.obfuscated.com",#012      "Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1",#012      "Accept-Language: fr-FR,fr;q=0.9,en;q=0.8",#012      "Referer: http://obfuscated",#012      "Connection: Keep-Alive",#012  },#012  worker = 0x7feece7efaa0 {#012    ws = 0x7feece7efcd8 { #012      id = "wrk",#012      {s,f,r,e} = {0x7feece7dd9f0,+16,(nil),+65536},#012    },#012    },#012    vcl = {#012      srcname = {#012        "input",#012        "Default",#012        "/obfuscated/img_backends.vcl",#012        "/obfuscated/img_directors.vcl",#012  

So we removed the vmod_var from our configuration as it wasn't really required anyway.

Then 13 minutes later, Varnish crashed again, now with a different, but similar assert error:

Apr  2 19:07:22 obfuscated /var/www/varnish[12750]: Child (18818) Panic message: Assert error in VRT_int_string(), cache_vrt.c line 324:#012  Condition((p = WS_Alloc(sp->http->ws, size)) != 0) not true.#012thread = (cache-worker)#012ident = Linux,2.6.32-5-amd64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012  0x439199: pan_backtrace+19#012  0x43946e: pan_ic+1ad#012  0x4435f7: VRT_int_string+88#012  0x7fe686b9ce1f: _end+7fe686507467#012  0x7fe686b9c99a: _end+7fe686506fe2#012  0x7fe686ba7424: _end+7fe686511a6c#012  0x4417bd: VCL_recv_method+6d#012  0x41d8f5: cnt_recv+262#012  0x41e68d: CNT_Session+5cb#012  0x43aa0a: wrk_thread_real+8cc#012sp = 0x7fed76034008 {#012  fd = 2357, id = 2357, xid = 2009537706,#012  client = 80.239.243.196 59852,#012  step = STP_RECV,#012  handling = deliver,#012  err_code = 200, err_reason = (null),#012  restarts = 0, esi_level = 0#012  flags = #012  bodystatus = 4#012  ws = 0x7fed76034080 { overflow#012    id = "sess",#012    {s,f,r,e} = {0x7fed76034c78,+16384,(nil),+16384},#012  },#012  http[req] = {#012    ws = 0x7fed76034080[sess]#012      "GET",#012      "/f/148682988.jpg",#012      "HTTP/1.1",#012      "User-Agent: Opera/9.80 (Android; Opera Mini/7.5.32193/29.3134; U; nl) Presto/2.8.119 Version/11.10",#012      "Host: img.obfuscated.com",#012      "Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1",#012      "Accept-Language: nl,en;q=0.9",#012      "Referer: obfuscated",#012      "Connection: Keep-Alive",#012      "X-OperaMini-Phone: Android #",#012      "X-OperaMini-Phone-UA: Mozilla/5.0 (Linux; U; Android 2.2.2; nl-be; Stockholm Build/HuaweiU8180) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",#012      "x-forwarded-for: obfuscated",#012      "Device-Stock-UA: Mozilla/5.0 (Linux; U; Android 2.2.2; nl-be; Stockholm Build/HuaweiU8180) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",#012      "X  

_______________________________________________
  varnish-misc mailing list
  varnish-misc at varnish-cache.org
  https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20130416/8241654f/attachment.html>