<div dir="ltr">Hi, <br><br>I have some questions regarding Varnish 4 and some problems we have with it's memory usage. <br><br>We are using Varnish 4 on a virtual server (CentOS 6.5) with 14 GB memory and 9 GB malloc. The TTL in use is 5m and the grace time is 60m. Varnish sits in front of an API that is responding with JSON only.<br>
<br>On the API servers we are using Nginx, and according to the body_bytes_sent variable in Nginx we can see that the average response size is 4.4 Kbyte and in 1 hour the API is sending around 300 Mbytes of data. Nginx is also compressing the response before sending it to Varnish. The number of requests to the API servers in 1 hour is between 60 000 and 70 000.<br>
<br>The problem we see is that when we reach around 73 000 objects in Varnish all the memory (9G) is used and the lru_nuked counter starts to increase. When Varnish is restarted most of the memory is used after 1 to 1.5 hour. As mentioned before the TTL is 5 minutes and the grace time is 60 minutes. I am really confused why we are seeing this behaviour, especially when the API servers is only sending around 300 Mbytes of data in 1 hour time. <br>
<br>I hope somebody can shed some light into this, thanks!<br><br><br>## Backend request ##<br><br>*   << BeReq    >> 808484921 <br>-   Begin          bereq 808484920 fetch <br>-   Timestamp      Start: 1403174559.768064 0.000000 0.000000<br>
-   BereqMethod    GET<br>-   BereqURL       request_url<br>-   BereqProtocol  HTTP/1.1<br>-   BereqHeader    Content-Type: application/hal+json<br>-   BereqHeader    Accept: application/hal+json<br>-   BereqHeader    Cneonction: close<br>
-   BereqHeader    Host: backend-servers:8080<br>-   BereqHeader    X-Forwarded-For: x.x.x.x<br>-   BereqHeader    Accept-Encoding: gzip<br>-   BereqHeader    X-Varnish: 808484921<br>-   VCL_call       BACKEND_FETCH<br>-   VCL_return     fetch<br>
-   Backend        46 backend backend3(x.x.x.x,,8080)<br>-   Timestamp      Bereq: 1403174559.768100 0.000036 0.000036<br>-   Timestamp      Beresp: 1403174560.085275 0.317211 0.317176<br>-   BerespProtocol HTTP/1.1<br>-   BerespStatus   200<br>
-   BerespResponse OK<br>-   BerespHeader   Server: nginx/1.6.0<br>-   BerespHeader   Date: Thu, 19 Jun 2014 10:42:46 GMT<br>-   BerespHeader   Content-Type: application/hal+json; charset=utf-8<br>-   BerespHeader   Transfer-Encoding: chunked<br>
-   BerespHeader   Connection: keep-alive<br>-   BerespHeader   Vary: Accept-Encoding<br>-   BerespHeader   Status: 200 OK<br>-   BerespHeader   X-Frame-Options: SAMEORIGIN<br>-   BerespHeader   X-XSS-Protection: 1; mode=block<br>
-   BerespHeader   X-Content-Type-Options: nosniff<br>-   BerespHeader   X-UA-Compatible: chrome=1<br>-   BerespHeader   Cache-Control: max-age=0, private, must-revalidate<br>-   BerespHeader   X-Runtime: 0.182126<br>-   BerespHeader   Content-Encoding: gzip<br>
-   TTL            RFC 0 -1 -1 1403174560 1403174560 1403174566 0 0<br>-   VCL_call       BACKEND_RESPONSE<br>-   TTL            VCL 300 10 0 1403174560<br>-   TTL            VCL 300 3600 0 1403174560<br>-   VCL_return     deliver<br>
-   Storage        malloc s0<br>-   ObjProtocol    HTTP/1.1<br>-   ObjStatus      200<br>-   ObjResponse    OK<br>-   ObjHeader      Server: nginx/1.6.0<br>-   ObjHeader      Date: Thu, 19 Jun 2014 10:42:46 GMT<br>-   ObjHeader      Content-Type: application/hal+json; charset=utf-8<br>
-   ObjHeader      Vary: Accept-Encoding<br>-   ObjHeader      Status: 200 OK<br>-   ObjHeader      X-Frame-Options: SAMEORIGIN<br>-   ObjHeader      X-XSS-Protection: 1; mode=block<br>-   ObjHeader      X-Content-Type-Options: nosniff<br>
-   ObjHeader      X-UA-Compatible: chrome=1<br>-   ObjHeader      Cache-Control: max-age=0, private, must-revalidate<br>-   ObjHeader      X-Runtime: 0.182126<br>-   ObjHeader      Content-Encoding: gzip<br>-   Fetch_Body     2 chunked stream<br>
-   Gzip           u F - 5876 21006 80 80 46942<br>-   Timestamp      BerespBody: 1403174560.085712 0.317648 0.000436<br>-   BackendReuse   46 backend3(x.x.x.x,,8080)<br>-   Length         5876<br>-   BereqAcct      471 0 471 657 5888 6545<br>
-   End<br><br><br>## default_vcl ##<br><br># This is a Varnish 4.x VCL file<br>vcl 4.0;<br>import directors;<br>import std;<br><br>backend backend1 {<br>   .host = "x.x.x.x";<br>   .port = "8080";<br>
   .probe = {<br>        .url = "/ping";<br>        .timeout   = 4s;<br>        .interval  = 5s;<br>        .window    = 5;<br>        .threshold = 4;<br>   }<br>}<br>backend backend2 {<br>   .host = "x.x.x.x";<br>
   .port = "8080";<br>   .probe = {<br>        .url = "/ping";<br>        .timeout   = 4s;<br>        .interval  = 5s;<br>        .window    = 5;<br>        .threshold = 4;<br>   }<br>}<br>backend backend3 {<br>
   .host = "x.x.x.x";<br>   .port = "8080";<br>   .probe = {<br>        .url = "/ping";<br>        .timeout   = 4s;<br>        .interval  = 5s;<br>        .window    = 5;<br>        .threshold = 4;<br>
   }<br>}<br><br># Below is an example redirector based on round-robin requests<br>sub vcl_init {<br>  new sservice = directors.round_robin();<br>  sservice.add_backend(backend1);<br>  sservice.add_backend(backend2);<br>  sservice.add_backend(backend3);<br>
}<br><br>acl local<br>{<br>    "x.x.x.x"/24;<br>}<br><br># Handle the HTTP request received by the client<br>sub vcl_recv {<br>  set req.backend_hint = sservice.backend();<br><br>    if (req.url ~ "^/ping$")<br>
    {<br>        if (client.ip ~ local)<br>            {<br>                return (synth(900, "pong"));<br>            }<br>  }<br><br>  if (! req.http.Authorization ~ "apitokens")<br>  {<br>    return (synth(401, "Restricted"));<br>
  }<br><br>  return (hash);<br>}<br><br>sub vcl_backend_response {<br>  set beresp.ttl = 5m;<br>  set beresp.grace = 60m;<br>  return (deliver);<br>}<br><br>sub vcl_deliver {<br>  if (obj.hits > 0) {<br>    set resp.http.X-Cache = "HIT";<br>
  } else {<br>    set resp.http.X-Cache = "MISS";<br>  }<br>  return (deliver);<br>}<br><br>sub vcl_synth {<br>  if (resp.status == 900)<br>  {<br>    set resp.status = 200;<br>    set resp.http.Content-Type = "text/plain; charset=utf-8";<br>
    synthetic("pong");<br>    return (deliver);<br>  }<br><br>  set resp.http.Content-Type = "application/hal+json";<br>  return (deliver);<br>}<br><br>sub vcl_backend_error {<br>  set beresp.http.Content-Type = "application/hal+json";<br>
  return (deliver);<br>}<br><br><br>## varnishstat -1 ##<br><br>MAIN.uptime             236760         1.00 Child process uptime<br>MAIN.sess_conn          217727         0.92 Sessions accepted<br>MAIN.sess_drop               0         0.00 Sessions dropped<br>
MAIN.sess_fail               0         0.00 Session accept failures<br>MAIN.sess_pipe_overflow            0         0.00 Session pipe overflow<br>MAIN.client_req_400                0         0.00 Client requests received, subject to 400 errors<br>
MAIN.client_req_411                0         0.00 Client requests received, subject to 411 errors<br>MAIN.client_req_413                0         0.00 Client requests received, subject to 413 errors<br>MAIN.client_req_417                0         0.00 Client requests received, subject to 417 errors<br>
MAIN.client_req              5316027        22.45 Good Client requests received<br>MAIN.cache_hit               2061461         8.71 Cache hits<br>MAIN.cache_hitpass                 0         0.00 Cache hits for pass<br>MAIN.cache_miss              3204801        13.54 Cache misses<br>
MAIN.backend_conn             107148         0.45 Backend conn. success<br>MAIN.backend_unhealthy             0         0.00 Backend conn. not attempted<br>MAIN.backend_busy                  0         0.00 Backend conn. too many<br>
MAIN.backend_fail                  0         0.00 Backend conn. failures<br>MAIN.backend_reuse           4253086        17.96 Backend conn. reuses<br>MAIN.backend_toolate           82192         0.35 Backend conn. was closed<br>
MAIN.backend_recycle         4335308        18.31 Backend conn. recycles<br>MAIN.backend_retry                 0         0.00 Backend conn. retry<br>MAIN.fetch_head                    0         0.00 Fetch no body (HEAD)<br>
MAIN.fetch_length                305         0.00 Fetch with Length<br>MAIN.fetch_chunked           4359871        18.41 Fetch chunked<br>MAIN.fetch_eof                     0         0.00 Fetch EOF<br>MAIN.fetch_bad                     0         0.00 Fetch bad T-E<br>
MAIN.fetch_close                   0         0.00 Fetch wanted close<br>MAIN.fetch_oldhttp                 0         0.00 Fetch pre HTTP/1.1 closed<br>MAIN.fetch_zero                    0         0.00 Fetch zero len body<br>
MAIN.fetch_1xx                     0         0.00 Fetch no body (1xx)<br>MAIN.fetch_204                     0         0.00 Fetch no body (204)<br>MAIN.fetch_304                     0         0.00 Fetch no body (304)<br>MAIN.fetch_failed               4994         0.02 Fetch body failed<br>
MAIN.pools                         2          .   Number of thread pools<br>MAIN.threads                     125          .   Total number of threads<br>MAIN.threads_limited               0         0.00 Threads hit max<br>
MAIN.threads_created           24761         0.10 Threads created<br>MAIN.threads_destroyed         24636         0.10 Threads destoryed<br>MAIN.threads_failed                0         0.00 Thread creation failed<br>MAIN.thread_queue_len              0          .   Length of session queue<br>
MAIN.busy_sleep                44391         0.19 Number of requests sent to sleep on busy objhdr<br>MAIN.busy_wakeup               44390         0.19 Number of requests woken after sleep on busy objhdr<br>MAIN.sess_queued               22381         0.09 Sessions queued for thread<br>
MAIN.sess_dropped                  0         0.00 Sessions dropped for thread<br>MAIN.n_object                  72729          .   N struct object<br>MAIN.n_vampireobject               0          .   N unresurrected objects<br>
MAIN.n_objectcore              79071          .   N struct objectcore<br>MAIN.n_objecthead              69256          .   N struct objecthead<br>MAIN.n_waitinglist             21819          .   N struct waitinglist<br>MAIN.n_backend                     3          .   N backends<br>
MAIN.n_expired               2977375          .   N expired objects<br>MAIN.n_lru_nuked             1336131          .   N LRU nuked objects<br>MAIN.n_lru_moved             1928525          .   N LRU moved objects<br>MAIN.losthdr                       0         0.00 HTTP header overflows<br>
MAIN.s_sess                   217727         0.92 Total Sessions<br>MAIN.s_req                   5316027        22.45 Total Requests<br>MAIN.s_pipe                        0         0.00 Total pipe<br>MAIN.s_pass                        0         0.00 Total pass<br>
MAIN.s_fetch                 3204801        13.54 Total fetch<br>MAIN.s_synth                   53251         0.22 Total synth<br>MAIN.s_req_hdrbytes       2351626120      9932.53 Request header bytes<br>MAIN.s_req_bodybytes               0         0.00 Request body bytes<br>
MAIN.s_resp_hdrbytes      3914065512     16531.79 Response header bytes<br>MAIN.s_resp_bodybytes    78724096530    332505.90 Reponse body bytes<br>MAIN.s_pipe_hdrbytes               0         0.00 Pipe request header bytes<br>
MAIN.s_pipe_in                     0         0.00 Piped bytes from client<br>MAIN.s_pipe_out                    0         0.00 Piped bytes to client<br>MAIN.sess_closed               47598         0.20 Session Closed<br>MAIN.sess_pipeline                 0         0.00 Session Pipeline<br>
MAIN.sess_readahead                0         0.00 Session Read Ahead<br>MAIN.sess_herd               2769201        11.70 Session herd<br>MAIN.shm_records           666122457      2813.49 SHM records<br>MAIN.shm_writes             27933605       117.98 SHM writes<br>
MAIN.shm_flushes                   0         0.00 SHM flushes due to overflow<br>MAIN.shm_cont                   1780         0.01 SHM MTX contention<br>MAIN.shm_cycles                  315         0.00 SHM cycles through buffer<br>
MAIN.sms_nreq                      0         0.00 SMS allocator requests<br>MAIN.sms_nobj                      0          .   SMS outstanding allocations<br>MAIN.sms_nbytes                    0          .   SMS outstanding bytes<br>
MAIN.sms_balloc                    0          .   SMS bytes allocated<br>MAIN.sms_bfree                     0          .   SMS bytes freed<br>MAIN.backend_req             4360234        18.42 Backend requests made<br>MAIN.n_vcl                         4         0.00 N vcl total<br>
MAIN.n_vcl_avail                   4         0.00 N vcl available<br>MAIN.n_vcl_discard                 0         0.00 N vcl discarded<br>MAIN.bans                          1          .   Count of bans<br>MAIN.bans_completed                1          .   Number of bans marked 'completed'<br>
MAIN.bans_obj                      0          .   Number of bans using obj.*<br>MAIN.bans_req                      0          .   Number of bans using req.*<br>MAIN.bans_added                    1         0.00 Bans added<br>
MAIN.bans_deleted                  0         0.00 Bans deleted<br>MAIN.bans_tested                   0         0.00 Bans tested against objects (lookup)<br>MAIN.bans_obj_killed               0         0.00 Objects killed by bans (lookup)<br>
MAIN.bans_lurker_tested            0         0.00 Bans tested against objects (lurker)<br>MAIN.bans_tests_tested             0         0.00 Ban tests tested against objects (lookup)<br>MAIN.bans_lurker_tests_tested            0         0.00 Ban tests tested against objects (lurker)<br>
MAIN.bans_lurker_obj_killed              0         0.00 Objects killed by bans (lurker)<br>MAIN.bans_dups                           0         0.00 Bans superseded by other bans<br>MAIN.bans_lurker_contention              0         0.00 Lurker gave way for lookup<br>
MAIN.bans_persisted_bytes               13          .   Bytes used by the persisted ban lists<br>MAIN.bans_persisted_fragmentation            0          .   Extra bytes in persisted ban lists due to fragmentation<br>MAIN.exp_mailed                        5722364        24.17 Number of objects mailed to expiry thread<br>
MAIN.exp_received                      5722364        24.17 Number of objects received by expiry thread<br>MAIN.hcb_nolock                        5266297        22.24 HCB Lookups without lock<br>MAIN.hcb_lock                          3204827        13.54 HCB Lookups with lock<br>
MAIN.hcb_insert                        3204827        13.54 HCB Inserts<br>MAIN.esi_errors                              0         0.00 ESI parse errors (unlock)<br>MAIN.esi_warnings                            0         0.00 ESI parse warnings (unlock)<br>
MAIN.vmods                                   2          .   Loaded VMODs<br>MAIN.n_gzip                            4083400        17.25 Gzip operations<br>MAIN.n_gunzip                          5065871        21.40 Gunzip operations<br>
MAIN.vsm_free                           972096          .   Free VSM space<br>MAIN.vsm_used                         83962512          .   Used VSM space<br>MAIN.vsm_cooling                             0          .   Cooling VSM space<br>
MAIN.vsm_overflow                            0          .   Overflow VSM space<br>MAIN.vsm_overflowed                          0         0.00 Overflowed VSM space<br>MGT.uptime                              307850         1.30 Management process uptime<br>
MGT.child_start                              2         0.00 Child process started<br>MGT.child_exit                               0         0.00 Child process normal exit<br>MGT.child_stop                               0         0.00 Child process unexpected exit<br>
MGT.child_died                               1         0.00 Child process died (signal)<br>MGT.child_dump                               0         0.00 Child process core dumped<br>MGT.child_panic                              1         0.00 Child process panic<br>
MEMPOOL.vbc.live                            88          .   In use<br>MEMPOOL.vbc.pool                            15          .   In Pool<br>MEMPOOL.vbc.sz_wanted                       88          .   Size requested<br>MEMPOOL.vbc.sz_needed                      120          .   Size allocated<br>
MEMPOOL.vbc.allocs                      107148         0.45 Allocations<br>MEMPOOL.vbc.frees                       107060         0.45 Frees<br>MEMPOOL.vbc.recycle                     105673         0.45 Recycled from pool<br>
MEMPOOL.vbc.timeout                      75842         0.32 Timed out from pool<br>MEMPOOL.vbc.toosmall                         0         0.00 Too small to recycle<br>MEMPOOL.vbc.surplus                       2032         0.01 Too many for pool<br>
MEMPOOL.vbc.randry                        1475         0.01 Pool ran dry<br>MEMPOOL.busyobj.live                        58          .   In use<br>MEMPOOL.busyobj.pool                        11          .   In Pool<br>MEMPOOL.busyobj.sz_wanted                65536          .   Size requested<br>
MEMPOOL.busyobj.sz_needed                65568          .   Size allocated<br>MEMPOOL.busyobj.allocs                 4391285        18.55 Allocations<br>MEMPOOL.busyobj.frees                  4391227        18.55 Frees<br>
MEMPOOL.busyobj.recycle                4380692        18.50 Recycled from pool<br>MEMPOOL.busyobj.timeout                 231100         0.98 Timed out from pool<br>MEMPOOL.busyobj.toosmall                     0         0.00 Too small to recycle<br>
MEMPOOL.busyobj.surplus                    916         0.00 Too many for pool<br>MEMPOOL.busyobj.randry                   10593         0.04 Pool ran dry<br>MEMPOOL.req0.live                           14          .   In use<br>
MEMPOOL.req0.pool                           10          .   In Pool<br>MEMPOOL.req0.sz_wanted                   65536          .   Size requested<br>MEMPOOL.req0.sz_needed                   65568          .   Size allocated<br>
MEMPOOL.req0.allocs                    1408010         5.95 Allocations<br>MEMPOOL.req0.frees                     1407996         5.95 Frees<br>MEMPOOL.req0.recycle                   1407941         5.95 Recycled from pool<br>
MEMPOOL.req0.timeout                    167174         0.71 Timed out from pool<br>MEMPOOL.req0.toosmall                        0         0.00 Too small to recycle<br>MEMPOOL.req0.surplus                         0         0.00 Too many for pool<br>
MEMPOOL.req0.randry                         69         0.00 Pool ran dry<br>MEMPOOL.sess0.live                          19          .   In use<br>MEMPOOL.sess0.pool                          10          .   In Pool<br>MEMPOOL.sess0.sz_wanted                    384          .   Size requested<br>
MEMPOOL.sess0.sz_needed                    416          .   Size allocated<br>MEMPOOL.sess0.allocs                    108788         0.46 Allocations<br>MEMPOOL.sess0.frees                     108769         0.46 Frees<br>
MEMPOOL.sess0.recycle                   108735         0.46 Recycled from pool<br>MEMPOOL.sess0.timeout                    76175         0.32 Timed out from pool<br>MEMPOOL.sess0.toosmall                       0         0.00 Too small to recycle<br>
MEMPOOL.sess0.surplus                        0         0.00 Too many for pool<br>MEMPOOL.sess0.randry                        53         0.00 Pool ran dry<br>MEMPOOL.req1.live                           14          .   In use<br>
MEMPOOL.req1.pool                           11          .   In Pool<br>MEMPOOL.req1.sz_wanted                   65536          .   Size requested<br>MEMPOOL.req1.sz_needed                   65568          .   Size allocated<br>
MEMPOOL.req1.allocs                    1409105         5.95 Allocations<br>MEMPOOL.req1.frees                     1409091         5.95 Frees<br>MEMPOOL.req1.recycle                   1408955         5.95 Recycled from pool<br>
MEMPOOL.req1.timeout                    168231         0.71 Timed out from pool<br>MEMPOOL.req1.toosmall                        0         0.00 Too small to recycle<br>MEMPOOL.req1.surplus                         0         0.00 Too many for pool<br>
MEMPOOL.req1.randry                        150         0.00 Pool ran dry<br>MEMPOOL.sess1.live                          19          .   In use<br>MEMPOOL.sess1.pool                          13          .   In Pool<br>MEMPOOL.sess1.sz_wanted                    384          .   Size requested<br>
MEMPOOL.sess1.sz_needed                    416          .   Size allocated<br>MEMPOOL.sess1.allocs                    108939         0.46 Allocations<br>MEMPOOL.sess1.frees                     108920         0.46 Frees<br>
MEMPOOL.sess1.recycle                   108788         0.46 Recycled from pool<br>MEMPOOL.sess1.timeout                    76251         0.32 Timed out from pool<br>MEMPOOL.sess1.toosmall                       0         0.00 Too small to recycle<br>
MEMPOOL.sess1.surplus                        0         0.00 Too many for pool<br>MEMPOOL.sess1.randry                       151         0.00 Pool ran dry<br>SMA.s0.c_req                          10059622        42.49 Allocator requests<br>
SMA.s0.c_fail                          1339270         5.66 Allocator failures<br>SMA.s0.c_bytes                    576850225314   2436434.47 Bytes allocated<br>SMA.s0.c_freed                    567228141680   2395793.81 Bytes freed<br>
SMA.s0.g_alloc                          145458          .   Allocations outstanding<br>SMA.s0.g_bytes                      9622083634          .   Bytes outstanding<br>SMA.s0.g_space                        41592782          .   Bytes available<br>
SMA.Transient.c_req                      26057         0.11 Allocator requests<br>SMA.Transient.c_fail                         0         0.00 Allocator failures<br>SMA.Transient.c_bytes                 11256624        47.54 Bytes allocated<br>
SMA.Transient.c_freed                 11256624        47.54 Bytes freed<br>SMA.Transient.g_alloc                        0          .   Allocations outstanding<br>SMA.Transient.g_bytes                        0          .   Bytes outstanding<br>
SMA.Transient.g_space                        0          .   Bytes available<br>VBE.backend1(x.x.x.x,,8080).vcls            4          .   VCL references<br>VBE.backend1(x.x.x.x,,8080).happy 11493186249049505791          .   Happy health probes<br>
VBE.backend1(x.x.x.x,,8080).bereq_hdrbytes    715578294      3022.38 Request header bytes<br>VBE.backend1(x.x.x.x,,8080).bereq_bodybytes            0         0.00 Request body bytes<br>VBE.backend1(x.x.x.x,,8080).beresp_hdrbytes    941776370      3977.77 Response header bytes<br>
VBE.backend1(x.x.x.x,,8080).beresp_bodybytes  17197789137     72638.07 Response body bytes<br>VBE.backend1(x.x.x.x,,8080).pipe_hdrbytes               0         0.00 Pipe request header bytes<br>VBE.backend1(x.x.x.x,,8080).pipe_out                    0         0.00 Piped bytes to backend<br>
VBE.backend1(x.x.x.x,,8080).pipe_in                     0         0.00 Piped bytes from backend<br>VBE.backend2(x.x.x.x,,8080).vcls                        4          .   VCL references<br>VBE.backend2(x.x.x.x,,8080).happy            14987979559889002495          .   Happy health probes<br>
VBE.backend2(x.x.x.x,,8080).bereq_hdrbytes      715272478      3021.09 Request header bytes<br>VBE.backend2(x.x.x.x,,8080).bereq_bodybytes             0         0.00 Request body bytes<br>VBE.backend2(x.x.x.x,,8080).beresp_hdrbytes     941339831      3975.92 Response header bytes<br>
VBE.backend2(x.x.x.x,,8080).beresp_bodybytes  17174050588     72537.80 Response body bytes<br>VBE.backend2(x.x.x.x,,8080).pipe_hdrbytes               0         0.00 Pipe request header bytes<br>VBE.backend2(x.x.x.x,,8080).pipe_out                    0         0.00 Piped bytes to backend<br>
VBE.backend2(x.x.x.x,,8080).pipe_in                     0         0.00 Piped bytes from backend<br>VBE.backend3(x.x.x.x,,8080).vcls                        4          .   VCL references<br>VBE.backend3(x.x.x.x,,8080).happy            4611686018427363327          .   Happy health probes<br>
VBE.backend3(x.x.x.x,,8080).bereq_hdrbytes      707848949      2989.73 Request header bytes<br>VBE.backend3(x.x.x.x,,8080).bereq_bodybytes             0         0.00 Request body bytes<br>VBE.backend3(x.x.x.x,,8080).beresp_hdrbytes     931448534      3934.15 Response header bytes<br>
VBE.backend3(x.x.x.x,,8080).beresp_bodybytes  16984021399     71735.18 Response body bytes<br>VBE.backend3(x.x.x.x,,8080).pipe_hdrbytes               0         0.00 Pipe request header bytes<br>VBE.backend3(x.x.x.x,,8080).pipe_out                    0         0.00 Piped bytes to backend<br>
VBE.backend3(x.x.x.x,,8080).pipe_in                     0         0.00 Piped bytes from backend<br>LCK.sms.creat                                                     0         0.00 Created locks<br>LCK.sms.destroy                                                   0         0.00 Destroyed locks<br>
LCK.sms.locks                                                     0         0.00 Lock Operations<br>LCK.smp.creat                                                     0         0.00 Created locks<br>LCK.smp.destroy                                                   0         0.00 Destroyed locks<br>
LCK.smp.locks                                                     0         0.00 Lock Operations<br>LCK.sma.creat                                                     2         0.00 Created locks<br>LCK.sma.destroy                                                   0         0.00 Destroyed locks<br>
LCK.sma.locks                                              18686630        78.93 Lock Operations<br>LCK.smf.creat                                                     0         0.00 Created locks<br>LCK.smf.destroy                                                   0         0.00 Destroyed locks<br>
LCK.smf.locks                                                     0         0.00 Lock Operations<br>LCK.hsl.creat                                                     0         0.00 Created locks<br>LCK.hsl.destroy                                                   0         0.00 Destroyed locks<br>
LCK.hsl.locks                                                     0         0.00 Lock Operations<br>LCK.hcb.creat                                                     1         0.00 Created locks<br>LCK.hcb.destroy                                                   0         0.00 Destroyed locks<br>
LCK.hcb.locks                                               6356203        26.85 Lock Operations<br>LCK.hcl.creat                                                     0         0.00 Created locks<br>LCK.hcl.destroy                                                   0         0.00 Destroyed locks<br>
LCK.hcl.locks                                                     0         0.00 Lock Operations<br>LCK.vcl.creat                                                     1         0.00 Created locks<br>LCK.vcl.destroy                                                   0         0.00 Destroyed locks<br>
LCK.vcl.locks                                               8976880        37.92 Lock Operations<br>LCK.sessmem.creat                                                 0         0.00 Created locks<br>LCK.sessmem.destroy                                               0         0.00 Destroyed locks<br>
LCK.sessmem.locks                                                 0         0.00 Lock Operations<br>LCK.sess.creat                                               217727         0.92 Created locks<br>LCK.sess.destroy                                             217689         0.92 Destroyed locks<br>
LCK.sess.locks                                                    0         0.00 Lock Operations<br>LCK.wstat.creat                                                   1         0.00 Created locks<br>LCK.wstat.destroy                                                 0         0.00 Destroyed locks<br>
LCK.wstat.locks                                            19771639        83.51 Lock Operations<br>LCK.herder.creat                                                  0         0.00 Created locks<br>LCK.herder.destroy                                                0         0.00 Destroyed locks<br>
LCK.herder.locks                                                  0         0.00 Lock Operations<br>LCK.wq.creat                                                      3         0.00 Created locks<br>LCK.wq.destroy                                                    0         0.00 Destroyed locks<br>
LCK.wq.locks                                               14797256        62.50 Lock Operations<br>LCK.objhdr.creat                                            3216824        13.59 Created locks<br>LCK.objhdr.destroy                                          3147552        13.29 Destroyed locks<br>
LCK.objhdr.locks                                           60262894       254.53 Lock Operations<br>LCK.exp.creat                                                     1         0.00 Created locks<br>LCK.exp.destroy                                                   0         0.00 Destroyed locks<br>
LCK.exp.locks                                              26069460       110.11 Lock Operations<br>LCK.lru.creat                                                     2         0.00 Created locks<br>LCK.lru.destroy                                                   0         0.00 Destroyed locks<br>
LCK.lru.locks                                              18404226        77.73 Lock Operations<br>LCK.cli.creat                                                     1         0.00 Created locks<br>LCK.cli.destroy                                                   0         0.00 Destroyed locks<br>
LCK.cli.locks                                                 78936         0.33 Lock Operations<br>LCK.ban.creat                                                     1         0.00 Created locks<br>LCK.ban.destroy                                                   0         0.00 Destroyed locks<br>
LCK.ban.locks                                               9477160        40.03 Lock Operations<br>LCK.vbp.creat                                                     1         0.00 Created locks<br>LCK.vbp.destroy                                                   0         0.00 Destroyed locks<br>
LCK.vbp.locks                                                124145         0.52 Lock Operations<br>LCK.backend.creat                                                 3         0.00 Created locks<br>LCK.backend.destroy                                               0         0.00 Destroyed locks<br>
LCK.backend.locks                                           8991937        37.98 Lock Operations<br>LCK.vcapace.creat                                                 1         0.00 Created locks<br>LCK.vcapace.destroy                                               0         0.00 Destroyed locks<br>
LCK.vcapace.locks                                                 0         0.00 Lock Operations<br>LCK.nbusyobj.creat                                                0         0.00 Created locks<br>LCK.nbusyobj.destroy                                              0         0.00 Destroyed locks<br>
LCK.nbusyobj.locks                                                0         0.00 Lock Operations<br>LCK.busyobj.creat                                           4391285        18.55 Created locks<br>LCK.busyobj.destroy                                         4391227        18.55 Destroyed locks<br>
LCK.busyobj.locks                                          33222445       140.32 Lock Operations<br>LCK.mempool.creat                                                 6         0.00 Created locks<br>LCK.mempool.destroy                                               0         0.00 Destroyed locks<br>
LCK.mempool.locks                                          17553267        74.14 Lock Operations<br>LCK.vxid.creat                                                    1         0.00 Created locks<br>LCK.vxid.destroy                                                  0         0.00 Destroyed locks<br>
LCK.vxid.locks                                                22801         0.10 Lock Operations<br>LCK.pipestat.creat                                                1         0.00 Created locks<br>LCK.pipestat.destroy                                              0         0.00 Destroyed locks<br>
LCK.pipestat.locks                                                0         0.00 Lock Operations<br><br></div>