malloc storage memory leak?

[Rogier R. Mulhuijzen]

I would go out and drop 700 bucks on 4x16GB for my server if I were you.

http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=100007952 
600336949&IsNodeId=1&name=64GB (4 x 16GB)

22TB of tiles is quite a bit, so maybe go through the logfiles a bit and 
see exactly how much having 50G of cache would gain you. And if the 
zoomlevel is in the URL of the tiles, I would maybe choose to just cache 
the outer N levels, so that inner levels don't blow away the cache of the 
outer. Or run two varnishes, one caching outer layers, like I just 
described and have a set amount of memory for that, and then run the 
second one with a smaller set, just to catch those requests for inner 
tiles that somehow are very popular.

Which brings me back to the fact that this is a -dev list (so your 
question really doesn't belong here) and I just had an idea.

This separating storage deal, I'm a bit behind on 3/master, so do we have 
stevedores exposed to VCL in any way? I recall storage hints, but not sure 
if there's anything using that beyond transient?

Anyhoo, it might be worth it to segregate objects by various properties 
and not have to run separate varnish instances. :)

Cheers,

 	Doc


On Mon, 23 Jul 2012, Cal Heldenbrand wrote:

> Thanks for the clarification Andreas.  I'm running around 600k objects in
> memory, which should be around 586MB of overhead.
>
> If I could ask you guys your opinion -- is there a better way to configure
> Varnish for my environment?  My backend is 22TB of mapping tiles, each file
> being anywhere from 100bytes to 3KB.  So a small 4GB cache results in just
> being an LRU, caching the most popular tiles.  Which makes outer zoom
> levels very fast, but misses on almost all of the low parcel levels.
>
> Thanks for any advice!
>
> --Cal
>
> On Mon, Jul 23, 2012 at 2:39 AM, Andreas Plesner Jacobsen <apj at mutt.dk>wrote:
>
>> On Thu, Jul 19, 2012 at 04:11:20PM -0500, Cal Heldenbrand wrote:
>>>
>>> Here's a screenshot of top.  This machine was set to malloc max at
>> 5500MB.
>>
>> And it hasn't passed that. Remember that there's additional overhead of
>> about
>> 1KB/object outside the actual storage backend.
>>
>>> SMA.s0.g_bytes        5595155188          .   Bytes outstanding
>>
>> It has allocated 5.5G
>>
>>> SMA.Transient.g_bytes            0          .   Bytes outstanding
>>
>> And isn't gobbling up transient space at the moment.
>>
>> I don't see a problem (in varnish at least).
>>
>> --
>> Andreas
>>
>