How to restrict ZFSin's total RAM usage

[imtiazdc]

We tried setting arc_max=15GB (with both arc_min=5GB and default) on a machine with 128GB of RAM but notice the RAM usage of ZFSin going up to 19.5GB and stabilizing around that figure. We observed the same behavior with different zvol sizes and different time durations. The total RAM usage of ZFSin seems to be 4-5 GB above the arc_max limit.

@lundman Any idea what could be consuming that additional 4-5 GB of RAM (on top of arc_max)? What and how do we investigate?

NOTE: We tried running verifier to see if there are any memory leaks but the driver doesn't unload upon uninstallation of ZFSin which doesn't help the verifier break into the debugger (so, even if there is a memory leak the verifier tool is unable to flag).

[lundman]

Right, so arc_max control the size of the ARC, which is the "main cache" on ZFS, but just that, a cache. There are other caches, and of course, a whole bunch of memory ZFS uses beyond that.

kmem/vmem in illumos is obviously constrained by physical memory, and will issue own pressure indicators when "available" memory is low. This doesn't really exist under macOS / Windows. We allocate memory from the OS, and let the OS send pressure indicators.

Perhaps we should discuss having a ceiling for kmem, and issue internal pressure, as well as, OS pressure - to stay below a limit.

[imtiazdc]

@lundman Is there a way to know the exact memory usage of ARC in ZFSin? In other words, how do we confirm that the arc_max limit is being honored by ARC?

[lundman]

I recommend running arcstat.pl to view the state of the ARC as you run, it will so it deflate and grow etc. I can run it under git-for-windows-bash shell without any extra installs.

The kstat segkmem_total_mem_allocated is all the memory ZFS has allocated (through spl) - which is pretty much all.