How the L2ARC works

L2ARC of ZFS is quite different from an ordinary L2 cache of a CPU. The following figure shows the structure of the caches of ZFS. L2ARC caches only blocks in the ARC read from HDDs. It doesn't cache dirty blocks in ARC written to HDDs. There is no data transfer between L2ARC and HDDs. When ZFS doesn't cache a requested block in ARC but  L2ARC, it copy the block to ARC and respond it.

L2ARC caches blocks almost evicted from ARC. ZFS periodically scans a limited range from each tail of four LRUs in ARC. It gathers blocks not cached yet in L2ARC until the total size of them reaches the size defined by l2arc_write_max. The default is 8MiB. The scan range equals the l2arc_write_max times the l2arc_headroom. The default of the l2arc_headroom is 2. A higher value makes the total of the gathered blocks near to the l2arc_write_max. The following figure based on the comment in the arc.c shows this process.

L2ARC forms one or more ring buffers. ZFS copies a hit block in L2ARC to ARC, but it doesn't move this to the head of L2ARC. It evicts old blocks from L2ARC to make room to copy new blocks. If L2ARC has multiple SSDs, each of them forms a separate ring buffer. ZFS uses them in the round robin manner.

There are some parameters to control the cycle of the scanning. ZFS chooses one of two cycles by the total size of the gathered blocks. If the total size is equal to or smaller than the half of the l2arc_write_max, it chooses the minutes of the l2arc_feed_secs. The default is 1. Otherwise it chooses the milliseconds of the l2arc_feed_min_ms. The default is 200. The cycle becomes slow if L2ARC caches enough blocks and the total size of gathered blocks is small. Otherwise the cycle becomes fast.

I investigated the behavior of L2ARC on our server with the dtrace script shown below and found a few tips. A higher value of the l2arc_headroom increases the speed of filling the cache. A higher value of the l2arc_write_max also increases the speed if the cache isn't enough filled. Otherwise it slows down the speed because of the slow scanning cycle.