Published 6/9/2011
Part III of III: More Tests to Validate Our Findings — Part III
In part II of our blog, we established that changes in memory consumed are inversely proportional to changes in memory reclaimed. In other words, the amount of memory reclaimed using balloon, swap and/or memory compression is inversely correlated to the amount of "Consumed" plus "SHRDSVD" and only the "Consumed" memory of VM is backed by physical memory.
When reclaiming memory via memory balloon, the balloon driver has no knowledge of the memory pinned is "Consumed" or "SHRDSVD". As such, when the amount of "SHRDSVD" is nonzero, the balloon driver must pin more memory from the VM in order to provide enough physical memory for the hypervisor to reclaim. If this process is not fast enough to satisfy the memory requirements, the hypervisor must resort to swap and/or memory compression.
To validate our theory, we plotted, in greater detail, the memory activities discussed in Fig. 1 of Part I of our blog series in order to visualize the above equation. Notice that the primary Y-axis represents the sum of the ballooned and swapped memory. The secondary Y-axis represents the VM memory shown in reversed value so that we could show the changes in VM memory in relation to the changes in ballooned and swapped memory as described in the previous differential equation.
Figure 3. Ballooned memory includes consumed and shared saved memory
The majority of the ballooned memory in the chart above was indeed coming from "Shared Saved" memory, not physical memory allocated to the VM. We also see from the above chart that the swapped memory reclaimed is coming directly from the "Consumed" memory!
Based on our initial findings, we thought there might be some discrepancy in VMware documentation relating to memory balloon. After extensive discussions with our good friend at VMware’s Performance group, we have the following agreements:
- The behavior we saw is specific to Windows VMs that are either freshly booted OR Windows VMs with deflated balloon driver. However, in all other circumstances when the VM memory is fully backed by host physical memory, the behavior explained in the VMware documentation still holds true.
- The behavior cited in the VMware docs holds true for all non-Windows VMs.
To prove the above statements, we performed another memory test now with VMs that were affected by the balloon memory with their memory fully backed by the host memory. The test result is very different than the one we showed in our Part I of the blog series. The amount of memory ballooned now is within our expectation. Furthermore, since memory balloon was able to reclaim enough physical memory, the amount of kernel swapping was nearly nonexistent as a result! The memory activities are shown in the following diagram:
Figure 4. Memory reclamation activities with VM memory fully backed
The above figure shows that the amount of memory balloon in the first half of the test is now much more reasonable. The large amount of memory balloon seen in the 2nd half of our tests is due to the additional memory activities imposed on the test system which will not be covered here. Since the hypervisor was able to reclaim enough memory via ballooning, there is no kernel swapping in the above figure.
We will plot the same memory reclamation activities as shown in Figure 4 as follows:
Figure 5 Memory reclamation activities
The above figure shows that when Windows VM’s memory is fully backed by host memory, the ballooned memory comes entirely from the "Consumed" memory. Furthermore, the above figure shows that the "shared saved" memory could also come from the deflated balloon memory.
In summary, the ballooned memory seen in Figure 1 of our first blog includes both the "Consumed" and "Shared Saved" memory of the VMs. Since only "Consumed" memory of the VM is backed by physical memory, the balloon driver must continue to pin more memory from the VM and this is why the amount of ballooned memory was high . Furthermore, when ESX cannot reclaim memory via Memory Balloon fast enough, ESX had to resort to swapping to service the demand.
The additional test shown in this blog post validates the fact that these are special cases for Windows VMs that are either freshly booted and/or with deflated balloon memory. So finally, we believe we now fully understand how memory ballooning actually works!
Author: YP Chien