Brain Dump: VMware Memory Management


  • Consumed Host Memory
    • amount of memory allocated to the virtual machine by the host
  • Active Guest Memory
      • amount of memory that is currently being used by the guest
    • Guest Level Paging
    • The memory transfer between the guest physical memory and the guest swap device
    • is driven by the guest operating system
  • Hypervisor Swapping
    • The memory transfer between guest physical memory and the host swap device
    • is driven by the hypervisor
  • Configured [Virtual Machine Memory] Size
    • a construct maintained by the virtualization layer for the virtual machine
    • it is the amount of memory that is presented to the guest operating system, but it is independent of the amount of physical RAM that is allocated to the virtual machine, which depends on the resource settings (shares, reservation, limit)
  • Shares
    • Specify the relative priority for a virtual machine if more than the reservation is available.
  • Reservation
    • a guaranteed lower bound on the amount of physical memory that the host reserves for the virtual machine, even when memory is over-committed
    • until the virtual machine accesses its full reservation, VMkernel can allocate any unused portion of its reservation to other virtual machines
    • after the guest’s workload increases and it consumes its full reservation, the guest is allowed to keep this memory, even if it becomes idle.
  • Limit
    • an upper bound on the amount of physical memory that the host can allocate to the virtual machine.

Memory Layers

  • There are three virtual memory layers in ESX
    1. guest virtual memory: a continuous virtual address space presented by the guest operating system to applications. It is the memory that is visible to the applications running inside the virtual machine.
    2. guest physical memory: the memory that is visible to the guest operating system running in the virtual machine. It is backed by host physical memory, which means the hypervisor provides a mapping from the guest to the host memory
    3. host physical memory: the memory that is visible to the hypervisor as available on the system


When the hypervisor runs multiple virtual machines and the total amount of the free host memory becomes low, none of the virtual machines will free guest physical memory because the guest operating system cannot detect the host’s memory shortage. Ballooning makes the guest operating system aware of the low memory status of the host.

  • A balloon driver is loaded into the guest operating system as a pseudo-device driver
  • it has no external interfaces to the guest operating system
  • it communicates with the hypervisor through a private channel
  • it polls the hypervisor to obtain a target balloon size if necessary

If the hypervisor needs to reclaim virtual machine memory, it sets a proper target balloon size for the balloon driver, making it “inflate” by allocating guest physical pages within the virtual machine.

Ballooning Example

  • four guest physical pages are mapped in the host physical memory
  • two of the pages are used by the guest application
  • the other two pages are in the guest operating system free list Note that since the hypervisor cannot identify the two pages in the guest free list, it cannot reclaim the host physical pages that are backing them

  • the hypervisor will set the target balloon size to two pages
  • the balloon driver allocates two guest physical pages inside the virtual machine and pins the
  • the balloon driver notifies the hypervisor the page numbers of the pinned guest physical memory
  • the hypervisor can safely reclaim this host physical memory because neither the balloon driver nor the guest operating system relies on the contents of these pages
  • when the hypervisor decides to deflate the balloon, the balloon driver de-allocates the pinned guest physical memory, which releases it for the guest’s applications

Much (if not all) of the information is this post is shamelessly ripped from VMware’s whitepaper, Understanding Memory Resource Management in VMware® ESXTM Server. and vSphere Resource Management Guide