|
Upgrading to VMWare Virtual Machine Hardware version 7.0 steals available memory
Our team recently updated each of our esx hosts to 4.0. However, we *did not* immediately upgrade the Virtual Machine Hardware version. We ran 4.0 for quite some time with no issues. After powering down the VM, upgrading to Virtual Machine Hardware v7.0, and powering back on we discovered that our VM had mysteriously "lost" ~ 800MB of memory. I've pasted data from two VMs below to illustrate this point.
This first VM is running Virtual Machine Hardware v4.0 and is reporting correct memory numbers all around. VM -1 ESX Version: 4.0.0,261974 Virtual Machine Hardware Version 4 Guest O.S: RHEL 5 Guest kernel: 2.6.18-194.3.1.el5 Virtual Machine Memory Settings: 4096MB Virtual Machine BIOS (Phoenix): System Memory 640kb / Extended Memory 4193280kb >free -m total used free shared buffers cached Mem: 3804 601 3203 0 61 474 -/+ buffers/cache: 64 3740 Swap: 4095 0 4095 >dmesg ---clip---- BIOS-provided physical RAM map: BIOS-e820: 0000000000010000 - 000000000009f800 (usable) BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved) BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved) BIOS-e820: 00000000000dc000 - 00000000000e4000 (reserved) BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 00000000efef0000 (usable) BIOS-e820: 00000000efef0000 - 00000000efeff000 (ACPI data) BIOS-e820: 00000000efeff000 - 00000000eff00000 (ACPI NVS) BIOS-e820: 00000000eff00000 - 00000000f0000000 (usable) BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved) BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) BIOS-e820: 00000000fffe0000 - 0000000100000000 (reserved) 2944MB HIGHMEM available. 896MB LOWMEM available. ---clip---- This second VM is missing memory. It is running Virtual Machine Hardware v7.0 (all else being equal). The memory that is reported is inconsistent and dependent on the source. VM - 2 ESX Version: 4.0.0,261974 Virtual Machine Hardware Version 7 Guest O.S: RHEL 5 Guest kernel: 2.6.18-194.3.1.el5 Virtual Machine Settings: 4096MB Virtual Machine BIOS (Phoenix): System Memory 640kb / Extended Memory 5241856kb >free -m total used free shared buffers cached Mem: 3034 281 2753 0 29 206 -/+ buffers/cache: 45 2988 Swap: 4095 0 4095 >dmesg ---clip---- BIOS-provided physical RAM map: BIOS-e820: 0000000000010000 - 000000000009f800 (usable) BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved) BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved) BIOS-e820: 00000000000dc000 - 00000000000e4000 (reserved) BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 00000000bfef0000 (usable) BIOS-e820: 00000000bfef0000 - 00000000bfeff000 (ACPI data) BIOS-e820: 00000000bfeff000 - 00000000bff00000 (ACPI NVS) BIOS-e820: 00000000bff00000 - 00000000c0000000 (usable) BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved) BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) BIOS-e820: 00000000fffe0000 - 0000000100000000 (reserved) BIOS-e820: 0000000100000000 - 0000000140000000 (usable) Warning only 4GB will be used. Use a PAE enabled kernel. 3200MB HIGHMEM available. 896MB LOWMEM available. ---clip---- What is interesting is that it looks like the BIOS is providing a memory map that includes address space which goes above and beyond the 4GB mark (0000000100000000). From what I understand, this is typical for instances wherein memory is being "remapped" by the BIOS. I also understand that the OS should be able to use this newly addressed space without having to go to a PAE kernel. What's confusing is that the the amount of memory that has been remapped is *not* equal to the amount of missing memory. There looks to be several factors involved Has anyone else seen this before? |
I am making an assumption here, but I have seen some newer Virtualization products start to use dynamic memory allocation, maybe it is somehting like that?
|
@scheidel21
Thanks for the response. We investigated ESX memory management to correct an issue we had a while back. As far as I know, ESX will dynamically allocate and reclaim memory using some pretty cool algorithms that depend on several variables. The balloon driver is particularly impressive. However, it's my understanding that all of this is managed at the hypervisor level. As far as the guest OS is concerned, it's convinced that its always had whatever memory it was originally configured with (ie. VM settings). That is, it has no visibility into resource management at the hypervisor level. I also think the dmseg output occurs long before esx has had a chance to invoke its memory management process. That is, the memory goes missing right out of the gate. |
Friends,
I'm ashamed that I missed this vmware kb article: http://kb.vmware.com/selfservice/mic...rnalId=1014006 ----clip---- Solution The PC architecture reserves a portion of the address space that is below 4GB for PCI devices. This space cannot be used for system memory. Also, note that this is one of the main reasons for guests not recognizing all allocated physical memory. This table lists the address space size available for system memory under 4GB for each hardware version. Hardware Version Reserved Memory Supported Memory 7 1024MB 3GB (3072MB) 6 512MB 3.5GB (3584MB) 4 256MB 3.75GB (3840MB) 3 496MB* 3.52GB (3600MB) ----clip---- ...and that's that |
Quote:
|
| All times are GMT -5. The time now is 10:27 PM. |