LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Virtualization and Cloud
User Name
Password
Linux - Virtualization and Cloud This forum is for the discussion of all topics relating to Linux Virtualization and Linux Cloud platforms. Xen, KVM, OpenVZ, VirtualBox, VMware, Linux-VServer and all other Linux Virtualization platforms are welcome. OpenStack, CloudStack, ownCloud, Cloud Foundry, Eucalyptus, Nimbus, OpenNebula and all other Linux Cloud platforms are welcome. Note that questions relating solely to non-Linux OS's should be asked in the General forum.

Notices

Reply
 
Search this Thread
Old 06-30-2010, 03:41 PM   #1
Lagos
LQ Newbie
 
Registered: Jun 2010
Location: Las Cruces, NM
Distribution: MAC OSX 10.6.4 / Fedora 12
Posts: 11

Rep: Reputation: 8
Exclamation Upgrading to VMWare Virtual Machine Hardware version 7.0 steals available memory


Our team recently updated each of our esx hosts to 4.0. However, we *did not* immediately upgrade the Virtual Machine Hardware version. We ran 4.0 for quite some time with no issues. After powering down the VM, upgrading to Virtual Machine Hardware v7.0, and powering back on we discovered that our VM had mysteriously "lost" ~ 800MB of memory. I've pasted data from two VMs below to illustrate this point.

This first VM is running Virtual Machine Hardware v4.0 and is reporting correct memory numbers all around.

VM -1
ESX Version: 4.0.0,261974
Virtual Machine Hardware Version 4
Guest O.S: RHEL 5
Guest kernel: 2.6.18-194.3.1.el5
Virtual Machine Memory Settings: 4096MB
Virtual Machine BIOS (Phoenix): System Memory 640kb / Extended Memory 4193280kb

>free -m
total used free shared buffers cached
Mem: 3804 601 3203 0 61 474
-/+ buffers/cache: 64 3740
Swap: 4095 0 4095

>dmesg
---clip----
BIOS-provided physical RAM map:
BIOS-e820: 0000000000010000 - 000000000009f800 (usable)
BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)
BIOS-e820: 00000000000dc000 - 00000000000e4000 (reserved)
BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 00000000efef0000 (usable)
BIOS-e820: 00000000efef0000 - 00000000efeff000 (ACPI data)
BIOS-e820: 00000000efeff000 - 00000000eff00000 (ACPI NVS)
BIOS-e820: 00000000eff00000 - 00000000f0000000 (usable)
BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000fffe0000 - 0000000100000000 (reserved)
2944MB HIGHMEM available.
896MB LOWMEM available.
---clip----

This second VM is missing memory. It is running Virtual Machine Hardware v7.0 (all else being equal). The memory that is reported is inconsistent and dependent on the source.

VM - 2
ESX Version: 4.0.0,261974
Virtual Machine Hardware Version 7
Guest O.S: RHEL 5
Guest kernel: 2.6.18-194.3.1.el5
Virtual Machine Settings: 4096MB
Virtual Machine BIOS (Phoenix): System Memory 640kb / Extended Memory 5241856kb

>free -m
total used free shared buffers cached
Mem: 3034 281 2753 0 29 206
-/+ buffers/cache: 45 2988
Swap: 4095 0 4095

>dmesg
---clip----
BIOS-provided physical RAM map:
BIOS-e820: 0000000000010000 - 000000000009f800 (usable)
BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)
BIOS-e820: 00000000000dc000 - 00000000000e4000 (reserved)
BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 00000000bfef0000 (usable)
BIOS-e820: 00000000bfef0000 - 00000000bfeff000 (ACPI data)
BIOS-e820: 00000000bfeff000 - 00000000bff00000 (ACPI NVS)
BIOS-e820: 00000000bff00000 - 00000000c0000000 (usable)
BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000fffe0000 - 0000000100000000 (reserved)
BIOS-e820: 0000000100000000 - 0000000140000000 (usable)
Warning only 4GB will be used.
Use a PAE enabled kernel.
3200MB HIGHMEM available.
896MB LOWMEM available.
---clip----

What is interesting is that it looks like the BIOS is providing a memory map that includes address space which goes above and beyond the 4GB mark (0000000100000000). From what I understand, this is typical for instances wherein memory is being "remapped" by the BIOS. I also understand that the OS should be able to use this newly addressed space without having to go to a PAE kernel. What's confusing is that the the amount of memory that has been remapped is *not* equal to the amount of missing memory. There looks to be several factors involved

Has anyone else seen this before?
 
Old 07-01-2010, 08:02 AM   #2
scheidel21
Senior Member
 
Registered: Feb 2003
Location: CT
Distribution: Debian PPC/i386/AMD64 6/7, Vista, XP , WIN7, Server 03/08
Posts: 1,287

Rep: Reputation: 97
I am making an assumption here, but I have seen some newer Virtualization products start to use dynamic memory allocation, maybe it is somehting like that?
 
Old 07-01-2010, 08:40 AM   #3
Lagos
LQ Newbie
 
Registered: Jun 2010
Location: Las Cruces, NM
Distribution: MAC OSX 10.6.4 / Fedora 12
Posts: 11

Original Poster
Rep: Reputation: 8
@scheidel21

Thanks for the response.

We investigated ESX memory management to correct an issue we had a while back. As far as I know, ESX will dynamically allocate and reclaim memory using some pretty cool algorithms that depend on several variables. The balloon driver is particularly impressive. However, it's my understanding that all of this is managed at the hypervisor level. As far as the guest OS is concerned, it's convinced that its always had whatever memory it was originally configured with (ie. VM settings). That is, it has no visibility into resource management at the hypervisor level. I also think the dmseg output occurs long before esx has had a chance to invoke its memory management process. That is, the memory goes missing right out of the gate.
 
Old 07-01-2010, 09:05 AM   #4
Lagos
LQ Newbie
 
Registered: Jun 2010
Location: Las Cruces, NM
Distribution: MAC OSX 10.6.4 / Fedora 12
Posts: 11

Original Poster
Rep: Reputation: 8
Friends,

I'm ashamed that I missed this vmware kb article:

http://kb.vmware.com/selfservice/mic...rnalId=1014006

----clip----
Solution
The PC architecture reserves a portion of the address space that is below 4GB for PCI devices. This space cannot be used for system memory. Also, note that this is one of the main reasons for guests not recognizing all allocated physical memory.

This table lists the address space size available for system memory under 4GB for each hardware version.
Hardware Version Reserved Memory Supported Memory
7 1024MB 3GB (3072MB)
6 512MB 3.5GB (3584MB)
4 256MB 3.75GB (3840MB)
3 496MB* 3.52GB (3600MB)
----clip----

...and that's that
 
1 members found this post helpful.
Old 01-02-2011, 01:29 PM   #5
archtoad6
Senior Member
 
Registered: Oct 2004
Location: Houston, TX (usa)
Distribution: MEPIS, Debian, Knoppix,
Posts: 4,727
Blog Entries: 15

Rep: Reputation: 231Reputation: 231Reputation: 231
Quote:
Originally Posted by Lagos View Post
I'm ashamed that I missed this vmware kb article:
....
...and that's that
Good for you for admitting it. That provides a solution & helps build the "Archive of Answers".
 
  


Reply

Tags
dmesg, esx, free, memory, rhel5, vmware, vsphere


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Vmware(Virtual Machine) on Ubuntu ahmed_as8 Linux - Newbie 4 01-29-2009 06:19 AM
vmware virtual machine slow in vmware client, faster viewer available? purplesocrates Linux - General 1 06-17-2008 10:40 AM
starting your own virtual machine in vmware 1dude1 Linux - Software 1 01-26-2007 01:55 PM
Vmware Virtual Machine Problem With Windows In Vmware jvdbossc Linux - Software 2 05-28-2004 06:54 AM
vmware virtual machine won't power on Castor Slackware 8 01-07-2004 09:35 AM


All times are GMT -5. The time now is 09:06 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration