LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 10-18-2007, 03:03 PM   #1
The00Dustin
Member
 
Registered: Jan 2006
Posts: 70

Rep: Reputation: 15
Fedora 7 (moonshine) xen not detecting >4GB RAM


I have a Core 2 Quad with 8 GiB of RAM. When I installed FC7 (2.6.22.9-91.fc7), it only detected 4GiB (BIOS detects 8GiB). I added mem=8G to the kernel line in grub.conf, and then Gnome System Monitor showed 7.1GiB (vs I think 4.0). I then changed that to mem=16G, and the Gnome System Monitor shows 7.8 GiB. /proc/meminfo MemTotal shows 8201832kB, so that seems about right. That said, the problem seems to be solved for the kernel mentioned above.
However, the virtualization kernel (2.6.20-2936.fc7xen) shows 3.1GiB in Gnome System Monitor and 3214456KB in /proc/meminfo MemTotal. Additionally, "xm info" shows 3327 for total memory, so the problem isn’t that it is there but not assigned to dom0. I tried the same thing with grub.conf for xen, but it did not work, mem=8G and mem=16G had no effect (this is shown to be a valid parameter in the Xen manual at [http://tx.downloads.xensource.com/do...00000000000000]). Since it didn’t work on the kernel line, I tried it on the module (vmlinuz) line that has the other FC grub arguments, and that did not work either; then I tried placing that on both lines (also to no avail). I also tried dom0_mem=8G on the kernel line, and that caused a kernel panic stating not enough physical memory (got the same thing with dom0_mem=4G).
Finally, as a side note, /proc/meminfo VmallocTotal shows 34359738367kB regardless of all of the above, and I don't know if that is normal or not (swap partition is 2GiB). Please let me know if there is any additional information that might be helpful, if you have any suggestions I might try in order to resolve this, or if this question would be better placed elsewhere in this forum. Thank you,
Dustin

Last edited by The00Dustin; 10-18-2007 at 03:04 PM.
 
Old 10-18-2007, 03:17 PM   #2
MegaSvensk
Member
 
Registered: Sep 2007
Posts: 43

Rep: Reputation: 15
Is the Fedora 64 bit?
 
Old 10-18-2007, 03:44 PM   #3
The00Dustin
Member
 
Registered: Jan 2006
Posts: 70

Original Poster
Rep: Reputation: 15
Yes, it is 64 bit. It is the x86_64 version and there was 8GiB in the machine during the initial install.
 
Old 10-18-2007, 03:57 PM   #4
jay73
LQ Guru
 
Registered: Nov 2006
Location: Belgium
Distribution: Ubuntu 11.04, Debian testing
Posts: 5,019

Rep: Reputation: 133Reputation: 133
If you are using memory mapping, you should set mem to total physical RAM + 2GB. It may also help to disable agp (agp = off).
 
Old 10-18-2007, 08:05 PM   #5
The00Dustin
Member
 
Registered: Jan 2006
Posts: 70

Original Poster
Rep: Reputation: 15
OK, when you say memory mapping, do you mean PAE? If so, I already have it on my agenda to check that in the morning (in /proc/cpuinfo and /boot/<appropriate>.config
Also, is the RAM+2G a standard thing or based on my 2GiB swap partition?
As for the arguments, Fedora itself seems to be working fine, and the problem seems to be with xen. I will look at the xen documentation again in the morning to see if I can figure out where to plug in agp=off (using PCI-Ex, luckily?), but I wanted to make sure you were suggesting this might help the xen kernel detect the memory in some way since the base (smp) kernel already is.
Thanks.

Last edited by The00Dustin; 10-18-2007 at 08:07 PM.
 
Old 10-18-2007, 08:26 PM   #6
jay73
LQ Guru
 
Registered: Nov 2006
Location: Belgium
Distribution: Ubuntu 11.04, Debian testing
Posts: 5,019

Rep: Reputation: 133Reputation: 133
No, memory mapping is not the same PAE, although both have the same task of expanding memory space. PAE is meant to do this for 32 bit kernels (by making them 34 bit if I remember correctly) while memory mapping is a feature in BIOS that is geared towards both 32 and 64 bit . If this feature weren't already enabled in your BIOS, you wouldn't have been able to see 8GB (you would have seen only two). So PAE is not something you should use if you are running a 64 bit kernel.

Part of your problem may be that you are telling your system that you have more RAM than you actually have. As I said, you need 8 GB physical + 2GB= 10GB. The additional 2GB is the effect of memory mapping, which reserves 2GB of the 4GB memory space and maps the remaining physical RAM above it. One side-effect of this is that agp may stop working (in case you wonder, agp does not mean that you have an AGP card - I have PCI-X and you can rest assured that my system will not boot unless I have agp=off).

Whether this will solve your problem, I don't know but I find it curious that your monitor is reporting 3.1GB. That is exactly what would happen if you booted your system with memory mapping disabled - in which case it would revert to 4GB memory space. Subtract the space that the kernel reserves for itself and there would be only about 3GB left for physical RAM.

Last edited by jay73; 10-18-2007 at 08:27 PM.
 
Old 10-18-2007, 08:40 PM   #7
The00Dustin
Member
 
Registered: Jan 2006
Posts: 70

Original Poster
Rep: Reputation: 15
Thank you for the information. All of that makes sense. I was saying I don't have an agp card and that was good because I was assuming that agp=off would render an agp card useless. I think the base (smp) fedora kernel might even count the memory correctly without the mem= argument now that it has detected it once with it. I don't know for sure, but I deleted it using append from grub, and the memory was still right (I was trying to confirm it said 4.0 and not 3.1 before I addded the argument). As for the xen fedora kernel, is it possible that the kernel doesn't have memory mapping enabled even though the bios does and that a custom compile with memory mapping on would do the trick? Would I need to turn anything else on with that? I ask because I will have to use 2.6.20-2936.fc7xen.config as my base config. Thanks,
Dustin
 
Old 10-18-2007, 09:10 PM   #8
The00Dustin
Member
 
Registered: Jan 2006
Posts: 70

Original Poster
Rep: Reputation: 15
I don't know exactly how linux installs work. I do know from experience that if one is to install linux on a machine with sata drives in compatibility mode in the bios and then want to change them to ahci mode he would have to compile a custom kernel to support ahci mode. I also know that if they were in ahci mode before the install the kernel installed would support them. What I don't know is whether there are a few various precompiled kernels on the install disc or whether the kernel is compiled on the fly. I would assume the earlier because otherwise it seems like kernel updates via package managers would be a bigger deal and would assume the update should be for the running custom kernel as opposed to the initially installed kernel. However, all of that aside, in addition to my question regarding kernel compilation and memory mapping, I am wondering if there is any argument I could pass to the installer that might cause it to install and work with 8GiB of RAM out of the box (vs initially needing the mem=10G kernel argument). I ask because if there were such an installer argument, it might resolve my xen problem while maintaining updatability and it might be just as quick and easy as compiling a new kernel anyway since I am not too far along on this project yet (I can log and redo all of my config changes easy enough). Let me know what's feasible of this and my previous post, and I will try what you have already suggested in the meantime (tomorrow). Thanks again.
 
Old 10-18-2007, 11:41 PM   #9
jay73
LQ Guru
 
Registered: Nov 2006
Location: Belgium
Distribution: Ubuntu 11.04, Debian testing
Posts: 5,019

Rep: Reputation: 133Reputation: 133
Well, one doesn't always need to recompile when certain functionality is missing. Quite often one can simply pass a boot argument or load an extra module and that will do the trick just as well.

All I can say is that the development of the fedora kernel is rather uneven. One version will support > 4GB without any special boot arguments and then the next version for some reason does not. I never had any issues running Fedora 6 but I had when I switched to Fedora 7. I have also seen cases where one specific kernel did not recognize mem=6GB while it had no trouble with mem=6144MB. Since you are using different versions, it's quite possible that you're looking at exactly one of those quirks. As for recompiling the kernel, I doubt whether that would make a difference. For a 32 bit system, yes, but a 64 bit kernel should natively support > 4GB. Is it possible that you have xen set up to run 32 bit systems? I seem to remember that Fedora 7 will reduce Dom0 to 32 bit in such a case - or so it did initially (I haven't really kept up with this part to say the least). That might explain why you are seeing the same amount that you would see if you installed a 32 bit kernel.
 
Old 10-19-2007, 09:08 AM   #10
The00Dustin
Member
 
Registered: Jan 2006
Posts: 70

Original Poster
Rep: Reputation: 15
Based on what you said, the first thing I did was check to see if the xen kernel was 32-bit:

#uname -a
Linux moonshine.domain.local 2.6.20-2936.fc7xen #1 SMP Fri Sep 21 11:56:20 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux

I believe that means it's 64 bit. I am still getting the same amount of memory showing, and I tried 10240M instead of 10G on both the kernel xen.gz and module vmlinuz lines of grub.conf. I also tried the previous version of the xen kernel to no avail. This might be a stretch, but perhaps with this output, you can tell me if it could be a kernel configuration problem and whether I might be able to try a module or recompile:

[~]# cat /boot/config-2.6.22.9-91.fc7 | grep -i mem
CONFIG_SHMEM=y
# CONFIG_TINY_SHMEM is not set
CONFIG_ARCH_DISCONTIGMEM_ENABLE=y
CONFIG_ARCH_DISCONTIGMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_SELECT_MEMORY_MODEL=y
# CONFIG_FLATMEM_MANUAL is not set
# CONFIG_DISCONTIGMEM_MANUAL is not set
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
# CONFIG_SPARSEMEM_STATIC is not set
CONFIG_SPARSEMEM_EXTREME=y
# Memory hotplug is currently incompatible with Software Suspend
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_BLK_DEV_UMEM=m
CONFIG_INPUT_FF_MEMLESS=y
CONFIG_INFINIBAND_USER_MEM=y
CONFIG_HAS_IOMEM=y
[~]# cat /boot/config-2.6.20-2936.fc7xen | grep -i mem
CONFIG_SHMEM=y
# CONFIG_TINY_SHMEM is not set
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
# CONFIG_SPARSEMEM_STATIC is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
# Memory Technology Devices (MTD)
CONFIG_BLK_DEV_UMEM=m
CONFIG_INPUT_FF_MEMLESS=y

I am not at all familiar with what these various switches might do, so I could easily be way off, but I am specifically curious as to whether or not the DISCONTIGMEM settings might be necessary (or the FFLATMEM settings a problem) based on how you explained that the memory is mapped.

I initially intended to use CentOS, but it doesn't support the network card in this machine, and while I managed to get it working in CentOS, I couldn't get it working in the xen kernel I had on CentOS (there may have also been some other incompatibilities I have since forgotten). When I found that Fedora 7 not only supported said card out of the box, but also supported states in iptables (in the prepackaged xen kernel, CentOS and Fedora both have for a long time in the standard kernels), I very quickly decided I would be going the Fedora route instead.
 
Old 10-19-2007, 09:39 AM   #11
The00Dustin
Member
 
Registered: Jan 2006
Posts: 70

Original Poster
Rep: Reputation: 15
I snipped this out of a fedora-xen mailing list:
Quote:
[root@yankee ~]# free
total used free shared buffers
cached
Mem: 6286336 1267508 5018828 0 310712
202928
-/+ buffers/cache: 753868 5532468
Swap: 8385920 220 8385700

[root@yankee ~]# uname -a
Linux yankee.domain.net 2.6.20-1.2962.fc6xen #1 SMP Tue Jun 19
19:10:51 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
Looks to me like my problem should be surmountable... The question of what needs done remains, though...

Last edited by The00Dustin; 10-19-2007 at 09:46 AM. Reason: remove domain from 3rd party info
 
Old 10-19-2007, 12:26 PM   #12
The00Dustin
Member
 
Registered: Jan 2006
Posts: 70

Original Poster
Rep: Reputation: 15
This also came out of the fedora-xen mailing list:
Quote:
I have 4 procs with 16G RAM and no mem paramters in grub at all, and it looks like this:

Linux h 2.6.20-2931.fc7xen #1 SMP Mon Aug 13 10:11:56 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux

# grep 'model name' /proc/cpuinfo
model name : AMD Opteron(tm) Processor 852
model name : AMD Opteron(tm) Processor 852
model name : AMD Opteron(tm) Processor 852
model name : AMD Opteron(tm) Processor 852

# grep Total /proc/meminfo
MemTotal: 13335552 kB
SwapTotal: 5261240 kB
VmallocTotal: 34359738367 kB

# xm info |grep total
total_memory : 16319
So that tells me that this should be working though the hardware is admittedly a bit different. Incidentally, I had that kernel on a test system that I hadn't wiped out yet, so I copied it to the quad core system I am working on. I did a diff /boot/config-2.6.20-2931.fc7xen /boot/config-2.6.20-2936.fc7xen and the only difference was the version number and date (both comment lines). I booted to /boot/config-2.6.20-2931.fc7xen and it made no difference (still 3.1GiB). The test machine was a Core 2 Duo with 2GiB of RAM. Both of these machines were installed using the F7 x86_64 DVD and Virtualization was selected on both during initial install. The machine with 8GiB had that during said install.
 
Old 10-19-2007, 05:02 PM   #13
jay73
LQ Guru
 
Registered: Nov 2006
Location: Belgium
Distribution: Ubuntu 11.04, Debian testing
Posts: 5,019

Rep: Reputation: 133Reputation: 133
Have you tried limiting memory just as an experiment by placing, say,

dom0_mem=8192

after the kernel /xen line in grub?

That should give you at least 6GB.

You should also have a look at the xen configuration files in /etc; it's quite possible that Fedora imposes a default memory limit on the host OS (dom0) in order to make sure that there is always enough RAM left for the guest systems.

Last edited by jay73; 10-19-2007 at 05:18 PM.
 
Old 10-19-2007, 08:43 PM   #14
The00Dustin
Member
 
Registered: Jan 2006
Posts: 70

Original Poster
Rep: Reputation: 15
Quote:
"xm info" shows 3327 for total memory
This is the physical memory in the machine as opposed to /proc/meminfo which would show memory allocated to dom0. Ballooning is enabled and functional (I started with dom0 @ 3327 and created a domU which took from that memory). Additionally, I came across this problem when I got an error stating not enough physical memory as I tried to create a domU with 1GiB after having already created a domU with 2GiB and having configured dom0_min_mem to 1024 (a bit high, but planning to run X and 4 domU's with some memory to spare anyway).
Quote:
I also tried dom0_mem=8G on the kernel line, and that caused a kernel panic stating not enough physical memory (got the same thing with dom0_mem=4G).
I couldn't get the exact kernel panic because the machine rebooted, but I believe I saw something to the effect of dom0_mem=-256M somewhere after the not enough physical memory available.
Don't take offense to my quotes, I know my posts are wordy, and quoting was just the quickest easiest way to answer your questions. Thanks.
 
Old 10-19-2007, 09:15 PM   #15
jay73
LQ Guru
 
Registered: Nov 2006
Location: Belgium
Distribution: Ubuntu 11.04, Debian testing
Posts: 5,019

Rep: Reputation: 133Reputation: 133
I can only say that this is a very puzzling case. Have you been able to find any pointers in dmesg or from the kernel logs?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
3.6GB RAM instead of 4GB ??? masterross Slackware 11 09-04-2008 08:07 AM
utilize 4gb of ram cynick Linux - General 2 04-20-2006 11:42 AM
266MHz, 128MB RAM, 4GB HD moo113 Linux - Distributions 4 04-16-2005 10:41 PM
What if you had 2GB RAM and 4GB HD? AC97Conquerer General 21 03-28-2005 07:21 PM
4GB RAM not showing eddie0uk Linux - Hardware 5 03-12-2005 07:08 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 06:43 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration