LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 09-30-2013, 06:49 PM   #1
sysbox
Member
 
Registered: Jul 2005
Posts: 117

Rep: Reputation: 15
/dev/shm not behaving right


I have CentOS 6.4 64-bit PC with 24 GB of memory. I typically use /dev/shm extensively for large temporary files. However, about a month ago, it started behaving wierd. When I copy large files to /dev/shm, the PC slows down and starts swapping to disk. If I continue copying large file to dev/shm, the machine would reboot.

I am trying to figure out what causes this problem, and how to fix it.

When this problem occurs, 'cat /proc/meminfo/' says MemFree is at almost zero. So, it looks like I'm running out of memory. But I'm not running any large memory processes.

When the machine reboots, MemFree shows almost 24 GB initially, but gradually goes to zero, and the 'Cached' entry increases towards 24 GB as I copy or create large files. So, it looks like the memory is being consumed by the disk cache.

But these problems all occurred suddenly about a month ago. This was right about the same time I configured a software RAID of 4 hard drives. So it's possible I changed something, but I don't remember.

Anyway, how can I figure out why this problem is occurring, or how can I figure out how to fix it. Does anyone have any ideas?
 
Old 10-01-2013, 12:02 PM   #2
Keith Hedger
Senior Member
 
Registered: Jun 2010
Location: Wiltshire, UK
Distribution: Linux From Scratch, Slackware64, Partedmagic
Posts: 2,482

Rep: Reputation: 622Reputation: 622Reputation: 622Reputation: 622Reputation: 622Reputation: 622
/dev/shm is a RAM disk so of course if you copy large files to it you will run out of memory.
 
Old 10-01-2013, 12:28 PM   #3
sysbox
Member
 
Registered: Jul 2005
Posts: 117

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Keith Hedger View Post
/dev/shm is a RAM disk so of course if you copy large files to it you will run out of memory.
Yes, I know that. The problem is that I can't copy large files to /dev/shm. For example, my /dev/shm is 18 GB in size, and the PC has 24 GB of physical memory. I should be able to copy 18 GB of files to /dev/shm. I used to be able to do that until about a month ago. Now, if I copy a single 1 GB file to /dev/shm, the machine starts swapping and crashes long before the copy is complete.

I'm trying to figure out what is causing the problem. What changed and how can I unchange it?
 
Old 10-01-2013, 05:38 PM   #4
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,714

Rep: Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280
Is /dev/shm a tmpfs mount?

If it is (which I think is likely) your system is subject to deadlock crashes.

This is especially likely if you don't have enough swap space.

With 24 GB of memory, you can only use 12 GB for a tmpfs IF and ONLY IF other users are not also using more than 12 GB of main memory (which also includes any shared memory segments, as that is what /dev/shm is supposed to be used for).

It gets even worse if you have two tmpfs mounts... each one gets a default of 12GB, so if you have /tmp mounted for tmpfs, they can trivially crash the system just by using up a lot of /tmp when you start copying those large files.

BTW, tmpfs mounts are nothing but cache memory. It can be paged out to swap, but only if there is a swap file available.

Technically, what you are doing is not what tmpfs was designed for, and as you found out, it doesn't do it very well...

It is possible the software raid is using a good bit of cache itself - it does have to compute parity (assuming raid 5), and such buffers are not shareable, so they would be allocated for use by the driver itself.

Last edited by jpollard; 10-01-2013 at 05:42 PM.
 
Old 10-02-2013, 09:21 AM   #5
sysbox
Member
 
Registered: Jul 2005
Posts: 117

Original Poster
Rep: Reputation: 15
Yes, /dev/shm is a tmpfs mount. I remount /dev/shm so it has 18 GB of space. There are no other users on this system. My software raid is RAID-0, so no parity bits.

Everything worked fine until a month ago. Then the same things I've been doing for 3 years stop working. Basically, even though /dev/shm has 18 GB of free space, the system crashes after I copy a single 1-GB file. So something changed. Do you know where I can look to find the answer? This is probably a deeper issue than most newbie questions.
 
Old 10-02-2013, 05:36 PM   #6
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,714

Rep: Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280
Even raid 0 has extra buffer usage - at least one per disk, and then there is read-ahead buffers, plus what ever is using the raid storage. Raid 0 provides faster I/O by spreading the data to different disks - which would put more pressure on the cache memory.

In an idle system though, things should still work.
 
Old 10-02-2013, 06:04 PM   #7
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,361
Blog Entries: 55

Rep: Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547
Quote:
Originally Posted by sysbox View Post
Everything worked fine until a month ago.
Then assess what happened a month ago in terms of (re-)configuration, software updates, etc, etc?
 
Old 10-03-2013, 08:00 AM   #8
sysbox
Member
 
Registered: Jul 2005
Posts: 117

Original Poster
Rep: Reputation: 15
A month ago I configured a Software RAID-0. It went fairly smoothly so I didn't change many things. But I can't remember for sure. I may have had to change something. However, there are no new files in the /etc directory tree so it's difficult to know what changed.
 
Old 10-03-2013, 08:05 AM   #9
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,714

Rep: Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280
You could always try disabling the raid (start with just not mounting it).
 
Old 10-03-2013, 12:06 PM   #10
sysbox
Member
 
Registered: Jul 2005
Posts: 117

Original Poster
Rep: Reputation: 15
Already tried that. I rebooted the system and never mounted the RAID, but the problem persists.
 
Old 10-03-2013, 06:09 PM   #11
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,714

Rep: Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280
Only thing left to try is to unload the raid software.
 
Old 10-04-2013, 09:23 AM   #12
sysbox
Member
 
Registered: Jul 2005
Posts: 117

Original Poster
Rep: Reputation: 15
I don't think I loaded any RAID software. mdadm is installed by default on the system.

Do you have suggestions on where else I can ask these question?

Thanks
 
Old 10-04-2013, 09:47 AM   #13
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,714

Rep: Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280
You can try the CentOS forums at http://www.centos.org/modules/newbb/

I'm not sure how much help they can be, but something has changed in your system (possibly due to updates) that has changed how memory is being utilized.

As I said before, tmpfs was never designed for what you are using it for - it can cause system deadlocks very easily and there is no way to prevent it other than limiting the size of tmpfs mounts.
 
Old 10-04-2013, 12:13 PM   #14
sysbox
Member
 
Registered: Jul 2005
Posts: 117

Original Poster
Rep: Reputation: 15
I have not done any updates sinec I initially installed the OS several months ago.

Was tmpfs not designed for this, or was /dev/shm not designed for this? Is there another way for me to utilize a large 18 GB RAMdisk filesystem? What's the proper way to do RAMdisk filesystems in Linux?
 
Old 10-04-2013, 02:18 PM   #15
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,714

Rep: Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280Reputation: 1280
It has nothing to do with /dev/shm - as the mount point for a tmpfs. /dev/shm is used to record files that describe shared memory segments. As such, using tmpfs for that never uses that much (just overhead for directories) the file sizes in the shm are the shared by all the processes connected to the segment. As shared memory segments are deallocated, these "files" disappear.

On my system (with two logins + gdm) there are 7 entries for shared memory segments (65 MB each) used by pulse audio, and one 1MB for a bunch of semaphores. Since these do occupy memory they will add to the usage of shm.

One possiblity is that these hang around longer than they should (pulse audio is not known (to me anyway) to be that well behaved about deallocations...). You can try deleting those that are not active (check with fuser).

The original ramdisk is being depreciated due to its internal limits - a fixed maximum size, non-pageable - and some kernels may not have it available (my fedora system doesn't even have the driver anymore). So putting an 18GB file in a tmpfs mount is not guaranteed to be in memory anyway - it would be paged out to disk as other usage of memory occurs.

I'm not even sure there IS a way - safely anyway. The only time I personally have used a tmpfs was for semaphore file - 0 length, quick for usage as file semaphores, and simple (even shell scripts can do that kind of locking).

Last edited by jpollard; 10-04-2013 at 02:24 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Terminal Emulation & /dev/pts and /dev/shm Issues on Kernel 3.4.3 unassailable Gentoo 8 10-28-2012 12:03 AM
differences betwne /dev/shm and kernel.shm hallask Linux - Server 2 09-23-2011 07:26 AM
[SOLVED] Issues with /dev/pts & /dev/shm, when LFS boots zer0signal Linux From Scratch 2 12-21-2010 08:14 AM
/dev/shm not mounted /dev busy compgenius999 Linux - Newbie 1 03-03-2010 12:12 PM
Sizes of /dev /dev/shm /lib/init/rw jmoschetti45 Linux - Server 0 01-25-2010 12:20 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 09:55 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration