segmentation fault error

dianarani · 02-18-2008, 05:34 AM

Hello,
I am using linux(kernel 2.6.9-5.0.3.EL).Unfortunately 2 days before all the cronjob was stopped and i am getting segmentation fault error,while starting and finding the status of all the services.
ERROR: service crond status
/sbin/service: line 68: 28349 Segmentation fault env -i LANG=$LANG PATH=$PATH TERM=$TERM "${SERVICEDIR}/${SERVICE}" ${OPTIONS}

while using mv command also i am getting the segmentation fault error.

Could anyone explain about this error?
I searched on this error and found it may be hardware error due to bad RAM memory.so now i have to run some diagnostic test to figure out the exact cause of problem.I am heard of memtest,but don't know how to run that in linux.... .Please help me...to find out some easy linux package or tool to test RAM.
I also tried a script in this link http://people.redhat.com/dledford/memtest.html.It took long time for me and also it make my server load to exceed threshold limit.So i gave off this script.Is there anyone get positive result using this script? Please help me to sort out the problem and to overcome that.

Did anyone get overcome this segmentation error?

Thanks,

unSpawn · 02-18-2008, 06:41 AM

Quote:

Originally Posted by dianarani

2 days before all the cronjob was stopped and i am getting segmentation fault error,

There's two things you should do:
- verify the integrity of your packages: 'rpm -qVa' and
- check your system logs for package changes, any errors, anything out of the ordinary.

Quote:

Originally Posted by dianarani

while using mv command also i am getting the segmentation fault error.

What does 'ldd /bin/mv' return?
What does 'touch /tmp/$$; strace -v /bin/mv -f /tmp/$$ /tmp/$$.mv' return?

dianarani · 02-18-2008, 07:10 AM

Thanks a lot for your fast reply and advise unspawn.

for ldd /bin/mv, i got output as,
libacl.so.1 => /lib/libacl.so.1 (0x0051b000)
libselinux.so.1 => /lib/libselinux.so.1 (0x00bcc000)
libc.so.6 => /lib/tls/libc.so.6 (0x00889000)
libattr.so.1 => /lib/libattr.so.1 (0x00bc2000)
/lib/ld-linux.so.2 (0x00870000)

and for this.,touch /tmp/$$; strace -v /bin/mv -f /tmp/$$ /tmp/$$.mv

brk(0) = 0x9f69000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_dev=makedev(3, 3), st_ino=3851518, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=120, st_size=49540, st_atime=2008/02/18-04:58:08, st_mtime=2006/11/22-05:37:15, st_ctime=2007/01/04-08:27:06}) = 0
old_mmap(NULL, 49540, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7ff3000
close(3) = 0
open("/lib/libacl.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\300\302"..., 512) = 512
fstat64(3, {st_dev=makedev(3, 3), st_ino=5079042, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=56, st_size=23572, st_atime=2008/02/18-04:58:08, st_mtime=2005/02/21-08:58:48, st_ctime=2006/11/22-04:05:31}) = 0
old_mmap(0x51b000, 20948, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x51b000
old_mmap(0x520000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x520000
close(3) = 0
open("/lib/libselinux.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\240\340"..., 512) = 512
fstat64(3, {st_dev=makedev(3, 3), st_ino=5080234, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=128, st_size=56320, st_atime=2008/02/18-04:58:08, st_mtime=2005/02/21-11:57:14, st_ctime=2006/11/19-13:23:38}) = 0
old_mmap(0xbcc000, 56172, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xbcc000
old_mmap(0xbd9000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd000) = 0xbd9000
close(3) = 0
open("/lib/tls/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0 \337\211"..., 512) = 512
fstat64(3, {st_dev=makedev(3, 3), st_ino=5080225, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=2392, st_size=1212936, st_atime=2008/02/18-04:58:08, st_mtime=2005/02/22-08:55:50, st_ctime=2006/11/19-13:23:34}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7ff2000
old_mmap(0x889000, 1215644, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x889000
old_mmap(0x9ac000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x123000) = 0x9ac000
old_mmap(0x9b0000, 7324, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x9b0000
close(3) = 0
open("/lib/libattr.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0000+\274"..., 512) = 512
fstat64(3, {st_dev=makedev(3, 3), st_ino=5080238, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=64, st_size=26343, st_atime=2008/02/18-04:58:08, st_mtime=2005/02/21-08:26:59, st_ctime=2006/11/19-13:23:43}) = 0
old_mmap(0xbc2000, 13504, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xbc2000
old_mmap(0xbc5000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0xbc5000
close(3) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7ff1000
mprotect(0x9ac000, 4096, PROT_READ) = 0
mprotect(0x885000, 4096, PROT_READ) = 0
set_thread_area({entry_number:-1 -> 6, base_addr:0xb7ff16c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
munmap(0xb7ff3000, 49540) = 0
access("/etc/selinux/", F_OK) = 0
brk(0) = 0x9f69000
brk(0x9f8a000) = 0x9f8a000
open("/etc/selinux/config", O_RDONLY) = 3
fstat64(3, {st_dev=makedev(3, 3), st_ino=3851492, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=447, st_atime=2008/02/18-04:25:31, st_mtime=2007/07/03-03:46:02, st_ctime=2007/07/03-03:46:02}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fff000
read(3, "# This file controls the state o"..., 4096) = 447
close(3) = 0
munmap(0xb7fff000, 4096) = 0
open("/proc/mounts", O_RDONLY) = 3
fstat64(3, {st_dev=makedev(0, 3), st_ino=967639056, st_mode=S_IFREG|0444, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=0, st_size=0, st_atime=2008/02/18-04:58:08, st_mtime=2008/02/18-04:58:08, st_ctime=2008/02/18-04:58:08}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fff000
read(3, "rootfs / rootfs rw 0 0\n/proc /pr"..., 1024) = 440
read(3, "", 1024) = 0
close(3) = 0
munmap(0xb7fff000, 4096) = 0
+++ killed by SIGSEGV +++

I hope the problem may be with the lower part of the output that is with opening /proc/filesystems and /usr/lib/locale/locale-archive.

Please explain me about these commands and help me more to get solution for this error...

dianarani · 02-18-2008, 07:13 AM

sorry it was not clear on my previous post.,

Thanks a lot for your fast reply and advise unspawn.

for ldd /bin/mv, i got output as,

libacl.so.1 => /lib/libacl.so.1 (0x0051b000)
libselinux.so.1 => /lib/libselinux.so.1 (0x00bcc000)
libc.so.6 => /lib/tls/libc.so.6 (0x00889000)
libattr.so.1 => /lib/libattr.so.1 (0x00bc2000)
/lib/ld-linux.so.2 (0x00870000)

and for this.,touch /tmp/$$; strace -v /bin/mv -f /tmp/$$ /tmp/$$.mv

brk(0) = 0x9f69000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_dev=makedev(3, 3), st_ino=3851518, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=120, st_size=49540, st_atime=2008/02/18-04:58:08, st_mtime=2006/11/22-05:37:15, st_ctime=2007/01/04-08:27:06}) = 0
old_mmap(NULL, 49540, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7ff3000
close(3) = 0
open("/lib/libacl.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\300\302"..., 512) = 512
fstat64(3, {st_dev=makedev(3, 3), st_ino=5079042, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=56, st_size=23572, st_atime=2008/02/18-04:58:08, st_mtime=2005/02/21-08:58:48, st_ctime=2006/11/22-04:05:31}) = 0
old_mmap(0x51b000, 20948, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x51b000
old_mmap(0x520000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x520000
close(3) = 0
open("/lib/libselinux.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\240\340"..., 512) = 512
fstat64(3, {st_dev=makedev(3, 3), st_ino=5080234, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=128, st_size=56320, st_atime=2008/02/18-04:58:08, st_mtime=2005/02/21-11:57:14, st_ctime=2006/11/19-13:23:38}) = 0
old_mmap(0xbcc000, 56172, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xbcc000
old_mmap(0xbd9000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd000) = 0xbd9000
close(3) = 0
open("/lib/tls/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0 \337\211"..., 512) = 512
fstat64(3, {st_dev=makedev(3, 3), st_ino=5080225, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=2392, st_size=1212936, st_atime=2008/02/18-04:58:08, st_mtime=2005/02/22-08:55:50, st_ctime=2006/11/19-13:23:34}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7ff2000
old_mmap(0x889000, 1215644, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x889000
old_mmap(0x9ac000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x123000) = 0x9ac000
old_mmap(0x9b0000, 7324, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x9b0000
close(3) = 0
open("/lib/libattr.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0000+\274"..., 512) = 512
fstat64(3, {st_dev=makedev(3, 3), st_ino=5080238, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=64, st_size=26343, st_atime=2008/02/18-04:58:08, st_mtime=2005/02/21-08:26:59, st_ctime=2006/11/19-13:23:43}) = 0
old_mmap(0xbc2000, 13504, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xbc2000
old_mmap(0xbc5000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0xbc5000
close(3) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7ff1000
mprotect(0x9ac000, 4096, PROT_READ) = 0
mprotect(0x885000, 4096, PROT_READ) = 0
set_thread_area({entry_number:-1 -> 6, base_addr:0xb7ff16c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
munmap(0xb7ff3000, 49540) = 0
access("/etc/selinux/", F_OK) = 0
brk(0) = 0x9f69000
brk(0x9f8a000) = 0x9f8a000
open("/etc/selinux/config", O_RDONLY) = 3
fstat64(3, {st_dev=makedev(3, 3), st_ino=3851492, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=447, st_atime=2008/02/18-04:25:31, st_mtime=2007/07/03-03:46:02, st_ctime=2007/07/03-03:46:02}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fff000
read(3, "# This file controls the state o"..., 4096) = 447
close(3) = 0
munmap(0xb7fff000, 4096) = 0
open("/proc/mounts", O_RDONLY) = 3
fstat64(3, {st_dev=makedev(0, 3), st_ino=967639056, st_mode=S_IFREG|0444, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=0, st_size=0, st_atime=2008/02/18-04:58:08, st_mtime=2008/02/18-04:58:08, st_ctime=2008/02/18-04:58:08}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fff000
read(3, "rootfs / rootfs rw 0 0\n/proc /pr"..., 1024) = 440
read(3, "", 1024) = 0
close(3) = 0
munmap(0xb7fff000, 4096) = 0
+++ killed by SIGSEGV +++

I hope the problem may be with the lower part of the output that is with opening /proc/filesystems and /usr/lib/locale/locale-archive.

Please explain me about these commands and help me more to get solution for this error...

unSpawn · 02-18-2008, 07:34 AM

Can you confirm you posted the whole strace because I don't see no "/proc/filesystems" or "/usr/lib/locale/locale-archive" in your output? BTW, please also pay attention to the first two things to check (integrity, logs). Any other details you may (not) think related to any of this would be appreciated.

dianarani · 02-18-2008, 07:44 AM

yes,i pasted the whole strace.when i ran strace on another server and compare, i found /usr/lib/locale/locale-archive and /proc/filesystem were missing in the output.I suspect it may be the problem.I checked the log and found nothing abnormal.
while verifying initscripts, i got reply as follows
.......T /bin/doexec
prelink: undo.c:699: prelink_undo: Assertion `dso->ehdr.e_entry == ehdr.e_entry' failed.
S.?....T /bin/ipcalc
.......T /bin/usleep
..5....T c /etc/inittab
S.5....T c /etc/rc.d/rc.local
.......T /sbin/consoletype
.......T /sbin/genhostid
.......T /sbin/getkey
.......T c /sbin/ifdown
.......T c /sbin/ifup
.......T /sbin/initlog
.......T /sbin/kmodule
.......T /sbin/minilogd
.......T /sbin/netreport
.......T /sbin/ppp-watch
.......T /sbin/redhat-support-check
.......T /sbin/service
.......T /sbin/setsysfont

Is it this initscripts(package) creating problem? Please help me to proceed more.....

dianarani · 02-18-2008, 10:17 PM

While verifying initscripts package using rpm -qVa initscripts i got the following failed message,
prelink: undo.c:699: prelink_undo: Assertion `dso->ehdr.e_entry == ehdr.e_entry' failed.

Is there any wrong on initscripts package? Please advice me on this...

dianarani · 02-19-2008, 12:13 AM

Is there any package(rpm) related to all services? Shall i solve this problem by reinstalling that package?

unSpawn · 02-19-2008, 09:55 AM

A segfault happens when an application tries to access a part of memory it shouldn't. If it happens to a single application it might be a programming error. If it suddenly happens to the whole system there's something else going on. Could be anything from missing dependency to partial upgrade to malicious activity. Unfortunately the strace and verify output don't give me a clue what's happening (and I don't get the prelink_undo host entry point error, does this happen with all packages you verified? Or did you only verify init scripts?). Let's focus a bit on what happened on the system up to the point you first noticed the segfaults. Look through your logfiles for package removal, installations and upgrades and anything else that looks "weird". While you're at it you might as well check for "weird" processes or unusual user activity around the time. BTW, if you're running RHEL-4 your kernel seems from 2005. Shouldn't you be running kernel 2.6.9-67.0.4.EL?

dianarani · 02-22-2008, 05:50 AM

Thanks a lot unspawn.On upgrading initscript rpm, i got my crond running as normal.
Even now i din't get some command(mv,mkdir,chown) working.these command getting(segmentation fault error).i have to check all the logs some more deep.
incidentally i noticed the segfaults by checking the cron as it stopped rsync the data,so while checking that i found this error.

One more problem i got now was none of the cron process end.that while i check the status of crond i got too many process as running.
Please say if you get any ideas on this...

unSpawn · 02-22-2008, 07:46 AM

I'll wait until you have conducted the rest of your checks. BTW, is this the same machine as the one you had problems with in your other threads?

dianarani · 02-23-2008, 12:10 AM

yes unspawn.

Electro · 02-23-2008, 02:01 AM

Probably you got a breach or someone installed a root kit. If you are not running AIDE or any file monitor, I suggest note your back ups as contaminated and do a complete re-install. If you are running a file monitor, probably it is hardware related such as not using ECC memory, hard drive going, power supply age.

ganeshbatwal · 02-27-2008, 05:28 AM

Dear all,
Here I 'm also facing the same type of problem in our Production Setup ,
I 'm not able to execute any commands .eg. ls -ltr , grep ,vi ...etc ,
Can any body provide the Solution for the Segmentation Fault Error .
How to resolve it ,
Is is possiable to recover the OS through Resecue CD's ?
Pls update ..

unSpawn · 02-27-2008, 06:44 AM

I don't know if this is a language barrier thing but you've not been reporting back what we need to see. Also you've not been updating this issue continuously, which leads me to believe you don't see the urgency of it. I agree it's time to boot a Live CD and do proper audting before doing anything else. Here's a list of checks you want to perform next to 'rpm -qVa': Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html.