LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 07-30-2010, 05:57 AM   #16
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197

Quote:
Originally Posted by Raghu140 View Post
cat /proc/slabinfo:
A few things in there were larger than I expected, but nothing was extreme.

When was that run? You seem to have an idea of when the failure is approaching and looking at the slabinfo when the failure is about to happen or starting to happen would be more informative than when the system is healthy.

Quote:
How to see that what is total size of kernel memory availabe?
I'm not sure. I did a web search for info about the file /proc/kcore and everything I found says that file represents a binary image of physical memory. But when I look at that file on various systems, it seems to represent a binary image of kernel virtual memory, so its size is the limit of the size of kernel virtual memory.

So what is the output of
ls -l /proc/kcore

Quote:
How to increase the kernel memory size?
It is a build time option when you recompile the kernel. Do you know how to recompile a kernel?

Quote:
Shall migrating to higher version of RHEL (5.3 etc) will help?
I know nothing about your system and the applications you run. I especially know nothing about the mercd driver that seems to be at the center of your problem.

Do you pay for support for this RHEL system? If you do, you should be asking Red Hat for some support. If you don't you ought to be using Centos instead of RHEL.

Maybe Switching to RHEL or Centos version 5 would help. Maybe what you're seeing is an old bug that was fixed long ago in RHEL itself or in the mercd driver. I don't know any of that stuff.

Is your hardware 64 bit capable? Do you have a good reason for running 32 bit RHEL rather than 64 bit? I think switching to 64 bit is more likely to fix the problem than switching just to version 5.
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 07-31-2010, 04:55 PM   #17
Raghu140
LQ Newbie
 
Registered: Oct 2009
Posts: 20

Original Poster
Rep: Reputation: 0
I have collected the vital stats when problem occured. Please have a look at it. You may able to deduce something out of it:-

Code:
freem
       
              total       used       free     shared    buffers     cached
Mem:          4053       4037         16          0        158       3670
-/+ buffers/cache:        209       3844
Swap:         6000          0       6000

*************************************************************************************
df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb6              20G  619M   18G   4% /
/dev/sdb1              99M   12M   82M  13% /boot
none                  2.0G     0  2.0G   0% /dev/shm
/dev/sdb2              62G   16G   43G  27% /home
/dev/sdb7             9.7G   68M  9.1G   1% /opt
/dev/sdb5              20G  3.0G   16G  17% /usr
/dev/sdb3              20G  268M   18G   2% /var

************************************************************************************
ifconfig

eth2      Link encap:Ethernet  HWaddr 00:30:64:08:C1:A6
          inet addr:10.100.108.69  Bcast:10.100.108.95  Mask:255.255.255.224
          inet6 addr: fe80::230:64ff:fe08:c1a6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1942724 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2163264 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:196009128 (186.9 MiB)  TX bytes:637743127 (608.1 MiB)
          Base address:0xb880 Memory:fda80000-fdaa0000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:5220447 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5220447 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1693155991 (1.5 GiB)  TX bytes:1693155991 (1.5 GiB)
*************************************************************************************

route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.100.108.64   0.0.0.0         255.255.255.224 U     0      0        0 eth2
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth2
0.0.0.0         10.100.108.65   0.0.0.0         UG    0      0        0 eth2
************************************************************************************

iptables -L -n -v

Chain INPUT (policy ACCEPT 10 packets, 2696 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 10 packets, 2696 bytes)
 pkts bytes target     prot opt in     out     source               destination

**************************************************************************************
cat /proc/cpuinfo

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5138  @ 2.13GHz
stepping        : 11
cpu MHz         : 2133.765
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dtss ht tm pbe nx lm constant_tsc pni monitor ds_cpl est tm2 xtpr
bogomips        : 4269.90

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5138  @ 2.13GHz
stepping        : 11
cpu MHz         : 2133.765
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dtss ht tm pbe nx lm constant_tsc pni monitor ds_cpl est tm2 xtpr
bogomips        : 4266.07

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5138  @ 2.13GHz
stepping        : 11
cpu MHz         : 2133.765
cache size      : 4096 KB
physical id     : 3
siblings        : 2
core id         : 6
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dtss ht tm pbe nx lm constant_tsc pni monitor ds_cpl est tm2 xtpr
bogomips        : 4266.13
processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5138  @ 2.13GHz
stepping        : 11
cpu MHz         : 2133.765
cache size      : 4096 KB
physical id     : 3
siblings        : 2
core id         : 7
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dtss ht tm pbe nx lm constant_tsc pni monitor ds_cpl est tm2 xtpr
bogomips        : 4266.14

**************************************************************************************

cat /proc/meminfo

MemTotal:      4151264 kB
MemFree:         16284 kB
Buffers:        161828 kB
Cached:        3757932 kB
SwapCached:          0 kB
Active:        3316776 kB
Inactive:       719684 kB
HighTotal:     3276544 kB
HighFree:         1024 kB
LowTotal:       874720 kB
LowFree:         15260 kB
SwapTotal:     6144820 kB
SwapFree:      6144820 kB
Dirty:            7944 kB
Writeback:           0 kB
Mapped:         143432 kB
Slab:            74596 kB
CommitLimit:   8220452 kB
Committed_AS:  1914288 kB
PageTables:       3556 kB
VmallocTotal:   106488 kB
VmallocUsed:      6420 kB
VmallocChunk:    99316 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

*********************************************************************************************************************

cat /proc/net/dev

Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
    lo:1702919443 5247802    0    0    0     0          0         0 1702919443 5247802    0    0    0     0       0          0
  eth0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  eth1:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  eth2:196945288 1952838    0    0    0     0          0         0 647852030 2177928    0    0    0     0       0          0
  eth3:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  sit0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0

*********************************************************************************************************************

cat /proc/interrupts

           CPU0       CPU1       CPU2       CPU3
  0:   32714438   32721191   32714201   32713570    IO-APIC-edge  timer
  1:          2          3          3          1    IO-APIC-edge  i8042
  8:          0          1          0          0    IO-APIC-edge  rtc
  9:          0          0          0          0   IO-APIC-level  acpi
 12:         17         12         12         17    IO-APIC-edge  i8042
 14:          0          0          0          0    IO-APIC-edge  libata
 15:         25          3          5          4    IO-APIC-edge  ide1
169:          0          0          0          0   IO-APIC-level  uhci_hcd, uhci_hcd
177:          0          0          0          0   IO-APIC-level  uhci_hcd
185:       1822    2524227    2846000       5483   IO-APIC-level  ehci_hcd, uhci_hcd, mercdintr
193:     455854      75408       3196     525536   IO-APIC-level  ioc0
201:     109601      20956       6495     124159   IO-APIC-level  ehci_hcd, uhci_hcd
209:      11773       2448       2324      12701   IO-APIC-level  uhci_hcd, mercdintr
233:    2982459          0          0          0         PCI-MSI  eth2
NMI:          0          0          0          0
LOC:  129134190  129134430  129137856  129137000
ERR:          0
MIS:          0

*************************************************************************************

vmstat

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0      0  16636 161352 3758148    0    0     2    28   23    11  1  1 97  1

*************************************************************************************

cat /proc/devices
Character devices:
  1 mem
  4 /dev/vc/0
  4 tty
  4 ttyS
  5 /dev/tty
  5 /dev/console
  5 /dev/ptmx
  7 vcs
 10 misc
 13 input
 29 fb
 36 netlink
 89 i2c
128 ptm
136 pts
162 raw
180 usb
253 mercd
254 ctimod

Block devices:
  1 ramdisk
  8 sd
  9 md
 22 ide1
 65 sd
 66 sd
 67 sd
 68 sd
 69 sd
70 sd
 71 sd
128 sd
129 sd
130 sd
131 sd
132 sd
133 sd
134 sd
135 sd
253 device-mapper
254 mdp

*************************************************************************************

top - 22:13:42 up 1 day, 12:23,  7 users,  load average: 0.22, 0.20, 0.18
Tasks: 118 total,   1 running, 116 sleeping,   0 stopped,   1 zombie
Cpu(s):  1.1% us,  1.9% sy,  0.0% ni, 96.1% id,  0.8% wa,  0.0% hi,  0.0% si
Mem:   4151264k total,  4135452k used,    15812k free,   161296k buffers
Swap:  6144820k total,        0k used,  6144820k free,  3758984k cached

*************************************************************************************

cat /proc/stat
cpu  408911 0 773353 50217252 330331 2915 0
cpu0 87787 0 184459 12542431 118181 202 0
cpu1 109435 0 214128 12547286 60982 1206 0
cpu2 124588 0 217934 12558696 30879 1274 0
cpu3 87099 0 156830 12568837 120288 231 0
intr 140817575 131081868 9 0 13 7 0 0 0 1 0 6 6 58 0 0 37 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5388483 0 0 0 0 0 0 0 1062535 0 0 0 0 0 0 0 261647 0 0 0 0 0 0 0 29282 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 2993619 0 0 0 0 0
ctxt 134827990
btime 1280479789
processes 4263951
procs_running 1
procs_blocked 0

**************************************************************************************
I took the stats when the problem occured. Free suggests that physical memory is ok. But the telephony card driver is saying unable to allocate memory. I am not able to decide where the issue is? Please help.
 
Old 07-31-2010, 05:57 PM   #18
Raghu140
LQ Newbie
 
Registered: Oct 2009
Posts: 20

Original Poster
Rep: Reputation: 0
Quote:
When was that run? You seem to have an idea of when the failure is approaching and looking at the slabinfo when the failure is about to happen or starting to happen would be more informative than when the system is healthy.
It generally happens 3 days after system restart, when load is about at its peak (30% - normal load for IVR). If i only restart the card driver then it runs for 8-9 hrs then again the problem starts. I will provide slabinfo next time when the problem occurs.Its an voice IVR application.

Code:
ls -lh /proc/kcore
-r-------- 1 root root 897M Jul 31 23:06 /proc/kcore

Quote:
It is a build time option when you recompile the kernel. Do you know how to recompile a kernel?
No. But i can do it. I have been working on linux for past 4 yrs. I am more of application designer/developer with telecom network understanding.

Quote:
Do you pay for support for this RHEL system? If you do, you should be asking Red Hat for some support. If you don't you ought to be using Centos instead of RHEL.
No. we dont. Yup i will consider it before putting such system in production. But the problem is that the current system is at remote loaction and any possibility of hardware/software change is very difficult.So my entire priority is in identify the root cause.

I have faced this issue earlier as well. But last time it was running RHEL 4.3 and my vender suggested to upgarde the O.S to RHEL 4.5. I did and we also replaced the chassis(server). The problem was resolved. But now we have RHEL 4.5 and we are again facing the issue. I am looking more for root cause to kill this issue once for all.


Quote:
Is your hardware 64 bit capable? Do you have a good reason for running 32 bit RHEL rather than 64 bit? I think switching to 64 bit is more likely to fix the problem than switching just to version 5.
I dont think so. Will consider 64 bit for now onwards.


I have also take then similar logs of system as posted earlier. But this time the system was working fine after the card driver restart. this is just for comparsion purpose. See if u can find anything:-

Code:
free -m
             total       used       free     shared    buffers     cached
Mem:          4053       4038         15          0        152       3685
-/+ buffers/cache:        200       3853
Swap:         6000          0       6000
**************************************************************************************
df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb6              20G  619M   18G   4% /
/dev/sdb1              99M   12M   82M  13% /boot
none                  2.0G     0  2.0G   0% /dev/shm
/dev/sdb2              62G   16G   43G  28% /home
/dev/sdb7             9.7G   68M  9.1G   1% /opt
/dev/sdb5              20G  3.0G   16G  17% /usr
/dev/sdb3              20G  267M   18G   2% /var

**************************************************************************************

ifconfig
eth2      Link encap:Ethernet  HWaddr 00:30:64:08:C1:A6
          inet addr:10.100.108.69  Bcast:10.100.108.95  Mask:255.255.255.224
          inet6 addr: fe80::230:64ff:fe08:c1a6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2014749 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2269242 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:202452852 (193.0 MiB)  TX bytes:716216325 (683.0 MiB)
          Base address:0xb880 Memory:fda80000-fdaa0000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:5478564 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5478564 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1769831977 (1.6 GiB)  TX bytes:1769831977 (1.6 GiB)

**************************************************************************************

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.100.108.64   0.0.0.0         255.255.255.224 U     0      0        0 eth2
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth2
0.0.0.0         10.100.108.65   0.0.0.0         UG    0      0        0 eth2

**************************************************************************************
iptables -L -n -v

Chain INPUT (policy ACCEPT 323K packets, 79M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 356K packets, 149M bytes)
 pkts bytes target     prot opt in     out     source               destination

**************************************************************************************

cat /proc/cpuinfo

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5138  @ 2.13GHz
stepping        : 11
cpu MHz         : 2133.765
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl est tm2 xtpr
bogomips        : 4269.90

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5138  @ 2.13GHz
stepping        : 11
cpu MHz         : 2133.765
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl est tm2 xtpr
bogomips        : 4266.07

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5138  @ 2.13GHz
stepping        : 11
cpu MHz         : 2133.765
cache size      : 4096 KB
physical id     : 3
siblings        : 2
core id         : 6
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl est tm2 xtpr
bogomips        : 4266.13

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5138  @ 2.13GHz
stepping        : 11
cpu MHz         : 2133.765
cache size      : 4096 KB
physical id     : 3
siblings        : 2
core id         : 7
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl est tm2 xtpr
bogomips        : 4266.14

****************************************************************************************************************************

cat /proc/meminfo

MemTotal:      4151264 kB
MemFree:         15196 kB
Buffers:        155624 kB
Cached:        3775052 kB
SwapCached:          4 kB
Active:        3262304 kB
Inactive:       776124 kB
HighTotal:     3276544 kB
HighFree:         1024 kB
LowTotal:       874720 kB
LowFree:         14172 kB
SwapTotal:     6144820 kB
SwapFree:      6144816 kB
Dirty:            1812 kB
Writeback:           0 kB
Mapped:         133872 kB
Slab:            73616 kB
CommitLimit:   8220452 kB
Committed_AS:  1907636 kB
PageTables:       3496 kB
VmallocTotal:   106488 kB
VmallocUsed:      6420 kB
VmallocChunk:    99316 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

*************************************************************************************************************************

cat /proc/net/dev

Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
    lo:1772920337 5488333    0    0    0     0          0         0 1772920337 5488333    0    0    0     0       0          0
  eth0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  eth1:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  eth2:202792040 2018629    0    0    0     0          0         0 720153624 2274779    0    0    0     0       0          0
  eth3:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  sit0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0

*************************************************************************************************************************

cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  0:   33404688   33413743   33414438   33407356    IO-APIC-edge  timer
  1:          2          3          3          1    IO-APIC-edge  i8042
  8:          0          1          0          0    IO-APIC-edge  rtc
  9:          0          0          0          0   IO-APIC-level  acpi
 12:         17         12         12         17    IO-APIC-edge  i8042
 14:          0          0          0          0    IO-APIC-edge  libata
 15:         25          3          5          4    IO-APIC-edge  ide1
169:          0          0          0          0   IO-APIC-level  uhci_hcd, uhci_hcd
177:          0          0          0          0   IO-APIC-level  uhci_hcd
185:       1838    2579661    2905017       6736   IO-APIC-level  ehci_hcd, uhci_hcd, mercdintr
193:     466468      76480       3226     537463   IO-APIC-level  ioc0
201:     111765      21576       6855     126563   IO-APIC-level  ehci_hcd, uhci_hcd
209:      12700       2699       2822      13971   IO-APIC-level  uhci_hcd, mercdintr
233:    3072471          0          0          0         PCI-MSI  eth2
NMI:          0          0          0          0
LOC:  131873870  131874041  131877241  131876386
ERR:          0
MIS:          0

****************************************************************************************************************************

vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0      4  16292 155228 3774408    0    0     2    28   28    17  1  1 97  1

***************************************************************************************************************************

cat /proc/devices
Character devices:
  1 mem
  4 /dev/vc/0
  4 tty
  4 ttyS
  5 /dev/tty
  5 /dev/console
  5 /dev/ptmx
  7 vcs
 10 misc
 13 input
 29 fb
 36 netlink
 89 i2c
128 ptm
136 pts
162 raw
180 usb
253 mercd
254 ctimod

Block devices:
  1 ramdisk
  8 sd
  9 md
 22 ide1
65 sd
 66 sd
 67 sd
 68 sd
 69 sd
 70 sd
 71 sd
128 sd
129 sd
130 sd
131 sd
132 sd
133 sd
134 sd
135 sd
253 device-mapper
254 mdp

*************************************************************************************************************************
Pls notice the Bolded words.

Code:
vm.percpu_pagelist_fraction = 0
vm.max_queue_depth = 0
vm.oom-kill = 1
vm.legacy_va_layout = 0
vm.vfs_cache_pressure = 100
vm.block_dump = 0
vm.laptop_mode = 0
vm.max_map_count = 65536
vm.min_free_kbytes = 949
vm.lower_zone_protection = 0
vm.hugetlb_shm_group = 0
vm.nr_hugepages = 0
vm.swappiness = 60
vm.nr_pdflush_threads = 2
vm.dirty_expire_centisecs = 3000
vm.dirty_writeback_centisecs = 500
vm.dirty_ratio = 30
vm.dirty_background_ratio = 7
vm.page-cluster = 3
vm.overcommit_ratio = 50
vm.overcommit_memory = 0
I was considering to vm.overcommit_memory = 2. I read in Redhat optimazation that it increases the ram availablity to the system. I dont know wheather it will help or not.
 
Old 08-01-2010, 02:58 PM   #19
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 4,070

Rep: Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897
Quote:
Originally Posted by Raghu140 View Post
I have faced this issue earlier as well. But last time it was running RHEL 4.3 and my vender suggested to upgarde the O.S to RHEL 4.5. I did and we also replaced the chassis(server). The problem was resolved.
Also consider the possibility that you didn't so much resolve the problem as delay its occurrence.

Quote:
I have collected the vital stats when problem occured
I don't believe that you recorded this at exactly the time that the problem started; was it just before or just after the problem actually started?

Quote:
Code:
vmstat

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0      0  16636 161352 3758148    0    0     2    28   23    11  1  1 97  1
I'm not sure that the output of vmstat is helping much but, if vmstat were to help, you'd have to do something other than this. The first line of vmstat probably only deceives about what is currently going on, so you need the multi-line output.

Quote:
Code:
top - 22:13:42 up 1 day, 12:23,  7 users,  load average: 0.22, 0.20, 0.18
Tasks: 118 total,   1 running, 116 sleeping,   0 stopped,   1 zombie
Cpu(s):  1.1% us,  1.9% sy,  0.0% ni, 96.1% id,  0.8% wa,  0.0% hi,  0.0% si
Mem:   4151264k total,  4135452k used,    15812k free,   161296k buffers
Swap:  6144820k total,        0k used,  6144820k free,  3758984k cached
Do you know what that zombie process was? not the driver for your telephony card, by any chance?

At this point, I have a suspicion that there is simply a bug, or maybe an incompatibility in the card driver (did you install it from a repo, did you build it yourself from a tarball or something else?), but I have no idea how to proceed further without more information.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Tutorial: Understanding OSPF Routing (part 2) LXer Syndicated Linux News 0 10-30-2008 04:40 AM
LXer: Tutorial: Networking 101: Understanding (and Using) ICMP LXer Syndicated Linux News 0 09-03-2008 10:00 PM
LXer: Tutorial: Networking 101: Understanding IP Addresses LXer Syndicated Linux News 0 07-23-2008 12:50 AM
understanding top gianh Linux - Software 4 07-25-2007 08:30 PM
Understanding the "Top" Command gauravdamri Linux - Newbie 1 02-07-2006 03:25 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 04:55 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration