weird problem - terminal not refreshing; sleep not terminating

vikas027 · 07-13-2009, 01:36 AM

Hi,

I am having a wierd problem in one of the Solaris server.

Code:

root@ussd # uname -a
SunOS ussd 5.8 Generic_108528-24 sun4u sparc SUNW,Sun-Fire-280R

The problem is that terminal is not refreshing when we run commands like prstat, tail -f "some log file", sar -u 1 10.

Also, sleep command is not terminating normally. I have to press ctrl+C or ctrl+Z to terminate it.

Code:

root@ussd # sleep 2
^C
root@ussd #

root@ussd # sleep 5
^Z
[1]+  Stopped                 sleep 5
root@ussd #

root@ussd # ps -aef | grep sleep
    root 22218     1  0 11:32:57 ?        0:00 sleep 2
    root 22311 21950  0 11:54:31 pts/1    0:00 sleep 5

Any suggestions, I am unable to troubleshoot this problem.

Please HELP !!!

jlliagre · 07-13-2009, 04:44 AM

You seem to have a suspended "sleep 2" process. Are you sure it is the one you just launched ?
What says "jobs" ?

vikas027 · 07-13-2009, 10:39 PM

Quote:

Originally Posted by jlliagre

You seem to have a suspended "sleep 2" process. Are you sure it is the one you just launched ?
What says "jobs" ?

No, I did not suspended the "sleep 2" proccess. It is that the prompt was not coming back after 2 seconds, so I have no other option than to forcefully break it.

And jobs has no output.

Code:

root@ussd # sleep 1
^Croot@ussd # jobs
root@ussd #

catkin · 07-14-2009, 12:14 AM

Is sleep aliased? Is it a builtin, in the shell you are using? If so does the sleep executable work as expected?

vikas027 · 07-14-2009, 01:43 AM

Quote:

Originally Posted by catkin

Is sleep aliased? Is it a builtin, in the shell you are using? If so does the sleep executable work as expected?

No, sleep is not aliased. It is builtin.

The problem is that terminal is not refreshing when we run commands like prstat, tail -f "some log file", sar -u 1 10

Even, crontab -e is not working.

catkin · 07-14-2009, 02:17 AM

No answers but maybe thinking out loud will help ...

The sleep you killed turned into a zombie. Normally (?) that means it had some output and nowhere to send it but sleep does no output. Does that mean it had something to communicate back to the process of the shell and was unable to do so?

What sort of terminal are you using? What is its device? What happens if you echo foo > <device>?

Are you able to try similar from some other terminal? The console itself?

jlliagre · 07-14-2009, 02:21 AM

That looks very strange.

What says:

Code:

ptree $$
truss -t !mmap,munmap -f sleep 1

?

vikas027 · 07-14-2009, 03:16 AM

Quote:

Originally Posted by catkin

What sort of terminal are you using? What is its device?

I have tried setting xterm and editor too. Here are some logs, that might help you out.

Code:

root@ussd-gw # stty -a
speed 38400 baud;
rows = 42; columns = 125; ypixels = 0; xpixels = 0;
csdata ?
eucw 1:0:0:0, scrw 1:0:0:0
intr = ^c; quit = ^\; erase = ^?; kill = ^u;
eof = ^d; eol = <undef>; eol2 = <undef>; swtch = <undef>;
start = ^q; stop = ^s; susp = ^z; dsusp = ^y;
rprnt = ^r; flush = ^o; werase = ^w; lnext = ^v;
-parenb -parodd cs8 -cstopb -hupcl cread -clocal -loblk -crtscts -crtsxoff -parext
-ignbrk brkint -ignpar -parmrk -inpck istrip -inlcr -igncr icrnl -iuclc
ixon -ixany -ixoff -imaxbel
isig icanon -xcase echo echoe echok -echonl -noflsh
-tostop echoctl -echoprt echoke -defecho -flusho -pendin iexten
opost -olcuc onlcr -ocrnl -onocr -onlret -ofill -ofdel


root@ussd-gw # env
EDITOR=vi
HOME=/
LD_LIBRARY_PATH=/usr/openwin/lib
LOGNAME=root
MAIL=/var/mail//root
MANPATH=:/usr/man:/usr/share/man
OPENWINHOME=/usr/openwin
PATH=/opt/sun/bin:/usr/sbin:/usr/bin:/usr/ccs/bin:/usr/openwin/bin:/usr/dt/bin:/usr/platform/SUNW,Sun-Fire-280R/sbin:/home/oracle/ora9i/bin
PS1=root@ussd #
PS2=root@ussd >
SHELL=/sbin/sh
SSH_CLIENT=10.2.29.44 35267 22
SSH_CONNECTION=10.2.29.44 35267 10.2.29.43 22
SSH_TTY=/dev/pts/1
TERM=xterm
TZ=Asia/Calcutta
USER=root

Quote:

Originally Posted by catkin

What happens if you echo foo > <device>?

I am sorry, unable to understand this.

Quote:

Originally Posted by catkin

Are you able to try similar from some other terminal? The console itself?

Yes, it works perfectly on other terminals of Solaris. I am sorry console is not available.

vikas027 · 07-14-2009, 03:17 AM

Quote:

Originally Posted by jlliagre

That looks very strange.

What says:

Code:

ptree $$
truss -t !mmap,munmap -f sleep 1

?

Please find the below logs

Code:

root@ussd # ptree $$
468   /usr/local/sbin/sshd
  23441 /usr/local/sbin/sshd -R
    23444 -sh
      23458 ptree 23444
root@ussd #
root@ussd #
root@ussd # truss -t !mmap,munmap -f sleep 1

23460:  execve("/usr/bin/sleep", 0xFFBEFD34, 0xFFBEFD40)  argc = 2
23460:  resolvepath("/usr/lib/ld.so.1", "/usr/lib/ld.so.1", 1023) = 16
23460:  open("/var/ld/ld.config", O_RDONLY)             Err#2 ENOENT
23460:  stat("/usr/openwin/lib/libc.so.1", 0xFFBEF454)  Err#2 ENOENT
23460:  stat("/usr/lib/libc.so.1", 0xFFBEF454)          = 0
23460:  open("/usr/lib/libc.so.1", O_RDONLY)            = 3
23460:  fstat(3, 0xFFBEF454)                            = 0
23460:  memcntl(0xFF280000, 113504, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
23460:  close(3)                                        = 0
23460:  stat("/usr/openwin/lib/libdl.so.1", 0xFFBEF454) Err#2 ENOENT
23460:  stat("/usr/lib/libdl.so.1", 0xFFBEF454)         = 0
23460:  open("/usr/lib/libdl.so.1", O_RDONLY)           = 3
23460:  fstat(3, 0xFFBEF454)                            = 0
23460:  close(3)                                        = 0
23460:  stat("/usr/platform/SUNW,Sun-Fire-280R/lib/libc_psr.so.1", 0xFFBEF264) = 0
23460:  open("/usr/platform/SUNW,Sun-Fire-280R/lib/libc_psr.so.1", O_RDONLY) = 3
23460:  fstat(3, 0xFFBEF264)                            = 0
23460:  close(3)                                        = 0
23460:  brk(0x00020F20)                                 = 0
23460:  brk(0x00022F20)                                 = 0
23460:  sigaction(SIGALRM, 0xFFBEFBB0, 0xFFBEFC30)      = 0
23460:  alarm(0)                                        = 0
23460:  sigaction(SIGALRM, 0xFFBEFB80, 0xFFBEFC30)      = 0
23460:  sigfillset(0xFF3428C8)                          = 0
23460:  sigprocmask(SIG_BLOCK, 0xFFBEFC20, 0xFFBEFC10)  = 0
23460:  alarm(1)                                        = 0
^Croot@ussd #

catkin · 07-14-2009, 07:27 AM

Quote:

Originally Posted by vikas027

I have tried setting xterm and editor too. Here are some logs, that might help you out.

Code:

root@ussd-gw # stty -a

Looks OK according to my limited knowledge of tty terminals. You could try

Code:

stty sane

Quote:

Originally Posted by vikas027

root@ussd-gw # env[/CODE]

You're using ssh and were assigned virtual tty /dev/pts/1.

So -- OK on other terminals (what sort of other terminals -- how do they differ from the problem one?) and OK for the shell but not for sleep, prstat, tail -f "some log file", sar -u 1 10 and crontab -e. That's quite a range of programs as regards terninal I/O: sleep does none, prstat I don't know, sar is line-orientated and crontab -e is full-screen!

Maybe we can get a bit more handle on the problem by trying a few more programs, looking for a pattern.

For example

Code:

echo foo >/dev/pts/1  <== assuming sshd assigns that device
echo foo /dev/tty
man sh  <== full screen
df  <== lines

jlliagre · 07-15-2009, 10:57 AM

This is the weirdest issue I remember being posted here ...

Can you post this script's output (to be run while a sleep command is stuck):

Code:

#!/bin/ksh
for s in $(pgrep sleep)
do
  ps -fLl -p $s
  for i in pstack pcred psig
  do
    echo === i ===
    /usr/proc/bin/$i $s
  done
done

vikas027 · 07-15-2009, 09:10 PM

Quote:

Originally Posted by jlliagre

This is the weirdest issue I remember being posted here ...

Can you post this script's output (to be run while a sleep command is stuck):

Code:

#!/bin/ksh
for s in $(pgrep sleep)
do
  ps -fLl -p $s
  for i in pstack pcred psig
  do
    echo === i ===
    /usr/proc/bin/$i $s
  done
done

Hi Jlliagre,

Pls find the below output of the script.

Code:

 F S      UID   PID  PPID   LWP  NLWP  C PRI NI     ADDR     SZ    WCHAN    STIME TTY     LTIME CMD
 8 S     root 22218     1     1     1  0  46 20        ?    119        ?   Jul 13 ?        0:00 sleep 2
=== i ===
22218:  sleep 2
 ff31ec58 sigsuspend (ffbefca8)
 ff2cef84 _libc_sleep (0, 0, 0, 0, 2, 10a7c) + f4
 00010a7c main     (fc00, ffff, 20df8, 0, 2, 0) + 10c
 00010958 _start   (0, 0, 0, 0, 0, 0) + b8
=== i ===
22218:  e/r/suid=0  e/r/sgid=0
=== i ===
22218:  sleep 2
HUP     default
INT     ignored
QUIT    ignored
ILL     default
TRAP    default
ABRT    default
EMT     default
FPE     default
KILL    default
BUS     default
SEGV    default
SYS     default
PIPE    default
ALRM    caught  0
TERM    default
USR1    default
USR2    default
CLD     default NOCLDSTOP
PWR     default
WINCH   default
URG     default
POLL    default
STOP    default
TSTP    default
CONT    default
TTIN    ignored
TTOU    default
VTALRM  default
PROF    default
XCPU    ignored
XFSZ    ignored
WAITING default
LWP     default
FREEZE  default
THAW    default
CANCEL  default
LOST    default
RTMIN   default
RTMIN+1 default
RTMIN+2 default
RTMIN+3 default
RTMAX-3 default
RTMAX-2 default
RTMAX-1 default
RTMAX   default

vikas027 · 07-15-2009, 09:11 PM

Hi All,

Seems like I found the culprit in /var/adm/messages.

We have some unusual errors there like.

Code:

kern.warning] WARNING: [AFT0] 3 soft errors in less than 24:00 (hh:mm) detected from Me
mory Module J0406

Transaction was a block operation.
Jun 25 12:06:07 ussd pcisch: [ID 956438 kern.info] dvma access, Memory safari command, address 00000000.7e2ee2d0, owned_
in not asserted.

secondary error from DVMA read transaction

I googled for these errors, seems like the PCI slot where the memory card is inserted is faulty. An engineer from SUN Microsystems is also working on it.

I will post if there is some other problem rather than memory card slot.

Thanks to ALL.

jlliagre · 07-16-2009, 03:00 AM

I doubt the RAM issue has anything to do with it.
Try that other script:

Code:

#!/bin/ksh
trap
for i in $(nawk 'BEGIN { for(i=0;i<32;i++) print i;}')
do
trap - $i
done
time sleep 2

vikas027 · 07-16-2009, 11:31 PM

Quote:

Originally Posted by jlliagre

I doubt the RAM issue has anything to do with it.
Try that other script:

Code:

#!/bin/ksh
trap
for i in $(nawk 'BEGIN { for(i=0;i<32;i++) print i;}')
do
trap - $i
done
time sleep 2

The script hangs, I guess because of sleep. Pls find the below logs.

Code:

root@ussd # ./a.sh > abc &
[1]     23591
root@ussd #
root@ussd # jobs -l
[1] + 23591      Running                 ./a.sh > abc &
root@ussd #
root@ussd # tail -f abc

^Croot@ussd # ls -l abc
-rw-r--r--   1 root     other          0 Jul 17 09:47 abc
root@ussd #
root@ussd # ps -aef | grep 23591
    root 23591 23586  0 09:47:18 pts/1    0:00 /bin/ksh ./a.sh
    root 23593 23591  0 09:47:18 pts/1    0:00 sleep 2