3rd Party Software Won't Run, Install kdump to troubleshoot?

JockVSJock · 02-28-2016, 09:26 AM

So I'm working with an outside vendor to install their software on a mix of RHEL 5/6 and mix of x32 and x86_64 servers.

The software will install, however it will start and all of the sudden their daemon stops working. The daemon doesn't have an entry under /etc/init.d, however the software does install to /opt.

Their tech support recommends installing kdump to try and troubleshoot. I thought that was only to troubleshoot the kernel?

Would it be worth to install kdump in order to troubleshoot?

pan64 · 02-28-2016, 09:51 AM

this question cannot be answered since we have no idea about that software at all. Probably it has some logs, or you can find something in /var/log

JockVSJock · 02-28-2016, 09:57 AM

Want to say this, originally when their daemon was dying, they kept telling me its SELinux fault. It wasn't.

I'm not used kdump before and was wondering what others have experienced.

thanks

pan64 · 02-28-2016, 11:45 AM

It still depends on you, on your knowledge. You need to learn how to use it and need to understand what's happening. Personally I would request more support.

JockVSJock · 02-29-2016, 07:37 AM

Quote:

Originally Posted by pan64

Personally I would request more support.

Honestly, I've gotten better support/information here by asking questions at Linux Questions, then some of the software support that we pay for...

John VV · 02-29-2016, 06:45 PM

a mix of rhel5 and 6 ( should be the up to date rhel 5.11 and 6.7 ) and 32 and 64 bit

that's a mess

Quote:

they kept telling me its SELinux fault. It wasn't.

10 years ago SE might have been a issue
back in 2005/6 SE was a bit new

NOT today ,

i take it you have read through the logs to find the errors

and you have ran "strace" on the program to see where it is faulting ( if applicable )

with a mixed ARCH of 32 and 64 bit
Are ALL the prerequisites installed ?
you did use yum to install the rpm's ??? right ?
How are you installing them ?

the same as above for the rhel5 and 6 mixing
are the el5 and el6 local repos set up like for the x86 and 64_x86 ?

basically is all the other rpm's installed as needed for
rhel5-32
rhel5-64
rhel6-32
rhel6-64

syg00 · 02-29-2016, 07:31 PM

Personally I think kdump should always be present. But not many agree.

I don't see how it would help if you aren't getting kernel oops to trigger the dump. You could use sysrq keys to force one, but why ?. Can you stand to have a reboot at the time of the failure ?.
Do you have debugging info available for that kernel(s) ?. Does the vendor if you are going to ship the dump file offsite ?.

JockVSJock · 02-29-2016, 08:11 PM

Quote:

Originally Posted by John VV

with a mixed ARCH of 32 and 64 bit
Are ALL the prerequisites installed ?

Yes, the 3rd party software is only .tgz and then launching a .sh file. No yum or RH Satellite involved. All servers are patched up and current.

The vendor has to give me the correct .tgz file based on if its RHEL 5/6 or x32 or x86_64.

strace is a good idea. I will give it a shot and report back.

The bad thing is that the previous admin didn't install kdump, however again, I don't think kdump is going to find an issue, its not a kernel issue.

syg00 · 03-01-2016, 01:06 AM

Maybe bump up the kernel printk level too while doing the test if you aren't getting anything in dmesg. As root

Code:

echo 7 > /proc/sys/kernel/printk

Do a cat first - first number (probably 4) is what you want to set it back to.

Valery Reznic · 03-02-2016, 06:32 AM

Quote:

Originally Posted by JockVSJock

So I'm working with an outside vendor to install their software on a mix of RHEL 5/6 and mix of x32 and x86_64 servers.

The software will install, however it will start and all of the sudden their daemon stops working. The daemon doesn't have an entry under /etc/init.d, however the software does install to /opt.

Their tech support recommends installing kdump to try and troubleshoot. I thought that was only to troubleshoot the kernel?

Would it be worth to install kdump in order to troubleshoot?

Is their software includes kernel module(s) or it just user space exes?

JockVSJock · 03-02-2016, 08:08 AM

Quote:

Originally Posted by syg00

Maybe bump up the kernel printk level too while doing the test if you aren't getting anything in dmesg. As root

Code:

echo 7 > /proc/sys/kernel/printk

Do a cat first - first number (probably 4) is what you want to set it back to.

Dumb question.

When I run

Code:

cat /proc/sys/kernel/printk 

3 4 1 7

I'm not familiar with printk, so I had to look, according to the following website, these numbers correspond to the following:

http://elinux.org/Debugging_by_printing

3 = current
4 = default
1 = minimum
7 = boot-time default

Just so I understand, am I only able to set that current number? Or are you able to specify which number you set?

syg00 · 03-02-2016, 08:15 AM

Just current.

JockVSJock · 03-02-2016, 08:28 AM

I am getting output of the software from dmesg, however I'm not sure if its an error or not.

Code:

[root@server varonis_strace_logs]# dmesg | grep varonis
varonis_drv: module license 'Proprietary. Send bug reports to support@varonis.com' taints kernel.
 [Varonis] varonis was register with major=253

If I were to bump up printk from 3 to 4, where would I see the output so I can further troubleshoot? Is it dmesg?

JockVSJock · 03-18-2016, 02:19 PM

So I'm revisiting this again, as I continue to get errors and try to troubleshoot. I've noticed that /var/log/messages is filling up with the following:

Code:

[root@server varonis]# tail -f /var/log/messages
2016-03-18T12:27:38.541532-05:00 server vrns_mond: Failed running /opt/varonis/vrns_mond - Operation not permitted
2016-03-18T13:24:35.987242-05:00 server vrns_mond: Failed running /opt/varonis/vrns_mond - Operation not permitted
2016-03-18T13:25:08.575069-05:00 server vrns_mond: Failed running /opt/varonis/vrns_mond - Operation not permitted
2016-03-18T13:25:41.172268-05:00 server vrns_mond: Failed running /opt/varonis/vrns_mond - Operation not permitted
2016-03-18T13:26:13.762016-05:00 server vrns_mond: Failed running /opt/varonis/vrns_mond - Operation not permitted
2016-03-18T13:26:46.373314-05:00 server vrns_mond: Failed running /opt/varonis/vrns_mond - Operation not permitted
2016-03-18T13:27:18.982556-05:00 server vrns_mond: Failed running /opt/varonis/vrns_mond - Operation not permitted
2016-03-18T13:27:51.585849-05:00 server vrns_mond: Failed running /opt/varonis/vrns_mond - Operation not permitted

The software is installed under /opt/varonis and I'm kicking it off as Root

Code:

-rwxr-x--- 1 root varonis   4129 Oct 31  2012 vrns_mon.sh

There is nothing out of the ordinary with SELinux and neither with acls.

pan64 · 03-21-2016, 04:09 AM

what is /opt/varonis/vrns_mond (is this a shell, binary, ??) You can check it with the command file.
you can try to strace it too.