Hi,

On CentOS 6.10, I couldn't run/start few processes (3rd party like x264, x265, Splunk etc.) as "root user" but no issues running them as a "Normal user". This is happening only on few machines in my environment. Is it some sort of security that was implemented on these boxes that prevents to be run/start as "root user"?

Please advise, Many thanks!!

How precisely are you launching these processes, and what happens if you try launching them as root?

Are you sure that the same operating system and application software versions are installed on all systems in your environment? In other words, can you see differences between the configurations of the servers that exhibit the problem and servers that don't?

Thanks berndbausch for your reply.

All machines are running CentOS 6.10 with selinux in "permissive" mode (i checked on each machine) with same patch level etc. There might be (not sure) slight (don't know what they are) config changes as they were built by another team.

When I run splunk ("/opt/splunkforwarder/bin/splunk") as root user it just stucks/hangs (waited for an hour or so) and the only way is to stop it by ctrl-C or close the putty session. Splunk won't start. But same thing I can run as "splunk" user (normal user) without any issues.

I think it might be some security settings.

Please advise!

There is no way to generally prevent root from running certain programs (except mandatory access control, i.e. SELinux, which we can exclude here, and AppArmor, which is unlikely to be enabled on Centos).

So I am thinking that this problem is caused by some Splunk configuration. I hoped you would get an error message on the command line. Since this is not the case, are there log files you could check? Is there a verbose or debug option when launching splunk (sadly I know very little about Splunk)?
By the way, shouldn't you run splunk with the start parameter?

If all else fails, you can run the command under strace. strace will show you the system calls that the process issues, which might give you a clue. For example, it shows you the files it attempts to open, and whether they were opened successfully or not. Be prepared for a lot of output.

Thanks again. Yes, we need to run it with start/stop/status parameters which i am doing correctly.

As you suggested, I used strace along with splunk by running and it never returned to prompt but writing continuously to /tmp/strace.out which has lots of "timeouts". I also ran the same as splunk user which ran in fraction of second without any issues. I attached both strace logs, Please see advise, thank you!!

Attached Files

	splunk_status_as_ROOT_user_struss-log.txt (52.2 KB, 13 views)
	splunk_status_as_splunk_user_struss-log.txt (10.2 KB, 10 views)

The two traces diverge around lines 110-120 after the call to getuid(). The non-root version then checks for the presence of some files under /opt/splunkforwarder and seems to execute the splunk daemon.

The root version does something totally different. Rather than looking at /opt/splunkforwarder, it opens a UNIX socket named /var/lib/samba/winbindd_privileged/pipe and reads from it in an endless loop. It seems that it reads as long as data is available at this socket, and data seems to be available all the time. Some of that data is ASCII, such as radiocymruprog (Welsh radio program? I am curious!), cas_assetreg, rad_rdincome, radcym_head (more Welsh) etc.

Judging from the socket's name, it's obviously part of Samba, the open-source version of Windows share server, and it references Winbind, which is some sort of name resolution protocol in Windows share environments.

My guess is that the problem only occurs on servers where Samba and/or Windbind are configured, or where Splunk is configured to do something with Winbind. But that's all I can offer with the information given.

Do you have winbind in /etc/nsswitch.conf? Then ensure there is files before winbind.

Thanks berndbausch & MadeInGermany

It seems "files" given priority over winbind

I also tried the Latest version of Splunk Forwarder and the behaviour is same.