How to track the user who has done something problematic in defined condition
 

There are two servers i.e server A & server B (both centOS) and 10 ubuntu systems. All in the same LAN

All users are logged into server A from their ubuntu systems and they all do their work regularly. All know root password of server A. Because They all login as root in server A frequently according to their work.

Problem :
1) A user logged into server A from his ubuntu system.
2) Run sudo -i and become root of server A.
3) Then He logged into server B as root. It didn't ask password because of authorized_keys.
4) Then he did something nasty on server B.
Now I want to find him. I have read /var/log/secure of both servers many times.
It is not helping me.

I know it is height of foolishness and no linux user can't beat it, at least in this century. But it has been done.

Please help to find the user.
Thanks
sam

I assume you know the time of the event on server B.

You should know which users had root on A at that time. That may be more than one. Depending on the details logged you may be able to see which tty was used for the ssh process .. and match that to the logged-in user.

I hope you can see by now the advantage of denying root remote login/command access in sshd (even with keys).

If the user is any good he'll have covered his tracks by editing anything in /var/log/* that would be relevant.

/var/log/secure and look for sudo sessions starting around the time of the event would be about all you'll get. Given how vulnerable your system is I'd think you're unlikely to ever find out.

This why using a remote log server is important. Nobody except security investigations should login on the log server.

And I would have said "...how vulnerable your systems are...". Anytime you have more than 4 people with uncontrolled root access, you have a severe problem.