Did you harden the box? If not that's what you should start with. Check out the
LQ FAQ: Security references, post #1 under "Checklists", "Securing", "Hardening, distro specific" and "Log analysis tools, resources".
I run a server with shell accounts and I noticed that any user can run X aplications (..) from his home
A shell server is a shell server and not a toybox. It shouldn't run X at all. You need to minimise chances people abuse applications by minimising the amount of applications installed. This means essentially all applications not necessary for the function of the box should be removed or protected (removal is better). If that's impossible you will need to 1. take away access rights for "world" on all dirs, configs, libs, binaries and other resources involved, 2. chown 'em to a separate group, 3. remove /usr/X11 from these users $PATH and 4. Install the Grsecurity kernel patch. It includes TPE (Trusted Path Execution) which means users will have no chance executing apps outside of the path *you* specify, options for denying users client and/or server socket operations, process and memory protection and ACL's for even more strict access definitions.
