I won't say it is impossible, but if everybody has shell access (or some other way to launch a program), I would think it would be incredibly difficult to do what you want. Even if you find some way to keep people from launching something that identifies itself as Shoutcast, somebody could get around it by calling the program something else. It would seem you would require whitelisting using digital signatures or something.
W/o using
noexec (and maybe even using it), it sounds like a very difficult problem to me. All I can do is wish you good luck. Maybe somebody else has a really clever idea.