I've been trying to figure out how failed calls to setuid causes insecurities. So far all ive been able to find is.. If a program doesnt check the return status, there is no way to tell 100% that priveleges were dropped.

But what things will cause this failure? Ive heard if the process limit has been reached it will fail, but what other things too?(and is that correct)

Its not optional, you must check return codes of system calls and handle errors if there are any.
The man page for setuid() lists two possible errno values and why they would happen.
Is there something other than what's listed in the man page that you are trying to find out?

A good example of my confusion comes from reading:
http://www.cs.berkeley.edu/~daw/pape...d-usenix02.pdf

It states that on solaris 8 and the current linux of the time during writing.. A call to setuid(getuid()) will not fully drop privileges unless the program is running as root. If you are not running as root, the saved uid will remain the same after that call.. Allowing an attacker to restore privileges.

I tested it locally on an old solaris 8 machine. It doesnt seem to be true. setuid(getuid()) fully drops the privileges. So i dont see why such a big deal was made about it. Unless the kernels are very old.. But in that case the kernels are probably vulnerable to other direct exploits.

It sounds like you understand it just fine, but I want to emphasize one important point just to be sure, that setuid(getuid()) does nothing when you run the program as root.

Edit: I should have been more clear, if you are logged in as root and run the program, or if you are starting the program from an init.d script, that's when it will do nothing. If the program's setuid flag is set then in will drop priv. only if its run by a non-root user.

Yep, that makes sense.
So lets say im using a solaris 8 machine, that acts the way described by the paper above..
How would I successfully drop my priv if iwas a setuid NON-root user.

Since setuid(getuid()) will not drop my saved privileges.

Heh, I should but didn't read the paper yet. Its 20 pages and when I get home from work I don't feel like reading that much.

On linux, the setuid(getuid()) will drop priv. in the situation you described.. I can't say for Solaris because I have not tried it. The solaris machines I have access to are older than solaris 8, possibly 6. There are other people around here that know the details of solaris much more than I.