You will already understand that a mutex cannot be acquired twice (because this would lead to recursion deadlock). In the kernel, the locking correctness validator can be used to check for such errors, as well as other locking errors such as inconsistent order of mutex acquisition.
Now rather than operate on individual mutex objects, the validator instead works with 'classes' of mutexes. This is because there may be thousands of mutexes all operating on the same type of object, and it is the ordering of the acquisitions of the classes that are of most relevance to validation (for example, in general we don't care which inode mutex was acquired when doing validation).
This generalization in the validator works in most cases, but there are some classes of objects where the kernel may acquire multiple mutexes from the same class, and where the ordering does matter. In order to be able to validate these, the
mutex_lock_nested was added. One can define a set of subclasses of a class that represent nesting levels, and the order of locking of the subclasses can then be validated (by calling the 'nested' mutex lock with the subclass as a parameter). It is not literally a recursive lock; the nesting occurs at the semantic level, from the point of view of the validator.
Refer to
lockdep-design.txt in the kernel documentation for a more detailed explanation.
