You probably want to pass the protection parameters to mmap() in the first place.
If you're not using mmap(), do be aware that a) this violates the POSIX standard and b) mprotect() works on the entire page (usually around 4kb) that happens to contain the address range you pass in; it's mostly used by the kernel internally, and by mmap().
The problem is that mprotect() uses a kernel-level function that allocates memory in the kernel-equivalent of the heap. Kernel memory allocation is subject to a number of different rules about where in RAM a block of memory can be allocated. If there doesn't happen to be any memory free in the right place, the allocation will fail. (For user-level memory, this would trigger the entire memory page to be swapped out and the memory allocation to be re-tried, but the kernel is primarily concerned with speed and so doesn't normally do this.)
The upshot of this is that allocation of kernel memory isn't guarenteed, and any function that requires allocating kernel memory, like mprotect(), can fail on memory, even if you technically have memory free in your system.
I would put this in a while loop, with an exit condition on timeout or when not returning ENOMEM. It might also help you to usleep() or schedule during the loop, as this will at least give the kernel a chance to swap out some RAM.