You seem to be under the impression that libraries and .h files are the same thing. Your title mentions library directories, then you ask about the directory containing .h files in the text.
A library and a .h file are NOT the same. A library contains object code that has the actual implementation of things. A .h file simply DECLARES the implementation of those things, it doesn't implement them.
So... even if you get your INCLUDE paths working, the next thing you will probably run into is linker errors because you don't have the LIBRARY paths setup. To tell the linker about a LIBRARY, you use -llibraryname and possibly -L/path/to/library if it is not in your search paths.
Sorry, if you are already aware of the distinction, but it was always a bit of a pet peeve of mine when people in my C++ classes in college would say they included a library, when what they really did was to include a header file...
I think the confusion might have come from the fact that the standard C/C++ libraries are usually included by default so they never had to deal with actual libraries before.