No. once on cache it remains there until the cache is rebuilded by another "ldconfig" command.
You can easilly verify this by writing a small C program that use a simple C function on a shared library for your own. - ok, maybe not so easilly
Put your small lib in a distinct folder, like "/usr/local/testlib", put this path in /etc/ld.so.conf, re-run "ldconfig", verify it is in cache with ldconfig -p, run your test program, and when it finish, verify your lib still is in the cache.
This is a good exercice and you will learn a lot about linking and shared libs if you do it by yourself.