A couple of thoughts.
Why not use a loop instead of xargs? The xargs has some limitations
that you won't experience using a loop. For example:
cat directories.dat | while read Dir; do ls "$Dir"; done
Also you don't mention anything about special characters. The * is
interpreted and possibly expanded into a set of files. If you have
space characters, tabs, slashes, back-slashes, question marks or
ampersands in any of the paths they will also be interpreted.