This from chapter 5.1 Introduction (from LFS 6.3, don't have 6.2 lying around any more):
After installing each package, delete its source and build directories, unless specifically instructed otherwise. Deleting the sources prevents mis-configuration when the same package is reinstalled later.
This tells you you _have_ to delete the source/build directories, not doing so might get you into trouble later on.
PS: spiky0011 said as much, but it wasn't worded strong enough in my opinion