I am assuming you're using the Nautilus file browser...
Maybe there are too many files in your home directory? I've heard reports of Nautilus choking on excessively long directory listings (like over 1000 files). Could be even worse if you have "show hidden files" turned on in your preferences, since your home directory probably has a bunch of those.
A quick and easy command to find out how many files/directories are in your home directory, open up a terminal and type:
ls -al ~/ | wc
The first number tells you how many there are.
If Nautilus just won't handle that many, I would suggest trying a different file browser, such as Endeavour Mark II