If it says 3.8G, then you're using 3.8G. Space adds up, even if it's only a few meg. du doesn't lie.
For the user with the 3.8G home directory, try:
du -csh /home/<the_user>/*
The -c option will add the sizes up for you and print the total usage at the end.
If you want to find out how much space your dotfiles are using, do, from within /home/<the_user>:
This will total up absolutely everything within the <the_user>'s home directory.
It sounds like the 3.8G of usage is just spread out rather than isolated to one or two directories/files.
If you want to quickly find the larger of these directories/files:
ls -rlash --sort=size
should work well. The larger files will be sorted toward the bottom.