My explanation may not be 100% on track with my thinking, so somebody please correct me if need be, but here goes:
The -d option for ls only lists the "directory entries instead of the contents"; which makes me think that it will only list the directories within the current working directory and not the files within. (I don't have access to a Linux box right now, so I can't test this theory out.)
The reason the size of the directory is only 4.0K is a nice reminder that you're using a UNIX-oriented system: everything is a file. Directories are just pointers to table entries, and what you're getting is the size of the file that is required to direct the computer to the appropriate "inode" table. Once formatted, a hard drive is nothing more than a giant inode table which says, "This file exists here, this directory refers to the container for these files...", etc.