If you are fairly new to Linux I would go for a small business solution called Zentyal
. It's based off Ubuntu and has file-sharing web filtering etc, so its pretty much an all-in-one solution and can be used as a home server.
If you want to build your own server then FreeBSD
would be a good choice, as would most flavours of Linux. FreeBSD is good just for the Handbook
A good howto for building a FreeBSD server I found recently is here
but the Samba part is still waiting to be done. Other useful resources, apart from here, are FreeBSD Forums