If you are fairly new to Linux I would go for a small business solution called
Zentyal. It's based off Ubuntu and has file-sharing web filtering etc, so its pretty much an all-in-one solution and can be used as a home server.
If you want to build your own server then
FreeBSD or
OpenBSD would be a good choice, as would most flavours of Linux. FreeBSD is good just for the
Handbook documentation.
A good howto for building a FreeBSD server I found recently is
here but the Samba part is still waiting to be done. Other useful resources, apart from here, are
FreeBSD Forums and
DaemonForums.