The SMB protocol has gone through numerous revisions since the first product using it, LAN Manager, was launched in the late 1980s. Like many network protocols, security came much as an afterthought. LAN Manager used to send passwords in the clear across the network until a weak LAN Manager hash mechanism was introduced. This in turn was replaced by
NTLM, and then improved upon with NTLM v2.
In a network without a centralized account database, a "workgroup" in Windows terms, the NTLM v2 mechanism is still used for authentication, and that's why Samba needs a separate password database. (Samba actually makes use of the user database in
/etc/passwd, it's just the passwords that must be stored separately.)
When a client PC attempts to access a SMB/CIFS share which requires authentication, the password is never sent over the network. Instead, a challenge/response mechanism is used, which involves creating a password hash and subjecting it to various mathematical operations. The server sends a "challenge" number, the client returns the result of a certain mathematical algorithm involving this number and the password hash, and the server performs the exact same operation and compares the results.
For this to work, both the server and the client must have access to either the unencrypted password (which a server should never store for security reasons), or a 128-bit MD4 password hash of that password. Since the hashes in
/etc/shadow aren't MD4 hashes, they are of no use to Samba, hence the need for a second password hash database. (DES hashes can be used as well, but since that algorithm is seriously outdated, the NTLM protocol in all modern Windows OSes will refuse to accept them unless specifically reconfigured to do so.)
Maintaining two separate password databases is indeed a hassle. It might be possible to configure PAM in such a way that
smbpasswd is called automatically whenever an account password is created or changed, but an even better idea would be to eliminate the need for two password databases. Linux can authenticate against just about anything via PAM, so if you configure Samba to be a so-called "Domain Controller" for either an old-style Windows NT Domain or an Active Directory Domain, both Linux and Samba can use that database for authentication.
As a bonus, you would then be able to log on to any computer in the network, be they Windows or Linux systems, using the credentials from the Windows Domain database. If you were to set up several Samba Domain Controllers, the entire account database would automatically be replicated to every Samba DC, passwords and all.
Oh, and if you go for an Active Directory Domain, the somewhat vulnerable MD4-based NTLM protocol is replaced by Kerberos, which has no known security vulnerabilities. Both Samba and Windows will fall back to NTLM v2 if a non-domain member is involved or if a resource is accessed via an IP address rather than a hostname, but you can disable that fallback mechanism if you so wish.
