File integrity in Samba/NFS environment
 

Hi,

I have a two server setup that acts as SMB as well as NFS servers in active/active configuration managed by CTDB(http://ctdb.samba.org/).

When a file is updated by SMB clients(followed by file-close), other SMB clients can see and modify the file. But when a NFS client (same user) updates the same file(followed by file-close), only one SMB server can see the updates. The clients mounting from other SMB server sees the file corrupted. BUT, when you again update the file from any CIFS clients(mounting from SMB server that can see the updates), the updates are visible on all the CIFS clients.

My first suspect was to do with NFS and SMB caching. I forced NFS server to export with sync option and same with SMB with the following, but does not help:

strict allocate = yes
strict locking = yes
strict sync = yes
sync always = yes

NFS exports:
/mnt/gpfs/nfsexport *(rw,no_root_squash,sync,fsid=222)

Tried NFS mount with sync:
node1:/mnt/gpfs/nfsexport /mnt/nfs nfs rw,tcp,hard,intr,sync,rsize=32768,wsize=32768,vers=3 0 0

smb.conf
[global]
workgroup = TESTDOMAIN2
realm = TESTDOMAIN2.LOCAL
netbios name = CTDB-NAS
server string = Clustered CIFS
security = ADS
auth methods = winbind, sam
password server = 172.16.2.25
private dir = /mnt/gpfs/CTDB_AD
passdb backend = tdbsam
client NTLMv2 auth = Yes
log level = 3 winbind:5 auth:10 passdb:5
syslog = 0
log file = /var/log/samba/log.%m
max log size = 10000
deadtime = 15
socket options = TCP_NODELAY SO_RCVBUF=16384 SO_SNDBUF=16384
use mmap = No
clustering = Yes
machine password timeout = 999999999
local master = No
dns proxy = No
ldap admin dn = cn=ldap,cn=Users,dc=testdomain2,dc=local
ldap idmap suffix = dc=testdomain2,dc=local
ldap suffix = dc=testdomain2,dc=local
idmap backend = ad
idmap uid = 5000-100000000
idmap gid = 5000-100000000
template homedir = /home/%D+%U
template shell = /bin/bash
winbind separator = +
winbind enum users = Yes
winbind enum groups = Yes
nfs4:acedup = merge
nfs4:chown = yes
nfs4:mode = special
gpfs:sharemodes = no
fileid:mapping = fsname
force unknown acl user = Yes
strict allocate = Yes
strict sync = Yes
sync always = Yes
blocking locks = No
strict locking = Yes
vfs objects = gpfs, fileid
[global-share]
comment = GPFS File Share
path = /mnt/gpfs/nfsexport
read only = No
inherit permissions = Yes
inherit acls = Yes

Any idea to what I may be doing wrong?

Thanks in Advance,
-Tim

Hey There,

I think I found an answer for you from samba.org. It's not mine, so, if it works, thank the gentleman at the link below :)

I would inline it but it's a lengthy explanation, and addresses a few of your concerns.

Hope it's of some help to you and best wishes,

http://lists.samba.org/archive/samba...ne/019414.html

, Mike

File integrity in Samba/NFS environment
 

Thanks Mike for the pointers.

With "oplocks = no", still we see the same behaviour of only some SMB clients seeing the file updates from NFS client (even after file is closed after updates). Using native file-system GPFS Linux client, all the SMB clients can see updates from the Linux client. I tried appending contents from NFS client via O_DIRECT flag (to bypass NFS caching), but does not seem to solve.

When SMB clients(which cannot see updates) tries to access the file after updated by NFS client, we get the following error:

"Cannot open the Z:\FileName_XYZ
Make sure a disk is in the drive you specified".

The SMB client run Windows 2003 64-bit and all of them have the latest updates.

Please note, it seems to be always the same SMB server that is having issues and when the SMB client mounting from the other SMB server updates the NFS file, all the SMB clients can access. The SMB servers are managed via CTDB authenticating via AD 2003.

Thoughts/comments/suggestions to resolve this will be greatly appreciated.

Thanks,
-Tim