Files and Folders just simply Disappear - weirdest thing yet
 

Hi All,

I've had this problem for a while and I'm still none the wiser on where it starts.

I have a directory full of files and folders which get rsync'd to every now and then by pulling down the new files and folders from a remote host (lots of them).

The problem I have is an rsync can complete and the files will all be there, but when I come back a little later those newly added or rsync'd files and folders have all gone like its somehow reverted everything to a the state everything in this particular folder was in before the rsync occurred (though i'm not sure if its everything or just most things... because 'find . -mmin -3600' . being the rsync destination path) does find some new files but newly copied across folders can simply vanish and files within existing folders also simply vanish.

When I assume the same thing occurs during an rsync this is clearer to see because as soon as it does hit the wall rsync then sends out this message for every respective file its got left in its list to copy over..
So to me this says something is somehow either deleting a folder at the local host between root level and destination rsync folder or renaming something or a symlink change or I have no idea what really. But every time you can see during the rsync with 'df -h' that disk space is being used and decreasing nicely as expected but then you hit what ever the hell this is and disk space is instantly returned to what it was before I started rsync'ing any files. Every time!

I've been trying all sorts, 'strace'ing different processes throughout the system to see if they are doing anything specific to the paths or anything of the rsync. 'ls -i' output of each folder until getting to the local destination folder to see if during an rsync the inode on any of the folders is changing mid way through which they don't appear to.

I've set-up and run auditd for a while to see if that picks-up on anything, auditd is shouting about all the right things during rsync but then when I hit the wall again there's nothing in auditd about any changes plus auditd doesn't even show (or perhaps see) attempts of the rsync trying to create temporary files in the path its auditing (also makes me think that somehow the path is getting lost/changed somewhere), so when in the above rsync out put you see the 'mkstemp' no such file errors, auditd isn't picking up or showing that any files are trying to be created? Perhaps I need to run auditd but at a directory level higher and keep going higher until at the root level to see if that tells me anything? But an 'ls' in a while loop with no sleep times constantly running on the folder path never fails but you do just suddenly and instantly see a whole load of files disappear!

Other than that I'm completely fresh out of ideas and would welcome some much needed help? Only other thing I can try is using chattr to force the non-delete bit or something, but i've not tried that yet.

Thanks in advanced for any help,
Best,
M

EDIT: OK - So i've run auditd on a file basis from a file that existed after the rsync had run for a while, I kill the rsync and the file still exists i wait a while and then it disappears later, no specific amount of time for the file to disappear its always changing. But I put auditd rules in place from the file destination going backwards NOT recursive as that's far to me data but just a rule on each folder and the file all the way upto the root level. So i've found something is running an 'ln' command which seems to be what's making me loose the files somewhere, but i'm not sure how still yet. I mean once the files "disappear" not even searching the entire file system can find the file 'find / -name file -type f', which I'd have thought if it was a symlinking type problem then the file would still be on the file system but just in another place or something, but in this instance it disappears completely? Even though symlinks might be changing this I think should be OK because the destination of files i'm rsyncing to isn't using any symlinks on the path EG:

/folder1/folder2/folder3/file

Rsync is:
rsync remotehost:/folder1/folder2/ /folder1/folder2/

Where folder1 folder2 or folder3 none of which are symlinks themselves, but there are symlinks that are being created that are linking to either folder1 folder2 or folder3 at the time the the files seem to end up disappearing, which to me shouldn't be a problem, I could understand it if of the path '/folder1/folder2/folder3/file' one of these folders was a symlink and that was changing half way through but thats not the case, its just links are being created to link to a folder on this path.

So, am I assuming correct:
You're running this command:and after a while the newly copied files in '/folder1/folder2' on your local host just disappear?
Is this what you wanted to say?

Regards,
Kostya

HI kostya,

Yes - basically that is it. Something is happening to cause the files to disappear, but I can't for the life of me work out what even with all the options that I've tried above so far.

Best,
M

Yea... I've had a couple of such exciting moments through my linux "career", so I can understand you.
Just wonder, is this rsync command of yours done from some script? If so, who owns the script and how is it executed? What's there in the script, besides the rsync command?
What, then, is your Selinux policy?

...especially that Selinux thing is known to cause files to disappear unexpectedly. Might check that, too.

I mean, just without going too deep into it, you might try first of all to disable selinux and see if it changes anything. Then, of course, you could fine-tune it a bit, cause in general selinux is a good thing to have enabled...

Hi Kostya,

Yea - its started of by an automated job from a Continuous Integration (CI) server, neither apparmour or SElinux are running. I think one final idea would be to set-up appArmour (I've not used SE Linux..), then I can trace any invalid steps the CI agent attempts to carry out and see what's going on. I've always felt the problem was related to the CI agent and the job its executing but I strace'd that particular process a long time ago now when i made a first attempt to see what it was doing and during that I couldn't see anything obvious with the files it plays with etc, (but there's so much output its possible to miss something i suppose). Does SElinux have something that would trace the attempts to change things in a similar to the cool way apparmour works and putting it into log mode but dooesn't stop the application from doing what it wants? I'd prefer to use apparmour but Debian no longer supports out of box and i don' really feel like going round the woods to get installed.

Thanks Again

With huge pieces of software like servers it is a good idea to know what exactly they do and how they interact with the rest of the system.
And to see what SELinux can do you need to check the manuals. It's a damn complicated thing...

But then again, there are so many integrated server solutions (says Google), that you may try different ones. Cause there are just AS many posts about disappearing files.
A good question will be, too, WHERE those folders are situated, where the rsync'ed files tend to disappear. Is it, per chance, another HDD with NTFS filesystem?

helptonewbie: can you please post the exact invocation of rsync you're using,
and tell us whether or not you're running in daemon mode?

Also, the file-names you presented us with very much look like temp-files.
Is that you being a secret squirrel about what you're copying, or are those
really temp-files your syncing?


Cheers,
Tink