LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-19-2012, 03:18 PM   #1
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Rep: Reputation: 39
Question Files and Folders just simply Disappear - weirdest thing yet


Hi All,

I've had this problem for a while and I'm still none the wiser on where it starts.

I have a directory full of files and folders which get rsync'd to every now and then by pulling down the new files and folders from a remote host (lots of them).

The problem I have is an rsync can complete and the files will all be there, but when I come back a little later those newly added or rsync'd files and folders have all gone like its somehow reverted everything to a the state everything in this particular folder was in before the rsync occurred (though i'm not sure if its everything or just most things... because 'find . -mmin -3600' . being the rsync destination path) does find some new files but newly copied across folders can simply vanish and files within existing folders also simply vanish.

When I assume the same thing occurs during an rsync this is clearer to see because as soon as it does hit the wall rsync then sends out this message for every respective file its got left in its list to copy over..
Code:
rsync: stat "/path/file.pLFghQ" failed: No such file or directory (2)
rsync: rename "/path/file.pLFghQ" -> "/path/file": No such file or directory (2)
rsync: mkstemp "/path/file1.nfeDwc" failed: No such file or directory (2)
rsync: mkstemp "/path/file2.KbHPNy" failed: No such file or directory (2)
rsync: mkstemp "/path/file3.kSJf9U" failed: No such file or directory (2)
.....
.....
So to me this says something is somehow either deleting a folder at the local host between root level and destination rsync folder or renaming something or a symlink change or I have no idea what really. But every time you can see during the rsync with 'df -h' that disk space is being used and decreasing nicely as expected but then you hit what ever the hell this is and disk space is instantly returned to what it was before I started rsync'ing any files. Every time!

I've been trying all sorts, 'strace'ing different processes throughout the system to see if they are doing anything specific to the paths or anything of the rsync. 'ls -i' output of each folder until getting to the local destination folder to see if during an rsync the inode on any of the folders is changing mid way through which they don't appear to.

I've set-up and run auditd for a while to see if that picks-up on anything, auditd is shouting about all the right things during rsync but then when I hit the wall again there's nothing in auditd about any changes plus auditd doesn't even show (or perhaps see) attempts of the rsync trying to create temporary files in the path its auditing (also makes me think that somehow the path is getting lost/changed somewhere), so when in the above rsync out put you see the 'mkstemp' no such file errors, auditd isn't picking up or showing that any files are trying to be created? Perhaps I need to run auditd but at a directory level higher and keep going higher until at the root level to see if that tells me anything? But an 'ls' in a while loop with no sleep times constantly running on the folder path never fails but you do just suddenly and instantly see a whole load of files disappear!

Other than that I'm completely fresh out of ideas and would welcome some much needed help? Only other thing I can try is using chattr to force the non-delete bit or something, but i've not tried that yet.

Thanks in advanced for any help,
Best,
M

EDIT: OK - So i've run auditd on a file basis from a file that existed after the rsync had run for a while, I kill the rsync and the file still exists i wait a while and then it disappears later, no specific amount of time for the file to disappear its always changing. But I put auditd rules in place from the file destination going backwards NOT recursive as that's far to me data but just a rule on each folder and the file all the way upto the root level. So i've found something is running an 'ln' command which seems to be what's making me loose the files somewhere, but i'm not sure how still yet. I mean once the files "disappear" not even searching the entire file system can find the file 'find / -name file -type f', which I'd have thought if it was a symlinking type problem then the file would still be on the file system but just in another place or something, but in this instance it disappears completely? Even though symlinks might be changing this I think should be OK because the destination of files i'm rsyncing to isn't using any symlinks on the path EG:

/folder1/folder2/folder3/file

Rsync is:
rsync remotehost:/folder1/folder2/ /folder1/folder2/

Where folder1 folder2 or folder3 none of which are symlinks themselves, but there are symlinks that are being created that are linking to either folder1 folder2 or folder3 at the time the the files seem to end up disappearing, which to me shouldn't be a problem, I could understand it if of the path '/folder1/folder2/folder3/file' one of these folders was a symlink and that was changing half way through but thats not the case, its just links are being created to link to a folder on this path.

Last edited by helptonewbie; 04-19-2012 at 04:49 PM.
 
Old 04-20-2012, 08:30 AM   #2
kostya
Member
 
Registered: Mar 2010
Location: Moscow, Russia
Distribution: Ubuntu Studio, antix(mepis), Fedora, FreeBSD
Posts: 173
Blog Entries: 5

Rep: Reputation: 17
So, am I assuming correct:
You're running this command:
Quote:
rsync remotehost:/folder1/folder2/ /folder1/folder2/
and after a while the newly copied files in '/folder1/folder2' on your local host just disappear?
Is this what you wanted to say?

Regards,
Kostya
 
Old 04-20-2012, 09:21 AM   #3
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Original Poster
Rep: Reputation: 39
HI kostya,

Yes - basically that is it. Something is happening to cause the files to disappear, but I can't for the life of me work out what even with all the options that I've tried above so far.

Best,
M
 
Old 04-24-2012, 02:06 AM   #4
kostya
Member
 
Registered: Mar 2010
Location: Moscow, Russia
Distribution: Ubuntu Studio, antix(mepis), Fedora, FreeBSD
Posts: 173
Blog Entries: 5

Rep: Reputation: 17
Yea... I've had a couple of such exciting moments through my linux "career", so I can understand you.
Just wonder, is this rsync command of yours done from some script? If so, who owns the script and how is it executed? What's there in the script, besides the rsync command?
What, then, is your Selinux policy?

...especially that Selinux thing is known to cause files to disappear unexpectedly. Might check that, too.
 
Old 04-24-2012, 02:09 AM   #5
kostya
Member
 
Registered: Mar 2010
Location: Moscow, Russia
Distribution: Ubuntu Studio, antix(mepis), Fedora, FreeBSD
Posts: 173
Blog Entries: 5

Rep: Reputation: 17
I mean, just without going too deep into it, you might try first of all to disable selinux and see if it changes anything. Then, of course, you could fine-tune it a bit, cause in general selinux is a good thing to have enabled...
 
Old 04-26-2012, 04:24 AM   #6
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Original Poster
Rep: Reputation: 39
Hi Kostya,

Yea - its started of by an automated job from a Continuous Integration (CI) server, neither apparmour or SElinux are running. I think one final idea would be to set-up appArmour (I've not used SE Linux..), then I can trace any invalid steps the CI agent attempts to carry out and see what's going on. I've always felt the problem was related to the CI agent and the job its executing but I strace'd that particular process a long time ago now when i made a first attempt to see what it was doing and during that I couldn't see anything obvious with the files it plays with etc, (but there's so much output its possible to miss something i suppose). Does SElinux have something that would trace the attempts to change things in a similar to the cool way apparmour works and putting it into log mode but dooesn't stop the application from doing what it wants? I'd prefer to use apparmour but Debian no longer supports out of box and i don' really feel like going round the woods to get installed.

Thanks Again
 
Old 04-29-2012, 07:49 PM   #7
kostya
Member
 
Registered: Mar 2010
Location: Moscow, Russia
Distribution: Ubuntu Studio, antix(mepis), Fedora, FreeBSD
Posts: 173
Blog Entries: 5

Rep: Reputation: 17
With huge pieces of software like servers it is a good idea to know what exactly they do and how they interact with the rest of the system.
And to see what SELinux can do you need to check the manuals. It's a damn complicated thing...

But then again, there are so many integrated server solutions (says Google), that you may try different ones. Cause there are just AS many posts about disappearing files.
A good question will be, too, WHERE those folders are situated, where the rsync'ed files tend to disappear. Is it, per chance, another HDD with NTFS filesystem?
 
Old 04-29-2012, 08:41 PM   #8
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,066
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
helptonewbie: can you please post the exact invocation of rsync you're using,
and tell us whether or not you're running in daemon mode?

Also, the file-names you presented us with very much look like temp-files.
Is that you being a secret squirrel about what you're copying, or are those
really temp-files your syncing?


Cheers,
Tink
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
How to import Simply Accounting files into Gnucash? jogl_z_8 Linux - Newbie 1 04-28-2011 11:35 AM
Weirdest thing as terminal gets stuck pgb205 Gentoo 1 02-14-2010 06:10 AM
find -exec command to recursively delete files and folders in folders with X name Joan Murt Linux - Newbie 2 07-08-2009 05:35 PM
Samba can create new files and folders but access denied in any new folders k.king Linux - Networking 2 01-15-2006 07:14 AM
Simply playin mp3 files... Kaelder Linux - Newbie 2 03-31-2003 03:16 PM


All times are GMT -5. The time now is 06:20 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration