LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-14-2014, 08:53 PM   #1
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,104

Rep: Reputation: 181Reputation: 181
Cool Copy and Remove files


hello guys, i copied some files to another folder and I notice it takes only few minutes.
But when I remove the files it takes more than an hour.

Now, I really don't know whether all the files was really copied.

I'm using SUSE Ent. Server 11

Any ideas guys, how Linux does the copying on the background? Or is this a normal operation?

Thanks.
 
Old 07-14-2014, 09:31 PM   #2
sag47
Senior Member
 
Registered: Sep 2009
Location: Orange County, CA
Distribution: Kubuntu x64, Raspbian, CentOS
Posts: 1,832
Blog Entries: 36

Rep: Reputation: 451Reputation: 451Reputation: 451Reputation: 451Reputation: 451
It is not a normal latency for that operation. Deleting files should be fairly instant. Try removing one file at a time and use strace. With strace you'll be able to see which system call the command is hanging up on. Iostat can show you some statistics as well. If top looks normal then there is no good reason for an rm operation to take longer than a copy operation unless it is enumerating of millions of tiny files.
 
1 members found this post helpful.
Old 07-14-2014, 10:22 PM   #3
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,104

Original Poster
Rep: Reputation: 181Reputation: 181
Thanks sag47, I was using putty when I was copying and removing files.

So there's a good possibility that all files were copied successfully?

I will try to check as what you suggested. Thanks for the idea.
 
Old 07-14-2014, 11:36 PM   #4
sag47
Senior Member
 
Registered: Sep 2009
Location: Orange County, CA
Distribution: Kubuntu x64, Raspbian, CentOS
Posts: 1,832
Blog Entries: 36

Rep: Reputation: 451Reputation: 451Reputation: 451Reputation: 451Reputation: 451
Quote:
Originally Posted by JJJCR View Post
So there's a good possibility that all files were copied successfully?
Unfortunately my crystal ball is in the shop :-). If you're concerned about the integrity of a file copy operation then you can do a few things.

If it's only a single file or a few files then run a checksum on the source/destination (md5sum command or any of the *sum commands). Find command one liner works well here for many files.

If you're copying copious amounts of files then you're better off using a utility which checks the integrity as part of the copying/syncing process. Rsync comes to mind for that.

Code:
rsync -av /src /dst
See rsync man page for option explanation. With rsync you can cancel the copy operation half way through and start it again later; it will pick up where it left off.

If you're possibly using putty over a flakey network consider using a terminal multiplexer such as screen or tmux. If you get disconnected from your session the copy operation will continue and when you connect to the server again you can resume the disconnected session to view the result of said operation.

Last edited by sag47; 07-14-2014 at 11:39 PM.
 
1 members found this post helpful.
Old 07-14-2014, 11:52 PM   #5
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,104

Original Poster
Rep: Reputation: 181Reputation: 181
hahah, crystall ball on the shop..

sorry for the vague question..

actually, i just use the cp command to copy the files and so far Linux OS just went back to the directory prompt without any error message.

And as usual during successful cp command it does this thing.

So cross fingers, I assumed that all the eml files (it could be hundred of them), were successfully copied.

And I deleted the files using the rm command, in order not to waste space or duplicate files.

But to my surprise it takes more than I expected. cp does it in few minutes and rm does it on hours.

Anyway, if user complains time to pull the backup. Thanks for the help.

Last edited by JJJCR; 07-14-2014 at 11:53 PM. Reason: edit
 
Old 07-15-2014, 12:11 AM   #6
sag47
Senior Member
 
Registered: Sep 2009
Location: Orange County, CA
Distribution: Kubuntu x64, Raspbian, CentOS
Posts: 1,832
Blog Entries: 36

Rep: Reputation: 451Reputation: 451Reputation: 451Reputation: 451Reputation: 451
Quote:
Originally Posted by JJJCR View Post
But to my surprise it takes more than I expected. cp does it in few minutes and rm does it on hours.

Anyway, if user complains time to pull the backup. Thanks for the help.
To me it sounds like there is a more serious underlying problem worth investigating. I recommend consulting with peers on the issue or continue hashing it out on LQ until you discover the source of the slow rm operation. What system call did the operation hang up on with strace? What type of file system do the files reside? What type of storage is the filesystem located? Is it a network file share mount? There's a lot more which could be investigated.
 
Old 07-15-2014, 01:37 AM   #7
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,104

Original Poster
Rep: Reputation: 181Reputation: 181
Cool

Quote:
Originally Posted by sag47 View Post
To me it sounds like there is a more serious underlying problem worth investigating. I recommend consulting with peers on the issue or continue hashing it out on LQ until you discover the source of the slow rm operation. What system call did the operation hang up on with strace? What type of file system do the files reside? What type of storage is the filesystem located? Is it a network file share mount? There's a lot more which could be investigated.
haven't run any diagnostic tool, the folder in which the file resides was running under Kerio user email folder.

I'm not really sure whether Kerio services was interfering with the rm command.

I did not try stopping the Kerio service and deleting the files. I remove the files while Kerio service was running.
 
Old 07-15-2014, 07:59 AM   #8
sag47
Senior Member
 
Registered: Sep 2009
Location: Orange County, CA
Distribution: Kubuntu x64, Raspbian, CentOS
Posts: 1,832
Blog Entries: 36

Rep: Reputation: 451Reputation: 451Reputation: 451Reputation: 451Reputation: 451
Quote:
Originally Posted by JJJCR View Post
haven't run any diagnostic tool, the folder in which the file resides was running under Kerio user email folder.

I'm not really sure whether Kerio services was interfering with the rm command.

I did not try stopping the Kerio service and deleting the files. I remove the files while Kerio service was running.
lsof is a good command to list any open file handles. Though in Linux you can delete a file and said file still takes up space as long as there's an open file handle. For that reason deleted files which have open file handles from programs can be recovered from /proc.
 
Old 07-15-2014, 03:24 PM   #9
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,604

Rep: Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241
It really depends on the filesystem.

Most linux native filesystems are quite fast at delete (using a tree or sparse list).

But non-native filesystems aren't so efficient - These usually (not always) store files in a directory as a simple linear file. Deleting a bunch of files in the wrong order (like always deleting the file at the beginning of the directory list) is slow. This is because the directory gets repacked for every delete. This is usually not a problem... unless you have thousands of files in a directory.

Now sometimes it can be due to the deallocation of data blocks - especially on a disk that is 100% full.
 
Old 07-15-2014, 08:18 PM   #10
frankbell
LQ Guru
 
Registered: Jan 2006
Location: Virginia, USA
Distribution: Slackware, Debian, Mint, OpenBSD
Posts: 11,358
Blog Entries: 12

Rep: Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739
How large are the files in question? More details about precisely what you are trying to accomplish might help clarify this: are the files are on the same or separate partitions, in a remote location, etc.

As a general observation, if you are copying a file to another location on the same partition, the file does not actually get rewritten. The pointers to the location of the file are simply changed. That's why the copy process appears to complete so quickly--nothing really gets copied, it just gets a new address.

If you copy a file to another partition or another physical device, that takes some time because the file must be written to the target partition and then removed from the source partition. It's recreating the data that takes time.

Last edited by frankbell; 07-15-2014 at 08:21 PM.
 
1 members found this post helpful.
Old 07-15-2014, 11:05 PM   #11
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,104

Original Poster
Rep: Reputation: 181Reputation: 181
Thumbs up

Quote:
Originally Posted by frankbell View Post
How large are the files in question? More details about precisely what you are trying to accomplish might help clarify this: are the files are on the same or separate partitions, in a remote location, etc.

As a general observation, if you are copying a file to another location on the same partition, the file does not actually get rewritten. The pointers to the location of the file are simply changed. That's why the copy process appears to complete so quickly--nothing really gets copied, it just gets a new address.

If you copy a file to another partition or another physical device, that takes some time because the file must be written to the target partition and then removed from the source partition. It's recreating the data that takes time.
Hi Frankbell, this would explain why copying was very fast. Thank you so much. It's good to know how the system works, but of course it would be the best to know how the developer did it.

Files were copied in the same partition but to a different location only.

Thanks again, Frankbell.
 
Old 07-16-2014, 09:26 PM   #12
sag47
Senior Member
 
Registered: Sep 2009
Location: Orange County, CA
Distribution: Kubuntu x64, Raspbian, CentOS
Posts: 1,832
Blog Entries: 36

Rep: Reputation: 451Reputation: 451Reputation: 451Reputation: 451Reputation: 451
You've still not answered critical questions which would help us help you troubleshoot your filesystem. Can you answer them please?
 
Old 07-16-2014, 09:59 PM   #13
frankbell
LQ Guru
 
Registered: Jan 2006
Location: Virginia, USA
Distribution: Slackware, Debian, Mint, OpenBSD
Posts: 11,358
Blog Entries: 12

Rep: Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739Reputation: 2739
Quote:
Hi Frankbell, this would explain why copying was very fast. Thank you so much. It's good to know how the system works, but of course it would be the best to know how the developer did it.
It's been like that since I first started mucking about with computers with DOS 3.2.

File information is stored in a table at the beginning of each partition. In NTFS, it's called a Master File Table. In Linux file systems, it's called inodes. In other file systems, it might be called something else.

When you move a file on a partition, all that happens is that that table gets changed. There's no reason the physically move the bits and bytes when changing a database entry (the table is ultimately a database) has the same result for, as my father would have said, all practical purposes.

As to who came up with the idea and why, I don't know. It's shrouded in the mists of Unix history.

A web search for "how file systems work" will turn up many useful links, if you want to pursue this further.

Glad I could help.

Last edited by frankbell; 07-16-2014 at 10:00 PM.
 
1 members found this post helpful.
Old 07-16-2014, 11:54 PM   #14
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,104

Original Poster
Rep: Reputation: 181Reputation: 181
Thanks Frankbell, sometimes need to refresh the memory on how things work.
 
Old 12-11-2014, 08:07 PM   #15
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,104

Original Poster
Rep: Reputation: 181Reputation: 181
Smile

Quote:
Originally Posted by sag47 View Post
Unfortunately my crystal ball is in the shop :-). If you're concerned about the integrity of a file copy operation then you can do a few things.

If it's only a single file or a few files then run a checksum on the source/destination (md5sum command or any of the *sum commands). Find command one liner works well here for many files.

If you're copying copious amounts of files then you're better off using a utility which checks the integrity as part of the copying/syncing process. Rsync comes to mind for that.

Code:
rsync -av /src /dst
See rsync man page for option explanation. With rsync you can cancel the copy operation half way through and start it again later; it will pick up where it left off.

If you're possibly using putty over a flakey network consider using a terminal multiplexer such as screen or tmux. If you get disconnected from your session the copy operation will continue and when you connect to the server again you can resume the disconnected session to view the result of said operation.
Thanks sag47.
 
  


Reply

Tags
copying, deleting, suse11


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
bash: search out files by extension, remove spaces, copy elsewhere jamtat Programming 25 10-31-2014 10:30 PM
[SOLVED] ssh-copy-id in reverse? How to remove an authorized account? wh33t Linux - Software 2 12-03-2013 01:50 AM
Opening a remove copy of firefox while a local one is running... assasukasse Linux - Software 3 02-24-2009 11:00 AM
in copy files or ls files the command want to invert select some files how to?? hocheetiong Linux - Newbie 3 06-27-2008 07:32 AM
Can anybody check this copy,remove and chown scripts? sathyguy Linux - Newbie 1 03-19-2007 12:05 AM


All times are GMT -5. The time now is 02:14 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration