LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 09-14-2013, 12:13 PM   #1
hesisaboury
Member
 
Registered: Dec 2010
Location: iran
Posts: 199

Rep: Reputation: 0
File size increase after copy(scp,rsync)


Hello,
i was trying backup from my virtual machine (openxen) using rsync , after backup i used du -sh at destination and source , unbelievably amount of disk usage at destination was double time of amount of data at source, i checked , for duplication data , nothing found ,here is my situation
source :
linux lenny , ext3 ,
destination:
linux os , ext4 (tested with ext3)
command :
rsync -a -v --delete -e ssh /mbox root@IP:/mbox
also used
scp -r /mbox root@IP:/mbox

i used ::: du -sh ::: df -h ::: for disk usage

Any Idea....
Thanks

Last edited by hesisaboury; 09-14-2013 at 12:14 PM.
 
Old 09-14-2013, 12:32 PM   #2
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,801

Rep: Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223
Without more detail I can only guess, but perhaps the source had one or more sparse files. By default, rsync does not try to preserve sparseness (use the "-S" or "--sparse" option to do that). I don't see any option in scp to preserve sparseness.
 
Old 09-15-2013, 12:56 PM   #3
hesisaboury
Member
 
Registered: Dec 2010
Location: iran
Posts: 199

Original Poster
Rep: Reputation: 0
Hi,
What details do u need rknichols?
i dont know what is going on , may be linux bug , the where i'm working used linux lenny 5.0.4 for most their projects, when i check some servers i see "du -sh" result has different value from"df -h" , it makes a big problem,

Last edited by hesisaboury; 09-15-2013 at 12:59 PM.
 
Old 09-15-2013, 04:37 PM   #4
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,801

Rep: Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223
Running du and df on a filesystem will very often yield different results. The du command walks through the directory tree and reports/sums the sizes of what it finds. The df command directly queries the filesystem and reports the number of blocks in use. Why would these be different? The most common cause is files that have been deleted from the directory tree but are still held open by some process(es). That would cause df to report higher usage than what du can see.

The other common cause is files that exist on the filesystem but are hidden under an active mount point. Let's say you decide to move /home from the root filesystem to a separate partition. You copy the current contents of /home to the new partition, and then mount the new partition on /home. All looks well, and /home now has lots of space from the new, presumably larger, partition. The catch is that you never deleted the files from the root filesystem. They still occupy space there, but "du --one-file-system" won't see them on the root partition while the new /home is mounted there.

I don't know whether either of these applies to your situation. Initially you reported different results for "du -sh" at the source and destination, which suggests a completely different issue. You would need to look at the output from "du --max-depth=N", with gradually increasing values for N, to try to see where the differences occur.
 
Old 09-15-2013, 08:45 PM   #5
jefro
Moderator
 
Registered: Mar 2008
Posts: 22,229

Rep: Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652Reputation: 3652
Post the exact command you used and include where you ran it from.
 
Old 09-29-2013, 11:53 AM   #6
dt64
Member
 
Registered: Sep 2012
Distribution: RHEL5/6, CentOS5/6
Posts: 218

Rep: Reputation: 38
Quote:
Originally Posted by man du
The du utility displays the file system block usage for each file argument and for each directory in the file hierarchy rooted in each directory argument.
Du you have same file systems, block size etc on all your systems involved?
Block size usage and file size are not exactly the same. If you have a small text file with e.g. 20 bytes, it would still use a full block (e.g. 4096 bytes).
 
Old 09-29-2013, 02:36 PM   #7
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,573

Rep: Reputation: 2143Reputation: 2143Reputation: 2143Reputation: 2143Reputation: 2143Reputation: 2143Reputation: 2143Reputation: 2143Reputation: 2143Reputation: 2143Reputation: 2143
Quote:
Originally Posted by dt64 View Post
Du you have same file systems, block size etc on all your systems involved?
Block size usage and file size are not exactly the same. If you have a small text file with e.g. 20 bytes, it would still use a full block (e.g. 4096 bytes).
^ this

And I noticed you didn't have -H set in your rsync command, which means any hard links on the source will be duplicated on the destination, resulting in double, triple, quadruple, etc. the disk usage (depending on how many hard links you had on the source).

Last edited by suicidaleggroll; 09-29-2013 at 02:38 PM.
 
Old 09-30-2013, 01:49 AM   #8
hesisaboury
Member
 
Registered: Dec 2010
Location: iran
Posts: 199

Original Poster
Rep: Reputation: 0
Hi,
source location is /mbox and it has only mail in it's directories , there is no hard link there , the strange thing is the size of folder inside and outside of that , different size , for example :
du -sh folder == 60M
cd folder
du -dh * ==> 40M

Thanks suicidaleggroll
 
Old 09-30-2013, 09:04 AM   #9
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,801

Rep: Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223Reputation: 2223
There are 2 possibilities. Either (a) the directory file itself is huge (unlikely, unless you have tens of thousands of users), or (b) there are dotfiles in that directory (the expansion of "*" will not include any filenames that begin with ".").
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
I think I need rsync or scp -- something to copy from http://... to /somewhere/else nithra Linux - Server 2 01-18-2011 07:09 AM
[SOLVED] increase a file size? Gibzon Linux - Newbie 10 12-02-2010 07:20 AM
increase the size of a file? Gibzon Linux - General 2 12-01-2010 05:25 AM
file size increase schamana AIX 5 09-18-2007 03:44 PM
SCP Copy File SSH petenyce Linux - Wireless Networking 2 10-05-2005 04:11 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 09:50 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration