LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 07-26-2008, 07:58 PM   #1
pentalive
Member
 
Registered: Jun 2005
Location: Sacramento, CA
Distribution: Many, Old and New
Posts: 124

Rep: Reputation: 15
dd Starts fast (10Mb/s) but slows down after 30gb (2.4mb/s)


I am asking here because the target drive in this case was a samba share on my file server.

I was using the "Partimage Is Not Ghost" PING distro to run my laptop. It was connected by 100baset via a switch to my fileserver (also running at 100 base T)

Once the SMB share was mounted I used this command to copy:

dd if=/dev/sda1 of=bk0726sda1 &

While I was in the mount point directory.

at times I did a kill -USR1 <pid> to see the IO stats..

The mb/s transfer rate started out above 10mb/s (for a backup of about 5 hours) but slowly dropped until it reached 2.4 mb/s after about 30gb.

The source partition is sda1 is over 200gb. It's the windows partition of my dual boot laptop.

Why the slowdown?

----------------- new information

I hooked up a usb2 external drive and dd my hard disk to it, it ran at a steady 6.8 mb/s the whole time with no slowdown.

still using ping (partimage is not ghost) shell and the dd command. No network, No Samba. Also I was:

dd if=sda of=sdb

instead of

dd if=sda1 of=\mnt\point\subdir\file

(I was going to do each partition separatly)


Well?

: ^ )

Last edited by pentalive; 07-27-2008 at 05:19 PM. Reason: New Information to add!
 
Old 07-27-2008, 07:28 PM   #2
stress_junkie
Senior Member
 
Registered: Dec 2005
Location: Massachusetts, USA
Distribution: Ubuntu 10.04 and CentOS 5.5
Posts: 3,873

Rep: Reputation: 331Reputation: 331Reputation: 331Reputation: 331
IMO 10 MB/second is slow, and slowing down to 2.4 MB/second is a sure sign of a failing hard drive.
 
Old 07-27-2008, 07:40 PM   #3
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 59
Experiment with different block sizes; you'll find a dramatic change in your disk to disk DD.

Try bonnie/bonnie++ to test out your file system performance. It will vary the blocksize and defeat any buffer cache affects.

Many people experience 10Mbit/sec Samba transfer performance; I don't see this, or a slowdown at large file sizes to be any indication of a disk problem. We also don't know anything about the fragmentation level of the Windows disk, nor its format. I don't know why performance suffers after large transfers, but I could hazard some guesses.
 
Old 07-27-2008, 08:19 PM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,271

Rep: Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028
Larger blksize would obviously help, but this has to be SMB. Sounds like the classic fill the funnel (fast), then the rate of (sustained) "top-up" is constrained by the rate at which it empties.
Sounds like a tuning opportunity there rather than at the disk itself.
 
Old 07-27-2008, 08:24 PM   #5
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 59
At slowdowns at or near 30G, that's an enormous funnel.
 
Old 07-27-2008, 09:17 PM   #6
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,269

Rep: Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028
Maybe the tgt disk starts to get nearly full/fragmented at that point, so it has to work harder to find diskblocks to write to.
 
Old 07-27-2008, 09:21 PM   #7
pentalive
Member
 
Registered: Jun 2005
Location: Sacramento, CA
Distribution: Many, Old and New
Posts: 124

Original Poster
Rep: Reputation: 15
Slowing for bad hard drive

Quote:
Originally Posted by stress_junkie View Post
IMO 10 MB/second is slow, and slowing down to 2.4 MB/second is a sure sign of a failing hard drive.
I know it probably does not matter but it is a brand new hard drive.

Also, I ran the backup twice (once as a device to file, once as a partition to file) and both times it slowed and both time it started "fast"
 
Old 07-27-2008, 09:23 PM   #8
pentalive
Member
 
Registered: Jun 2005
Location: Sacramento, CA
Distribution: Many, Old and New
Posts: 124

Original Poster
Rep: Reputation: 15
Disk full

Quote:
Originally Posted by chrism01 View Post
Maybe the tgt disk starts to get nearly full/fragmented at that point, so it has to work harder to find diskblocks to write to.
500gb mostly empty drive for target

250gb mostly empty drive for source

I know dd would copy even the empty parts...
 
Old 07-27-2008, 09:25 PM   #9
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 59
That's what I was thinking too, but didn't want to spend much time guessing. With a virgin FS, I can't see any reason why > 2.4meg/s can't be achieved.

I'd like to see the OP perform more controlled benchmarks both read-side, and write-side. Finding the knee in the performance curves can be revealing. Certainly something better than sending a signal to dd.
 
Old 07-27-2008, 11:50 PM   #10
pentalive
Member
 
Registered: Jun 2005
Location: Sacramento, CA
Distribution: Many, Old and New
Posts: 124

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Mr. C. View Post
That's what I was thinking too, but didn't want to spend much time guessing. With a virgin FS, I can't see any reason why > 2.4meg/s can't be achieved.

I'd like to see the OP perform more controlled benchmarks both read-side, and write-side. Finding the knee in the performance curves can be revealing. Certainly something better than sending a signal to dd.
What do you make of the mb/s remaining steady with the external USB2 disk drive? 6.3mb/s the whole time... no slowdown.

Could something about the way Samba works cause longer or more seeks as the dd progresses?

What would you suggest for testing?

Last edited by pentalive; 07-27-2008 at 11:51 PM. Reason: missed original question...
 
Old 07-28-2008, 12:37 AM   #11
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 59
6.3mb/sec is very slow.

$ dd if=/dev/zero of=/home/mrc/tmp/BIG bs=4096 count=100000
100000+0 records in
100000+0 records out
409600000 bytes transferred in 7.445 secs (55016789 bytes/sec)

You have a number of components when you are testing Samba. You are testing the performance of:

source disk and file system reads
target disk writes
network
samba
target file system writes over samba

and must consider system activity during benchmarking.

Now, I've seen all too many times where users have had poor Samba performance, and it turns out to be a problem in any of the above areas.

Last edited by Mr. C.; 07-28-2008 at 12:40 AM.
 
Old 07-28-2008, 01:25 AM   #12
pentalive
Member
 
Registered: Jun 2005
Location: Sacramento, CA
Distribution: Many, Old and New
Posts: 124

Original Poster
Rep: Reputation: 15
There was no other system activity in either case - except and occasional kill -USR1.

And it's not the specific speed I am asking about - it's the steady drop in speed over Ethernet
and samba vs. the steady speed over usb2. The specific speed over USB2 (6.8 mb/s), I used to know
the speeds of USB1 and 2 and it may be that the drive is actually only capable of USB1..

Well I was just curios anyway - I guess it's just mysterious.

Thanks for the suggestions and help.
 
Old 07-28-2008, 01:31 AM   #13
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,271

Rep: Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028
If you have access to a 2.6.25+ kernel with latencytop selected, there is an associated userspace tool that provides some handy numbers. Not sure what you'll get for SMB shares, but might be worth a try.
 
Old 07-28-2008, 02:08 AM   #14
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 59
Given that USB 1 has a 1.5 or 12 Mbit max speed, there's no way that is a USB1 device.

I understand and know what you're after, but don't have an (easy) answer for you.

I understand you are trying to figure out why there is an apparent performance drop...somewhere (you think it is only in very large Samba transfers). But what actual data do we have to go on? How can anyone here draw any conclusions?

We do see that the best performance you get is about 6.8M/sec. How can Samba give you > 10M/sec then? This alone tells me you testing is problematic.
 
Old 07-28-2008, 02:31 AM   #15
pentalive
Member
 
Registered: Jun 2005
Location: Sacramento, CA
Distribution: Many, Old and New
Posts: 124

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Mr. C. View Post
Given that USB 1 has a 1.5 or 12 Mbit max speed, there's no way that is a USB1 device.

I understand and know what you're after, but don't have an (easy) answer for you.

I understand you are trying to figure out why there is an apparent performance drop...somewhere (you think it is only in very large Samba transfers). But what actual data do we have to go on? How can anyone here draw any conclusions?

We do see that the best performance you get is about 6.8M/sec. How can Samba give you > 10M/sec then? This alone tells me you testing is problematic.
The 6.8m/s is over USB only. It's steady and it removes the network and samba and the server from the event.

I understand I have not given you much to go on - it's all I have. Unless you want the six specific data points I wrote down. They consist of bytes copies and seconds since start. I don't know how to test it any other way.

I guess I was hoping someone else had seen this too, and perhaps even overcome it.

We can call it closed now.. Thanks
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Xterm (any X terminal) starts fast, but initializes very slow Kevschef Linux - General 3 11-09-2006 12:36 PM
10Mb/s and 100Mb/s on same network plisken General 2 05-07-2003 10:28 AM
Onstream IDE 30GB Installation jeucken Linux - Hardware 2 08-01-2002 12:53 PM
Harddisk problems, seems to forget about the last 30Gb... Thymox General 7 05-21-2002 10:15 AM
30gb Linux Installation vcheah Linux - Software 2 12-06-2001 07:28 PM


All times are GMT -5. The time now is 07:00 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration