LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices


Reply
  Search this Thread
Old 10-09-2017, 07:06 PM   #1
usao
Member
 
Registered: Dec 2011
Location: Chandler, AZ
Posts: 283

Rep: Reputation: Disabled
How to disable filesystem buffer for a single filesystem?


I have a 2TB SSD drive which im writing data to using 'scp'.
The 'scp' command finished but in trynig to umount the drive, I found that I had to run a 'sync' command to flush the buffers.
The disk remained at 100% utilization for almost 1.5 hours while the sync command ran.
Unfortunately, this caused me to miss a window for shipping the drive out.

So, how can I disable the filesystem buffer to prevent this crazylong sync command?
I would much rather have a slower transfer rate but be able to get the drive out of the system on schedule than have faster transfer and miss my delivery.
 
Old 10-09-2017, 07:26 PM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 17,077

Rep: Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609
Short answer - you can't.
How is the SSD connected ?.
 
Old 10-09-2017, 07:45 PM   #3
usao
Member
 
Registered: Dec 2011
Location: Chandler, AZ
Posts: 283

Original Poster
Rep: Reputation: Disabled
via USB 2.0, the drive is a Samsung 2tb consumer drive with a usb adapter.
This is for our offsite backups, so pick-up tome is important.

Code:
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdg               0.00     0.00    0.20   54.80     0.00    27.40  1020.31   141.10 2569.24  601.00 2576.42  18.18 100.00

Last edited by usao; 10-09-2017 at 07:53 PM.
 
Old 10-09-2017, 07:52 PM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 17,077

Rep: Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609Reputation: 2609
Thought so - in that case you can do some things to help. Read this. It's old, but still relevant.
 
Old 10-09-2017, 08:16 PM   #5
usao
Member
 
Registered: Dec 2011
Location: Chandler, AZ
Posts: 283

Original Poster
Rep: Reputation: Disabled
That's conceivable since the box has 1tb ram...
 
Old 10-09-2017, 08:48 PM   #6
michaelk
Moderator
 
Registered: Aug 2002
Posts: 17,596

Rep: Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325
I might be over simplifying things but USB 2.0 is 480Mbps. How much data are you writing to the drive? If you are writing 300GB then it will take about 1.5 hours. You can easily calculate the approx time it will take to transfer and write to the drive so you can plan accordingly.
 
Old 10-09-2017, 09:28 PM   #7
usao
Member
 
Registered: Dec 2011
Location: Chandler, AZ
Posts: 283

Original Poster
Rep: Reputation: Disabled
Well, I get about 27 MB/s or just over 50% theoretical bandwidth.
Ive been adjusting the dirty_bites downward slowly to avoid stalling the ssh copy.
Got it down to 90GB and heading down to 10GB.

Code:
[root@hp6 vm]# for x in $(seq 800 -1 100); do echo ${x}00000000 > dirty_bytes; sleep 10; done
Seems to be working, the I/O throughput is slower but not stalling, which is important as resuming a scp command is a problem because it doesnt purge copied files and doesnt go in alphabetical order either, so its difficult to restart. Once the files are transferred I can go back and generate a list of files to delete, then swap drives and resume the scp.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
How to disable filesystem cache? nawwar Linux - Kernel 20 01-22-2016 03:49 AM
New Linux filesystem: bcachefs - a general purpose COW filesystem jeremy Linux - News 1 08-21-2015 12:19 PM
script to move file from local filesystem to hadoop filesystem callis07 Linux - Enterprise 4 07-19-2014 08:37 AM
LXer: SSHFS(Secure SHell FileSystem): Securely Mount remote filesystem over ssh LXer Syndicated Linux News 0 11-13-2013 07:30 PM
DISCUSSION: Virtual Filesystem: Building a Linux Filesystem from an Ordinary File mchirico LinuxAnswers Discussion 0 10-28-2004 10:35 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel

All times are GMT -5. The time now is 10:11 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration