LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-13-2015, 06:25 AM   #1
Abdul Basit
LQ Newbie
 
Registered: Apr 2015
Posts: 10

Rep: Reputation: Disabled
Smile Performance of fwrite() on usb mass storage device?


Hi,
I am trying to write to usb flash drive using a C program. I have just created some files in USB device and doing my tasks using fopen,fread and fwrite on those files. I was just wondering what is the standard data rate that can be achieved using fwrite() on a file in usb. Secondly what is better way of transferring data to USB flash drive using C Program? Obviously I am a newbie

Regards
AB
 
Old 05-13-2015, 07:04 AM   #2
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,602

Rep: Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241
It really depends on what you are doing.

A USB storage device is treated no differently than any other storage device... Disks are disks.

It usually is fastest to use the largest buffer...

But it sucks for random block I/O, which is much better in small sizes.

If all you are doing is copying a file to a USB flash drive, how about just using cp?
 
Old 05-13-2015, 07:13 AM   #3
Abdul Basit
LQ Newbie
 
Registered: Apr 2015
Posts: 10

Original Poster
Rep: Reputation: Disabled
Thanks for instant reply

thanks jpollard,
My code reads data in the form a kernel module in the form of 4KB chunks and these chunks are written to files in USB. Expected data rate is 32MB/s and I am using Usb3.0 flash drive. I was wondering whether fwrite() can serve the purpose?

Regards
 
Old 05-13-2015, 08:34 AM   #4
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,602

Rep: Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241
Why not just use cp?

4KB chunks may not be the best size - 8/16/32k may be better, but that depends on the filesystem and the device...

Whether you use fwrite/write or anything else, is up to you. No single one is better.
 
Old 05-13-2015, 08:53 AM   #5
Abdul Basit
LQ Newbie
 
Registered: Apr 2015
Posts: 10

Original Poster
Rep: Reputation: Disabled
Thanks,

My implementation is i.e. there are 4 files in USB disk.
file0.txt
file1.txt
file2.txt
file3.txt

each time data from kernel module arrives in form of 4KB it has to be appended to a file (in round robin pattern). So I guess cp might not work as it will replace the file with new file. 'cp' takes directory locations (files) as parameter where I have only buffers in my program that will be copied to files.Also each time 'cp' is called fopen and fclose will also be called that might slow down overall operation. So I think I have only choice of fwrite() as per my understanding (which might be questionable ).
 
Old 05-13-2015, 09:20 AM   #6
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,602

Rep: Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241
The speed of fwrite is irrelevent.

Concatenating modules doesn't exactly make any sense - they can't be used that way.

The problem of speed depends on the device. As I understand it, the flash is divided up into rather large blocks - to add a single 4K block to a file requires reading up to 64k, replacing the one 4K update, then writing the 64K somewhere else (wear leveling). So 4K could be rather bad on the device.

And that is much slower than a cp would be, and a lot less overhead (and wear and tear).

But you can use whatever you want. The system will handle it.
 
Old 05-13-2015, 01:10 PM   #7
exvor
Senior Member
 
Registered: Jul 2004
Location: Phoenix, Arizona
Distribution: Gentoo, LFS, Debian,Ubuntu
Posts: 1,537

Rep: Reputation: 87
Data rates on flash media vary greatly and it depends on the type of write your attempting to do. Since these are 4k blocks your probably doing non sequential writes to the device. In that case then writes will be rather slower in magnitude to reads. USB flash drives are probably the worst for performance in this area considering they are usually a low cost solution. However there is variance even in them with regards to quality of flash memory used.

Like jpollard said writing data to the device in 4k blocks is probably a bad idea because of the general wear and tear on the device ( why putting a swap file on flash is also not good ).

You could however instead of writing to a file could create a buffer for the data until it reaches a particular size and then dump to the disk when full or whatnot. Several ways that could be implemented.
 
  


Reply

Tags
file operators, usb disk


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Mass storage device USB xeebeeeeeee Linux - Hardware 10 02-13-2008 01:11 PM
USB Mass storage Device purefan Linux - Hardware 15 06-06-2005 02:56 PM
usb - mass storage device d1l2w3 Mandriva 7 01-10-2005 11:10 PM
USB Mass Storage Device ebasi Linux - Hardware 2 12-23-2004 03:06 PM
Usb mass storage device laguna Linux - Hardware 4 08-12-2002 03:16 AM


All times are GMT -5. The time now is 04:58 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration