Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum. |
Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
04-15-2009, 03:37 AM
|
#1
|
LQ Newbie
Registered: Dec 2008
Posts: 17
Rep:
|
Selection of a file system for a specific task
I need to choose a file system for a specific tasks I need to perform to collect data for some research.
The data are raw images, each 0.75MiB coming in at a rate of 30Hz. They are raw images (i.e. completely uncompressed raw coding from the camera) and need to remain as such - we have tried some on-the-fly decoding but it far too intensive for the embedded system that needs to capture the images (PM-1.6GHz EBX board).
I am experimenting with several ways to capturing the data. I have a SATA card, or have external USB HDDs, and I can successfully capture said data (formatted FAT) albeit with a fairly high CPU usage (especially on USB).
I want to lower the CPU usage for capturing the data, and the loading tends to increase when the images are being written to disk.
What file system do you think is optimal for this sort of task? Low-CPU is the prime candidate, and I need little in the way of other features.
|
|
|
04-15-2009, 10:02 PM
|
#2
|
Member
Registered: Oct 2006
Location: brisbane - australia
Distribution: ubuntu
Posts: 335
Rep:
|
when I quickly calculate the rate at which you want it to be captured, I come at 22,5Mb/s. Which is not quiet the maximum throughput of usb 2.0 but close enough; you must have a pretty good system to keep up with that. Personally I would not do such a task on a machine with those specs, if it is reliability/performance you want. I find fire wire (IEEE1394) worked seamless for me on a P4 2.4Ghz 512Mb ram writing to a FAT32 (baring in mind the 2Gb max file size limit) storage partition. the system remained responsive and my rate was 25Hz.
|
|
|
04-15-2009, 10:07 PM
|
#3
|
Senior Member
Registered: Jan 2006
Posts: 4,363
Rep: 
|
How much data are we talking about in total? sub 4GB? I suspect this will be more about the storage device than it will be about the format.
|
|
|
04-16-2009, 12:41 AM
|
#4
|
LQ Newbie
Registered: Dec 2008
Posts: 17
Original Poster
Rep:
|
Quote:
Originally Posted by wabbalee
when I quickly calculate the rate at which you want it to be captured, I come at 22,5Mb/s. Which is not quiet the maximum throughput of usb 2.0 but close enough;
|
Ah yes, I should comment on this a little.
The camera capture programs (corander) actually saves to a ramdisk. Another process actually moves the files from the ramdisk to **two** USB2 drives, alternating between them. It writes up to 900 images per directory, then creates another directory.
In an experiment, I had a SATA (1.5Gbs) cardbus card that I plugged into the system and was able to save directly to same, although it was dropping the off frame (haven't tried yet doing the same thing via the ramdisk as described above).
I can actually do that with two cameras (one at 15Hz, the other at 30Hz), with two instances of coriander, each saving to a different SATA drive as a proof of comcept.
It was formatted FAT32 so that (at the time) it could be directly read by anyone with Windows, but that's no longer the issue, so I'm too see if there is anything better for the same purpose.
These cameras can run for an hour or more at time and can capture enormous amounts of data - it was close to 200GB in our last field test.
|
|
|
04-16-2009, 01:14 AM
|
#5
|
Member
Registered: Oct 2006
Location: brisbane - australia
Distribution: ubuntu
Posts: 335
Rep:
|
I hear ext4 is promising in its capabilities like stability, reliability and performance. I also read it is quicker than its predecessor ext3 and it will be the default fs for the upcoming new release of Ubuntu which should be due for download soon.
and yes, capturing full uncompressed frames needs a lot of hard drive space and very consistent and high bus speeds accompanied with a fair amount of continues cpu use. imho you would not be able to do much else with that system while it does this unless you have multiple cores.
|
|
|
04-16-2009, 01:29 AM
|
#6
|
LQ Guru
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678
Rep: 
|
I would assume that a non journaling system like ext2 would be quicker, since I guess this must take processor power. Only valid if you don't feel you need journaling of course.
|
|
|
04-16-2009, 02:02 AM
|
#7
|
Member
Registered: Oct 2006
Location: brisbane - australia
Distribution: ubuntu
Posts: 335
Rep:
|
It did cross my mind.
|
|
|
04-16-2009, 03:15 AM
|
#8
|
LQ Newbie
Registered: Dec 2008
Posts: 17
Original Poster
Rep:
|
Quote:
Originally Posted by billymayday
I would assume that a non journaling system like ext2 would be quicker, since I guess this must take processor power. Only valid if you don't feel you need journaling of course.
|
ext2 did cross my mind. Journaling is not necessary.
How does ext2 perform against more esoteric file systems such as jfs (which is reputed to have low cpu usage), especially considering the job I want to do.
Keep it coming 
|
|
|
04-16-2009, 06:43 PM
|
#9
|
Member
Registered: Oct 2006
Location: brisbane - australia
Distribution: ubuntu
Posts: 335
Rep:
|
imo the best and quickest way to find out is to try and see for yourself on a dedicated drive/partition.
|
|
|
All times are GMT -5. The time now is 10:45 AM.
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|