Linux - SoftwareThis forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I need to choose a file system for a specific tasks I need to perform to collect data for some research.
The data are raw images, each 0.75MiB coming in at a rate of 30Hz. They are raw images (i.e. completely uncompressed raw coding from the camera) and need to remain as such - we have tried some on-the-fly decoding but it far too intensive for the embedded system that needs to capture the images (PM-1.6GHz EBX board).
I am experimenting with several ways to capturing the data. I have a SATA card, or have external USB HDDs, and I can successfully capture said data (formatted FAT) albeit with a fairly high CPU usage (especially on USB).
I want to lower the CPU usage for capturing the data, and the loading tends to increase when the images are being written to disk.
What file system do you think is optimal for this sort of task? Low-CPU is the prime candidate, and I need little in the way of other features.
when I quickly calculate the rate at which you want it to be captured, I come at 22,5Mb/s. Which is not quiet the maximum throughput of usb 2.0 but close enough; you must have a pretty good system to keep up with that. Personally I would not do such a task on a machine with those specs, if it is reliability/performance you want. I find fire wire (IEEE1394) worked seamless for me on a P4 2.4Ghz 512Mb ram writing to a FAT32 (baring in mind the 2Gb max file size limit) storage partition. the system remained responsive and my rate was 25Hz.
when I quickly calculate the rate at which you want it to be captured, I come at 22,5Mb/s. Which is not quiet the maximum throughput of usb 2.0 but close enough;
Ah yes, I should comment on this a little.
The camera capture programs (corander) actually saves to a ramdisk. Another process actually moves the files from the ramdisk to **two** USB2 drives, alternating between them. It writes up to 900 images per directory, then creates another directory.
In an experiment, I had a SATA (1.5Gbs) cardbus card that I plugged into the system and was able to save directly to same, although it was dropping the off frame (haven't tried yet doing the same thing via the ramdisk as described above).
I can actually do that with two cameras (one at 15Hz, the other at 30Hz), with two instances of coriander, each saving to a different SATA drive as a proof of comcept.
It was formatted FAT32 so that (at the time) it could be directly read by anyone with Windows, but that's no longer the issue, so I'm too see if there is anything better for the same purpose.
These cameras can run for an hour or more at time and can capture enormous amounts of data - it was close to 200GB in our last field test.
I hear ext4 is promising in its capabilities like stability, reliability and performance. I also read it is quicker than its predecessor ext3 and it will be the default fs for the upcoming new release of Ubuntu which should be due for download soon.
and yes, capturing full uncompressed frames needs a lot of hard drive space and very consistent and high bus speeds accompanied with a fair amount of continues cpu use. imho you would not be able to do much else with that system while it does this unless you have multiple cores.
I would assume that a non journaling system like ext2 would be quicker, since I guess this must take processor power. Only valid if you don't feel you need journaling of course.
I would assume that a non journaling system like ext2 would be quicker, since I guess this must take processor power. Only valid if you don't feel you need journaling of course.
ext2 did cross my mind. Journaling is not necessary.
How does ext2 perform against more esoteric file systems such as jfs (which is reputed to have low cpu usage), especially considering the job I want to do.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.