LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 04-15-2009, 03:37 AM   #1
damien_d
LQ Newbie
 
Registered: Dec 2008
Posts: 17

Rep: Reputation: 0
Selection of a file system for a specific task


I need to choose a file system for a specific tasks I need to perform to collect data for some research.

The data are raw images, each 0.75MiB coming in at a rate of 30Hz. They are raw images (i.e. completely uncompressed raw coding from the camera) and need to remain as such - we have tried some on-the-fly decoding but it far too intensive for the embedded system that needs to capture the images (PM-1.6GHz EBX board).

I am experimenting with several ways to capturing the data. I have a SATA card, or have external USB HDDs, and I can successfully capture said data (formatted FAT) albeit with a fairly high CPU usage (especially on USB).

I want to lower the CPU usage for capturing the data, and the loading tends to increase when the images are being written to disk.

What file system do you think is optimal for this sort of task? Low-CPU is the prime candidate, and I need little in the way of other features.
 
Old 04-15-2009, 10:02 PM   #2
wabbalee
Member
 
Registered: Oct 2006
Location: brisbane - australia
Distribution: ubuntu
Posts: 335

Rep: Reputation: 38
when I quickly calculate the rate at which you want it to be captured, I come at 22,5Mb/s. Which is not quiet the maximum throughput of usb 2.0 but close enough; you must have a pretty good system to keep up with that. Personally I would not do such a task on a machine with those specs, if it is reliability/performance you want. I find fire wire (IEEE1394) worked seamless for me on a P4 2.4Ghz 512Mb ram writing to a FAT32 (baring in mind the 2Gb max file size limit) storage partition. the system remained responsive and my rate was 25Hz.
 
Old 04-15-2009, 10:07 PM   #3
lazlow
Senior Member
 
Registered: Jan 2006
Posts: 4,363

Rep: Reputation: 172Reputation: 172
How much data are we talking about in total? sub 4GB? I suspect this will be more about the storage device than it will be about the format.
 
Old 04-16-2009, 12:41 AM   #4
damien_d
LQ Newbie
 
Registered: Dec 2008
Posts: 17

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by wabbalee View Post
when I quickly calculate the rate at which you want it to be captured, I come at 22,5Mb/s. Which is not quiet the maximum throughput of usb 2.0 but close enough;
Ah yes, I should comment on this a little.

The camera capture programs (corander) actually saves to a ramdisk. Another process actually moves the files from the ramdisk to **two** USB2 drives, alternating between them. It writes up to 900 images per directory, then creates another directory.

In an experiment, I had a SATA (1.5Gbs) cardbus card that I plugged into the system and was able to save directly to same, although it was dropping the off frame (haven't tried yet doing the same thing via the ramdisk as described above).

I can actually do that with two cameras (one at 15Hz, the other at 30Hz), with two instances of coriander, each saving to a different SATA drive as a proof of comcept.

It was formatted FAT32 so that (at the time) it could be directly read by anyone with Windows, but that's no longer the issue, so I'm too see if there is anything better for the same purpose.

These cameras can run for an hour or more at time and can capture enormous amounts of data - it was close to 200GB in our last field test.
 
Old 04-16-2009, 01:14 AM   #5
wabbalee
Member
 
Registered: Oct 2006
Location: brisbane - australia
Distribution: ubuntu
Posts: 335

Rep: Reputation: 38
I hear ext4 is promising in its capabilities like stability, reliability and performance. I also read it is quicker than its predecessor ext3 and it will be the default fs for the upcoming new release of Ubuntu which should be due for download soon.

and yes, capturing full uncompressed frames needs a lot of hard drive space and very consistent and high bus speeds accompanied with a fair amount of continues cpu use. imho you would not be able to do much else with that system while it does this unless you have multiple cores.
 
Old 04-16-2009, 01:29 AM   #6
billymayday
LQ Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
I would assume that a non journaling system like ext2 would be quicker, since I guess this must take processor power. Only valid if you don't feel you need journaling of course.
 
Old 04-16-2009, 02:02 AM   #7
wabbalee
Member
 
Registered: Oct 2006
Location: brisbane - australia
Distribution: ubuntu
Posts: 335

Rep: Reputation: 38
It did cross my mind.
 
Old 04-16-2009, 03:15 AM   #8
damien_d
LQ Newbie
 
Registered: Dec 2008
Posts: 17

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by billymayday View Post
I would assume that a non journaling system like ext2 would be quicker, since I guess this must take processor power. Only valid if you don't feel you need journaling of course.
ext2 did cross my mind. Journaling is not necessary.

How does ext2 perform against more esoteric file systems such as jfs (which is reputed to have low cpu usage), especially considering the job I want to do.

Keep it coming
 
Old 04-16-2009, 06:43 PM   #9
wabbalee
Member
 
Registered: Oct 2006
Location: brisbane - australia
Distribution: ubuntu
Posts: 335

Rep: Reputation: 38
imo the best and quickest way to find out is to try and see for yourself on a dedicated drive/partition.
 
  


Reply

Tags
file, performance, system


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
File system selection for a volatile drive (ext2 or tmpfs?) agkunkle Linux - Embedded & Single-board computer 5 04-18-2009 12:00 PM
wxPython: file browser with multiple file selection: possible? BrianK Programming 0 01-11-2008 05:46 PM
file selection munna_dude Programming 0 03-23-2007 04:08 AM
Users, groups, permissions - specific task question Doctafonk Linux - General 2 04-21-2004 09:05 PM
Operating system selection screen BigBrittDOGG Linux - Software 3 08-10-2003 06:19 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 09:39 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration