Dear Forum Members:

I've got a problem with a linux-based data-acquisition and
processing system I'm developing. Before describing the problem,
I'm going to provide some details of the host linux system.

------------------------------------------------------------------
* Computer Type: Dell Precision 690, 64-bit dual-core Xeon processor,
32 GB physical memory.
* OS: RHEL 4, kernel 2.6.9-78.0.8.ELsmp #1 SMP
* Compilers: gcc and Intel's icc
* Name: whisper2
* Data Acquisition Board: ICS 554B, 100 MHz max ADC rate, with four
Graychip digital downconverters. Output complex sampling rates
from 24.414 kHz to 12.5 MHz in dyadic steps.
* External Mass Storage: Wiebetech 4 TB SCSI RAID, operating as a RAID
0 device, with 512 MB main memory, and a single Volume Group (4 TB)
and a single Cache Volume (427 MB).
* Interface to RAID: VMIC UltraSCSI 320 SCSI controller
------------------------------------------------------------------

The whisper2 computer is used to acquire blocks of data and process
them with my signal processing algorithms for signal detection and
parameter estimation. In this mode of operation, real-time processing
is not required. Blocks of data are collected and processed, and the
processing time of a block dictates the time of the next collected block.

whisper2 is also used to collect large coherent (no missed
samples) blocks of data and store them on the external RAID device.
The OS is not a real-time OS. The performance measure for streaming long
data records to external RAID is whether or not the internal sample
buffer on the ICS ADC device overflows. That is, whether or not the
system can keep up with the incoming samples.

This works well for all programmable sampling rates up to 6.25 MHz.
For the fastest rate of 12.5 MHz (12.5 million complex samples per
second, each complex sample comprises six bytes), the internal
ADC buffer overflows. Streaming the data to an internal SCSI disk
produces much fewer overflows than streaming to the external SCSI RAID
device.

My problem is to adjust the cached linux I/O parameters and the
multithreaded acquisition-program parameters in order to minimize the
number of overflows for the 12.5 MHz rate.

Red Hat has suggested some values for the cached I/O system in linux,
and I've set those parameters, but it hasn't helped much. Wiebetech
doesn't know what to do.

I've also reverted the filesystem type for both the internal SCSI
disk and the external SCSI RAID from ext3 to ext2 to remove journaling
from the equation.

My question is how to get the highest continuous data-streaming rate
without taking the step of using an official real-time linux OS.
I'm hoping to get some help from the forum, or a pointer to a more
relevant forum.

Thanks in advance for any light you can shed.

Chad

Sounds like about 75MB per sec of data throughput.
How much data needs to be read in this manner at one time, or in one session? Would it be possible to have a very large ram disk to write to, and then write out the ram disk at a time when the sampling was not being done?

Alex:

Thanks for the reply. You got the basic rate right: 12.5e6 * 6 = 75 MB/s.

I can get by with about 1000 seconds of coherent collection. I would
like, however, to be able to stream to the external RAID at that rate
indefinitely, limited only by the available space left on the device.

So, the RAM disk idea is not quite practical (1000 s * 75 MB/s = 75 GB),
although I can fit 64 GB of memory into the host, giving a collection
time of approximately 14 minutes (860 s). I will have to resort to this
if I can't get the streaming-to-RAID to work.

In fact, the mode of operation in which I collect a block then process
then collect another, etc., writes the data to memory, and this generally
works with very few ADC overflows. So, you are sync'ed to my problem,
I think. What I want is the ability to control linux's cached I/O
subsystem so that it favors sustained high-speed writes over its
complex caching system.

When I use simple C programs to measure the write speed to the SCSI
RAID device, I get something between 50 and 100 MB/s, although I am
expecting more like 300 MB/s.

Any other ideas?

Thanks very much,

Chad

You might try an XFS filesystem on your array.

Quakeboy02:

Thanks for the interesting tip. I see that XFS is not a part of RHEL 4 or 5,
but can possibly be added. Do you have experience with the "Direct I/O"
and "Guaranteed Rate I/O" features?

Chad

What is the results of "free"? Meaning that the amount of ram available to cache is more important than the total amount of ram.

What are the results of " hdparm -tT /dev/sdb"? Adjust sdb to whatever drive you are using. Unless you have a configuration/driver problem, with your raid setup I would expect to see numbers well over 100MB/s.

Are there any other operations going on with the machine? I am just thinking that you may be overloading the bus, rather than the array.

Chad, it was just a thought, and probably not worth pursuing. You say that an internal works better but not perfectly, so lazlow probably is on to something with the bus overflow idea. One idea would be to setup your storage box as a JBOD using only one disk, with a non-journaling filesystem, and see if it can take the data rate. I'm afraid I don't have *the* answer, though.

I also suggest XFS. It is designed for high sustain throughput rates. I use it on all my desktops and notebooks because it is the only file system that can handle throughput that I need to playback raw video with out using RAID to achieve near 50 megabytes per second. For an improve performance, could place the journal on a different drive.

Your system lacks performance for its memory system. FBDIMM is not fast because it is technically slower than traditional DDR2 and consumes too much power.

A RAM Disk will not work in Linux because it looses performance from trying to cache and buffer it.

I think your system right now does not contain a true hardware RAID controller. It is more like a software RAID controller. I suggest buy a different computer that does not come with integrated storage controller. I suggest spend the money on a 3ware or Areca hardware RAID PCIe controllers. Use Western Digital VelociRaptor hard drives with hardware RAID controllers to achieve very high throughput while providing low latency.

There is no such thing as a low latency Linux kernel. A low latency OS is BeOS. Getting drivers to control your data acquisition card and today's hardware will take some work in BeOS.

If you think you have overflow problems, it is most likely the module or driver that controls the data acquisition is giving you problems. I suggest re-write the software, so it knows what to do when a piece of data is not in a compatible format to throw it away or keep it as is and send it to a different location to be process later. The software could then include SIMD instructions like MMX or SSE to improve performance.

I doubt it is the bus giving you problems because it has enough bandwidth to handle the data. If you do not have a lot of cards installed, could increase the latency or how long the bus stays on each card before moving to the next card.

Electo

I did not think any of the SCSI raid solutions were software raid, just Sata and pata.

lazlow:

After booting, free gives:

spooner@whisper2>free
total used free shared buffers cached
Mem: 32957388 304904 32652484 0 19240 176192
-/+ buffers/cache: 109472 32847916
Swap: 4891784 0 4891784

Here are the results of hdparm applied to one of the partitions of the
RAID device:
root@whisper2 spooner]# hdparm -tT /dev/sda1

/dev/sda1:
Timing cached reads: 4836 MB in 2.00 seconds = 2418.37 MB/sec
Timing buffered disk reads: 226 MB in 3.01 seconds = 75.12 MB/sec
[root@whisper2 spooner]# hdparm -tT /dev/sda1

/dev/sda1:
Timing cached reads: 4824 MB in 2.00 seconds = 2412.37 MB/sec
Timing buffered disk reads: 226 MB in 3.01 seconds = 75.12 MB/sec

Regarding other operations going on simultaneously, I've tried to shut down
all non-essential services and the machine is dedicated to this application.
There are no other disk-access processes going on that are initiated by
me during these collection times.

It sounds like the numbers from hdparm are not what you expected
(assuming you were talking about the buffered disk reads and not
the cached ones). As we can see, the buffered disk read parameter
is right at the desired rate of 75 MB/s, which I guess is consistent
with the fact that the A/D buffer (FIFO) overflows don't happen
every time they could, just pretty often.

How can I get that number higher without using XFS?

Thanks,

Chad

Electro:

Thank you for your help. It appears that you agree with Quakeboy02 that I should
strongly consider converting the RAID to XFS from ext2.

Based on all the very helpful comments on this thread, my current
plan is to upgrade to RHEL 5 and then to construct an XFS filesystem
on the RAID device.

But it would be good to resolve the issue of lazlow's expected higher
buffered read number from hdparm.

Thanks!

Chad

My best guess is that either those drives are very old (older tech is slower) or that there is a configuration problem with your Raid setup. Here are my hdparm results from a plain (non raid) sata drive:

This was done while I was burning a dvd iso from that drive (I forgot I was burning it). The drive is just a plain Seagate 7200.10 320gb. The last I checked they were running about $50 each. The newer models will run 100MB/sec and cost under a $100 for 500GB.

EXT2/3 is designed for reliability and stability. It does not feature high bandwidth like XFS. The reliability and stability of XFS is about second or fights for second with JFS.

Upgrading the present distribution does not have to be done to support XFS. Re-compiling the kernel with XFS support and installing XFS utilities is all you need to use XFS as your file system.

The utility hdparm is not a true benchmark for hard drive performance. There are many factors for performance. I suggest use iozone to do a true benchmark. If you use XFS, read and write performance will be close to equal of the amount of RAM you have.

Seagate drives are not good for sustain rates. They are OK for short period of writes and reads. For long periods Western Digital and Hitachi are better because they have higher sustain rates. To handle 75 MB per second, I suggest about eight hard drives in RAID-0 to provide enough throughput even in the worst of conditions. Though if four channels are being saved separately, a RAID-5 or RAID-6 will have to be used because it can handle multiple writes compared to RAID-0 which can only handle one write at a time.

Something is terribly wrong here with my setup and I'm not
sure that changing filesystem types is the whole answer.

I just timed a /bin/cp of an 11 GB file to the RAID
device. It took 10 minutes:

spooner@whisper2>time cp /disk2/.ism_6.25MHz_5min /mnt/Whisper2RAID_A/Archives/
0.622u 21.791s 10:06.64 3.6% 0+0k 752+21994944io 0pf+0w

Copying the file from internal SCSI to internal SCSI took 20 seconds:

spooner@whisper2>time cp /disk2/.ism_6.25MHz_5min /disk2/test_cp2
0.639u 21.211s 0:21.87 99.8% 0+0k 152+21994184io 0pf+0w

The reported system and user times are about the same, but the
elapsed time is quite different. The file gets copied in both
cases.

This thread is starting to wander, but if this new information
helps you have any further insights, I'd be happy to learn
about them.

Chad

After rereading my last post it does look a little like an advertisement for seagate. It was NOT intended to be. With any current drive from any manufacture you should see similiar results. What I was trying to point out was that my drive is not anything special, it is just a typical (current) drive.

While I do believe changing file systems can buy you some headroom, I suspect you have other(non file system based) issues. Unless those drives are 10(?) year old technology you should be seeing at least 100MB/sec in a Raid0 array(my 160GB pata drives in a raid0 array run 110MB/sec and those drives date to whenever 160GB drives would have been in the "sweet spot", 2002?). Having the drive system at the limit of what you are doing will not work (as you have seen) becuase the system will always(on an occasional basis) be doing stuff other than your specific task (system overhead). When you posted your hdparm results it appeared to be from only one drive. I assume that is the drive you are having issues with (the external drive). It may be helpful to compare what your internal drive setups hdparm numbers looks like. There were a lot of updates made to the kernel between RH4 and RH5 in the way sda type drives are handled, but I am not at all sure that this will cure the problem. Maybe contacting RH again with this information about the (very low to me) hdparm results may be of some help.

Edit: all my drives are running default Ext3 file systems. When I used XFS, Rizer(never could spell) and others I did see speed gains, but not enough to make it worth the hassle for me.