So I _USED_ to have a really beefy computer... you know... 3 years ago... But recently I noticed a problem has cropped up that is driving me insane...

I used to have an areca 1220ix raid controller configured with two arrays. the "root" array (raid0 comprising 2 SSDs for performance) and then a "home" array (raid 5 comprising 3 750GB drives). On both arrays lived a pair of LUKS partitions with LVMs in them, the "home" crypto container had just the home partition, while the "root" container had / and swap.
in addition to these raid containers, I also had a /boot drive separate from the raid arrays.

When my raid controller died on me (I don't presently have the scratch to buy a new controller) I opted to build a software raid config with mdadm, then LVM my '/home' '/' and 'swap' partitions inside a pair of luks partition. The only layer I have added here is the linux raid partitions. I figured my system was still powerful enough to handle the added load... WOW was I wrong...

I have an external storage drive I use for backups and such. that drive is also encrypted (I know, I know...) when I do any data transfers from that drive, my system slows to an immediate crawl. conky appears to freeze so I cant really tell what is happening, but top indicates no process is going out of control... my processors (8 core proc) stay at low load, but everything I have up that is graphical in nature (gui) literally pauses (I like to listen to youtube and play minecraft while I play with my data )... I can move my mouse, and exit to console, but that's it. if I wait for the transfer to complete, my system returns to normal.

the best I can tell the loss of performance is having the system have to perform raid, LVM, and crypto simultaneously in this manner is simply robbing everything else of ram.

so I am sitting here thinking, I have 8 processor cores... and I really don't need that many at ANY point... also I have 16 gigs of ram...

So here's the question: is it possible to dedicate a core of my processor to the raid system, and a core to dm-crypt? is it possible to segregate a section of my ram off and give it exclusively to mdadm/dm-crypt? ...and prevent them from useing anything else)?

Look at iotop when the system freezes like that.

I will try this... if/when it gives me troubles again... suppriseingly, right now, it seems to be not freezing...

well isn't this just neat. iotop pauses when the system freezes up. when the system recovers, iotop "lurches" changing the entries rather violently, I cant catch what the offenders are... I still believe this is because of my choice to run LVM on crypto on soft-raid... if that is the correct line of thinking, my fix is to either fork out for a new hardware raid controller (been eyeballing the new areca 1883ix... prety...), or abandon either the crypto layer or the raid layer...

...unless anyone can tell me it's possible (and how) to "dedicate" a chunk of ram and a processor core to mdadm exclusively...

Seems to me you're reacting to symptoms, not the underlying problem(s).

Do some diag:
- is the external always involved when the slowdown/lockup occcurs ?.
- does it also occur if you only write to the external (ie, not a backup from the LVM/crpyt filesystem)
- does dmesg show any issues ?.
- run latencytop

Edit: no you can't dedicate processors/RAM to the software layers you have imposed on the system. They run in kernel space (even if kernel threads), and AFAIK you can't use (yet more) containers in cgroups to corral them. But I haven't tried ...

I have been diagnosing this issue since it began 6 months ago... I have reinstalled my OS, switched distros several times, even tried windows and BSD--erg and blarg...
While Windows does not exibit the symptoms, I'm fairly convinced this is because, though it is using fakeraid, I didn't have a satisfactory way to encrypt the drives. while I can encrypt the main OS partition (truecrypt), and I can encrypt my "/home" array and mount it as an additional drive in windows, that "/home" partition isn't really the home parition for windows... the "documents and settings" (or "user" for win 7 and up) folder is not easily moved to an seperate drive and I found I was working harder on picking through registry keys to move a stupid folder than troubleshooting my problem... so eff that.
All other distros exibited the same issues. this includes, Suse, Debian, LMDE, Mint, Fedora, Arch, PCBSD (NEVER again...), and red hat (enterprise... and yes, I have an entitlement, but I don't want to use it on this, they tend to throw up their hands and proclaim hardware failure for things like this).

no, it's just more obvious. the following are tasks I perform which will _usually_ generate a freeze:
> youtube playing (firefox) while transmission is open - this applies to any streaming service such as Hulu or vimeo. the freeze is momentary, but my system will freeze durring video caching consistantly, creating a kind of studder effect. IE: freeze for 30 seconds, system catches up. freeze again for another 30, catch up, etc.
> listening to music (clementine) while writing in libreoffice writer (usually have 3 or so windows of writer open) - I can keep typing all day, and the text I type will appear on page, however whenever the song changes in clementine, the systme pauses untill the next song is playing then the things I have typed will populate on the page.
> moving files to and from any of my myriad of drives (/, /home, /boot, /media, thumbdrives, external HDDs, etc.). - doesn't matter if I use thunnar or terminal to perform the task, doesn't matter what drive I'm transfering to or from, the system pauses on each and every file in the chain of files to be moved. the trouble is far more obvious on the one external drive, I believe, because both it and my /home partition are encrypted making the system deal with encryption twice.
> playing a DVD in VLC while killing some kerbals...
> listening to music while surfing the interwebz...
> playing minecraft while playing an .avi
> etc. etc. etc.
the pause is only obviuious if I try to do anything dynamic, like typing in a text document, or scrolling either on the web or in a file browser, or text doc, or anything that scrolls... or playing a game. additionally, the pause only occures when I have multiple things open, and ONLY if one of those things is reading/writing to one of the drives which are both encrypted and in RAID. for example, I can dd an image file from the encrypted external drive to a thumbdrive and I will not notice any lag/pause.
Sure as crap does! but none of them stand out for this issue...
> I have a faulty memory card reader, still works, kinda...
> I have a radeon 5785 running eyefinity (6 disp.) config on the radeon driver (I cant be bothered to load the firmware or the catalyst driver, besides this isn't a video issue)
> I have a realtek network card which spouts on about missing ucode and such...
> I have a gigabyte Motherboard which has WAY too much periferal hardware showing up in dmesg, none of which is faulty, just a lot of padding in DMESG...

If you'd like you can see my DMESG here (LINKY). but I'm prety sure there's nothing to see...

latencytop is only vailable to me using an "unsupported" package (AUR, I'm using Arch at the moment)... I'd really like to avaiod yet another reinstall... I am using SSds for the OS partition... meh, regardless, I did manage to get some useful-ish info from iotop, when the pause occures I get a spike on one of the applications described above (firefox, transmission, or clementine, or one of their assiciated processes like firefox-flpl or clementine-gtk), the application which hits the top is only there for the briefest of moments and it is only after the pause has lifted. Still their appearance only pushes the notion this is an I/O issue related to too much hapening on the affected drives...

If you insist, I am willing to install and run latencytop, but I'm still prety sure my troubleshooting is sound... if not, let me know and I'll install/run it.

dang! Oh well... I can get a raid controller, just wanted to be cheep.

TL;DR - I'm trying to make my PC do things it just cant do with the hardware I have in it... I need to get muhself a new raid controller.

EDIT: there are a lot of spelling and punctuation errors in this post... I can't be bothered to install a proper spellchecker/dictionary for firefox and I'm too lazy to open up libreoffice. It's my day off... I'll sleep some more, then correct the typos later.

You're running a stripe array on your OS and a mirror for /root?

Have you tested each drive's performance individually? On the arrays? I'm guessing when you benchmark them the system freezes? There are a million ways to benchmark drives, how'd you go about doing it? I'm be curious to see the results from hdparm.

hdparm -t /dev/sd{a,b,c,d}
hdpart -t /dev/md{127,126}

Also, how did your Areca RAID card fail?

You're using ext4 for the filesystems?

I'm interested in anything RAID so forgive me if I ask too many questions.

Personallly I don't have an issue with AUR packages - easy enough to check the pkgbuild to satisfy yourself. And if you vote it up, it may get promoted and you won't have to worry in future ...
You need more pertinent performance data - iotop will give you an idea maybe, but something like collectl will give you good historical data you can analyse later. However if the problem is really "deep", anything running in userspace will only see the effects - but even that will be useful data if you see kernel threads listed as delayed for example.
There are tools that trace the various functions within the kernel itself, but none are user-friendly.

sda1 is /boot - nothing fancy there.
SDB, SDC are / - Raid 0 array (stripe). both drives are Sata3 SSDs
SDD, SDE, SDF are /home - Raid 5 array. these drives don't have to be super fast like the OS drives so they are just 7300 RPM sata 2 drives (750GB each for a total 1.5TB array)

I just use hdparm, though I usually only test the /home drives as they are the ones with the appearant problem...

here's the results of # hdparm -t /dev/sd{b,c}
to be honest, I did not expect this diferance in performance at all... this shouldn't effect the /home partition... could it?
anyway, results of # hdparm -t /dev/sd{d,e,f}
and these numbers are about half what I was getting with my old areca... but then, the benchmark was on the array and not the individual drives.

Finally, the results of # hdparm -t /dev/md{127,126}
md127 is showing about on par with what I was getting with my old areca, and again md126 is definitely under performing compared with the numbers from my old areca.

The on-board memory overheated and put my arrays in bypass mode. without that added buffer, the arrays became unstable as fsck. I got tired of rebuilding the home array and forceing the root array online almost daily. that's why I will never get a 1#20 model EVAR again... depending on what model you get, some of their higher-end models have a DIMM slot with a replacable/upgradable ram stick... like the 1882ix

Yeah... I have no reason why... I have always used ext3... I just switched with no real reason, I know there are better performing file systems (btrfs for one) but I have had zero bit rot with ext over the years...

Not at all!

It's not so much that the AUR has "bad" packages... I would think that my use of raid/LVM/Crypto in conjunction with an external encrypted drive would reveal my penchant for paranoia...

I'll look into it...

Does the system lock up when you copy files from md127 to the external?

The slow performance of sdb may be due to outdated firmware. smartctl -a
I don't think it has anything to do with your freezing problem. It seems to be affecting the performance of your stripe.

The slower performance of md126 could have to do with the chunk size you set for the array.

I'd like to see you log iotop to a file before you reproduce the problem. iotop -bo > /tmp/file

FOUND IT!!!

my swappiness was 60... I have a 30 gig swap partition with 16 gig of ram... yeah...

reduced swappiness to 1 and problem solved. with that much ram, I can probably drop swappiness to zero... but, what the heck.

UPDATE:
Holy crap, I nearly forgot how snappy my PC is!!! I'm currently running an rsync of data from my external to my Home partition, listening to some tunes, re-installing steam (soon to be loading Kerbal), and playing minecraft... and no lag!

I missed this...

Nice find.

With 16 Gig you could even try getting rid of the swap altogether - maybe test with swapoff first.
Swappiness is just a recommendation indicator - nothing magic about 1 versus zero. I recommend (mainframe) enterprise users always use zero.

With modern desktop kernels, this shouldn't happen - something odd going on there.