small file performance situation

notsure · 02-01-2014, 10:08 PM

I have zoneminder virtualized running debian. It currently uses a dedicated 3TB HDD that is local to the ESX host. The server runs great, no complaints.

I have 2TB (LVM,ext4) full of jpg files 16KB-35KB, many directories containing 1000-5000 files each.

A transfer of 480GB (56million files) to the current 2TB volume took 48hours.

Code:

root@zoneminder:~# time ls -R1 /var/www/zm/events/* | wc -l 158578363

real    75m11.510s
user    16m22.881s
sys     2m22.853s

I'm just curious if anyone has suggestions to increase performance. I tried tweaking the filesystem but didn't notice any increase watching iotop.

Code:

fstab:
/dev/VolGroup1/lv3T     /var/www/zm     ext4    defaults,noatime,data=writeback         0 0

(some of you may notice I labeled the volume 3T instead of 2T, oops)

DinoFly · 02-01-2014, 11:59 PM

You should have considered using reiserfs for performance on huge amount of small files. You will definitely see the difference.

syg00 · 02-02-2014, 01:12 AM

Having a filesystem more attuned to small file packing will help, but you still have to do all the I/O - twice. That's an awful lot of interrupts. And are we talking 2 separate ESX (physical) devices here ?. On separate busses ?. Are there any other guests ?
LVM adds another layer of block device driver to the stack that already consists of ESX, then Linux device driver then VFS. And various levels of caching and I/O schedulers. Let's hope you're not also getting NFS/CIFS into the mix as well.

If this is going to be a regular occurrence, I would get it out from being a guest, and I wouldn't use LVM. Reiser was the answer to this yeas ago, maybe not so much now. Maybe phoronix will have an up-to-date benchmark runoff.

notsure · 02-02-2014, 01:34 AM

Thanks guys, I'll check out reiser and ditch LVM.