I am running a ClarkConnect Community 4.2 gateway. This is a Redhat 2.26 kernel Linux system.

I get the error above error when I run "dmesg":

EXT3-fs warning (device dm-0): ext3_dx_add_entry: Directory index full!

I have run out of disk space on my /var mount point. This is a 10GB volume group.

I have extended the volume group and volume name to 25 GB and is now trying to resize the file system for the mail server to restore function to the file systems and mail server (postfix).

I am having serious problems running resize2fs as it indicates that the /var directory is busy and canot be unmounted.

The mount command show:

mounted partitions:
[root@Assurance mapper]# mount
/dev/sdb1 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
/dev/sda2 on /home type ext3 (rw)
/dev/sdb2 on /usr type ext3 (rw)
/dev/mapper/VolGroup00-LogVol00 on /var type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

The following device is the one that needs to be resized:

/dev/mapper/VolGroup00-LogVol00 on /var type ext3 (rw)

Results of several commands ie. pvscan and dumpconfigure
lvm> pvscan
PV /dev/sdb3 VG VolGroup00 lvm2 [138.28 GB / 125.56 GB free]
PV /dev/sda3 VG VolGroup00 lvm2 [119.62 GB / 119.62 GB free]
Total: 2 [257.91 GB] / in use: 2 [257.91 GB] / in no VG: 0 [0 ]

Dumpconfig:

devices {
dir="/dev"
scan="/dev"
preferred_names=[]
filter="a/.*/"
cache_dir="/etc/lvm/cache"
cache_file_prefix=""
write_cache_state=1
sysfs_scan=1
md_component_detection=1
ignore_suspended_devices=0
}
activation {
missing_stripe_filler="/dev/ioerror"
reserved_stack=256
reserved_memory=8192
process_priority=-18
mirror_region_size=512
mirror_log_fault_policy="allocate"
mirror_device_fault_policy="remove"
}
global {
umask=63
test=0
units="h"
activation=1
proc="/proc"
locking_type=1
fallback_to_clustered_locking=1
fallback_to_local_locking=1
locking_dir="/var/lock/lvm"
}
shell {
history_size=100
}
backup {
backup=1
backup_dir="/etc/lvm/backup"
archive=1
archive_dir="/etc/lvm/archive"
retain_min=10
retain_days=30
}
log {
verbose=0
syslog=1
overwrite=0
level=0
indent=1
command_names=0
prefix=" "

fuser run with the -v switch shows:
[root@Assurance /]# fuser -v /var

USER PID ACCESS COMMAND
/var root kernel mount /var

when i run umount i get the following:
[root@Assurance ~]# umount /var -l
umount: /var: not mounted
[root@Assurance ~]#

but I get this when I run fsck /var

/var mount issues:
fsck.ext3: Device or resource busy while trying to open /dev/VolGroup00/LogVol00
Filesystem mounted or opened exclusively by another program?

Finally fuser shows the following:

[root@Assurance log]# fuser -kv /var

USER PID ACCESS COMMAND
/var root 4692 ..c.. locate
root 4693 ..c.. less
root 4727 ..c.. less
root kernel mount /var
No automatic removal. Please use umount /var


Does anyone have and idea what all this is saying or mean.

I have checked this forum for solutions but no one has a problem similar to this one.

I would appreciate any help or suggestions.

Thanks in advance.

cgreen6911

I am running a ClarkConnect Community 4.2 gateway. This is a Redhat 2.26 kernel Linux system.

I get the error above error when I run "dmesg":

EXT3-fs warning (device dm-0): ext3_dx_add_entry: Directory index full!

I have run out of disk space on my /var mount point. This is a 10GB volume group.

I have extended the volume group and volume name to 25 GB and is now trying to resize the file system for the mail server to restore function to the file systems and mail server (postfix).

I am having serious problems running resize2fs as it indicates that the /var directory is busy and canot be unmounted.

The mount command show:

mounted partitions:
[root@Assurance mapper]# mount
/dev/sdb1 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
/dev/sda2 on /home type ext3 (rw)
/dev/sdb2 on /usr type ext3 (rw)
/dev/mapper/VolGroup00-LogVol00 on /var type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

The following device is the one that needs to be resized:

/dev/mapper/VolGroup00-LogVol00 on /var type ext3 (rw)

Results of several commands ie. pvscan and dumpconfigure
lvm> pvscan
PV /dev/sdb3 VG VolGroup00 lvm2 [138.28 GB / 125.56 GB free]
PV /dev/sda3 VG VolGroup00 lvm2 [119.62 GB / 119.62 GB free]
Total: 2 [257.91 GB] / in use: 2 [257.91 GB] / in no VG: 0 [0 ]

Dumpconfig:

devices {
dir="/dev"
scan="/dev"
preferred_names=[]
filter="a/.*/"
cache_dir="/etc/lvm/cache"
cache_file_prefix=""
write_cache_state=1
sysfs_scan=1
md_component_detection=1
ignore_suspended_devices=0
}
activation {
missing_stripe_filler="/dev/ioerror"
reserved_stack=256
reserved_memory=8192
process_priority=-18
mirror_region_size=512
mirror_log_fault_policy="allocate"
mirror_device_fault_policy="remove"
}
global {
umask=63
test=0
units="h"
activation=1
proc="/proc"
locking_type=1
fallback_to_clustered_locking=1
fallback_to_local_locking=1
locking_dir="/var/lock/lvm"
}
shell {
history_size=100
}
backup {
backup=1
backup_dir="/etc/lvm/backup"
archive=1
archive_dir="/etc/lvm/archive"
retain_min=10
retain_days=30
}
log {
verbose=0
syslog=1
overwrite=0
level=0
indent=1
command_names=0
prefix=" "

fuser run with the -v switch shows:
[root@Assurance /]# fuser -v /var

USER PID ACCESS COMMAND
/var root kernel mount /var

when i run umount i get the following:
[root@Assurance ~]# umount /var -l
umount: /var: not mounted
[root@Assurance ~]#

but I get this when I run fsck /var

/var mount issues:
fsck.ext3: Device or resource busy while trying to open /dev/VolGroup00/LogVol00
Filesystem mounted or opened exclusively by another program?

Finally fuser shows the following:

[root@Assurance log]# fuser -kv /var

USER PID ACCESS COMMAND
/var root 4692 ..c.. locate
root 4693 ..c.. less
root 4727 ..c.. less
root kernel mount /var
No automatic removal. Please use umount /var


Does anyone have and idea what all this is saying or mean.

I have checked this forum for solutions but no one has a problem similar to this one.

I would appreciate any help or suggestions.

Thanks in advance.

cgreen6911[/QUOTE]

It may have more to do with the options used when formatting the ext3 filesystem than with the size of the partition. Try seeing if you have an "ext3" filesystem, or look for an ext3.txt file in the Kernel documentation.

Also see if you have 10s of thousands of empty files or empty directories on the filesystem that shouldn't be. When I used SuSE 10.1, something went wrong with the update system and there were tons of empty directories that were being created.

jschuwal,

As you suggested I have looked into my systems for a lot of small files. i noticed that my
/var/cache/apt/archives folder have several GBs of rpm files from apt updates.

I am new to Linux. Is it ok to delete these archive files?

If so, I will be able to delete thse files and get quite a bit of space back until I can resolve how to resize my filesystem.

Is is ok to delete thse files or anr they needed for the system?

cgreen6911

The error message was that the index (directory) was full and not the filesystem itself. I think it may be OK to delete these files. They will be re-downloaded if needed. You may want to investigate whether this indicates a problem, or if there is a cache limit you can set.

Be sure to look at hidden directories and files as well (in /var/cache). I think that is where kerry beagle and uncompressed man pages are cached as well.

I have been chasing my tail on this issue for the last 48 hours.

I get this error when I run dmesg:

EXT3-fs warning (device dm-0): ext3_dx_add_entry: Directory index full!

I am not at all familiar with working with filesystems.

I know I need to unmount the filesystem before I can do anything to check the condition of the dir_index on the filesystem. However, I get the errors listed below:

[root@Assurance /]# umount /dev/mapper/VolGroup00-LogVol00
umount: /var: device is busy
umount: /var: device is busy

[root@Assurance /]# umount -l /dev/mapper/VolGroup00-LogVol00
[root@Assurance /]# e2fsck /dev/mapper/VolGroup00-LogVol00
e2fsck 1.39 (29-May-2006)
e2fsck: Device or resource busy while trying to open /dev/mapper/VolGroup00-LogVol00
Filesystem mounted or opened exclusively by another program?

How do you unmount a /dev/mapper raided filesystem?

How do you fix the, "EXT3-fs warning (device dm-0): ext3_dx_add_entry: Directory index full!" issue?

Any help or suggestions would be useful and appreciated.

cgreen6911

try:

df -i

See if you have any free inodes left on that mounted partition.

Also searches on the web indicate this might be something to do with have a large number of files in one directory.

I assume you've got 10 gigs of old emails in a postfix queue .

If you need to unmount /var normally fuser -m /var will let you know whats running on it, course you fuser -k as you've mentioned too.

Running in single user mode if practical might be most beneficial. If you get desperate you can of course comment out /var in fstab and run in single user mode to the fsck the drive and/or resize2fs it.

Also bear in mind that running fsck on a filesystem with a directory containing an overwhelming number of files might end up just making your machine run out of ram, forcing you to reboot and/or corrupting the filesystem your trying to grow.

Thank you for the suggestions.

I will find a way to boot into single user mode. If that does not work, what is the risk to commenting out the /etc/fstab and removing /var to run fsck? I want to avoid corruption at any cost.

cgreen6911

By the way, this is the results when I run Fuser -m /var. I get hundreds of processes using the /var filesystem. Not a simple taks to kill all of these processes.

root@Assurance /]# fuser -m /var
/var: 1077 1077c 1490 1490c 1502 1502c 1660 1660c 2017 2017c 2033 2033c 2199 2199c 2313 2403 2403c 2404 2404c 2409 2409c 2471 2471c 2527 2536c 2571 2571c 2576 2576c 2606 2681m 2685 2685m 2688 2688m 2690 2690m 2772 2772m 2809 2853 2853c 2880 2880c 2891 2891c 2901c 2906c 2914 2914c 2915 2915c 2917 2917c 2918 2918c 2924c 2931 2931c 2932 2932c 2937 2937c 2949 2949c 2957 2957c 2959 2959c 2965 2965c 2967 2967c 2971 2971c 2972 2972c 2973 2973c 2974 2974c 2976 2976c 2977 2977c 2983 2983c 2984 2994 2994c 3004 3004c 3005 3005c 3007 3007c 3008 3008c 3010 3010c 3012 3012c 3020 3020c 3021 3021c 3022 3022c 3023 3023c 3024 3024c 3025 3025c 3026 3026c 3031 3031c 3032 3032c 3033 3033c 3036 3036c 3038 3038c 3041 3041c 3110 3374 3518 3877 3877m 3906 3906m 3910 3910m 3911 3911m 5253 5661 6762 6762c 7013 7013c 9918 9918c 10001 10001c 11227 11227c 11232 11232c 11607 11607c 11608 11608c 11760 11760c 11765 11765c 11766 11766c 11777 11777c 11778 11778c 11779 11779c 11970 11970c 11978 11978c 11979 11979c 11989 11989c 11991 11991c 12198 12198m 12546 12546c 12555 12555c 12557 12557c 12559 12559c 12560 12560c 12715 12715c 12716 12716c 12717 12717c 12719 12719c 12721 12721c 12725 12725c 12727 12727c 12731 12731c 12734 12734c 12735 12735c 12739 12739c 12744 12744c 12752 12752c 12753 12753c 12757 12757c 12759 12759c 12760 12760c 12767 12767c 12769 12769c 12941 12941c 13111 13111m 13112 13112c 13950 13950c 13951 13951c 13952 13952c 13953 13953c 13954 13954c 13955 13955c 13956 13956c 13957 13957c 14110 14110c 15070 15070c 16634 16634c 18647 18647c 20328 20328c 21401 21401c 21416 21416c 21418 21418c 22697 22697c 23031 23031c 23037 23037c 23130 23130c 23131 23131c 23136 23136c 23138 23138c 23145 23145c 23146 23146c 23148 23148c 23151 23151c 23157 23157c 23211 23211c 23308 23308c 23311 23311c 23313 23313c 23321 23321c 23323 23323c 23325 23325c 23326 23326c 23327 23327c 23329 23329c 23330 23330c 23331 23331c 23332 23332c 23333 23333c 23357 23357c 23358 23358c 23359 23359c 23389 23389c 23391 23391m 23442c 23932 23932c 23937 23937c 23940 23940c 23945 23945c 23950 23950m 24110 24110c 24111 24111c 24126 24126c 24283 24283c 24284 24284c 24285 24285c 24286 24286c 24287 24287c 24288 24288c 24290 24290c 24294 24294c 24295 24295c 24296 24296c 24297 24297c 24298 24298c 24299 24299c 24300 24300c 24301 24301c 24302 24302c 24303 24303c 24305 24305c 24311 24311c 24312 24312c 24313 24313c 24314 24314c 24315 24315c 24316 24316c 24317 24317c 24318 24318c 24319 24319c 24320 24320c 24322 24322c 24323 24323c 24327 24327c 24328 24328c 24329 24329c 24330 24330c 24331 24331c 24332 24332c 24333 24333c 24334 24334c 24335 24335c 24337 24337c 24338 24338c 24339 24339c 24340 24340c 24798 24798c 24807 24807c 25770 25770c 27980 27980c 29674 29674c 29829 29829c 29839 29839c 30787 30787c 32309 32309c

cgreen6911

as Deleriux suggested,
run df -i against /var and check whether all ur inodes are taken up!
if so run a find against /var to check for empty files

#find /var -size 0

if you find any files which are not needed, remove them

#find /var -size 0 -ok rm -f {} \;

Make sure the files you are removing is not needed!

PS: Was there any error when u tried extending the volume group?

Thanks for the follow-up. I did not understand the inode issue. I have run and have appear to have additional inodes available as indicated below:

[root@Assurance ~]# df -i /var
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup00-LogVol00
1310720 835460 475260 64% /var

I will complete the other task and see what happens and will continue this project.

cgreen6911

By the way, I got no errors when I extended the volume Group. I extended it to 20GBs no problem.

cgreen6911

Attached is a sample of the files that have 0 length from my system. Except for the ones which have temp in them, which ones are safe to delete?

/var/state/apt/lists/lock
/var/run/dbus/system_bus_socket
/var/run/saslauthd/mux.accept
/var/run/saslauthd/mux
/var/webconfig/tmp/sess_bbb5a81b79d89c5444dff5a7a8d3d013
/var/webconfig/htdocs/include/phpsysinfo/includes/index.html
/var/webconfig/htdocs/include/phpsysinfo/includes/xml/index.html
/var/webconfig/htdocs/include/phpsysinfo/includes/os/index.html
/var/webconfig/htdocs/include/phpsysinfo/includes/mb/index.html
/var/webconfig/htdocs/include/phpsysinfo/includes/lang/index.html
/var/webconfig/htdocs/include/phpsysinfo/templates/aq/images/index.html
/var/webconfig/htdocs/include/phpsysinfo/templates/aq/index.html
/var/webconfig/reports/spyware/03/16/error
/var/webconfig/reports/spyware/03/5/error
/var/webconfig/reports/snort/tmp.YYTG23599
/var/webconfig/reports/snort/tmp.GTKT23806
/var/webconfig/reports/snort/tmp.Pnlf23598
/var/webconfig/reports/snort/03/31/sids
/var/webconfig/reports/snort/init
/var/log/spooler.3
/var/log/mysqld.log
/var/log/snort/logrotate.3
/var/log/snort/logrotate.2
/var/log/spooler.2
/var/log/spooler.4
/var/log/spooler
/var/log/mysqld.log.2
/var/log/cups/access_log
/var/log/snortsam.2
/var/log/system-mysql/mysqld.log.3
/var/log/system-mysql/mysqld.log.2
/var/log/iptraf/rvnamed.log
/var/log/spooler.1
/var/log/mysqld.log.1
/var/cache/apt/archives/lock

I have deleted hundreds of these files so far but is still getting the same error below:
"EXT3-fs warning (device dm-0): ext3_dx_add_entry: Directory index full!"

Finally, when i run df on the /dev/mapper or raid drive, it shows that there is only 399K left on the filesystem. I believe that this is the source of my problem.

I have tried sveral ways to unmount this drive to run e2fsck on it but i get the error that the filesystem is busy.

when i go to single user mode iget the error:

e2fsck 1.39 (29-May-2006)
e2fsck: Device or resource busy while trying to open /dev/mapper/VolGroup00-LogVol00
Filesystem mounted or opened exclusively by another program?

I believe that if i can resize this filesystem that I will be able to resolve this issue.

What is the correct process for unmounting a raid or /dev/mapper filesystem?

Any one have any ideas or a process?

Thanks in advance for your help.

cgreen6911

I have not been able to resolve the "Directory index full" condition on my system and am experiencing poor performance and my email is not working.

I don't believe that this is an inode issue since I have over 36% of the inode resources available. See below:

[root@Assurance ~]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sdb1 128000 23519 104481 19% /
/dev/sda1 13104 34 13070 1% /boot
none 194373 1 194372 1% /dev/shm
/dev/sda2 3840000 49441 3790559 2% /home
/dev/sdb2 1281024 83670 1197354 7% /usr
/dev/mapper/VolGroup00-LogVol00
1310720 836236 474484 64% /var

However, when I run a df I get the following results:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb1 1003795 146643 806148 16% /
/dev/sda1 102711 7727 89765 8% /boot
none 777492 0 777492 0% /dev/shm
/dev/sda2 30226637 12368235 16322588 44% /home
/dev/sdb2 10078190 1537289 8028830 17% /usr
/dev/mapper/VolGroup00-LogVol00
10318745 9566300 228174 98% /var

My /dev/mapper/VolGroup00-LogVol00 filesystem is at 98% of its capacity!

I believe that this is my issue.

See note from prior post for results of efforts to umount and do a fsck of this filesystem. I really have been chasing my tail on this issue!

This situation is complicated by the fact that I have SATA drives and my operating system has setup my filesystem with /dev/mapper!

I have not been able to figure out how to umount this device to resize the filesystem to add additional space.

Does the work to expand the file system have to be done offline?

Managing filesystems top this level is new to me, I do not have the experience at this time to develop a recover plan.

Does anyone have an idea how I should proceed assuming that this is a filesystem issue and not an inode issue?

I would appeciate any help or ideas, I have not had email for over a week and now my firewall needs to be rebooted twice a day just to keep the internet working.

Thanks in advance.

cgreen6911