LVM: Building a LV and a Filesystem to just fit

malcor · 11-12-2007, 12:49 PM

The idea: Write a script that will analyze a directory and build a LV/FS to fit that directory perfectly with no wasted space.

The problem: Getting the sizing right. Default settings make too many inodes and waste space. Analyzing the directory with find just seems to get too close and then some overhead of creating the FS makes it just not fit either on inodes or space. In general it seems more art than algorithm.

I'm using RHEL4 and LVM2.

:-Dan

acid_kewpie · 11-14-2007, 02:15 AM

the style and content of this reads just like a university final year project to me. as such we aren't here to do your wrk for you, especially as you've done nothing but recite it, and haven't even asked a question at all...

malcor · 11-14-2007, 08:52 AM

Thanks I guess. I never had much university and certainly never made it to any final year. At least we agree that the problem is challenging. I had considered posting all my trial and error methods, but it is really just guess work and I was concerned that my attempts would tempt someone to make minor tweaks to my trail and failures, instead of making an unprejudiced suggestion.

I'm not looking for anyone to do my work for me, I would like someone who understands the interactions between the LVM and the filesystem to point me in the right direction.

XavierP · 11-14-2007, 02:35 PM

Your best bet is to post up what you have done already. Otherwise we'll end up suggesting things you have tried. It also may be that you were on the right track with one of your ideas. We'll never know unless you tell us.

malcor · 11-15-2007, 08:55 PM

This is my trial and error script and a few examples. I'm sure the trick is in figuring out some factor to add to the size and inode counts, but I'd like it to be based on reason and not trial and error.

Code:

     1  #!/bin/bash
     2
     3  DIR=${1}
     4  NAME=${DIR##*/}
     5
     6  if [[ ! -d ${DIR} ]]; then
     7          echo "[${DIR}] is not a directory"
     8          exit 2
     9  fi
    10
    11  declare -i total=0 cnt=0
    12  while read blocks;do
    13          (( ++cnt ))
    14          (( total += blocks ))
    15  done < <( /usr/bin/find ${DIR} -printf "%k\n" )
    16
    17  echo "inodes=${cnt}, 1kBlocks=${total}"
    18
    19  set -x
    20  lvcreate --name ${NAME} --size ${total}k ReadOnly
    21  [[ $? -ne 0 ]] && exit 1
    22  mke2fs -N ${cnt} -L ${NAME} /dev/ReadOnly/${NAME}
    23  [[ $? -ne 0 ]] && exit 1
    24  mkdir /testing/${NAME}
    25  mount /dev/ReadOnly/${NAME} /testing/${NAME}
    26  [[ $? -ne 0 ]] && exit 1
    27  df -P /testing/${NAME};df -Pi /testing/${NAME}
    28  cd ${DIR}
    29  tar cf - . | (cd /testing/${NAME};tar xf -)
    30  df -P /testing/${NAME};df -Pi /testing/${NAME}
    31  echo "umount /testing/${NAME};lvremove /dev/ReadOnly/${NAME}"

This run is small (and it fails). The actual volumes we expect to create are in the 60-80GB range (but the runs take a long time and always fail with this script:

Code:

# ./MakeImgVol.sh  /cacheimg/raidu0/pcl/v007
inodes=205, 1kBlocks=9716
+ lvcreate --name v007 --size 9716k ReadOnly
  Rounding up size to full physical extent 12.00 MB
  Logical volume "v007" created
+ [[ 0 -ne 0 ]]
+ mke2fs -N 205 -L v007 /dev/ReadOnly/v007
mke2fs 1.35 (28-Feb-2004)
Filesystem label=v007
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
208 inodes, 12288 blocks
614 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=12582912
2 block groups
8192 blocks per group, 8192 fragments per group
104 inodes per group
Superblock backups stored on blocks:
        8193

Writing inode tables: done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
+ [[ 0 -ne 0 ]]
+ mkdir /testing/v007
mkdir: cannot create directory `/testing/v007': File exists
+ mount /dev/ReadOnly/v007 /testing/v007
+ [[ 0 -ne 0 ]]
+ df -P /testing/v007
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/ReadOnly-v007     12253       108     11531       1% /testing/v007
+ df -Pi /testing/v007
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/ReadOnly-v007     208      11     197    6% /testing/v007
+ cd /cacheimg/raidu0/pcl/v007
+ tar cf - .
+ cd /testing/v007
+ tar xf -
tar: ./v00707700v/img2c914.tif: Cannot open: No space left on device
tar: ./v00705400v: Cannot mkdir: No space left on device
tar: ./v00705400v/img2de5e.tif: Cannot open: No such file or directory
tar: ./v00705400v/img2ded2.tif: Cannot open: No such file or directory
tar: ./v00705400v/img2df3a.tif: Cannot open: No such file or directory
tar: ./v00705400v/img2dfe9.tif: Cannot open: No such file or directory
tar: ./v00705400v/img2e027.tif: Cannot open: No such file or directory
tar: Error exit delayed from previous errors
+ df -P /testing/v007
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/ReadOnly-v007     12253      9134      2505      79% /testing/v007
+ df -Pi /testing/v007
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/ReadOnly-v007     208     208       0  100% /testing/v007
+ echo 'umount /testing/v007;lvremove /dev/ReadOnly/v007'
umount /testing/v007;lvremove /dev/ReadOnly/v007

Success with a larger volume:

Code:

# ./MakeImgVol.sh  /cacheimg/raidu0/pcl/v000
inodes=6417, 1kBlocks=399540
+ lvcreate --name v000 --size 399540k ReadOnly
  Rounding up size to full physical extent 392.00 MB
  Logical volume "v000" created
+ [[ 0 -ne 0 ]]
+ mke2fs -N 6417 -L v000 /dev/ReadOnly/v000
mke2fs 1.35 (28-Feb-2004)
Filesystem label=v000
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
6664 inodes, 401408 blocks
20070 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67633152
49 block groups
8192 blocks per group, 8192 fragments per group
136 inodes per group
Superblock backups stored on blocks:
        8193, 24577, 40961, 57345, 73729, 204801, 221185

Writing inode tables: done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 35 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
+ [[ 0 -ne 0 ]]
+ mkdir /testing/v000
+ mount /dev/ReadOnly/v000 /testing/v000
+ [[ 0 -ne 0 ]]
+ df -P /testing/v000
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/ReadOnly-v000    400452      2062    378320       1% /testing/v000
+ df -Pi /testing/v000
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/ReadOnly-v000    6664      11    6653    1% /testing/v000
+ cd /cacheimg/raidu0/pcl/v000
+ tar cf - .
+ cd /testing/v000
+ tar xf -
+ df -P /testing/v000
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/ReadOnly-v000    400452    384437         0     100% /testing/v000
+ df -Pi /testing/v000
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/ReadOnly-v000    6664    6427     237   97% /testing/v000
+ echo 'umount /testing/v000;lvremove /dev/ReadOnly/v000'
umount /testing/v000;lvremove /dev/ReadOnly/v000

malcor · 11-16-2007, 11:54 AM

Here is an example of failure with a larger volume and a small tweak to the mke2fs command (on line 22) in the script to reduce the blocks "reserved for the super user" from the default 5% to 0% (didn't seem to make any difference):

22 mke2fs -m 0 -O sparse_super,filetype -N ${cnt} -L ${NAME} /dev/ReadOnly/${NAME}

The man page says that "sparse_super,filetype" are defaults. Just making sure.

Note: The size of this volume is in the range we expect to see.

Code:

# ./MakeImgVol.sh  /cacheimg/raidu0 
inodes=1122588, 1kBlocks=69711504
+ lvcreate --name raidu0 --size 69711504k ReadOnly
  Rounding up size to full physical extent 66.48 GB
  Logical volume "raidu0" created
+ [[ 0 -ne 0 ]]
+ mke2fs -m 0 -O sparse_super,filetype -N 1122588 -L raidu0 /dev/ReadOnly/raidu0
mke2fs 1.35 (28-Feb-2004)
Filesystem label=raidu0
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1123584 inodes, 17428480 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=20971520
532 block groups
32768 blocks per group, 32768 fragments per group
2112 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424

Writing inode tables:   done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
+ [[ 0 -ne 0 ]]
+ mkdir /testing/raidu0
mkdir: cannot create directory `/testing/raidu0': File exists
+ mount /dev/ReadOnly/raidu0 /testing/raidu0
+ [[ 0 -ne 0 ]]
+ df -P /testing/raidu0
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/ReadOnly-raidu0  69568904     53272  69515632       1% /testing/raidu0
+ df -Pi /testing/raidu0
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/ReadOnly-raidu0 1123584      11 1123573    1% /testing/raidu0
+ cd /cacheimg/raidu0
+ tar cf - .
+ cd /testing/raidu0
+ tar xf -
tar: ./pcl/v102/v10271900v/imge126a.tif: Wrote only 4608 of 10240 bytes
tar: Skipping to next header
tar: ./pcl/v102/v10271900v/imge1423.tif: Cannot write: No space left on device
tar: Skipping to next header
tar: ./pcl/v102/v10271900v/imge156d.tif: Cannot write: No space left on device
tar: Skipping to next header
... over 4,000 messages like this ...
tar: ./pcl/v988/v98816750/imgd1ca9.tif: Cannot open: No such file or directory
tar: ./pcl/v988/v98816750/imgd1d94.tif: Cannot open: No such file or directory
tar: Error exit delayed from previous errors
+ df -P /testing/raidu0
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/ReadOnly-raidu0  69568904  69568904         0     100% /testing/raidu0
+ df -Pi /testing/raidu0
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/ReadOnly-raidu0 1123584 1118460    5124  100% /testing/raidu0
+ echo 'umount /testing/raidu0;lvremove /dev/ReadOnly/raidu0'
umount /testing/raidu0;lvremove /dev/ReadOnly/raidu0