LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 11-12-2007, 12:49 PM   #1
malcor
LQ Newbie
 
Registered: Apr 2005
Location: Lake Forest, CA
Distribution: RHEL
Posts: 7

Rep: Reputation: 0
LVM: Building a LV and a Filesystem to just fit


The idea: Write a script that will analyze a directory and build a LV/FS to fit that directory perfectly with no wasted space.

The problem: Getting the sizing right. Default settings make too many inodes and waste space. Analyzing the directory with find just seems to get too close and then some overhead of creating the FS makes it just not fit either on inodes or space. In general it seems more art than algorithm.

I'm using RHEL4 and LVM2.

:-Dan
 
Old 11-14-2007, 02:15 AM   #2
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,397

Rep: Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963
the style and content of this reads just like a university final year project to me. as such we aren't here to do your wrk for you, especially as you've done nothing but recite it, and haven't even asked a question at all...
 
Old 11-14-2007, 08:52 AM   #3
malcor
LQ Newbie
 
Registered: Apr 2005
Location: Lake Forest, CA
Distribution: RHEL
Posts: 7

Original Poster
Rep: Reputation: 0
Thanks I guess. I never had much university and certainly never made it to any final year. At least we agree that the problem is challenging. I had considered posting all my trial and error methods, but it is really just guess work and I was concerned that my attempts would tempt someone to make minor tweaks to my trail and failures, instead of making an unprejudiced suggestion.

I'm not looking for anyone to do my work for me, I would like someone who understands the interactions between the LVM and the filesystem to point me in the right direction.
 
Old 11-14-2007, 02:35 PM   #4
XavierP
Moderator
 
Registered: Nov 2002
Location: Kent, England
Distribution: Lubuntu
Posts: 19,176
Blog Entries: 4

Rep: Reputation: 430Reputation: 430Reputation: 430Reputation: 430Reputation: 430
Your best bet is to post up what you have done already. Otherwise we'll end up suggesting things you have tried. It also may be that you were on the right track with one of your ideas. We'll never know unless you tell us.
 
Old 11-15-2007, 08:55 PM   #5
malcor
LQ Newbie
 
Registered: Apr 2005
Location: Lake Forest, CA
Distribution: RHEL
Posts: 7

Original Poster
Rep: Reputation: 0
Post Script and Eaxmples

This is my trial and error script and a few examples. I'm sure the trick is in figuring out some factor to add to the size and inode counts, but I'd like it to be based on reason and not trial and error.

Code:
     1  #!/bin/bash
     2
     3  DIR=${1}
     4  NAME=${DIR##*/}
     5
     6  if [[ ! -d ${DIR} ]]; then
     7          echo "[${DIR}] is not a directory"
     8          exit 2
     9  fi
    10
    11  declare -i total=0 cnt=0
    12  while read blocks;do
    13          (( ++cnt ))
    14          (( total += blocks ))
    15  done < <( /usr/bin/find ${DIR} -printf "%k\n" )
    16
    17  echo "inodes=${cnt}, 1kBlocks=${total}"
    18
    19  set -x
    20  lvcreate --name ${NAME} --size ${total}k ReadOnly
    21  [[ $? -ne 0 ]] && exit 1
    22  mke2fs -N ${cnt} -L ${NAME} /dev/ReadOnly/${NAME}
    23  [[ $? -ne 0 ]] && exit 1
    24  mkdir /testing/${NAME}
    25  mount /dev/ReadOnly/${NAME} /testing/${NAME}
    26  [[ $? -ne 0 ]] && exit 1
    27  df -P /testing/${NAME};df -Pi /testing/${NAME}
    28  cd ${DIR}
    29  tar cf - . | (cd /testing/${NAME};tar xf -)
    30  df -P /testing/${NAME};df -Pi /testing/${NAME}
    31  echo "umount /testing/${NAME};lvremove /dev/ReadOnly/${NAME}"


This run is small (and it fails). The actual volumes we expect to create are in the 60-80GB range (but the runs take a long time and always fail with this script:

Code:
# ./MakeImgVol.sh  /cacheimg/raidu0/pcl/v007
inodes=205, 1kBlocks=9716
+ lvcreate --name v007 --size 9716k ReadOnly
  Rounding up size to full physical extent 12.00 MB
  Logical volume "v007" created
+ [[ 0 -ne 0 ]]
+ mke2fs -N 205 -L v007 /dev/ReadOnly/v007
mke2fs 1.35 (28-Feb-2004)
Filesystem label=v007
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
208 inodes, 12288 blocks
614 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=12582912
2 block groups
8192 blocks per group, 8192 fragments per group
104 inodes per group
Superblock backups stored on blocks:
        8193

Writing inode tables: done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
+ [[ 0 -ne 0 ]]
+ mkdir /testing/v007
mkdir: cannot create directory `/testing/v007': File exists
+ mount /dev/ReadOnly/v007 /testing/v007
+ [[ 0 -ne 0 ]]
+ df -P /testing/v007
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/ReadOnly-v007     12253       108     11531       1% /testing/v007
+ df -Pi /testing/v007
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/ReadOnly-v007     208      11     197    6% /testing/v007
+ cd /cacheimg/raidu0/pcl/v007
+ tar cf - .
+ cd /testing/v007
+ tar xf -
tar: ./v00707700v/img2c914.tif: Cannot open: No space left on device
tar: ./v00705400v: Cannot mkdir: No space left on device
tar: ./v00705400v/img2de5e.tif: Cannot open: No such file or directory
tar: ./v00705400v/img2ded2.tif: Cannot open: No such file or directory
tar: ./v00705400v/img2df3a.tif: Cannot open: No such file or directory
tar: ./v00705400v/img2dfe9.tif: Cannot open: No such file or directory
tar: ./v00705400v/img2e027.tif: Cannot open: No such file or directory
tar: Error exit delayed from previous errors
+ df -P /testing/v007
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/ReadOnly-v007     12253      9134      2505      79% /testing/v007
+ df -Pi /testing/v007
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/ReadOnly-v007     208     208       0  100% /testing/v007
+ echo 'umount /testing/v007;lvremove /dev/ReadOnly/v007'
umount /testing/v007;lvremove /dev/ReadOnly/v007
Success with a larger volume:

Code:
# ./MakeImgVol.sh  /cacheimg/raidu0/pcl/v000
inodes=6417, 1kBlocks=399540
+ lvcreate --name v000 --size 399540k ReadOnly
  Rounding up size to full physical extent 392.00 MB
  Logical volume "v000" created
+ [[ 0 -ne 0 ]]
+ mke2fs -N 6417 -L v000 /dev/ReadOnly/v000
mke2fs 1.35 (28-Feb-2004)
Filesystem label=v000
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
6664 inodes, 401408 blocks
20070 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67633152
49 block groups
8192 blocks per group, 8192 fragments per group
136 inodes per group
Superblock backups stored on blocks:
        8193, 24577, 40961, 57345, 73729, 204801, 221185

Writing inode tables: done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 35 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
+ [[ 0 -ne 0 ]]
+ mkdir /testing/v000
+ mount /dev/ReadOnly/v000 /testing/v000
+ [[ 0 -ne 0 ]]
+ df -P /testing/v000
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/ReadOnly-v000    400452      2062    378320       1% /testing/v000
+ df -Pi /testing/v000
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/ReadOnly-v000    6664      11    6653    1% /testing/v000
+ cd /cacheimg/raidu0/pcl/v000
+ tar cf - .
+ cd /testing/v000
+ tar xf -
+ df -P /testing/v000
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/ReadOnly-v000    400452    384437         0     100% /testing/v000
+ df -Pi /testing/v000
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/ReadOnly-v000    6664    6427     237   97% /testing/v000
+ echo 'umount /testing/v000;lvremove /dev/ReadOnly/v000'
umount /testing/v000;lvremove /dev/ReadOnly/v000
 
Old 11-16-2007, 11:54 AM   #6
malcor
LQ Newbie
 
Registered: Apr 2005
Location: Lake Forest, CA
Distribution: RHEL
Posts: 7

Original Poster
Rep: Reputation: 0
Any file system gurus out there?

Here is an example of failure with a larger volume and a small tweak to the mke2fs command (on line 22) in the script to reduce the blocks "reserved for the super user" from the default 5% to 0% (didn't seem to make any difference):

22 mke2fs -m 0 -O sparse_super,filetype -N ${cnt} -L ${NAME} /dev/ReadOnly/${NAME}

The man page says that "sparse_super,filetype" are defaults. Just making sure.

Note: The size of this volume is in the range we expect to see.

Code:
# ./MakeImgVol.sh  /cacheimg/raidu0 
inodes=1122588, 1kBlocks=69711504
+ lvcreate --name raidu0 --size 69711504k ReadOnly
  Rounding up size to full physical extent 66.48 GB
  Logical volume "raidu0" created
+ [[ 0 -ne 0 ]]
+ mke2fs -m 0 -O sparse_super,filetype -N 1122588 -L raidu0 /dev/ReadOnly/raidu0
mke2fs 1.35 (28-Feb-2004)
Filesystem label=raidu0
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1123584 inodes, 17428480 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=20971520
532 block groups
32768 blocks per group, 32768 fragments per group
2112 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424

Writing inode tables:   done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
+ [[ 0 -ne 0 ]]
+ mkdir /testing/raidu0
mkdir: cannot create directory `/testing/raidu0': File exists
+ mount /dev/ReadOnly/raidu0 /testing/raidu0
+ [[ 0 -ne 0 ]]
+ df -P /testing/raidu0
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/ReadOnly-raidu0  69568904     53272  69515632       1% /testing/raidu0
+ df -Pi /testing/raidu0
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/ReadOnly-raidu0 1123584      11 1123573    1% /testing/raidu0
+ cd /cacheimg/raidu0
+ tar cf - .
+ cd /testing/raidu0
+ tar xf -
tar: ./pcl/v102/v10271900v/imge126a.tif: Wrote only 4608 of 10240 bytes
tar: Skipping to next header
tar: ./pcl/v102/v10271900v/imge1423.tif: Cannot write: No space left on device
tar: Skipping to next header
tar: ./pcl/v102/v10271900v/imge156d.tif: Cannot write: No space left on device
tar: Skipping to next header
... over 4,000 messages like this ...
tar: ./pcl/v988/v98816750/imgd1ca9.tif: Cannot open: No such file or directory
tar: ./pcl/v988/v98816750/imgd1d94.tif: Cannot open: No such file or directory
tar: Error exit delayed from previous errors
+ df -P /testing/raidu0
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/ReadOnly-raidu0  69568904  69568904         0     100% /testing/raidu0
+ df -Pi /testing/raidu0
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/ReadOnly-raidu0 1123584 1118460    5124  100% /testing/raidu0
+ echo 'umount /testing/raidu0;lvremove /dev/ReadOnly/raidu0'
umount /testing/raidu0;lvremove /dev/ReadOnly/raidu0
 
  


Reply

Tags
lvm, scripting


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LVM and shrinking the root filesystem coreychch Linux - Newbie 4 05-21-2012 01:47 AM
Resizing a root LVM filesystem eigoteacher Linux - Newbie 7 08-23-2007 09:36 AM
Unify 2 filesystem with LVM oswaldospe Linux - Enterprise 0 09-18-2006 01:15 PM
DISCUSSION: Virtual Filesystem: Building a Linux Filesystem from an Ordinary File mchirico LinuxAnswers Discussion 0 10-28-2004 10:35 PM
bootsector location with lvm filesystem flv7a Linux - General 1 11-07-2003 10:53 AM


All times are GMT -5. The time now is 11:30 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration