Extend /dev/sda space

almond_84 · 01-31-2019, 06:20 AM

Hello,
Please would you help me on this,
I want to extend the space for /dev/sda but I am afraid.

When I execute

Code:

[root@NewGen ~]# lsblk -f
NAME                         FSTYPE      LABEL      UUID                                   MOUNTPOINT
fd0
sda                          LVM2_member            17PjYe-4DcP-z7ty-VTjt-4Qpf-g50p-UE5lQj
└─vg_oracle-lv_oracle (dm-2) ext4                   4a279124-c1e3-46b6-9301-929016cfc299   /u01/app/oracle/oradata
sdb
├─sdb1                       ext4                   3da4ab80-66c8-474a-9ed0-4ea9d7d40c8f   /boot
├─sdb2                       LVM2_member            wi8Fsf-YdyG-xuSp-Rbuq-CoRv-T8oH-EDfedw
│ ├─vg_newgen-lv_root (dm-0) ext4                   e4ba1017-3d3a-45a8-8af3-874cedce30df   /
│ └─vg_newgen-lv_swap (dm-1) swap                   e524a9f6-3409-415f-bd75-b19d140ec341   [SWAP]
├─sdb3                       ext4        dms_images 67d47fb5-50b9-4cf5-b9b4-0c9dfa56c333
└─sdb4                       LVM2_member            wIUc6V-Coc0-DtbK-3go3-hKCE-cSkb-dgn8VJ
  └─vg_newgen-lv_root (dm-0) ext4                   e4ba1017-3d3a-45a8-8af3-874cedce30df   /
sr0

I see that /dev/sda is being used, but when I run fdisk -l, I cannot see any primary partition created, for example /dev/sda1..
:

Code:

[root@NewGen ~]# fdisk -l

Disk /dev/sdb: 451.0 GB, 450971566080 bytes
255 heads, 63 sectors/track, 54827 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0003f0aa

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          64      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sdb2              64       15666   125316096   8e  Linux LVM
/dev/sdb3           15667       22192    52420095   83  Linux
/dev/sdb4           22193       54827   262140637+  8e  Linux LVM

Disk /dev/sda: 536.9 GB, 536870912000 bytes
255 heads, 63 sectors/track, 65270 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_newgen-lv_root: 388.3 GB, 388342218752 bytes
255 heads, 63 sectors/track, 47213 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_newgen-lv_swap: 8405 MB, 8405385216 bytes
255 heads, 63 sectors/track, 1021 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_oracle-lv_oracle: 214.7 GB, 214744170496 bytes
255 heads, 63 sectors/track, 26107 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Please, would you please explain to me what is the meaning of this?
Does this mean that the physical partition /dev/sda has never been extended ?

Also, I need to extend the space on that physical partition.

I have increased the space in VMware, and now I want to create in Linux the physical partition /dev/sda1 as Linux LVM 8e. Is this ok?

Many Thanks!!

ehartman · 01-31-2019, 07:50 AM

Quote:

Originally Posted by almond_84

Does this mean that the physical partition /dev/sda has never been extended ?

Also, I need to extend the space on that physical partition.

I have increased the space in VMware, and now I want to create in Linux the physical partition /dev/sda1 as Linux LVM 8e. Is this ok?

Many Thanks!!

/dev/sda is NOT a partition, it is the whole disk (and thus is as large AS the disk).
In your case that disk has not been partitioned but made into an LVM "Physical Volume",
in which lv's (logical volumes) have been created. Their device files look like /dev/dm* (device mapper) and are managed through the LVM tools.

Find a good tutorial about LVM2, especially in combination with your distribution.

MensaWater · 01-31-2019, 08:00 AM

lsblk is showing that you are using LVM2. LVM = Logical Volume Manager

In LVM you add disks or partitions as PVs (Physical Volumes) to a VG (Volume Group). You can then subdivide the VG into LVs (Logical Volumes).

Since fdisk shows no partitions on sda it means the entire disk is a PV. The line:

Quote:

vg_oracle-lv_oracle (dm-2) ext4 4a279124-c1e3-46b6-9301-929016cfc299 /u01/app/oracle/or

Shows the VG's name is vg_oracle and it has one LV named lv_oracle which is an ext4 filesystem mounted as /u01/app/oraccle/or.

Since sda was used as the PV there is no additional space available from sda to add a partition.

However, since your filesystem is in LVM you may be able to extend the LV if the VG has free space. Failing that you'd have to add another disk to the system since lsblk shows sda fully allocated to LVM and fdisk -l shows sdb fully allocated to partitions (some of which are in LVM but a separate VG named vg_newgen). With a new disk you can add it as a PV to vg_oracle.

Run the following commands and provide the output:
vgs = Shows your defined VGs and whether any space is available to add to the LVs
pvs = Shows which devices (disks or partitions are in use)
lvs = Shows your defined LVs with sizes
df -hP = Shows your mounted filesystem directories with sizes and the devices (e.g. LVs) they are mounted from.

You might want to read up on LVM and the manual page is a good place to start. Type:
man lvm

almond_84 · 01-31-2019, 11:01 AM

Hi,

Results below:

Code:

[root@NewGen ~]# vgs
  VG                       #PV        #LV       #SN        Attr                        VSize                     VFree
  vg_newgen       2             2             0             wz--n-                   369.50g                0
  vg_oracle          1             1             0             wz--n-                   200.00g                0


[root@NewGen ~]# pvs
  PV                       VG                        Fmt        Attr        PSize                     PFree
  /dev/sda            vg_oracle            lvm2      a--          200.00g                 0
  /dev/sdb2         vg_newgen        lvm2      a--          119.51g                0
  /dev/sdb4         vg_newgen        lvm2      a--          249.99g                0

[root@NewGen ~]# lvs
  LV                        VG                         Attr                       LSize                      Pool       Origin    Data%  Meta% Move    Log         Cpy%Sync       Convert
  lv_root               vg_newgen        -wi-ao----             361.67g
  lv_swap             vg_newgen        -wi-ao----             7.83g
  lv_oracle            vg_oracle             -wi-ao----             200.00g

[root@NewGen ~]# df -hP
Filesystem                                                        Size        Used      Avail      Use%     Mounted on
/dev/mapper/vg_newgen-lv_root             356G     130G     222G     37%       /
tmpfs                                                                 3.8G      2.0G      1.8G      53%       /dev/shm
/dev/mapper/vg_oracle-lv_oracle            197G     184G     3.4G      99%       /u01/app/oracle/oradata
/dev/sdb1                                                         477M   64M       384M    15%       /boot
172.16.10.32:/backup                                   300G     265G   36G          89%       /mnt/backup

Thank you!

MensaWater · 01-31-2019, 03:07 PM

That output confirms you've allocated all of both disks to 2 separate VGs and have allocated 100% of VGs to LVs. There is no space in partitions.

It does show that you allocated your root (/) filesystem as 356GB but it contains only 130GB of files/data and has 222GB free. That filesystem is in VG, vg_newgen, which is comprised of two separate PVs, /dev/sdb2 & /dev/sdb4.

Also I notice your df output doesn't show anything for /dev/sdb3 but your lsblk earlier shows it formated for ext4 (not as part of LVM). Do you know what that is used for?

Ideally what you would do at this point is add a new disk to the system (are these internal disks? You can find out by running lsscsi command.

If you can't add another disk your options are:
1) If /dev/sdb3 is unused you could change its type to LVM2 then pvcreate it then vgextend it into vg_oracle (type "man" on each of the commands for details, e.g. "man pvcreate". Do NOT do this unless you're sure it isn't used for the images the label implies.

2) You could work on reducing the size of the root filesytem and its LV to only use the space allowed by sdb4 then plan to pvmove to move extents from /dev/sdb2 to /dev/sdb4, use vgremove to remove sdb2 from vg_newgen then vgextend to add to sdb2 to vg_oracle. Since this is the root filesystem you'd likely have to plan on reducing it in single user mode. Not a light undertaking.

3) If there are subdirectories in /u01/app/oracle/oradata filesystem that aren't actually data (e.g. the oracle binaries themselves) you could cheat by moving them to a new subdirectory of root (/) such as /u01/app/oracle/ora2 then make a symbolic link of that to the subdirectory they had in the above filesystem:
e.g. If you had /u01/app/oracle/oradata/appl_top you could create /u02/app/oracle/ora2/appl_top then link them.
I suspect however, your binaries and non data files are already in the root filesystem somewhere under /u02/app/oracle.

Any of these would require careful planning. For option 2 you'd have to shutdown everything. For option 3 you'd likely have to shutdown Oracle.

This is why I say option 1 is the ideal way to go. The only downtime would be to add a physical disk to the system. All the configuration after that can be done online. If it is a SAN disk it doesn't even require the downtime to add the LUN as that can be done online.

syg00 · 01-31-2019, 06:20 PM

I suspect the OP has made the virtual disk bigger - /dev/sda is 536.9 GB but the pv is only 200 G; probably the original size of the disk. "pvresize /dev/sda" will expand the pv to the size of the disk. Then the vg and lv can be enlarged.

MensaWater · 02-01-2019, 08:21 AM

Quote:

Originally Posted by syg00

I suspect the OP has made the virtual disk bigger - /dev/sda is 536.9 GB but the pv is only 200 G; probably the original size of the disk. "pvresize /dev/sda" will expand the pv to the size of the disk. Then the vg and lv can be enlarged.

D'oh! I completely missed that because I saw the VG was the same as the PV. You're right - fdisk shows the disk, /dev/sda, is much larger than the PV.

He'd need to do a pvresize

A good write-up I found some time ago on that by Earl C. Ruby III is at:
http://earlruby.org/2010/10/increasi...out-rebooting/

Since multipathing doesn't appear to be in use the OP can ignore that part.

Quote:

Increasing the size of an LVM Physical Volume (PV) while running multipathd — without rebooting
Posted on 2010-10-17 by Earl C. Ruby III

If you’re using the Linux Logical Volume Manager (LVM) to manage your disk space it’s easy to enlarge a logical volume while a server is up and running. It’s also easy to add new drives to an existing volume group.

But if you’re using a SAN the underlying physical drives can have different performance characteristics because they’re assigned to different QOS bands on the SAN. If you want to keep performance optimized it’s important to know what physical volume a logical volume is assigned to — otherwise you can split a single logical volume across multiple physical volumes and end up degrading system performance. If you run out of space on a physical volume and then enlarge a logical volume you will split the LV across two or more PVs. To prevent this from happening you need to enlarge the LUN, tell multipathd about the change, then enlarge the PV, then enlarge the LV, and finally enlarge the file system.

I have three SANs at the company where I work (two Pillar Axioms and a Xyratex) which are attached two two fibrechannel switches and several racks of blade servers. Each blade is running an Oracle database with multiple physical volumes (PVs) grouped together into a single LVM. The PVs are tagged and as logical volumes (LVs) are added they’re assigned to the base physical volume with the same tag name as the logical volume. That way we can assign the PV to a higher or lower performance band on the SAN and optimize the database’s performance. Oracle tablespaces that contain frequently-accessed data get assigned to a PV with a higher QOS band on the SAN. Archival data gets put on a PV with a lower QOS band.

We run OpenSUSE 11.x using multipathd to deal with the multiple fiber paths available between each blade and a SAN. Since each blade has 2 fiber ports for redundancy, which are attached to two fiber switches, each of which is cross-connected to 2 ports on 2 different controllers on the SAN, so there are 4 different fiber paths that data can take between the blade and the SAN. If any path fails, or one port on a fiber card fails, or one fiber switch fails, multipathd re-routes the data using the remaining data paths and everything keeps working. If a blade fails we switch to another blade.

If we run out of space on a PV I can log into the SAN’s administrative interface and enlarge the size of the underlying LUN, but getting the operating system on the blade to recognize the fact that more physical disk space was available is tricky. LVM’s pvresize command would claim that it was enlarging the PV, but nothing would happen unless the server was rebooted and then pvresize was run again. I wanted to be able to enlarge physical volumes without taking a database off-line and rebooting its server. Here’s how I did it:

First log into the SAN’s administrative interface and enlarge the LUN in question.
Open two xterm windows on the host as root
Gather information – you will need the physical device name, the multipath block device names, and the multipath map name. (Since our setup gives us 4 data paths for each LUN there are 4 multipath block device names.)
List the physical volumes and their associated tags with pvs -o +tags:

# pvs -o +tags
PV VG Fmt Attr PSize PFree PV Tags
/dev/dm-1 switch lvm2 a- 500.38G 280.38G db024-lindx,lindx
/dev/dm-10 switch lvm2 a- 1.95T 801.00G db024-ldata,ldata
/dev/dm-11 switch lvm2 a- 81.50G 0 db024-mindx,mindx
/dev/dm-12 switch lvm2 a- 650.00G 100.00G db024-reports,reports
/dev/dm-13 switch lvm2 a- 51.25G 31.25G db024-log,log
/dev/dm-14 switch lvm2 a- 450.12G 50.12G db024-home,home
/dev/dm-15 switch lvm2 a- 1.76T 342.00G db024-q_backup,q_backup
/dev/dm-16 switch lvm2 a- 1.00G 640.00M db024-control,control
/dev/dm-2 switch lvm2 a- 301.38G 120.38G db024-dbs,dbs
/dev/dm-3 switch lvm2 a- 401.88G 101.88G db024-cdr_data,cdr_data
/dev/dm-5 switch lvm2 a- 450.62G 290.62G db024-archlogs,archlogs
/dev/dm-6 switch lvm2 a- 40.88G 22.50G db024-boot,boot
/dev/dm-7 switch lvm2 a- 51.25G 1.25G db024-rbs,rbs
/dev/dm-8 switch lvm2 a- 51.25G 27.25G db024-temp,temp
/dev/dm-9 switch lvm2 a- 201.38G 161.38G db024-summary,summary

Find the device that corresponds to the LUN you just enlarged, e.g. /dev/dm-11
Run multipath -ll, find the device name in the listing. The large hex number at the start of the line is the multipath map name and the sdX block devices after the device name are the multipath block devices. So in this example the map name is 2000b080112002142 and the block devices are sdy, sdan, sdj, and sdbc:

2000b080112002142 dm-11 Pillar,Axiom 500
[size=82G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
\_ 0:0:5:9 sdy 65:128 [active][ready]
\_ 1:0:4:9 sdan 66:112 [active][ready]
\_ round-robin 0 [prio=20][enabled]
\_ 0:0:4:9 sdj 8:144 [active][ready]
\_ 1:0:5:9 sdbc 67:96 [active][ready]

Next get multipath to recognize that the device is larger:
For each block device do echo 1 > /sys/block/sdX/device/rescan:

# echo 1 > /sys/block/sdy/device/rescan
# echo 1 > /sys/block/sdan/device/rescan
# echo 1 > /sys/block/sdj/device/rescan
# echo 1 > /sys/block/sdbc/device/rescan

In the second root window, pull up a multipath command line with multipathd -k
Delete and re-add the first block device from each group. Since multipathd provides multiple paths to the underlying SAN, the device will remain up and on-line during this process. Make sure that you get an ‘ok’ after each command. If you see ‘fail’ or anything else besides ‘ok’, STOP WHAT YOU’RE DOING and go to the next step.

multipathd> del path sdy
ok
multipathd> add path sdy
ok
multipathd> del path sdj
ok
multipathd> add path sdj
ok

If you got a ‘fail’ response:
Type exit to get back to a command line.
Type multipath -r on the command line. This should recover/rebuild all block device paths.
Type multipath -ll | less again and verify that the block devices were re-added.
At this point multipath may actually recognize the new device size (you can see the size in the multipath -ll output). If everything looks good, skip ahead to the pvresize step.
In the first root window run multipath -ll again and verify that the block devices were re-added:

2000b080112002142 dm-11 Pillar,Axiom 500
[size=82G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
\_ 1:0:4:9 sdan 66:112 [active][ready]
\_ 0:0:5:9 sdy 65:128 [active][ready]
\_ round-robin 0 [prio=20][enabled]
\_ 1:0:5:9 sdbc 67:96 [active][ready]
\_ 0:0:4:9 sdj 8:144 [active][ready]

Delete and re-add the remaining two block devices in the second root window:

multipathd> del path sdan
ok
multipathd> add path sdan
ok
multipathd> del path sdbc
ok
multipathd> add path sdbc
ok

In the first root window run multipath -ll again and verify that the block devices were re-added.
Tell multipathd to resize the block device map using the map name:

multipathd> resize map 2000b080112002142
ok

Press Ctrl-D to exit multipathd command line.
In the first root window run multipath -llagain to verify that multipath sees the new physical device size. The device below went from 82G to 142G:

2000b080112002142 dm-11 Pillar,Axiom 500
[size=142G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
\_ 0:0:5:9 sdy 65:128 [active][ready]
\_ 1:0:4:9 sdan 66:112 [active][ready]
\_ round-robin 0 [prio=20][enabled]
\_ 0:0:4:9 sdj 8:144 [active][ready]
\_ 1:0:5:9 sdbc 67:96 [active][ready]

Finally, get the LVM volume group to recognize that the physical volume is larger using pvresize:

# pvresize /dev/dm-11
Physical volume "/dev/dm-11" changed
1 physical volume(s) resized / 0 physical volume(s) not resized
# pvs -o +tags
PV VG Fmt Attr PSize PFree PV Tags
/dev/dm-1 switch lvm2 a- 500.38G 280.38G db024-lindx,lindx
/dev/dm-10 switch lvm2 a- 1.95T 801.00G db024-ldata,ldata
/dev/dm-11 switch lvm2 a- 141.50G 60.00G db024-mindx,mindx
/dev/dm-12 switch lvm2 a- 650.00G 100.00G db024-reports,reports
/dev/dm-13 switch lvm2 a- 51.25G 31.25G db024-log,log
/dev/dm-14 switch lvm2 a- 450.12G 50.12G db024-home,home
/dev/dm-15 switch lvm2 a- 1.76T 342.00G db024-q_backup,q_backup
/dev/dm-16 switch lvm2 a- 1.00G 640.00M db024-control,control
/dev/dm-2 switch lvm2 a- 301.38G 120.38G db024-dbs,dbs
/dev/dm-3 switch lvm2 a- 401.88G 101.88G db024-cdr_data,cdr_data
/dev/dm-5 switch lvm2 a- 450.62G 290.62G db024-archlogs,archlogs
/dev/dm-6 switch lvm2 a- 40.88G 22.50G db024-boot,boot
/dev/dm-7 switch lvm2 a- 51.25G 1.25G db024-rbs,rbs
/dev/dm-8 switch lvm2 a- 51.25G 27.25G db024-temp,temp
/dev/dm-9 switch lvm2 a- 201.38G 161.38G db024-summary,summary

pvs shows that /dev/dm-11 is now 141.5G.

At this point you can enlarge any logical volumes residing on the underlying physical volume without splitting the logical volume across multiple (non-contiguous) physical volumes using lvresize and enlarge the file system using the file system tools, e.g. resize2fs.

If you ran out of space, your LVs were split across multiple PVs, and you need to coalesce a PV onto a single LV use pvmove to move the physical volume to a single device.

I've not actually done that because typically we just add additional LUNs on physical servers or additional virtual disks from the hypervisor to virtual servers. We then just add the additional LUN or virtual disk as a PV to appropriate VG.

almond_84 · 02-01-2019, 10:23 AM

Hello,

Hm.. I am a little confused now.
What I have done till now is: I have increased the size of /dev/sda from 200GB to 536.9GB from VMware.
Now, I want to create a new primary partition type 8e named /dev/sda1, and then increase VG and then LV.

Is this OK? or I am missing something?
Many thanks for your support!

MensaWater · 02-01-2019, 11:40 AM

Quote:

Originally Posted by almond_84

Hello,

Hm.. I am a little confused now.
What I have done till now is: I have increased the size of /dev/sda from 200GB to 536.9GB from VMware.
Now, I want to create a new primary partition type 8e named /dev/sda1, and then increase VG and then LV.

Is this OK? or I am missing something?
Many thanks for your support!

No. Your sda is NOT partitioned currently. The entire original sda size of 200 GB was used as PV with no partitioning done. If you try to partition sda you risk blowing away the LVM information already on the disk.

As noted by syg00:

Quote:

/dev/sda is 536.9 GB but the pv is only 200 G; probably the original size of the disk. "pvresize /dev/sda" will expand the pv to the size of the disk. Then the vg and lv can be enlarged.

My post was just a follow up to his. I was showing a person who wrote up doing such expansion. I then noted that I typically don't increase virtual disk sizes but instead add new ones. You've already increased the virtual disk size so you need to go the pvresize route.

LQParsons · 02-08-2019, 09:02 AM

This was helpful to me:

How to Increase the size of a Linux LVM by expanding the virtual machine disk

https://www.rootusers.com/how-to-inc...-machine-disk/

MensaWater · 02-08-2019, 10:06 AM

Quote:

Originally Posted by LQParsons

This was helpful to me:

How to Increase the size of a Linux LVM by expanding the virtual machine disk

https://www.rootusers.com/how-to-inc...-machine-disk/

Unfortunately that link assumes the disk is partitioned which is not the case for the OP.

To re-iterate to OP:
Do NOT try to partition /dev/sda - you are using the entire disk as a PV, not partitions from it. If you try to partition it you risk destroying existing LVM information. Just do the pvresize and other steps.