LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Debian
User Name
Password
Debian This forum is for the discussion of Debian Linux.

Notices


Reply
  Search this Thread
Old 08-04-2019, 03:00 AM   #1
jsep
LQ Newbie
 
Registered: Dec 2017
Posts: 4

Rep: Reputation: Disabled
/boot out of diskspace during Strech => Buster upgrade


Hit a bit of a problem during Buster upgrade.

At the final steps of the upgrade it was exited when /boot run out of diskspace while installing initramfs-tools.

System is a small backup server with following configuration
Raid1 with sda & sdb for OS with all physical space reserved
md0 = /boot 100MB
md1 = swap 2GB
md2 = / 230GB
md3 = /data (raid5, 4 disks (sdc-sdf))
ext4 on all disks.
System is currently running with the latest default strech kernel and it has not been rebooted. Backup done for /etc & /data so that if all goes terribly wrong I donít loose anything else than some extrawork to build it up again

To recover from this situation my idea was to decrease swap by 500MB and extend /boot by 400MB (just to leave some safe margin). I would like to do this on the currently running system without rescue-cd as the server is somewhat far away from me. Of course the system could most likely be started with the old strech kernel but something needs to be done for the too small /boot partition for it to be future proof.

The steps I planned were:
1. umount swap & /boot
2. decrease swap space by 500MB
3. extend /boot with 400MB and extend the filesystem
4. run disk checks
5. complete the installation with some magical apt command

Does the above make any sense or am I already doomed? Any pointers would be appreciated.
 
Old 08-05-2019, 12:10 PM   #2
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware & Android
Posts: 10,560

Rep: Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172
Just looking at that, boot is too small, and probably swap as well. On Mint (Part of the debian/ubuntu family) the initrd is 56M, and the initrd-tree is 30 Megs. You certainly won't fit 2 of those on 100MB. Likewise, swap looks small unless your memory is so big you don't need swap.

I'm running slackware with an 11M initrd on it's kernel and no initrd on my home brewed kernels. The initrd doesn't need the kitchen sink in there, just enough to get / mounted ro so the modules can be loaded. The initrd is mounted on /, and the drive is mounted on / so the initrd goes away then anyhow.

I ran into trouble with only 100MB on /boot. I have 1G now, and I'm using 140M of it. What's your ram like? I think it's time to backup & re partition. I believe you can make swap a file, which could let you use some spare space by deleting the swap partition and adding it to boot, then make a file for swap. Google that, because I've never done it.
 
Old 08-06-2019, 03:13 AM   #3
jsep
LQ Newbie
 
Registered: Dec 2017
Posts: 4

Original Poster
Rep: Reputation: Disabled
Thanks for the comment! Actually now that you said it, it might be easier to create swap as a file and release the whole md1 swap partition for /boot.

Here's some additional information from the system. As said it's a small headless fileserver system with very limited SW installed.

RAM = 2GB

root@hdserver:~# df -hT |grep md
/dev/md2 ext4 227G 84G 132G 39% /
/dev/md0 ext4 95M 75M 13M 86% /boot

root@hdserver:~# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid10]
md0 : active raid1 sdb1[1] sda1[0]
104320 blocks [2/2] [UU]

md1 : active raid1 sdb2[1] sda2[0]
2048192 blocks [2/2] [UU]

md2 : active raid1 sdb3[1] sda3[0]
242043200 blocks [2/2] [UU]

md3 : active raid5 sdc1[0] sdf1[3] sde1[2] sdd1[1]
4395407808 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>

root@hdserver:~# fdisk -l /dev/sda
Disk /dev/sda: 232.9 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WDC WD2500AAJS-0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0007d768

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 63 208844 208782 102M fd Linux raid autodetect
/dev/sda2 208845 4305419 4096575 2G fd Linux raid autodetect
/dev/sda3 4305420 488392064 484086645 230.9G fd Linux raid autodetect

root@hdserver:~# fdisk -l /dev/sdb
Disk /dev/sdb: 232.9 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WDC WD2500AAJS-0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00040ca0

Device Boot Start End Sectors Size Id Type
/dev/sdb1 * 63 208844 208782 102M fd Linux raid autodetect
/dev/sdb2 208845 4305419 4096575 2G fd Linux raid autodetect
/dev/sdb3 4305420 488392064 484086645 230.9G fd Linux raid autodetect

So with the help of googling the steps would go approx. like this:

Create swap to /
swapoff
fallocate -l 2G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
add to /etc/fstab: /swapfile swap swap defaults 0 0 (remove old swap partition info)
swapon --show

Do changes to the disks:

md0 = /boot
md1 = swap
md2 = /

Start by unmounting /dev/md1.

Fail and remove the devices (partitions) making up /dev/md1: mdadm /dev/md1 --fail /dev/sda2 --remove /dev/sda2 (and same for /dev/sdb2).

Then stop /dev/md1: mdadm --stop /dev/md1.

Fail and remove /dev/sda1: mdadm /dev/md0 --fail /dev/sda1 --remove /dev/sda1.

Within, e.g., parted remove /dev/sda2 and extend /dev/sda1 to occupy the unpartitioned space created.

Add /dev/sda1 back to /dev/md0: mdadm /dev/md0 --add /dev/sda1. Watch cat /proc/mdstat; only when [UU], move to the next step.

Fail and remove /dev/sdb1, then remove /dev/sdb2 and resize /dev/sdb1. Then add /dev/sdb1 back to /dev/md0 and wait for [UU] again.

Grow the array: mdadm --grow /dev/md0 --size=max. Wait for [UU] again.

Resize the filesystem: resize2fs /dev/md0

Sounds like a plan, eh?

Last edited by jsep; 08-08-2019 at 12:45 PM.
 
Old 08-06-2019, 03:41 AM   #4
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 12,456
Blog Entries: 9

Rep: Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370
Once you got that figured out...
Quote:
Originally Posted by jsep View Post
5. complete the installation with some magical apt command
Probably apt won't let you continue, complaining about broken/unconfigured packages etc.
I once had this problem when running an upgrade over a flaky wlan connection.
I ran all of these commands:
Code:
sudo apt-get update --fix-missing
sudo apt-get install -f
sudo dpkg --configure -a
sudo apt-get clean
sudo apt-get update
AFAIR in that order (probably several times)
This article has some more info.
 
Old 08-06-2019, 12:07 PM   #5
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware & Android
Posts: 10,560

Rep: Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172
Quote:
Originally Posted by jsep
Thanks for the comment! Actually now that you said it, it might be easier to create swap as a file and release the whole md1 swap partition for /boot.
{SNIP!}
Resize the filesystem: resize2fs /dev/md0

Sounds like a plan, eh?
Yeah, sounds like you thought about it in depth, which is more that I did .
The raid thing on the swap is confusing. Raid what? It's only one partition.

Anyhow, if you have space to spare, I'd make swap file bigger - 5G?
It may be dynamic anyhow.

Also, if anything is going to be upset by a change in disk numbers, I might leave a small md1 there, but idle or doing nothing of consequence. Allocate 1G /boot, and 1G=do nothing. That way, partition numbers stay the same, and you needn't mention the do nothing partition in fstab, or mount it.

I believe, in fact that you can add md1 to swap by having several swap partitions. In the context of tiny capacity but physically huge disks back in the 1970s, and tiny ram due to limits in packaging and chip fab, several swap files were often needed.

Last edited by business_kid; 08-06-2019 at 12:08 PM.
 
Old 08-08-2019, 12:40 PM   #6
jsep
LQ Newbie
 
Registered: Dec 2017
Posts: 4

Original Poster
Rep: Reputation: Disabled
Excellent! Got it sorted.

Few things popped up during the way:
1. resizing of the swap partition was not an option as the free disk space was not contiguous to /boot. So I had to remove the whole swap partition from /dev/sda and /dev/sdb with parted
2. Resizing the /boot to 1GB was easily done with parted. I left 1GB free for possible future use
3. changes were not visible to the kernel before reboot i.e. the extended disk area
4. I had to reboot the system to the old working Debian 9.9 kernel and work from the console. After the reboot /dev/md0 and the filesystem were possible to be expaded
5. I had to renew the /etc/mdadm/mdadm.conf with mdadm ódetail óscan >> /etc/mdadm/mdadm.conf and comment out the old disk configuration from the file
6. I had to bring manually /dev/sda1 next to /dev/sdb1 after which the mirroring was done succesfully
7. Reinstalling the missing packages were done with ondohoís spells. Thanks for those
8. swap is now on / and is 5GB of size

Everything is up and running and no data was lost. So Iím super happy!

Thanks for all your hints and comments!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
GDM could not write a new authorization entry to disk. Possibly out of diskspace. Error: No space left on device mc2017 Linux - Server 5 08-24-2017 03:44 PM
[SOLVED] Out of Diskspace on 30 Gig? Senathon Linux From Scratch 7 09-07-2016 12:52 PM
how do I clean my root dir out? i have no diskspace in it and i cant log in. ethereal27 Ubuntu 3 07-14-2006 10:48 PM
Dell Latitude D600 strech problem koenig Linux - Laptop and Netbook 0 10-14-2004 07:48 AM
dual monitors - how can I strech desktop xanas3712 Linux - Newbie 3 04-30-2004 09:48 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Debian

All times are GMT -5. The time now is 05:36 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration