LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 02-01-2022, 01:19 PM   #1
jim_cliff11
LQ Newbie
 
Registered: Jan 2004
Posts: 21

Rep: Reputation: 0
BTRFS nightmare.... Help!


Hi,

My SDA3 btrfs filesystem is saying it's full, and not allowing samba, webmin, apache etc to run.

6 months ago I had the exact same issue... (https://www.linuxquestions.org/quest...ed-4175696383/

In the end I reformatted and installed a fresh copy of OpenSuse 15.3. I gave btrfs the benefit of the doubt by again configuring my root filesystem to us it. But this time with a larger partition size of 60GB. I'm now kicking myself.... as I'm back with the same issue and our business server is down.

'snapper list' gives me
Code:
The config 'root' does not exist. Likely snapper is not configured.
See 'man snapper' for further instructions.
Thus, I'm guessing I'm not using snapper and shouldn't have the problem of snapshot files consuming space.

df -h gives:

Code:
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           3.2G  297M  2.9G  10% /run
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/sda3        60G   57G     0 100% /
/dev/sda3        60G   57G     0 100% /boot/grub2/i386-pc
/dev/sda3        60G   57G     0 100% /boot/grub2/x86_64-efi
/dev/sda3        60G   57G     0 100% /opt
/dev/sda3        60G   57G     0 100% /root
/dev/sda3        60G   57G     0 100% /srv
/dev/sda3        60G   57G     0 100% /tmp
/dev/sda3        60G   57G     0 100% /usr/local
/dev/sda3        60G   57G     0 100% /var
/dev/sda4       1.0T  1.8G 1022G   1% /home
/dev/sda5       4.4T  103G  4.3T   3% /big
/dev/sdb1       1.9T   99G  1.8T   6% /exthdd
tmpfs           1.6G     0  1.6G   0% /run/user/0
'du -hsc /boot /opt /root /srv /usr/local /var /tmp' gives:

Code:
47M     /boot
0       /opt
168K    /root
3.7M    /srv
45M     /usr/local
620M    /var
0       /tmp
715M    total
btrfs filesystem usage / gives:

Code:
Overall:
    Device size:                  60.00GiB
    Device allocated:             60.00GiB
    Device unallocated:            1.00MiB
    Device missing:                  0.00B
    Used:                         56.50GiB
    Free (estimated):                0.00B      (min: 0.00B)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               74.34MiB      (used: 0.00B)

Data,single: Size:55.94GiB, Used:55.94GiB
   /dev/sda3      55.94GiB

Metadata,DUP: Size:2.00GiB, Used:290.81MiB
   /dev/sda3       4.00GiB

System,DUP: Size:32.00MiB, Used:16.00KiB
   /dev/sda3      64.00MiB

Unallocated:
   /dev/sda3       1.00MiB
I've heard about the BTRFS huge metadata allocation issue, but I have no idea how to resolve it. Unfortunately, I'm just the user of a btrfs filesystem and far from an expert.

It seems daft to me that this is the second time in less than a year my btrfs filesystem has done this. There MUST be a way to fix it, free up space and prevent any future recurrence? BTRFS from my understanding is a common filesystem, so either everyone has this issue, or I've made a fundamental mistake when configuring mine (both times).

Any help is appreciated, I really don't want to re-install again (if I do, this time in ext4).

Thanks
 
Old 02-01-2022, 01:25 PM   #2
jim_cliff11
LQ Newbie
 
Registered: Jan 2004
Posts: 21

Original Poster
Rep: Reputation: 0
Update,

I've tried running 'btrfs balance start -m /' and after 5 seconds I get:

Code:
Done, had to relocate 2 out of 63 chunks
'df -h' then gives me 5% free space:

Code:
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           3.2G  329M  2.9G  11% /run
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/sda3        60G   57G  3.1G  95% /
/dev/sda3        60G   57G  3.1G  95% /boot/grub2/i386-pc
/dev/sda3        60G   57G  3.1G  95% /boot/grub2/x86_64-efi
/dev/sda3        60G   57G  3.1G  95% /opt
/dev/sda3        60G   57G  3.1G  95% /root
/dev/sda3        60G   57G  3.1G  95% /srv
/dev/sda3        60G   57G  3.1G  95% /tmp
/dev/sda3        60G   57G  3.1G  95% /usr/local
/dev/sda3        60G   57G  3.1G  95% /var
/dev/sda4       1.0T  1.8G 1022G   1% /home
/dev/sda5       4.4T  103G  4.3T   3% /big
/dev/sdb1       1.9T   99G  1.8T   6% /exthdd
tmpfs           1.6G     0  1.6G   0% /run/user/0
Something tells me this will soon fill up due to exactly the same reasons
 
Old 02-01-2022, 01:41 PM   #3
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 26,636

Rep: Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965
Quote:
Originally Posted by jim_cliff11 View Post
Update,
I've tried running 'btrfs balance start -m /' and after 5 seconds I get:
Code:
Done, had to relocate 2 out of 63 chunks
'df -h' then gives me 5% free space:
Code:
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           3.2G  329M  2.9G  11% /run
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/sda3        60G   57G  3.1G  95% /
/dev/sda3        60G   57G  3.1G  95% /boot/grub2/i386-pc
/dev/sda3        60G   57G  3.1G  95% /boot/grub2/x86_64-efi
/dev/sda3        60G   57G  3.1G  95% /opt
/dev/sda3        60G   57G  3.1G  95% /root
/dev/sda3        60G   57G  3.1G  95% /srv
/dev/sda3        60G   57G  3.1G  95% /tmp
/dev/sda3        60G   57G  3.1G  95% /usr/local
/dev/sda3        60G   57G  3.1G  95% /var
/dev/sda4       1.0T  1.8G 1022G   1% /home
/dev/sda5       4.4T  103G  4.3T   3% /big
/dev/sdb1       1.9T   99G  1.8T   6% /exthdd
tmpfs           1.6G     0  1.6G   0% /run/user/0
Something tells me this will soon fill up due to exactly the same reasons
Probably; 60GB when you have a LOT of disk space to play with seems odd. Have you looked in /var and /tmp?? Log files and journal files can get fairly big, and if you have run updates, you've got several kernels installed too.

Run "rpm -qa | grep -i kernel", and see what's installed. Then run "uname -a" to get the current running kernel...you should be able to remove the older kernels (and kernel development headers, too, if installed), to gain more room. You can also try "zypper purge-kernels" to trim things down, if you do find several older kernels installed.

As a stopgap measure, you may want to symlink /var/log off to one of your larger volumes, but that'd be a last resort for me. Personally, I'd copy everything from /home to /big, reinstall everything (and resizing the root volume on sda3) to be something more along the lines of 250GB, and resize /home accordingly. Don't format /big, of course, and just copy things back to /home when you're done.
 
Old 02-01-2022, 03:46 PM   #4
jim_cliff11
LQ Newbie
 
Registered: Jan 2004
Posts: 21

Original Poster
Rep: Reputation: 0
Thanks for the help.

I've found the following kernels:

Code:
kernel-firmware-ti-20210208-2.4.noarch
kernel-firmware-nvidia-20210208-2.4.noarch
kernel-firmware-liquidio-20210208-2.4.noarch
kernel-firmware-bluetooth-20210208-2.4.noarch
kernel-firmware-realtek-20210208-2.4.noarch
kernel-firmware-mwifiex-20210208-2.4.noarch
kernel-firmware-i915-20210208-2.4.noarch
kernel-firmware-ath10k-20210208-2.4.noarch
kernel-firmware-usb-network-20210208-2.4.noarch
kernel-firmware-serial-20210208-2.4.noarch
kernel-firmware-prestera-20210208-2.4.noarch
kernel-firmware-network-20210208-2.4.noarch
kernel-firmware-media-20210208-2.4.noarch
kernel-firmware-intel-20210208-2.4.noarch
kernel-firmware-brcm-20210208-2.4.noarch
kernel-firmware-ath11k-20210208-2.4.noarch
kernel-firmware-sound-20210208-2.4.noarch
kernel-firmware-qlogic-20210208-2.4.noarch
kernel-firmware-nfp-20210208-2.4.noarch
kernel-firmware-mediatek-20210208-2.4.noarch
kernel-firmware-iwlwifi-20210208-2.4.noarch
kernel-firmware-chelsio-20210208-2.4.noarch
kernel-firmware-atheros-20210208-2.4.noarch
kernel-firmware-all-20210208-2.4.noarch
kernel-default-extra-5.3.18-59.5.2.x86_64
kernel-default-optional-5.3.18-59.5.2.x86_64
kernel-firmware-radeon-20210208-2.4.noarch
kernel-firmware-mellanox-20210208-2.4.noarch
kernel-firmware-dpaa2-20210208-2.4.noarch
kernel-firmware-amdgpu-20210208-2.4.noarch
nfs-kernel-server-2.1.1-10.18.1.x86_64
kernel-firmware-ueagle-20210208-2.4.noarch
kernel-firmware-platform-20210208-2.4.noarch
kernel-firmware-marvell-20210208-2.4.noarch
kernel-firmware-bnx2-20210208-2.4.noarch
kernel-default-5.3.18-59.5.2.x86_64
purge-kernels-service-0-8.3.1.noarch
'zypper purge-kernels' gives me:

Code:
Reading installed packages...

Preparing to purge obsolete kernels...
Configuration: latest,latest-1,running
Running kernel release: 5.3.18-59.5-default
Running kernel arch: x86_64

Resolving package dependencies...
Nothing to do.
How do I go about safely removing these kernels? I'm running
Code:
5.3.18-59.5-default #1 SMP Wed Jun 2 08:21:36 UTC 2021 (eaf040d) x86_64 x86_64 x86_64 GNU/Linux
'du -sh *' of the /var subvolume:

Code:
97M     adm
4.0K    agentx
158M    cache
0       crash
259M    lib
4.0K    lock
33M     log
4.0K    mail
0       opt
4.0K    run
43M     spool
0       tmp
36M     webmin
'du -sh *' of /tmp:

Code:
0       systemd-private-9214e7d5c7cd403e8e095e43e9d69adb-chronyd.service-KfD3gi
0       systemd-private-9214e7d5c7cd403e8e095e43e9d69adb-systemd-logind.service-efpwbh
Although there are other folder in /tmp:

Code:
.ICE-unix   .X11-unix  .font-unix  systemd-private-9214e7d5c7cd403e8e095e43e9d69adb-chronyd.service-KfD3gi
.Test-unix  .XIM-unix  .webmin     systemd-private-9214e7d5c7cd403e8e095e43e9d69adb-systemd-logind.service-efpwbh
.

I've run 'rpm --rebuilddb' and 'zypper clean --all' with no drastic effect. Is there an obvious way to check snapshots is not enabled?

So far, there's nothing to point to heavy disk usage by a particular file or directory. I don't understand how I can resolve this apart from reinstalling and using an ext4 filesystem.

Thanks
 
Old 02-01-2022, 04:44 PM   #5
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 26,636

Rep: Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965
Quote:
Originally Posted by jim_cliff11 View Post
Thanks for the help. I've found the following kernels:
Code:
kernel-firmware-ti-20210208-2.4.noarch
kernel-firmware-nvidia-20210208-2.4.noarch
kernel-firmware-liquidio-20210208-2.4.noarch
kernel-firmware-bluetooth-20210208-2.4.noarch
kernel-firmware-realtek-20210208-2.4.noarch
kernel-firmware-mwifiex-20210208-2.4.noarch
kernel-firmware-i915-20210208-2.4.noarch
kernel-firmware-ath10k-20210208-2.4.noarch
kernel-firmware-usb-network-20210208-2.4.noarch
kernel-firmware-serial-20210208-2.4.noarch
kernel-firmware-prestera-20210208-2.4.noarch
kernel-firmware-network-20210208-2.4.noarch
kernel-firmware-media-20210208-2.4.noarch
kernel-firmware-intel-20210208-2.4.noarch
kernel-firmware-brcm-20210208-2.4.noarch
kernel-firmware-ath11k-20210208-2.4.noarch
kernel-firmware-sound-20210208-2.4.noarch
kernel-firmware-qlogic-20210208-2.4.noarch
kernel-firmware-nfp-20210208-2.4.noarch
kernel-firmware-mediatek-20210208-2.4.noarch
kernel-firmware-iwlwifi-20210208-2.4.noarch
kernel-firmware-chelsio-20210208-2.4.noarch
kernel-firmware-atheros-20210208-2.4.noarch
kernel-firmware-all-20210208-2.4.noarch
kernel-default-extra-5.3.18-59.5.2.x86_64
kernel-default-optional-5.3.18-59.5.2.x86_64
kernel-firmware-radeon-20210208-2.4.noarch
kernel-firmware-mellanox-20210208-2.4.noarch
kernel-firmware-dpaa2-20210208-2.4.noarch
kernel-firmware-amdgpu-20210208-2.4.noarch
nfs-kernel-server-2.1.1-10.18.1.x86_64
kernel-firmware-ueagle-20210208-2.4.noarch
kernel-firmware-platform-20210208-2.4.noarch
kernel-firmware-marvell-20210208-2.4.noarch
kernel-firmware-bnx2-20210208-2.4.noarch
kernel-default-5.3.18-59.5.2.x86_64
purge-kernels-service-0-8.3.1.noarch
'zypper purge-kernels' gives me:
Code:
Reading installed packages...
Preparing to purge obsolete kernels...
Configuration: latest,latest-1,running
Running kernel release: 5.3.18-59.5-default
Running kernel arch: x86_64

Resolving package dependencies...
Nothing to do.
How do I go about safely removing these kernels? I'm running
Code:
5.3.18-59.5-default #1 SMP Wed Jun 2 08:21:36 UTC 2021 (eaf040d) x86_64 x86_64 x86_64 GNU/Linux
If you look at the packages, you have ONE kernel installed...the one you're using. The others are firmware and other associated kernel files for your system.
Quote:
'du -sh *' of the /var subvolume:
Code:
97M     adm
4.0K    agentx
158M    cache
0       crash
259M    lib
4.0K    lock
33M     log
4.0K    mail
0       opt
4.0K    run
43M     spool
0       tmp
36M     webmin
'du -sh *' of /tmp:
Code:
0       systemd-private-9214e7d5c7cd403e8e095e43e9d69adb-chronyd.service-KfD3gi
0       systemd-private-9214e7d5c7cd403e8e095e43e9d69adb-systemd-logind.service-efpwbh
Although there are other folder in /tmp:
Code:
.ICE-unix   .X11-unix  .font-unix  systemd-private-9214e7d5c7cd403e8e095e43e9d69adb-chronyd.service-KfD3gi
.Test-unix  .XIM-unix  .webmin     systemd-private-9214e7d5c7cd403e8e095e43e9d69adb-systemd-logind.service-efpwbh
.
I've run 'rpm --rebuilddb' and 'zypper clean --all' with no drastic effect. Is there an obvious way to check snapshots is not enabled?
Running "snapper list" should; documentation:
https://doc.opensuse.org/documentati...a-snapper.html
Quote:
So far, there's nothing to point to heavy disk usage by a particular file or directory. I don't understand how I can resolve this apart from reinstalling and using an ext4 filesystem.
You may have update packages remaining; "sudo zypper clean --all" should remove the already-installed update packages (if any), and clear the cache. Beyond that, you're going to have to look to see what's taking up space. SOMETHING is...start with /root, and start traversing things. Lots of suggestions on how to do it, but running:
Code:
sudo du -a / | sort -n -r | head -n 5
...will do it. It will take a while to run, though.
 
Old 02-05-2022, 10:17 AM   #6
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 2,798

Rep: Reputation: 1201Reputation: 1201Reputation: 1201Reputation: 1201Reputation: 1201Reputation: 1201Reputation: 1201Reputation: 1201Reputation: 1201
Do you have deleted files in use?
Code:
lsof | grep -w deleted
(This concerns all file system types.)
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] docker on btrfs using much space in /var/lib/docker/btrfs taumeister Linux - Containers 2 01-25-2018 04:13 PM
btrfs filesystem df /mnt/btrfs/ shows bothe data and metdata with raid1 and single. gonus Linux - Software 3 07-17-2014 02:48 PM
LXer: Rollback To A Working State With btrfs + apt-btrfs-snapshot On Ubuntu 12.10 LXer Syndicated Linux News 0 11-25-2012 10:42 AM
ext4 vs btrfs, and with btrfs do I still need to use LVM? blue_k Linux - Software 4 08-17-2012 11:31 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 02:35 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration