Ubuntu upgrade failed because root partition is full

Deemar · 06-18-2017, 11:11 PM

I forgot to clear out my old kernels from my server as it was updating and now the root partition is full. I got halfway through an upgrade and it stopped because it ran out of space. How do I fix this?

Code:

ubuntu@ip-172-31-0-59:/boot$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            477M     0  477M   0% /dev
tmpfs           100M  4.3M   95M   5% /run
/dev/xvda1       99G   70G   25G  74% /
tmpfs           496M     0  496M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           496M     0  496M   0% /sys/fs/cgroup
cgmfs           100K     0  100K   0% /run/cgmanager/fs
tmpfs           100M     0  100M   0% /run/user/1000

Code:

ubuntu@ip-172-31-0-59:/boot$ df -i
Filesystem      Inodes   IUsed  IFree IUse% Mounted on
udev            122079     371 121708    1% /dev
tmpfs           126791     476 126315    1% /run
/dev/xvda1     6553600 6553599      1  100% /
tmpfs           126791       1 126790    1% /dev/shm
tmpfs           126791       5 126786    1% /run/lock
tmpfs           126791      18 126773    1% /sys/fs/cgroup
cgmfs           126797      14 126783    1% /run/cgmanager/fs
tmpfs           126797       4 126793    1% /run/user/1000

Code:

ubuntu@ip-172-31-0-59:/boot$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
You might want to run 'apt-get -f install' to correct these.
The following packages have unmet dependencies:
 linux-headers-4.4.0-79-generic : Depends: linux-headers-4.4.0-79 but it is not installed
E: Unmet dependencies. Try using -f.

Code:

  Could not open file /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_xenial-security_main_source_Sources.xz - open (28: No space left on device) [IP: 91.***.**.*** **]

Code:

ubuntu@ip-172-31-0-59:/boot$ ls -l
total 46768
-rw-r--r-- 1 root root  1245659 Mar 31 11:14 abi-4.4.0-72-generic
-rw-r--r-- 1 root root   190236 Mar 31 11:14 config-4.4.0-72-generic
drwxr-xr-x 5 root root     4096 Jun 18 21:01 grub
-rw-r--r-- 1 root root 35474648 Jun 18 21:01 initrd.img-4.4.0-72-generic
-rw------- 1 root root  3882277 Mar 31 11:14 System.map-4.4.0-72-generic
-rw------- 1 root root  7083248 Mar 31 11:14 vmlinuz-4.4.0-72-generic

syg00 · 06-18-2017, 11:18 PM

Deleting kernels isn't going to help. How on earth did you consume that number of inodes ?. Start by cleaning out whatever is eating them.

Deemar · 06-18-2017, 11:28 PM

I'm not really sure where to start, any ideas how I can free it up?

syg00 · 06-18-2017, 11:36 PM

Try this, it should give you some ideas (may take a few minutes to complete). Biggest will be listed first, they will give biggest buy-back.

Code:

sudo du / -xh --inodes | sort -hr | head -n 25

Deemar · 06-19-2017, 12:14 AM

Quote:

Originally Posted by syg00

Try this, it should give you some ideas (may take a few minutes to complete). Biggest will be listed first, they will give biggest buy-back.

Code:

sudo du / -xh --inodes | sort -hr | head -n 25

I've been running it for about 20 minutes and nothing has happened.

Code:

ubuntu@ip-***-**-*-**:~$ sudo du / -xh --inodes | sort -hr | head -n 25

Habitual · 06-19-2017, 07:56 AM

It's searching the entire file system.
Wait a few more...to completion.

Deemar · 06-19-2017, 02:23 PM

Quote:

Originally Posted by Habitual

It's searching the entire file system.
Wait a few more...to completion.

Okay I'm running it again right now.
I did notice one other thing, I found so many files under /var/tmp that it wouldn't even list the files in the folder, it just hung when I tried to run "ls" on it. I was able to delete some using the -U parameter to return them unordered and that freed up enough nodes to complete the autoremove function and left me with about 10,000 nodes available. I came back to it a few hours later and once again there were 0 nodes available, something ate them up. I have since shut down Apache2 since it's just a personal server but I'm wondering what else I can try here.

I'll post the results of that command after it completes.

Deemar · 06-19-2017, 02:52 PM

Code:

packet_write_wait: Connection to ****** port 22: Broken pipe

I'll have to wait until I get home, it's dropping my work machine for some reason. I'll run it again tonight.

Jjanel · 06-19-2017, 07:18 PM

Yes, it's a 'runaway' something. Here's some ideas for 'catching' it, from this search:
linux record what process creates a file
lsof, inotify, audit, ... Even chmod on the dir so pgm will choke&die

(with error msg hopefully, tho chmod could/would break other things!)

You can save a LOT of time by starting the du /var/tmp (instead of / IF that's where they are)

Puzzle heaven

Best wishes! Let us know, esp. how reSolved, for future searchers!

Barkester · 06-19-2017, 08:28 PM

I moved to another distro when it happened to me.

I think a clean install would be your most realistic option in any case whatever you use. Simply think how many hours you may waste on a fix when a reinstall can be done in how long? And its pretty easy.

Unless you just wanna try it for the project, anyhow.

Enjoy.

Deemar · 06-20-2017, 06:53 PM

Quote:

Originally Posted by Habitual

It's searching the entire file system.
Wait a few more...to completion.

It has been running for about 20 hours now and it still isn't finished.

syg00 · 06-20-2017, 07:53 PM

Quote:

Originally Posted by Deemar

I did notice one other thing, I found so many files under /var/tmp that it wouldn't even list the files in the folder, it just hung when I tried to run "ls" on it.

Where they all Apache ?.

You're in a bit of an awkward situation - too big to list, and deleting will also take "forever". Either will also consume a lot of slab space for the dentry and inode caches. I would just delete everything in /var/tmp - there should be nothing of consequence there, but there might be; it's (supposedly) for less transient data.
Maybe even give it a small partition of its own - just to see easily what goes in there.

Deemar · 06-21-2017, 12:21 AM

What's the easiest way to delete all files in a folder without doing a recursive "rm?" It seems to hang if I just try and delete them all, this is what I've been running every few minutes:

Code:

sudo ls -U | head -5000 | sudo xargs -d '\n' rm -rf --

That deletes the files without ordering them first.
Here is the result of deleting and then running a nodes check about every 10 minutes afterwards, the count is good but then slowly goes back towards zero.

Code:

ubuntu@ip-**-**-0-**:/var/tmp$ sudo ls -U | head -5000 | sudo xargs -d '\n' rm -rf --
ubuntu@ip-**-**-0-**:/var/tmp$ df -i
Filesystem      Inodes   IUsed  IFree IUse% Mounted on
udev            122038     370 121668    1% /dev
tmpfs           126788     494 126294    1% /run
/dev/xvda1     6553600 6539733  13867  100% /
tmpfs           126788       1 126787    1% /dev/shm
tmpfs           126788       5 126783    1% /run/lock
tmpfs           126788      18 126770    1% /sys/fs/cgroup
cgmfs           126794      14 126780    1% /run/cgmanager/fs
tmpfs           126794       4 126790    1% /run/user/1000
ubuntu@ip-**-**-0-**:/var/tmp$ sudo ls -U | head -5000 | sudo xargs -d '\n' rm -rf --
ubuntu@ip-**-**-0-**:/var/tmp$ df -i
Filesystem      Inodes   IUsed  IFree IUse% Mounted on
udev            122038     370 121668    1% /dev
tmpfs           126788     485 126303    1% /run
/dev/xvda1     6553600 6544392   9208  100% /
tmpfs           126788       1 126787    1% /dev/shm
tmpfs           126788       5 126783    1% /run/lock
tmpfs           126788      18 126770    1% /sys/fs/cgroup
cgmfs           126794      14 126780    1% /run/cgmanager/fs
tmpfs           126794       4 126790    1% /run/user/1000
ubuntu@ip-**-**-0-**:/var/tmp$ df -i
Filesystem      Inodes   IUsed  IFree IUse% Mounted on
udev            122038     370 121668    1% /dev
tmpfs           126788     485 126303    1% /run
/dev/xvda1     6553600 6544392   9208  100% /
tmpfs           126788       1 126787    1% /dev/shm
tmpfs           126788       5 126783    1% /run/lock
tmpfs           126788      18 126770    1% /sys/fs/cgroup
cgmfs           126794      14 126780    1% /run/cgmanager/fs
tmpfs           126794       4 126790    1% /run/user/1000
ubuntu@ip-**-**-0-**:/var/tmp$ df -i
Filesystem      Inodes   IUsed  IFree IUse% Mounted on
udev            122038     370 121668    1% /dev
tmpfs           126788     485 126303    1% /run
/dev/xvda1     6553600 6549251   4349  100% /
tmpfs           126788       1 126787    1% /dev/shm
tmpfs           126788       5 126783    1% /run/lock
tmpfs           126788      18 126770    1% /sys/fs/cgroup
cgmfs           126794      14 126780    1% /run/cgmanager/fs
tmpfs           126794       4 126790    1% /run/user/1000

syg00 · 06-21-2017, 12:32 AM

Do it from a liveCD - that way whatever is creating the files won't be a factor.

Deemar · 06-21-2017, 07:14 PM

Quote:

Originally Posted by syg00

Do it from a liveCD - that way whatever is creating the files won't be a factor.

I can't, it's an Amazon EC2 instance.