Huge tarball-what gives?
 

My O/S is Ubuntu 10.04.

While working on another problem and following guidance I found on the net, I created a tarball of my /home directory. My /home consists of three directories and their "sizes":

lost + found - ? items
myusername - 18 items
tester - 9 items

checking properties, it shows

lost + found - unreadable
myusername - 10.5 GB
tester - 32.2MB

(dispensing with "tester" which if I remember correctly, I set up to work on a long ago problem, but didn't actually use it. It is essentially empty.)

The important thing is that in addition to the three directories above, there is the "home.tar.gz" tarball which I created using this command (from within /home) :

sudo tar -cjf home.tar.gz myusername

and that tarball is 20.6 GB or twice the size of /myusername. How can this be?

Also can I delete the /tester directory without causing a problem, and if so, can I simply right click in Gnome and delete?

Finally can I delete /lost & found without causing a problem?

What does a du /home/username show?

pages and pages listing directories and subdirectories. Is there something specific I should look for? Thanks.

Use du -hs /home/username instead. The -s is for "summary" (so no voluminous listing of directories) and the -h for "human readable"

Thanks for the update. It returned:

Would this be 25 Gigs?

Check if there are any hidden files/directories in /home/username ?

You can delete /tester directory, but first check if there is any important data of yours.

And there is no need to delete lost + found directory.

This is a known and annoying bug reported here: https://bugs.launchpad.net/ubuntu/+s...lus/+bug/33883. The best way to determine the actual size of a directory is by means of the command line and the du command, as suggested by frankbell and Catkin above.

Regarding the tester home directory the cleaner way to delete it is by removing the tester user.

Instead, regarding the lost+found directory, it's there for a reason: it is related to ext2/ext3/ext4 filesystems and serves to store pieces of corrupted/broken files, generated for example upon a system crash or a sudden and unexpected shutdown. At the next boot the system will forcedly check the filesystems that are not in a clean state and it will try to repair corrupted files, if any. Therefore, better to leave it alone in order to assure the proper behaviour of your system.

Yes.

That implies ~15 GB in hidden directories and files if the linked bug is the cause of the discrepancy. Not impossible but an unusually large amount. This find command will list files over 250 MB for you to investigate (you may want to increase or decrease that 250 in case you get too many or too few files listed to be useful.

I am very anxious to get back to work and have taken the following steps in order to get the files across to the NEW computer.

I removed "tester" as a user (since it was never even used) and moved /oldhome out of my /home/username (into /opt) to clean the /home directory up and to reduce its total size.

I want to zip (the now much smaller) /home again, but this time have the tarball file placed outside of /home as it is created to avoid a "double compression" issue.

I have tried several variations of tar with various switches and destinations, but nothing wants to work. Have read the tar man pages, which is way too technical for me, and google is not being helpful. Could I trouble someone to please extend (and correct if needed) the line of code below to have the tarball show up in /opt (where there is space for it (or if /opt is not a good place to put it, your suggestion):

where "currenthome" will be the tarball, "/home" will be the current home folder that is being tarballed, and XYZ will be whatever else is needed to actually get the tarball into /opt (or some other place you think better. I also recognize that XYZ may not be properly located in the line of code. I just put it there to represent the additional step of placing the tarball elsewhere.)

Many thanks.

Anywhere with enough space would be a suitable temporary location. /tmp and /var/tmp are specifically for temporary files but may be emptied on shutdown/reboot.

The tar command you posted would add XYZ to the archive. You can do what you intend by something like
.tgz is conventional for compressed tar archives.

I want to thank the posters who added to my knowledge of du.

For what it's worth, in my experience, there is little utility to moving hidden configuration files from an old computer to a new one except when they are files I've configured away from their default settings.

For example, I will always copy my Fluxbox menu, startup, and init files; my dot-opera directory with bookmarks, mailstore, and so on; my .pan2 directory with my newsgroup subscriptions and configuration; and my LibreOffice templates, because I've tailored them to my needs and preferences. I'll also grab my /etc/samba/smb.conf file and my rc.firewall script because I expect to reuse them.

I don't copy over any configuration files that I've left at default. Not copying over config directories and files for programs I expect to leave at default greatly decreases the size of the data needing to be transferred.

That nailed it. Thanks so much.

Arrarrgh! This is becoming more complicated. I thought that I had deleted the old home.tar.gz from the /home directory, but had not. So the new tar file of /home contained the previous tar file. (The new tar file is 32GB!)

So I deleted the new tar file from /opt, then deleted the old home.tar.gz from /home. Then emptied the trash.

Then logged out and at the intitial login screen, pressed Alt-Ctrl-F1 to bring up a command line terminal. But Alt-Ctrl-F1 now doesn't work.

I rebooted a couple of times, then powered off, but Alt-Ctrl-F1 still doesn't work. So I then logged in, brought up a console and tried to create the tar from the console, but it failed. Arrrrrgh!

So now I need to get Alt-Ctrl-F1 working again at the Login screen. Any ideas?

EDIT: Found it! Ctrl key on keyboard failed! What are the chances of that? Thanks.

Still bashing away and seem to be no closer to success. In fact I am in a terrible muddle. Here is a summary of the history:

in old computer I ran:
(put into opt to keep from creating the tarball within the directory being zipped),

then from within /opt
which failed with error message: "permission denied, please try again" (i.e., would not accept my password [which I know is correct and retried several times]), then
which worked

Then in /home at the new computer, I renamed /username /username2, moved username120305.tar.gz into /home and unpacked it.

Now when I boot up new computer, I put in my password at login as normal, but then get the following three error messages on a blank desktop (color only, no icons)

1)Could not update ICEauthority file /home/username/ICEauthority
2)There is a problem with the configuration server. (/usr/lib/libgconf2-4/gconf-sanity-check-2 exited with status 256)
3)Nautilus could not create the following required folders: /home/username/Desktop, /home.username.nautilus (Before running Nautilus, please create these folders, or set permissions such that Nautilus can create them.)

To add to the confusion, I cannot bring back /username2 because it is gone, and I have no idea where.

So my first question is, can anyone see anything in the above that caused this whole process to go pear shaped?

Finally could someone please give me the command to locate a specific directoryname anywhere on the computer (still looking for /username2)

I am very much a noob at the command line, so more information rather than less will be so appreciated (I hope to learn something from all this) Thanks.

Given that the tar archive contained /home, you should have moved it to / and unpacked it. Maybe it worked OK and would have worked from anywhere because it explicitly contained /home with the leading /.

The errors could be caused by ownerships and permissions on the new files. Is your username and its group name the same on old and new computers? You could look in /etc/passwd and /etc/group to check.

Can you log in at a psuedo-terminal (accessed by Alt+Ctrl+F1 from the graphical logon screen?

If so and you create a file in /tmp and examine its username and group name, do they match what is in you new home directory? If you change to you home directory is it the expected /home/username? These commands can be used:
The two cd comands should take you to the same directory.

This command will search for the lost directory: sudo find / -type d -iname username2

catkin, I was very careful to set up exactly the same username and pw from the old on the new computer. Yes, I switch between GUI to (psuedo) terminal by logging out and using Alt+Ctrl+F1 to bring up a terminal view, then sudo restart to get back to the login to open Gnome.

I cannot locate either /passwd or /group in /etc? Am working on locating those directories.

I suspected that where I went wrong yesterday was to have zipped the /home directory instead of /home/myusername, so today I made a new tarball of /myusername (in /opt) on OLD,then:

went to the new computer and using the find command you gave me, located the "missing" /myusername2 file which wasn't missing at all. I was just looking at the wrong end of lines of the list. (good grief!) Anyway renamed /myusername and replaced it by renaming /myusername2 as /myusername, all this to restore the NEW computer to it's original state before yesterday's changes described earlier.

This "re-enabled" my ability to scp the tarball over to /myusername on NEW, then:
-renamed /myusername
-moved myusername.tar.gz into /home and unzipped it, then rebooted.

No cigar. Exactly the same 3 error messages as I listed last night.

Any ideas?

It's OK to log in at a virtual terminal while also logged in to the GUI desktop -- except when messing with files used by the desktop as suggested below!

/etc/passwd and /etc/group are files.

The error messages in post 14 do not look quite right. If possible, please copy and paste rather than type in (it's easier too).

I'm not running Gnome desktop so it is difficult to compare things on my system.

Working in a virtual terminal (when not logged in to the GUI desktop) ...

You could try mv ~/.ICEauthority ~/.ICEauthority.aside and then try logging into the GUI desktop.

How does the output of these commands (run either in a virtual terminal or in a GUI terminal emulator in the desktop) compare on old and new systems:
(one of ~/nautilus ~/.nautilus will not exist)

I am going to need to spend a bit of time with your latest to make sure that I understand it, but in the meantime, here is something that I just came up with:

On OLD computer in /home and using: ls -l, it reports:
drwx------ 2 root root 16384 date time lost+found
drwxr-xr-x 47 username username 4096 date time username

On NEW computer:
drwx------ 2 root root 16384 date time lost+found
drwxr-xr-x 47 root root 4096 date time home

I do not know, but would like to know, what is the significance of
username username in OLD vs
root root in NEW (i.e. I think the "root root" should be "username username" and do not know why the "root root" is there, but may have to do with the naming of the tarball?)

but certainly the "home" in the NEW must be problematic as this should be "username" as it was in OLD

I tried renaming home to username, but still have the same three error message upon login and so the "root root" must also be significant.

Does this help diagnose the problem?

The old computer's directory layout is very unusual. The conventional layout is /home (owned by root:root) and a subdirectory for each user having the same name as the user's name, owned by the user and with the same group as the user's primary group (on some systems having the same name as the user name). The lost+found directory was explained by colucix in post 7.

The Internet is a wild and dangerous place on which it is prudent not to disclose personal information but items such as user names, group names and timestamps are not usually regarded as dangerous to reveal. Unless you have special reasons for not revealing them, it would be easier to help you if you could copy and paste them verbatim.

Each user's home directory is set in the /etc/passwd file in the 6th :-delimited field. Their primary group is denoted by a number in the 4th field. The corresponding name comes from the /etc/group file by looking up the line wiyh that number in the 3rd field and taking the name from the first.

Catkin, Your continuing assistance is very much appreciated. I hope to be able to resolve this today although I realize that differences in time zones may interfere. As I attempt to copy the console output as you suggest, I would like to better understand the issues you mentioned.

You said, So do I understand that the setup on the NEW computer appears to be the conventional one? Also you wrote "/home" whereas the console on NEW shows just "home" and I wonder if the "/" is significant in terms of it working or not, or is that just you making sure to express that it is a directory not a file?

Also What is the best way for me to examine each of those files without possibly inadvertently changing something?

Also if someone else wants to add their comments, you are most welcome to do so. Thanks.

/home is used to show that it is a directory directly under the root of the file system (/).

When you wrote "I cannot locate either /passwd or /group in /etc" it was technically wrong -- you should have written "I cannot locate either passwd or group in /etc".

Can you give more detail on "the console on NEW shows just "home""? Which command are you running to make it show that?

One convention sometimes used to show that something is a directory and not a file is to put a trailing / on it. The convention is not widely enough used to be much use, though.

For a read-only view of files you can use cat /etc/passwd or, for a page at a time less /etc/passwd then space to page forward and q to quit.

Sometimes it's like this on LQ -- if a thread is progressing it ends up as a two-way conversation which in this case of widely separated time zones is not ideal.

I have the contents of the passwd in view, but it appears that I need to ask you what "in the 6th :-delimited field. Their primary group is denoted by a number in the 4th field" means. I can't see anything that I can match up to that description. Thanks.

As soon as I get this sorted, I will copy the info regarding permissions and any lines in passwd and group that appear relevant and post it here (don't want to reproduce the entire contents from those two because of length, but can if helpful to you?)

Here, I think, is what you have requested to see:

From OLD computer:

robert@Intel-i3-550:~$ ls -l /home
total 20
drwx------ 2 root root 16384 2010-11-20 12:37 lost+found
drwxr-xr-x 47 robert robert 4096 2012-03-07 17:43 robert
robert@Intel-i3-550:~$ ls -l /home/robert
total 84
-rw-r--r-- 1 robert robert 352 2011-07-29 05:48 AFRV-order 110726
-rw-r--r-- 1 robert robert 0 2011-02-04 18:58 Band Vista Q's
-rw-r--r-- 1 robert robert 1430 2011-07-26 13:41 Bookkeeper Responsibilities 110726.txt
drwxr-xr-x 24 robert robert 4096 2012-03-06 13:33 Desktop
drwxr-xr-x 2 robert robert 4096 2011-11-08 09:47 Documents
drwxr-xr-x 2 robert robert 20480 2012-03-07 16:53 Downloads
-rw-r--r-- 1 robert robert 2447 2011-06-19 12:37 DR Res email list
-rw-r--r-- 1 robert robert 179 2010-11-20 12:43 examples.desktop
drwxr-xr-x 2 robert robert 4096 2011-01-27 21:44 Music
-rw-r--r-- 1 robert robert 3113 2011-02-23 07:41 NotSxSW-13-16th
-rw-r--r-- 1 robert robert 4595 2011-02-23 07:46 NotSxSW-17-18th
-rw-r--r-- 1 robert robert 3113 2011-02-23 07:41 NotSxSW-Sunday 13
drwxr-xr-x 2 robert robert 4096 2010-11-20 21:19 Pictures
drwxr-xr-x 2 robert robert 4096 2010-11-20 21:19 Public
-rw-r--r-- 1 robert robert 819 2011-12-13 06:29 screen shot.txt
drwxr-xr-x 2 robert robert 4096 2010-11-20 21:19 Templates
drwxr-xr-x 2 robert robert 4096 2010-11-20 21:19 Videos


Because I can't do anything in NEW after I log in (this the problem I am trying to solve), I am restricted to the console (pseudo) or booting from a CD (which is maybe what I should do). Anyway the output from NEW is this (I have typed it but triple checked every character and am certain that it is 100% correct, including the dates and times):

robert@Intel-i3-550:~$ id
robert@robert-desktop:~$ ls -l /home
total 28
drwx------ 2 root root 16384 2010-11-20 12:39 lost+found
drwxr-xr-x 47 robert robert 4096 2012-03-07 12:38 robertorig
drwxr-xr-x 3 root root 4096 2012-03-07 12:45 robert
robert@robert-desktop:~$

(where "robertorig" is the original /home/user that I set up on NEW and later renamed when I wanted to unpack the tarball, which was unpacked into /home) (BTW, this is where I think I went wrong, just unsure how)

Also please note that there are none of the many files listed in NEW that show up in the "ls -l /home" output from OLD (at the top) and maybe another clue to what has gone wrong?

AND, I think I am making some progress figuring out the passwd and group thing, but let me know if I have failed to include anything you need to see. Here are selected line in the outputs from OLD:

robert@Intel-i3-550:~$ id
uid=1000(robert) gid=1000(robert) groups=1000(robert),4(adm),20(dialout),24(cdrom),46(plugdev),111(lpadmin),119(admin),122(sambashare)

robert@Intel-i3-550:~$ groups
robert adm dialout cdrom plugdev lpadmin admin sambashare

robert@Intel-i3-550:~$ grep robert /etc/passwd
robert:x:1000:1000:Robert,,,:/home/robert:/bin/bash

robert@Intel-i3-550:~$ grep robert /etc/group
adm:x:4:robert
dialout:x:20:robert
cdrom:x:24:robert
plugdev:x:46:robert
lpadmin:x:111:robert
admin:x:119:robert
robert:x:1000:
sambashare:x:122:robert
robert@Intel-i3-550:~$

cut from /etc/passwd

root:x:0:0:root:/root:/bin/bash
robert:x:1000:1000:Robert,,,:/home/robert:/bin/bash

and from group:

root:x:0:
adm:x:4:robert
dialout:x:20:robert
cdrom:x:24:robert
plugdev:x:46:robert
lpadmin:x:111:robert

These would seem to be the relevant lines but I am usure how to interpret them:

from passwd:
robert:x:1000:1000:Robert,,,:/home/robert:/bin/bash

from group:
adm:x:4:robert

Does this shed any light?

Here's the problem:
Hopefully fix by, as root

YES!!!!!! I'm so happy! Catkin, thank you most sincerely for hanging in there with me until this got resolved, and thank you for your insight. I have been beating my head against this wall of a problem for far longer than I like to admit.

The proposed solution worked. May I trouble you just a little more to please explain what you perceived the problem to be (i.e., "here's the problem"),

and what the corrective code actually did. I recognize chown as permission related but don't understand what was accomplished.

Thanks again, a hundred times.

Best regards,
robert

Great we got there in the end :)

Personal users need to be able to write to their home directories.

There's a good page about ownership and permissions here.