Building a Debian Wheezy/Ubuntu Precise Mirror using apt-mirror

bennetfox · 02-10-2013, 06:57 PM

Hello everyone!

I'm trying to build a mirror of Debian Wheezy 7.0 and Ubuntu Precise 12.04.1 on my local area network and I am using apt-mirror to do all of the downloading. I've done the mandatory reading and research, but I'm having a problem with the configuration file of apt-mirror. when I run an apt-get update on the local client, apt-get responds with:

Code:

$ sudo apt-get update

(...cut for length...)

W: Failed to fetch http://greenfox.foxden.lan/debian/dists/wheezy/updates/main/source/Sources  404  Not Found

W: Failed to fetch http://greenfox.foxden.lan/debian/dists/wheezy/updates/contrib/source/Sources  404  Not Found

W: Failed to fetch http://greenfox.foxden.lan/debian/dists/wheezy/updates/non-free/source/Sources  404  Not Found

W: Failed to fetch http://greenfox.foxden.lan/debian/dists/wheezy/updates/main/binary-i386/Packages  404  Not Found

W: Failed to fetch http://greenfox.foxden.lan/debian/dists/wheezy/updates/contrib/binary-i386/Packages  404  Not Found

W: Failed to fetch http://greenfox.foxden.lan/debian/dists/wheezy/updates/non-free/binary-i386/Packages  404  Not Found

W: Failed to fetch http://greenfox.foxden.lan/debian/dists/wheezy/contrib/i18n/Translation-en  404  Not Found

W: Failed to fetch http://greenfox.foxden.lan/debian/dists/wheezy/main/i18n/Translation-en  404  Not Found

W: Failed to fetch http://greenfox.foxden.lan/debian/dists/wheezy/non-free/i18n/Translation-en  404  Not Found

E: Some index files failed to download. They have been ignored, or old ones used instead.

This is my /etc/apt/mirror.list file:

Code:

$ /etc/apt/mirror.list

$ cat /etc/apt/mirror.list
############# config ##################

set base_path    /home/apt-mirror
set mirror_path  $base_path/mirror
set skel_path    $base_path/skel
set var_path     $base_path/var
set cleanscript $var_path/clean.sh
set defaultarch  i386
set postmirror_script $var_path/postmirror.sh
set run_postmirror 0
set nthreads     20
set _tilde 0

############# end config ##############

############# Debian 7.0 Wheezy ################

deb http://debian.osuosl.org/debian/ wheezy main contrib non-free
deb-src http://debian.osuosl.org/debian/ wheezy main contrib non-free
deb http://security.debian.org/ wheezy/updates main contrib non-free
deb-src http://security.debian.org/ wheezy/updates main contrib non-free
deb http://debian.osuosl.org/debian/ wheezy-updates contrib non-free main

This is the sources.list file I am using on the local machine:

Code:

$ cat /etc/apt/sources.list
deb http://greenfox.foxden.lan/debian/ wheezy main contrib non-free
deb-src http://greenfox.foxden.lan/debian/ wheezy main contrib non-free

deb http://greenfox.foxden.lan/debian/ wheezy/updates main contrib non-free
deb-src http://greenfox.foxden.lan/debian/ wheezy/updates main contrib non-free
deb http://greenfox.foxden.lan/debian/ wheezy-updates contrib non-free main

Did I overlook something in my configurations or do I have a typo somewhere that I'm just not seeing?

Any help would be greatly appreciated!

lykwydchykyn · 02-11-2013, 01:19 PM

Not sure about all of the problem, but the translation files do not automatically download; this is a known bug in apt-mirror.

I added the following to my postmirror script:

Code:

cd /srv/apt-mirror/mirror/ftp.us.debian.org/debian/dists/wheezy

for directory in contrib main non-free; do                                                                                                                     
    cd $directory                                                                                                                                              
    mkdir i18n 2>/dev/null                                                                                                                                     
    cd i18n                                                                                                                                                    
    rm Translation-en*                                                                                                                                         
    wget http://ftp.us.debian.org/debian/dists/wheezy/$directory/i18n/Translation-en.bz2                                                                       
    cd ../../                                                                                                                                                  
done

You'll want to adapt it to your install, of course (my stuff is in a different directory than yours). Not sure about the other errors since I don't mirror sources. I assume the apt-mirror script has run at least once?

bennetfox · 02-11-2013, 04:13 PM

It's funny you say that because yesterday in amongst my Googling I ran across a page that talked about my error and how the author of apt-mirror has been informed of it, but hasn't fixed the issue yet. I saw a few solutions on how to fix the issue by inserting code into the source but that's just a bit past my level of expertise (I'm not an uber-techie-geek yet, I'm still learning!

). Your solution is something I can implement and I do greatly appreciate it!

Since apt-mirror isn't functioning fully to my expectations, I might take a look at the ftp scripts using rsync to keep my local mirrors updated instead.

Thank you again for your help!

bennetfox · 02-15-2013, 06:19 PM

I'm still having troubles mirroring the wheezy/updates locally. Every time I run apt-get update on my local debian machines, I'm presented with this:

Code:

Err http://greenfox wheezy/updates/main Sources
  404  Not Found
Err http://greenfox wheezy/updates/contrib Sources
  404  Not Found
Err http://greenfox wheezy/updates/non-free Sources
  404  Not Found
Err http://greenfox wheezy/updates/main i386 Packages
  404  Not Found
Err http://greenfox wheezy/updates/contrib i386 Packages
  404  Not Found
Err http://greenfox wheezy/updates/non-free i386 Packages
  404  Not Found

Any help would be greatly appreciated!

lykwydchykyn · 02-15-2013, 07:15 PM

Maybe the mirror itself is having trouble synchronizing?

Have you tried running:

Code:

su apt-mirror -c apt-mirror

As root in a terminal to see if everything mirrors without error?

Also, can you post the apache configuration for the web server?

bennetfox · 02-16-2013, 02:59 PM

Hmm... Now this is interesting. I tried the command you suggested and this is the response I get:

Code:

# su - apt-mirror -c apt-mirror
Downloading 195 index files using 20 threads...
apt-mirror: can't write to intermediate file (index-urls.0) at /usr/bin/apt-mirror line 210.

When I normally run apt-mirror, I usually do it from the command line like this:

Code:

$ sudo apt-mirror

Hopefully soon we can get to the bottom of this! Any help would be appreciated!

lykwydchykyn · 02-16-2013, 06:55 PM

If you run:

Code:

sudo apt-mirror

The apt-mirror runs as root and all the files will be owned by root. The cron job it ships with runs as "apt-mirror", so it won't update properly because "apt-mirror" (the user) can't overwrite root's files.

You probably need to recursively chown all the files in the apt-mirror repository to apt-mirror, then try the su version of the command again.

can you list the contents of your /home/apt-mirror/mirrors directory?

bennetfox · 02-17-2013, 03:06 PM

Directory listing of /home/apt-mirror:

Code:

:/home/apt-mirror/mirror$ ls -la
total 36
drwxr-xr-x 9 apt-mirror apt-mirror 4096 Feb 10 15:21 .
drwxr-xr-x 5 apt-mirror apt-mirror 4096 Feb  6 18:23 ..
drwxr-xr-x 3 root       root       4096 Feb  9 14:35 archive.canonical.com
drwxr-xr-x 3 root       root       4096 Feb  6 18:29 debian.osuosl.org
drwxr-xr-x 3 root       root       4096 Feb  9 15:27 dl.google.com
drwxr-xr-x 3 root       root       4096 Feb  9 15:27 download.virtualbox.org
drwxr-xr-x 3 root       root       4096 Feb  7 09:20 extras.ubuntu.com
drwxr-xr-x 3 root       root       4096 Feb 10 15:21 security.debian.org
drwxr-xr-x 3 root       root       4096 Feb  6 18:29 ubuntu.osuosl.org

I also changed the ownerships to reflect the apt-mirror user:

Code:

$ sudo chown -R apt:mirror:apt-mirror apt-mirror/

Now my apt-mirror/mirrors directory looks like this:

Code:

$ ls -la
total 36
drwxr-xr-x 9 apt-mirror apt-mirror 4096 Feb 10 15:21 .
drwxr-xr-x 5 apt-mirror apt-mirror 4096 Feb  6 18:23 ..
drwxr-xr-x 3 apt-mirror apt-mirror 4096 Feb  9 14:35 archive.canonical.com
drwxr-xr-x 3 apt-mirror apt-mirror 4096 Feb  6 18:29 debian.osuosl.org
drwxr-xr-x 3 apt-mirror apt-mirror 4096 Feb  9 15:27 dl.google.com
drwxr-xr-x 3 apt-mirror apt-mirror 4096 Feb  9 15:27 download.virtualbox.org
drwxr-xr-x 3 apt-mirror apt-mirror 4096 Feb  7 09:20 extras.ubuntu.com
drwxr-xr-x 3 apt-mirror apt-mirror 4096 Feb 10 15:21 security.debian.org
drwxr-xr-x 3 apt-mirror apt-mirror 4096 Feb  6 18:29 ubuntu.osuosl.org

I just ran apt-mirror with the "su apt-mirror -c apt-mirror" command and everything completed successfully.

However I am still having the same error about apt-get update not being able to find the wheezy/updates repositories.

Code:

Err http://greenfox wheezy/updates/main Sources        
  404  Not Found
Err http://greenfox wheezy/updates/contrib Sources
  404  Not Found
Err http://greenfox wheezy/updates/non-free Sources
  404  Not Found
Err http://greenfox wheezy/updates/main i386 Packages
  404  Not Found
Err http://greenfox wheezy/updates/contrib i386 Packages
  404  Not Found
Err http://greenfox wheezy/updates/non-free i386 Packages
  404  Not Found

lykwydchykyn · 02-18-2013, 12:09 AM

Ok, I think I see the problem here.

"wheezy/updates" comes from a separate server than the other wheezy packages (security.debian.org). Because of this, it's in a different folder under the "mirrors" folder.

I don't know how you are sharing these out with your webserver, if it's directly or with aliases or symlinks or what, but the (webroot)/debian folder cannot point to *both* the main debian repository directory and the security server directory.

You need to make the security.debian.org folder available in it's own path on your web server, then adjust your client's sources.list file accordingly.

I don't know if I explained that clearly, let me know.

bennetfox · 02-18-2013, 03:52 PM

THANK YOU for explaining that to me! After reading your message, I looked at my /etc/apt/sources.list and noticed the slight differences. I've got the mirror directories linked to the /var/www directory so it's all accessable via http. I dug around a little bit in the directory structure and saw the different directories and I created a new symlink and the problem was solved!

Code:

$ sudo ln -s /home/apt-mirror/mirror/security.debian.org/ /var/www/security

apt-get update and apt-get upgrade went off without a hitch!

Now my next question is this: how do I add the debian/netinst mirror to it so I can mirror the network installation of Debian and Ubuntu? I would like to have the ability to install and upgrade a new installation of Debian/Ubuntu using local mirror resources instead of downloading software or updates from the Internets.

lykwydchykyn · 02-18-2013, 10:24 PM

Just add:

Code:

deb http://ftp.us.debian.org/debian/ wheezy main/debian-installer

To your sources. I have this on a server at work along with the netboot installer, works great. You have to manually keep the netboot installer up-to-date whenever there's a new point release, or else you get an error during install. Otherwise it works great.

bennetfox · 02-19-2013, 04:05 PM

Added the aforementioned line to my mirrors.list, resynched, and tried a network install and it went off without any issues! Thank you so much for your help!

What would I add to the mirrir.list file to accomplish the same thing with a Ubuntu network installation? I've been googling around all morning and haven't been able to find my answer!

lykwydchykyn · 02-20-2013, 10:32 AM

Here's what I've got on my mirror for Precise:

Code:

#Precise32
deb http://us.archive.ubuntu.com/ubuntu/ precise main restricted universe multiverse
deb http://us.archive.ubuntu.com/ubuntu/ precise-updates main restricted universe multiverse
deb http://us.archive.ubuntu.com/ubuntu/ precise-backports main restricted universe multiverse
deb http://us.archive.ubuntu.com/ubuntu/ precise-security main restricted universe multiverse
deb http://us.archive.ubuntu.com/ubuntu/ precise main/debian-installer 


#Precise64

deb-amd64 http://us.archive.ubuntu.com/ubuntu/ precise main restricted universe multiverse
deb-amd64 http://us.archive.ubuntu.com/ubuntu/ precise-updates main restricted universe multiverse
deb-amd64 http://us.archive.ubuntu.com/ubuntu/ precise-backports main restricted universe multiverse
deb-amd64 http://us.archive.ubuntu.com/ubuntu/ precise-security main restricted universe multiverse
deb-amd64 http://us.archive.ubuntu.com/ubuntu/ precise main/debian-installer

Keep in mind if you want to install 64-bit, multiarch is configured by default so it needs both the 64-bit and 32-bit repos. Kind of a pain, unfortunately.

bennetfox · 02-21-2013, 07:09 PM

I haven't forgotten about you, I've just been swamped with getting apt-mirror to function correctly as well as working on other projects as well!

Thank you so much for the mirror.list file. I'm curious if you also have one for Debian?

Let me see if I got this right, you have to mirror both 1386 and 64 bit distributions even if you only use just one?

lykwydchykyn · 02-22-2013, 03:56 PM

Quote:

Originally Posted by bennetfox

I haven't forgotten about you, I've just been swamped with getting apt-mirror to function correctly as well as working on other projects as well!

Thank you so much for the mirror.list file. I'm curious if you also have one for Debian?

Yeah, but I just mirror squeeze, and I'm pulling updates from the web:

Code:

#Debian Squeeze 
deb http://ftp.us.debian.org/debian/ squeeze main contrib non-free
deb http://ftp.us.debian.org/debian/ squeeze main/debian-installer 
deb http://ftp.us.debian.org/debian/ squeeze-updates main contrib non-free

Quote:

Let me see if I got this right, you have to mirror both 1386 and 64 bit distributions even if you only use just one?

If you are using 64 bit with multiarch enabled (I think that's default) then you need the 32bit repos or you'll get errors. If you are just running 32bit or if you turn off multiarch, you can just do whichever one.