Hey guys,

I am having an issue with Debootstrap using my own mirror of debian. Using apt-mirror, apt-ftparchive and GPG, once a day my personal mirror server synchronizes with "http://ftp.us.debian.org/debian" using apt-mirror. Following that, I have a perl script that confirms the MD5s of the Packages files and the listed packages within the Packages file. If everything goes well and the MD5 checks pass, I sign the Release with my own GPG/Trusted key. Another thing to note is apt-mirror is "cleaning" the mirror, so theoretically the two (mine and ftp.us.debian.org's repos) should match exactly, other than the fact I signed the Release file instead of Debian.

Currently, when I launch up a debootstrap using ftp.us.debian.org as my repo, debootstrap works exactly as expected and I eventually have a working Debian base system. When I use my self-signed mirror instead of ftp.us.debian.org, I get installation failures due to dependency issues and have to add an "--exclude" line with several packages to my debootstrap command to allow debootstrap to complete. I've done some investigation into what conflicts specifically, but I feel that is a symptom of the problem, and not the problem.

When I download both "Packages" files from my repo and Debian's, grep for the "Package: " line, pipe through 'sort' and 'uniq', and then run a diff on the two, the repos match exactly package-wise. They both have an exact match for unique packages:

"tim@timmey:~/tmp$ diff Packages-my-debian-repo.list Packages-debian.list
tim@timmey:~/tmp$ wc -l Packages-my-debian-repo.list Packages-debian.list
22025 Packages-my-debian-repo.list
22025 Packages-debian.list
44050 total
"

Apt versions from repo server:

"(14:30 tim@repo:/data/repo/config) dpkg -l |grep apt
ii apt 0.7.20.2+lenny1 Advanced front-end for dpkg
ii apt-file 2.1.5 APT package searching utility -- command-lin
ii apt-utils 0.7.20.2+lenny1 APT utility programs
ii aptitude 0.4.11.11-1~lenny1 terminal-based package manager
ii libapt-pkg-perl 0.1.22+b1 Perl interface to libapt-pkg
"

Debootstrap versions from server trying debootstrap:

"s1d:~# dpkg -l|grep debootstrap
ii cdebootstrap 0.5.4 Bootstrap a Debian system
ii debootstrap 1.0.10lenny1 Bootstrap a basic Debian system
"

/etc/apt/mirror.list:

"(14:26 tim@repo:~) cat /etc/apt/mirror.list|egrep -v "(^$|^#)"
set nthreads 20
set _tilde 0
deb http://ftp.us.debian.org/debian lenny main
deb http://security.debian.org/ lenny/updates main
clean http://ftp.us.debian.org/debian
clean http://security.debian.org/
skip-clean http://ftp.us.debian.org/debian/doc/
skip-clean http://ftp.us.debian.org/debian/tools/
skip-clean http://ftp.us.debian.org/debian/project/
skip-clean http://ftp.us.debian.org/debian/indices/
"

The config for apt-ftparchive is fairly typical and I can confirm that area is working nicely. We use binary caching and generate each component/subdir of the repo's pool/ dir. Release, Packages, all the MD5s, etc are all working flawlessly. This issue only shows when using debootstrap.

Has anyone else ran into this issue. Is there something special "ftp.us.debian.org" is doing to support deboostrap that I am not?

I can gladly provide more config files and data that anyone would like to look at. I didn't want to spam my post with lots of long configs. Any help, ideas, experienencies, guesses would be greatly appreciated!

Cheers,

Tim

More info:

The debootstrap command being ran (--foreign has no effect enabled or disabled):

"debootstrap --foreign \
--include locales \
--components main,security lenny /mnt/fl http://myrepo.com/debian"

- /mnt/fl = an LVM formatted with ext3.
- "security" component is a mirror of debian's security repo which is done using the same procedure and is a component of our repo. Removing this has no effect to the issue.

The error deboostrap crashes on over and over:

"I: Unpacking iptables...
I: Unpacking iproute...
I: Unpacking ifupdown...
I: Unpacking wget...
I: Unpacking libtasn1-3...
W: Failure while installing base packages. This will be re-attempted up to five times."

If I chroot into the dir after deboostrap fails, the following wants to be installed:

"s1d:~# chroot /mnt/fl
s1d:/# apt-get -f install
Reading package lists... Done
Building dependency tree... Done
Correcting dependencies... Done
The following extra packages will be installed:
binutils cpp-4.1 g++-4.1 gcc-4.1 gcc-4.1-base libc6 libc6-dev libstdc++6-4.1-dev linux-libc-dev
Suggested packages:
binutils-doc gcc-4.1-locales g++-4.1-multilib gcc-4.1-doc gcc-4.1-multilib libmudflap0-dev glibc-doc manpages-dev
libstdc++6-4.1-doc
Recommended packages:
gcc c-compiler
The following NEW packages will be installed:
binutils cpp-4.1 g++-4.1 gcc-4.1 gcc-4.1-base libc6-dev libstdc++6-4.1-dev linux-libc-dev
The following packages will be upgraded:
libc6
1 upgraded, 8 newly installed, 0 to remove and 2 not upgraded.
75 not fully installed or removed.
Need to get 18.1MB of archives.
After this operation, 45.6MB of additional disk space will be used.
Do you want to continue [Y/n]?
"

Cheers,

Tim

I'm starting to think my issue might be related to "override files". ftp.us.debian.org uses override files in the "/indices" directory, which I do not. This overrides the package definition of packages when enabled at apt-ftparchive "Packages" file generation time. It seems ftp.us.debian.org's "Packages.gz" contains these overrides while mine does not. In apt-ftparchive I also have "OverrideDir" commented out for this repo.

Generating new repo data now using debian's override files and "OverrideDir" enabled with apt-ftparchive. That will take a while. I can provide a resolution (to my own issue, lol) if that works.

By the way: Override files in general are very sparingly documented for anyone running into similar issues.

Cheers,

Tim

It was in fact "override files" that was the issue. Once I pulled down the override files from "http://ftp.us.debian.org/debian/indices/override.*.gz" to my repo and set apt-ftparchive up to recognize them, everything was resolved. The override file made my Packages file generate with several packages that were previously "important" set to "optional". After that, I removed the --exclude line from my debootstrap and everything worked flawlessly.

I only needed to get the override files for lenny/main, but others could be used. Here is the "Tree" section of my apt-ftparchive.conf, those two override lines at the bottom are what I added to my config:

"Tree "dists/lenny" {

// This is a space separated list of sections which appear under
// the distribution, typically this is something like main
// contrib non-free
Sections "main security";

// This is a space separated list of all the architectures that
// appear under search section. The special architecture 'source'
// is used to indicate that this tree has a source archive.
Architectures "amd64";

// Overrides:
BinOverride "/data/repo/debian/indices/override.lenny.$(SECTION)";
ExtraOverride "/data/repo/debian/indices/override.lenny.extra.$(SECTION)";

}
"

Solved this one myself! Cheers!

Tim