LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Debian
User Name
Password
Debian This forum is for the discussion of Debian Linux.

Notices

Reply
 
Search this Thread
Old 06-21-2010, 04:40 PM   #1
lil_drummaboy
Member
 
Registered: May 2003
Location: Victoria, CANADA!
Distribution: OpenBSD, Slackware, Debian
Posts: 85

Rep: Reputation: 15
Post Problems with Debootstrap and apt-mirror-based Debian mirror


Hey guys,

I am having an issue with Debootstrap using my own mirror of debian. Using apt-mirror, apt-ftparchive and GPG, once a day my personal mirror server synchronizes with "http://ftp.us.debian.org/debian" using apt-mirror. Following that, I have a perl script that confirms the MD5s of the Packages files and the listed packages within the Packages file. If everything goes well and the MD5 checks pass, I sign the Release with my own GPG/Trusted key. Another thing to note is apt-mirror is "cleaning" the mirror, so theoretically the two (mine and ftp.us.debian.org's repos) should match exactly, other than the fact I signed the Release file instead of Debian.

Currently, when I launch up a debootstrap using ftp.us.debian.org as my repo, debootstrap works exactly as expected and I eventually have a working Debian base system. When I use my self-signed mirror instead of ftp.us.debian.org, I get installation failures due to dependency issues and have to add an "--exclude" line with several packages to my debootstrap command to allow debootstrap to complete. I've done some investigation into what conflicts specifically, but I feel that is a symptom of the problem, and not the problem.

When I download both "Packages" files from my repo and Debian's, grep for the "Package: " line, pipe through 'sort' and 'uniq', and then run a diff on the two, the repos match exactly package-wise. They both have an exact match for unique packages:

"tim@timmey:~/tmp$ diff Packages-my-debian-repo.list Packages-debian.list
tim@timmey:~/tmp$ wc -l Packages-my-debian-repo.list Packages-debian.list
22025 Packages-my-debian-repo.list
22025 Packages-debian.list
44050 total
"

Apt versions from repo server:

"(14:30 tim@repo:/data/repo/config) dpkg -l |grep apt
ii apt 0.7.20.2+lenny1 Advanced front-end for dpkg
ii apt-file 2.1.5 APT package searching utility -- command-lin
ii apt-utils 0.7.20.2+lenny1 APT utility programs
ii aptitude 0.4.11.11-1~lenny1 terminal-based package manager
ii libapt-pkg-perl 0.1.22+b1 Perl interface to libapt-pkg
"

Debootstrap versions from server trying debootstrap:

"s1d:~# dpkg -l|grep debootstrap
ii cdebootstrap 0.5.4 Bootstrap a Debian system
ii debootstrap 1.0.10lenny1 Bootstrap a basic Debian system
"

/etc/apt/mirror.list:

"(14:26 tim@repo:~) cat /etc/apt/mirror.list|egrep -v "(^$|^#)"
set nthreads 20
set _tilde 0
deb http://ftp.us.debian.org/debian lenny main
deb http://security.debian.org/ lenny/updates main
clean http://ftp.us.debian.org/debian
clean http://security.debian.org/
skip-clean http://ftp.us.debian.org/debian/doc/
skip-clean http://ftp.us.debian.org/debian/tools/
skip-clean http://ftp.us.debian.org/debian/project/
skip-clean http://ftp.us.debian.org/debian/indices/
"

The config for apt-ftparchive is fairly typical and I can confirm that area is working nicely. We use binary caching and generate each component/subdir of the repo's pool/ dir. Release, Packages, all the MD5s, etc are all working flawlessly. This issue only shows when using debootstrap.

Has anyone else ran into this issue. Is there something special "ftp.us.debian.org" is doing to support deboostrap that I am not?

I can gladly provide more config files and data that anyone would like to look at. I didn't want to spam my post with lots of long configs. Any help, ideas, experienencies, guesses would be greatly appreciated!

Cheers,

Tim

Last edited by lil_drummaboy; 06-21-2010 at 04:43 PM.
 
Old 06-21-2010, 04:57 PM   #2
lil_drummaboy
Member
 
Registered: May 2003
Location: Victoria, CANADA!
Distribution: OpenBSD, Slackware, Debian
Posts: 85

Original Poster
Rep: Reputation: 15
More info:

The debootstrap command being ran (--foreign has no effect enabled or disabled):

"debootstrap --foreign \
--include locales \
--components main,security lenny /mnt/fl http://myrepo.com/debian"

- /mnt/fl = an LVM formatted with ext3.
- "security" component is a mirror of debian's security repo which is done using the same procedure and is a component of our repo. Removing this has no effect to the issue.

The error deboostrap crashes on over and over:

"I: Unpacking iptables...
I: Unpacking iproute...
I: Unpacking ifupdown...
I: Unpacking wget...
I: Unpacking libtasn1-3...
W: Failure while installing base packages. This will be re-attempted up to five times."

If I chroot into the dir after deboostrap fails, the following wants to be installed:

"s1d:~# chroot /mnt/fl
s1d:/# apt-get -f install
Reading package lists... Done
Building dependency tree... Done
Correcting dependencies... Done
The following extra packages will be installed:
binutils cpp-4.1 g++-4.1 gcc-4.1 gcc-4.1-base libc6 libc6-dev libstdc++6-4.1-dev linux-libc-dev
Suggested packages:
binutils-doc gcc-4.1-locales g++-4.1-multilib gcc-4.1-doc gcc-4.1-multilib libmudflap0-dev glibc-doc manpages-dev
libstdc++6-4.1-doc
Recommended packages:
gcc c-compiler
The following NEW packages will be installed:
binutils cpp-4.1 g++-4.1 gcc-4.1 gcc-4.1-base libc6-dev libstdc++6-4.1-dev linux-libc-dev
The following packages will be upgraded:
libc6
1 upgraded, 8 newly installed, 0 to remove and 2 not upgraded.
75 not fully installed or removed.
Need to get 18.1MB of archives.
After this operation, 45.6MB of additional disk space will be used.
Do you want to continue [Y/n]?
"

Cheers,

Tim

Last edited by lil_drummaboy; 06-21-2010 at 06:44 PM.
 
Old 06-21-2010, 07:04 PM   #3
lil_drummaboy
Member
 
Registered: May 2003
Location: Victoria, CANADA!
Distribution: OpenBSD, Slackware, Debian
Posts: 85

Original Poster
Rep: Reputation: 15
I'm starting to think my issue might be related to "override files". ftp.us.debian.org uses override files in the "/indices" directory, which I do not. This overrides the package definition of packages when enabled at apt-ftparchive "Packages" file generation time. It seems ftp.us.debian.org's "Packages.gz" contains these overrides while mine does not. In apt-ftparchive I also have "OverrideDir" commented out for this repo.

Generating new repo data now using debian's override files and "OverrideDir" enabled with apt-ftparchive. That will take a while. I can provide a resolution (to my own issue, lol) if that works.

By the way: Override files in general are very sparingly documented for anyone running into similar issues.

Cheers,

Tim

Last edited by lil_drummaboy; 06-21-2010 at 07:11 PM.
 
Old 06-22-2010, 06:12 PM   #4
lil_drummaboy
Member
 
Registered: May 2003
Location: Victoria, CANADA!
Distribution: OpenBSD, Slackware, Debian
Posts: 85

Original Poster
Rep: Reputation: 15
It was in fact "override files" that was the issue. Once I pulled down the override files from "http://ftp.us.debian.org/debian/indices/override.*.gz" to my repo and set apt-ftparchive up to recognize them, everything was resolved. The override file made my Packages file generate with several packages that were previously "important" set to "optional". After that, I removed the --exclude line from my debootstrap and everything worked flawlessly.

I only needed to get the override files for lenny/main, but others could be used. Here is the "Tree" section of my apt-ftparchive.conf, those two override lines at the bottom are what I added to my config:

"Tree "dists/lenny" {

// This is a space separated list of sections which appear under
// the distribution, typically this is something like main
// contrib non-free
Sections "main security";

// This is a space separated list of all the architectures that
// appear under search section. The special architecture 'source'
// is used to indicate that this tree has a source archive.
Architectures "amd64";

// Overrides:
BinOverride "/data/repo/debian/indices/override.lenny.$(SECTION)";
ExtraOverride "/data/repo/debian/indices/override.lenny.extra.$(SECTION)";

}
"

Solved this one myself! Cheers!

Tim
 
  


Reply

Tags
file


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: When you install Debian without a mirror, you need to edit /etc/apt/sources.lis LXer Syndicated Linux News 0 10-09-2009 08:00 PM
apt-mirror doesn't create appropriate mirror ??? abd_bela Debian 1 09-30-2009 04:23 PM
LXer: Burning Debian packages and repositories to disc with APTonCD and apt-mirror LXer Syndicated Linux News 0 07-03-2007 06:01 PM
LXer: netselect-apt - tip to find the fastest Debian mirror LXer Syndicated Linux News 0 05-23-2007 09:01 AM
LXer: How To Create A Local Debian/Ubuntu Mirror With apt-mirror LXer Syndicated Linux News 0 01-04-2007 05:33 PM


All times are GMT -5. The time now is 01:09 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration