Have you ever thought about using rsync?
You can keep your packages up to date with it.
It will only grab the files that have changed since your last rysnc.
Also in most mirrors, there is a Packages file under the main, contrib and non-contrib dirs
under the /pub/debian/dists/<version>/<arch>/
If you really wanted to gather that info from it you could get those three files, cat them into
one and then just grep out the word "Package" which should give all of the official package names
for the whole os.
those are my thoughts.