Hi,
I've been googling a lot to try and find some piece of software to allow mirroring a directory on an HTTP or FTP server without having to keep it up to date manually or by cron jobs and keeping it from downloading files that are never used on the mirror.

The basic idea is that it acts a sort of proxy between a local system and a software repository, for example the Ubuntu apt sources, which are over 200 GB large. The software would, upon a regular request, begin downloading the file and save it locally while also redirecting it to the requesting system. If another system then requests the same file, the software would then be able to deliver it directly from its local cache.

I know this sounds very similar to a normal caching proxy, but they usually only store small files and it would require a different setup than just setting the mirror site to the local server, which would allow a central setup of the best mirror.

I really hope I've explained this well enough, as I haven't been able to find anything through Google, which may be due to me not using the right terms for the problem.

So my question is: has anyone heard of any application like this?

Thanks,
- Marcus

If you are using a RH based system you can use the createrepo package, set yum priorities for the local repo to be first, and set yum so it downloads to your local repo directory.

Well, I was looking for a general solution, so that I would be able to implement it in different scenarios.
But I do have one scenario, which is the most important one:

An Arch Linux server is supposed to run a mirror for a bunch of Ubuntu hosts that are wasting a lot of bandwidth on downloading the same packages over and over again. But maintaining a full Ubuntu mirror with its continually expanding repositories of more than 200 GB also seems like wasting bandwidth, as we only use a tiny fraction of those packages.

- Marcus