[SOLVED] Trying to stand up a local mirror for my company, issues with rsync

I am roughly basing what I am trying to do around this article: https://blog.programster.org/centos-...entos-7-mirror

tldr:
Step 1. rsync with a CentOs mirror and a EPEL mirror
Step 2. stand up company internal CentOs mirror with nginx
Step 3. Profit

I have been going through and trying to pick mirrors near me to rsync with, but it seems all of them at one point or another hang on a file and die. I checked with the networking team, the corporate firewall isn't known to be blocking my rsync, and I am able to get a lot of files before it chokes, so that's not it. Sometimes switching mirrors fixes the problem, sometimes it doesn't.

Here is an example of the command I am running to sync EPEL:

sudo rsync -avSHPz --delete rsync://mirror.team-cymru.com/epel/7/x86_64 ./CentOSEpelMirror/7/x86_64

which yields the following error:

x86_64/Packages/o/openarena-0.8.8-7.el7.noarch.rpm
335,753,632 83% 26.23MB/s 0:00:02
rsync: connection unexpectedly closed (3910004 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [receiver=3.1.2]
rsync: connection unexpectedly closed (522056 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.2]


The only thing I can think of is that it's an issue with WSL, since that is how I am rsyncing, but I was wondering if anyone had any additional input for me. I was about to write a python script to parse the mirror list and get me all the rsync urls, and then just write a bash script that loops over all of them whenever rsync chokes and dies.

The syntax looks wrong. Try it with a dry run first before deleting anything. Mind the slashes, too.

rsync -avSHPz --dry-run user@mirror.team-cymru.com:/epel/7/x86_64/ ./CentOSEpelMirror/7/x86_64/

That will update from the remote machine mirror.team-cymru.com to the local directory ./CentOSEpelMirror/7/x86_64/ Does that destination directory exist in the relative path?

Substitute "user" for the actual remote account. I see that SSH keys appear to not be used there. They would be a good idea. However, keep in mind that if things were set up properly that remote root access was disabled.

By the way, you write that you are using CentOS. What does the Linux Subsystem for Windows (WSL) have to do with any of that?

Originally Posted by Turbocapitalist The syntax looks wrong. Try it with a dry run first before deleting anything. Mind the slashes, too. rsync -avSHPz --dry-run user@mirror.team-cymru.com:/epel/7/x86_64/ ./CentOSEpelMirror/7/x86_64/ That will update from the remote machine mirror.team-cymru.com to the local directory ./CentOSEpelMirror/7/x86_64/ Does that destination directory exist in the relative path? Substitute "user" for the actual remote account. I see that SSH keys appear to not be used there. They would be a good idea. However, keep in mind that if things were set up properly that remote root access was disabled. By the way, you write that you are using CentOS. What does the Linux Subsystem for Windows (WSL) have to do with any of that?

So the rsync mirrors for CentOS that I am contacting do not use ssh. There is an rsync daemon on port 873 of the remote server I am contacting via tcp. I also tried the with and without slashes, but no dice. The destination directory exists on that path, yes. I am rsyncing from WSL, but I am building a CentOS mirror, since my work computer runs good ol Winblows. I will try the dry run though and see what happens.

WSL is a weird distro and as far as I know is using paravirtualization, to add to the strangeness. So a lot of things might not work, which I expect is a part of their plan.

Back to rsync, the following should work if WSL is not too broken:

test -e ./CentOSEpelMirror/7/x86_64/ || mkdir -p ./CentOSEpelMirror/7/x86_64/
rsync -avSHPz --delete rsync://mirror.team-cymru.com/epel/7/x86_64/ ./CentOSEpelMirror/7/x86_64/

Keep in mind that the . stands in for the current directory. You may wish to use absolute paths instead.

If that does not work, then try on a normal distro like CentOS, Debian, Devuan, etc on bare metal.

While it seems you can run a webserver in WSL I think it would be a lot easier to create a local repository with a real distribution using virtualization software like VirtualBox.

I have not played with WSL but without knowing where or how it uses . as suggested use an absolute path.

Or install a web server i.e. IIS and create a local repository on Windows itself.

Originally Posted by michaelk While it seems you can run a webserver in WSL I think it would be a lot easier to create a local repository with a real distribution using virtualization software like VirtualBox. I have not played with WSL but without knowing where or how it uses . as suggested use an absolute path. Or install a web server i.e. IIS and create a local repository on Windows itself.

relative paths have expected behavior in WSL. I am going to copy over my mirror to an air-gapped vm once I rsync the packages, and that will host the mirror for the machines on that network, but as it sits I can't use a vm to rsync the remote mirror down for security reasons. Maybe I am misunderstanding, but IDK how IIS is going to fix my problem with copying a remote mirror to my virtual machine.

Originally Posted by Turbocapitalist WSL is a weird distro and as far as I know is using paravirtualization, to add to the strangeness. So a lot of things might not work, which I expect is a part of their plan. Back to rsync, the following should work if WSL is not too broken: test -e ./CentOSEpelMirror/7/x86_64/ || mkdir -p ./CentOSEpelMirror/7/x86_64/ rsync -avSHPz --delete rsync://mirror.team-cymru.com/epel/7/x86_64/ ./CentOSEpelMirror/7/x86_64/ Keep in mind that the . stands in for the current directory. You may wish to use absolute paths instead. If that does not work, then try on a normal distro like CentOS, Debian, Devuan, etc on bare metal.

Getting back to you from earlier, the dry-run actually helped a lot with the stability, I am still having some issues but they aren't nearly as bad as before. I also swapped over to rsyncing to a destination in my linux homedir which is supposedly better for some reason. It is still having some of that weird behavior though. I will look into using a real linux distro, but due to security policy around here I doubtful that I will be able to do so.

Ok. It seems then that you're stuck in a shop with resellers in place of support. Just keep in mind that WSL is not intended to give an honest demonstration of what a real distro could do. It appears to work barely enough to give Windows users a bad experience and have them turn back to the familiar, eschewing the modern for another work generation. Thus the abysmally low adoption rate for WSL. So, you might be able to get it to Rsync, you might not. Keep us posted.

A mirror of a repository is not important data, in that it it can be easily replaced if lost and it is cryptographically signed so does not need to be locked away. So if you have a hardware budget even a simple Raspberry Pi with a USB hard drive would do. Or an old desktop computer with a new or newish hard drive would do just as well.

However, I'm guessing that this repository is to be mirrored on an existing web server?

Originally Posted by Turbocapitalist Ok. It seems then that you're stuck in a shop with resellers in place of support. Just keep in mind that WSL is not intended to give an honest demonstration of what a real distro could do. It appears to work barely enough to give Windows users a bad experience and have them turn back to the familiar, eschewing the modern for another work generation. Thus the abysmally low adoption rate for WSL. So, you might be able to get it to Rsync, you might not. Keep us posted. A mirror of a repository is not important data, in that it it can be easily replaced if lost and it is cryptographically signed so does not need to be locked away. So if you have a hardware budget even a simple Raspberry Pi with a USB hard drive would do. Or an old desktop computer with a new or newish hard drive would do just as well. However, I'm guessing that this repository is to be mirrored on an existing web server?

So I went home and did an experiment in my homelab, WSL is 10/10 the issue. Maybe WSL 2 will be better since it contains the whole linux kernel. Who knows?

So at least now I have the evidence to show my manager and security that the issue is definitely with winblows, and I am sure they will be able to stand up a vm somewhere to do this. Their issue is not with the security of the mirror itself, but more of if you let your users do whatever they want on your corporate network, then if someone malicious comes in it is much easier for them to attack your network.

Anyways thanks for the help you all.