LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > CentOS
User Name
Password
CentOS This forum is for the discussion of CentOS Linux. Note: This forum does not have any official participation.

Notices


Reply
  Search this Thread
Old 05-21-2020, 10:57 AM   #1
BeepCluster8765
LQ Newbie
 
Registered: May 2020
Posts: 5

Rep: Reputation: Disabled
Red face Trying to stand up a local mirror for my company, issues with rsync


I am roughly basing what I am trying to do around this article: https://blog.programster.org/centos-...entos-7-mirror

tldr:
Step 1. rsync with a CentOs mirror and a EPEL mirror
Step 2. stand up company internal CentOs mirror with nginx
Step 3. Profit

I have been going through and trying to pick mirrors near me to rsync with, but it seems all of them at one point or another hang on a file and die. I checked with the networking team, the corporate firewall isn't known to be blocking my rsync, and I am able to get a lot of files before it chokes, so that's not it. Sometimes switching mirrors fixes the problem, sometimes it doesn't.

Here is an example of the command I am running to sync EPEL:

sudo rsync -avSHPz --delete rsync://mirror.team-cymru.com/epel/7/x86_64 ./CentOSEpelMirror/7/x86_64

which yields the following error:

x86_64/Packages/o/openarena-0.8.8-7.el7.noarch.rpm
335,753,632 83% 26.23MB/s 0:00:02
rsync: connection unexpectedly closed (3910004 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [receiver=3.1.2]
rsync: connection unexpectedly closed (522056 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.2]


The only thing I can think of is that it's an issue with WSL, since that is how I am rsyncing, but I was wondering if anyone had any additional input for me. I was about to write a python script to parse the mirror list and get me all the rsync urls, and then just write a bash script that loops over all of them whenever rsync chokes and dies.
 
Old 05-21-2020, 01:23 PM   #2
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 4,839
Blog Entries: 3

Rep: Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399
The syntax looks wrong. Try it with a dry run first before deleting anything. Mind the slashes, too.

Code:
rsync -avSHPz --dry-run user@mirror.team-cymru.com:/epel/7/x86_64/ ./CentOSEpelMirror/7/x86_64/
That will update from the remote machine mirror.team-cymru.com to the local directory ./CentOSEpelMirror/7/x86_64/ Does that destination directory exist in the relative path?

Substitute "user" for the actual remote account. I see that SSH keys appear to not be used there. They would be a good idea. However, keep in mind that if things were set up properly that remote root access was disabled.

By the way, you write that you are using CentOS. What does the Linux Subsystem for Windows (WSL) have to do with any of that?
 
Old 05-21-2020, 02:02 PM   #3
BeepCluster8765
LQ Newbie
 
Registered: May 2020
Posts: 5

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
The syntax looks wrong. Try it with a dry run first before deleting anything. Mind the slashes, too.

Code:
rsync -avSHPz --dry-run user@mirror.team-cymru.com:/epel/7/x86_64/ ./CentOSEpelMirror/7/x86_64/
That will update from the remote machine mirror.team-cymru.com to the local directory ./CentOSEpelMirror/7/x86_64/ Does that destination directory exist in the relative path?

Substitute "user" for the actual remote account. I see that SSH keys appear to not be used there. They would be a good idea. However, keep in mind that if things were set up properly that remote root access was disabled.

By the way, you write that you are using CentOS. What does the Linux Subsystem for Windows (WSL) have to do with any of that?
So the rsync mirrors for CentOS that I am contacting do not use ssh. There is an rsync daemon on port 873 of the remote server I am contacting via tcp. I also tried the with and without slashes, but no dice. The destination directory exists on that path, yes. I am rsyncing from WSL, but I am building a CentOS mirror, since my work computer runs good ol Winblows. I will try the dry run though and see what happens.
 
Old 05-21-2020, 02:09 PM   #4
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 4,839
Blog Entries: 3

Rep: Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399
WSL is a weird distro and as far as I know is using paravirtualization, to add to the strangeness. So a lot of things might not work, which I expect is a part of their plan.

Back to rsync, the following should work if WSL is not too broken:

Code:
test -e ./CentOSEpelMirror/7/x86_64/ || mkdir -p ./CentOSEpelMirror/7/x86_64/
rsync -avSHPz --delete rsync://mirror.team-cymru.com/epel/7/x86_64/ ./CentOSEpelMirror/7/x86_64/
Keep in mind that the . stands in for the current directory. You may wish to use absolute paths instead.

If that does not work, then try on a normal distro like CentOS, Debian, Devuan, etc on bare metal.
 
Old 05-21-2020, 03:06 PM   #5
michaelk
Moderator
 
Registered: Aug 2002
Posts: 20,076

Rep: Reputation: 3374Reputation: 3374Reputation: 3374Reputation: 3374Reputation: 3374Reputation: 3374Reputation: 3374Reputation: 3374Reputation: 3374Reputation: 3374Reputation: 3374
While it seems you can run a webserver in WSL I think it would be a lot easier to create a local repository with a real distribution using virtualization software like VirtualBox.

I have not played with WSL but without knowing where or how it uses . as suggested use an absolute path.

Or install a web server i.e. IIS and create a local repository on Windows itself.

Last edited by michaelk; 05-21-2020 at 03:14 PM.
 
Old 05-21-2020, 04:51 PM   #6
BeepCluster8765
LQ Newbie
 
Registered: May 2020
Posts: 5

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by michaelk View Post
While it seems you can run a webserver in WSL I think it would be a lot easier to create a local repository with a real distribution using virtualization software like VirtualBox.

I have not played with WSL but without knowing where or how it uses . as suggested use an absolute path.

Or install a web server i.e. IIS and create a local repository on Windows itself.
relative paths have expected behavior in WSL. I am going to copy over my mirror to an air-gapped vm once I rsync the packages, and that will host the mirror for the machines on that network, but as it sits I can't use a vm to rsync the remote mirror down for security reasons. Maybe I am misunderstanding, but IDK how IIS is going to fix my problem with copying a remote mirror to my virtual machine.
 
Old 05-21-2020, 04:55 PM   #7
BeepCluster8765
LQ Newbie
 
Registered: May 2020
Posts: 5

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
WSL is a weird distro and as far as I know is using paravirtualization, to add to the strangeness. So a lot of things might not work, which I expect is a part of their plan.

Back to rsync, the following should work if WSL is not too broken:

Code:
test -e ./CentOSEpelMirror/7/x86_64/ || mkdir -p ./CentOSEpelMirror/7/x86_64/
rsync -avSHPz --delete rsync://mirror.team-cymru.com/epel/7/x86_64/ ./CentOSEpelMirror/7/x86_64/
Keep in mind that the . stands in for the current directory. You may wish to use absolute paths instead.

If that does not work, then try on a normal distro like CentOS, Debian, Devuan, etc on bare metal.
Getting back to you from earlier, the dry-run actually helped a lot with the stability, I am still having some issues but they aren't nearly as bad as before. I also swapped over to rsyncing to a destination in my linux homedir which is supposedly better for some reason. It is still having some of that weird behavior though. I will look into using a real linux distro, but due to security policy around here I doubtful that I will be able to do so.
 
Old 05-21-2020, 09:40 PM   #8
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 4,839
Blog Entries: 3

Rep: Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399Reputation: 2399
Ok. It seems then that you're stuck in a shop with resellers in place of support. Just keep in mind that WSL is not intended to give an honest demonstration of what a real distro could do. It appears to work barely enough to give Windows users a bad experience and have them turn back to the familiar, eschewing the modern for another work generation. Thus the abysmally low adoption rate for WSL. So, you might be able to get it to Rsync, you might not. Keep us posted.

A mirror of a repository is not important data, in that it it can be easily replaced if lost and it is cryptographically signed so does not need to be locked away. So if you have a hardware budget even a simple Raspberry Pi with a USB hard drive would do. Or an old desktop computer with a new or newish hard drive would do just as well.

However, I'm guessing that this repository is to be mirrored on an existing web server?
 
Old 05-22-2020, 10:25 AM   #9
BeepCluster8765
LQ Newbie
 
Registered: May 2020
Posts: 5

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
Ok. It seems then that you're stuck in a shop with resellers in place of support. Just keep in mind that WSL is not intended to give an honest demonstration of what a real distro could do. It appears to work barely enough to give Windows users a bad experience and have them turn back to the familiar, eschewing the modern for another work generation. Thus the abysmally low adoption rate for WSL. So, you might be able to get it to Rsync, you might not. Keep us posted.

A mirror of a repository is not important data, in that it it can be easily replaced if lost and it is cryptographically signed so does not need to be locked away. So if you have a hardware budget even a simple Raspberry Pi with a USB hard drive would do. Or an old desktop computer with a new or newish hard drive would do just as well.

However, I'm guessing that this repository is to be mirrored on an existing web server?
So I went home and did an experiment in my homelab, WSL is 10/10 the issue. Maybe WSL 2 will be better since it contains the whole linux kernel. Who knows?

So at least now I have the evidence to show my manager and security that the issue is definitely with winblows, and I am sure they will be able to stand up a vm somewhere to do this. Their issue is not with the security of the mirror itself, but more of if you let your users do whatever they want on your corporate network, then if someone malicious comes in it is much easier for them to attack your network.

Anyways thanks for the help you all.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Red Hat - How An Open Source Software Company Became 34,000 Million Dollars Company LXer Syndicated Linux News 0 11-06-2018 04:11 PM
.kvm roms (centos) - rsync'd local mirror slight issue. kevinyeandel Linux - Software 1 11-04-2013 04:08 PM
Can't open company website from outside only when inside the company lan perfectpol7 Linux - Server 6 01-16-2012 09:42 AM
apt-mirror doesn't create appropriate mirror ??? abd_bela Debian 1 09-30-2009 04:23 PM
LXer: How To Create A Local Debian/Ubuntu Mirror With apt-mirror LXer Syndicated Linux News 0 01-04-2007 05:33 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > CentOS

All times are GMT -5. The time now is 03:58 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration