Some more Rsync questions
 

ok so i am a total noob at linux
i managed to copy a file using rsync from one of my linux boxes
to another one over the internet...i just have a few more questions and would appreciate it if you could just give me a push in the right direction...

this was a simple command from my terminal...but i realise if i want to backup a server with many directories i will need to create a bash script right? i need the rsync script to run at a certain time, i will use cron to do this...this i understand. what i dont understand is this...

how can i get the script to login with the remote linux box automatically...obviously i dont want to have to type in a password every time the backup runs.

are bash scripts simply text files? do they have to be saved with a .sh extension...where do they get placed? does it matter? does the cron just find the script and execute it?

lastly i dont think i have ssh configured properly between the machines as it gives an error about keys or something...whats the most secure way to implement ssh?

thanks for all your reading

rsync is, imho, best run over an ssh tunnel. Here the automatic login is achieved via the use of a preshared ssh key (utterly unrelated to rsync here)... plenty of guides showing this setup - https://www.digitalocean.com/communi...rsync-over-ssh

a script can be called anything and placed anywhere. the extension doesn't matter. any executed file will be presumed to be a shell script, but placing a shebang:
as the very first line will make it explicitly use that interpreter instead of the default sh (so, python, perl, bash, ruby etc...) You'd just usually set it as an executable file After that, you can just run "/where/ever/that/file/is/script.sh" in a prompt and it will run.

the cron job finds it by the fact that you would usually provide the absolute path of it. Whilst you can modify the environment that cron uses to look for files, just specifying the full path is much clearer an simpler.

Thanks for that info...i havent been this excited about computing for a long time now...really enjoying linux and doing stuff like this. I was first messing about with centos and now have debian installed too. Are the distros really very different to use/work with in a work environment? Am i better off sticking to a specific distro? Thanks again for the help...

check out the links in my sig for howto create the ssh keys you will need, there is also some remote backup scripts out there that use rsync that are easy enough to manipulate.

Concerning distribution differences, while the base system is usually very similar there can be very important differences between different distributions, not the least of which are program versions available in the package management repository (if it has one), binary vs compile at install time package management, different default paths, installation methodologies, etc. I recommend figuring out which distribution meets your needs at work and stick with it while learning others on non-production or non-mission-critical applications.

Just to give a few arbitrary examples of major differences between distributions:

Gentoo's Portage package management system is incredibly expansive, compiles everything (and I mean everything) you install on the system using it from source by default (though I think there may be a way to force it to use binary packages too), and uses a "use flags" system which allows you to customize everything down to the smallest degree, even if your configurations conflict with each other, so long as you write the flags properly. This allows it to use bleeding edge versions of software specifically configured for your system (A gentoo installation is closer to a custom personal distribution than a universal structure), but means you're waiting until tomorrow to see that X, KDE environment you just entered the install command for and are going to be climbing that learning curve and doing lots of reading, potentially getting lost a little on the way.

Debian, on the other hand, uses apt which does binary distribution by default (e.g. no compilation, fast, but not as custom built to your system and potentially slower and an older version due to availability in the repository) While derivatives use this system, they tend to vastly differ in default software available. Additionally, Debian doesn't necessarily do things the same way other distros do, meaning different default paths for certain things. However it's an easier to learn package management system compared to Gentoo and the binary packages tend to give instant gratification when you need to install something which is one reason why so many distros designed for people used to windows are based on it (ubuntu, mint, etc.) as it allows them to retain a lot of the expectations they have from Windows (binary packages, not a lot of loading time for installations, etc.)

Slackware has no automatic global package manager to handle dependency management for you. You learn about the software and install the prerequisites yourself (after compiling in many cases, but not necessarily) But you'll learn quite a lot, know exactly what is on the system (because you put it there personally) and it'll run until the hardware breaks once it's set up properly. It also tends to be very traditional in how things are done and upgrades aren't as obvious without the package management other distros rely on.

But they all utilize the same general folder structure for the base system and can be classified as GNU/Linux operating systems.

thanks for that information i guess they really arent kidding when they say open source offers choice.
i noticed you didnt mention red hat at all...isnt it true that most corporate systems are red hat/red hat based?

what i really ment to ask is, despite the differences you mentioned, if i learned say slackware backwards, would i pretty much be able to administer a wide range of linux servers etc in the real world.

again thanks for the reply

The primary reason I didn't mention RedHat is because I don't have as much experience with it compared to some of the others. My employer utilizes an almost exclusively Microsoft environment so I'm due to go back and play with it a bit. The last time I messed with it was back before I was doing things at work using Linux and I didn't like it as much for my personal machines as some others at that time.

That being said, even with the differences involved between distributions, yes, the majority of skills you learn in administrating a linux system do carry over between them. In spite of the plethora of choice out there, the core of the system is still GNU/Linux and while there are a couple which don't stick to traditional paths or file placement, they still keep things in the same place as everyone else for the most part and it's still a Linux kernel in there and the GNU software collection providing the basic tools.

thanks for that info paladin,

lieb...there is a ssh guide you have posted at the beginning it says "Unless otherwise specified all of these tasks are performed on server A."

which is server A??? the server that is sending data...or receiving data from a remote host...sorry if it seems a silly question

sn

"Server A" is the client, but yeah... doesn't actually say that does it??!

Linux Distros
 

Nearly every Linux distro is a debian fork. Some notable exceptions are Red Hat/Fedora, Gentoo/Sabayan, Slackware and SuSE/opensuse. Linux Mint, Arch Linux, Knoppix, Mepis, Ubuntu, Kubuntu, etc. are all debian forks. If debian didn't exist, none of those would either.

If you want to learn server operation, you probably want to do away with the X server, because server-based machines don't require an X server. They're not designed to be fun or anything like that.

And, computer operators follow corporate policies on automation and documentation of it. Every so often, especially when a new system is installed, the company (any company) will hunt down a premiere operator to write all kinds of automation scripts.

The new guy is thinking, 'Two more months of this and I can just hit one key every five minutes and everything will work by itself. Won't that be great!'

The company is thinking, 'As soon as this guy finishes the difficult stuff, we can make a severance agreement with him and hire grunts to push one key every five minutes.'

So, you just can't win. But programmers tend to get abused less, because they're more dangerous. They can destroy an entire corporation. They're not ideologically motivated to do that, but you don't want to push them too far, because it has happened!

And, it isn't worth learning server administration for your home-based network. It's like becoming an auto mechanic to fix your own vehicle.

That's really quite an overstatement.

Reality says far far far too many enterprise linux servers run X constantly, and is engrained in their operation. In my experience it's more "Changing stuff is expensive! All the project management and cross charging required would invalidate any long term savings. Let's just keep doing things as they are."

While the discussion about differences between distributions and how prolific different distributions are can go on for a while, one of my favorite sources for this stuff has become the Linux Documentation Project.

http://www.tldp.org

There is a ton of information here which spans from beginner stuff through to the really arcane. One of the sections I'd recommend to get a good solid base in Linux which will be usable with pretty much any distribution would be the Introduction to Linux Guide. Just go to the Guides section and scroll down a little and you'll find it.

I have a pdf version of it on my kindle that I'm reading through to re-learn a lot of basics and I think it would be a good thing for a beginner who really wants to get a good understanding of the basics.

ok thanks to everyone who posted...i have learned a lot over the last few weeks...i just have a last question regarding rsync...here goes

if i have server A and Server B located 20km's away...and im using Rsync to backup +- 200GB of data from A to B (only backing up changes) , should I also have a 2TB hdd connected to server B that takes a local full backup everyday??? cause surely if its only backing up the changes and a file becomes corrupt (ie it cant be opened) the changes will copy over to my only copy of the data? whats the best practice?

thanks again

Some more Rsync questions
 

as far as good disaster recovery, you should have historical backups of some kind to protect against this. whether you perform a local backup at the site of server a or do weekly full backups which transfer to server b and are synced to a long term storage location to be retained for some reasonable amount of time or some other option you think it's better for you is partially dependant on what is available as a storage location.

I would recommend the second option. additionally, if you do weekly full backups it could also make your incremental backups smaller as they would be the difference from a more recent full backup, hence fewer differences.

wow, thank you for pointing that out. Ill fix that ASAP. Server A is the local server, not the remote. server B is typically remote. I apologize and thank you for pointing out my oversight.

It has been adjusted. Again please forgive my oversight. I try to help where and when I can.