LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 04-30-2019, 12:12 PM   #1
Beren329
LQ Newbie
 
Registered: Oct 2014
Posts: 26

Rep: Reputation: Disabled
"Personal Cloud" via ssh, rsync?


Hello All!

Not a newbie to Linux, but a newbie to more advanced networking here.

Now that Dropbox is limiting the number of devices with which one can use their service, I'd really like to move to an easy way of simply hosting my own files, well, at home.

I've looked into some more complex options, such as an NAS or Owncloud server, but those seem fairly involved for a relatively simple personal-use solution.

I'm already quite familiar with accessing my home "server" machine remotely via ssh and transferring files via rsync, so I keep thinking: shouldn't there be a relatively easy way to create a personal "cloud" simply by automating rsync to keep certain folders synced across machines, even my laptop when I'm out working? It seems like there ought to be an easy way to do this--or at least easier than manually using r-sync every time I switch machines.

Any advice or ideas?
 
Old 04-30-2019, 02:16 PM   #2
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,310
Blog Entries: 3

Rep: Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722
How much data are you talking about, both the total to be stored and the amount changed daily?

For a generic answer, I'd just go with a home-built NAS and sshfs and/or rsync. For the NAS I'd also make sure that the underlying file system would be OpenZFS using RAID-Z1 or RAID-Z2. However because of OpenZFS' problematic licensing you have to add that extra in most cases.

There are other options like Syncthing and InterplanetaryFileSystem but I have no direct experience.

The Coda file system was almost an option a long time ago. I read about someone considering rejuvenating it but don't know where it stands now.
 
1 members found this post helpful.
Old 04-30-2019, 02:28 PM   #3
Beren329
LQ Newbie
 
Registered: Oct 2014
Posts: 26

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
How much data are you talking about, both the total to be stored and the amount changed daily?

For a generic answer, I'd just go with a home-built NAS and sshfs and/or rsync. For the NAS I'd also make sure that the underlying file system would be OpenZFS using RAID-Z1 or RAID-Z2. However because of OpenZFS' problematic licensing you have to add that extra in most cases.

There are other options like Syncthing and InterplanetaryFileSystem but I have no direct experience.

The Coda file system was almost an option a long time ago. I read about someone considering rejuvenating it but don't know where it stands now.

Thanks for the reply!

As to your question, I'm not talking about tons of data. I have a lot stored, but the amount I actually want constantly updated across devices isn't that huge: certainly less than 5gb; once the folders in question are initially synced, probably less than 1gb on a daily basis.

Which is why I'm thinking there's gotta be an easier way than going through the rigamarole of setting up an NAS. I already have a linux box that I use for storage, and have port 22 forwarded so that I can ssh in or use rsync via my public ip address. Very often I just use rsync to grab an individual file remotely from that "storage" machine (which also runs a Minecraft server). I'm just thinking there must be a simpler way to, for example, start a script on startup that compares and updates a directory. But I'm new enough at this that I'm unsure of the best method.

Thanks!
 
Old 04-30-2019, 02:29 PM   #4
Beren329
LQ Newbie
 
Registered: Oct 2014
Posts: 26

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
How much data are you talking about, both the total to be stored and the amount changed daily?

For a generic answer, I'd just go with a home-built NAS and sshfs and/or rsync. For the NAS I'd also make sure that the underlying file system would be OpenZFS using RAID-Z1 or RAID-Z2. However because of OpenZFS' problematic licensing you have to add that extra in most cases.

There are other options like Syncthing and InterplanetaryFileSystem but I have no direct experience.

The Coda file system was almost an option a long time ago. I read about someone considering rejuvenating it but don't know where it stands now.

Thanks for the reply!

As to your question, I'm not talking about tons of data. I have a lot stored, but the amount I actually want constantly updated across devices isn't that huge: certainly less than 5gb; once the folders in question are initially synced, probably less than 1gb on a daily basis.

Which is why I'm thinking there's gotta be an easier way than going through the rigamarole of setting up an NAS. I already have a linux box that I use for storage, and have port 22 forwarded so that I can ssh in or use rsync via my public ip address. Very often I just use rsync to grab an individual file remotely from that "storage" machine (which also runs a Minecraft server). I'm just thinking there must be a simpler way to, for example, start a script on startup that compares and updates a directory. I'm guessing that I'm thinking more along the lines of a very simple, personal, peer-to-peer setup rather than some kind of server-client setup. But I'm new enough at this that I'm unsure of the best method.

Thanks!
 
Old 04-30-2019, 02:34 PM   #5
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,310
Blog Entries: 3

Rep: Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722Reputation: 3722
Unison was another option but I'm not sure it was developed any more.

Your existing setup might be ok. Though for the central machine, I'd still suggest considering a move to OpenZFS RAID-Z1 or -Z2 for the file system. OpenZFS has checksumming so that when the drive goes bad, the error is detected and, hopefully, corrected.

Rysnc can be automated easily enough with SSH keys but be sure to make incremental backups from your central machine often. Adding the --update option to the --archive option will be quite helpful in reducing the risk of overwriting the wrong file.
 
Old 04-30-2019, 07:58 PM   #6
frankbell
LQ Guru
 
Registered: Jan 2006
Location: Virginia, USA
Distribution: Slackware, Ubuntu MATE, Mageia, and whatever VMs I happen to be playing with
Posts: 19,326
Blog Entries: 28

Rep: Reputation: 6142Reputation: 6142Reputation: 6142Reputation: 6142Reputation: 6142Reputation: 6142Reputation: 6142Reputation: 6142Reputation: 6142Reputation: 6142Reputation: 6142
Don't make this more complicated than it needs to be. A server is, at the heart of it, a computer that stores and serves data for other computers in a network.

What you need is a computer in your network with adequate hard drive space, network access, and file-sharing. It could even be a Raspberry Pi with a beefy external HDD. You could indeed use ssh, scp, rsync, and samba shares or NFS.

A web search for "raspberry pi as file server" will turn up a number of articles which I think you will find helpful, even if you end up using something other than a Pi for this.
 
1 members found this post helpful.
Old 05-01-2019, 02:10 PM   #7
Beren329
LQ Newbie
 
Registered: Oct 2014
Posts: 26

Original Poster
Rep: Reputation: Disabled
Thanks for the good advice folks!
 
  


Reply

Tags
rsync, rsync+ssh, server



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Cloud 5: Dispelling cloud myths, the cloud security excuse, and the fight for cloud supremacy LXer Syndicated Linux News 0 06-20-2014 04:12 PM
LXer: Cloud 5: NSA not killing cloud, cloud IT jobs, rise of cloud brokers LXer Syndicated Linux News 0 03-02-2014 09:51 AM
LXer: Cloud 5: Netflix's cloud-connected brain, 5 cloud myths and from cloud to fog LXer Syndicated Linux News 0 02-21-2014 02:20 PM
LXer: It's a cloud, cloud, cloud, cloud world LXer Syndicated Linux News 0 07-23-2013 05:40 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 04:08 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration