LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 04-05-2013, 08:26 AM   #1
Ulysses_
Senior Member
 
Registered: Jul 2009
Posts: 1,303

Rep: Reputation: 57
Cloud storage possibilities in linux


In NFS the data is only stored in the server's hard drive. Not so with dropbox, where a duplicate of the data is stored in your local drive. If there is no local drive, dropbox is not an option, right? At least not with files larger than available ram.

Is there any other cloud storage solution for linux that does not keep a duplicate of the data locally?

Any workarounds to make dropbox work this way, so it can handle files larger than the available ram when there is no hard drive?
 
Old 04-06-2013, 06:22 PM   #2
MS3FGX
LQ Guru
 
Registered: Jan 2004
Location: NJ, USA
Distribution: Slackware, Debian
Posts: 5,852

Rep: Reputation: 361Reputation: 361Reputation: 361Reputation: 361
Perhaps I'm not understanding your question, but "cloud storage" is nothing more than an online account where you can store files. It's FTP with a pretty interface. You don't need to keep a local copy of the file to use a cloud storage service like Dropbox, and if you did, that would completely nullify the point of the service.

So perhaps we should start with clarifying what it is you are actually trying to do here.
 
Old 04-06-2013, 06:40 PM   #3
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 11,225

Rep: Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320
You want a cloud storage provider that gives you a network drive to mount? An actual NFS/Samba/you-get-the-idea network drive, and not one that "syncs"? Er...

Google tells me that owncloud.org supports mounting via webdav, and its homepage lists several providers implementing it: http://owncloud.org/providers/ No, I've never used any of them.

Quote:
Any workarounds to make dropbox work this way, so it can handle files larger than the available ram when there is no hard drive?
Why are you having this strange requirement?

Last edited by dugan; 04-06-2013 at 07:49 PM.
 
Old 04-07-2013, 01:48 PM   #4
Ulysses_
Senior Member
 
Registered: Jul 2009
Posts: 1,303

Original Poster
Rep: Reputation: 57
You find it strange that a pc can work without a hard drive? A live CD attempts that. Or that a live CD pc can modify remote files larger than its ram size? NFS is for this purpose.

MS3FGX, dropbox's executable keeps a copy of the files locally and synchronizes with the server copy with something like rsync. There must be a reason, probably immediate availability of your data on all your hardware (smartphones, windows pc's, tablets etc), and bandwidth efficiency as only changes are transfered.

I'm looking for online storage that appears like a local folder ie mounted as a local folder, but is only stored in the server, just like NFS.

And it must be available in the cloud, ie from unlimited bandwidth servers with unlimited storage where you get charged more automatically as your data grows and as your transfers grow, like amazon's S3 charges.

Any cloud service that you can mount with sshfs-fuse perhaps?

Last edited by Ulysses_; 04-07-2013 at 01:52 PM.
 
Old 04-07-2013, 02:01 PM   #5
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 11,225

Rep: Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320
First, I've already pointed you towards some storage providers that fit your exact requirements. I am not impressed that you've completely ignored them.

Second, as you've already been asked twice, what exactly is the use case here? You're booting computers from livecds, either assuming they have no hard drives or intentionally not mounting the hard drives that they have, and using them to work remotely on files that are larger than the RAM size of the workstations? First, LOL and second, why? How large are the files and how much RAM do the workstations have? Why don't the workstations have hard drives? Why are you working from livecds? Do you have any idea how ridiculous your "requirement" sounds when you omit information like this? Information, btw, which you've been asked for multiple times? If you decide that your why is none of our business, then our answer will be "well, then we can't help you."

Third, have you actually confirmed that editing a file larger than the client computer's RAM size is feasible? If that's a requirement, I would expect to be able to SSH/RDesktop/etc remote access the server, which had better have enough RAM, and then work on the file from there.

Fourth, I'd like to know why just getting a rack server and adding hard drives to it as needed (and then, when it comes to it, getting a second rack server and making your own cloud), which is what everyone else would do, isn't the best solution here. If your answer is "I need unlimited storage and bandwidth" then it translates to "I don't know what I need."

Last edited by dugan; 04-08-2013 at 11:59 AM.
 
Old 04-08-2013, 12:26 PM   #6
Ulysses_
Senior Member
 
Registered: Jul 2009
Posts: 1,303

Original Poster
Rep: Reputation: 57
Since you're putting so much attention to the use case I am morally obliged to detail it for you. Hope you find it interesting.

Live CDs have security applications. You can boot a severely infected computer without worrying about malware logging what you type to access your remote data. Also when you power it off there is no trace of your activity left. So subsequent boots of the infected o/s cannot steal anything from you. This is useful when using public computers, where previous users are wreckless with the sites they visit and have no qualms about executing trojans etc. But also when using a computer you share with family members, perhaps you do not trust it to be free from infection, so when you want to access the bank or pay online you boot the live CD, just to be on the safe side.

US military personnel are encouraged to boot a live CD on public computers if they have to use such computers, where a hardened live CD is prepared for this purpose, here it is:

http://www.spi.dod.mil/lipose.htm

Myself I do not trust the US military so I am making my own hardened live CD. So far I am plugging usb drives when I need access to private data, which of course nullifies one of the advantages of the live CD as malware can now infect the usb drive and stay there, so usb drives are only plugged strictly when very trusted sites are visited, and only if access to the data is absolutely essential.

Instead of a usb drive, why not hold data in a server somewhere, and be just as strict in only visiting very trusted sites when you mount the server directory. If AES256 or whatever is used properly, the data cannot be eavesdropped. However, the plaintext is available to the owner of the server, and we do not want that.

To avoid this privacy issue, people have suggested that public storage is used in conjunction with truecrypt. So you store the truecrypt container in the server and only decrypt on your end. Here's how it is done with dropbox:

http://tonyonsecurity.com/2012/08/05...box-truecrypt/

When a change is made in one of your user files, a small part of the truecrypt container file is modified by truecrypt. The dropbox application takes care of synchronizing its copy of the container with the server copy efficiently, transmitting just the changes.

You cannot use dropbox if you do not have a local drive to store the duplicate, or a ram drive large enough to store the duplicate. The container is twice as large as the maximum amount of data you store in it, so it can easily get too large for the live CD created ram drive.

If the server offers NFS access, you can just mount the folder where the container resides on the server and use truecrypt on the remote container directly, no need for a local duplicate. NFS takes care of only transmitting modified sectors or other chunks depending on the filing system underneath.

But NFS is very insecure and no wonder it is not offered by server space providers online, so I am told.

Someone recommended sshfs-fuse instead, where if you have a shell account on a server with ssh support, then your space on the server can be mounted as a local directory, and your truecrypt container is accessible directly from the truecrypt application, just like with NFS.

However, this requires a shell account on a specific server, which doesn't sound like cloud to me. Cloud computing is about transparent scalability where you do not manually migrate to a bigger server when storage resources or bandwidth resources are not enough for you (risking compatibility and security problems due to the new hardware and software), but migration is done automatically for you. So there is zero cost and effort migrating. It's as if you have unlimited space and bandwidth. Your credit card is charged automatically depending on resource usage.

Quote:
I'd like to know why just getting a rack server and adding hard drives to it as needed (and then, when it comes to it, getting a second rack server and making your own cloud), which is what everyone else would do, isn't the best solution here.
Because you then get concerned with hardware and administration which is not the task for the user of online storage space, or if others do it for you, you only gain storage space but no extra bandwidth or cpu power, and sooner or later you have to migrate. But in the cloud migration is automatic, not your concern.

Quote:
If your answer is "I need unlimited storage and bandwidth" then it translates to "I don't know what I need."
I know what I need now, my amount of data is known. But I do not know what I will need in the future.

Quote:
How large are the files and how much RAM do the workstations have?
About 1 TB at present if this is important. From what I said above it shouldn't be, and not all data has to go to the server. Neither should RAM matter, as it is not the bottleneck, and RAM available is unknown as the live CD is for use on public computers too.

Quote:
have you actually confirmed that editing a file larger than the client computer's RAM size is feasible?
It's not about editing as in using an editor, but modifying small parts of a very large file, as done by truecrypt.

Quote:
Originally Posted by dugan View Post
First, I've already pointed you towards some storage providers that fit your exact requirements. I am not impressed that you've completely ignored them.
I was put off by webdav being mentioned because I did not know how to mount it and it has a terrible reputation for security:

http://www.networkworld.com/news/201...11-webdav.html

Then I went to a couple of these providers' sites and they didn't look much like cloud to me, cloud was mentioned by them more like a marketing ploy, to say they have big scalable hardware or something. Are you aware of amazon S3? That is true cloud computing. If someone offers cloud storage like this for a low price or for free, I'd like to know.

If free providers for this exist, maybe I'll make my own version of truecrypt that works redundantly with several containers over several cloud providers, so any crappy free providers do not hurt too much. Maybe software raid over mounted directories already exists?

Last edited by Ulysses_; 04-08-2013 at 01:34 PM.
 
Old 04-08-2013, 12:42 PM   #7
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 11,225

Rep: Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320
The service you're asking about does not exist. Period. If you want something similar, you'll have to set up your own private cloud and create it for yourself. You can also probably also make money selling access to other people. Which mount protocol did you want, btw, since you've already ruled out NFS, fuse-sshfs, webdav, and any of the syncing ones?

If I were in your position, though, I would go for the ssh-fuse + shell account on a server option.

Quote:
Are you aware of amazon S3?
Post #3 originally recommended an S3 account mounted as a network drive with SS3FS. I removed that recommendation when I found out that that SS3FS syncs just like Dropbox does.

Quote:
Hope you find it interesting.
I did. Thanks.

Last edited by dugan; 04-08-2013 at 01:47 PM.
 
Old 04-08-2013, 01:49 PM   #8
Ulysses_
Senior Member
 
Registered: Jul 2009
Posts: 1,303

Original Poster
Rep: Reputation: 57
Quote:
Originally Posted by dugan View Post
Which mount protocol did you want, btw, since you've already ruled out NFS, fuse-sshfs, webdav, and any of the syncing ones?
Can't fuse-sshfs be used on a dynamically allocated server?
 
Old 04-08-2013, 02:07 PM   #9
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 11,225

Rep: Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320Reputation: 5320
Can't help you there. I have never set up a cloud.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Is there any cloud storage that propely handles symlikns? froff Linux - Software 4 03-14-2013 04:16 PM
Open source Cloud storage package? Nick_C Linux - Networking 4 02-13-2013 06:46 PM
LXer: Red Hat details next Linux and storage platforms for cloud, big data era LXer Syndicated Linux News 0 06-29-2012 05:40 PM
LXer: How to setup Wuala secure cloud storage in Ubuntu 11.10 and Linux Mint 12 LXer Syndicated Linux News 0 12-09-2011 08:30 PM
cloud storage via ftp recneps_divad Linux - Networking 7 12-08-2011 12:01 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 06:24 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration