LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-15-2019, 01:31 AM   #16
rnturn
Senior Member
 
Registered: Jan 2003
Location: Illinois (SW Chicago 'burbs)
Distribution: openSUSE, Raspbian, Slackware. Previous: MacOS, Red Hat, Coherent, Consensys SVR4.2, Tru64, Solaris
Posts: 2,802

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550

Quote:
Originally Posted by Turbocapitalist View Post
Use HTTP or HTTPS not FTP. HTTP/HTTPS are more efficient. And one does not "just upload" 1.5 TB. It might be worth trying again with the shipment, even if only to get an official negative answer on record.
I'm wondering what kind of data is the OP talking about? If the company isn't careful, they could run afoul of laws that are in place to protect certain kinds of data---for example, anything the would be covered by HIPAA. I would ask that workplace naysayer to reconsider not allowing the physical transfer via FedEx or the like (the modern equivalent of a station wagon full of 9-track tapes). The company's internal auditor and data security team may have some things to say about the preferred way to transmit data between the company and outside entities. The places where I've worked, when data had to be sent to or from "business partners" it had to be across a VPN or, at the very least, encrypted before being sent across the 'net.
 
3 members found this post helpful.
Old 12-16-2019, 10:05 PM   #17
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,359

Rep: Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751
There may be security considerations, as above.

Sticking to the original qn, as it is in 7 dirs, I'd probably just produce one tar.gz file for each of the 7 dirs and then rsync them one at a time.
You can play with the compression settings - many tools eg gzip, bzip, xz have a parameter of value 1-9 which means higher the num, higher the compression, but more processing time reqd (to compress at least)

Reasons:
1 split into 7 logical separate sets is easier for knowing what data is where (for sender & receiver people).
Eg you could start processing the data (if reqd) as each set arrives, without waiting (if that makes sense)

2 rsync will cope (recover) if the cxn drops out, which, even with 7 smaller sets, may happen, depending on what the sender & receiver have as cxns.
 
Old 12-17-2019, 04:14 AM   #18
ivnit
LQ Newbie
 
Registered: Nov 2019
Location: Italy
Distribution: Fedora, Ubuntu
Posts: 8

Rep: Reputation: Disabled
Maybe not useful, but why not using zstd to compress a file? It is VERY fast! Many many times more than xz or gzip.
Then you can split the file, doing something like:
Code:
zstd -c <your big file> | split -b200M
By this command you compress the file and create compressed pieces of it (in this case 200MB each).
When done, you find files named xaa, xab, xac, etc.
You can send the files, so in case of error, you must transfer again only 200MB.

At the other site, the command could be:
Code:
cat x?? >> "original file name"
and then

Code:
unzstd "original file name"
Of course, everything said about the network speed is valid. But, zstd is VERY fast so you don't have to wait for hours, when compressing. The same for uncompressing.
You could check how big the compressed file is and, based on your network speed, you can understand how long it takes.

Using public repositories (like git, gdrive, etc.) could be anyway against the company police. Be careful!

PS zstd can be installed in Fedora by dnf. For other distro it should be available. Maybe, someone here can help about that.
 
Old 12-17-2019, 04:18 AM   #19
Hermani
Member
 
Registered: Apr 2018
Location: Delden, NL
Distribution: Ubuntu
Posts: 261
Blog Entries: 3

Rep: Reputation: 113Reputation: 113
I feel OP should consider taking this up with his management again. Getting 1.5 TB over the internet in this way is possible but prone to errors. I've just been migrating 4 TB of data between servers on the same LAN and even then it is not a one-off affair. Rsync was my friend.

You might consider putting the data in an encrypted Veracrypt container for safety and then ship it to its destination on a drive (preferably a SSD) after talking to your management and explain that online transfer of 1.5 TB of data is just not feasable - you can't put an ocean liner on a truck.

Last edited by Hermani; 12-17-2019 at 04:19 AM.
 
Old 12-17-2019, 04:32 AM   #20
Shadow_7
Senior Member
 
Registered: Feb 2003
Distribution: debian
Posts: 4,137
Blog Entries: 1

Rep: Reputation: 874Reputation: 874Reputation: 874Reputation: 874Reputation: 874Reputation: 874Reputation: 874
What part of the data are they "browsing"?

That amount of data over a network is a bad day. Recalling my dialup days of 10MB-15MB an HOUR. Sure I could do 10GB of downloads a month, with a persistent 24/7 connection. Even on my almost broadband connection now, it still takes about 4 hours to get a 4GB ISO for a distro. Might it just be easier to make that data available over a shell on a dedicated machine that only has ssh and that data, since they're "browsing". Otherwise physical media via snail mail will probably trump most "network" options. At least for MOST places in the USA.

Corporate entities would probably prefer that they come in person and sit down at a terminal so said data, that's probably sensitive, never leaves the campus. Sell them that first with transport and accommodations, and the disk drive plus postage becomes an easier sell. Or just abandoning the project entirely and taking a new less time consuming route.

Depending on the contents of that data, there are compression options better than .zip. Like rar or bz2. But high compression takes a while on modest hardware. And you need enough storage for the source data and the destination file(s). Plus the ability of the receiver of the data to uncompress it. Easy if they're well versed linux users. Perhaps a squashfs option so they can mount the compressed data and browse without having to uncompress it directly. Depends on how tech savvy (and *nix) they are.

Last edited by Shadow_7; 12-17-2019 at 04:33 AM.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Can rsync work with 10 terabyte data? ufmale Linux - Software 3 06-22-2008 06:09 PM
copy terabyte of data from one machine to another ufmale Linux - Newbie 3 05-16-2008 12:17 AM
LXer: Kernel space: How to use a terabyte of RAM LXer Syndicated Linux News 0 03-20-2008 01:10 AM
problem installing red hat 5.1 on a 3 terabyte HP proliant server MightyU Red Hat 2 01-10-2008 10:29 AM
Suggestions for SuSe 10.2 LVM partioning for 4 drive terabyte server faron_young Linux - Newbie 3 05-08-2007 06:44 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 04:53 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration