Linux - GeneralThis Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Distribution: SuSE Linux Open/Enterprise, Red Hat, Ubuntu
Posts: 147
Rep:
run checksum on stdout?
This is probably a dumb question, but I can't seem to figure out how to run an md5sum on a tar file that is generated via stdin. Is this not possible? I always end up with the same checksum value no matter what files I set to have tarred.
Distribution: SuSE Linux Open/Enterprise, Red Hat, Ubuntu
Posts: 147
Original Poster
Rep:
Both of you have a better idea than I had. I'm basically trying to transfer millions of small files from one server to another using tar to store those files. I thought it would be easier to stream the tar file via stdout, and untar on the fly, but doing a checksum on that isn't exactly straight forward. I think I will end up creating the tar files on the source and destination, run the checksum and then delete both files afterwards. It just means the process will take longer.
Distribution: SuSE Linux Open/Enterprise, Red Hat, Ubuntu
Posts: 147
Original Poster
Rep:
rsync and probably any other utility will have to "walk" each individual file. That kills the transfer speed a lot. If I transfer a few large tar files it will increase the transfer speed. I would've used rsync if there weren't 40+ million files.
Another possibility which I have used for a few applications: http://freecode.com/projects/checkcrc
You can pipe anything through it and it will remain unaltered, but a CRC will be outputted to stderr.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.