Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Introduction to Linux - A Hands on Guide
This guide was created as an overview of the Linux Operating System, geared toward new users as an exploration tour and getting started guide, with exercises at the end of each chapter.
For more advanced trainees it can be a desktop reference, and a collection of the base knowledge needed to proceed with system and network administration. This book contains many real life examples derived from the author's experience as a Linux system and network administrator, trainer and consultant. They hope these examples will help you to get a better understanding of the Linux system and that you feel encouraged to try out things on your own.
Click Here to receive this Complete Guide absolutely free.
I was wondering if some one can help me out with my problem.
I have a C program which outputs a text file test.out which contains hexadecimal addresses. The file size is a little more than 100GB. The problem is I don't have enough space to store this file on my hard drive. The space I have on my drive is only 72GB. Can anyone tell me how to compress this file on the fly so that I can store this on my hard disk?
Again, after I get this file test.out in the compressed format like in .gz or .bz2 I want to use this as an input to a shell script..for example ./simulator < test.out.gz. I will be really grateful if someone helps me out with this as well.
(the '-c' flag is just so that gzip will use stdin/stdout).
Thanks Neo and Chris for your promptness, but the problem is my C program says the output will be test.out (this is stated in the C code) and when I pipe the way you suggested I get both the file, the test.out and the testing.out.gz. The test.out shows all the required stuff but the testing.out.gz just says "done..." and shows no text (the hexadecimal address I am looking for).
Thank you very much. You are just great and a wonderful person. I thought I would end-up with no solution for my question but you did it all! Kudos to you and you deserve it.
If you have time please let me know what this mkfifo does. I ran a test program and everything works just fine. I am ready to run few large programs and hopefully they'll give me similar results.
I saw something wierd happening though. When I type the command mkfifo test.out and run I don't see that the memory is consumed but I do see that the output of my C program, if it's test.out is greyed out and then when I use the head command to look at the content of test.out (the grayed) it doesn't show anything but the zipped file does have everything.
I use df -h to know the amount of memory being consumed.
The mkfifo creates an object that can be used as a data pipe from a source to a sink. Although it looks like a file, it exists on the file system as a name only, and does not use up any other space. Any data put into it by one process goes into a memory buffer, and is consumed by the process that you set up at the other end.
Originally Posted by dimpu
when I use the head command to look at the content of test.out it doesn't show anything but the zipped file does have everything.
In the example, the gzip has already grabbed the data out of the pipe, so head does not see anything inside test.out (remember that test.out is now a pipe object, not a file).
Neo, I want to know how many minutes/ hours should a benchmark (you know, they are huge programs)take to run using your technique. I remember it use to take few hours to produce the text file but now it just takes few seconds. Do you think it is working fine? Also I want to know what command should I use to find the size of a single file?
I am running several benchmarks but I first started with "Cactus". I've written a program and that took almost 5 hours to run.
The other program which is the original program took just a few seconds (with a zipped output) but if I run the same program without using gzip feature it takes an hour or two or a little more therefore I am a little skeptical. Also this output is an input to my memory simulator.
The memory simulator with a zipped output shows a million instruction was fetched but with the test.out output which is a direct output file it shows an instruction fetch of 100 million in just a few minutes.
It's quite possible that you are right. There is a lot of repetition I believe.
In my second question I wanted to know in terms of size (GB/ MB) of a single file; like, suppose I have several files in my directory and I want to find the size of the output file test.out then what command should I use?
There's no way to predict the size of the compressed file; it depends on the content and compression algorithm and compression factor. Both gzip and bzip2 have a compression flag with values 1-9. Better compression = slower to create. Start with some small files and see how the trend goes.
see the man pages.