Hello
I have a directory structure like the following, each subfolder contains about 10 .txt files

filings/000104/1042391
filings/000104/1042392
filings/000105/1052391
filings/000105/1052222
.......

How do I recursively grep through all files in the filings directory and gzip them, while preserving directory structure in the resultant gz file

1.I want to find all files that contain two strings on the same line, 'FORM TYPE' and '99D' right now the following line does the job for me

grep 'FORM TYPE' *.txt |grep '99D'

2.I want to tar and bzip2 all of the above files with maximal compression , but I don't know how to incorporate the piped sequential-grep command here, as the command in paragraph 1. Right now I am using the following:

tar cvf - ./11a.txt | bzip2 -v9 - > ~/myname.tar.bz2

3.I want to preserve directory structure once the gz file is extracted, so that filings/000105/1052222 directory is created once the file is extracted

You can write a script to do this. Personally, I'd approach it in exactly the steps you laid out:

1. Since you have a top level to start with, you can run a find on it for any .txt files. That gives you the path and file name, so you can...
2. ..feed it into the grep statements, and if it's found....
3. ..put the file name (the variable you found going in) into an array, to feed into...
4. ..the tar command.

That said, read the man page on tar...you don't need to pipe it to bzip (see the -j option). What have you written so far?

1 members found this post helpful.

Welcome to LQ!

You can do that all in one line with only small variation from what you have shown.

First, no need for two greps if 'FORM TYPE' and '99D' are on the same line.

Next, you can use grep to only produce the matching file names, see man grep, -l option.

Same to make it recursive, man grep, -r.

Then you can pipe that directly into the tar command, man tar, -T reading from stdin. You can also bzip that in the tar comman and redirect to your file.

So something like this should do it...

2 members found this post helpful.

Thanks for the reply. But I need two greps because they could be spaces or other characters between 'FORM TYPE' and '99D',
how do I handle it with one grep?

Mind posting the complete answer please, including how to bzip2 with maximal compression? Thanks.

Sir I know the bare minimum of bash script. Mind give me the answer please?
I am not competent enough to write this on my own.

Not true - grep accepts regex; basic will suffice if the 'FORM TYPE' is always first on the line. Like soUsing the hints given above you should be able to resolve this yourself.

If you are concerned that your two search terms can occur in either order in a line, then you can search for both with

Let's see if you can figure it yourself from the hints given first! LQ members are always happy to help you learn, but you learn best by doing.

And in a way, we have given you the complete answer already, all you need to do is think about what has been posted, maybe read the referenced man pages for more complete details (but the answer is already given).

Grep simply matches regular expressions in the the files, so all you need is a regular expression to match the text you are looking for.

Regular expressions are just a kind of shorthand notation for matching arbitrary strings of characters, and you are already using it to match each expression. All you need is to modify your regular expression to match both expressions. Others have already given the basic hint, '.*', which matches zero or more of any character.

There are many regular expression tutorials online, I found this one with a quick search, and it includes the ability to test your knowledge by submitting your own regular expressions and testing the result.

The man page for tar will provide all the information needed to tweak that part of the command pipeline to you exact requirements.

Give it a try and tell us what you figure out!

My attempt is following, I removed *.txt since I want to search all subdirectories and there is no .txt file in current directory, not sure I need to do tar -cv instead of tar -cvf, is missing 'f' a problem?

grep -rl ".*FORM TYPE.*99D" | tar -T - -cv | bzip2 -v9 - > ~/99D.tar.bz2 >~/log.txt

I got an ambiguous output error, it actually worked without the final part on log.txt. How could I keep the stdout in this situation with a log.txt logfile?

you added two redirection of stdout, which cannot work. What do you want to put into log.txt?
probably you need to redirect stderr to log.txt.

1 members found this post helpful.

That is progress, nice to see!

If you want to include the bzip without the final pipe, see the -j option to tar as mentioned by TB0ne above. But the way you have it is OK too, and gives you all the bzip2 options if you want them.

If you want the file list to go to log.txt you can use tee with the log filename (see man tee as usual). The basic form would be grep ...| tee ... | tar ...

On the other hand, if you want error messages redirect stderr into log.txt (hint: 2>log.txt).

If you want the list of files and all status and errors, you should be able to use tee as described above and append stderr to log.txt at the end (or send to a separate file, errs.txt). I haven't tested tee+append but you can do so easily, right!

And you never will be until you actually do it. Read the "Question Guidelines" link in my posting signature. We are all happy to help you if you're stuck, but we aren't going to write your scripts/do your work for you, and hand it to you.

There are THOUSANDS of easily found scripting guides you can find with a very brief internet search...exactly like the one you did to find this site. Start there.

1 members found this post helpful.

well then, thanks for nothing, I got it on my own.

Please try to be respectful of the time and effort provided by others such as TB0ne to help you learn to resolve your issues. LQ is all about learning from others!

If you feel entitled to free handouts and are scornful of those trying to help in other ways, then you may be in the wrong place. From the link in TB0ne's sig, the LQ Welcome page:

Please consider the spirit of that statement as the basis of LQ participation, and try to be respectful of those offering help, if not thankful for the guidance received.

We look forward to your continuing participation in that same spirit, and hope that you will also learn along the way so that you may also help others in your turn!

My apologies. Must have mis-read TB0ne's comment as one that is extremely condescending. Appreciations to all those who actually had substantive input.