[SOLVED] Combine several commands which write their output to different files

georgi · 10-11-2013, 08:56 AM

Hi all,

I have large files with url-s ending on "|<number>" which is the Page Rank, for example

Code:

http://www.machinokairo.com/2012/05/post-39.html|2

I am using "grep" to sort out all url-s in a particular way: first, remove all ending on "|0" and write the out put to a file, then remove all ending on "|1" and write the output to a new file and so on up to "|5". Each time I remove certain PR and have the rest in separate file. For now I use the following commands to do that

Code:

grep --invert-match "|0" sitelist >> sitelist_PR1.txt
grep --invert-match "|1" sitelist_PR1.txt >> sitelist_PR2.txt
.
.
grep --invert-match "|5" sitelist_PR1.txt >> sitelist_PR6.txt

I will greatly appreciate if someone points how to automate this process with CLI or bash script.

sundialsvcs · 10-11-2013, 09:00 AM

Just ">>" them to the same filename. ">>" means append, whereas ">" means replace.

Or, if for other reasons you want them to remain separate, run the commands then "cat file1 file2 file3 > outputfile"

georgi · 10-11-2013, 09:05 AM

Quote:

Originally Posted by sundialsvcs

Just ">>" them to the same filename. ">>" means append, whereas ">" means replace.

Or, if for other reasons you want them to remain separate, run the commands then "cat file1 file2 file3 > outputfile"

Yes, I would like to keep each of the files as separate and to make the whole process with one run of a script or command line.

TenTenths · 10-11-2013, 09:13 AM

Code:

#!/bin/bash

for PR in {0..9} ; do
 grep "|${PR}$" /path/to/input/sitelist > /path/to/output/sitelist_PR${PR}.txt
done

Will loop through the numbers 0 to 9

Then tries to grep for the occurance of a bar and that number at the end of each line in the input file and puts them in the output file accordingly.

georgi · 10-11-2013, 01:05 PM

Quote:

Originally Posted by TenTenths

Code:

for PR in {0..9} ; do

Will loop through the numbers 0 to 9

Then tries to grep for the occurance of a bar and that number at the end of each line in the input file and puts them in the output file accordingly.

Ten, your script might be the one I need, but when I first looked at it I noticed that you wrote PR in the text inside. I have "|<number>" in my text files, like in the example

Code:

|2

at the end. The number is actual PR of that site. So, I need to deal with "|".

TenTenths · 10-11-2013, 01:48 PM

in my example PR is a variable in the "for" loop it is refered to in the main line as ${PR} so first time round the grep command will look for "|0$" where $ tells grep to match it at the end of the line. Next time through the loop it becomes "|1$" etc.

georgi · 10-11-2013, 01:56 PM

the script creates only empty files

TenTenths · 10-12-2013, 07:40 AM

worked fine for me when I created an input file in the firnat you gave.

georgi · 10-12-2013, 07:55 AM

Quote:

Originally Posted by TenTenths

worked fine for me when I created an input file in the firnat you gave.

still have empty files although numbered correctly.
anyway, thanks for your efforts and time.

ntubski · 10-12-2013, 10:26 AM

TenTenth's grep expression is bit stricter than the original. I can think of a couple of cases where this could cause failure:

Does "sitefile" use DOS/Windows line endings (post output of file sitefile)?

Can the page rank be more than one digit?

georgi · 10-12-2013, 01:21 PM

Quote:

Originally Posted by ntubski

TenTenth's grep expression is bit stricter than the original. I can think of a couple of cases where this could cause failure:

Does "sitefile" use DOS/Windows line endings (post output of file sitefile)?

Can the page rank be more than one digit?

sitefile is converted from windows and your point is excellent. Now I formatted the proper way and the script works.
The point is that script gives me output in a different way than desired i.e. each output file contains only url-s with PR from its name
sitelist_PR1.txt contains only url-s with PR1.

ntubski · 10-12-2013, 01:52 PM

Untested:

Code:

awk -F\| '{for (i=1; i <= 5; i++) { if ($2 >= i) print > ("sitelist_PR" i ".txt") }}' sitelist

georgi · 10-12-2013, 01:58 PM

Quote:

Originally Posted by ntubski

Untested:

Code:

awk -F\| '{for (i=1; i <= 5; i++) { if ($2 >= i) print > ("sitelist_PR" i ".txt") }' sitelist

Code:

awk: cmd. line:1: {for (i=1; i <= 5; i++) { if ($2 >= i) print > ("sitelist_PR" i ".txt") }
awk: cmd. line:1:                                                                          ^ unexpected newline or end of string

ntubski · 10-12-2013, 03:41 PM

Ah, missed the closing brace (fixed in edit).

georgi · 10-12-2013, 04:46 PM

Quote:

Originally Posted by ntubski

Untested:

Code:

awk -F\| '{for (i=1; i <= 5; i++) { if ($2 >= i) print > ("sitelist_PR" i ".txt") }}' sitelist

this did the job. thank you.