read third string from second line of each file

dani1234 · 03-07-2019, 06:53 AM

Hi,
I have 100K files and want to read third and 4th string of each file and stored in a single file .

see sample below file content i want in the output file 0002 20190228

Code:

VUP gfppgdht
GHT HT 00002 20190228 20190227 100122 N 0
GHT CVD JYT 20190228 359 001 1

next file is so the outputfile only 00010 20190228 should come

Code:

VGT HT_TXTGHP
HTR HT 00010 20190228 20190227 100122 Y 
HTR BHY JYT 20190228 359 001 1

Like this for all 100K files output in a single file ..

AWK or SED can do?

hydrurga · 03-07-2019, 06:56 AM

Just a note that for both these examples, according to your code, the highlighted strings are the 5th and 6th strings in their respective files, not the 3rd and 4th.

joe_2000 · 03-07-2019, 07:02 AM

Many ways would achieve this, but...

when you iterate over all files,

Code:

head -n 2

will give you the top two lines of a file.

Code:

tail -n 1

can be used to obtain the bottom line of the two.
Use cut on the result to get the third and fourth string.

@hydrurga: According to the title this is about the second line of each file.

hydrurga · 03-07-2019, 07:09 AM

Quote:

Originally Posted by joe_2000

@hydrurga: According to the title this is about the second line of each file.

Ah, I didn't see the title (which appears not to give the complete story, anyway

). I was just going by the contents of the opening post. My bad. Thanks Joe.

dani1234 · 03-07-2019, 07:29 AM

I just pasted the header of the file , so how we can do by using AWK or SED and combining loop with 100K.

hydrurga · 03-07-2019, 07:32 AM

Quote:

Originally Posted by dani1234

I just pasted the header of the file , so how we can do by using AWK or SED and combining loop with 100K.

You may get someone who gives you the code straight out, but usually how it works on here is that you are asked to have a go at it yourself and then are helped to get it right. So, what have you tried so far? Can you simplify the problem, for example, by printing out every second line of each file to stdout? If you get that right then you can build on it.

joe_2000 · 03-07-2019, 07:33 AM

Quote:

Originally Posted by dani1234

I just pasted the header of the file , so how we can do by using AWK or SED and combining loop with 100K.

Why do you insist on using awk or sed?
Did you look at my previous post?
What have you tried for yourself?

pan64 · 03-07-2019, 08:08 AM

you can use awk easily, you only need to check the line number and print 3rd and 4th values of all files passed (no sed required). But obviously you can do it in sed too.

rtmistler · 03-07-2019, 08:15 AM

@dani1234,

Given your span of time on the LQ site, I feel you should understand how LQ works and how best to pose a technical question. Asking for others to provide you with code is not the best way to proceed. Please review the site guidelines about how to ask questions and offer some additional background for your question, such as illustrating what you have tried already to solve your problem.

teckk · 03-07-2019, 03:59 PM

Code:

var="VUP gfppgdht
GHT HT 00002 20190228 20190227 100122 N 0
GHT CVD JYT 20190228 359 001 1"

awk 'FNR==2{print $3, $4}' <<< "$var"

sed -sn 2p <<< "$var" | cut -d " " -f3,4

head -n2 <<<"$var" | tail -n1 | cut -d " " -f3,4

Code:

#array of file names
list=(file1 file2 file3)

#loop through them and do something
for i in "${list[@]}"; do
    awk 'FNR==2{print $3, $4}' "$i"
done

pan64 · 03-08-2019, 12:59 AM

Quote:

Originally Posted by teckk

Code:

sed -sn 2p <<< "$var" | cut -d " " -f3,4

head -n2 <<<"$var" | tail -n1 | cut -d " " -f3,4

would be better to solve with a single awk or sed. No need to use pipes.

Quote:

Originally Posted by teckk

Code:

#array of file names
list=(file1 file2 file3)

#loop through them and do something
for i in "${list[@]}"; do
    awk 'FNR==2{print $3, $4}' "$i"
done

actually the for loop is superfluous here:

Code:

awk 'script' filelist

should do the same. If we have no "command line too long" error.

joe_2000 · 03-08-2019, 01:56 AM

Building on pan64's excellent comments:

If you are going to use a "for" loop, better do it this way:

Code:

for filename in ./*; do
    # do stuff such as
    echo $filename
done

The advantage being that it is simpler and doesn't break when filenames contain whitespaces.
If you don't have all files in one directory, use find.

allend · 03-08-2019, 06:48 AM

If you use a bash shell and have files more than one subdirectory deep, you can set the globstar shell option. When wrapped in parentheses, the commands are run in a subshell, so that the parent shell stays the same when the subshell exits.

Code:

(shopt -s globstar; for f in **/*; do awk 'FNR==2{print $3,$4;exit}' "$f"; done)

I prefer this construction as it will handle filenames containing spaces, as noted above.
I have added an exit statement to the awk for efficiency, which might be measurable for 100K of files.