grep | xargs -I echo $(foo; bar; echo $(fee; fi; fo; fum)) == questionable output..

GrapefruiTgirl · 12-07-2010, 02:25 PM

http://www.linuxquestions.org/questi...5/#post4183237

The above post is the inspiration for this experiment. Read it if you like, but I'm going to try to make that not necessary, by explaining it like it's a brand new problem.

Also, I KNOW this code is terrible, and is not the way to do this. I simply want to understand why it doesn't work - other methods of doing the job are not needed. Learning experience only! So..

The input file (filename = "tags") contains the below:

Code:

root@reactor: cat tags
This is about soccer #soccer_generic #soccer_intro . More information is here in more text.
This is line 2 #another_hash_tag #hastag_2 . And here is even more text.
Here's another line to increase count of tag: #hastag_2
root@reactor:

I expect my output to resemble the following:

Code:

#soccer_generic (1)
#soccer_intro (1)
#another_hash_tag (1)
#hastag_2 (2)
#hastag_2 (2)

Yes, I am aware that there's a duplicate line in the output - for this exercise it doesn't matter.
What I actually get for output, is this:

Code:

#soccer_generic (0)
#soccer_intro (0)
#another_hash_tag (0)
#hastag_2 (0)
#hastag_2 (0)

So notice the counts are not happening. Here's the code:

Code:

grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; echo "($(grep -c "'$var'" tags))");

Now, a demonstration about why I am confused about the output. I'm going to add an echo statement to the code, so we can see the command that's supposedly being run inside the subshell; observe the output:

Code:

grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; echo "($(echo grep -c "'$var'" tags))");
#soccer_generic (grep -c '#soccer_generic' tags)
#soccer_intro (grep -c '#soccer_intro' tags)
#another_hash_tag (grep -c '#another_hash_tag' tags)
#hastag_2 (grep -c '#hastag_2' tags)
#hastag_2 (grep -c '#hastag_2' tags)

So it looks to me like the command should be working. I can run one of those greps:

Code:

root# grep -c '#another_hash_tag' tags
1

So why doesn't the "1" appear when the whole command is run? Things I have tried include assorted combinations of :

grep | xargs -n 1 ...
grep | xargs -L 1 ...
grep | xargs -n 1 -L 1 ...
grep | xargs -d "\n" ...
grep -Z | xargs -0 ...
echo $(grep) | xargs ...
echo "$(grep)" | xargs ...
xargs ... <<<$(grep)
xargs ... <<<"$(grep)"

Every combination of those above, I've mixed and tried, and probably more. Been to the xargs manpage repeatedly, and grep's too. If I'm missing something obvious, that's cool

- just tell me where it is!

Also tried grep -E, -e but didn't fiddle much with grep since the output appears such that grep is working fine as is.

Also tried without that subshell in there at the end, like so (and took out the brackets desired in the output, for code clarity):

Code:

root@reactor: grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; grep -c "'$var'" tags;)
#soccer_generic 0
#soccer_intro 0
#another_hash_tag 0
#hastag_2 0
#hastag_2 0
root@reactor: grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; echo grep -c "'$var'" tags;)
#soccer_generic grep -c '#soccer_generic' tags
#soccer_intro grep -c '#soccer_intro' tags
#another_hash_tag grep -c '#another_hash_tag' tags
#hastag_2 grep -c '#hastag_2' tags
#hastag_2 grep -c '#hastag_2' tags
root@reactor:

So still no go - seems the subshell itself is not the problem...

I've tried setting various shellopts, like dotglob/extglob/globstar, but I admit to not really having much of a grip on these anyways yet, as I rarely use them. I don't think they have anything to do with this, but you may correct me.

Tried removing the single quotes inside grep -c "'$var'" to no avail. Tried no quotes, double quotes, and both together on var='{}' to no avail.

Tried sticking `eval` into various places, with and without the subshell near the end. Made no difference:

Code:

root@reactor: grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; eval grep -c "'$var'" tags;)
#soccer_generic 0
#soccer_intro 0
#another_hash_tag 0
#hastag_2 0
#hastag_2 0
root@reactor: grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; eval echo grep -c "'$var'" tags;)
#soccer_generic grep -c #soccer_generic tags
#soccer_intro grep -c #soccer_intro tags
#another_hash_tag grep -c #another_hash_tag tags
#hastag_2 grep -c #hastag_2 tags
#hastag_2 grep -c #hastag_2 tags
root@reactor:

So, what do you folks think? Again - I don't in reality want a good, working way to do this job; I simply want to understand what exactly is causing this way to not work. Further: the "0" being returned from the problem area - do you suppose it's a zero as in "No errors, exit 0", or is it the result of `grep -c blah` finding zero occurrences of "blah"? Or something else?

FWIW it's Slackware64 -current.

ntubski · 12-07-2010, 03:19 PM

Code:

grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; echo "($(grep -c "'$var'" tags))");

The $() gets evaluated before xargs is run so we can look at it separately:

Code:

~/tmp$ echo $(export var='{}'; printf "%s" "$var "; echo "($(grep -c "'$var'" tags))")
{} (0)

The inner $() gets evaluated as part of the outer, so we can look at that separately keeping in mind that it has var={} in its environment:

Code:

~/tmp$ export var='{}' ; echo "$(grep -c "'$var'" tags)"
0

Now let's look at the arguments that grep is receiving

Code:

~/tmp$ export var='{}' ; echo -c "'$var'" tags
-c '{}' tags

So grep is reporting that it finds 0 '{}' (including the quotes) in tags.

Dark_Helmet · 12-07-2010, 03:31 PM

ntubski beat me to it, but...

I was going to suggest testing your xargs substitution--use "line" instead of "{}" for the -I argument. The word "line" is in your sample data. So, if the xargs sustitution works, your output (0's) would remain the same. When I ran it though, the result came back as all 2's--which is exactly how many times "line" appears in the sample data.

So, at the very least, it would have indicated a problem with the substitution.

Though, again, ntubski gave a thorough explanation.

GrapefruiTgirl · 12-07-2010, 03:46 PM

Quote:

Originally Posted by Dark_Helmet

I was going to suggest testing your xargs substitution--use "line" instead of "{}" for the -I argument. The word "line" is in your sample data.

Not quite sure I understand how the code would look, or work, with these ideas; would you please show me how you strung the code together using your "line" idea?

crts · 12-07-2010, 04:07 PM

Hi,

another idea how to verify what grep "sees" is to echo $var into a temporary file:

Code:

grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; echo "$var" > tmp);

After the command finishes you can do

Code:

$ cat tmp
{}
$

If you do
echo '{}' > tags

then grep should always return 1. But you will have to loose the single-quotes around '$var' in your grep statement for that. Otherwise grep will search for a a single-quoted pair of braces like
'{}'

Code:

grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; echo "($(grep -c "$var" tags))");

[EDIT]

Quote:

Not quite sure I understand how the code would look, or work, with these ideas;

I think, Dark_Helmet is suggesting something like:

Code:

grep -o '#\w*' tags |\
 xargs -I line echo $(export var='line'; printf "%s" "$var "; echo "($(grep -c "$var" tags))");

Dark_Helmet · 12-07-2010, 04:12 PM

Sure thing. It's almost a one-to-one replacement of '{}' with 'line'--you also need to delete the single quotes in the nested subshell:

Code:

$ grep -o '#\w*' tags | xargs -Iline echo $(export var='line'; printf "%s" "$var "; echo "($(grep -c "$var" tags))");

The output of that should be:

Code:

#soccer_generic (2)
#soccer_intro (2)
#another_hash_tag (2)
#hastag_2 (2)
#hastag_2 (2)

It doesn't give your original, expected results, but it does give you different results than all 0's. It was meant to help track down the root cause of the problem--not solve it.

My thought process was this:
1. The grep -c command works at the terminal.
2. When bundled with your larger command, the grep -c command relied on variable substitution
3. Something seemed to be wrong with the substitution. So, the first step is to verify that a substitution, any substitution, is actually happening.

To do that, I looked at your sample data. I noticed the word "line" was used (i.e. count > 0) and was not used on every line (i.e. count < 3). There may have been other, better choices (e.g. "more", "information", etc.).

So, if xargs was substituting stdin whenever it saw "line" and you still got all 0's, then the problem is not with variable substitution. If, however, the substitution is not happening, then "line" will be passed through to the grep -c command, and the output should be 2.

Not sure if I explained the reasoning very well...

GrapefruiTgirl · 12-07-2010, 04:48 PM

Quote:

Originally Posted by Dark_Helmet

Sure thing. It's almost a one-to-one replacement of '{}' with 'line'--you also need to delete the single quotes in the nested subshell:

Code:

$ grep -o '#\w*' tags | xargs -Iline echo $(export var='line'; printf "%s" "$var "; echo "($(grep -c "$var" tags))");

The output of that should be:

Code:

#soccer_generic (2)
#soccer_intro (2)
#another_hash_tag (2)
#hastag_2 (2)
#hastag_2 (2)

Thanks for the additional explanation. It does indeed return '2', but the whole output is a little odd. I got similar output during my experimentation earlier today, except that I DID get the #hash* things printed, exceot they were missing the letter "n", which I believe is/was due to a newline problem. Here's what I get from your command:

Code:

root@reactor: grep -o '#\w*' tags | xargs -Iline echo $(export var='line'; printf "%s" "$var "; echo "($(grep -c "$var" tags))");
li e (2)
li e (2)
li e (2)
li e (2)
li e (2)
root@reactor:

Do you get the same?

Other repliers - thank you for the explanations & information thus far. I am still playing with this and considering all input given.

ntubski · 12-07-2010, 05:55 PM

Code:

~/tmp$ grep -o '#\w*' tags | xargs -Iline echo $(export var='line'; printf "%s" "$var "; echo "($(grep -c "$var" tags))");
#soccer_generic (2)
#soccer_intro (2)
#another_hash_tag (2)
#hastag_2 (2)
#hastag_2 (2)

Not sure how you got anything else.

The correct way of doing this is to ask xargs to run a subshell:

Code:

~/tmp$ grep -o '#\w*' tags | xargs -I{} sh -c 'printf "$0 (%d)\n" $(grep -c "$0" tags)' {}
#soccer_generic (1)
#soccer_intro (1)
#another_hash_tag (1)
#hastag_2 (2)
#hastag_2 (2)

grail · 12-07-2010, 06:37 PM

Thanks to ntubski for clearing up the correct way to do it, but I still have a further question which relates to what you said in your first post:

Quote:

Originally Posted by ntubski

The $() gets evaluated before xargs is run so we can look at it separately

I tried the following 2 tests and the results surprised me:

Code:

$ grep -o '#\w*' tags | xargs -I{} echo $(printf "%s" {})
#soccer_generic
#soccer_intro
#another_hash_tag
#hastag_2
#hastag_2

So on this test I thought, based on above information, that the {} should not have returned a value?

Code:

$ grep -o '#\w*' tags | xargs -I{} echo $(grep -c {} tags)
0
0
0
0
0

So this seems to now support the previous information??

I realise I am being silly here somehow, but please explain why one moment I am getting substitution and the next it does not?

Dark_Helmet · 12-07-2010, 06:49 PM

Quote:

Originally Posted by GrapefruiTgirl

Do you get the same?

Nope. I get the same thing ntubski does:

Code:

#soccer_generic (2)
#soccer_intro (2)
#another_hash_tag (2)
#hastag_2 (2)
#hastag_2 (2)

I'm not sure why you get "li e" -- if it were a newline issue, then I would expect your output to be:

Code:

li
e (2)
li
e (2)
. . .

Quote:

Originally Posted by grail

I tried the following 2 tests and the results surprised me:

Code:

grep -o '#\w*' tags | xargs -I{} echo $(printf "%s" {})

The first thing that happens is the "$(printf "%s" {})" -- the result of which is "{}}. At that point, the command becomes:

Code:

grep -o '#\w*' tags | xargs -I{} echo {}

At this point, xargs replaces each {} with stdin. And that is why you get the output you gave.

GrapefruiTgirl · 12-07-2010, 06:52 PM

By adding bash into the mix (and that -F for grep, for good measure) the problem goes away. This is what I would have liked the initial example to do, and is more compact too. Only remaining problem is that it doesn't notice dupes, so `sort` and `uniq` would come into play.

Code:

sasha@reactor: grep -o '#\w*' tags | xargs -I@ bash -c 'echo "@ ($(grep -Fc "@" tags))"'
#soccer_generic (1)
#soccer_intro (1)
#another_hash_tag (1)
#hastag_2 (2)
#hastag_2 (2)

PRE-SUBMIT-EDIT: I've been having this post ready to submit for like 45+ mins now, so some of you are ahead of what I'm posting here. Specifically ntubski: I figured out the bash/shell thing, sometime between my last post and your most recent post, though mine looks a little different; I'll have a closer look once I submit this.. Also, (@ ntubski & *Helmet) I must have had a copy/paste typo in the xargs -Iline version; it does work!

grail · 12-07-2010, 07:02 PM

Thanks DH ... I knew it was going to be something simple, but like GGirl I had been playing with so many combos I could not see the wood for the forest