LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 12-07-2010, 02:25 PM   #1
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
Post grep | xargs -I echo $(foo; bar; echo $(fee; fi; fo; fum)) == questionable output..


http://www.linuxquestions.org/questi...5/#post4183237

The above post is the inspiration for this experiment. Read it if you like, but I'm going to try to make that not necessary, by explaining it like it's a brand new problem.

Also, I KNOW this code is terrible, and is not the way to do this. I simply want to understand why it doesn't work - other methods of doing the job are not needed. Learning experience only! So..

The input file (filename = "tags") contains the below:
Code:
root@reactor: cat tags
This is about soccer #soccer_generic #soccer_intro . More information is here in more text.
This is line 2 #another_hash_tag #hastag_2 . And here is even more text.
Here's another line to increase count of tag: #hastag_2
root@reactor:
I expect my output to resemble the following:
Code:
#soccer_generic (1)
#soccer_intro (1)
#another_hash_tag (1)
#hastag_2 (2)
#hastag_2 (2)
Yes, I am aware that there's a duplicate line in the output - for this exercise it doesn't matter.
What I actually get for output, is this:
Code:
#soccer_generic (0)
#soccer_intro (0)
#another_hash_tag (0)
#hastag_2 (0)
#hastag_2 (0)
So notice the counts are not happening. Here's the code:
Code:
grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; echo "($(grep -c "'$var'" tags))");
Now, a demonstration about why I am confused about the output. I'm going to add an echo statement to the code, so we can see the command that's supposedly being run inside the subshell; observe the output:
Code:
grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; echo "($(echo grep -c "'$var'" tags))");
#soccer_generic (grep -c '#soccer_generic' tags)
#soccer_intro (grep -c '#soccer_intro' tags)
#another_hash_tag (grep -c '#another_hash_tag' tags)
#hastag_2 (grep -c '#hastag_2' tags)
#hastag_2 (grep -c '#hastag_2' tags)
So it looks to me like the command should be working. I can run one of those greps:
Code:
root# grep -c '#another_hash_tag' tags
1
So why doesn't the "1" appear when the whole command is run? Things I have tried include assorted combinations of :

grep | xargs -n 1 ...
grep | xargs -L 1 ...
grep | xargs -n 1 -L 1 ...
grep | xargs -d "\n" ...
grep -Z | xargs -0 ...
echo $(grep) | xargs ...
echo "$(grep)" | xargs ...
xargs ... <<<$(grep)
xargs ... <<<"$(grep)"

Every combination of those above, I've mixed and tried, and probably more. Been to the xargs manpage repeatedly, and grep's too. If I'm missing something obvious, that's cool - just tell me where it is!

Also tried grep -E, -e but didn't fiddle much with grep since the output appears such that grep is working fine as is.

Also tried without that subshell in there at the end, like so (and took out the brackets desired in the output, for code clarity):
Code:
root@reactor: grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; grep -c "'$var'" tags;)
#soccer_generic 0
#soccer_intro 0
#another_hash_tag 0
#hastag_2 0
#hastag_2 0
root@reactor: grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; echo grep -c "'$var'" tags;)
#soccer_generic grep -c '#soccer_generic' tags
#soccer_intro grep -c '#soccer_intro' tags
#another_hash_tag grep -c '#another_hash_tag' tags
#hastag_2 grep -c '#hastag_2' tags
#hastag_2 grep -c '#hastag_2' tags
root@reactor:
So still no go - seems the subshell itself is not the problem...

I've tried setting various shellopts, like dotglob/extglob/globstar, but I admit to not really having much of a grip on these anyways yet, as I rarely use them. I don't think they have anything to do with this, but you may correct me.

Tried removing the single quotes inside grep -c "'$var'" to no avail. Tried no quotes, double quotes, and both together on var='{}' to no avail.

Tried sticking `eval` into various places, with and without the subshell near the end. Made no difference:
Code:
root@reactor: grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; eval grep -c "'$var'" tags;)
#soccer_generic 0
#soccer_intro 0
#another_hash_tag 0
#hastag_2 0
#hastag_2 0
root@reactor: grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; eval echo grep -c "'$var'" tags;)
#soccer_generic grep -c #soccer_generic tags
#soccer_intro grep -c #soccer_intro tags
#another_hash_tag grep -c #another_hash_tag tags
#hastag_2 grep -c #hastag_2 tags
#hastag_2 grep -c #hastag_2 tags
root@reactor:
So, what do you folks think? Again - I don't in reality want a good, working way to do this job; I simply want to understand what exactly is causing this way to not work. Further: the "0" being returned from the problem area - do you suppose it's a zero as in "No errors, exit 0", or is it the result of `grep -c blah` finding zero occurrences of "blah"? Or something else?

FWIW it's Slackware64 -current.
 
Old 12-07-2010, 03:19 PM   #2
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian
Posts: 2,455

Rep: Reputation: 843Reputation: 843Reputation: 843Reputation: 843Reputation: 843Reputation: 843Reputation: 843
Code:
grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; echo "($(grep -c "'$var'" tags))");
The $() gets evaluated before xargs is run so we can look at it separately:
Code:
~/tmp$ echo $(export var='{}'; printf "%s" "$var "; echo "($(grep -c "'$var'" tags))")
{} (0)
The inner $() gets evaluated as part of the outer, so we can look at that separately keeping in mind that it has var={} in its environment:
Code:
~/tmp$ export var='{}' ; echo "$(grep -c "'$var'" tags)"
0
Now let's look at the arguments that grep is receiving
Code:
~/tmp$ export var='{}' ; echo -c "'$var'" tags
-c '{}' tags
So grep is reporting that it finds 0 '{}' (including the quotes) in tags.
 
Old 12-07-2010, 03:31 PM   #3
Dark_Helmet
Senior Member
 
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 369Reputation: 369Reputation: 369Reputation: 369
ntubski beat me to it, but...

I was going to suggest testing your xargs substitution--use "line" instead of "{}" for the -I argument. The word "line" is in your sample data. So, if the xargs sustitution works, your output (0's) would remain the same. When I ran it though, the result came back as all 2's--which is exactly how many times "line" appears in the sample data.

So, at the very least, it would have indicated a problem with the substitution.

Though, again, ntubski gave a thorough explanation.

Last edited by Dark_Helmet; 12-07-2010 at 04:20 PM.
 
Old 12-07-2010, 03:46 PM   #4
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Original Poster
Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
Quote:
Originally Posted by Dark_Helmet View Post
I was going to suggest testing your xargs substitution--use "line" instead of "{}" for the -I argument. The word "line" is in your sample data.
Not quite sure I understand how the code would look, or work, with these ideas; would you please show me how you strung the code together using your "line" idea?
 
Old 12-07-2010, 04:07 PM   #5
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,604

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446
Hi,

another idea how to verify what grep "sees" is to echo $var into a temporary file:
Code:
grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; echo "$var" > tmp);
After the command finishes you can do
Code:
$ cat tmp
{}
$
If you do
echo '{}' > tags

then grep should always return 1. But you will have to loose the single-quotes around '$var' in your grep statement for that. Otherwise grep will search for a a single-quoted pair of braces like
'{}'

Code:
grep -o '#\w*' tags |\
 xargs -I{} echo $(export var='{}'; printf "%s" "$var "; echo "($(grep -c "$var" tags))");
[EDIT]
Quote:
Not quite sure I understand how the code would look, or work, with these ideas;
I think, Dark_Helmet is suggesting something like:
Code:
grep -o '#\w*' tags |\
 xargs -I line echo $(export var='line'; printf "%s" "$var "; echo "($(grep -c "$var" tags))");

Last edited by crts; 12-07-2010 at 04:09 PM.
 
Old 12-07-2010, 04:12 PM   #6
Dark_Helmet
Senior Member
 
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 369Reputation: 369Reputation: 369Reputation: 369
Sure thing. It's almost a one-to-one replacement of '{}' with 'line'--you also need to delete the single quotes in the nested subshell:
Code:
$ grep -o '#\w*' tags | xargs -Iline echo $(export var='line'; printf "%s" "$var "; echo "($(grep -c "$var" tags))");
The output of that should be:
Code:
#soccer_generic (2)
#soccer_intro (2)
#another_hash_tag (2)
#hastag_2 (2)
#hastag_2 (2)
It doesn't give your original, expected results, but it does give you different results than all 0's. It was meant to help track down the root cause of the problem--not solve it.

My thought process was this:
1. The grep -c command works at the terminal.
2. When bundled with your larger command, the grep -c command relied on variable substitution
3. Something seemed to be wrong with the substitution. So, the first step is to verify that a substitution, any substitution, is actually happening.

To do that, I looked at your sample data. I noticed the word "line" was used (i.e. count > 0) and was not used on every line (i.e. count < 3). There may have been other, better choices (e.g. "more", "information", etc.).

So, if xargs was substituting stdin whenever it saw "line" and you still got all 0's, then the problem is not with variable substitution. If, however, the substitution is not happening, then "line" will be passed through to the grep -c command, and the output should be 2.

Not sure if I explained the reasoning very well...

Last edited by Dark_Helmet; 12-07-2010 at 04:18 PM.
 
Old 12-07-2010, 04:48 PM   #7
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Original Poster
Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
Quote:
Originally Posted by Dark_Helmet View Post
Sure thing. It's almost a one-to-one replacement of '{}' with 'line'--you also need to delete the single quotes in the nested subshell:
Code:
$ grep -o '#\w*' tags | xargs -Iline echo $(export var='line'; printf "%s" "$var "; echo "($(grep -c "$var" tags))");
The output of that should be:
Code:
#soccer_generic (2)
#soccer_intro (2)
#another_hash_tag (2)
#hastag_2 (2)
#hastag_2 (2)
Thanks for the additional explanation. It does indeed return '2', but the whole output is a little odd. I got similar output during my experimentation earlier today, except that I DID get the #hash* things printed, exceot they were missing the letter "n", which I believe is/was due to a newline problem. Here's what I get from your command:
Code:
root@reactor: grep -o '#\w*' tags | xargs -Iline echo $(export var='line'; printf "%s" "$var "; echo "($(grep -c "$var" tags))");
li e (2)
li e (2)
li e (2)
li e (2)
li e (2)
root@reactor:
Do you get the same?

Other repliers - thank you for the explanations & information thus far. I am still playing with this and considering all input given.
 
Old 12-07-2010, 05:55 PM   #8
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian
Posts: 2,455

Rep: Reputation: 843Reputation: 843Reputation: 843Reputation: 843Reputation: 843Reputation: 843Reputation: 843
Code:
~/tmp$ grep -o '#\w*' tags | xargs -Iline echo $(export var='line'; printf "%s" "$var "; echo "($(grep -c "$var" tags))");
#soccer_generic (2)
#soccer_intro (2)
#another_hash_tag (2)
#hastag_2 (2)
#hastag_2 (2)
Not sure how you got anything else.

The correct way of doing this is to ask xargs to run a subshell:
Code:
~/tmp$ grep -o '#\w*' tags | xargs -I{} sh -c 'printf "$0 (%d)\n" $(grep -c "$0" tags)' {}
#soccer_generic (1)
#soccer_intro (1)
#another_hash_tag (1)
#hastag_2 (2)
#hastag_2 (2)
 
Old 12-07-2010, 06:37 PM   #9
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,516

Rep: Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896
Thanks to ntubski for clearing up the correct way to do it, but I still have a further question which relates to what you said in your first post:
Quote:
Originally Posted by ntubski
The $() gets evaluated before xargs is run so we can look at it separately
I tried the following 2 tests and the results surprised me:
Code:
$ grep -o '#\w*' tags | xargs -I{} echo $(printf "%s" {})
#soccer_generic
#soccer_intro
#another_hash_tag
#hastag_2
#hastag_2
So on this test I thought, based on above information, that the {} should not have returned a value?
Code:
$ grep -o '#\w*' tags | xargs -I{} echo $(grep -c {} tags)
0
0
0
0
0
So this seems to now support the previous information??

I realise I am being silly here somehow, but please explain why one moment I am getting substitution and the next it does not?
 
Old 12-07-2010, 06:49 PM   #10
Dark_Helmet
Senior Member
 
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 369Reputation: 369Reputation: 369Reputation: 369
Quote:
Originally Posted by GrapefruiTgirl
Do you get the same?
Nope. I get the same thing ntubski does:
Code:
#soccer_generic (2)
#soccer_intro (2)
#another_hash_tag (2)
#hastag_2 (2)
#hastag_2 (2)
I'm not sure why you get "li e" -- if it were a newline issue, then I would expect your output to be:
Code:
li
e (2)
li
e (2)
. . .
Quote:
Originally Posted by grail
I tried the following 2 tests and the results surprised me:
Code:
grep -o '#\w*' tags | xargs -I{} echo $(printf "%s" {})
The first thing that happens is the "$(printf "%s" {})" -- the result of which is "{}}. At that point, the command becomes:
Code:
grep -o '#\w*' tags | xargs -I{} echo {}
At this point, xargs replaces each {} with stdin. And that is why you get the output you gave.
 
Old 12-07-2010, 06:52 PM   #11
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Original Poster
Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
By adding bash into the mix (and that -F for grep, for good measure) the problem goes away. This is what I would have liked the initial example to do, and is more compact too. Only remaining problem is that it doesn't notice dupes, so `sort` and `uniq` would come into play.

Code:
sasha@reactor: grep -o '#\w*' tags | xargs -I@ bash -c 'echo "@ ($(grep -Fc "@" tags))"'
#soccer_generic (1)
#soccer_intro (1)
#another_hash_tag (1)
#hastag_2 (2)
#hastag_2 (2)
PRE-SUBMIT-EDIT: I've been having this post ready to submit for like 45+ mins now, so some of you are ahead of what I'm posting here. Specifically ntubski: I figured out the bash/shell thing, sometime between my last post and your most recent post, though mine looks a little different; I'll have a closer look once I submit this.. Also, (@ ntubski & *Helmet) I must have had a copy/paste typo in the xargs -Iline version; it does work!

Last edited by GrapefruiTgirl; 12-07-2010 at 06:55 PM.
 
Old 12-07-2010, 07:02 PM   #12
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,516

Rep: Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896
Thanks DH ... I knew it was going to be something simple, but like GGirl I had been playing with so many combos I could not see the wood for the forest
 
  


Reply

Tags
expansion, grep, pipeline, subshell, xargs


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Grep with Echo Myiagros Linux - General 3 04-26-2010 10:12 AM
When I use xargs with echo, why does the output contain an extra space? web_janitor Linux - General 2 08-20-2009 04:18 PM
grep -v for foo or bar Viper Chief Linux - General 2 05-07-2008 02:52 AM
ls | echo, I got blank, why can't echo take the 2nd seat in a pipeline? elinuxqs Linux - Newbie 6 11-24-2006 08:25 AM
BASH: How to NOT echo to screen with "if echo $x | grep ".*"; then" eur0dad Programming 9 07-27-2006 02:14 PM


All times are GMT -5. The time now is 03:33 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration