LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 06-28-2010, 06:36 AM   #16
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,604

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446

Quote:
Originally Posted by David the H. View Post
xargs should also work (once you take care of the word separation issue), as I believe it will process the input in suitably-sized chunks if necessary.
Hi,

how would you propose to solve the space issue in this case? The only way xargs can cope with this, afaik, is to use the -0 option. However, grep has only the -Z option which according to the man page
Quote:
-Z, --null
Output a zero byte (the ASCII NUL character) instead of the character that normally follows a file name. For example, grep -lZ outputs a zero byte after each file name instead of the usual newline. This option makes the output unambiguous, even in the presence of file names containing unusual characters like newlines. This option can be used with commands like find -print0, perl -0, sort -z, and xargs -0 to process arbitrary file names, even those that contain newline characters.
So no option to zero-terminate an "ordinary" match, unless I missed something.
[EDIT]
As for the chunk-sized processing, yes I can acknowledge this behaviour. I recently had such a case with a huge amount of input data. It would output a couple of screens, pause for a few seconds and then continue processing.

Last edited by crts; 06-28-2010 at 06:43 AM.
 
Old 06-28-2010, 06:10 PM   #17
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946
To tell the truth, I'm still a novice myself when it comes to the intricacies of xargs. But it looks like you at least have the option to set the xargs delimiter to newline (-d "\n"). If that's not enough you process the input with sed or something and add some other delimiter to the end of each entry that xargs could use.

By the way, I see that xargs --show-limits will tell you how big your command buffer is, and there are several options for controlling how much input is processed at any one time.
 
1 members found this post helpful.
Old 06-28-2010, 06:35 PM   #18
ryan858
Member
 
Registered: Feb 2009
Distribution: Slackware 12.2
Posts: 37

Rep: Reputation: 16
Couldn't the white spaces just be escaped? e.g. /media/My\ Book/...

This could be accomplished by simply replacing " " (space) with "\ " (backslash-space) in the logfile. Any decent text editor could do that in a few seconds.

Last edited by ryan858; 06-28-2010 at 06:36 PM.
 
Old 06-29-2010, 04:32 AM   #19
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,604

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446
Quote:
Originally Posted by David the H. View Post
To tell the truth, I'm still a novice myself when it comes to the intricacies of xargs. But it looks like you at least have the option to set the xargs delimiter to newline (-d "\n"). If that's not enough you process the input with sed or something and add some other delimiter to the end of each entry that xargs could use.

By the way, I see that xargs --show-limits will tell you how big your command buffer is, and there are several options for controlling how much input is processed at any one time.
Hi,

tried it with the -d option and it worked fine. Seems that it can be done with a one liner after all.
 
Old 06-29-2010, 09:06 AM   #20
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946
Quote:
Originally Posted by ryan858 View Post
Couldn't the white spaces just be escaped? e.g. /media/My\ Book/...

This could be accomplished by simply replacing " " (space) with "\ " (backslash-space) in the logfile. Any decent text editor could do that in a few seconds.
Yes, that's a possibility. But it seems to me like an unnecessary brute-force approach when xargs can handle it internally with the proper delimiter configured.

Also, filenames are generally ok, but let's say instead that you want to grab and process all the lines containing the word "alias" from a file like this (a sample bashrc).
Code:
#!/bin/bash

# This is a sample bashrc

# User specific aliases
alias ll='ls -l --color=auto'
alias la='ls -A --color=auto'

other stuff, etc...
Look at what happens when I try to escape the spaces in the input:
Code:
sed -n '/alias/ s/ /\\ /gp' sample_bashrc | xargs printf "[%s]\n"
[# User specific aliases and functions]
[alias ll=ls\ -l\ --color=auto]
[alias la=ls\ -A\ --color=auto]
Admittedly this is a useless example, but imagine that it's being sent to a script or something instead. First of all, I needed to escape it twice just so that one backslash went through to the final command. But also notice how the quotation marks in the input have protected some of the escapes, so that xargs doesn't recognize them. And the quotes themselves have disappeared.

Setting the -d delimiter, however, disables xargs' default processing, so that everything else is ignored as part of the string and only that delimiter means anything special.
Code:
grep 'alias' sample_bashrc | xargs -d "\n" printf "[%s]\n"
[# User specific aliases and functions]
[alias ll='ls -l --color=auto']
[alias la='ls -A --color=auto']
Now everything works as intended.

Last edited by David the H.; 06-29-2010 at 09:12 AM. Reason: minor change for readability
 
Old 06-30-2010, 11:30 PM   #21
GlennsPref
Senior Member
 
Registered: Apr 2004
Location: Brisbane, Australia
Distribution: Mageia Studio-13.37 Kubuntu.
Posts: 3,325
Blog Entries: 33

Rep: Reputation: 199Reputation: 199
Smile Well Done

Well Done, and thank you for posting the solution that worked for you.

Regards Glenn
 
Old 07-01-2010, 09:57 AM   #22
tange
LQ Newbie
 
Registered: Jul 2010
Posts: 13

Rep: Reputation: 9
Simple solution using GNU Parallel

It seems a better solution is using GNU Parallel http://www.gnu.org/software/parallel/ as it uses \n as separator by default.

Code:
grep '/media/My Book/HDD_Toshiba' logfile.log | parallel rm
Watch the intro video to GNU Parallel: http://www.youtube.com/watch?v=OpaiGYxkSuQ
 
1 members found this post helpful.
Old 07-01-2010, 02:46 PM   #23
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946
Neat. I hadn't heard of parallel before. Some of the options look very useful.

Strangely though, it doesn't seem to be in the Debian repositories. So while it may be more convenient to use in cases like this, it seems that it's not something that you can depend on to be available everywhere, unlike like xargs, which is a standard core utility.
 
Old 07-01-2010, 03:26 PM   #24
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,604

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446
Quote:
Originally Posted by David the H. View Post
I hadn't heard of parallel before.
That is probably because it hasn't been around that long, if this archive (http://ftp.gnu.org/gnu/parallel/) contains the whole release history.
Scroll down to the maintainer on this (http://www.gnu.org/software/parallel/) site. It appears that the poster is also the author/maintainer of the program. From the linked youtube video I'd say that the main purpose is to enhance performance for computers using a multicore cpu. Since this is seemingly a rather "young" program I would also recommend to stick with xargs, especially because the development focus was probably not on replacing xargs. So substituting xargs for parallel might be a bit premature at this point.
 
Old 07-02-2010, 07:07 AM   #25
tange
LQ Newbie
 
Registered: Jul 2010
Posts: 13

Rep: Reputation: 9
History of GNU Parallel

Parallel has been around since 2002 and had been in production at least since 2005. The reason for the apparent short history is that Parallel quite recently was accepted as a GNU tool and thus renamed to GNU Parallel.

GNU parallel did not have one but two main objectives: Replace xargs and run commands in parallel.

See the full history on http://www.gnu.org/software/parallel/history.html

(Yes, I am the author of GNU Parallel)
 
1 members found this post helpful.
  


Reply

Tags
batch, delete, parallel, script, xargs


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Shell script to delete folders and files dynamically and recursively rjbaca Linux - General 1 06-21-2010 11:26 AM
How can I use Shell script to edit a data at a particular location in a txt file? leena_d Programming 30 02-08-2010 12:43 AM
Shell script to automatically delete files with the same name as the parent directory pratap.iisc Programming 9 10-12-2009 10:17 AM
Help: Create a shell script to move only files wich has stopped growing proxmity Linux - Newbie 8 07-28-2009 12:40 PM
How to create/delete temp/backup files through a shell script ? Sid2007 Programming 4 10-17-2007 01:55 PM


All times are GMT -5. The time now is 08:04 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration