LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices



Reply
 
Search this Thread
Old 04-24-2010, 12:59 PM   #1
danielbmartin
Senior Member
 
Registered: Apr 2010
Location: Apex, NC, USA
Distribution: Ubuntu
Posts: 1,165

Rep: Reputation: 306Reputation: 306Reputation: 306Reputation: 306
Replicate a field


Hello.

I have a file containing text. I want to replicate a specific field.
For example, I might want to append a copy of the second word of each line to the end of that line.

Have:
Once upon a midnight dreary, while I pondered weak and weary,
Over many a quaint and curious volume of forgotten lore,

Want:
Once upon a midnight dreary, while I pondered weak and weary, upon
Over many a quaint and curious volume of forgotten lore, many

Is there a Linux command which will do this?
I seek a basic command, not awk, not Perl, because I haven't learned those things yet.

Daniel B. Martin
 
Old 04-24-2010, 01:10 PM   #2
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 729Reputation: 729Reputation: 729Reputation: 729Reputation: 729Reputation: 729Reputation: 729
First, let's define some terms: There are no "Linux commands". There are shell commands---BASH being the most common shell---and there are a bazillion utilities, applications, etc.

For text manipulation, common utilities include SED, AWK, and Perl. (Maybe Python also)

The BASH man pages will tell you about the commands built-in to BASH.

Second, I do not recommend posting a question here, and then placing restrictions on what solutions are offered. In fact, since you are talking about fields, I suspect that AWK may be one of the better choices.

I assume you want to do this on a line by line basis. Thus, you cannot simply use one tool to grab a word into a variable, and then make a second pass to add that variable to the end of the line.
 
Old 04-26-2010, 02:19 AM   #3
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,004
Blog Entries: 11

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
Indeed ... this one screams "awk" and "perl" at the top of
its lungs...

Code:
awk '{print $0", "$2}' file
You could use a shell script and treat each line like so:
Code:
...
scnd=$( echo $line|sed 's/[[:space:]][[:space:]]*/ /g'| cut -d" " -f2)
echo ${line}", "${scnd}
...
The sed is in there in case there's a few consecutive spaces or tabs
in the line which would throw "cut" off.

Personally I find the awk version cleaner and more concise.


Cheers,
Tink
 
Old 04-26-2010, 02:39 AM   #4
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,563
Blog Entries: 29

Rep: Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179
Quote:
Originally Posted by danielbmartin View Post
Is there a Linux command which will do this?
I seek a basic command, not awk, not Perl, because I haven't learned those things yet.
pixellany usefully defined terms. Since you didn't exclude bash (bash commands could be held as "basic"), here's a pure bash solution
Code:
#!/bin/bash

while read line
do
    array=( $line )
    echo $line ${array[1]}
done <  input.txt
EDIT: or, more neatly
Code:
#!/bin/bash
while read -a array
do
    echo ${array[*]} ${array[1]}
done <  input.txt

Last edited by catkin; 04-26-2010 at 02:41 AM.
 
Old 04-26-2010, 07:30 AM   #5
danielbmartin
Senior Member
 
Registered: Apr 2010
Location: Apex, NC, USA
Distribution: Ubuntu
Posts: 1,165

Original Poster
Rep: Reputation: 306Reputation: 306Reputation: 306Reputation: 306
Quote:
Originally Posted by pixellany View Post
... I do not recommend posting a question here, and then placing restrictions on what solutions are offered. ...
I respect your expertise and long service to this forum. This is a counterargument which you may find reasonable.

This is the Newbie Forum. I am a newbie, learning Linux on my own. I can't learn all of it at once, so I'm starting with what I mistakenly called Linux commands. Commands such as sed and grep are so powerful that I want to develop competence and confidence with them before moving on to awk or Perl.

If I place no bounds on solutions some members will produce awk or Perl solutions. Then they feel betrayed when I won't use their hard work. That's because I am unwilling to use code that I don't understand.

Daniel B. Martin
 
Old 04-26-2010, 09:01 AM   #6
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 729Reputation: 729Reputation: 729Reputation: 729Reputation: 729Reputation: 729Reputation: 729
Quote:
Originally Posted by danielbmartin View Post
I respect your expertise and long service to this forum. This is a counterargument which you may find reasonable.

This is the Newbie Forum. I am a newbie, learning Linux on my own. I can't learn all of it at once, so I'm starting with what I mistakenly called Linux commands. Commands such as sed and grep are so powerful that I want to develop competence and confidence with them before moving on to awk or Perl.

If I place no bounds on solutions some members will produce awk or Perl solutions. Then they feel betrayed when I won't use their hard work. That's because I am unwilling to use code that I don't understand.

Daniel B. Martin
I totally understand your point of view---and I especially agree with the last sentence.

The only thing I can offer is that the work required to apply the wrong tool often eclipses the work required to learn the right tool. I have personally demonstrated this by coming up with some totally convoluted SED code and then watching the AWK experts swoop in with something far better.

I recommend learning all of the most common tools in the depth required to get your work done. In my case, I know SED and GREP well enough to know what problems will be difficult or even impossible. From this, I know when I need to dig back into AWK and learn a bit more.
 
Old 04-26-2010, 09:18 AM   #7
MTK358
LQ 5k Club
 
Registered: Sep 2009
Posts: 6,443
Blog Entries: 3

Rep: Reputation: 714Reputation: 714Reputation: 714Reputation: 714Reputation: 714Reputation: 714Reputation: 714
Code:
$ echo "this is test text" | sed -r 's:^([^ \t]+[ \t]+)([^ \t]+)(.*)$:\1\2\3, \2:'
this is test text, is
 
Old 04-26-2010, 09:28 AM   #8
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 729Reputation: 729Reputation: 729Reputation: 729Reputation: 729Reputation: 729Reputation: 729
I love it!!!! Another SED fanatic is released into the world.

This eloquently demonstrates my point above:
Quote:
The only thing I can offer is that the work required to apply the wrong tool often eclipses the work required to learn the right tool. I have personally demonstrated this by coming up with some totally convoluted SED code and then watching the AWK experts swoop in with something far better.
But then MTK's SED solution is NOT convoluted at all---it is a very simple and elegant use of backreferences.
 
Old 04-26-2010, 10:10 AM   #9
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,692

Rep: Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987
I think we can do it a little different, but same result:
Code:
echo "this is test text" | sed -r 's:[ \t]+([^ \t]+).*:&, \1:'
 
Old 04-26-2010, 10:28 AM   #10
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
Yet another different approach...
Code:
paste -d' ' file <(cut -d' ' -f2 file)
but maybe too specific for the example shown in the original post.
 
1 members found this post helpful.
Old 04-26-2010, 11:14 AM   #11
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,563
Blog Entries: 29

Rep: Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179
Quote:
Originally Posted by danielbmartin View Post
This is the Newbie Forum. I am a newbie, learning Linux on my own. I can't learn all of it at once, so I'm starting with what I mistakenly called Linux commands. Commands such as sed and grep are so powerful that I want to develop competence and confidence with them before moving on to awk or Perl.
I respect and understand your position; I would like to offer a counter argument.

There are many commands in the toolset, each with pros and cons for solving various problems. I doubt that any of us are totally fluent with them all. It is not necessary, even if possible, to completely master each before moving on to the next. Another approach is to learn simple usage of an increasing number and gradually extend that knowledge as convenient, as need arises.

This problem suits awk particularly well, allowing Tinkster to offer the simple and comparatively comprehensible
Code:
awk '{print $0", "$2}' file
Hoping to tempt you, it breaks down like this:
  1. awk <string> file means run awk with program <string>, taking input from file.
  2. awk processes each line in turn.
  3. An awk program comprises patterns and actions; when the pattern matches the line the action is performed.
  4. In this case no pattern is given; for awk that matches all lines.
  5. The action is contained in { }.
  6. awk puts the whole line in variable $0 and parses the line into $1, $2, $3 ... words according to its word separator.
  7. The default word separator is a space.
  8. awk's print function prints its arguments to standard output, by default the terminal.
  9. In awk, literal strings are given in double quotes.
  10. awk concatenates adjacent strings.
  11. Thus $0", "$2 is the whole line, followed by comma and space followed by the second word of the line. For every line of file, awk prints that to standard output.
 
1 members found this post helpful.
Old 04-27-2010, 08:03 PM   #12
danielbmartin
Senior Member
 
Registered: Apr 2010
Location: Apex, NC, USA
Distribution: Ubuntu
Posts: 1,165

Original Poster
Rep: Reputation: 306Reputation: 306Reputation: 306Reputation: 306
Quote:
Originally Posted by catkin View Post
... This problem suits awk particularly well, allowing Tinkster to offer the simple and comparatively comprehensible
Code:
awk '{print $0", "$2}' file
Hoping to tempt you, it breaks down like this:
  1. awk <string> file means run awk with program <string>, taking input from file.
  2. awk processes each line in turn.
  3. An awk program comprises patterns and actions; when the pattern matches the line the action is performed.
  4. In this case no pattern is given; for awk that matches all lines.
  5. The action is contained in { }.
  6. awk puts the whole line in variable $0 and parses the line into $1, $2, $3 ... words according to its word separator.
  7. The default word separator is a space.
  8. awk's print function prints its arguments to standard output, by default the terminal.
  9. In awk, literal strings are given in double quotes.
  10. awk concatenates adjacent strings.
  11. Thus $0", "$2 is the whole line, followed by comma and space followed by the second word of the line. For every line of file, awk prints that to standard output.
Thank you for the detailed explanation. It whets my appetite for learning awk.

Some respondents misread the original post. The objective is to append the second word in each line to that line. There was no need for an additional comma. With that clarification, several of the offered code segments could be simplified.

Daniel B. Martin
 
Old 04-27-2010, 08:08 PM   #13
danielbmartin
Senior Member
 
Registered: Apr 2010
Location: Apex, NC, USA
Distribution: Ubuntu
Posts: 1,165

Original Poster
Rep: Reputation: 306Reputation: 306Reputation: 306Reputation: 306
Quote:
Originally Posted by colucix View Post
Yet another different approach...
Code:
paste -d' ' file <(cut -d' ' -f2 file)
but maybe too specific for the example shown in the original post.
Love it! One line of code, and more readable than some of the other suggested code segments. (More readable, at least, to this newbie.)

Technical Excellence may be defined as "completeness of function coupled with economy of means." Your solution qualifies as TE!

For extra credit: show how the output may be directed to a file rather than standard output.

Daniel B. Martin
 
Old 04-28-2010, 03:53 AM   #14
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,563
Blog Entries: 29

Rep: Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179
Quote:
Originally Posted by danielbmartin View Post
show how the output may be directed to a file rather than standard output.
Standard output can be directed to a file using the output redirection operator ">" as in
Code:
ls > my_file
Sometimes a command produces standard error as well. If you want it inn the same file then
Code:
command > my_file 2>&1
where "2>&1" means "send standard error (stream 2) to the same place as standard input (&1) is going".
In case you want them in different files
Code:
command > my_file.stdout 2> my_file.stderr
 
Old 04-28-2010, 07:52 AM   #15
MTK358
LQ 5k Club
 
Registered: Sep 2009
Posts: 6,443
Blog Entries: 3

Rep: Reputation: 714Reputation: 714Reputation: 714Reputation: 714Reputation: 714Reputation: 714Reputation: 714
> redirects stdout to a file.

2> redirects stderr to a file.

&> redirects both stdout and stderr to a file.
 
  


Reply

Tags
processing, text


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] awk: how to print a field when field position is unknown? elfoozo Programming 12 08-18-2010 04:52 AM
Can't replicate the site using wget ravi.xolve Linux - Software 4 01-20-2010 12:42 AM
awk printing from Nth field to last field sebelk Programming 2 01-08-2010 10:39 AM
php question, how do I get a return from a field within a field? cherrington Programming 11 04-29-2009 02:27 AM
How DNS works? - how does it replicate? neocontrol Linux - Networking 1 08-23-2007 08:56 AM


All times are GMT -5. The time now is 08:07 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration