LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-16-2014, 07:43 AM   #1
sigint-ninja
Member
 
Registered: Feb 2011
Location: Republic Of Ireland
Distribution: Debian,Centos,Slackware
Posts: 449

Rep: Reputation: 29
Where are you using awk, grep and sed?


Hi guys,

i understand the functions and fundamental differences between grep sed and awk...but cannot understand where you would utilise them in the real world...

i know configuration files are manipulated via the command line...
but would you really need to replace keywords like the with THE
these are the examples my book is giving me regarding these types of functions using grep sed and awk

are these commands used more for scripting?

i know grep is useful in a pipe command if you looking up specifics on something for example

lspci (lists all pci hardware)
lspci |grep vga (could list all data pertaining to vga within pci devices etc...this i get

but where practically does sed and awk come in? office workers arent using text files to store data are they?

thanks

Last edited by sigint-ninja; 10-16-2014 at 07:48 AM.
 
Old 10-16-2014, 09:41 AM   #2
rtmistler
Moderator
 
Registered: Mar 2011
Location: Sutton, MA. USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu
Posts: 4,087
Blog Entries: 10

Rep: Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521
I use grep in the command line to find stuff. I've sparingly used it as part of scripts. If you check the return from grep as shown in the manual page, it varies depending on whether the find itself failed or whether the command was issued in error, so one has to be careful.

I use sed to modify files from within a script or to globally change something in multiple files from the command line.

Never used awk, however my impression is that it's more highly used than sed; just from me seeing scripts where it's used.

In my case I'm a developer so I'm less about writing scripts except to provide for duplication of a process needed as part of a release.
 
Old 10-16-2014, 10:07 AM   #3
sycamorex
LQ Veteran
 
Registered: Nov 2005
Location: London
Distribution: Slackware64-current
Posts: 5,811
Blog Entries: 1

Rep: Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191
Quote:
Originally Posted by sigint-ninja View Post
Hi guys,

i understand the functions and fundamental differences between grep sed and awk...but cannot understand where you would utilise them in the real world...

i know configuration files are manipulated via the command line...
but would you really need to replace keywords like the with THE
these are the examples my book is giving me regarding these types of functions using grep sed and awk

are these commands used more for scripting?

i know grep is useful in a pipe command if you looking up specifics on something for example

lspci (lists all pci hardware)
lspci |grep vga (could list all data pertaining to vga within pci devices etc...this i get

but where practically does sed and awk come in? office workers arent using text files to store data are they?

thanks
While the functionality of the three tools overlap in many cases, typically you'd use:
awk) for data files organised in columns
sed) operations on strings in files organised in lines
grep) operations on lines (but only matching, not manipulating them like sed)

The above is a massive oversimplification and I'd characterise them in that way. The tool you choose should depend on the job.

Most often awk/sed would be used to modify a log files/ lines of code, etc. where the number of files or changes makes manual editing unfeasible.
 
Old 10-16-2014, 10:13 AM   #4
sigint-ninja
Member
 
Registered: Feb 2011
Location: Republic Of Ireland
Distribution: Debian,Centos,Slackware
Posts: 449

Original Poster
Rep: Reputation: 29
thanks for that guys...

sycamorex can you link me to or give me information that elaborates what you talking about...or a more real world example
or could i just go and google each command and play with it...i am just finding it difficult to place all of this in the real world
is it something you would only dive deep into once you were placed in a job???
 
Old 10-16-2014, 10:24 AM   #5
sycamorex
LQ Veteran
 
Registered: Nov 2005
Location: London
Distribution: Slackware64-current
Posts: 5,811
Blog Entries: 1

Rep: Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191Reputation: 1191
For examples of problems that people wanted to solve using sed/awk, just use the search option of this forum (look for sed or awk), you'll see threads with people's problems and solutions.


The best sed tutorial out there
http://www.grymoire.com/Unix/Sed.html
http://www.grymoire.com/Unix/Awk.html
http://www.grymoire.com/Unix/Grep.html

You'll also need to be familiar with regular expressions:
http://www.grymoire.com/Unix/Regular.html
 
Old 10-16-2014, 10:29 AM   #6
TenTenths
Senior Member
 
Registered: Aug 2011
Location: Dublin
Distribution: Centos 5 / 6 / 7
Posts: 2,166

Rep: Reputation: 751Reputation: 751Reputation: 751Reputation: 751Reputation: 751Reputation: 751Reputation: 751
Quote:
Originally Posted by sycamorex View Post
While the functionality of the three tools overlap in many cases, typically you'd use:
awk) for data files organised in columns
sed) operations on strings in files organised in lines
grep) operations on lines (but only matching, not manipulating them like sed)
grep/sed are great for doing things programmatically.
Quote:
Originally Posted by sigint-ninja View Post
or a more real world example
here's a command line I was running on a number of servers to change values in a config file based on the number of processors on a system.

[Edit:I'll split the command line out into the individual parts to make it a bit easier to read!]
Code:
grep 'loadavg\.\|uptime\.' conftab
sed s/$(grep "LOAD.loadavg.warn" conftab)/*:LOAD.loadavg.warn::@0-$(expr $(grep ^processor /proc/cpuinfo | wc -l) / 2 )/g -i conftab
sed s/$(grep "LOAD.loadavg.crit" conftab)/*:LOAD.loadavg.crit::@0-$(grep ^processor /proc/cpuinfo | wc -l)/g -i conftab


Quote:
Originally Posted by sigint-ninja View Post
is it something you would only dive deep into once you were placed in a job???
I use grep, awk and sed on pretty much a daily basis!

Last edited by TenTenths; 10-16-2014 at 10:31 AM.
 
Old 10-16-2014, 10:59 AM   #7
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,250

Rep: Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684
I follow the same line as others that there is always a right tool for the job. As far as 'real world' examples, just search this forum and you will find plenty of instances where people
have used all the commands you are looking at. I think it will also show you the diversity of the commands and maybe an idea on which is best for which jobs.
 
Old 10-16-2014, 11:18 AM   #8
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 2,959

Rep: Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268
I too use sed and grep on a daily basis. Here's a one-liner that finds the names of all files that are shell scripts in and below the current directory and stores that in a shell variable Files:
Code:
Files=$(find . -type f -exec file {} + | sed -ne 's/:.*shell script.*//p')
And no, it's not very robust -- file names with embedded white space or colon characters will be mishandled. I know I don't have any shell scripts with names like that.

awk is most commonly used in scripts, though I'll occasionally type a one-liner there, too. The examples I have tend to be complex (that's why they're in scripts). I've attached one that parses named fields out of a CSV file and prints them in a user-defined format.
Attached Files
File Type: txt csvextract.txt (2.7 KB, 1 views)
 
Old 10-16-2014, 11:31 AM   #9
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,258

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
Here's a little one-liner to grab the current temperature of a hard disk, using both grep and awk
Code:
/usr/sbin/smartctl -d sat -a /dev/sda | grep Temperature | awk '{ print $10 }'
This could then be nested inside a script that probes the temperature of all HDDs on a regular basis (say through a cron job) and takes action as necessary.
 
1 members found this post helpful.
Old 10-16-2014, 11:36 AM   #10
metaschima
Senior Member
 
Registered: Dec 2013
Distribution: Slackware
Posts: 1,982

Rep: Reputation: 490Reputation: 490Reputation: 490Reputation: 490Reputation: 490
I prefer awk to sed. Its syntax is clearer and it can probably do more than sed. However, for convenience reasons I'll use sed for substitutions. For most everything else I'll use awk.

I use grep for finding words in files for convenience reasons. It is very useful, but not as versatile as awk. Awk is actually a scripting language and very powerful one at that. If you want even more power, try perl.

Specific uses of awk would be parsing tables and CSV files as said above.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] replace grep -o with sed or awk nicomoresi Linux - Newbie 20 08-08-2011 12:13 PM
grep help or sed or awk dmchess Linux - Software 4 09-29-2010 07:53 PM
[SOLVED] Help using awk,sed and grep shakes82 Programming 34 07-08-2010 12:12 AM
[SOLVED] awk, sed, grep and paragraphs ThinkLinux Linux - Newbie 3 04-09-2010 02:22 PM
help with grep/sed/awk nikunjbadjatya Programming 8 02-17-2010 08:29 PM


All times are GMT -5. The time now is 05:12 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration