LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-03-2016, 06:39 PM   #1
lamont0
LQ Newbie
 
Registered: Feb 2016
Posts: 20

Rep: Reputation: Disabled
Using awk to filter output from the ls command


Good evening all. I am working on a assignment and would like some guidance.
The task is to use awk to list all files less than 32 bytes using output from ls -lR /home.
So I plan to redirect the ls -lR output to a file and then filter out the lines that do no
have a file name. This would be a filter on position $9. Hopefully this will leave me
with files and dirrectories. Next filter out the directories. Then filter out the files
that are greater than 31 bytes. Finally print the list. Does this sound like a reasonable plan?
The first step is easy: ls -lR /home > awk_data 2>/dev/null.
OK I never used awk so my terminology is weak. Searching finds a lot of results but nothing
that I can relate to. So my first step is to filter out all the lines that do not have a
filename and rewrite the data file. Are there any good examples to be found or search terms to
use.
 
Old 04-03-2016, 08:14 PM   #2
lamont0
LQ Newbie
 
Registered: Feb 2016
Posts: 20

Original Poster
Rep: Reputation: Disabled
Using

Stumbled across some very useful information that looks like it is a good solution.
This is what I am using.
Code:
#!/bin/bash
#=======================================================
# Script Name:  list_small_files
# By:           MJN
# Date:         04-03-2016
# Purpose:      Use awk to search recursively thru the /home directory
#               to display file less than 32 bytes using ls -lR /home.
# Command Line: ./list_small_files
# =====================================================l
trap "rm awk_data awk_data2 awk_data3 2> /dev/null; exit" 0 1 2 3
ls -lR /home > awk_data 2>/dev/null
awk 'length($9) > 0' awk_data > awk_data2
awk -F, '$1 !~ /drwx/' awk_data2 > awk_data3
awk  '{ if($5 < 32) print $0}' awk_data3 > awk_data4
awk '{print $0}' awk_data4
 
Old 04-04-2016, 05:23 AM   #3
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,437

Rep: Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842
I know you have marked this as solved, but thought I would point out a few things.

1. Just as you have created a bash script using the shebang '#!/bin/bash', you can also create an awk script using '#!/usr/bin/awk -f'

2. Whether using the above advice or keeping with your bash script and then using awk, all of your steps could be handled in a single call to awk

3. Using awk on the output of 'ls -l' is not recommended as a file name with spaces in it will return incorrect values when looking at $9 in the output. Also, on different machines (like mine)
$9 is not the last field on the line

4. You also can simply pipe your ls output to awk which would also negate the need for temp files

Looking at current code:
Code:
awk 'length($9) > 0' awk_data > awk_data2 # Wasn't sure why this was here but then remembered you are using 'ls -lR' and so some of the output has zero file names.  Would suggest looking up NF variable
awk -F, '$1 !~ /drwx/' awk_data2 > awk_data3 # What was the point here of changing the input separator to a comma?? Your regex could also be just the letter 'd' or if wanting to be more correct '^d'
awk  '{ if($5 < 32) print $0}' awk_data3 > awk_data4 # Like previous test, the print $0 is not needed - - awk '$5 < 32'
awk '{print $0}' awk_data4 # What is the purpose of this??  This will print exactly the same as the line above
Using your values (as I said ls does not display file name in $9 for me), the above could have been:
Code:
ls -lR | awk 'NF == 9 && $1 !~ /^d/ && $5 < 32'
caveat: As said above, this script assumes no white space in file names and that output has file names in the ninth field
 
Old 04-04-2016, 08:20 AM   #4
mikenash
Member
 
Registered: Dec 2014
Posts: 81

Rep: Reputation: Disabled
Thread: [SOLVED] Using awk to filter output from the ls command

Great, many thanks. I definitely appreciate the review, helpful suggestions, and alternative suggestions. I will be looking at this and learning. Thanks again!
 
Old 04-04-2016, 09:00 AM   #5
mikenash
Member
 
Registered: Dec 2014
Posts: 81

Rep: Reputation: Disabled
Thread: [SOLVED] Using awk to filter output from the ls command

Thank you grail. All your comments were correct. These mistakes were in translating information I found to act on my specific goal. I received the results I needed and wanted to report back to the forum on my progress. I planned to closely review the code to learn and refine it. I greatly appreciate your experience and review of my code. I am curious about your answer. I wanted to add 2>/dev/nul to suppress any error output but this did not work. Still investigating.
 
Old 04-04-2016, 09:42 AM   #6
mikenash
Member
 
Registered: Dec 2014
Posts: 81

Rep: Reputation: Disabled
Greetings. I figured out the redirection situation. Awk does not produce the error or care about it. I add the redirection to the output of the ls command.
 
Old 04-04-2016, 11:52 AM   #7
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,437

Rep: Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842
I am not sure I understand what error you are trying to suppress? Awk will only error if the code you write causes it to try a task it cannot perform (like divide by zero) and ls also
wouldn't throw any errors I can think of??

For future attempts, you may also want to bookmark this site

Glad you were able to get some use from the comments
 
Old 04-04-2016, 12:48 PM   #8
MadeInGermany
Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 277

Rep: Reputation: 111Reputation: 111
The pipe | connects the output of one command with the input of the next command.
Code:
ls -lR | awk ...
This is much easier and safer, compared to a redirection to/from a temporary file.
 
Old 04-04-2016, 03:46 PM   #9
mikenash
Member
 
Registered: Dec 2014
Posts: 81

Rep: Reputation: Disabled
Hello, thanks for all the replies.
The error was from the ls command so I place the redirection after the ls command.
Quote:
ls: cannot open directory /home/linadm: Permission denied
ls: cannot open directory /home/syslog: Permission denied
 
Old 04-04-2016, 04:37 PM   #10
MadeInGermany
Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 277

Rep: Reputation: 111Reputation: 111
Aha, you mean you redirect the stderr, e.g. to a "discard" device like this
Code:
ls -lR 2>/dev/null | awk ...
 
Old 04-04-2016, 06:47 PM   #11
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,437

Rep: Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842Reputation: 2842
As you should only ever have access to your own home directory, unless root, perhaps you should limit the ls to your own.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Filter output of a AWS EC2 command using sed or cut in bash legolasthehansy Linux - Newbie 8 10-25-2015 04:04 AM
[SOLVED] Awk: How to pass/store a command output in variable? shivaa Linux - Newbie 8 01-05-2013 12:30 PM
print output in desire format using awk or cut command dipanjan Linux - General 1 12-15-2012 01:59 AM
awk not parse the output of nc command! mythcat Linux - General 6 09-30-2011 03:00 AM
[SOLVED] Using bash command to execute output of awk sylye Programming 12 04-03-2011 04:37 AM


All times are GMT -5. The time now is 08:25 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration