LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 06-22-2010, 04:08 AM   #1
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,546
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
bash: how to discard unwanted stdin?


Hello

Often in bash we read lines from stdin in a loop and implicitly discard the remaining stdin by terminating the loop. Is it possible to discard it without terminating the loop? It could lead to smaller code.

Here's an example which uses two loops and below is the same algorithm assuming unwanted stdin can be discarded
Code:
found=
while read destination gateway _
do
    [[ $destination = default ]] && found=yes && break
done <<< "$( route )"

if [[ $found ]]; then
    found=
    ping -c 1 -q $gateway >/dev/null # populate ARP cache
    while read address _ hw_address _
    do
        [[ $address = $gateway ]]  && found=yes && break
    done <<< "$( arp )"
fi
Code:
found=
while read destination gateway _
do
    if [[ $destination = default ]]; then
        ping -c 1 -q $gateway >/dev/null # populate ARP cache
        <discard unwanted stdin>
        while read address _ hw_address _
        do
            [[ $address = $gateway ]] && found=yes && break
        done <<< "$( arp )"
    fi
done <<< "$( route )"
 
Old 06-22-2010, 04:23 AM   #2
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 15

Rep: Reputation: 233Reputation: 233Reputation: 233
Quote:
Originally Posted by catkin View Post
Code:
found=
while read destination gateway _
do
    if [[ $destination = default ]]; then
        ping -c 1 -q $gateway >/dev/null # populate ARP cache
        <discard unwanted stdin>
        while read address _ hw_address _
        do
            [[ $address = $gateway ]] && found=yes && break
        done <<< "$( arp )"
    fi
done <<< "$( route )"
If I get what you meant, perhaps that can be done this way:
Code:
while read destination gateway _
do
    if [[ $destination = default ]]; then
        ping -c 1 -q $gateway >/dev/null # populate ARP cache

        while read something; do continue; done  # add some conditions here.  there's a trick if you need the last input to the following loop

        while read address _ hw_address _
        do
            [[ $address = $gateway ]] && found=yes && break
        done <<< "$( arp )"
    fi
done <<< "$( route )"
or the other way around
Code:
while read -u 3 destination gateway _
do
    if [[ $destination = default ]]; then
        ping -c 1 -q $gateway >/dev/null # populate ARP cache

        while read -u 4 address _ hw_address _
        do
            [[ $address = $gateway ]] && found=yes && break
        done 4< <(exec arp)
    fi
done 3< <(exec route)
What input from route are you trying to discard by the way?

Last edited by konsolebox; 06-22-2010 at 04:25 AM.
 
1 members found this post helpful.
Old 06-22-2010, 04:39 AM   #3
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,546
Blog Entries: 28

Original Poster
Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Thanks konsolebox, the second method using unit/fd numbers is elegant

It works without the "exec"s in the process substitution subshells too, as in done 4< <(arp) instead of done 4< <(exec arp)

EDIT: I'm only interested in the default gateway line from route output; once that is found the rest is irrelevant.

Last edited by catkin; 06-22-2010 at 04:41 AM.
 
Old 06-22-2010, 04:50 AM   #4
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 15

Rep: Reputation: 233Reputation: 233Reputation: 233
Quote:
Originally Posted by catkin View Post
It works without the "exec"s in the process substitution subshells too, as in done 4< <(arp) instead of done 4< <(exec arp)
It's really just optional. I just prefer not to summon another subprocess.
Quote:
EDIT: I'm only interested in the default gateway line from route output; once that is found the rest is irrelevant.
You might also find this pattern helpful (just in case):
Code:
while read LINE; do
    if [[ <expression to find line> ]]; then
        while
            < run the statements here that will process LINE >
            read LINE
        do
            continue  # or just : if you like
        done
    fi
done

Last edited by konsolebox; 06-22-2010 at 04:53 AM.
 
1 members found this post helpful.
Old 06-22-2010, 09:58 AM   #5
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,546
Blog Entries: 28

Original Poster
Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by konsolebox View Post
It's really just optional. I just prefer not to summon another subprocess.
Ah, I get you; with the bare (arp) a shell will be forked into the subshell and will then fork+exec to run arp, whereas using (exec arp), a shell will be forked into the subshell and will then exec arp thus saving a fork call. One more word in the code, one less (relatively expensive) system call in the execution. Nice.

Quote:
Originally Posted by konsolebox View Post
You might also find this pattern helpful
Very helpful -- experimenting with it I found my OP was solving a problem that didn't exist; bash is smart enough to save and restore "here strings"

First the experimental/demonstration script:
Code:
#! /bin/bash

# How do nested read while loops behave with "here string"s?

while read LINE
do
    echo "Outer: LINE is '$LINE'"
    if [[ $LINE = 2 ]]; then
        while read LINE
        do
            echo "Inner: LINE is '$LINE'"
            [[ $LINE = B ]] && break
        done <<< "$( printf 'A\nB\nC\n' )"
    fi
done <<< "$( printf '1\n2\n3\n' )"
The output is
Code:
Outer: LINE is '1'
Outer: LINE is '2'
Inner: LINE is 'A'
Inner: LINE is 'B'
Outer: LINE is '3'
Thus showing that bash saved the outer "here string" when it started the inner loop and restored it when it exited the inner loop.

Taking advantage of this feature, a better version of the get_default_gateway_MAC.sh script (better = simpler and does not need process substitution so more portable):
Code:
found=
while read destination gateway _
do
    if [[ $destination = default ]]; then
        ping -c 1 -q $gateway >/dev/null    # Populate ARP cache
        while read address _ hw_address _
        do
            [[ $address = $gateway ]] && found=yes && break 2
        done <<< "$( exec arp )"
    fi
done <<< "$( exec route )"
 
Old 06-22-2010, 11:58 PM   #6
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 15

Rep: Reputation: 233Reputation: 233Reputation: 233
Quote:
Originally Posted by catkin View Post
Code:
<<< "$( exec route )"
With larger outputs we should also be careful with this method since bash will most probably (as I've found out with other shells as well) allocate all the output of route first then store it in memory. It always depends on the implementation but it doesn't hurt to be more careful.
 
1 members found this post helpful.
Old 06-23-2010, 02:39 AM   #7
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,516

Rep: Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896
So try not to smack me about too much, but as opposed to the question asked about discarding stdin I thought I would just give an alternative for the script:
Code:
def=$(route | awk '/default/{print $2}')

ping -c 1 -q $def >/dev/null

arp | awk -v chk=$def '$1 == chk{print "found"}'
 
Old 06-23-2010, 07:55 AM   #8
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,546
Blog Entries: 28

Original Poster
Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by konsolebox View Post
With larger outputs we should also be careful with this method since bash will most probably (as I've found out with other shells as well) allocate all the output of route first then store it in memory. It always depends on the implementation but it doesn't hurt to be more careful.
Thanks for the perspective -- so better to use process substitution when the command output is large.
 
Old 06-23-2010, 08:21 AM   #9
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,546
Blog Entries: 28

Original Poster
Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by grail View Post
So try not to smack me about too much, but as opposed to the question asked about discarding stdin I thought I would just give an alternative for the script:
Code:
def=$(route | awk '/default/{print $2}')

ping -c 1 -q $def >/dev/null

arp | awk -v chk=$def '$1 == chk{print "found"}'
Hello grail, nice to see you passing by the thread and thanks for the alternative

Is that alternative functionally equivalent, though? The intention is to get the default gateway's hardware address, not its IP address. Does it handle the case where there is no "default" in the routing table? Does it set $found to indicate the validity of the address?

The default gateway's hardware address will be used by a netbook's boot script as a "good enough" way of identifying the LAN so it can do LAN-specific initialisation. Being boot script code, performance is significant so the sweet spot between coding complexity and the fork+exec count is more towards the coding side.

Your alternative is tighter and easier to maintain than mine (natch!) but will take more resources to run, which matters when booting an Atom-based system!

Last edited by catkin; 06-23-2010 at 08:21 AM. Reason: :-) to :)
 
Old 06-23-2010, 09:12 AM   #10
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,516

Rep: Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896
hmmmm ... I am guessing there must be a lot more to the code that I have not seen as my output is the same, except I have output the word found instead
of yes and not attached to a variable which I know you would have no trouble in doing.

I would be curious, and I will premiss this by saying I am no guru, about using more resources in mine than yours.
Are we saying that the use of awk is a bigger hit than the 2 while loops being iterated over?

And in answer to your questions, again from the original script presented:
Quote:
The intention is to get the default gateway's hardware address, not its IP address.
I see the hardware address being obtained but not used, hence I only returned the true statement 'found'
Quote:
Does it handle the case where there is no "default" in the routing table?
To be honest, in its current state there is obviously no control structure, but again, a simple if can handle this.
Also I would point out that if your script hits a route with no 'default' it will exit and do nothing.
Quote:
Does it set $found to indicate the validity of the address?
Again back to my original statement that supplying the last line as an assignment to a variable, or better yet a control, perhaps 'if', can take care of all that.

I guess my initial point was that maybe instead of discarding values we can maybe be smarter about what values we are getting?

I hope I am being constructive as I always value your opinions and very happy to learn

cheers
grail
 
Old 06-24-2010, 12:09 PM   #11
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,546
Blog Entries: 28

Original Poster
Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by grail View Post
hmmmm ... I am guessing there must be a lot more to the code that I have not seen as my output is the same, except I have output the word found instead
of yes and not attached to a variable which I know you would have no trouble in doing.
Here it is in context
Code:
    # Get the default gateway hardware address
    found=
    while read destination gateway _
    do
        if [[ $destination = default ]]; then
            ping -c 1 -q $gateway >/dev/null # populate ARP cache
            while read address _ hw_address _
            do
                [[ $address = $gateway ]] && found=yes && break
            done <<< "$( exec arp )"
        fi
    done <<< "$( exec route )"

    # LAN-specific actions, based on identification by default gateway hardware address
    if [[ $found ]]; then
        case $hw_address in
            '<some MAC address, not published on the Internet!>' )
                echo 'Mounting "home" LAN networked file systems'
                <some network file system mounts>
                sleep 1
                # Start the OpenSSH SSH daemon
                if [ -x /etc/rc.d/rc.sshd ]; then
                    echo 'Starting OpenSSH SSH daemon'
                    /etc/rc.d/rc.sshd start
                fi
                ;;
            * )
                echo "Gateway HW address $hw_address not configured in rc.local"
        esac
    else
        echo 'Gateway HW address not found' >&2
    fi
EDIT: the above code fails if the default gateway IP address is resolved to a name that is longer than the column width provided by the route command. The following modification solves this problem by not resolving names
Code:
    # Get the default gateway hardware address
    found=
    while read destination gateway _
    do
        if [[ $destination = '0.0.0.0' ]]; then
            ping -c 1 -q $gateway >/dev/null # populate ARP cache
            while read address _ hw_address _
            do
                [[ $address = $gateway ]] && found=yes && break
            done <<< "$( exec arp -n )"
        fi
    done <<< "$( exec route -n )"

Quote:
Originally Posted by grail View Post
I would be curious, and I will premiss this by saying I am no guru, about using more resources in mine than yours.
Are we saying that the use of awk is a bigger hit than the 2 while loops being iterated over?
Yes

fork+exec is "expensive" and bash runs fast when processing small (say < 5 kB) strings. To be sure (and to get a feel for where awk's faster processing of large amounts of text begins to outweigh the fork+exec cost) testing would be necessary. The ping would have to be removed (very slow compared to the rest of the code) and to be measurable the code would have to be put in a loop; in the case of this code I don't think there are any buffering effects that would affect the results (as there would be when doing file I/O).

Quote:
Originally Posted by grail View Post
And in answer to your questions, again from the original script presented:

I see the hardware address being obtained but not used, hence I only returned the true statement 'found'

To be honest, in its current state there is obviously no control structure, but again, a simple if can handle this.
Also I would point out that if your script hits a route with no 'default' it will exit and do nothing.

Again back to my original statement that supplying the last line as an assignment to a variable, or better yet a control, perhaps 'if', can take care of all that.
All true! You ave me bang to rights guvnor, an no mistake!

Doing nothing when the routing table has no default route is intended -- this is a "convenience" facility so it's not worth analysing rc.inet1.conf to see if there should be a default route or re-running DHCP client as might be appropriate on a mission-critical, robust system.

Quote:
Originally Posted by grail View Post
I guess my initial point was that maybe instead of discarding values we can maybe be smarter about what values we are getting?

I hope I am being constructive as I always value your opinions and very happy to learn
If there was a lot of data to search then I would go with being smarter about getting the values but for the small amount of data output by the route command (on a personal netbook) I believe that trawling through it line by line until getting what is needed is faster without requiring overly complex code.

Always a pleasure to debate with you; debate is a great way to learn different ways, to see other perspectives, to increase the breadth and depth of understanding and thus become better at what we do. Just so long as you remember that I'm always right!

Last edited by catkin; 09-13-2010 at 09:46 PM. Reason: bash indentation correction
 
Old 06-24-2010, 08:51 PM   #12
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,516

Rep: Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896
I was talking to a network guy here at work (i am still learning) and he has said that to the best of his knowledge the 'default' will never not be in the routing table
(just passing on what was said )

So to this end, this might help for later testing, this is the same with the changes:
Code:
def=$(route | awk '/default/{print $2}')

ping -c 1 -q $def >/dev/null

case $(arp | awk -v chk=$def '$1 == chk{print $3}') in # I assume we are using case as there may be other addresses later??
    '<some MAC address, not published on the Internet!>' )
        echo 'Mounting "home" LAN networked file systems'
        <some network file system mounts>
        sleep 1
        # Start the OpenSSH SSH daemon
        if [ -x /etc/rc.d/rc.sshd ]; then
            echo 'Starting OpenSSH SSH daemon'
            /etc/rc.d/rc.sshd start
        fi
        ;;
    * )
        echo "Gateway HW address $hw_address not configured in rc.local"
esac
Assuming my friend here is correct your last echo would never get executed, so I have left it off.

Quote:
Just so long as you remember that I'm always right!
For now ... lol
 
Old 06-24-2010, 09:30 PM   #13
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,546
Blog Entries: 28

Original Poster
Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by grail View Post
I was talking to a network guy here at work (i am still learning) and he has said that to the best of his knowledge the 'default' will never not be in the routing table
(just passing on what was said )
That is not correct as shown by this terminal session
Code:
root@CW8:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
localnet        *               255.255.255.0   U     0      0        0 eth0
loopback        *               255.0.0.0       U     0      0        0 lo
default         192.168.168.1   0.0.0.0         UG    1      0        0 eth0
root@CW8:~# route del default
root@CW8:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
localnet        *               255.255.255.0   U     0      0        0 eth0
loopback        *               255.0.0.0       U     0      0        0 lo
That is slightly artificial but could happen by accident. More realistically, we have had several examples here on LQ in which questioner's troubles were traced to a missing default gateway.

Regards the case statement -- yes, more cases are envisaged.
 
  


Reply

Tags
bash, read, stdin


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
BASH: Is it possible to take STDIN? microsoft/linux Programming 8 12-30-2011 04:21 PM
redirecting stdin in bash script artur Programming 3 12-09-2011 06:07 AM
[SOLVED] [bash] sort string and discard duplicates hashbang#! Programming 10 08-21-2009 06:17 AM
bash stdin redirect Yustu Programming 3 03-13-2009 07:01 PM
BASH -copy stdin to stdout (replace cat) (bash browser) gnashley Programming 4 07-21-2008 01:14 PM


All times are GMT -5. The time now is 11:47 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration