LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-17-2014, 08:52 AM   #1
bangnagr
Member
 
Registered: Jan 2013
Posts: 48

Rep: Reputation: 4
Output redirection to file until n number of lines are appended.


Hello all

I'm writing a script where i want to get the size of the package (any). Depending on the size of the package, my script's logic changes.

My logic for getting file size is as follows:
  1. Ask user for package name:
  2. wget the file, but redirect output to a text file.
  3. grep 'Length' keyword from text file (5th line) | awk for 2nd field and assign it to a variable.
  4. and finally, delete partially downloaded package and output redirected text file.

Works great!! For example, in my script, i use:

Code:
timeout 10s wget http://some-package-file &> some-output-file

if grep -q 'Length' some-output-file ; then
  VARIABLE="$(grep 'Length' some-output-file | awk '{print $2}')"
fi
But the problem is - depending on the speed of network/server, not always i get the 5th line of output. Sometimes some server are too slow and takes more time to fill the output.

So i want output redirection to happen until the 5 lines are appended to text file, and then only exit. That way, i can count on that text file.

Any help really appreciated.

Thanks.

Last edited by bangnagr; 08-17-2014 at 08:54 AM.
 
Old 08-17-2014, 09:27 AM   #2
keefaz
LQ Guru
 
Registered: Mar 2004
Distribution: Slackware
Posts: 5,387

Rep: Reputation: 397Reputation: 397Reputation: 397Reputation: 397
You mean you start wget in background, scan its redirected output and then possibly kill wget to stop it downloading, delete partial download etc?
 
Old 08-17-2014, 09:34 AM   #3
bangnagr
Member
 
Registered: Jan 2013
Posts: 48

Original Poster
Rep: Reputation: 4
Quote:
Originally Posted by keefaz View Post
You mean you start wget in background, scan its redirected output and then possibly kill wget to stop it downloading, delete partial download etc?
Exactly. I use this technique not only for this, but also for most of the other programs.

Last edited by bangnagr; 08-17-2014 at 09:35 AM.
 
Old 08-17-2014, 12:40 PM   #4
keefaz
LQ Guru
 
Registered: Mar 2004
Distribution: Slackware
Posts: 5,387

Rep: Reputation: 397Reputation: 397Reputation: 397Reputation: 397
Did you consider start wget with --spider option

man wget
Code:
       --spider
           When invoked with this option, Wget will behave as a Web spider,
           which means that it will not download the pages, just check that
           they are there.
The ouput should contain same content as if wget was downloading something, but in some server "content length" is not indicated though

This way you start wget --spider as normal (in foreground), scan output, then wait for it to end

Last edited by keefaz; 08-17-2014 at 12:41 PM.
 
1 members found this post helpful.
Old 08-17-2014, 01:26 PM   #5
bangnagr
Member
 
Registered: Jan 2013
Posts: 48

Original Poster
Rep: Reputation: 4
Quote:
Originally Posted by keefaz View Post
Did you consider start wget with --spider option
No. didn't knew about --spider. Thanks for that. At least it solved wget's size problem.

I usually apply this technique for most of the programs, where they display most of the information on to screen while they are running. That on screen information is gold mine of data. This problem is some what similar that is discussed here and here.

As i'm from different programming background, i'm struggling on how to do them in bash script.

Thanks
 
Old 08-20-2014, 01:26 PM   #6
bangnagr
Member
 
Registered: Jan 2013
Posts: 48

Original Poster
Rep: Reputation: 4
OK, I made some improvements. I come up with this script. This scripting is working, but I’m pretty much sure even though logic 'maybe' correct, there will be better way to do it. Will anybody please tell me how can I improve on this.

Here I’m using wget as example, but I intend to use this logic for any command which does not provide certain information prior, but gives them when it is running.

The logic is as follows:
  1. Create a empty temp file.
  2. Run any command and redirect output to that temp file.
  3. Wait for the required keyword to be written to that temp file.
  4. When written, use those information.
  5. Stop/break/kill the running process (command).
  6. Delete temp file and partial downloaded file.

Code:
#!/bin/bash

touch outputfile

while true ; do
  wget http://releases.ubuntu.com/14.04.1/ubuntu-14.04.1-desktop-amd64.iso -O ubu.iso &> outputfile &
  sleep 1

  if grep -q 'Length' outputfile ; then
    echo "Got it"
    echo "Size: $(awk '/Length/{ print $2 }' outputfile)"
    echo "Type: $(awk '/Length/{ print $4 }' outputfile)"
    break
  fi  
done

killall wget
rm ubu.iso
rm outputfile
Thank you.

Last edited by bangnagr; 08-20-2014 at 01:27 PM.
 
Old 08-28-2014, 08:59 AM   #7
mddnix
Member
 
Registered: Mar 2013
Location: Bangalore, India
Distribution: Redhat, Arch, Ubuntu
Posts: 512

Rep: Reputation: 139Reputation: 139
Here it is

Code:
#!/bin/bash

wget http://releases.ubuntu.com/14.04.1/ubuntu-14.04.1-desktop-amd64.iso -O ubu.iso &> outputfile &
myPID=$!
while true ; do
    if grep -q 'Length' outputfile ; then
        echo "Size: $(awk '/Length/{ print $2 }' outputfile)"
        echo "Type: $(awk '/Length/{ print $4 }' outputfile)"
        break
    fi
done    
disown "$myPID"
kill -9 "$myPID"
rm -f ubu.iso outputfile
 
Old 08-28-2014, 09:02 AM   #8
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 8,122

Rep: Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270Reputation: 2270
do not use kill -9 if not really necessary! a simple kill <pid> is in most cases sufficient.
 
1 members found this post helpful.
Old 08-28-2014, 09:09 AM   #9
mddnix
Member
 
Registered: Mar 2013
Location: Bangalore, India
Distribution: Redhat, Arch, Ubuntu
Posts: 512

Rep: Reputation: 139Reputation: 139
Quote:
Originally Posted by pan64 View Post
do not use kill -9 if not really necessary! a simple kill <pid> is in most cases sufficient.
Thanks.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Trying to understand the shell. Output-redirection into a file markush Programming 3 01-19-2014 03:04 PM
[SOLVED] bash: I/O redirection is being buffered on output to a file kirk5000 Programming 5 06-14-2012 02:38 AM
Output Redirection - Trying to output to screen and file? helptonewbie Linux - Newbie 7 03-19-2009 08:05 AM
output number of blank lines tjgadu Linux - Newbie 7 06-09-2005 05:01 PM
cat: output specific number of lines mikeshn Linux - Software 3 12-31-2003 01:15 PM


All times are GMT -5. The time now is 11:01 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration