LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-13-2012, 01:11 AM   #1
soupmagnet
LQ Newbie
 
Registered: Sep 2012
Posts: 27

Rep: Reputation: Disabled
piping gzip into split


I'm working on a shell script for an Android device that continually saves the output of logcat to series of files for later reference. The script as it stands is as so:

Code:
logcat | split -l 1000 - /some_dir/log-
I need to somehow pipe gzip into it but it doesn't seem as though either the shell or Android is responsive to wildcards or regular expressions...or maybe I'm doing it wrong.

If I write the code as so:

Code:
logcat | split -l 1000 - /some_dir/log- | gzip -c /some_dir/log-*
...it returns something to the effect of "gzip: some_dir/log-* : No such file or directory", even though there are files in that directory that match that pattern.

I would like to zip each file being made if possible. What am I missing? Will regular expressions like [^\.gz$] in this situation? (I'm not even sure if that's the correct usage to elliminate files already gzipped).
 
Old 12-13-2012, 04:12 AM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 9,184

Rep: Reputation: 2673Reputation: 2673Reputation: 2673Reputation: 2673Reputation: 2673Reputation: 2673Reputation: 2673Reputation: 2673Reputation: 2673Reputation: 2673Reputation: 2673
the problem with your example is that split will write files, nothing will be piped to gzip.
you will need to first compress and split:
http://stackoverflow.com/questions/1...z-zip-or-bzip2
http://www.linuxquestions.org/questi...-files-347840/
http://techlogbook.wordpress.com/200...iles-in-linux/
or you will need to write a small script to compress files after splitting (on the fly). That would require some perl or similar knowledge
 
Old 12-13-2012, 10:45 AM   #3
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Arch
Posts: 3,131

Rep: Reputation: 1317Reputation: 1317Reputation: 1317Reputation: 1317Reputation: 1317Reputation: 1317Reputation: 1317Reputation: 1317Reputation: 1317Reputation: 1317
If you have GNU split you can use the --filter option:

Code:
logcat | split -l 1000 --filter='gzip > $FILE' - /some_dir/log-
But it's quite likely Android has a smaller version of split that doesn't support that.
 
Old 12-13-2012, 03:36 PM   #4
soupmagnet
LQ Newbie
 
Registered: Sep 2012
Posts: 27

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by ntubski View Post
If you have GNU split you can use the --filter option:

Code:
logcat | split -l 1000 --filter='gzip > $FILE' - /some_dir/log-
But it's quite likely Android has a smaller version of split that doesn't support that.
You're right. Android doesn't seem to support '--filter'


I've decided it's probably best to run a separate script to check the contents of "/some_dir" for anything NOT ending in .gz and zip thos files accordingly, but I'm having trouble with regex.



Let's say, in "/some_dir", files are listed as such:

log-aa.gz
log-ab.gz
log-ac.gz
log-ad
log-ae
log-af

Entering:
Code:
ls *.gz
..will obviously output
log-aa.gz
log-ab.gz
log-ac.gz

But I need to match everything BUT that which contains or ends in ".gz". I know I have to somehow get "negative lookahead" to work, but it doesn't seem to want to. Taking the most simplistic approach (to avoid problems with the "."), I tested a few commands like:

Code:
ls "(?!gz)"
Code:
ls "*(?!gz)"
Code:
ls "(?!gz)$"
Code:
ls "*(?!gz)$"
...all of which return "No such file or directory", including on my laptop (so as to elliminate Android as the culprit).

I understand "plenty" about matching strings with regex, just not inversely. What am I missing? I am pretty confused in the correct usage of negative and positive lookahead and I haven't found much that is either useful or easily understood.


Eventually I would like to include the "." in the pattern because as you can tell, there will eventually be a file saved as "log-gz" which would be matched against the "gz" argument.

Quote:
Originally Posted by pan64
That would require some perl or similar knowledge
As far as I am aware, Android doesn't support Perl so that may be out of the question.

[edit:] Also, the script is logging the output of 'logcat' which is continuously running. Without 'split' the files saved are too large to be displayed in one text file, so zipping the file and then splitting it would be pointless because each file only would have the beginning output of 'logcat' and not the most recent. As it stands now, a file will be created and will filled up until it gets to 1000 lines and creates a new file with each new file containing the most recent data.

Last edited by soupmagnet; 12-13-2012 at 04:00 PM.
 
Old 12-13-2012, 08:08 PM   #5
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,356

Rep: Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367
This seems to work
Code:
cd tmp
for file in *
do
	if [[ $file != *.gz ]]
	then
		gzip $file
	fi
done
Code:
#before
 ls
log-aa.gz  log-ab.gz  log-ac.gz  log-ad  log-ae  log-af  t.t

#After
 ls
log-aa.gz  log-ab.gz  log-ac.gz  log-ad.gz  log-ae.gz  log-af.gz  t.t.gz
 
1 members found this post helpful.
Old 12-13-2012, 10:45 PM   #6
soupmagnet
LQ Newbie
 
Registered: Sep 2012
Posts: 27

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by chrism01 View Post
This seems to work
Code:
cd tmp
for file in *
do
	if [[ $file != *.gz ]]
	then
		gzip $file
	fi
done
Code:
#before
 ls
log-aa.gz  log-ab.gz  log-ac.gz  log-ad  log-ae  log-af  t.t

#After
 ls
log-aa.gz  log-ab.gz  log-ac.gz  log-ad.gz  log-ae.gz  log-af.gz  t.t.gz
I tried using a 'for' loop but it never worked for me (improper use of variables I'm guessing).

Regardless, I worked out something that could be done all in one script.

Code:
function1(){

while true; do

    logcat | split -l 1000 /some_dir/log-

done

}

function2(){

while true; do

    var=`find /some_dir -ipath "*-??"`

    sleep 1           #to avoid an error when the script starts
    gzip $var
    sleep 3599        #script will check every hour for unzipped files

done

}

function1 | function2
So far it's working exactly as it should. Thanks anyway for everyone's help.

Last edited by soupmagnet; 12-14-2012 at 07:36 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Split a gzip file ZAMO Linux - Server 6 08-21-2012 10:30 AM
split a gzip file based on size and have a prefix without changing extension gubbu Linux - Newbie 3 01-18-2010 05:09 PM
[gzip] appears in process table after gzip command completes redmanDBA Linux - General 0 02-26-2008 06:12 AM
Piping dd through gzip to image Win2k Partition sr_25 Linux - Newbie 2 07-08-2007 02:43 PM
Zip/Gzip "split" archives? Megamieuwsel Linux - General 2 01-10-2004 09:59 AM


All times are GMT -5. The time now is 09:22 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration