LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-05-2013, 01:49 AM   #1
ibarra
LQ Newbie
 
Registered: Apr 2013
Posts: 5

Rep: Reputation: Disabled
Using awk command to select date range to ftp


hi.. just joined in.
im trying to ftp log files from remote host to my local machine. before i am using winscp, however, just recently i can no longer use it, says that the files in the directory are too long to list. so, i thought of having a simple script using awk to get what was just being requested to be ftp'd.

my objective is: select the range by month (e.g Dec 2012 - Feb 2013), move it a temp directory, then ftp the entire directory to the local machine.

as of to date, i am stuck with just printing the file of only one month.

#ls -lrt *.log | awk '/Feb/ {print $6 $7 $8 $9}'

your inputs are highly appreciated
 
Old 04-05-2013, 02:54 AM   #2
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,241

Rep: Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325
Code:
ls -lrt *.log | awk '/Feb/ {print  $9}'

BTW, as far as GUIs go, have you tried Filezilla?
 
Old 04-05-2013, 03:01 AM   #3
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,256

Rep: Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686
My main concern is the use of processing ls output as this can have issues, especially if there is white space in any of the file names.

You mention you wish to move the files to a new location and then upload the directory so I am not sure why you are returning multiple fields?
May I suggest you look at using find and you can research either using the name format to retrieve your files and / or the creation / modification times.
 
Old 04-05-2013, 03:38 AM   #4
ibarra
LQ Newbie
 
Registered: Apr 2013
Posts: 5

Original Poster
Rep: Reputation: Disabled
@chrism01: no haven't tried it.

@grail: i have included the other fields so i could see the time-stamps, since i was only able to print the files. can you suggest a sample script using find? i am just wondering why would i use find when i already know where are the target files to be ftp'd. besides awk can be more useful on filtering specific fields.
 
Old 04-05-2013, 04:22 AM   #5
ibarra
LQ Newbie
 
Registered: Apr 2013
Posts: 5

Original Poster
Rep: Reputation: Disabled
how i will be able to set the output of the awk command to a variable.
like: a='awk output'; then cp a /home/ or something of the sort.
 
Old 04-05-2013, 05:10 AM   #6
ibarra
LQ Newbie
 
Registered: Apr 2013
Posts: 5

Original Poster
Rep: Reputation: Disabled
this is my progress (without progressing )


ls -lrt | awk '/Feb/ {print $6" "$7" "$8" "$9}' | xargs -I -t cp {} test
cp: cannot stat `{}': No such file or directory
cp: cannot stat `{}': No such file or directory
cp: cannot stat `{}': No such file or directory
cp: cannot stat `{}': No such file or directory
cp: cannot stat `{}': No such file or directory
 
Old 04-15-2013, 03:21 AM   #7
ibarra
LQ Newbie
 
Registered: Apr 2013
Posts: 5

Original Poster
Rep: Reputation: Disabled
input from other members will be highly appreciated.
looking forward to have this issue solved. thanks
 
Old 04-15-2013, 08:49 AM   #8
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,241

Rep: Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325
As I pointed out, you only need $9 ie the filename. It doesn't make sense to 'cp' the other fields ls -lrt returns.
 
Old 04-15-2013, 09:37 AM   #9
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,604

Rep: Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241
The "-I" option to xargs needs a parameter --- Add {} to it so that xargs knows where the substitution is to take place.
 
Old 04-15-2013, 09:41 AM   #10
shivaa
Senior Member
 
Registered: Jul 2012
Location: Grenoble, Fr.
Distribution: Sun Solaris, RHEL, Ubuntu, Debian 6.0
Posts: 1,800
Blog Entries: 4

Rep: Reputation: 286Reputation: 286Reputation: 286
Post

Can you try find cmd:
Code:
~$ find .  -name "*.log" -mtime -150 -mtime +60 -exec cp '{}' /path/to/dir \;

Last edited by shivaa; 04-15-2013 at 09:42 AM.
 
1 members found this post helpful.
Old 04-15-2013, 10:14 AM   #11
millgates
Member
 
Registered: Feb 2009
Location: 192.168.x.x
Distribution: Slackware
Posts: 840

Rep: Reputation: 380Reputation: 380Reputation: 380Reputation: 380
or, if you want to search by date,

Code:
touch -t 201202010000 /tmp/min
touch -t 201304010000 /tmp/max
find . -maxdepth 1 -mindepth 1 -newer /tmp/min ! -newer /tmp/max -exec cp -t dest/dir {} +
 
Old 04-18-2013, 02:26 PM   #12
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
I second grail's suggestion on avoiding parsing ls. There are almost always better techniques available for what you want to do.

What do the filenames look like, exactly? If the names have dates included in them, then it shouldn't be too hard to parse out the ones you want, either with find or shell globbing.

It would be even better if the dates are in the ISO 8601 standard structure (non-hyphenated is best).

On the other hand, if you need to locate the files by mtime, find is definitely the way to go, as shivva and millgates have demonstrated. You shouldn't need to use xargs then, just the built-in -exec action.

Speaking of which, if you have gnu find available, check out the -newerXY option, which lets you bypass creating a reference file first:

Code:
find . -maxdepth 1 -mindepth 1 -newermt 201202010000 ! -newermt 201304010000 -exec cp -t dest/dir {} +
 
Old 04-18-2013, 02:41 PM   #13
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,029

Rep: Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845Reputation: 845
also, instead of using the antiquated insecure and un-scriptable ftp command, you can backtick the output of the previous command into scp and cron it whenever you like:
Code:
scp `find . -newermt 201202010000 ! -newermt 201304010000` user@host:/whatever/floats/your/boat

Last edited by schneidz; 04-18-2013 at 02:44 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Can tar do a date range? ghughes5669 Linux - Software 5 10-25-2012 06:03 PM
[SOLVED] Converting the contents of a column with the output of the date command using awk mystupidquestion Programming 4 10-02-2011 05:22 PM
Awk script to select range from file jeesun Linux - General 8 11-26-2009 05:46 AM
How to select TCP_Ports Range sachinsharma10 Linux - Networking 4 09-06-2007 06:14 AM
tar a date range rayia Programming 3 01-03-2006 08:52 AM


All times are GMT -5. The time now is 08:51 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration