LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 11-14-2021, 12:37 PM   #1
sean mckinney
LQ Newbie
 
Registered: Mar 2004
Posts: 26

Rep: Reputation: 0
tail help


As an addition to what I was doing / am doing in https://www.linuxquestions.org/quest...54#post6299554

I want to output the last 20 lines of all the csv's in one folder to a csv in another folder.

I think

tail -n 20 *.csv > folder2/last-20s.csv

works beautifully, I am currently using a windows computer and can't check the actual command I wrote, hence "I think".

But, in the process of writing each qualifying line to last-20s.csv, I would like to, if possible, add
1) the count for total number of lines in all the csv's processed to that instant.
2) the count for number of lines processed to that instant in the individual csv being processed.
3) the name of the individual csv being processed.

If the terminology is applicable to "tail" the above might amount to NR, FNR and FILENAME.
If relevant and necessary, I would be using awk or gawk as it/they is/are the only programming language/s with which I have any familiarity.
For reference there are 2000+ csv's being scanned with a total of well over 4,000,000 lines.
Any help would be gratefully received.

Last edited by sean mckinney; 11-14-2021 at 12:47 PM.
 
Old 11-14-2021, 01:33 PM   #2
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 3,037

Rep: Reputation: Disabled
Quote:
Originally Posted by sean mckinney View Post
I think

tail -n 20 *.csv > folder2/last-20s.csv

works beautifully
The result won't be a CSV anymore unless you also specify -q, because tail prints names of processed files by default:
Code:
$ tail -n3 <(seq 5) <(seq 20)
==> /dev/fd/63 <==
3
4
5

==> /dev/fd/62 <==
18
19
20
Code:
$ tail -qn3 <(seq 5) <(seq 20)
3
4
5
18
19
20
Quote:
If relevant and necessary, I would be using awk or gawk as it/they is/are the only programming language/s with which I have any familiarity.
Yes, I guess using awk is more reasonable in this case considering your requirements. Although emulating tail with awk is rather inefficient.

With gawk, I'd try something like
Code:
gawk -vOFS=, 'ENDFILE{print NR,FNR,FILENAME; print}' *.csv
The second print could be replaced with something different, e.g. with
Code:
system("tail -n20 " FILENAME)
But then again, miller has both tail and the awk-like variables.

Last edited by shruggy; 11-14-2021 at 02:31 PM.
 
Old 11-14-2021, 03:03 PM   #3
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,971
Blog Entries: 3

Rep: Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102
Perl might do the job.

Code:
perl -a -n -e '
        BEGIN {
               $file=$ARGV;
        } 

        if($file ne $ARGV) { 
               print "\n$file Count: $c\n"; 
               print(join("\n", @a),"\n"); 
               $file=$ARGV; 
               $c=0;
        } 
        $c++; 
        push(@a, $F[0]); 
        shift(@a) if($#a>=20);  

        END { 
               print "\n$file Count: $c\n"; 
               print(join("\n", @a),"\n");
        }
' *.csv
Thus it can be made to work a little like AWK, but with more flexibility.

Last edited by Turbocapitalist; 11-14-2021 at 03:19 PM. Reason: off-by-one
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
tail the output of tail -n 1 raj k yadav Linux - Newbie 5 02-07-2010 12:26 AM
How to make newer "tail" behave like older "tail" rylan76 Linux - Software 4 12-07-2007 05:27 AM
tail multiple files not working in AIX - Help atmguy AIX 4 05-31-2004 11:22 PM
Help with 'tail' command! Beppe83 Linux - Software 3 04-07-2004 09:11 AM
root-tail -help Streams Linux - Software 2 04-26-2003 05:16 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 12:00 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration