LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-14-2015, 03:45 PM   #1
Haba2015
LQ Newbie
 
Registered: Mar 2015
Posts: 10

Rep: Reputation: Disabled
How to filter multiple csv files according to a complete rows


Hi Everyone, I have multiple csv files(>100). They are rain-gauge stations files for precipitation measurement. In these files, the numbers of stations are not equal(i.e. there are missing stations). I want only the stations that are present in all the files. The files have unique station id in column #3. I want to ask if this is possible in Linux?

It may be something along: for h in *.cvs; do sed '?????' $h > rippe_$h && mv rippe_$h $h.xls ; done
 
Old 04-14-2015, 04:27 PM   #2
Lnthink
Member
 
Registered: May 2010
Location: Lafayette, LA
Distribution: Ubuntu, RH, Fedora
Posts: 38

Rep: Reputation: 8
You've got the right idea...

for i in *.cvs
do
cut -f3 -d"," $i
done | sort | uniq

This gives you a unique list of all the items in column 3 in all of the .cvs files in the current directory, assuming that you file delimiters are commas ",".

I just realized - this should also work:
cat *.cvs | cut -f3 -d"," | sort | uniq

As well this should work too:
cut -f3 -d"," *.cvs | sort | uniq

-- note: the uniq command (filters for uniqueness) doesn't work with unsorted data, thus the sort command in front of it.

Hope that these options help get you along in your task,
this should give you the list of all stations in all of the files that have reported/recorded data in this dump set of .cvs files.

Last edited by Lnthink; 04-14-2015 at 04:35 PM.
 
  


Reply

Tags
linux command


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] How to script csv editing? Remove rows from csv file that do not contain certain text ingram87 Linux - Software 9 08-03-2012 12:45 PM
consolidating rows from csv file noony123 Linux - Newbie 3 12-14-2011 07:47 AM
[SOLVED] AWK (or TCL/TK): Matching rows and columns between multiple files Euler2 Programming 6 05-30-2011 06:31 PM
Create 1 csv file from multiple txt files richmur Programming 10 09-03-2008 01:28 PM
Combine output of multiple files in one CSV file say_hi_ravi Programming 4 07-17-2008 03:04 AM


All times are GMT -5. The time now is 12:59 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration