Script help required

davidcummings · 07-16-2014, 08:47 AM

Hi,
I am new to the Linux scene and I am using a raspberry Pi to log air traffic radar. I was wondering if someone could help me with a bit of script code?

Code:

 
#!/bin/bash
/bin/mv /root/dump1090/radar.txt /root/logs/in/radar.txt
/bin/touch /root/dump1090/radar.txt
/bin/awk '!_[$1]++' /root/logs/in/radar.txt > /root/logs/in/processed.txt
/bin/rm /root/logs/in/radar.txt
/bin/mv /root/logs/in/processed.txt /root/logs/out/"radar-`date +%Y%m%d`.txt"
/sbin/restart

my problem is here :

Code:

/bin/awk '!_[$1]++' /root/logs/in/radar.txt > /root/logs/in/processed.txt

what I want it to do is go through the radar.txt file, remove any duplicates and save out the result as processed.txt

before i try and run it is the above syntax correct?

I would also like the file to have a date in the format of yyyy-mm-dd added to the end of each line after a comma:

current:
gf621a
83hfs8

wanted:
gf621a,2014-07-16
83hfs8,2014-07-16

and the file saved out as CSV. is this possible?

rtmistler · 07-16-2014, 03:13 PM

Quote:

Originally Posted by davidcummings

I am new to the Linux scene and I am using a raspberry Pi to log air traffic radar.

Welcome to LQ! You may wish to ask that this thread be moved to the Programming forum for some better exposure; however moderators may also make that determination on their own. Either case, people will still see it so don't be too overly concerned.

Quote:

Originally Posted by davidcummings

Code:

/bin/awk '!_[$1]++' /root/logs/in/radar.txt > /root/logs/in/processed.txt

what I want it to do is go through the radar.txt file, remove any duplicates and save out the result as processed.txt

before i try and run it is the above syntax correct?

Correct by my reckoning. I'm not too familiar with awk; however I created a text file containing the following:

Code:

abc
abc
123
456

I then ran your awk command against it and the result was one of the duplicate "abc" lines was removed. Quick Edit Update: To go further, I thought since me not knowing awk too well that I'd also add a second duplicate, and therefore added another line "123" and it removed both duplicates. So I feel the command as you have it scripted will work properly.

Quote:

Originally Posted by davidcummings

I would also like the file to have a date in the format of yyyy-mm-dd added to the end of each line after a comma:

I believe the format you chose is fine, you may also use:

Code:

date +%F

And if you wish a COMMA versus a DASH, then you need to change "radar-" to say "radar,". Personally I prefer the DASH because I think COMMA in a file name is not too great.

Quote:

Originally Posted by davidcummings

and the file saved out as CSV. is this possible?

The files are in ... some form of record type? You have performed an awk call to remove duplicate entries, similarly you can use awk, sed, or something to edit entries to place them in CSV form. I would then instead of using a .txt extension, use a .csv extension. The matter there is how difficult it is, but as far as processing a record, you can do this. If you show an illustration of a record and the desired way you'd change it to be comma separated, then people can suggest a command form which will do this.

suicidaleggroll · 07-16-2014, 05:29 PM

One suggestion, from a maintainability point of view, is to drop all of the hard coded path names and use variables instead. For example:

Code:

#!/bin/bash
/bin/mv /root/dump1090/radar.txt /root/logs/in/radar.txt
/bin/touch /root/dump1090/radar.txt
/bin/awk '!_[$1]++' /root/logs/in/radar.txt > /root/logs/in/processed.txt
/bin/rm /root/logs/in/radar.txt
/bin/mv /root/logs/in/processed.txt /root/logs/out/"radar-`date +%Y%m%d`.txt"
/sbin/restart

becomes

Code:

#!/bin/bash
start_dir=/root/dump1090
in_dir=/root/logs/in
out_dir=/root/logs/out

/bin/mv ${start_dir}/radar.txt ${in_dir}/radar.txt
/bin/touch ${start_dir}/radar.txt
/bin/awk '!_[$1]++' ${in_dir}/radar.txt > ${in_dir}/processed.txt
/bin/rm ${in_dir}/radar.txt
/bin/mv ${in_dir}/processed.txt ${out_dir}/"radar-`date +%Y%m%d`.txt"
/sbin/restart

That way in the future, when you want to move something (and you probably will), you can do it in one place instead of five.

Also, you're using a lot of intermediate files, and I'm not really seeing why. Couldn't you just do:

Code:

#!/bin/bash
start_dir=/root/dump1090
out_dir=/root/logs/out

/bin/awk '!_[$1]++' ${start_dir}/radar.txt > ${out_dir}/"radar-`date +%Y%m%d`.txt"
/bin/rm ${start_dir}/radar.txt
/bin/touch ${start_dir}/radar.txt
/sbin/restart

Also, you're using a lot of absolute path names to standard executables. Are you planning on running this script through a cron job? Even then, /bin is usually in the default cron PATH and can be left off. Or if you're worried about it, you can modify the PATH directly in the script:

Code:

#!/bin/bash
export PATH=$PATH:/bin:/sbin
start_dir=/root/dump1090
out_dir=/root/logs/out

awk '!_[$1]++' ${start_dir}/radar.txt > ${out_dir}/"radar-`date +%Y%m%d`.txt"
rm ${start_dir}/radar.txt
touch ${start_dir}/radar.txt
restart

TenTenths · 07-17-2014, 02:02 AM

Also take a look at

Code:

man uniq

for dropping duplicates in files.
Using this as a test file:

Code:

abc
abc
123
456
abc

Code:

uniq test.txt

abc
123
456
abc

Code:

uniq -u test.txt

123
456
abc

However the more you learn about awk the better!