2 files diff help

2byt3s · 04-21-2020, 02:01 PM

I need some help with a bash script to do the following. I can do simple stuff in bash but this one is beyond me.

I have 2 files, one has a set of domains and urls only such as
www.google.com
www.fakeurl.com/xmltcs/test.php

the other contains another list some of the urls match but are not in the same order and are commented using the # to notate not to be processed:

www.google.com # comment not to be processed
www.wikihow.com/index.html # whatever goes here

I need to parse the two and merge them, removing any duplicates but keeping the comments if there are any.

Any ideas?

Turbocapitalist · 04-21-2020, 02:15 PM

I would use join if you're not going to escalate to perl. The input files would need to be sorted first for that. See the options -j and -a in "man join".

An advanced approach would be to use process substition to sort the files on the way in to join without changing anything on the disk.

syg00 · 04-21-2020, 06:45 PM

Might as well remove the not to be processed lines with something like "grep -v" - save some post-processing.

2byt3s · 04-22-2020, 01:59 PM

I wish I could remove them, however it is a requirement to meet a certain standard that was adopted.

ondoho · 04-22-2020, 02:08 PM

bash:

read the file in line by line
remove comments like so: ‘line=“${line%%#*}”’

edit:
probably best if you show us what you wrote so far.

syg00 · 04-22-2020, 06:29 PM

Quote:

Originally Posted by 2byt3s

I wish I could remove them, however it is a requirement to meet a certain standard that was adopted.

I was suggesting the removal be done along with the sort (piping) in the process substitution suggested above. That only affects the data your script sees, not the actual source data.

scasey · 04-22-2020, 08:05 PM

Quote:

Originally Posted by ondoho

probably best if you show us what you wrote so far.

Agreed. The OP needs to show us what they’ve tried..