[SOLVED] Heavy Parsing of Text File

individual · 06-30-2019, 06:27 PM

Quote:

Originally Posted by FOSSilized_Daemon

I hate to bother you again, but I am having some odd issues I can't understand. The script is now

Code:

#!/bin/sh

echo "configuring /etc/hosts..."

# download the hosts lists
torsocks wget -q --show-progress -i host-list.txt

# merge all files into a temporary text file
cat * > tmp.txt

# filter hosts to only include valid hostnames
sed 's/#.*//' tmp.txt | grep -Eo '([a-zA-Z0-9-]+)(\.[a-zA-Z0-9-]+){1,}' | grep -Ev '[0-9]{1,3}(\.[0-9]{1,3}){3}' > tmp2.txt

# format the file to align urls under host-name and put 0.0.0.0 under ip-address
sort -u tmp2.txt | sed 's/^/                /' > tmp.txt
cat tmp.txt | awk '$1="0.0.0.0"' >> tmp3.txt

# add formatting header
sed '1 s/^/#<ip-address>    <host-name>/' tmp2.txt > host

echo "/etc/hosts configured"

and this script works as expected up to

Code:

sort -u tmp2.txt | sed 's/^/                /' > tmp.txt

after that things get weird. I am using

Code:

cat tmp.txt | awk '$1="0.0.0.0"' >> tmp3.txt

to put the ip next to all URLs. This places the ip under <ip-address> (which is column one), however after this is done I cat tmp3.txt and all the hostnames are gone. I need to have the ip next to all hostnames and I am unsure how else to do this. I am also very confused on how this is remove all hostnames. I know I am likely missing something obvious, but do you notice anything which would cause this?

EDIT:

Doing

Code:

sort -u tmp2.txt | sed 's/^/0.0.0.0         /' > tmp.txt

doesn't work it just inserts ip spaces ip

Nevermind, I made a mistake all good

Is everything working properly now? The reason the AWK command was doing that is the fact you have leading spaces on the line. By default AWK treats whitespace as the delimiter, so saying $1="0.0.0.0" really means "replace the first non-whitespace column with 0.0.0.0." If you really want to use AWK for that, you'll need to do something like this:

Code:

awk '{ print "0.0.0.0        "$1 }' tmp.txt

Just another tip for good scripting--avoid useless uses of cat. Most if not all UNIX commands accept a file as a command-line argument.

anon033 · 06-30-2019, 08:55 PM

Quote:

Originally Posted by individual

Is everything working properly now? The reason the AWK command was doing that is the fact you have leading spaces on the line. By default AWK treats whitespace as the delimiter, so saying $1="0.0.0.0" really means "replace the first non-whitespace column with 0.0.0.0." If you really want to use AWK for that, you'll need to do something like this:

Code:

awk '{ print "0.0.0.0        "$1 }' tmp.txt

Just another tip for good scripting--avoid useless uses of cat. Most if not all UNIX commands accept a file as a command-line argument.

It's all up and working thank you so much! I have an LQ question though, I am working on an iptables setup and did make one post in Security about it (I am having some design issues) this script and the iptables one are both part of my install and configure scripts: https://gitlab.com/Puffles_the_Drago...er/dur-install what forum would I make a full post about iptables script? The reason for the post is issues with DNS and ssh for my laptop (I have a configuration I just need help with some issues I am trying my best to figure out). Is this a security question or programming? I did ask it in secuirty but only response was saying iptables is old (but the Linux config I am using this for uses a grsecure fork + SELinux extensions). I am still learning how best to ask on LQ and where to ask and just need a point in the right direction. Not to plug the post, the post I did make is here: https://www.linuxquestions.org/quest...es-4175656625/ (I apologize if this is off-topic, but I just don't want to make a wrong post for a section).

ntubski · 07-01-2019, 06:46 AM

Quote:

Originally Posted by FOSSilized_Daemon

I can't for the life of me fix one thing though, some of these files are written on msdos and other machines and thus having annoying encoding at the end (^W) I looked online, but none of the solutions will remove these symbols. For example:

Code:

's/^W//g'

now I know that to sed 's/^W/' means remove all lines starting with W, but I can't find how to remove this symbol. From what I can find it appears to be a carriage return from old msdos encoding.

It sounds like this is an End-of-Transmission-Block character. Which isn't quite the same as the carriage return line-ending stuff. I guess you could use bash ANSI-C Quoting to match it, as in

Code:

sed $'s/\x17//g'

(unless you've misread M as W (upside down monitor?

), in which case it is a carriage return after all)