ProgrammingThis forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Hello everyone! I am very excited as I finally started working on some much needed scripts. The one I am currently working on is used to configure /etc/hosts for blocking basically trashware. I was recommended amazing lists over in the security part of LQ and am currently working on the script part. I have this script currently
it's a lot, I know. This is all working 100% fine (takes a bit, but I expected that). Now is that hard part, just a recap of what I have done so far:
1) First I gathered all the lists I need
2) I merge all these files into a temporary file called "tmp.txt"
3) I sorted tmp.txt and removed all the duplicates and put the results into "hosts.tmp"
4) I add a comment for formatting to the top of the file and save it back into "tmp.txt"
What I need to do next is go into hosts and:
1) remove all # and everything that isn't a URL (this will be done before I add the comment)
2) under the ip-adress column insert:
Code:
0.0.0.0
3) move all the URLs under host.name
I am having a hard time doing this part and need some help. If anyone has any docs, samples or even a way to do one of these I would love some help. Thank you so much! I am doing this in generic UNIX script so things like sed -i don't work as sed -i is different in GNU sed then well every other sed.
I got it all working:
Code:
#!/bin/sh
echo "configuring /etc/hosts..."
# download the hosts lists
torsocks wget -q --show-progress -i host-list.txt
# merge all files into a temporary text file
cat * >> tmp.txt
# remove all duplicate hosts and comments
cat tmp.txt | uniq -iu > hosts.tmp
# replace all ips with 0.0.0.0
cat tmp.txt | awk '$1="0.0.0.0"' > hosts.tmp
# add formatting header
sed '1 s/^/#<ip-address> <host-name>/' hosts.tmp > tmp.txt
# move to /etc/hosts
mv tmp.txt /etc/hosts
echo "/etc/hosts configured"
I suggest you use the commands below in regards to your numbered tasks.
I do it from memory and I have not tested, so you better ensure you adapt them to your real needs and environment. man pages exist for a reason.
1) wget your are using, adding cd before to change to a working directory will help as suggested
2) cat * > tmp.txt
3) uniq -iu tmp.txt > tmp_uniq.txt
4) sed -i ... tmp_uniq.txt, please figure out what the ... should be replaced with
1) insert a grep -v '^#' after the uniq above
2) sed -i ... tmp_uniq.txt, with similar comment for ...
And I thought host file had passed its use-by date. I haven't used one in years for blocking like that.
This seems (massively) overly manual. What about something like pihole to handle everything on the LAN ?.
Have a look at how this script does it.
Do not reinvent the wheel (and don't ask us to help you reinvent the wheel).
I am not reinventing the wheel, I am writing a script to do a task and there isn't anything that works for me and my system and is fully portable across ALL UNIX systems, is pure UNIX shell script (not written in perl, python or BASH programming languages) and uses tor. I didn't mean to come across asking you all to solve this for me, I just needed some pointers. I got everything working (need to fix some issues because some of these lists aren't formatted and windows leaves stupid encodings in the files ._.), but thank you for your advice.
And I thought host file had passed its use-by date. I haven't used one in years for blocking like that.
This seems (massively) overly manual. What about something like pihole to handle everything on the LAN ?.
My goal is to see just how much I can do with the bare system and a few tools. It's half annal systemness and half let's see how far we can get with this.
Since you solved it on your own, I'll show you my solution to it.
Code:
#!/usr/bin/env dash
files="$(cat - <<'EOF')"
PUT ALL OF YOUR URLS HERE
EOF
# pipe the URLs to xargs, which will call wget with output silenced.
# pipe the downloaded content to sort -u and store it in a temporary file.
echo "$files" | xargs -I{} wget -q -O - {} | sort -u > tmp.txt
# filter hosts to only include valid hostnames.
sed 's/#.*//' tmp.txt | grep -Eo '([a-zA-Z0-9-]+)(\.[a-zA-Z0-9-]+){1,}' | grep -Ev '[0-9]{1,3}(\.[0-9]{1,3}){3}' > tmp2.txt
# format the file to align urls under host-name and put 0.0.0.0 under ip-address.
sort -u tmp2.txt | sed 's/^/ /' > tmp.txt
printf "<ip-address> <host-name>\n0.0.0.0\n" | cat - tmp.txt > hosts
# remove temporary files.
rm tmp*
Since you solved it on your own, I'll show you my solution to it.
Code:
#!/usr/bin/env dash
files="$(cat - <<'EOF')"
PUT ALL OF YOUR URLS HERE
EOF
# pipe the URLs to xargs, which will call wget with output silenced.
# pipe the downloaded content to sort -u and store it in a temporary file.
echo "$files" | xargs -I{} wget -q -O - {} | sort -u > tmp.txt
# filter hosts to only include valid hostnames.
sed 's/#.*//' tmp.txt | grep -Eo '([a-zA-Z0-9-]+)(\.[a-zA-Z0-9-]+){1,}' | grep -Ev '[0-9]{1,3}(\.[0-9]{1,3}){3}' > tmp2.txt
# format the file to align urls under host-name and put 0.0.0.0 under ip-address.
sort -u tmp2.txt | sed 's/^/ /' > tmp.txt
printf "<ip-address> <host-name>\n0.0.0.0\n" | cat - tmp.txt > hosts
# remove temporary files.
rm tmp*
There are only two things I just can't figure out. I can't for the life of me fix one thing though, some of these files are written on msdos and other machines and thus having annoying encoding at the end (^W) I looked online, but none of the solutions will remove these symbols. For example:
Code:
's/^W//g'
now I know that to sed 's/^W/' means remove all lines starting with W, but I can't find how to remove this symbol. From what I can find it appears to be a carriage return from old msdos encoding.
The other is I need to move all the hostnames under column two (some files have only hostnames and no ip, so when I replace all of column one with "0.0.0.0" it removes a lot of hostnames). I can't seem to find or figure out doing this. Not all of them start with "www" or "http" or any specific string so I can't find all lines with X and move them to Y. How would I go about this?
now I know that to sed 's/^W/' means remove all lines starting with W, but I can't find how to remove this symbol.
When talking about regex, yes, but that's not what you're trying to do here. ^W (or ^M as I'm looking at it in vim) signifies carriage return/line feed (CRLF), which is the line terminator Windows uses. Tr (man tr for more info) can help you, though.
Code:
tr -d '\r'
That means "delete all carriage return characters."
EDIT: For your second problem, that's what sed 's/^/ /' in my script does. It moves all lines 16 spaces to the right. Of course, you'll need to do that before placing the header in the file.
EDIT2: I misread your question. You can insert 0.0.0.0 in the same sed statement, but modify the number of spaces.
Code:
sort -u tmp2.txt | sed 's/^/0.0.0.0 /' > tmp.txt
Last edited by individual; 06-30-2019 at 04:15 PM.
When talking about regex, yes, but that's not what you're trying to do here. ^W (or ^M as I'm looking at it in vim) signifies carriage return/line feed (CRLF), which is the line terminator Windows uses. Tr (man tr for more info) can help you, though.
Code:
tr -d '\r'
That means "delete all carriage return characters."
EDIT: For your second problem, that's what sed 's/^/ /' in my script does. It moves all lines 16 spaces to the right. Of course, you'll need to do that before placing the header in the file.
EDIT2: I misread your question. You can insert 0.0.0.0 in the same sed statement, but modify the number of spaces.
When talking about regex, yes, but that's not what you're trying to do here. ^W (or ^M as I'm looking at it in vim) signifies carriage return/line feed (CRLF), which is the line terminator Windows uses. Tr (man tr for more info) can help you, though.
Code:
tr -d '\r'
That means "delete all carriage return characters."
EDIT: For your second problem, that's what sed 's/^/ /' in my script does. It moves all lines 16 spaces to the right. Of course, you'll need to do that before placing the header in the file.
EDIT2: I misread your question. You can insert 0.0.0.0 in the same sed statement, but modify the number of spaces.
Code:
sort -u tmp2.txt | sed 's/^/0.0.0.0 /' > tmp.txt
I hate to bother you again, but I am having some odd issues I can't understand. The script is now
Code:
#!/bin/sh
echo "configuring /etc/hosts..."
# download the hosts lists
torsocks wget -q --show-progress -i host-list.txt
# merge all files into a temporary text file
cat * > tmp.txt
# filter hosts to only include valid hostnames
sed 's/#.*//' tmp.txt | grep -Eo '([a-zA-Z0-9-]+)(\.[a-zA-Z0-9-]+){1,}' | grep -Ev '[0-9]{1,3}(\.[0-9]{1,3}){3}' > tmp2.txt
# format the file to align urls under host-name and put 0.0.0.0 under ip-address
sort -u tmp2.txt | sed 's/^/ /' > tmp.txt
cat tmp.txt | awk '$1="0.0.0.0"' >> tmp3.txt
# add formatting header
sed '1 s/^/#<ip-address> <host-name>/' tmp2.txt > host
echo "/etc/hosts configured"
and this script works as expected up to
Code:
sort -u tmp2.txt | sed 's/^/ /' > tmp.txt
after that things get weird. I am using
Code:
cat tmp.txt | awk '$1="0.0.0.0"' >> tmp3.txt
to put the ip next to all URLs. This places the ip under <ip-address> (which is column one), however after this is done I cat tmp3.txt and all the hostnames are gone. I need to have the ip next to all hostnames and I am unsure how else to do this. I am also very confused on how this is remove all hostnames. I know I am likely missing something obvious, but do you notice anything which would cause this?
RE: Widows text files and Linux.
Two more options I'm aware of:
1. Do an ascii file transfer in sftp (or ftp). That will change the line-end character to match the destination OS.
(so if transferring from Windows to Linux, the Windows CRLF will be converted to the Linux LF, and vice-versa)
2. Run the file through dos2unix to convert the line-end characters (or unix2dos going the other way)
This is not to say that using tr or other methods already posted aren't good...just tossing out a couple other options.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
