LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - General (https://www.linuxquestions.org/questions/linux-general-1/)
-   -   How to remove forward slash and everything after with sed or awk? (https://www.linuxquestions.org/questions/linux-general-1/how-to-remove-forward-slash-and-everything-after-with-sed-or-awk-4175455309/)

CaptainDerp 03-23-2013 07:09 PM
How to remove forward slash and everything after with sed or awk?
 
Ok guys, Ive got a list of urls and I need to parse just the domains from them and get rid of all the trailing crap

Ive got a list of these lines like this

somebullshit.com/some/crap/i/want/to/remove
someothershit.biz/crap/poop.html
random.ru/shiz/myboody.php
moar.ch/caca.html
someotherstuff.pl/blah/blah/blah

And I need to end up with

somebullshit.com
someothershit.biz
random.ru
moar.ch
someotherstuff.pl

Also this is a massive list of over 2 million urls, so, its not limited to .com, .ru. .ch, .pl, and .biz. So it needs to implicitly remove the first firward slash and everything after it on each line

Thank you to the wizard who knows the answer in advance.

colucix 03-23-2013 08:06 PM
Code:
sed 's:/.*::' file

freebsd_Rules_All_OSes 03-23-2013 08:10 PM
If your list is formatted with no http:// at the beginning then this should work.
Code:
sed 's/\/.*//g' file_with_urls
This will give you a preview of the output. If you are happy with the results then redirect the output to another file.

Code:
sed 's/\/.*//g' file_with_urls > output_file
or

Code:
sed -i 's/\/.*//g' file_with_urls
The -i will overwrite the original file with the new changes

CaptainDerp 03-23-2013 09:04 PM
sed 's:/.*::' file

Did exactly what I needed

also I didnt know about the -i option thats awesome,

thanks guys.


Hmm I have another more complicated one, any care to have a swing at it?


Ok, so this huge list of sites is like this.

laui.somesite.com
lau-immobilien.de
laurapausini.fanspace.it
lauraroebuck.com
laurenserect.ru
laurentianbankz.ca
laurianoalmeida.sites.uol.com.br
lavasoftupdate.com
lavl-vicky.com
lavvckpordclbduy.ru

I need a way to remove subdomains, but without destroying items like .co.uk,

So I figure I need to remove anything before .*.* but only on lines that do not contain multiple entries like .co.uk and .co.nz

The whole idea is to have a list of top level domains. But retaining .co.nz, and .co.uk and all similar extensions

chrism01 03-25-2013 01:15 AM
Unfortunately, I think you'll need a list of all 2 level domain codes to keep ie your program needs to know if eg '.com.br' is a TLD (ie only want .br) or an allowable 2 level one?
Similarly, given .it is ccTLD for italy, is there an equiv to .co.uk eg .co.it?
For tha I'd use Perl, although others might use awk.
I think sed would be a stretch for this more general problem.

This is a problem where the code needs to know stuff, as opposed to just truncating at a known marker.


All times are GMT -5. The time now is 02:13 AM.